Matthew Zyskowski 1 Quanyan Zhu 2

Transcription

1 Matthew Zyskowski 1 Quanyan Zhu 2 1 Decision Science, Credit Risk Office Barclaycard US 2 Department of Electrical Engineering Princeton University

2 Outline

3 Outline

4 I Modern control theory with game-theoretic formalizations permits the design of decentralized versus centralized control mechanisms under both deterministic and stochastic dynamics. I From this arises the need to quantify efficiency loss of decentralized mechanisms, and also provide criteria for designing efficient mechanism. I The Price of Anarchy (PoA) and Price of Information (PoI) measures quantify efficiency loss, and both have been successfully used for control design. I This research proposes a new measure for efficiency loss, the Variance of Anarchy (VoA), inspired by PoA and PoI. I This research proposes a mixed objective PoA-VoA optimization design method with supporting simulation results for queuing server problems.

5 Outline

6 I Players i and j send data packets to the queue at rate d i and d j, respectively. I Players i and j are serviced by the queue at the rate w i s r and w j s r, respectively. I Service rate s r is chosen in order to regulate the queue length q l at a certain level. I Assume each user assigned a fixed proportion of total available bandwidth: P i w i = 1. I Assume that users have perfect measurement of s(t), but occasionally differ from allotted bandwidth due to fluctuations.

7 (cont.) I Let d i (t) denote the rate of source i at time t, and introduce u i (t) :=d i (t) w i s r (t) as control (action) variable of source i I Queue build-up is governed by the differential equation: NX q l (t) = u i (t), (1) i=1 I Assume queue is relatively tightly controlled so that bottleneck queue size stays around some desired level q l I Consider the shifted variable x(t) :=q l (t) q l, which satisfies the following differential equation which is the shifted version of (1): ẋ(t) = NX u i (t) x(0) =x 0. (2) i=1

8 (cont.) I Consider stochastic version of original model with additive white noise for random network phenomena, such as dropped packets and demand surges dx = NX u i (t)dt + dw(t), x 0 = E P0 {x(t 0 )} (3) i=1 I Above x 2 R n, u i 2 R n, and w(t) is one-dimensional Wiener process with correlation of increments E[(w( 1 ) w( 2 ))(w( 1 ) w( 2 )) T ]=W 1 2. I Use (3) to motivate two particular problems: the noncooperative problem and the team (or social welfare) problem.

9 - Optimal Control I Consider the scalar stochastic differential equations given by dx = 1 N NX u i (t)dt + dw(t), x 0 = E P0 {x(t 0 )}, x 2 R. i=1 (4) I Suppose each network user chooses his or her demand u i to optimize the statistical characterization of a random cost below. Z tf 1 J i (u i )= N x(t) u i (t) 2 dt + s i (x(t c f )) i t 0 (5) I Assume that n = 1, N = 2, s 1 (x) =s 2 (x) =x 2, W = 1, and c 1 = c 2 = 2.

10 - Cumulants I Suppose that control inputs are linear, state-feedback u i (t) =k i (t)x(t), k i (t) 2 R, t 2 [t 0, t f ], i = 1, 2. (6) I Assumptions make problem fit LQG framework, so E{J j x(t 0 )=x 0 } = apple j 1 ( ) =hj 1 ( )x d j 1 ( ), Var{J j x(t 0 )=x 0 } = apple j 2 ( ) =hj 2 ( )x d j 2 ( ). (7) I Here h j i ( ), d j i ( ) for j = 1, 2 and i = 1, 2 satisfy dh j 1 ( ) d = (k 1( )+k 2 ( ))h j 1 ( ) 1 2 (k j( )) = f 1(h j, k j ), dh j 2 ( ) d = (k 1( )+k 2 ( ))h j 2 ( ) 4(hj 1 ( ))2 = f 2 (h j, k j ), d d d j 1 ( ) =hj 1 ( ), d d d j 2 ( ) =hj 2 ( ), 2 [t 0, t f ], h j 1 (t f )=1, h j 2 (t f )=0, d j 1 (t f )=0, d j 2 (t f )=1. (8)

11 - Cumulants I Suppose that control inputs are linear, state-feedback u i (t) =k i (t)x(t), k i (t) 2 R, t 2 [t 0, t f ], i = 1, 2. (9) I Assumptions make problem fit LQG framework, so E{J j x(t 0 )=x 0 } = apple j 1 ( ) =hj 1 ( )x d j 1 ( ), Var{J j x(t 0 )=x 0 } = apple j 2 ( ) =hj 2 ( )x d j 2 ( ). (10) I Here h j i ( ), d j i ( ) for j = 1, 2 and i = 1, 2 satisfy dh j 1 ( ) d = (k 1( )+k 2 ( ))h j 1 ( ) 1 2 (k j( )) = f 1(h j, k j ), dh j 2 ( ) d = (k 1( )+k 2 ( ))h j 2 ( ) 4(hj 1 ( ))2 = f 2 (h j, k j ), d d d j 1 ( ) =hj 1 ( ), d d d j 2 ( ) =hj 2 ( ), 2 [t 0, t f ], h j 1 (t f )=1, h j 2 (t f )=0, d j 1 (t f )=0, d j 2 (t f )=1. (11)

12 - Target Cumulants I The LQG optimal control problem minimizes the objective function E{J}. The resulting cumulants are denoted as apple j i,lqg under the control k j LQG. I The 2CC optimal control problem minimizes the objective function E{J} + µ Var{J}. The resulting cumulants are denoted as apple j i,2cc under the control k j 2CC. I For well-posedness, µ>0 is required.

13 - Target Cumulants I Let the target cost cumulants for players j = 1, 2 with 0 apple apple 1 be apple j i (, ) =(1 ) applej i,lqg ( )+ applej i,2cc ( ), (12) I The quantities apple j i,lqg ( ) and applej i,2cc ( ) are given by apple j i,lqg ( ) = h j i,lqg ( )x d j i,lqg ( ), apple j i,2cc ( ) = h j i,2cc ( )x d j i,2cc ( ), i = 1, 2, I The quantities h j j i,lqg ( ), d dh j 1,LQG d i,lqg ( ) are determined by = f 1 (h j LQG, k j LQG ), dhj 2,LQG d = f 2 (h j LQG, k j LQG ) I The quantities h j j i,2cc ( ), d i,2cc ( ) are determined by dh j 1,2CC d = f 1 (h j 2CC, k j 2CC ), dhj 2,2CC d = f 2 (h j 2CC, k j 2CC )

14 - I Define normalized variates Z j, Z j as Z j = J j apple j 1 (t 0) apple j 2 (t 0), Zj = J j j apple 1 (t 0, ) 1/2 apple j 2 (t = a j Z j + b j 0, ) 1/2 Z j = apple j 2 (t 0)! 1/2 apple j 2 (t 0, ) {z } a j J applej 1 (t 0) apple j 2 (t 0) 1/2 {z } Z j + applej 1 (t 0) apple j 1 (t 0, ) apple j 2 (t 0, ) 1/2 {z } b j I Let p Zj and p Zj represent best Gaussian density approximations, p Zj (z) 1 p 2 exp z 2 2, p Zj ( z) a j p Zj (a j z + b j ). I The Kullback-Leibler divergence between density approximations can be expressed as convex function of cumulants and targets KLD(p Zj (z), p Zj (z)) = g " apple j 1 (t 0) apple j 2 (t 0) #, " apple j 1 (t 0, ) apple j 2 (t 0, ) #!.

15 - I Consider the mean-variance cost density-shaping optimization problem for Player j = 1, 2, min j apple {KLD(p 1,applej Zj (z), (z))} p Zj 2 subject to: (eq. of motion for cumulants/targets). (13) I The control solution for Player j = 1, 2 is given by kj (, ) = 2 h j 1 ( )!h j2 ( )/@apple j 1 ( ) ( ). j 2 I This control input to the system ensures the cost s density is closest to the target density among linear controls.

16 - Optimal Control I Consider the scalar stochastic differential equations given by dx T = u(t)dt + dw(t), x 0 = E P0 {x T (t 0 )}. (15) I Suppose each network user chooses his or her demand u i to optimize the statistical characterization of a random cost below. Z tf J T (u) = x T (t) 2 + c 1 + c 2 u(t) 2 dt + s(x T (t c 1 c f )). 2 t 0 (16) I Assume that n = 1, N = 2, s(x) =2x 2, c 1 = c 2 = 2, and W = 1

17 - Cumulants I Suppose that control inputs are linear, state-feedback u(t) =k(t)x T (t), t 2 [t 0, t f ]. (17) I Assumptions make problem fit LQG framework, so E{J T x(t 0 )=x 0 } = apple T 1 ( ) =ht 1 ( )x d T 1 ( ), Var{J T x(t 0 )=x 0 } = apple T 2 ( ) =ht 2 ( )x d T 2 ( ), (18) I Here h T i ( ), d T i ( ) satisfy dh1 T ( ) = 2k( )h1 T ( ) (k( )) 2 1 = f 1 (h T, k T ), d dh2 T ( ) = 2k( )h2 T ( ) 4(h1 T ( )) 2 = f 2 (h T, k T ), d d d d 1 T ( ) =h1 T d ( ), d d 2 T ( ) =h2 T ( ), 2 [t 0, t f ], h1 T (t f )=1, h2 T (t f )=0, d1 T (t f )=0, d2 T (t f )=1. (19)

18 - Target Cumulants I Let the team s target cost cumulants with 0 apple apple 1 be apple T i (, ) =(1 ) apple T i,lqg ( )+ applet i,2cc ( ) (20) I The quantities apple T i,lqg ( ) and applet i,2cc ( ) are given by apple T i,lqg ( ) = h T i,lqg ( )x d T i,lqg ( ), apple T i,2cc ( ) = h i,2cc T ( )x d i,2cc T ( ), i = 1, 2, I The quantities h i,lqg T ( ), d T dh T 1,LQG d i,lqg ( ) are determined by = f 1 (h T LQG, k T LQG ), dht 2,LQG d = f 2 (h T LQG, k T LQG ) I The quantities h T i,2cc dh T 1,2CC d ( ), d T i,2cc ( ) are determined by = f 1 (h T 2CC, k T 2CC ), dht 2,2CC d = f 2 (h T 2CC, k T 2CC )

19 - I Define normalized variates Z T, Z T as Z T = J T apple T 1 (t 0) apple T 2 (t 0), ZT = J T apple T 1 (t 0, ) 1/2 apple T 2 (t = a 0, ) 1/2 T Z T + b T apple T 1/2 Z T = 2 (t 0 ) apple T 2 (t J applet 1 (t 0) 0, ) apple T 2 {z } (t + applet 1 (t 0) apple T 1 (t 0, ) 0) 1/2 apple T 2 {z } (t 0, ) 1/2 {z } a T Z T b T I Let p ZT and represent best Gaussian density p ZT approximations, p ZT (z) p 1 z 2 exp, ( z) a p ZT T p ZT (a T z + b T ). 2 2 I The Kullback-Leibler divergence between density approximations can be expressed as convex function of cumulants and targets KLD(p ZT (z), p ZT (z)) = g apple apple T 1 (t 0 ) apple T 2 (t 0) apple apple T, 1 (t 0, ) apple T 2 (t 0, ).

20 - I Consider the mean-variance cost density-shaping optimization problem for the team, min apple T 1,apple T 2 {KLD(p ZT (z), p ZT (z))} subject to: (eq. of motion for cumulants/targets). (21) I The control solution for team is given by kt (, ) = 2 h T 1 ( )!h T2 ( )/@apple T 1 ( ) ( T 2 (22) I This control input to the system ensures the cost s density is closest to the target density among linear controls

21 Outline

22 of Anarchy I Price of Anarchy (PoA) is a measure of efficiency loss in a control problem due to noncooperation; it can be defined as PoA, E P JT {J T (u )} P N i=1 E P {J (23) Ji i(ui )}. I PoA can be viewed as ratio of team cost s mean to sum of means of player costs. This measure should be bounded below and above, 0 < PoA < 1. I Using higher-order statistics, define the Variance of Anarchy (VoA) measure as VoA, Var P JT {J T (u )} P N i=1 Var P {J (24) Ji i(ui )}.

23 PoA-VoA Optimization I Given framework for noncooperative and team (social welfare) optimal control problems, consider the optimization min 2[0,1] PoA( ) 1 + VoA( ) 1 subject to: (eq. of motion for cumulants/targets - noncooperative), (eq. of motion for cumulants/targets - team) (25) I Choosing controls per (25) will yield noncooperative control solutions for Players 1 and 2 that minimize efficiency loss due to noncooperation while maintaining a high level of confidence in the PoA metric.

24 Outline

25 Step 1: Solve PoA-VoA Optimization I With 2CC and LQG, the equations of motion (13) and (21) can be solved iteratively for 2 [0, 1] to solve PoA-VoA Optimization; = 0.1 is optimal. PoA( ) 1 + VoA( ) * =

26 Step 2: Simulate Team and Non-Coop I Team versus noncooperative controls differ significantly over [0, 100 sec] under 2CC control I RMS measures show that 2CC is approximately 302% of I PoA-VoA Control seems to have minimized efficiency loss and its statistical variability x *(t) x 2CC (t) t (sec) NonCooperative Team t (sec)

27 Outline

28 I Introduced VoA measure to quantify a level of confidence in PoA metrics for efficiency loss in dynamic games due to decentralized mechanisms. I Proposed a new design method, the PoA-VoA optimization, for decentralized control algorithms using both measures. I Simulation results support that this design method is viable, however other criteria might be included (e.g. robust stability). I Work completed under finite-horizon, full-observation assumptions for 2 players. I Next steps include pursuing formalizations for the N player case. I Next steps might include investigation of infinite-horizon and partial observation settings.