Final Project Report for EE599 (Special Topics: Decision Making in Networked Systems)

Transcription

1 Final Project Report for EE599 (Special Topics: Decision Making in Networked Systems) Project Title: Applications of Dynamic Games in Queues Shivasadat Navabi Sohi This report was prepared for the Fall-2014 EE599 class project (Due Date 12/10/2014). Fall 2014 Abstract Queuing theory formalizes the study of waiting lines and queues in a mathematical framework. While it was originally developed in the context of operations research to analyze logistics of business related decisions and operations management more systematically, it has frequently been adopted in other contexts such as telecommunications, computer networks, road traffic, etc as well to leverage queuing models for gaining insight into the way these systems function. Traditionally, customers behavioral and cognitive factors were treated as exogenous parameters in the queuing models constructed for studying queuing networks; thus, classical models typically failed to properly capture the impacts of interaction of users with queuing systems. However, system dynamics are clearly influenced by the decisions made by the customers entering the queuing system to receive some sort of service. Game-theoretic approaches can then be leveraged to properly incorporate impacts of customers behavior on queuing networks dynamics which can potentially lead to construction of truly predictive models of dynamics of these complex systems. Following such a notion, in this project we provide examples of different settings that involve some sort of queuing phenomenon in which strategic decision makers can affect evolution of system dynamics through the actions that they choose. This work is based on the models discussed in the survey paper by Altman in 2005 [2]. I Introduction Queuing networks may comprise individual decision makers, central controllers or a combination of both. As queuing models become more sophisticated to better represent more realistic scenarios, decision making tends to be more complicated at both individual and central levels as more factors have to be taken into account while devising strategies. Decentralized structure of the systems, uncertainty in the underlying dynamics and availability of limited and imperfect information can further complicate the decision making procedure. 1

2 Decisions that are typically faced with by both individuals and central controllers in a queuing system can in general be categorized as follows: 1. Whether or not to queue? The very decision of whether or not to queue typically arises when new jobs are decided to be submitted to a network or not. One example would be admission of packets in different nodes of a communication network. Other examples will be discussed in more details later. 2. When to queue? Decision problems that are studied under this category would concern determination of the best time for arriving into a queue to receive a service. These decisions are typically taken based on the information about statistics of the total number of jobs submitted to the system to receive service. However, individuals decisions about their arrival time will not be publicly available most of the times. 3. Where to queue? Routing decisions in both telecommunication networks and transportation networks are well-known examples of decisions of this type. Having the information about congestion state and demands load at different nodes of a network is critical in picking the optimal routing policy. 4. How much to queue? Determination of the amount of the data that can be transmitted through a network as well as the rate at which the information should be sent are the principal questions that should be answered under this category. Once again information about the amount of data that are already queued at different nodes of a network, which represents congestion state at different parts, is required to adjust the rate and amount of data that is to be sent over the network in a way that the imposed cost 1 is minimized. Different examples illustrating the aforementioned types of decision problems will be provided and discussed. We will also investigate certain properties of these game-theoretic models including existence and uniqueness, structure, certain behavioral tendencies and overall efficiency of the emerged equilibria to gain more insight into the structure of problems. This report is organized in the following way: we will first introduce dynamic games. In section III we will briefly talk about game theoretic analysis in networked systems. Section IV is devoted to a general overview of basic concepts of queuing theory and some related terminology. In section V examples of each of the introduced decision problems will be provided and discussed in greater details. We will then conclude by summarizing key points and possible future directions to pursue in this context. II Dynamic Games Static game models are used to analyze situations where players specify their actions once and stick to it afterwards. In fact, there is no dynamic information structure involved in static form games in the sense that no new information is learned during the process. Dynamic game frameworks on the other hand are suitable for studying decision making in dynamic situations. There are essentially two different paradigms for perceiving dynamic structures in competitive settings: 1 Cost is used as a general term here and its precise definition can differ from one context to another 2

3 Extensive form games: this framework is established to study sequential decision making problems in systems comprising multiple self-interested decision makers whose objectives may or may not conflict. Repeated games: this class of dynamic analysis in game theory is devoted to studying dynamics induced through repeatedly playing the same one-stage game in possibly multi-agent settings. The dynamic 2 models that will be discussed in this report are analyzed from the first perspective. II.I Extensive Form Games As discussed in [14], an extensive form game can be formally defined to have four major components as follows: A set of players, I = {1,, I} A set H of sequences: histories h 0 = s 0 = (s 0 1,, s 0 I ) initial history stage 0 action profile h 1 = s 0 history after stage 0. h k+1 = (s 0, s 1,, s I ) H = {h k } H = 0 H k history after stage k set of all possible stage k histories set of all possible histories A set of pure strategies for each player, s i = {s k i } k=0 A set of preferences for each player, u i : H k+1 R III Network Games Players decisions can depend on each others actions through a networked structure. Here we bring the Pigou s example [Pigou 1920] from [14] as a motivating example to better illustrate the situation. Figure 1: Pigou s example As illustrated in figure 1, one unit of traffic is to be transmitted from the source node on the left to the destination node on the right. There are two alternative routes that can be used for this purpose. Each 2 Some of the examples that will be discussed later are static form games 3

4 of these links have a cost 3 function associated with them which indicates the cost that is posed through transportation of 1 unit of traffic from left side to the right side. The cost on the first link is a function of the congestion or load imposed to it and is linear in traffic while the cost of link 2 is a constant number, i.e., no matter how much traffic is transported through the second link, the fixed amount of cost (= 1) will be posed. The problem is now determination of the optimal routing policy that possibly incurs the minimum amount of cost. The optimal routing policy can be sought from two different perspectives: Social optimal problem Non-cooperative setting Social Optimal Problem Here the goal is to minimize the aggregate cost in the system through finding the routing policy that minimizes the global cost. This problem can be formulated in the following way: {x 1, x 2} = arg min C total (x s ) = arg min { x 1+x 2=1 x 1+x 2=1 2 i=1 l i (x s i )x s i } = arg min { x x 2 } (1) x 1+x 2=1 By taking derivatives of the total cost with respect to x 1 and x 2 and setting them equal to zero (considering the constraint x 1 + x 2 = 1) we will get: {x 1, x 2} = { 1 2, 1 2 } (2) Meaning that from a social optimization viewpoint the optimal policy would be to equally divide the load between the two links. The minimum of the aggregate cost is then achieved at {x 1, x 2} = { 1 2, 1 2 }: min x 1+x 2 1 C system(x s ) = i l i (x s i )x s i = = 3 4 (3) Non-Cooperative Framework Now we consider the non-cooperative setting where each self-interested player is only concerned about minimizing their own cost and disregard how their decision may affect the total cost in the system. In this case, one can easily see that the socially optimal policy ({x 1, x 2} = { 1 2, 1 2 }) cannot be a Nash equilibrium. Suppose that we have players with infinitesimal traffic loads 4 to transmit. Since their load is very small they will clearly prefer not to use link 2 essentially because they do not want to pay the considerable cost associated with link 2 (=1) for their negligible traffic load ( x 1). Hence, the unique Nash equilibrium of this game turns out to be: {x W E 1, x W E 2 } = {1, 0}. The superscript W E stands for Wardrop Equilibrium which is another name for the Nash equilibrium in this context, named after John Glen Wardrop who formalized this equilibrium concept in the context of transportation. We can now calculate the total cost of the system incurred after adopting this equilibrium routing policy {x W 1 E, x W E} 2 = {1, 0}: C e q(x W E ) = i l i (x W i E )x W i E = = 1 (4) 3 Typically in the context of road traffic, cost can be thought of as vehicle s delay 4 Such games that comprise an infinite number of infinitesimal players are typically called non-atomic games 4

5 So, from the results in equations (3) and (4) it is observed that the total cost at a system with self interested users is 4 3 times larger than the cost incurred through socially optimal policy. It is then formally argued that this is the worst-case factor of loss of efficiency in case all possible network topologies are considered and all costs are linear. In fact, this notion that equilibria tend to exhibit an overall inefficiency from a gloabl point of view was then formalized through introduction of the concept of Price of Anarcy[11] which defines the largest efficiency loss factor over all routing instances (R ), taking all possible network topologies as well as all delay functions from a certain family (such as linear in the Pigou example) into account: P oa = inf R R C(x s (R)) C(x W E (R)) (5) IV Queuing Theory In this section we briefly review some of the basic concepts from Queuing theory ([13]) and present common assumptions that are made in analyzing queuing systems. M/G/s Model The basic model used to represent a typical queuing system is M/G/s where s indicates existence of s identical servers in the system. G stands for General, meaning that service rate distribution is not specified. M holds for Markovian modulated with Poisson meaning that arrivals enter the system according to a Poisson distribution and hence, the inter-arrival times are exponentially distributed. M/M/1 is probably the simplest model of a queuing system in which there is only one server to respond to requests, arrivals follow a Poisson distribution and service rate is exponentially distributed. Customers Decision Types Customers may exhibit different types of behavior in a queuing system such as declining to join a queue which is referred to as balking and deciding to leave a queue after joining it which is referred to as reneging. It should be noted here that arrival process should not be confused with joining process; arrival process alludes to the customers entering the system which does not necessarily mean that the users that have entered the system will queue somewhere in the system to receive service. Joining process on the other hand refers to the process of the customers who have joined the queue. It can then be inferred that a disparity between arrival and joining rates implies occurrence of balking decisions among customers. Service Discipline First-Come First-Served (FCFS) is probably the most well-known service discipline practiced in queuing systems. There are however networks which are operated base on a Last-Come First-Served (LCFS) discipline as well. Other types of discipline might also be applied for responding to demands in different contexts. Service Times Typically service times are assumed to be i.i.d.. If we consider the expected service time to be equal to µ 1 and the expected inter-arrival time to be λ 1 then system utilization factor can be written as ρ = λ µ which is defined as the ratio between average service time and average inter-arrival time. Often times for stability reasons, ρ < 1 which implies that customers are served faster than they arrive. 5

6 V Queuing Game Models In this section we provide examples of queuing decision problems that can be modeled within a game-theoretic framework. V.I To queue or not to queue? PC-MF game [4] is an example of these types of decision problems in computer networks. In PC-MF game customers that need to recieve service have to decide whether to queue at a fast serving but public facility that can be represented as a Main Frame (MF) or to be served at a slowly serving but private equipment which can be thought of as a personal computer (PC). The service discipline used by the MF here is of processor sharing type in which the processor capacity is shared between all the customer queued there. Figure 2: PC-MF game V.I.I The Model In PC-MF game it is assumed that when the individuals arrive at the system, they can observe the length of the queue at the MF and based on this information they can decide whether or not to queue there to be served at MF. Here is the way to model this problem: Inter-arrival times i.i.d with mean λ 1 Service at MF exponential with rate µ(x) (typically µ(x) = µ) µ(x) Service intensity per customer: x (typically µ x ) Expected PCs service time: θ 1 Now, let s define: X(t) := the number of customers at MF at time t T k, k 0 := arrival time of job C k, where 0 < T 0 < T 1 < T 2 < u k := the strategy for C k, probability of joining joining MF if X(T k ) = x π := (u 0, u 1, ) multi-strategy profile of all customers w k := service duration of customer C k W k (x, π) := expected service time of C k, given that x customers are in the queue when she arrives at MF V k (x, π) := expected service time of C k, if she chooses to be served at MF, given that x customers are in the queue when she arrives at MF 6

7 x denotes the number of customers queued at the MF for service. Given this model and considering the expected service time of an individual as the performance measure, the objective function to be minimized so as to yield the k th self-interested user s optimal policy can be formulated as: W k (x, π) = u k (x)v k (x, π) }{{} + (1 u k (x))θ 1 }{{} expected service time of C k at MF expected service time of C k at PC (6) It should be noted that V k depends also on the decisions of subsequent customers through the multistrategy profile π := (u 0, u 1, ) which comprises the decisions of the customer that will arrive the system after the k th step ({u l, l > k}) in addition to the ones already queued there. This effect is caused by the processor sharing discipline that is used by MF here. Defition: Threshold Policies For any 0 q 1 and integer L 0, the decision rule u is an [L, q]-threshold rule if 1 if x < L u(x) = q if x = L 0 if x > L (7) If an individual follows a threshold decision rule as defined in (7), she will join the queue at MF with probability 1 if x < L. If x = L she will choose to queue at MF with probability q and she will choose to be served by the PC with probability 1 if x > L. A threshold policy is then characterized by the two parameters L and q and can be denoted by [g] where g = L + q for representation convenience. Theorem: (i) For any equilibrium policy π = (u 0, u 1, ), each decision rule u k is a threshold rule. (ii) A symmetric equilibrium policy π = (u, u, ) exists, is unique, and u is a threshold rule. Basic steps of proof: (i) For every policy π and k 0, one shows that V k (x, π) is strictly increasing in x. (ii) Assume that all jobs other than C k use a threshold policy [g] = [L, q]. Then V k (x, [g] ) can be shown to be: 1. strictly increasing in g, and 2. continuous in g So it can be seen that when all users apply the threshold policy [g], the threshold value for the best responding decision rule is decreasing in that [g] = L + q which means that this individual is encouraged to balk as the number of the people queued at MF increases. So, when the system is not at equilibrium, the individuals may be incentivized to take an action that is opposite of others which leads to emergence of the Avoiding the Crowd tendency among the users. This makes sense essentially because some mechanism for keeping the length of the queue at MF bellow some certain threshold value should be adopted since this would be beneficial to the self-interested users. 7

8 V.II When to queue? We discuss the phone call retrial ([5],[7]) example borrowed from the context of games in retrial queues. Getting the busy signal while trying to make a phone call somewhere can indicate either of the two possibilities: the destination is busy with another phone call or the line is blocked due to congestion. When a line is found busy by some user, often times they may try to make retrial phone calls. In this case, the call is said to be in orbit. We are then interested in determining the optimal rate of making retrial calls from both global and individual viewpoints. The problem can then be modeled as follows: The Model: Calls arrival Poisson with average rate λ Service rates i.i.d. with mean τ and finite variance σ 2 Let S 2 := τ 2 + σ 2 and ρ := λτ Times between retrials of the i th call in orbit exponential 5 with expected value 1 θ i c cost for 1 retrial w cost of waiting per unit of time In [12] Kulkarni showed that the expected time spent in orbit when the parameter θ is the same for all retrials is: Social Optimization Problem W = ρ ( ) 1 1 ρ θ + S2 2τ Using (8), the average cost per call will be formulated as follows: (w + cθ)w = ρ ( cs 2 1 ρ 2τ θ + w ) + ρ ( ) ws 2 θ 1 ρ 2τ + c The global minimizer will then be of the following form: θ = 2wτ/c It is interesting to note here that the socially optimal retrial rate θ as obtained in (10) is independent of the calls arrival rate λ. If we plug θ in (9), it can be seen that waiting cost and retrial cost will coincide ( cs2 2τ θ = w θ ). Non-cooperative framework In a non-cooperative framework where each self-interested user is concerned with minimization of their own waiting time in orbit, the equilibrium retrial rate can be computed by assuming that an individual is interested in minimizing her waiting time g(θ, γ) through finding the best γ when all other individuals are making retrials with the rate θ. The equilibrium rate is then shown to have the following form: S (8) (9) (10) 5 Exponential distribution is frequently used to model processes in telecommunications, particularly in telephony to model duration of phone calls. Due to its memoryless property, eponential distribution leads to straightforward Markovian structure which is desirable: P (X > t + s X > t) = P (X > s) for any t, x > 0 8

9 θ e = wρ + w 2 ρ w(1 ρ)(2 ρ)/s 2 4c(1 ρ) (11) It can be seen that θ e as given in (11) is increasing in λ and as λ 1 σ, θ e. This implies that within a non-cooperative framework, users tend to make retrials at a faster rate as the rate of arrivals (λ) into the line increases, so as to maximize their chances of connecting the destination. Also, it can be shown that the ratio between the total cost in the system at θ e and cost at θ tends to infinity as λ increases from which it can be inferred that the equilibrium solution can lead to high inefficiencies in the system from a global point of view. Using (10) and (11), it can be shown that θ e > θ but as ρ 0, which is achieved by either tending service rate τ to 0 or tending the arrivals rate λ to 0, then θ e θ. It is also observed that both θ e and θ will increase as the uncertainty in service rate becomes larger (σ ). The non-cooperative model that was discussed here is actually a static form game, essentially because the users are not making new observations in the process to update their decision based upon that. From (11) also it can be seen that the equilibrium rate is characterized by the parameters all of which are known a priori. V.III Where to queue? In a network-like structure various performance measures such as packet loss probabilities in a communication network, or delay in a transportation network can be affected by the congestion state at different parts of the network and congestion itself is a function of the routing decisions taken by both individuals and central controllers. Therefore, decisions regarding which path to choose in a network to transmit a message or transport a certain amount of traffic between two nodes is critical in the overall performance of the system. These types of problem are studies in the context of traffic assignment in networks. Wardrop has done fundamental works related to traffic assignment in the context of road traffic and the principles he developed using a game model which comprises a continuum number of agents 6 where the effects of one player s decisions can be insignificant on other players costs, are also applied in other contexts. The game might be played by atomic players rather than atomless ones where atomic players can be represented by big organizations in the context of telecommunications or transportation networks. Atomic players will then decide about the different routs that they can choose in a network to transmit various portions of their total flow over each ([9],[15]). Figure 3: Traffic Assignment Problem We will discuss the gas station game [6] borrowed from the road traffic context. 6 Such player are typically referred to as atomless players 9

10 V.III.I Gas Station Game In this game, there are two gas stations on a highway. The drivers arrive the first gas station and observe the length of the queue there and based on this information they will then decide to queue there or leave the first station for the second one. An analog of this situation is also observed in choosing routing paths in a communication network where the information about congestion state at a downstream node may become available with so much delay that can make those information useless at time of making decisions. Probability distribution of the congestion state in the second queue depends on the routing policy and can be computed by the router. The equilibrium in this game can then be obtained using the joint probability distribution of the congestion states at both queues. In this game, we assume a threshold structure for the decision rules as defined earlier in (7) and here it is characterized by parameters (n, r). In sequel of holding this assumption that all players will adopt such threshold decision rule, through solving the steady-state probabilities of the underlying continuous Markov chain, steady state joint probability distribution of the congestion states can be obtained. Figure 4: Gas station game The Model: Service time at queue i exponential with parameter µ i Global inter-arrival times i.i.d. exponential with parameter λ i Users will then estimate the length of the queue at the second station, based on the observed queue length at first station: E i [X 2 ] = E[X 2 X 1 = i] (12) Customers will then choose to be served at the queue in which they will have to wait for lesser amount of time: T n,r (i, 1) := i + 1 µ 1 E i[x 2 ] + 1 µ 2 := T n,r (i, 2) (13) In order to compute the routing policy, the user will then need to know the routing policy used by all previous users as the probability distribution of the congestion state in second queue is characterized by users decision rule. If based on the decision problem formulated in (13) the user s decision rule coincide with [g] = (n, r) policy applied by others as well, then [g] = (n, r) is the Nash equilibrium of this game. In this game also the Avoiding the Crowd 7 behavior is observed when system is not at equilibrium. 7 The optimal response against the decision rule [g] = (n, r) used by others is monotonically decreasing in g = n + r. 10

11 It is argued in [6] that when users adopt the equilibrium policy, the first station ends up making higher revenue than the second stations and hence, availability of further information about the congestion state in the first queue turns out to be of additional benefit to the first station. In this case, it is not yet clear whether station 2 can apply some pricing mechanism to incentivize the customers to prefer to be served at the second station. Finding the optimal decision rule in such framework is an interesting problem to further pursue. One possible idea would be to involve other types of costs so as to be able to formulate the optimal routing policy problem in a pricing framework. If we consider γ i to be the price of service (e.g., filling up the gas tank in this context) at the i th queue and consider w to be the cost of waiting per unit of time, the routing decision problem can be formulated 8 as: C(i, 1) := w ( ) i + 1 µ 1 }{{} cost of waiting in queue 1 + γ 1 }{{} service price at queue 1 w ( ) Ei [X 2 ] + 1 µ 2 } {{ } cost of waiting in queue 2 + γ }{{} 2 := C(i, 2) (14) service price at queue 2 Where, C(i, j) indicates total cost of service (including waiting times) at j th queue when there are i customers in the first queue. From (14) it can be inferred that gas station 2 can potentially reduce its service price to the extent that it provides sufficient motivation for drivers to leave the first queue and queue at station 2 while they do not have precise information about the congestion state at second station. It should be noted that the cost of the time spent on getting to queue 2 from station 1 is not incorporated in the formulation in (14). V.IV Queues with Priority In this section we discuss an example of a queueing problem from [1] and [8] where customers arrive into the system according to a Poisson distribution with average rate λ. Service rate is exponentially distributed with parameter µ and there is a single server in the system yet two queues: one with strictly high priority and the other with low priority. Upon arrival, players will observe the number of customers in each queue and decide whether to join the high priority queue or the low priority one, given that the customers queued at the low priority line will be served only after all customers in the high priority line are served. However, the customers need to purchase the high priority for the payment of θ. Let (i, j) denote the state of the system where i is the number of people queued at high priority line while j indicates the number of people queued for the low priority. Figure 5: Games in queues with priority 8 This formulation is not borrowed from any where. 11

12 Monotonicity Property[1]: If for some policy used by everybody it is optimal for an individual to purchase priority when the system state is (i, j), then she must purchase priority at (r, j), r > i. In sequel of this principle, it can also be inferred that if at some state (0, m) it is optimal not to purchase priority, then it is also optimal not to buy one at the state (0, i), i m. Now, letting n 1 to be the largest state of this form, after reaching the state (0, n 1) it will then be optimal to purchase priority starting at (0, n) after which we move to (1, n). One of the interesting consequences of the monotonicity property is that so long as the system is at state (i, n), i 1 the length of the queue at low priority line will neither decrease nor increase. Essentially because at such a state, the customers will purchase high priority and the customers in the low priority line will not be served since high priority line is not empty yet. It is argued in [1] that when all customers except for one individual adopt the threshold rule [g], then the optimal threshold for that individual will be non-decreasing in g, implying a "Follow the Crowd" kind of behavior. It is further argued in [8] that uniqueness of the equilibrium cannot be established for this problem and that there may be up to 1 1 ρ, ρ = λ µ pure threshold Nash equilibria in addition to other mixed equilibria. V.V How much to queue? Decisions of this type have to do with the transmission rate and amount of data that is to be sent over a network which can also be viewed as finding effective strategies for controlling congestion state within a network. In communication systems there are two main approaches to address this problem: Transmission rate control in sources Window based[10] flow control In the first framework congestion control is achieved through controlling transmission rate of different sources in networks where bandwidth allocations are not very strict. The second approach is commonly used in Internet applications where the sender transmits multiple frames (depending on the size of the window in the receiver side) to the receiver and waits for the acknowledgement from the receiver. This mechanism tends to be more efficient in terms of throughput speed. Here we provide an example from the first framework. V.VI Competitive Flow Control In this problem M sources want to transmit their information through a single bottleneck queue. A linear quadratic differential 9 game model[3] is then constructed to seek the optimal policy that can maximize each sender s throughput while satisfying certain constraints. 9 Differential game is a framework that formalizes dynamic analysis in settings in which players have conflicting goals and a system of differential equations is then used to describe dynamics of the game. Differential games can be viewd as a generalization of the notions of optimal control where the system may comprise multiple self-interested decision makers each of whom aims at maximizing their own benefits through controlling state of the system 12

13 Figure 6: Competitive flow control problem The Model: s(t) server s rate r m (t) controlled input rate of user m q(t) instantaneous queue rate Q target size of the queue a m s(t) bandwidth share for user m where, M m=1 a m = 1 a m denotes the m th user s allocated portion of the available bandwidth in the system. Let s define: x(t) := q(t) Q state of the queue u m (t) := r m (t) a m s(t) shifted control Dynamics of the system can then be formulated as follows: dx M dt = M (r m a m s) = u m (15) m=1 It should be noted that the non-linearities[3] associated with the extreme cases in the system state (empty queue or full queue) are not reflected 10 in (15). Intuitively it can be seen that the rate of the change in the queue s shifted length is equivalent to the summation of all (positive or negative) disparities between users throughput rate (r m (t)) and their actual bandwidth share a m (t). m=1 Here, we consider the policies employed by senders to be history dependent of the following form: u m (t) = µ m (t, x t ), t [0, ), u m U m (16) Where, µ m is piecewise continuous in its first argument and piecewise Lipschitz continuous in its second argument. There are two types of costs to consider here: Cost of deviation from the target queue length Q Cost of deviation from the allocated bandwidth to each user There are two different ways to linearly combine these two costs and formulate the objectives: 10 In the derived optimal control mechanism it is shown that there the developed control is symmetric with respect to positive or negative deviations around the target queue length 13

14 1. N1: m th user (m M = {1,, M}) minimizes her individual cost J N1 m (u) = 0 1 } x(t) 2 + u m (t) 2 {{} c m cost of }{{} deviation from Q cost of deviation from bandwidth share dt (17) 2. N2: m th controller (m M) minimizes its individual cost J N2 m (u) = 0 1 M x(t) 2 }{{} split queue length deviation cost + 1 u m (t) 2 c m dt (18) As can be seen from (18), the cost of deviation from the target length Q is equally split among the M users in N2 formulation of the cost. The problem of determining the Nash equilibrium would be formulated as finding the equilibrium multipolicy profile µ := (µ 1,, µ M ) through the following infimization11 : J N1 m (µ ) := inf µ m U m J N1 m ([µ m µ m]) (19) Similar equilibriunm formulation can be used for N2 objective as well. The equilibrium problem formulated in (19) seeks the best policy for m th user when all other players use the decision rule µ. It can then be shown [3] that the competitive flow control game model as described, has a unique Nash equilibrium of the following form: Where, µ Ni,m(x) = β Ni m x, m = 1,, M, i = 1, 2 (20) β N2 m = β N1 m = M m=1 ( M m=1 β N1 β N1 m ) ( M m=1 m / ( M M m=1 β N1 m β N1 m / M ) 2 c m (21) ) 2 c m M Now, we can evaluate the two types of objective formulations ((17), (18)) at the equilibrium policy: (22) J N2 m J N1 m (µ N1) = βn1 m x 2 (23) c m (µ N2) = βn2 m x 2 = 1 Jm N1 (µ c N1) (24) m M 11 In general, the decision space (U m) is considered to be infinite in this problem. 14

15 Where βm N1 and βm N2 are given by (21) and (22) respectively. From (24) it can be seen that splitting the cost of deviation from the target queue length Q among the M users has led to the relation J N2 m (µ N2 ) = 1 M Jm N1 (µ N1 ) between the two cost objectives defined in (17) and (18) respectively. In the symmetric case where c m = c j := c for all m, j M we will get: c βm N1 = 2M 1 βm N2 c = M(2M 1) (25) (26) And the N1 and N2 costs can be evaluated using (25) and (26) for the symmetric case. In this game model[3], delays that may occur in availability of information to users as well as effects of presence of noise in users observations were not incorporated. These effects can be reflected through including some stochasticity assumptions on fluctuations that may occur in the available bandwidth throughout network operation. In that case notions from robust controller design can be employed in formulating optimal routing policies. Continuous availability of information to the users is another assumption underpinning the presented analysis whereas in practical cases the information may become available in certain time instances only. Considering these additional details can lead to development of more sophisticated models for designing flow control policies in networks. VI S-Modular Games As discussed in the models presented in previous sections, most of the queuing systems when analyzed from a game-theoretic view, tend to exhibit either of the Avoiding the Crowd or Joining the Crowd behaviors while computing best response policies. Having such insights into the structure of problems can be quite useful while investigating convergence of non-equilibrium states to equilibria. Such considerations will be discussed in this section from [16] and [17]. VI.I Example of Super-Modularity In this example the system comprises a set queues in tandem where each queue has a server with controlled serving speed. The utility function formulated for each player comprises rewards for the throughput and penalty for the delay. In [17] it is argued that even though the players actions are not coordinated in any direct way, super-modularity 12 ensures their incentives to be compatible, meaning that their strategies tend to increase or decrease in the same direction. We consider the following model for a simple system with 2 queues in tandem: The Model: service times i.i.d. exponential with parameter µ i, i = 1, 2 12 The utility f i for player i is super-modular if and only if where, S i is the strategy space and min(x, y) = x y f i (x y) + f i (x y) f i (x) + f i (y), x, y S i 15

16 Let µ i u for some constant u Assume also that server one has an infinite source of input jobs and that there is buffers between the two servers. The throughput is then given by µ 1 µ 2. It is then shown in [17] that the expected number of jobs in the buffer can be obtained as: µ 1 µ 2 µ 1 if µ 1 < µ 2 (27) The expected number of jobs in the buffer will be infinity otherwise (µ 1 µ 2 ). Let: p i (µ 1 µ 2 ) profit of server i c i (µ i ) operating cost g(.) inventory cost Players utilities can then be formulated as: Strategy spaces are given by: ( ) µ1 f 1 (µ 1, µ 2 ) := p 1 (µ 1 µ 2 ) c 1 (µ 1 ) g µ 2 µ 1 ( ) µ1 f 2 (µ 1, µ 2 ) := p 2 (µ 1 µ 2 ) c 2 (µ 1 ) g µ 2 µ 1 (28) (29) S 1 (µ 2 ) = {µ 1 : 0 µ 2 µ 2 } (30) S 2 (µ 1 ) = {µ 2 : µ 1 µ 2 u} (31) VI.II Example of Sub-Modularity: Flow Control Consider a queuing center with c servers with 1 unit of service rate each. There are 2 input streams to the system with Poisson arrival rates having average rates λ 1 and λ 2. 2 players control rates of the streams. If all servers are busy, further arrivals to the system are blocked and lost. The blocking probability is given by Erlang loss formula 13 : Where, λ = λ 1 + λ 2 User i s utility will then be given by: [ c B(λ) = λc c! k=0 ] 1 λ k (32) k! 13 The probability that a new customer arriving to a queuing system is rejected because all servers are busy is: P = Em/m! m Where E is the total traffic in Erlang, offered to m identical parallel servers in the system i=0 E i i! 16

17 f i = r i (λ i ) c i ( λb(λ) ) (33) }{{} total loss rate Where c i is assumed to be convex increasing and then it can be argued [17] that f i are sub-modular 14. VII Conclusion In this work we presented several examples from telecommunications and transportation contexts and discussed some of the most frequently arose decision problems in queuing systems, which are faced with by both individuals and central controllers. We then borrowed notions from queuing theory and used various game models to analyze the interaction between strategic self-interested customers and queuing networks from a game-theoretic perspective. We studied existence and uniqueness of possible equilibria in these problems and discussed certain properties associated with equilibria such as Avoiding the Crowd and Joining the Crowd behaviors. We showed that in the provided queuing models the optimal strategies typically tend to exhibit a threshold like structure. In some cases we investigated overall efficiency in a system at equilibrium strategy profiles of decision makers and compared the results with socially optimal results to gain further insight into the way these systems function. In the end, we briefly introduced sub-modularity and super-modularity porperties of a function and discussed S-Modular game framework that can be used to provide further insight into the relation between non-equilibrium and equilibrium states of a system. It should be noted that general form and even existence of Nash equilibria in several of these game models still remains an open problem. 14 The utility f i for player i is sub-modular if and only if f i (x y) + f i (x y) f i (x) + f i (y), x, y S i where, S i is the strategy space and min(x, y) = x y. Intuitively, sub-modularity of a function f that maps a subset of a set S to a real value, implies that adding an element to a smaller subset of S makes a bigger difference to the values of f than adding it to a larger subset of S. Such a property is referred to as diminishing return property 17

18 References [1] Igal Adiri and Ury Yechiali. Optimal priority-purchasing and pricing decisions in nonmonopoly and monopoly queues. Operations Research, 22(5): , [2] Eitan Altman. Applications of dynamic games in queues. In Advances in Dynamic Games, pages Springer, [3] Eitan Altman and Tamer Basar. Multiuser rate-based flow control. Communications, IEEE Transactions on, 46(7): , [4] Eitan Altman and Nahum Shimkin. Individual equilibrium and learning in processor sharing systems. Operations Research, 46(6): , [5] Amie Elcan. Optimal customer return rate for an m/m/1 queueing system with retrials. Probability in the Engineering and Informational Sciences, 8(04): , [6] Refael Hassin. On the advantage of being the first server. Management Science, 42(4): , [7] Refael Hassin and Moshe Haviv. On optimal and equilibrium retrial rates in a queueing system. Probability in the Engineering and Informational Sciences, 10(02): , [8] Refael Hassin and Moshe Haviv. Equilibrium threshold strategies: The case of queues with priorities. Operations Research, 45(6): , [9] Alain Haurie and Patrice Marcotte. On the relationship between nash cournot and wardrop equilibria. Networks, 15(3): , [10] Yannis A Korilis and Aurel A Lazar. On the existence of equilibria in noncooperative optimal flow control. Journal of the ACM, 42(3), [11] Elias Koutsoupias and Christos Papadimitriou. Worst-case equilibria. In STACS 99, pages Springer, [12] Vidyadhar G Kulkarni. On queueing systems with retrials. Journal of Applied Probability, pages , [13] Larry J LeBlanc, Edward K Morlok, and William P Pierskalla. An efficient approach to solving the road network equilibrium traffic assignment problem. Transportation Research, 9(5): , [14] Asu Ozdaglar and Ishai Menache. Network Games: Theory, Models, and Dynamics. Morgan & Claypool Publishers, [15] P Patriksson. The traffic assignment problem: models and methods [16] Donald M Topkis. Equilibrium points in nonzero-sum n-person submodular games. SIAM Journal on Control and Optimization, 17(6): , [17] David D Yao. S-modular games, with queueing applications. Queueing Systems, 21(3-4): ,