Performance Evaluation

Transcription

1 Performance Evaluation ( ) Contents lists avaible at ScienceDirect Performance Evaluation journal omepage: Dynamic service migration and workload sceduling in edge-clouds Raul Urgaonkar a,, Siqiang Wang b, Ting He a, Murtaza Zafer c, Kevin Can d, Kin K. Leung b a IBM T. J. Watson Researc Center, Yorktown Heigts, NY, USA b Department of Electrical and Electronic Engineering, Imperial College London, UK c Nyansa Inc., Palo Alto, CA, USA d US Army Researc Laboratory, Adelpi, MD, USA a r t i c l e i n f o a b s t r a c t Article istory: Avaible online xxxx Keywords: Edge-clouds Service migration Stocastic optimization Markov decision processes Edge-clouds provide a promising new approac to significantly reduce network operational costs by moving computation closer to te edge. A key callenge in suc systems is to decide were and wen services sould be migrated in response to user mobility and demand variation. Te objective is to optimize operational costs wile providing rigorous performance guarantees. In tis paper, we model tis as a sequential decision making Markov Decision Problem (MDP). However, departing from traditional solution metods (suc as dynamic programming) tat require extensive statistical knowledge and are computationally proibitive, we develop a novel alternate metodology. First, we establis an interesting decoupling property of te MDP tat reduces it to two independent MDPs on disjoint state spaces. Ten, using te tecnique of Lyapunov optimization over renewals, we design an online control algoritm for te decoupled problem tat is provably cost-optimal. Tis algoritm does not require any statistical knowledge of te system parameters and can be implemented efficiently. We validate te performance of our algoritm using extensive trace-driven simutions. Our overall approac is general and can be applied to oter MDPs tat possess a simir decoupling property Elsevier B.V. All rigts reserved. 1. Introduction Te increasing popurity of mobile applications (suc as social networking and poto saring) running on andeld devices is putting a significant burden on te capacity of cellur and backaul networks. Tese applications are generally comprised of a front-end component running on te andeld and a back-end component (tat performs data processing and computation) tat typically runs on te cloud. Wile tis arcitecture enables applications to take advantage of te on-demand feature of cloud computing, it also introduces new callenges in te form of increased network overead and tency. A promising approac to address tese callenges is to move suc computation closer to te network edge. Here, it is envisioned tat entities (suc as basestations in a cellur network) closer to te network edge would ost smallersized cloud-like infrastructure distributed across te network. Tis idea as been variously termed as Cloudlets [1], Fog Corresponding autor. addresses: rurgaon@us.ibm.com (R. Urgaonkar), siqiang.wang11@imperial.ac.uk (S. Wang), te@us.ibm.com (T. He), murtaza.zafer.us@ieee.org (M. Zafer), kevin.s.can.civ@mail.mil (K. Can), kin.leung@imperial.ac.uk (K.K. Leung). ttp://dx.doi.org/ /j.peva / 2015 Elsevier B.V. All rigts reserved.

2 2 R. Urgaonkar et al. / Performance Evaluation ( ) Computing [2], Edge Computing [3], and Follow Me Cloud [4], to name a few. Te trend towards edge-clouds is expected to accelerate as more users perform a majority of teir computations on andelds and as newer mobile applications get adopted. One of te key design issues in edge-clouds is service migration: sould a service currently running in one of te edgeclouds be migrated as te user locations cange, and if yes, were? Tis question stems from te basic tradeoff between te cost of service migration vs. te reduction in network overead and tency for users tat can be acieved after migration. Wile conceptually simple, it is callenging to make tis decision in an optimal manner because of te uncertainty in user mobility and request patterns. Because edge-clouds are distributed at te edge of te network, teir performance is closely reted to user dynamics. Tese decisions get even more complicated wen te number of users and applications is rge and tere is eterogeneity across edge-clouds. Note tat te service migration decisions affect workload sceduling as well (and vice versa), so tat in principle tese decisions must be made jointly. Te overall problem of dynamic service migration and workload sceduling to optimize system cost wile providing enduser performance guarantees can be formuted as a sequential decision making problem in te framework of MDPs [5,6]. Tis approac, altoug very general, suffers from several drawbacks. First, it requires extensive knowledge of te statistics of te user mobility and request arrival processes tat can be impractical to obtain in a dynamic network. Second, even wen tis is known, te resulting problem can be computationally callenging to solve. Finally, any cange in te statistics would make te previous solution suboptimal and require recomputing te optimal solution. In tis paper, we present a new metodology tat overcomes tese drawbacks. Our approac is inspired by te tecnique of Lyapunov optimization [7,8] wic is a general framework for designing optimal control algoritms for non-mdp problems witout requiring any knowledge of te transition probabilities. Specifically, tese are problems were te cost functions and control decisions are functionals of states tat evolve independently of te control actions. However, as we will sow ter, tis condition does not old for te joint service migration and workload sceduling problem studied in tis paper. A key contribution of tis work is to develop a metodology tat enables us to still apply te Lyapunov optimization tecnique to tis MDP wile preserving its attractive features. 2. Reted work Te general problem of resource allocation and workload sceduling in cloud computing systems using te framework of stocastic optimization as been considered in several recent works. Specifically, [9] considers a stocastic model for a cloud computing cluster, were requests for virtual macines (VMs) arrive according to a stocastic process. Eac VM request is specified in terms of a vector of resources (suc as CPU, memory and storage space) and its duration and must be pced on pysical macines (PMs) subject to a set of vector packing constraints. Ref. [9] defines te notion of te capacity region of te cloud system and sows tat te MaxWeigt algoritm is trougput optimal. Virtual macine pcement utilizing sadow routing is studied in [10,11], were virtual queues are introduced to capture more complicated packing constraints. Ref. [12] considers a joint VM pcement and route selection problem were in addition to packing constraints, te traffic load between te VMs is also considered. All tese works consider a traditional cloud model were te issue of user mobility and resulting dynamics is not considered. As discussed before, tis issue becomes crucial in edge-clouds and introduces te need for dynamic service migration tat incurs reconfiguration costs. Te presence of tese costs fundamentally canges te underlying resource allocation problem from a non-mdp to an MDP for wic te tecniques used in tese works are no longer optimal. Te impact of reconfiguration or switcing cost as been considered in some works recently. Specifically, [13] considers te problem of dynamic rigt-sizing of data centers were te servers are turned ON/OFF in response to te time-varying workloads. However, suc switcing incurs cost in terms of te dey associated wit switcing as well te impact on server life. Ref. [13] proposes an online algoritm tat is sown to ave a 3-competitive ratio wile explicitly considering switcing costs. A simir problem involving geograpic load bancing is considered in [14] using a receding orizon control framework. Ref. [15] focuses on a wireless sceduling problem wit reconfiguration dey wile [16] studies te reconfiguration problem from a queueing teory perspective and derives analytical expressions for te performance. All te approaces in [14 16] assume knowledge of te statistics of te underlying system wile [13] considers a single data center wit omogeneous servers. Our work differs from all tese because we explicitly consider te reconfiguration costs associated wit service migrations wile treating a very general model for a distributed edge-cloud system. Te metodology used in tis paper is inspired by te framework of Lyapunov optimization over renewals proposed in [17]. Tis framework extends te Lyapunov optimization approac of [7,8] to treat constrained MDPs. Te basic idea involves converting a constrained MDP into a sequence of unconstrained stocastic sortest pat problems (SSPs) [5,6] tat are solved over consecutive renewal frames. However, solving te resulting SSPs typically still requires knowledge of te underlying probability distributions and can be computationally proibitive for rge state spaces. In tis work, we also make use of te framework of Lyapunov optimization over renewals. However, instead of directly applying te tecnique of [17], we first establis a novel decoupling property of our MDP wic sows tat it can be decoupled into two independent MDPs tat evolve on disjoint state spaces. Tis crucial property enables us to apply te framework of [17] in suc a way tat te resulting algoritms are simple deterministic optimization problems (rater tan stocastic sortest pat problems) tat can be solved efficiently witout any knowledge of te underlying probability distributions. For example, one of te components of our overall algoritm involves solving a deterministic sortest pat

3 R. Urgaonkar et al. / Performance Evaluation ( ) 3 Fig. 1. Illustration of our edge-cloud model sowing te collection of edge-clouds, back-end cloud, and mobile users. problem instead of an SSP every renewal frame. As suc, te resulting solution is markedly different from cssical dynamic programming based approaces and does not suffer from te associated curse of dimensionality or convergence issues. 3. Problem formution We consider an edge-cloud system comprised of M distributed edge-clouds and one back-end cloud tat togeter ost K applications (see Fig. 1). Te system also consists of N users tat generate application requests over time. Te collection of edge and back-end clouds supports tese applications by providing te computational resources needed to serve user requests. Te users are assumed to be mobile wile te edge and back-end clouds are static. We assume a time-slotted model and use te notion of service and application intercangeably in tis paper System model Mobility model: Let L n (t) denote te location of user n in slot t. Te collection of all user locations in slot t is denoted by vector l(t). We assume tat l(t) takes values from a finite (but potentially arbitrarily rge) set L. Furter, l(t) is assumed to evolve according to an ergodic discrete time Markov cain (DTMC) over te states in L wit transition probabilities denoted by p ll for all l, l L. Application request model: Denote te number of requests for application k generated by user n in slot t by A kn (t) and te collection {A kn (t)} for all k, n by vector a(t). Simir to l(t), we assume tat a(t) takes values from a finite (but potentially arbitrarily rge) set A. We furter assume tat for all k, n tere exist finite constants A max kn suc tat A kn (t) A max kn for all t. Te process a(t) is also assumed to evolve according to an ergodic DTMC over te states in A wit transition probabilities q aa for all a, a A. All application requests generated at eac user are routed to a selected subset of te edge-clouds for servicing. Tese routing decisions incur transmission costs and are subject to certain constraints as discussed below. User-to-edge-cloud request routing: Let r knm (t) denote te number of application k requests from user n tat are routed to edge-cloud m in slot t and let r(t) denote te collection {r knm (t)} for all k, n, m. Routing of tese requests incurs a transmission cost of r knm (t)c knm (t) were c knm (t) is te unit transmission cost tat can depend on te current location of te user L n (t), te application index k, as well as te edge-cloud index m. More generally, it could also depend on oter uncontrolble factors suc as background backaul traffic and wireless fading, but we do not consider tese for simplicity. Denote te sum total transmission cost incurred in slot t by C(t), i.e., C(t) = knm r knm(t)c knm (t). For eac (k, m), we denote by R (t) te total number of application k requests received by edge-cloud m in slot t, i.e., R (t) = N n=1 r knm(t). We assume tat te maximum number of requests for an application k tat can be routed to edge-cloud m in any slot is upper bounded by R max. Given tese assumptions, te routing decisions r(t) are subject to te following constraints A kn (t) = 0 N n=1 M r knm (t) m=1 r knm (t) R max k, n k, m were (1) captures te assumption tat no request buffering appens at te users. In addition to (1) and (2), tere can be oter location-based constraints tat limit te set of edge-clouds were requests from user n can be routed given its location L n (t). Given l(t) = l, a(t) = a, denote te feasible request routing set by R(l, a). We assume tat R(l, a) for all l L, a A. (1) (2)

4 4 R. Urgaonkar et al. / Performance Evaluation ( ) Application configuration of edge-clouds: For all k, m, define application pcement variables H (t) as 1 if edge-cloud m osts application k in slot t, H (t) = 0 else. Te collection {H (t)} is denoted by te vector (t). Tis defines te application configuration of te edge-clouds in slot t and determines te local service rates {µ (t)} offered by tem in tat slot. An application s requests can only be serviced by an edge-cloud if it osts tis application in tat slot. Tus, µ (t) = 0 if H (t) = 0. Wen H (t) = 1, ten µ (t) is assumed to be a general non-negative function ϕ ( ) of te vector (t), i.e., µ (t) = ϕ ((t)). Tis results in a very general model tat can capture corretions between te service rates of co-located applications. A special case is were µ (t) depends only on H (t). For simplicity, we assume tat µ (t) is a deterministic function of (t) and use µ (t) to mean ϕ ((t)). Furter, we assume tat tere exist finite constants µ max suc tat µ (t) µ max for all t. An edge-cloud is typically resource constrained and may not be able to ost all applications. In general, osting an application involves creating a set of virtual macines (VMs) or execution containers (e.g., Docker) and assigning tem a vector of computing resources (suc as CPU, memory and storage) from te pysical macines (PMs) in te edge-cloud. We say tat an application configuration (t) is feasible if tere exists a VM-to-PM mapping tat does not viote any resource constraints. Te set of all feasible application configurations is denoted by H and is assumed to be finite. We also assume tat tere is a system-wide controller tat can observe te state of te system and cange te application configuration over time by using tecniques suc as VM migration and replication. Tis enables te controller to adapt in response to te system dynamics induced by user mobility as well as demand variations. However, suc reconfiguration incurs a cost tat is a function of te degree of reconfiguration. Given any two configurations a, b H, te cost of switcing from a to b is denoted by W ab. For simplicity, we assume tat it is possible to switc between any two configurations a, b H and tat W ab is upper bounded by a finite constant W max. We furter assume, witout loss of generality, tat W ab W ac + W cb a, b, c H. Te st assumption is valid if W ab is te minimum cost required to switc from a to b, Tis is because if W ab > W ac + W cb, ten we could carry out te reconfiguration from a to b by switcing from a to c and ten to b, acieving lower cost. Denote te switcing cost incurred in slot t by W(t). For simplicity, we assume tat switcing incurs no dey wile noting tat our model can be extended to consider suc deys (for example, by setting te local service rates to zero during tose slots wen a reconfiguration is underway). Request queues at te edge-clouds: As illustrated in Fig. 1, every edge-cloud m maintains a request queue U (t) per application k tat buffers application k requests from all users tat are routed to edge-cloud m. Requests in U (t) can get serviced locally in a slot by edge-cloud m if it osts application k in tat slot. In addition, buffered requests in U (t) can also be routed to te back-end cloud wic is assumed to ost all applications at all times. However, tis incurs additional back-end transmission cost as discussed ter. Te queueing dynamics for U (t) is given by U (t + 1) = max[u (t) µ (t) υ (t) + R (t), 0] (5) were υ (t) denotes te number of requests from U (t) tat are transmitted to te back-end cloud in slot t and µ (t) is te local service rate. It is assumed tat te requests in U (t) are serviced in a FIFO manner. Te collection of all queue backlogs {U (t)} is denoted by te vector U(t). From (5), note tat requests generated in a slot can get service in te same slot. It sould also be noted tat requests can be routed to U (t) in a slot even if H (t) = 0. Edge-cloud to back-end cloud request routing: Te back-end cloud is assumed to ost all applications at all times. However, transmitting requests to te back-end may incur very ig costs and terefore it is desirable to maximize te fraction of requests tat can be serviced locally by te edge-clouds. Let υ (t) denote te number of requests from U (t) tat are transmitted to te back-end cloud in slot t and let υ(t) denote te collection {υ (t)} for all k, m. Routing of υ (t) incurs a transmission cost of υ (t)e (t) were e (t) is te unit back-end transmission cost tat can depend on te application index as well as te edge-cloud index. Simir to request routing costs c knm (t), e (t) can also depend on oter uncontrolble factors (suc as background backaul traffic), but we only consider te average impact of tese for simplicity. Since bot te edge-clouds and te back-end cloud are static, we ave e (t) = e for all t. We assume tat υmax tere exist finite constants υ max suc tat υ (t) υ max for all t. Furter, Rmax wic models te baseline scenario were all requests are serviced only by te back-end cloud. Denote te set of all υ(t) tat satisfy tese constraints by V and te sum total back-end transmission cost incurred in slot t by E(t), i.e., E(t) = υ (t)e. We assume tat te back-end cloud as sufficient processing capacity suc tat it can service all requests in υ(t) wit negligible dey. Tus, queueing in te back-end becomes trivial and is ignored. It sould be noted tat in our model any user request tat is eventually serviced by te back-end cloud is transmitted first to an edge-cloud. Performance objective: Given tis model, our goal is to design a control algoritm for making request routing decisions at te users and edge-clouds as well as application reconfiguration decisions across te edge-clouds so tat te time-average overall transmission and reconfiguration costs are minimized wile serving all requests wit finite dey. Specifically, we assume tat te time-average dey for te requests in eac queue U (t) sould not exceed d avg, were d avg is a finite constant. Tis can be formuted as a constrained Markov Decision Problem (MDP) [5,6] as sown in Section 3.2. Timing of events in a slot: We assume te following sequence of events in a slot. At te start of slot t, te controller observes te queue backlogs U(t), new arrivals a(t), user locations l(t), and te st configuration (t 1). Ten it makes a (3) (4)

5 R. Urgaonkar et al. / Performance Evaluation ( ) 5 reconfiguration decision tat transitions te configuration state to (t) and tis determines te local service rates offered in slot t. Ten te controller makes user to edge-cloud and edge-cloud to back-end cloud routing decisions. Te queue backlogs U(t + 1) at te start of te next slot evolve according to (5) MDP formution Te control problem described in Section 3.1 can be formuted as a constrained MDP over te joint state space (l(t), a(t), (t), U(t)). It is well-known tat if tis problem is feasible, ten an optimal control policy for tis MDP can be obtained by searcing over te css of stationary, randomized control algoritms tat take control actions purely as a function of te system states [5,6]. Specifically, consider a control algoritm tat operates as follows. First, given st state = (l(t 1) = l, a(t 1) = a, (t 1) =, U(t 1) = u ) and current location, arrival, and queue backlog states l(t) = l, a(t) = a and U(t) = u, te control algoritm cooses current configuration (t) = wit probability z, tereby transitioning te state to = (l(t) = l, a(t) = a, (t) =, U(t) = u). Note tat te routing decisions taken in te st slot togeter wit its configuration determine U(t) troug te queueing equations in (5). Denote te resulting transition probability from U(t 1) = u to U(t) = u by s u. Ten te total expected reconfiguration u cost incurred wen transitioning from state is given by W = p l l q a a s u u z W. (6) Next, given current state = (l(t) = l, a(t) = a, (t) =, U(t) = u), te control algoritm cooses routing vector r(t) = r wit probability x (r) subject to r R(l, a). Tis incurs a total expected transmission cost given by C = r knm c knm. (7) r R(l,a) x (r) knm Finally, given current state = (l(t) = l, a(t) = a, (t) =, U(t) = u), it cooses back-end routing vector υ(t) = υ wit probability y (υ) subject to υ V and tis incurs a total expected back-end transmission cost given by E = y (υ) υ V υ e. (8) Let us denote te steady state probability of being in state under tis policy by π. Ten, te overall time-average expected transmission plus reconfiguration cost is given by C + E + W = π C + π E + π W. (9) Let U and R denote te time-average expected values of U (t) and R (t) under tis control algoritm. By Little s Teorem, we ave tat te average dey D for te requests in queue U (t) satisfies U = R D. In order to meet te average dey constraint, we need tat D = U R d avg k, m. Te constrained MDP optimization searces for an optimal policy tat minimizes C + E + W subject to meeting te average dey constraint. Assuming tat te problem is feasible, let c, e, and w denote te optimal time-average user-toedge-cloud transmission cost, edge-cloud-to-back-end cloud transmission cost, and total reconfiguration cost respectively. Solving tis optimization is extremely callenging and quickly becomes intractable due to te complexity of te state space under traditional solution metods (suc as value iteration [5,6]). Furter, tese tecniques require knowledge of te transition probabilities of te user mobility and request arrival processes tat may not be known a priori. In te following, we develop an alternate metodology for tackling tis problem tat overcomes tese callenges. Specifically, we take te following approac. (1) We first rex tis MDP by repcing te time-average dey constraints by queue stability constraints. Tis results in an MDP wose state space involves only (l(t), a(t), (t)). (2) For tis rexed MDP, we prove a novel decoupling property wic sows tat it can be decoupled into two independent MDPs tat evolve on disjoint state spaces. (3) Tis decoupling property enables us to develop an online control algoritm tat does not require any knowledge of te probability distributions, yet can acieve (arbitrarily) close to optimal cost wile providing worst-case dey guarantees. Before proceeding, we make te following assumption about te above (non-rexed) MDP. Let R, µ and υ respectively denote te time-average request arrival rate, local service rate, and back-end routing rate for queue U (t) under te optimal control policy. For all k, m for wic R > 0, define ϵ = µ + υ R and let ϵ = min ϵ. Ten we assume tat ϵ is strictly positive, i.e., ϵ > 0. Note tat, in general, in a queue wit stocastic arrivals and service rates, if te average arrival rate is not smaller tan te service rate, ten te average dey becomes unbounded. Terefore tis assumption is not very restrictive.

6 6 R. Urgaonkar et al. / Performance Evaluation ( ) 4. MDP rexation and decoupling Consider a rexation of te original MDP discussed in Section 3.2 were we repce te average dey constraints by te following queue stability constraints k, m. µ + υ R ϵ if R > 0 (10) were R, µ and υ respectively denote te time-average expected arrival rate, local service rate and back-end routing rate under any control algoritm. It can be sown tat meeting tese constraints ensures tat all queues are rate stable [8]. Furter, we add te constraints tat C = c, E = e, and W = w. Tat is, we enforce te time-average transmission and switcing costs under te rexed problem to matc tose under te optimal solution to te original MDP. It is clear tat tis problem is a rexation of te original MDP since te solution to te original MDP is feasible for tis problem. However, a solution to te rexed problem will not necessarily satisfy te average dey constraints. An optimal stationary, randomized control algoritm for te rexed problem can be defined simirly to te original MDP and is described in Appendix A. Te motivation for considering tis rexation is tat, unlike te original MDP, it suffices to consider te reduced state space defined by (l(t), a(t), (t)) for tis problem. Tis follows by noting tat none of te constraints involve te queue backlogs. Furter, te rexed problem as an interesting decoupling property (discussed next) tat can be leveraged to design an online control algoritm tat can acieve close to optimal cost wile providing explicit worst-case dey guarantees (as sown in Section 5) Decoupling te rexed MDP We now sow tat a decoupled control algoritm is optimal for te rexed MDP defined above. Specifically, under tis decoupled algoritm, te control decisions for user request routing are taken purely as a function of (l(t), a(t)), tose for application reconfiguration are taken purely as a function of (t), and te back-end routing decisions are taken in i.i.d. manner every slot, independent of all states. As a result, under tis algoritm, te states (l(t), a(t)) and (t) become decoupled and evolve independently of eac oter. Note tat, in general, wen searcing for te optimal policy for te rexed MDP, one must consider te css of algoritms were te control decisions are taken as a function of te joint state (l(t), a(t), (t)). Under suc algoritms, te states (l(t), a(t)) and (t) would be coupled and teir evolution would not be independent. Tus, it is noteworty tat suc a decoupled policy exists. We next specify te decoupled control algoritm and sow tat it acieves te same time-average cost as te rexed MDP (and ence te original MDP). Te decoupled algoritm is defined in terms of te control decision probabilities (and resulting steady-state values) of te optimal solution to te rexed MDP. We use te superscript xdp to indicate te control actions and resulting steady state probabilities of te optimal solution to te rexed MDP wile te superscript dec is used for te decoupled control algoritm. We use te sortand notation = (l(t) = l, a(t) = a, (t) = ) and = (l(t 1) = l, a(t 1) = a, (t 1) = ). Let denote te steady-state probability of state under te optimal solution to te rexed MDP and sums tis over all states for a given configuration. Define H as te set of all > 0. Te decoupled algoritm as te following components: configuration states for wic Reconfiguration policy: Te reconfiguration policy is defined by probabilities θ dec wic denote te probability of switcing to configuration given tat te configuration in te st slot was. Tese probabilities are given by l a p l l q a a z xdp θ dec = if, H, l a 0 else. (11) Routing policy: Given l(t) = l, a(t) = a, coose a routing vector r R(l, a) wit probability ζ dec ζ dec (r) = x xdp (r) if 0 else > 0, (r) given by (12) were sums over all states for wic l(t) = l and a(t) = a. Back-end routing policy: In eac slot t, coose a back-end routing vector υ V wit probability ϑ dec (υ) given by ϑ dec (υ) = y xdp (υ). (13)

7 R. Urgaonkar et al. / Performance Evaluation ( ) 7 Let us denote te time-average arrival and service rates for queue U (t) under te decoupled algoritm by R dec, µdec and υ dec respectively. Ten we ave te following. Teorem 1. For te decoupled control algoritm defined by (11), (12), and (13), te following old: (1) Te time-average reconfiguration cost is equal to w. (2) Te time-average transmission cost is equal to c. (3) Te time-average back-end routing cost is equal to e. (4) For eac queue U (t), te time-average arrival and service rates R dec, µdec MDP and satisfy (10), i.e., µ dec + υdec Proof. See Appendix B. Rdec ϵ if Rdec > 0. and υdec are equal to tose under te rexed We empasize tat te time-average arrival and service rates R dec, µdec, and υdec need not be equal to te corresponding values for te original MDP, i.e., R, µ and υ dec. Teorem 1 can be intuitively expined by noting tat te probability θ is cosen to be equal to te fraction of time tat te rexed MDP cooses to switc to configuration given tat te st configuration was, in steady state. Simirly, ζ dec (r) is cosen to be equal to te fraction of time te rexed MDP cooses routing vector r R(l, a) given tat te current user location and request arrival states are (l, a), in steady state, and te same applies to te back-end routing decisions. Tus, it can be seen tat te decoupled control algoritm tries to matc te time-average costs of te rexed policy wile meeting te queue stability constraints. Note tat under te decoupled control algoritm, te reconfiguration and local servicing decisions are a function only of te configuration state wile te routing is only a function of te user location and arrival states (l, a). It sould also be noted tat in our model, te tter states (l, a) evolve on teir own, independent of te control actions of tis algoritm. On te oter and, te evolution of te configuration state is completely governed by tis control algoritm. Finally, we note tat te decoupled control algoritm is expressed in terms of te steady state probabilities and control actions of te optimal solution of te rexed MDP wic is itself ard to calcute and requires knowledge of te statistics of te mobility or arrival processes tat may not be avaible. However, our objective is not to calcute tis control algoritm explicitly. Rater, we will use its existence to obtain an alternate online control algoritm tat will track te performance of tis control algoritm. Te online algoritm does not require any knowledge of te statistics of te mobility or arrival processes and can be implemented efficiently. Furter, te online algoritm stabilizes all queues and provides worst-case dey bounds for all requests. Recall tat H is te set of all configuration states for wic l a > 0. It can be sown tat te reconfiguration policy (11) of te decoupled algoritm results in a finite state Markov cain M over te states in H (see Lemma 2 in Appendix B). Furter, all states H are positive recurrent. Suppose te Markov cain M is in configuration at time t and let T dec denote te time spent in oter configurations before returning to (recurrence time). Ten, by basic renewal teory [18], te following olds for all t. t+t dec 1 t+t dec 1 E µ dec(τ) E W dec (τ) τ=t = µ dec E, τ=t T dec = w. (14) E T dec Furter, te first and second moments of te recurrence times, i.e., E T dec 5. Online control algoritm and E (T dec) 2 are bounded. We now present an online control algoritm tat makes joint request routing and application configuration decisions as a function of te system state (l(t), a(t), (t), U(t)). However, unlike traditional MDP solution approaces suc as dynamic programming [5,6], tis algoritm does not require any knowledge of te transition probabilities tat govern te system dynamics. In addition to te request queues U (t), for eac (k, m) tis algoritm maintains te following dey-aware queues tat are used to provide worst-case dey guarantees for user requests (as sown ter in Teorem 3) and are simir to te dey-aware queues used in [8]. max[z (t) µ Z (t + 1) = (t) υ (t) + σ, 0] if U (t) > µ (t) + υ (t), 0 if U (t) µ (t) + υ (t) were 0 σ υ max are control parameters tat affect te dey guarantees offered by tis algoritm. Our algoritm also uses a control parameter V > 0 tat affects a cost-dey tradeoff made precise in Teorem 3. Denote te collection {Z (t)} by Z(t) and te collection {σ } by σ. We assume tat all request queues U (t) and dey-aware queues Z (t) are initialized to 0 at t = 0. As sown in te following, our online algoritm is designed to ensure tat all request and deyaware queues remain bounded for all t and tis guarantees a deterministic worst-case dey bound for eac request. Simir to te decoupled control algoritm defined by (11) (13), tis algoritm consists of decoupled components for routing and reconfiguration decisions. Te control decisions in eac component are made independently but tey are weakly coupled troug te queue backlogs U(t) and Z(t). In te following, we describe eac of tese components in detail. Te performance guarantees provided by our algoritm are presented in Section 6. (15)

8 8 R. Urgaonkar et al. / Performance Evaluation ( ) 5.1. User-to-edge-cloud request routing We first describe te user-to-edge-cloud routing component of te algoritm. In eac slot t, te routing decisions {r knm (t)} are obtained by solving te following optimization problem. Minimize U (t) + Vc knm (t) r knm (t) k,m subject to {r knm (t)} R(l, a) n were R(l, a) is defined by constraints (1), (2), r knm (t) Z 0 k, n, m, and oter location-based constraints as discussed in Section 3.1. Te resulting problem is an integer linear program (ILP) in te variables r knm (t). Furter, te problem is separable across k, i.e., it is optimal to solve K suc problems separately, one per application k. Wen R max (16) N n=1 A kn(t) for all k, m, te above optimization as a particurly simple solution tat can be obtained independently for eac user n and can be calcuted in closed-form as follows. For eac (k, n), set r knm (t) = A kn (t) for te particur edge-cloud m tat user n can route to (given its current location L n (t)) and tat minimizes U (t) + Vc knm (t). Set r knm (t) = 0 for all m m. Note tat c knm (t) depends on te current user location (L n (t)) as well as te indices of te application (k) and te edge-cloud (m). Tis algoritm can be viewed as a more general version of te Join te Sortest Queue policy wic uses only queue lengts. In contrast, ere a weigted sum of queue lengt and transmission cost is used to determine te sortest queue. More generally, (16) can be mapped to variants of matcing problems on bipartite graps. For example, consider te case were A max kn = 1 for all k, n, R max = 1 for all k, m, and N M. Ten (16) becomes an instance of te minimum weigt matcing problem on a bipartite grap formed between te N users and M edge-clouds. Tis can be solved in polynomial time using well-known metods (suc as in [19]). For more general cases, (16) becomes a generalized assignment problem tat is NP-ard. However, efficient constant factor approximation algoritms are known for suc problems [20]. As we sow in Teorem 3, using any suc approximation algoritm instead of te optimal solution to (16) ensures tat te overall cost of te online algoritm is witin te same approximation factor of te optimal cost. It sould be noted tat te routing component of te control algoritm considers only te current user location, request arrival, and queue backlog states to make decisions and is terefore myopic. Furter, it does not require any knowledge of te mobility/arrival model. We also note tat it does not depend on te application configuration state Edge-cloud to back-end cloud request routing Te back-end routing decisions {υ (t)} are obtained by solving te following optimization problem every slot. Minimize Ve U (t) Z (t) υ (t) subject to 0 υ (t) min[u (t), υ max ] k, m. (17) Tis problem is separable across (k, m) and as a simple solution given by υ (t) = min[u (t), υ max ] wen U (t) + Z (t) > Ve and υ (t) = 0 else. Simir to request routing, te back-end routing algoritm considers only current queue backlogs (as e is a constant) and does not require any knowledge of te user mobility or request arrival model. Furter, it does not depend on te application configuration state. Te structure of te back-end routing decisions results in te following bounds on U (t) and Z (t). Lemma 1. Suppose υ max for all t. Rmax for all k, m. Ten, under te back-end routing decisions resulting from (17), te following old U (t) U max = Ve + R max Z (t) Z max = Ve + σ. (18) (19) Proof. We sow tat (18) olds using induction. First, (18) olds for t = 0 since all queues are initialized to 0. Now suppose U (t) U max for some t > 0. Ten, we sow tat U (t + 1) U max. We ave two cases. First, suppose U (t) Ve. Ten, from queueing equation (5), it follows tat te maximum value tat U (t + 1) can ave is U (t) + R max Ve + R max = U max. Next, suppose Ve < U (t) U max. Ten, we ave tat U (t) + Z (t) > Ve and te solution to (17) cooses υ (t) = min[u (t), υ max ]. Since υmax Rmax, from queueing equation (5) it follows tat U (t + 1) U (t) U max. Te bound (19) follows simirly and its proof is omitted for brevity. In Teorem 3, we sow tat for any σ request tat gets routed to U (t). > 0, te above bounds result in deterministic worst case dey bounds for any

9 R. Urgaonkar et al. / Performance Evaluation ( ) 9 Fig. 2. Illustration of te directed acyclic grap on te application configuration states over a renewal frame. Frame f starts at slot t f wit te configuration canging from 0 to one of 1, 2, 3 and ends at slot t f + 4 wen te configuration becomes 0 again Application reconfiguration Te tird component of te online algoritm performs application reconfigurations over time. We first define te notion of a renewal state under tis reconfiguration algoritm. Consider any specific state 0 H and designate it as te renewal state. Te application reconfiguration algoritm presented in tis section is designed to operate over variable lengt renewal frames were eac frame starts wit te initial configuration 0 (excluded from te current frame) and ends wen it returns to te state 0 (included in te current frame). All application configuration decisions for a frame are made at te start of te frame and are recalcuted for eac new frame as a function of te queue backlogs at te start of te frame. Note tat te system configuration in te st slot of eac frame is 0. Eac visit to 0 defines a renewal event and initiates a new frame tat starts from te next slot and sts until (and including) te slot wen te next renewal event appens as illustrated by an example in Fig. 2. Te renewal event and te resulting frame lengt are fully determined by te configuration decisions of te reconfiguration algoritm, i.e., tey are deterministic functions of te configuration decisions. In te following, we denote te lengt of te f t renewal frame by T f and te starting slot of te f t renewal frame by t f. Note tat T f = t f +1 t f. For simplicity, we assume t 0 = 0. Recall tat H is te set of all configuration states for wic l a > 0. In principle, any state in H can be cosen to be te renewal state 0. However, H itself may not be known a priori. Furter, in practice, 0 sould be cosen as te configuration tat is likely to be used frequently by te optimal policy for te rexed MDP presented in Section 4. Here, we assume tat te reconfiguration algoritm can select a renewal state 0 H and leave te determination of optimal selection of 0 for future work. Let te collection of queue backlogs at te start of renewal frame f be denoted by {U (t f )} and {Z (t f )}. Ten te reconfiguration algoritm makes decisions on te frame lengt T f and te application configurations [(t f ), (t f + 1),..., (t f + T f 1)] by solving te following optimization at t f. Minimize T f 1 1 Jτ + VW(t f + τ) T f τ=0 G (t f, τ) subject to (t f + T f 1) = 0 (t f + τ) H \ 0 τ {0,..., T f 2} T f 1 (20) were G (τ, t f ) = U (t f ) + Z (t f ) µ (t f + τ) denotes te queue-lengt weigted service rate, W(t f + τ) denotes te reconfiguration cost incurred in slot (t f + τ), and J = J were J is a constant defined as J 2(µ max + υmax )2 + σ 2 + (Rmax )2. Note tat te constraint (t f + T f 1) = 0 enforces te renewal condition. Note also tat wen te frame starts (τ = 0), te configuration in te previous slot t f 1 was 0. Te problem above minimizes te ratio of te sum total penalty earned in te frame (given by te summation multiplying 1/T f above) to te lengt of te frame. Te penalty term is a sum of V times te reconfiguration costs (VW(t f + τ)) and te Jτ terms minus te queue-lengt weigted service rates ( G (t f, τ)). Since te sum of te Jτ terms grows quadratically wit frame size, tis discourages te use of longer frames. Note also tat since te overall objective only uses te queue backlog values at te start of te frame and since all te oter terms are deterministic functions of te sequence of configurations [(t f ),..., (t f + T f 1)], te optimization problem (20) can be mapped to a deterministic sortest pat problem involving T f stages. Specifically, as illustrated in Fig. 2, consider a directed acyclic grap wit T f +1 stages, one node eac in te first and te st stage (corresponding to configuration 0 ), and H 1 nodes per stage in all oter stages. For a fixed T f, te objective in (20) corresponds to finding te minimum cost pat from te first to te st node, were te weigt of eac directed edge ( i, j ) tat is τ + 1 ops away from te first node is equal to te terms Jτ + VW(t f + τ) G (t f, τ). Here, W(t f + τ) is te switcing cost between te configurations i and j wile G (t f, τ) corresponds to te queue-lengt weigted service rate acieved using configuration j. Given a T f, tis as a complexity O( H 2 T f ) and optimally solving (20) would require searcing over all T f 1 since T f itself is

10 10 R. Urgaonkar et al. / Performance Evaluation ( ) an optimization variable in tis problem. We next caracterize an important property of te optimal solution to (20) tat results in a significantly lower complexity Complexity reduction Consider any solution to (20) tat results in a frame lengt of T f and a configuration sequence given by [(t f ),..., (t f + T f 2), 0 ]. Ten we ave te following. Teorem 2. An optimal solution to (20) can be obtained by restricting to te css of policies tat perform eiter two or no reconfigurations per frame. Furter, te reconfigurations (if any) appen only in te first slot and te st slot of te frame. Proof. First consider te case 1 T f 2. By definition te st configuration in te frame must be 0. Simirly, te configuration in te previous slot before te start of te frame is 0. Tere can be at most one more configuration between tese. Tus, tere can be at most two reconfigurations in te frame. Next, consider te case T f > 2. Let te queue backlogs at te start of frame be U(t f ), Z(t f ) and suppose te optimal configuration sequence is given by [(t f ),..., (t f + T f 2), 0 ]. Denote te set of configurations in tis sequence by Ω(U(t f ), Z(t f )) and define opt (U(t f ), Z(t f )) as te configuration from tis sequence tat minimizes te following: opt (U(t f ), Z(t f )) = arg min U (t f ) + Z (t f ) µ (t f + τ). (21) Ω(U(t f ),Z(t f )) Now consider an alternate configuration sequence given by [ opt (u f ),..., opt (u f ), 0 ]. Te value of te summation in te objective of (20) under tis sequence cannot be rger tan tat under te sequence [(t f ),..., (t f + T f 2), 0 ]. Tis is because opt (U(t f ), Z(t f )) minimizes te terms G (t f, τ) by (21) wile te total reconfiguration cost 1 τ=0 VW(t f +τ) in te alternate sequence cannot exceed te total reconfiguration cost under [(t f ),..., (t f +T f 2), 0 ] by property (4). Te teorem follows by noting tat at most two reconfigurations are needed in te alternate sequence, one at te beginning and one at te end of te frame. For a given frame lengt T f, Teorem 2 reduces te complexity of solving (20) from O( H 2 T f ) to O( H ) since we only need to searc for one configuration per frame. In Section 5.3.2, we sow tat te reconfiguration algoritm can be furter simplified by finding a closed-form expression for te optimal frame lengt given a configuration. Tis frame lengt is O U (t f ) + Z (t f ) wic sows tat te frame lengt is always bounded, given tat {U (t f )} and {Z (t f )} are bounded (see Lemma 1). We also discuss special cases were (20) can be mapped to bipartite grap matcing problems. We will sow tat simir to te routing component, using any approximation algoritm instead of te optimal solution to (20) still ensures tat te overall cost of te reconfiguration algoritm is witin te same constant factor of te optimal cost (Teorem 3, part 3). We analyze te performance of te overall control algoritm, including te components for request routing (Sections 5.1 and 5.2) and te component for application reconfiguration in Section 6. Before proceeding, we note tat simir to te routing components, te reconfiguration algoritm does not require any statistical knowledge of te request arrival or user mobility processes. Furter, it depends only on te queue backlog at te start of te frame, tereby decoupling it from te routing decisions in tat frame. However, unlike te routing components tat make decisions on a per slot basis (using current system states), te reconfiguration algoritm computes te sequence of configurations once per frame (at te start of te frame) and implements it over te course of te frame Calcuting te optimal frame lengt Given tat a configuration state is used in a frame, we sow tat te optimal frame lengt T opt () can be easily calcuted, tereby furter reducing complexity. Specifically, let us denote te values of various terms in te objective of (20) wen configuration 0 or is used as follows: Θ( 0 ) = G 0 (t f, τ) = (U (t f ) + Z (t f ))µ 0, Θ() = G (t f, τ) = (U (t f )+Z (t f ))µ, W sum = W 0 +W 0, were µ 0, µ denote te service rate µ wen configurations 0 and are used. Also, let ˆB = J /2. Ten, in order to calcute te optimal frame lengt, we ave two cases. If no reconfiguration is done, ten frame lengt is 1 and te objective of (20) becomes Θ( 0 ). Else, if a reconfiguration is done, te frame lengt is at least 2 and te objective of (20) can rewritten as Θ( 0 ) + Θ()(T f 1) + ˆB (T f 1) + VW sum min. T f 2 T f Ignoring te constant terms, te above can be simplified to min ˆBT f + VW sum + Θ( 0 ) Θ(). T f 2 T f If VW sum + Θ( 0 ) Θ() 0, ten te optimal frame lengt is T f = 2. Else, by taking derivative, we get tat te optimal VW T f is one of te two integers closest to sum +Θ( 0 ) Θ(). Define T ˆB f () as te optimal frame lengt given tat a switcing

11 R. Urgaonkar et al. / Performance Evaluation ( ) 11 to configuration is done. Ten, to calcute te overall optimal frame lengt T opt (), we compare te value of te objective of (20) under T f () wit tat under frame lengt one (i.e., Θ( 0)) and select te one tat results in a smaller objective Bipartite matcing Note tat a complexity of O( H ) can still be ig if tere are a rge number of possible configurations. For example, consider te case were at most one application can be osted by any edge-cloud and were exactly one instance of eac application is allowed per slot across te edge-clouds. Assume M K. In tis case, H = M! wic is exponential in (M K)! M, K. Now assume tat te total reconfiguration cost is separable across applications, te service rate of eac edge-cloud is independent of configurations at te oter edge-clouds, and te oter settings are te same as in te above example. Ten (20) can be reduced to a maximum weigt matcing problem on a bipartite grap formed between K applications and M edge-clouds. Let m k denote te edge-cloud osting application k in te renewal state 0. Ten in te bipartite grap, te weigt for any edge (k, m) (wen m m k ) for a given frame lengt T f becomes (U (t f ) + Z k (t f ))µ k + (T f k 1)(U (t f ) + Z (t f ))µ V(W k,m k,m + W k,m,m ) J T f (T f 1) were W k 2 k,m k,m (W k,m,m ) denotes te reconfiguration cost k associated wit moving application k from edge-cloud m (m) k to edge-cloud m (m k ). Wen m = m k, te weigt for te edge (k, m) is simply (U (t f ) + Z k (t f ))µ k. k For a given T f, te optimal configuration tat solves (20) can be obtained by finding te maximum weigt matcing on tis bipartite grap and tis is polynomial in M, K [19]. Tis is in contrast to simply searcing over all H tat is exponential in M, K. For more general cases were eac application can ave multiple instances and multiple application instances are allowed on eac edge-cloud, te problem becomes a generalized assignment problem and constant factor approximation algoritms exist [20], as long as different application instances can be considered separately. 6. Performance analysis We now analyze te performance of te online control algoritm presented in Section 5. Tis is based on te tecnique of Lyapunov optimization over renewal periods [8,17] were we compare te ratio of a weigted combination of te Lyapunov drift and costs over a renewal period and te lengt of te period under te online algoritm wit te same ratio under a stationary algoritm tat is queue backlog independent. Tis stationary algoritm is defined simirly to te decoupled control algoritm given by (11) (13) and we use te subscript stat to denote its control actions and te resulting service rates and costs. Ten te reconfiguration and routing decisions are defined by probabilities θ stat, ζ stat (r), and ϑ stat (υ) tat are cosen to be equal to θ dec (r), and ϑ dec (υ) respectively. If te resulting expected total service rate of any dey-, ζ dec aware queue Z (t) is less tan σ, ten its back-end request routing is augmented by coosing additional υ (t) in an i.i.d. manner suc tat te expected total service rate becomes σ. It can be sown tat te resulting time-average backend routing cost under tis algoritm is at most e + (σ) were (σ) = max[σ µ dec υdec, 0]e. By comparing te Lyapunov drift plus cost of te online control algoritm over renewal frames wit tis stationary algoritm, we ave te following. Teorem 3. Suppose te online control algoritm defined by (16), (17), and (20) is implemented wit a renewal state 0 H using control parameters V > 0 and 0 σ υ max for all k, m. Denote te resulting sequence of renewal times by t f were f {0, 1, 2,...} and let T f = t f +1 t f denote te lengt of frame f. Assume t 0 = 0 and tat U (t 0 ) = 0, Z (t 0 ) = 0 for all k, m. Ten te following bounds old. (1) Te time-average expected transmission plus reconfiguration costs satisfy tf lim F F 1 f =0 E C(τ) + W(τ) + E(τ) F 1 f =0 E T f were B = (1 + Υ dec 0 )J /2 + δ(r max )2, Υ dec 0 = E 1 + c + w + e + (σ) + V 1) 0 T dec (T dec 0 E T dec 0 B (22), (σ) = max[σ µ dec υdec, 0]e and δ is an O(log V) parameter tat is a function of te mixing time of te Markov cain defined by te user location and request arrival processes wile all oter terms are constants (independent of V ). (2) For all k, m, te worst-case dey d max d max U max + Z max σ for any request routed to queue U (t) is upper bounded by = 2Ve + R max + σ σ. (23)

12 12 R. Urgaonkar et al. / Performance Evaluation ( ) (3) Suppose we implement an algoritm tat approximately solves (16) and (20) resulting in te following bound for all slots for some ρ 1 U (t) + Vc knm (t) r apx (t) ρ knm U (t) + Vc knm (t) r opt knm (t) (24) n n and te following bound for every renewal frame 1 T apx f T apx 1 f τ=0 B + ρ T opt f ρj τ U (t f ) + Z (t f ) µ apx (t f + τ) + VW apx (t f + τ) T opt 1 f τ=0 J τ U (t f ) + Z (t f ) µ opt (t f + τ) + VW opt (t f + τ) (25) for some constant B were te subscripts apx and opt denote te control decisions and resulting costs under te approximate algoritm and te optimal solution to (16) and (20) respectively. Ten te time-average expected transmission plus reconfiguration costs under tis approximation algoritm is at most 1 + B + (R max ρ c + w + e + (σ) + V wile te dey bounds remain te same as (23). Proof. See Appendix C. + σ )υ max B (R max + σ )υ max + V Discussion on performance tradeoffs: Our control algoritm offers tradeoffs between cost and dey performance guarantees troug te control parameters V and σ. For a given V and σ, te bound in (22) implies tat te time-average expected transmission plus reconfiguration costs are witin an additive term (σ) + O(log V/V) term of te optimal cost wile (23) bounds te worst case dey by O(V/σ ). Tis sows tat by increasing V and decreasing σ, te time-average cost can be pused arbitrarily close to optimal at te cost of an increase in dey. Tis cost-dey tradeoff is simir to te results in [7,8] for non-mdp problems. Tus, it is noteworty tat we can acieve simir tradeoff in an MDP setting. Note tat if tere exists σ > 0 k, m suc tat σ µ dec + υdec, ten (σ) = 0 and te tradeoff can be expressed purely in terms of V. Also note tat since te average dey is upper bounded by te worst case dey, our algoritm provides an additive approximation wit respect to te cost c +w +e and a multiplicative approximation wit respect to te average dey d avg of te optimal solution to te original MDP defined in Section 3.2. In practice, setting σ = 0 k, m sould yield good dey performance even toug (23) becomes unbounded. Tis is because our control algoritm ensures tat all queues remain bounded even wen σ = 0 (see Lemma 1). Tis ypotesis is confirmed by te simution results in te next section. 7. Evaluations We evaluate te performance of our control algoritm using simutions. To sow bot te teoretical and real-world beaviors of te algoritm, we consider two types of user mobility traces. Te first is a set of syntetic traces obtained from a random-walk user mobility model wile te second is a set of real-world traces of San Francisco taxis [21]. We assume tat te edge-clouds are co-located wit a subset of te basestations of a cellur network. A exagonal symmetric cellur structure is assumed wit 91 cells in total as sown in Fig. 3. Out of te 91 basestations, 10 ost edge-clouds and tere are 5 applications in total. For simplicity, eac edge-cloud can ost at most one application in any slot in te simution. Furter, tere can be only one active instance of any application in a slot. Te transmission and reconfiguration costs are defined as a function of te distance (measured by te smallest number of ops) between different cells. Wen a user n in cell l routes its request to te edge-cloud in cell l, we define its transmission cost as trans n (l, l dist(l, l ) = ), if l l 0, if l = l (27) were dist(l, l ) is te number of ops between cells l and l. Te reconfiguration cost of different applications is assumed to be independent. For any application k tat is moved from te edge-cloud in cell l to te edge-cloud in cell l, te reconfiguration cost for tis specific application is defined as recon k (l, l κ( dist(l, l ) = )), if l l 0, if l = l (28) were κ is a weigting factor to compare te reconfiguration cost to te transmission cost. Te total reconfiguration cost is te sum of reconfiguration costs across all k. In te simutions, we consider two cases in wic κ takes te values 0.5 and (26)

13 R. Urgaonkar et al. / Performance Evaluation ( ) 13 Fig. 3. Illustration of te exagonal cellur structure sowing distance between 2 cells. 1.5 respectively, to represent cases were te reconfiguration cost is smaller/rger tan te transmission cost. Bot cases can occur in practice depending on te amount of state information te application as to transfer during reconfiguration. Te back-end routing cost is fixed as a constant 2 for eac request. Eac user generates requests for an application according to a fixed probability λ per slot. However, te number of active users in te system can cange over time. Tus, te aggregate request arrival rate across all users for an application varies as a function of te number of active users in a slot. In our study of syntetic mobility traces, we assume tat te number of users is fixed to 10 and all of tem are active. However, te real-world mobility trace as a time-varying number of active users. In bot cases, λ is te time-average (over te simution duration) aggregate arrival rate per application per slot, wile te edge cloud service rate for an active application instance is 1 per slot, and te back-end cloud service rate for eac application is 2 per slot. Te request arrivals are assumed to be independent and identically distributed among different users, and tey are also independent of te past arrivals and user locations. We note tat optimally solving te original or even rexed MDP for tis network is igly callenging. Terefore, we compare te performance of our algoritm wit tree alternate approaces tat include never/always migrate policies and a myopic policy. In te never migrate policy, eac application is initially pced at one particur edge-cloud and reconfiguration never appens. User requests are always routed to te edge-cloud tat osts te corresponding application. In te always migrate policy, user requests are always routed to te edge-cloud tat is closest to te user and reconfiguration is performed in suc a way tat te queues wit te rgest backlogs are served first (subject to te constraint tat eac edge-cloud can only ost one application). We also assume tat te request arrival rate λ is known in te never and always migrate policies. If λ > 1, te arrival rate exceeds te edge-cloud capacity, and te requests tat are queued in edge-clouds are probabilistically routed to te back-end cloud, were te probability is cosen suc tat te average arrival rate to edgeclouds does not exceed te service rate at edge clouds. Finally, te myopic policy considers te transmission, reconfiguration, and back-end routing costs jointly in every slot. Specifically, in eac slot, it calcutes a routing and configuration option tat minimizes te sum of tese tree types of costs in a single slot, were it is assumed tat a user routes its request eiter to te back-end cloud or to te edge-cloud tat osts te application after possible reconfiguration. Te online algoritm itself is implemented by making use of te structure of te optimal solution as discussed in Section 5. Specifically, we implement te request routing part (16) by solving te bipartite max-weigt matcing problem as discussed in Section 5.1 wile te application reconfiguration part (20) uses te tecniques in Sections and Because te proposed online algoritm is obtained using a (loose) upper bound on te drift-plus-penalty terms, te actual drift-pluspenalty value can be muc smaller tan te upper bound. We take into account tis fact by adjusting te constant terms J in (20). We set J = 0.2 in te simution wic is a reasonably good number tat we found experimentally Syntetic traces We first evaluate te performance of our algoritm along wit te tree alternate approaces on syntetic mobility traces. Te syntetic traces are obtained assuming random-walk user mobility. Specifically, at te beginning of eac slot, a user moves to one of its neigboring cells wit probability 1/7 for eac cell, and it stays in te same cell wit probability 1/7. Wen te number of neigboring cells is less tan six, te corresponding probability is added to te probability of staying in te same cell. Suc a mobility model can be described as a Markov cain and terefore our teoretical analysis applies. Tere are 10 users in tis simution, and we simute te system for 100,000 slots. Te average queue lengt and te average transmission plus reconfiguration plus back-end routing costs over te entire simution duration are first studied for different values of te control parameters V as well as {σ }. Specifically, we set all σ to te same value σ wic is cosen from σ {0, 0.1, 0.5}. Te performance results for all four algoritms under tese scenarios are sown in Fig. 4 for bot values of κ, were we set λ = We can see from te results tat, for eac fixed σ, te queue lengts and cost values under te Lyapunov algoritm follow te O(V, log V/V) trend as suggested by te bounds (22) and (23). Te impact of te value of σ is also as predicted by tese bounds. Namely, a smaller value of σ yields rger queue lengts and lower costs, wile a rger value of σ yields smaller queue lengts and iger costs. Wen comparing all four algoritms in Fig. 4(a), (b) were κ = 0.5, it can be seen tat

14 14 R. Urgaonkar et al. / Performance Evaluation ( ) a b c d Fig. 4. Average queue lengts and costs for syntetic user mobility wit different V and σ values. Subfigures (a) and (b) are results for κ = 0.5, and subfigures (c) and (d) are results for κ = 1.5. a b c d Fig. 5. Average queue lengts and costs for syntetic user mobility wit different λ values. Subfigures (a) and (b) are results for κ = 0.5, and subfigures (c) and (d) are results for κ = 1.5. wile te never/always migrate policies ave smaller queue backlogs, tey incur more cost tan te Lyapunov algoritm. Note tat, unlike te Lyapunov algoritm, none of te alternate approaces offer a mecanism to trade off queue backlog (and ence average dey) performance for a reduction in cost. For te case κ = 1.5, simir beavior is seen as illustrated by Fig. 4(c), (d). We next study te queue lengts and costs under different values of te arrival rate λ, were we fix V = 100 and σ = 0. Results are sown in Fig. 5. We can see tat wit te myopic policy, te queue lengts are very rge and in fact become unbounded. Tis is because te myopic policy does not try to matc te edge-cloud arrival rate wit its service rate, and it is also independent of te queue backlog. Because te one-slot cost of routing to an edge-cloud is usually lower tan routing to te back-end cloud, an excessive amount of requests is routed to edge-clouds exceeding teir service capacity. Te never and always migrate policies ave low queue backlogs because we matced te request routing wit te service rate of edge-clouds, as expined earlier. However, tey incur iger costs as sown in Fig. 5(b), (d). More importantly, tey require prior knowledge on te arrival rate, wic it is usually difficult to obtain in practice Real-world mobility To study te performance under more realistic user mobility, we use real-world traces of San Francisco taxis [21] tat is a collection of GPS coordinates of approximately 500 taxis collected over 24 days in te San Francisco Bay Area. In our simution, we select a subset of tis data tat corresponds to a period of 5 consecutive days. We set te distance between basestations (center of cell) to 1000 m, and te exagon structure is pced onto te geograpical location. User locations are ten mapped to te cell location by considering wic cell te user lies in. In tis dataset, tere are 536 unique users in total, and not all of tem are active at a given time. Te number of active users at any time ranges from 0 to 409, and 278 users are active on average. We assume tat only active users generate requests suc tat te average arrival rate over te entire duration is λ = 0.95 for eac application. Wit tis model, wen te number of active users is rge (small), te instantaneous arrival rate can be iger (lower) tan te edge-cloud service rate. Te underlying mobility pattern in tis scenario can be quite different from a stationary Markov model and exibits non-stationary beavior. We set te timeslot lengt as 1 s and fix V = 100, σ = 0 for te Lyapunov algoritm. Te purpose of tis simution is to study te temporal beavior of queue lengts and cost values under our algoritm and compare wit te alternate approaces. We find tat wile te queue lengts cange retively slowly, te per slot costs fluctuate rapidly. Terefore, we measure te moving average of te costs over an interval of size 6000 s for all algoritms. Figs. 6 and 7 sow te results respectively for te case κ = 0.5 and κ = 1.5, and te average values across te entire time duration are given in Table 1. Tere are several noteworty observations. From Table 1, we can see tat even toug σ = 0, te average queue lengt under te Lyapunov approac is significantly lower tan all oter approaces, wile te cost of te Lyapunov approac is lower tan all oter approaces wen κ = 0.5 and only sligtly iger tan te never migrate and myopic policies wen κ = 1.5. Tis confirms tat te proposed Lyapunov algoritm as promising performance wit real-world user traces. As sown in Figs. 6 and 7, te cost results sow a noticeable diurnal beavior wit 5 peaks and valleys tat matc wit te

15 R. Urgaonkar et al. / Performance Evaluation ( ) 15 Table 1 Average values for trace-driven simution. Policy Queue lengts (κ = 0.5) Costs (κ = 0.5) Queue lengts (κ = 1.5) Costs (κ = 1.5) Lyapunov Never migrate Always migrate Myopic a b Fig. 6. Instantaneous queue lengts and moving average of costs for trace-driven simution wit κ = 0.5. a b Fig. 7. Instantaneous queue lengts and moving average of costs for trace-driven simution wit κ = day simution period. Te cost of te Lyapunov algoritm becomes iger tan some oter approaces at peaks, wic is mainly due to te presence of back-end routing. At te same time, owever, te difference between te queue lengt of te Lyapunov algoritm and te oter approaces is also rger at suc peaks. We see tat te Lyapunov approac as te lowest variation in its queue lengt, wic is a consequence of our design goal of bounding te worst-case dey. Te queue lengts of te oter approaces fluctuate more, and te always migrate policy appears to be unstable as te queue backlogs grow unbounded. 8. Conclusions In tis paper, we ave developed a new approac for solving a css of constrained MDPs tat possess a decoupling property. Wen tis property olds, our approac enables te design of simple online control algoritms tat do not require any knowledge of te underlying statistics of te MDPs, yet are provably optimal. Te resulting solution is markedly different from cssical dynamic programming based approaces and does not suffer from te associated curse of dimensionality or convergence issues. We applied tis tecnique to te problem of dynamic service migration and workload sceduling in te emerging area of edge-clouds and sowed ow it results in an efficient control algoritm for tis problem. Our overall approac is promising and could be useful in a variety of oter contexts. Acknowledgments Tis researc was sponsored in part by te US Army Researc Laboratory and te UK Ministry of Defence and was accomplised under Agreement Number W911NF Te views and conclusions contained in tis document are

16 16 R. Urgaonkar et al. / Performance Evaluation ( ) tose of te autor(s) and sould not be interpreted as representing te official policies, eiter expressed or implied, of te US Army Researc Laboratory, te US Government, te UK Ministry of Defence or te UK Government. Te US and UK Governments are autorized to reproduce and distribute reprints for Government purposes notwitstanding any copyrigt notation ereon. Appendix A. Rexed MDP formution An optimal stationary, randomized control algoritm for te rexed MDP can be defined as follows. First, given st state = (l(t 1) = l, a(t 1) = a, (t 1) = ) and current states l(t) = l, a(t) = a, it cooses configuration (t) = wit probability z were = (l(t) = l, a(t) = a, (t) = ). Te total expected reconfiguration cost incurred wen transitioning from state is given by W = p l l q a a z W. (29) Next, given current state = (l(t) = l, a(t) = a, (t) = ), it cooses a routing vector r(t) = r wit probability x (r) subject to te constraint tat r R(l, a) and incurs a total expected transmission cost of C = r knm c knm. (30) r R(l,a) x (r) knm Finally, given current state = (l(t) = l, a(t) = a, (t) = ), it cooses a back-end routing vector υ(t) = υ wit probability y (υ) subject to υ V and tis incurs a total expected back-end transmission cost given by E = y (υ) υ V υ e. (31) Let us denote te steady state probability of being in state under tis policy by π. Ten, te overall time-average expected transmission plus reconfiguration cost satisfies C + E + W = π C + π E + π W = c + e + w. (32) Note tat te rexed MDP still as a ig-dimensional state space making it impractical to solve using standard tecniques. Furter, we still need knowledge of all transition probabilities. Appendix B. Cost optimality of decoupled MDP Here we prove Teorem 1 troug a series of lemmas. For notational convenience, we use and intercangeably trougout tis section. Recall tat H is te set of all configuration states for wic l a > 0. Ten for all, H, we define matrix F = (θ dec dec ) were θ is given by (11). Lemma 2. F is a stocastic matrix. Proof. In order for F = (θ dec ) to be a stocastic matrix, we need H θ dec = 1 for all H. Fix an H. Using (11), we ave = θ dec = H l a p l l q a a z xdp l a p l l q a a z xdp = 1. H H θ dec = l a = l a Since tis olds for all H, te lemma follows. From Lemma 2, it follows tat F can be tougt of as te transition probability matrix for te Markov cain over te set H of configuration states tat results from te reconfiguration policy (11) of te decoupled control algoritm. Lemma 3. For all H, te fraction of time spent in configuration under te reconfiguration policy (11) of te decoupled control algoritm is equal to te fraction of time in configuration under te rexed MDP. Proof. First, consider any state H. Let π F denote te steady state probability for state in te Markov cain defined by transition probability matrix F. Tis is well-defined and exists according to Lemma 2. It is well known tat tese steady state probabilities can be obtained as te unique solution to te system of equations π F = H π F θ dec for all. We now

17 R. Urgaonkar et al. / Performance Evaluation ( ) 17 sow tat coosing π F = l a l a for all H satisfies tis system of equations. Upon substituting π F = l a l a, te RHS H π F θ dec becomes π F θdec H = H l a = H l a l a θ dec l a = H l a p l l q a a z xdp l a, l a = l a l a l a p l l q a a z xdp l a, l a l a H l a p l l q a a z xdp l a, = were we used (11) in te second step. Simirly, upon substituting π F = l a l a, te LHS π F becomes: π F = l a l a =. Tis implies tat π F = for all H. Tis sows tat te fraction of time spent in configuration under te reconfiguration policy (11) of te decoupled control algoritm, i.e., π F is equal to wic is te fraction of time in configuration under te rexed MDP. For te case wen H \ H, Lemma 3 follows by noting tat te fraction of time spent in configuration in bot cases is 0. Lemma 4. Te time-average expected reconfiguration cost under te decoupled control algoritm is equal to te time-average expected reconfiguration cost of te rexed MDP, i.e., w. Proof. Te time-average expected reconfiguration cost under policy (11) is H π F (from Lemma 3) and using (11), we ave: π F H H θ dec W = H = H = H H were te st step follows from (29) and (32). Teorem 1 part 1 follows from Lemma 4. l a p ll q aa z xdp l a p ll q aa z xdp,l a W p ll q aa z xdp,l a W l a =,l a W W xdp = w H θ dec W. Substituting π F = Lemma 5. Under te routing policy defined by (12), te time-average transmission cost of te decoupled algoritm is equal to tat under te rexed MDP, i.e., c. Proof. Let π denote te fraction of time te decoupled MDP is in state (l(t) = l, a(t) = a). Te time-average transmission cost under te routing policy defined by te probabilities in (12) is given by x xdp r knm c knm π r R(l,a) = π ζ dec (r) r knm c knm = knm r R(l,a) π r R(l,a) x xdp (r) r knm c knm knm = π (r) knm C xdp = C xdp = c were in te second step we sum over only tose (l, a) for wic π > 0. In te second st step above, we used te fact tat π = H, i.e., te fraction of time te rexed MDP is in state (l(t) = l, a(t) = a) is equal to π. Tis is because tese states evolve independent of te control actions of eiter control algoritms. Tis sows Teorem 1 part 2. Lemma 6. Under te back-end routing policy defined by (13), te time-average back-end transmission cost of te decoupled algoritm is equal to tat under te rexed MDP, i.e., e.

18 18 R. Urgaonkar et al. / Performance Evaluation ( ) Proof. Te time-average transmission cost under te back-end routing policy defined by (13) is given by ϑ dec (υ) υ e = y xdp (υ) υ e υ V υ V = y xdp (υ) υ e = E xdp = e υ V were we used (31) in te second st step. Tis sows Teorem 1 part 3. Finally, we sow tat time-average arrival and service rates are equal to tose under te rexed MDP. Lemma 7. Te time-average arrival and service rates for eac queue U (t) under te decoupled MDP are equal to tose under te rexed MDP. Proof. Te fraction of time a routing vector r R(l, a) is cosen under te rexed MDP is given by te decoupled MDP, tis becomes x xdp (r) π ζ dec (r) = π = x xdp (r). x xdp (r). Under Tis sows tat te time-average arrival rate to any queue is te same under bot algoritms. A simir result olds for average local service rates since tese are only a function of te fraction of time spent in any configuration wic is te same under bot MDPs. Finally, te fraction of time a back-end routing vector υ V is cosen under te rexed MDP is given by y xdp (υ). Under te decoupled MDP, tis is equal to ϑ dec (υ) = y xdp (υ) by definition (13). Teorem 1 part 4 follows by noting tat te time-average arrival and service rates to eac queue U (t) under te decoupled MDP satisfy (10). Appendix C. Proof of Teorem 3 Let Q (t) = (U(t), Z(t)) denote te collection of queue backlogs in slot t. Define L(Q (t)) 1 2 U 2 (t) + Z 2 (t) as a Lyapunov function of Q (t). Using (5) and (15), we can bound te one-slot difference L(Q (t + 1)) L(Q (t)) as L(Q (t + 1)) L(Q (t)) B u + Bz U (t) µ (t) + υ (t) R (t) Z (t) µ (t) + υ (t) σ were B u = ((µmax + υmax )2 + (R max )2 )/2 and B z = ((µmax + υmax )2 + σ 2 )/2. Note tat at any renewal time t f, te application configuration (t f ) = 0. For ease of notation, we denote te collection of all queue backlogs, user locations, and arrival vectors at te start of renewal frame f by (t f ). Define te frame-based conditional Lyapunov drift (t f ) as te expected cange in te value of L(Q (t)) over te f t frame given initial state (t f ), i.e., (t f ) E L(Q (t f +1 )) L(Q (t f )) (t f ). In te following, we use te sortand notation {.} to represent te conditional expectation E. (t f ). Following te metodology in [8,17] and using te one-slot difference bound along wit deyed queueing equations, we can obtain te following upper bound on te frame-based conditional Lyapunov drift plus V times te conditional expected tf transmission plus reconfiguration cost, i.e., (t f ) + V C(τ) + W(τ) + E(τ) in terms of te control decisions under any algoritm (see Appendix D). X (T f ) U (t f ) + Z (t f ) tf µ (τ) tf U (τ) + Z (τ) υ (τ) + tf Z (τ)σ + tf tf U (τ)r (τ) + V Ec C(τ) + W(τ) + E(τ) (33)

19 R. Urgaonkar et al. / Performance Evaluation ( ) 19 were X (T f ) (B u + Bz )T f + T f (T f 1)(µ max + υmax )µmax. Given any location and arrival states and queue backlogs, te routing decisions under te online control algoritm minimize VC(t) + U (t)r (t) in every slot over all oter algoritms, including te backlog independent stationary algoritm. Simirly, te back-end routing decisions under te online control algoritm minimize VE(t) (U (t) + Z (t))υ (t) in every slot over all oter algoritms, including te stationary algoritm. Denoting tese decisions under te stationary algoritm by R stat costs by C stat (t) and E stat tf (t), it follows tat we can upper bound te value of (t f ) + V under te online control algoritm by X (T f ) U (t f ) + Z (t f ) tf µ (τ) tf U (τ) + Z (τ) υ stat (τ) + tf Z (τ)σ tf tf + V C stat (τ) + W(τ) + E stat (τ) + U (τ)r stat (τ) Next, we use te following bounds obtained using deyed queue equations (details in Appendix E): tf U (τ)υ stat (τ) U (t f )υ stat E (T f 1) c + (µ max 2 tf tf tf tf tf C stat (τ) E stat (τ) Z (τ)υ stat (τ) Z (t f )υ stat Z (τ)σ U (τ)r stat (τ) Z (t f )σ + + U (t f )(R stat δt f (R max + (T f 1) (µ max 2 (T f 1) σ αγ δ ) )2 + T f (T f 1) 2 (t) and υstat (t) and te resulting C(τ) + W(τ) + E(τ). (34) + υmax )υmax + υmax )υmax (35) (36) (37) (R max )2 + x (t f, t f +1 ) (38) (c + βγ δ ) + Ec y(tf, t f +1 ) (39) (e + (σ)) were x (t f, t f +1 ) and y(t f, t f +1 ) are defined as x (t f, t f +1 ) = t f +δ 1 tf +1+δ 1 U (τ)r stat (τ) +1 t f +δ 1 U (τ)r stat (τ), y(t f, t f +1 ) = t f +1 +δ 1 C stat (τ) +1 C stat (τ) and α 0, β 0, 0 < γ < 1 are constants tat caracterize ow fast te Markov cain defined by te user location and request arrival processes mixes to its steady-state distribution and δ 0 is an integer wose value will be specified in te following. Substituting tese, we get tf (t f ) + V X (T f ) C(τ) + W(τ) + E(τ) U (t f ) + Z (t f ) υ stat U (t f ) + Z (t f ) tf µ (τ) + Z (t f )σ (40)

20 20 R. Urgaonkar et al. / Performance Evaluation ( ) + tf U (t f )(R stat + αγ δ ) + V(c + e + βγ δ (σ)) + V Ec W(τ) + x (t f, t f +1 ) + V y(tf, t f +1 ) (41) were X (T f ) = T f + T f (T f 1) J /2 + δt f (R max )2. Next, note tat under te application reconfiguration and servicing decisions of te online control algoritm (20), te following olds for any (t f ). tf + V Ec X (T f ) η X stat ( U (t f ) + Z (t f ) tf ) η µ (τ) U (t f ) + Z (t f ) t f +T stat 1 f µ stat W(τ) (τ) + ηv t f +T stat 1 f W stat (τ) were η = {T f }. Here, µ stat(τ), W stat (τ), T stat f represent te control decisions and te resulting reconfiguration cost and T stat f frame lengt under te backlog independent stationary algoritm defined earlier. Tis follows by noting tat te online control algoritm is designed to minimize te ratio of te left and side to te frame lengt over all possible algoritms. Next, note tat te reconfiguration policy of te stationary algoritm is dependent only on te configuration state and is independent of all states in (t f ). Tus, te conditional expectation can be repced by regur expectation and te retionsips in (14) can be used since te reconfiguration policies of te decoupled control algoritm and te stationary algoritm are identical. Plugging tese, te above can be rewritten as X (T f ) U (t f ) + Z (t f ) tf tf µ (τ) + V Ec W(τ) (42) η X stat ( ) U (t f ) + Z (t f ) µ stat + Vw. (43) Using tis, te rigt and side of (41) can be bounded as η X stat ( Z (t f )(µ stat ) + υstat U (t f )(µ stat + υstat Rstat αγ δ ) σ ) + V(c + e + w + βγ δ + (σ)) + x (t f, t f +1 ) + V y(tf, t f +1 ). (44) Note tat te frame lengt T stat f under te stationary algoritm is independent of te initial state (t f ). Furter, te reconfiguration policy for te stationary algoritm and te decoupled algoritm is identical. Tus, te conditional expectation can been removed and te first term can be rewritten as η were Υ dec 0 X = E T dec stat ( ) = 1) 0 (T dec 0 E T dec 0 B + Z (t f )(µ stat = E X stat ( E T stat f ) E = E (1 + Υ dec 0 )J /2 + δ(r max )2 X dec (T 0 ) T dec 0. Denoting B = (1 + Υ dec 0 )J /2 + δ(r max )2, (44) can be rewritten as U (t f )(µ stat + υstat + υstat Rstat αγ δ ) σ ) + V(c + e + w + βγ δ + (σ)) x (t f, t f +1 ) + V y(tf, t f +1 ). (46) (45)

21 R. Urgaonkar et al. / Performance Evaluation ( ) 21 Note tat under te stationary algoritm, if R stat from Teorem 1 part 4 and te fact tat υ stat if R stat > 0. For tose k, m for wic µstat tat µ stat + υstat tf (t f ) + V σ. Tus, we ave > 0, ten µstat + υstat Rstat µdec + υdec Rdec ϵ > 0. Tis follows 2α/ϵ. Tus, by coosing δ log, we ave tat µstat 1/γ Rstat αγ δ ϵ/2 = 0, te α can be set to 0. Also, te stationary algoritm is designed suc υdec = Rstat C(τ) + W(τ) + E(τ) B + V(c + e + w + βγ δ + (σ)) + x (t f, t f +1 ) + V y(tf, t f +1 ). (47) Taking te expectation of bot sides and summing over f {0, 1, 2,..., F 1} yields F 1 tf E {Φ(U(t F ))} + V E C(τ) + W(τ) + E(τ) F 1 B E T f f =0 F 1 + V(c + e + w + βγ δ + (σ)) F 1 E T f + E x (t f, t f +1 ) F 1 + V E y(t f, t f +1 ). (48) Tis can be rearranged to get F 1 tf E C(τ) + W(τ) + E(τ) f =0 f =0 f =0 f =0 F 1 (c + e + w + βγ δ + (σ)) + F 1 f =0 f =0 E T f + V f =0 E x (t f, t f +1 ) F 1 + E y(t f, t f +1 ). V f =0 E T f F 1 B F 1 Te st two terms vanis wen we divide bot sides by f =0 E T f and take limit F. Te bound in Teorem 3 part log Vβ 1 follows by coosing δ. It can be seen tat tis term is O(log V). log 1/γ Part 2 of Teorem 3 follows by a direct application of Lemma 5.5 from [8]. Finally, part 3 of Teorem 3 is obtained by performing a frame-based conditional Lyapunov drift analysis for te approximate algoritm simir to te analysis above wile using te retions (24), (25) and is omitted for brevity. Appendix D. Frame-based Lyapunov drift bound Te one-slot difference L(Q (τ + 1)) L(Q (τ)) is upper bounded by B u + Bz U (t) µ (t) + υ (t) R (t) Z (t) µ (t) + υ (t) σ. Summing over t {t f,..., t f +1 1} yields te following bound on te frame-level difference L(Q (t f +1 )) L(Q (t f )): (B u + Bz )T f t f t f (U (τ) + Z (τ))υ (τ) + (U (τ) + Z (τ))µ (τ) + t f From (5), it follows tat U (τ) U (t f ) (τ t f )(µ max Z (τ) Z (t f ) (τ t f )(µ max + υmax (U (τ) + Z (τ))µ (τ) t f t f U (τ)r (τ) Z (τ)σ. (49) + υmax) for all τ t f. Simirly from (15), it follows tat ) for all τ t f. Using tis, te second term above can be lower bounded as t f (U (t f ) + Z (t f ))µ (τ) t f t f (τ t f )(µ max (U (t f ) + Z (t f ))µ (τ) T f (T f 1)(µ max f =0 + υmax)µ (τ) + υmax )µmax (33) follows by using te above in (49), adding te V t f (C(τ)+W(τ)+E(τ)) term to bot sides, and taking conditional expectation given (t f ).

22 22 R. Urgaonkar et al. / Performance Evaluation ( ) Appendix E. Deyed backlog bounds We first sow (35). From (5), it follows tat U (τ) U (t f ) (τ t f )(µ max t f tf +1 1 tf U (τ)υ stat (τ) U (t f )υ stat (τ) (τ t f )(µ max t f U (t f )υ stat (τ) + T f (T f 1) 2 (µ max + υmax) for all τ t f. Using tis, + υmax )υstat(τ) + υmax )υmax. (50) Te bound in (35) is obtained by summing (50) over all k, m, taking te conditional expectation of bot sides and using te fact tat te back-end routing decisions under te stationary algoritm are i.i.d. every slot. Te bounds (36) and (37) can be sown using a simir procedure and we omit te details for brevity. Next, we sow (38). For any integer δ 0, we can express t f t f tf +δ 1 tf U (τ)r stat (τ) = +1+δ 1 tf U (τ)r stat (τ) + +1+δ 1 U (τ)r stat (τ) +δ U (τ)r stat (τ) in terms of δ-sifted terms as +1 U (τ)r stat (τ) t f +1 +δ 1 = x (t f, t f +1 ) + U (τ)r stat (τ). (51) +δ From (5), it follows tat U (τ) U (t f ) + (τ t f )R max bounded as t f +1 +δ 1 +δ tf +1+δ 1 tf U (t f )R stat (τ) + +1+δ 1 (τ t f )R max Rstat(τ) +δ for all τ t f. Using tis, te second term above can be upper +δ U (t f )R stat (τ) + t f +1 +δ 1 Now consider te conditional expectation of te first term on te RHS above, i.e., +δ integer δ 0 δt f + T f (T f 1) 2 t f +1 +δ 1 t f +1 +δ 1 U (t f )R stat (τ) = U (t f ) R stat (τ) U (t f )(R stat + αγ δ ) +δ +δ (R max )2. U (t f )R stat (τ). We ave for any were te st step follows from te application of Lemma 8 and using te fact tat te control decisions R stat (τ) under te routing policy of te stationary algoritm are functions purely of te Markov cain defined by te user location and request arrival processes. (38) is obtained by summing (51) over all k, m, taking te conditional expectation, and applying te bound in (52). A simir approac can be used to sow (39). Finally, (40) follows by noting tat te back-end routing decisions under te stationary algoritm are taken in an i.i.d. manner every slot and te resulting time-average back-end routing cost is at most e + (σ). Lemma 8 (Markov Cain Convergence [22]). Let Z(t) be a finite state, discrete time ergodic Markov cain. Let S denote its state space and let {π s } s S be te steady state probability distribution. Let f (Z(t)) be a positive random function of Z(t). Define f = j S π jm j were m j E {f (Z(t)) Z(t) = j}. Ten tere exist constants α, γ suc tat for all integers d 0, we ave E {f (Z(t)) Z(t d) = i} f + sm max αγ d were m max max j S m j and s = card{s}. (52) References [1] M. Satyanarayanan, Cloudlets: At te leading edge of cloud-mobile convergence, in: Proceedings of te 9t International ACM Sigsoft Conference on Quality of Software Arcitectures, Ser. QoSA 13, ACM, New York, NY, USA, 2013, pp [2] F. Bonomi, R. Milito, J. Zu, S. Addepalli, Fog computing and its role in te Internet of tings, in: Proceedings of te First Edition of te MCC Worksop on Mobile Cloud Computing, ACM, 2012, pp [3] S. Davy, J. Famaey, J. Serrat-Fernandez, J. Gorrico, A. Miron, M. Dramitinos, P. Neves, S. Latre, E. Gosen, Callenges to support edge-as-a-service, IEEE Commun. Mag. 52 (1) (2014) [4] T. Taleb, A. Ksentini, Follow me cloud: interworking federated clouds and distributed mobile networks, IEEE Netw. 27 (5) (2013) [5] D.P. Bertsekas, Dynamic Programming and Optimal Control, second ed., Atena Scientific, [6] M.L. Puterman, Markov Decision Processes: Discrete Stocastic Dynamic Programming, first ed., Jon Wiley & Sons, Inc., [7] L. Georgiadis, M.J. Neely, L. Tassius, Resource allocation and cross-yer control in wireless networks, Found. Trends Netw. 1 (1) (2006) [8] M.J. Neely, Stocastic Network Optimization wit Application to Communication and Queueing Systems, Morgan and Cypool Publisers, 2010.

23 R. Urgaonkar et al. / Performance Evaluation ( ) 23 [9] S. Maguluri, R. Srikant, L. Ying, Stocastic models of load bancing and sceduling in cloud computing clusters, in: INFOCOM, 2012 Proceedings IEEE, Marc 2012, pp [10] Y. Guo, A. Stolyar, A. Walid, Sadow-routing based dynamic algoritms for virtual macine pcement in a network cloud, in: INFOCOM, 2013 Proceedings IEEE, April 2013, pp [11] Y. Guo, A.L. Stolyar, A. Walid, Online algoritms for joint application-vm-pysical-macine auto-scaling in a cloud, in: Te 2014 ACM International Conference on Measurement and Modeling of Computer Systems, Ser. SIGMETRICS 14, ACM, 2014, pp [12] J.W. Jiang, T. Lan, S. Ha, M. Cen, M. Ciang, Joint VM pcement and routing for data center traffic engineering, in: INFOCOM, IEEE, 2012, pp [13] M. Lin, A. Wierman, L.L.H. Andrew, E. Tereska, Dynamic rigt-sizing for power-proportional data centers, IEEE/ACM Trans. Netw. 21 (5) (2013) [14] M. Lin, Z. Liu, A. Wierman, L.L.H. Andrew, Online algoritms for geograpical load bancing, in: Proceedings of te 2012 International Green Computing Conference, IGCC, Ser. IGCC 12, Wasington, DC, USA, 2012, pp [15] G. Celik, E. Modiano, Sceduling in networks wit time-varying cannels and reconfiguration dey, IEEE/ACM Trans. Netw. 23 (1) (2015) [16] A. Gandi, S. Doroudi, M. Harcol-Balter, A. Sceller-Wolf, Exact analysis of te M/M/K/setup css of Markov cains via recursive renewal reward, SIGMETRICS Perform. Eval. Rev. 41 (1) (2013) [17] M. Neely, Dynamic optimization and learning for renewal systems, IEEE Trans. Automat. Control 58 (1) (2013) [18] S.M. Ross, Introduction to Probability Models, nint ed., Academic Press, Inc., Orndo, FL, USA, [19] T.H. Cormen, C. Stein, R.L. Rivest, C.E. Leiserson, Introduction to Algoritms, second ed., McGraw-Hill Higer Education, [20] D.B. Smoys, E. Tardos, An approximation algoritm for te generalized assignment problem, Mat. Program. 62 (3) (1993) [21] M. Piorkowski, N. Sarafijanovic-Djukic, M. Grossguser, A parsimonious model of mobile partitioned networks wit clustering, in: Communication Systems and Networks and Worksops, COMSNETS First International, Jan 2009, pp [22] R. Urgaonkar, M.J. Neely, Opportunistic sceduling wit reliability guarantees in cognitive radio networks, IEEE Trans. Mob. Comput. 8 (6) (2009) Raul Urgaonkar is a Researc Staff Member wit te Cloud-Based Networks group at te IBM T.J. Watson Researc Center. He is currently a task lead on te US Army Researc Laboratory (ARL) sponsored Network Science Colborative Tecnology Alliance (NS CTA) program. He is also a Primary Researcer in te US/UK International Tecnology Alliance (ITA) researc programs. His researc is in te areas of stocastic optimization, algoritm design, and optimal control wit applications to communication networks and cloud-computing systems. Before joining IBM researc, Dr. Urgaonkar was a Scientist wit te Network Researc group at Rayteon BBN Tecnologies were e worked on several government funded projects, including te NS CTA and ITA programs. He obtained is Masters and P.D. degrees from te University of Soutern California in 2005 and 2011 respectively and is Bacelor s degree (all in Electrical Engineering) from te Indian Institute of Tecnology Bombay in Siqiang Wang received te B.Eng. and M.Eng. degrees from Norteastern University, Cina, respectively in 2009 and He is currently working toward te P.D. degree at te Department of Electrical and Electronic Engineering, Imperial College London, United Kingdom. In te summers of 2014 and 2013, e was at IBM T.J. Watson Researc Center, Yorktown Heigts, NY, United States. In te autumn of 2012, e was at NEC Laboratories Europe, Heidelberg, Germany. His researc interests include dynamic control mecanisms, optimization algoritms, protocol design and prototyping, wit applications to mobile cloud computing, ybrid and eterogeneous networks, ad-oc networks, and cooperative communications. He as over 20 scorly publications, and as served as a tecnical program committee (TPC) member or reviewer for a number of international journals and conferences. Ting He received te B.S. degree in computer science from Peking University, Cina, in 2003 and te P.D. degree in electrical and computer engineering from Cornell University, Itaca, NY, in In 2007, Ting joined te IBM T.J. Watson Researc Center, were se is currently a Researc Staff Member in te Network Analytics Researc Group. At IBM, se as worked as a primary researcer and task lead in several researc programs including te International Tecnology Alliance (ITA) program funded by US ARL and UK MoD, te ARRA program funded by NIST, and te Social Media in Strategic Communication (SMISC) program funded by DARPA. Her work is in te broad areas of network modeling, statistical inference, and information teory. He is a senior member of IEEE. Se as served as te Membersip co-cair of ACM N2Women and te TPC of a range of communications and networking conferences, including IEEE INFOCOM, IEEE SECON, IEEE/ACM IWQoS, IEEE MILCOM, IEEE ICNC, IFIP Networking, etc. Se received te Outstanding Contributor Award from IBM Researc in 2009 and Se received te Best Paper Award at te 2013 International Conference on Distributed Computing Systems (ICDCS), a Best Paper Nomination at te 2013 Internet Measurement Conference (IMC), and te Best Student Paper Award at te 2005 International Conference on Acoustic, Speec and Signal Processing (ICASSP). In scool, se was an Outstanding College Graduate of Beijing Area and an Outstanding Graduate of Peking University in 2003, and a winner of te Excellent Student Award of Peking University from 1999 to Murtaza Zafer received te B.Tec. degree in Electrical Engineering from te Indian Institute of Tecnology, Madras, in 2001, and te P.D. and S.M. degrees in Electrical Engineering and Computer Science from te Massacusetts Institute of Tecnology in 2003 and 2007 respectively. He currently works at Nyansa Inc., were e leads te analytics portfolio of te company. Prior to tis, e was a Senior Researc Engineer at Samsung Researc America, were is researc focused on macine learning, deep-neural networks, big data and cloud computing. From e was a Researc Scientist at te IBM T.J. Watson Researc Center, New York, were is researc focused on computer and communication networks, data-analytics and cloud computing. He was a tecnical lead on several researc projects in te US UK funded multi-institutional International Tecnology Alliance program wit empasis on fundamental researc in mobile wireless networks. He as previously worked at te Corporate R&D center of Qualcomm Inc. and at Bell Laboratories, Alcatel-Lucent Inc., during te summers of 2003 and 2004 respectively. Zafer serves as an Associate Editor for te IEEE Network magazine. He is a co-recipient of te Best Student Paper award at te International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt) in 2005, a recipient of te Siemens and Pilips Award in 2001 and a recipient of several invention acievement awards at IBM Researc.

24 24 R. Urgaonkar et al. / Performance Evaluation ( ) Kevin Can is researc scientist wit te Computational and Information Sciences Directorate at te US Army Researc Laboratory (Adelpi, MD). Previously, e was an ORAU postdoctoral researc fellow at ARL ( ). His researc interests are in network science, wit past work in dynamic networks, trust and distributed decision making and quality of information troug ARL s Network Science Colborative Tecnology Alliance and Network and Information Sciences International Tecnology Alliance. Prior to ARL, e received a P.D. in Electrical and Computer Engineering (ECE) and MSECE from Georgia Institute of Tecnology (Atnta, GA) in 2003 and 2008, respectively. He also received a BS in ECE/EPP from Carnegie Mellon University (Pittsburg, PA) in Kin K. Leung received is B.S. degree from te Cinese University of Hong Kong in 1980, and is M.S. and P.D. degrees from University of California, Los Angeles, in 1982 and 1985, respectively. He joined AT&T Bell Labs in New Jersey in 1986 and worked at its successors, AT&T Labs and Lucent Tecnologies Bell Labs, until Since ten, e as been te Tanaka Cair Professor in te Electrical and Electronic Engineering (EEE), and Computing Departments at Imperial College in London. He is te Head of Communications and Signal Processing Group in te EEE Department. His current researc focuses on protocols, optimization and modeling of various wireless networks. He also works on multi-antenna and cross-yer designs for tese networks. He received te Distinguised Member of Tecnical Staff Award from AT&T Bell Labs (1994), and was a co-recipient of te Lancester Prize Honorable Mention Award (1997). He was elected an IEEE Fellow (2001), received te Royal Society Wolfson Researc Merits Award ( ) and became a member of Academia Europaea (2012). He also received several best paper awards, including te IEEE PIMRC 2012 and ICDCS He as actively served on conference committees. He serves as a member ( ) and te cairman ( ) of te IEEE Fellow Evaluation Committee for Communications Society. He was a guest editor for te IEEE JSAC, IEEE Wireless Communications and te MONET journal, and as an editor for te JSAC: Wireless Series, IEEE Transactions on Wireless Communications and IEEE Transactions on Communications. Currently, e is an editor for te ACM Computing Survey and International Journal on Sensor Networks.