IN RECENT years, global mobile data traffic has experienced

Transcription

1 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL FEMOS: Fog-Enabled Mltitier Operations Schedling in Dynamic Wireless Networks Shang Zhao, Yang Yang, Fellow, IEEE, Ziy Shao, Member, IEEE, XimeiYang, Ha Qian, Senior Member, IEEE, and Cheng-Xiang Wang, Fellow, IEEE Abstract Fog compting has recently emerged as a promising techniqe in content delivery wireless networks to alleviate the heavy brsty traffic brdens on backhal connections. In order to improve the overall system performance, in terms of network throghpt, service delay and fairness, it is very crcial and challenging to jointly optimize node assignments at control tier and resorce allocation at access tier nder dynamic ser reqirements and wireless network conditions. To solve this problem, in this paper, a fog-enabled mltitier network architectre is proposed to model a typical content delivery wireless network with heterogeneos node capabilities in compting, commnication, and storage. Frther, based on Lyapnov optimization techniqes, a new online low-complexity algorithm, namely fogenabled mltitier operations schedling (FEMOS), is developed to decompose the original complicated problem into two operations across different tiers. Rigoros performance analysis derives the tradeoff relationship between average network throghpt and service delay, i.e., [O(1/V), O(V)] with a control parameter V, nder FEMOS algorithm in dynamic wireless networks. For different network sizes and traffic loads, extensive simlation reslts show that FEMOS is a fair and efficient algorithm for all ser terminals and, more importantly, it can offer mch better performance, in terms of network throghpt, service delay, and qee backlog, than traditional node assignment and resorce allocation algorithms. Index Terms 5G, delay, fog compting, Internet-of-Things (IoT), qality of experience (QoE), throghpt. Manscript received November 3, 2017; revised Janary 30, 2018; accepted Febrary 13, Date of pblication Febrary 21, 2018; date of crrent version April 10, This work was spported in part by the National Natral Science Fondation of China nder Grant and Grant , in part by the Science and Technology Commission of Shanghai Mnicipality nder Grant and Grant 17DZ , and in part by the EPSRC nder Grant EP/L020009/1. Part of this work was presented at the IEEE Global Commnications Conference (Globecom), Singapore, Dec [1]. (Corresponding athor: Yang Yang.) S. Zhao, Y. Yang, and X. Yang are with the Shanghai Institte of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai , China, with the University of Chinese Academy of Sciences, Beijing , China, and also with the Shanghai Institte of Fog Compting Technology, Shanghai , China ( yang.yang@wico.sh). Z. Shao is with the School of Information Science and Technology, ShanghaiTech University, Shanghai , China, and also with the Shanghai Institte of Fog Compting Technology, Shanghai , China. H. Qian is with the Shanghai Advanced Research Institte, Chinese Academy of Sciences, Shanghai , China, and also with the Shanghai Institte of Fog Compting Technology, Shanghai , China. C.-X. Wang is with the Institte of Sensors, Signals and Systems, School of Engineering and Physical Sciences, Heriot-Watt University, Edinbrgh EH14 4AS, U.K. Digital Object Identifier /JIOT I. INTRODUCTION IN RECENT years, global mobile data traffic has experienced explosive growth. It is expected to grow to 49 exabytes per month by 2021, a sevenfold increase over 2016 [2]. Crrent wireless technologies, sch as 4G and Wi-Fi, do not have localized data analysis and processing capabilities so that they cannot handle sch a brsty traffic increase. As machine-type commnications have been adopted in ftre 5G networks [3] [5], new flexible network architectres and service strategies are desperately needed to spport more and more data-centric and delay-sensitive Internet-of-Things (IoT) applications [6], sch as smart city, environment srveillance, intelligent manfactring, and atonomos driving. For ftre 5G and IoT applications in complex environments, raw measrement data of mltiple radio channels at different commnication scenarios can be fond at an open-sorce website: [7]. If only centralized clod compting architectre is applied to those varios IoT applications, it is envisaged that the nderlayer commnication networks, especially backhal connections, will face heavy brsty traffic brdens and experience dramatic performance degradation. On the other hand, the Moore s Law has significantly driven down the prices of compting and storage devices, more and more smart network nodes and ser terminals (UTs) are deployed and connected into modern commnication networks. They provide a rich collection of biqitos local compting, commnication, and storage resorces. In view of this technological trend, the concept of fog compting is proposed to enable compting anywhere along the clod-to-thing continm [8], [9]. In other words, fog-enabled network architectre and services can effectively leverage those local resorces to spport fastgrowing data-centric and delay-sensitive IoT-applications in regional environments, ths redcing backhal traffic transmissions and centralized compting needs, and at the same time, improving the overall network throghpt performance and sers qality of experience (QoE) [10] [12]. Withot loss of generality, let s consider a content delivery wireless network consisting of heterogeneos smart nodes with different compting, commnication, and storage capabilities. As UTs are moving arond and can make reqests of any contents at anytime anywhere, it is obvios that poplar contents shold be placed in mltiple neighboring nodes of a UT according to their resorces and capabilities. In doing so, most content delivery reqests are handled in local network segments, service delay, and backhal traffic transmission can be greatly redced, ths minimizing the needs for expensive c 2018 IEEE. Personal se is permitted, bt repblication/redistribtion reqires IEEE permission. See for more information.

2 1170 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018 centralized compting resorces. Besides traffic localization and service delay, the overall network throghpt and fairness among different UTs are crcial performance metrics for network operators and service providers, and therefore need to be analyzed and improved simltaneosly. As an interesting and promising way to relieve the backhal pressres and redce the service latency, the cacheenabled content delivery networks have been widely stdied. Baştǧ et al. [13] explored the significant gains in terms of the otage probability and the average delivery rate by having cache-enabled small base station. Li et al. [14] designed distribted caching optimization algorithms via belief propagation to minimize the downloading latency. Li et al. [15] stdied the cache placement problem in fog-radio access network (F-RAN) and developed transmission aware cache placement strategies. Most of those works mainly focs on file placement in the design of content delivery networks, while dynamic association between the UTs and access nodes were rarely investigated, which also directly affects UTs QoE, and therefore is the focs of this paper. On the other hand, the topic abot taking the advantages of available compting and storage resorces at network edges has attracted significant attentions recently. Mao et al. [16] provided a comprehensive srvey of the state-of-the-art mobile edge compting (MEC) research with a focs on joint radioand-comptational resorce management. In [17], a joint radio and comptational resorce management algorithm for mltiser MEC system is proposed, which efficiently tilized the powerfl comptation resorce at the MEC server. Shanmgam et al. [18] focsed on minimizing the transmission delay throgh taking advantages of the distribted storage capacities at the network edges. Shih et al. [19] introdced the F-RAN architectre which can provide ltra low-latency service. Pang et al. [20] stdied the latency-driven cooperative task compting in F-RAN networks, where the F-RAN node can offload its comptation tasks to the nearby loadfree F-RAN nodes. Different from previos research works, which mainly focsed on improving the tilization of compting or storage resorces at network edges, this paper applies the concept of fog compting to deal with a more complex mltitier network architectre with heterogeneos node capabilities and dynamic network resorces, in terms of compting power, storage capacity, transmission power, and commnication bandwidth. Specifically, in this paper, the content delivery wireless network is modeled with access tier and control tier. 1) A node in access tier is typically located close to the UTs and is called fog access node (FAN). An FAN caches a sbset of poplar contents depending on its limited compting and storage capabilities. De to restricted transmission power and dynamic wireless environment, the commnication channels between an FAN and its neighboring UTs are nreliable and time-varying. 2) A node in control tier manages a grop of FANs throgh reliable bt expensive backhal connections, and is called fog control node (FCN). An FCN is mch more powerfl in terms of compting power and storage capacity than FAN. However, the FCN is physically not possible or economically not feasible to commnicate directly with any UTs. It is more efficient for an FCN to execte control operations for achieving regional performance objectives with a centralized approach. Under this realistic mltitier system model, it is very challenging to simltaneosly address the following problems in real-time. 1) When poplar contents are randomly cached at different FANs, how to identify the most feasible FAN for every UT s reqest in order to maximize network throghpt and global fairness. 2) Under dynamic wireless network conditions and fading channel characteristics, how to effectively allocate commnication bandwidth for associated FAN-UT pairs in order to minimize service delay. In particlar, or main contribtions are smmarized as follows. 1) Based on the concept of fog compting, a fog-enabled mltitier network architectre is proposed to model a content delivery wireless network, which takes fll advantages of heterogeneos node capabilities in compting, commnication, and storage. 2) Based on this network model and Lyapnov optimization techniqes, an online (real-time) low-complexity fogenabled mltitier operations schedling (FEMOS) algorithm is developed to jointly optimize node assignments at control tier and resorce allocation at access tier. Specifically, the inherent mixed nonlinear integer programming problem in this mltitier network architectre is flly investigated and its strctral information is exploited to decompose the original complicated problem into two operations, i.e., centralized assignment of access node (CAAN) scheme at the FCN and dynamic online bandwidth allocation (DOBA) scheme at FANs. 3) A general analytical framework is then developed to evalate the performance of FEMOS algorithm and, more importantly, to characterize the tradeoff relationship between average network throghpt and service delay. Compared with traditional node assignment and resorce allocation algorithms, simlation reslts show that FEMOS can offer mch better performance, in terms of network throghpt and service delay. The rest of this paper is organized as follows. The system model is presented in Section II. Under the fog-enabled network architectre, the problem is formlated and the responding analytical framework is developed in Section III. The online FEMOS algorithm is proposed in Sections IV. The performance analysis for the proposed algorithm is condcted in Section V and the simlation reslts are shown in Section VI. Section VII concldes this paper. II. SYSTEM MODEL We consider a fog-enabled mltitier network with heterogeneos nodes as shown in Fig. 1, which involves an FCN tier, an FAN tier, and a set of mltiple stationary or low-mobility UTs in the region nder consideration. Each UT possesses small storage capacity, minor commnication ability, and little or no comptation ability. UTs reqest files to be downloaded

3 ZHAO et al.: FEMOS IN DYNAMIC WIRELESS NETWORKS 1171 TABLE I SUMMARY OF KEY NOTATIONS Fig. 1. Sample fog-enabled mltitier network with three FANs and five UTs. from FAN tier throgh wireless links. De to the restricted transmission power and dynamic wireless environment, the commnication channels between an FAN and its neighboring UTs are nreliable and time-varying. Each FAN in FAN tier is eqipped with limited storage capacity, medim comptation ability bt strong commnication ability. All of them cached a sbset of poplar files. Throgh reliable backhal links, FANs are connected with an FCN, which is next to the clod and core network, has the global information abot the network and is the server of the file library. The FCN has both sfficient storage capacity and powerfl comptation ability. We emphasize here that in or model, the backhal links transmit control information from the FCN to FANs and files cached at each FANs are only refreshed at off-peak times. To be clear, if the UT reqested file is not cached on the FANs, the FAN will not fetch the file from the operation center throgh the backhal link. Under these conditions, we can see more clearly the benefit of dynamic assignment of FAN and resorce schedling. Denote the set of FANs as H, the set of UTs as U, and the file library as F. The large scale fading and small scale fading coefficients seen by each FAN are assmed to be mtally independent. We assme that the network operates in a slotted system, indexed by t {0, 1, 2,...} and the time slot length is T.TakeFig.1 along with Fig. 2 as an example, where H = {I, II, III}, U = {1, 2, 3, 4, 5}. Dring each slot, UT broadcasts its content reqest for file with only one type f F, to the FAN tier. The arrived bt not yet served reqests will be qeed in the reqest bffers at the FCN, as shown in Fig. 2. The FCN determines the dynamic FAN assignment for UTs at beginning of each slot, to optimize the network throghpt in a memoryless pattern, based on the network states, reqest qee length, and disregarding all sch previos decisions. Each UT reqested files will be then transmitted by its associated FAN throgh the wireless link. The qee length of crrent nserved reqest bffers will in trn inflence the FCN s decision abot FAN assignment in the next slot. Each FAN then independently implements its per-slot schedling policy inclding the bandwidth and service rate allocation over the UTs associated with it. Consistent with the above setting, the reqested file f can only be downloaded from the FAN that has cached it. The service rate wold be zero if the reqested file is not cached on the FAN that the UT is associated with. For ease of reference, we list the key notation of or system model in Table I. A. FAN Assignment and File Placement Define the FAN assignment (UT-FAN association) as a bipartite graph G = (U, H, E), where E contains edges for the pairs (, h) sch that there exists a potential transmission link between FAN h H and UT U. We assme G varies in different time slots. Let X(t) denote a U H association matrix of G between UTs and FANs in time slot t, where U ( H ) denotes the cardinality of the set U(H), X(t) [x h (t)],h. Here x h (t) = 1 if (, h) E, and 0 otherwise. Define the file placement (FAN-file association) as a bipartite graph G = (H, F, Ẽ), where edges (h, f ) Ẽ indicates that files with type f are cached in FAN h. The file set cached at each FAN is N, N F, with N different file types. In this paper, we focs on dynamic FAN assignment and the resorce schedling problem. As said before, the backhal pdates the storages at a time scale mch larger than the time scale of UTs placing file reqests. Therefore, assming fixed file placement (FAN-file association) is jstified. Let Y denote a H F file placement matrix of G and Y [y hf ] h,f. Here, y hf = 1if(h, f ) Ẽ, and 0 otherwise. We assme that each UT can be associated with at most one FAN and each FAN can associate with at most M UTs in one time slot. Ths X(t) shold be chosen from the feasible set A A = {X(t) {0, 1} U H x } h(t) M h H h H x. h(t) 1 U (1) B. UT Traffic Model All UTs are assmed to generate file reqest traffic randomly in each time slot, and this traffic generation is independent of the FCN s operation. Let A(t) denote the reqest arrival vector in time slot t and A T (t) [A 1 (t),...,a U (t)], where random variable A (t) (with the nit kbits) denotes the reqested amont in time slot t and the operation ( ) T denotes vector transposition. Here we

4 1172 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018 (a) (b) (c) Fig. 2. Illstration of dynamic FAN assignment. (a) t = 1. (b) t = 2. (c) t = 3. assme that A (t) is independent identically distribted (i.i.d.) with E{A (t)} =λ, and there exists a positive constant A max sch that 0 A (t) A max. Let I(t) denote the U F reqested file type matrix in time slot t and I(t) [I f (t)],f.herei f (t) = 1ifthe reqested file type by UT is f in time slot t, and 0 otherwise. We assme that each UT can reqest at most only one type of file in one time slot, which means that the row weight of I(t) is at most 1. The reqested probability of each file f F is sbject to Zipf distribtion [21]. C. Transmission Model The wireless channels between UTs and FANs are assmed to be flat fading channels [22], and all FANs transmit at constant power. We assme that the additive white Gassian noise at the UTs follows Gassian distribtion with N (0,σ 2 ).Note that the maximm service rate of UT can be obtained if it has been allocated the total bandwidth by its associated FAN. Then the maximm backlog that can be served in time slot t over link (, h) E is given by C h (t) ( )] P h g h (t) s h = T B h (t) E [log σ 2 + h H\h P h g h (t) s h 2 where B h (t) is the total bandwidth of FAN h in time slot t, P h is the transmit power of FAN h, g h (t) is the large scale fading from FAN h to UT which contains pathloss and shadow, and s h is the small scale fading which follows the Rayleigh distribtion. For simplicity, crrently implemented rate adaption schemes [23], [24] are consistent in assming, slowly varying pathloss coefficients g h (t) change across slots in an i.i.d. manner, and each FAN h being aware of g h (t) for all U at the beginning of each time slot t. We also assme that each FAN h serves its associated UTs by sing orthogonal FDMA or TDMA, which is consistent with most crrent wireless standards. Let ν h (t) be the proportion of bandwidth allocated to UT by FAN h. Then ν h (t) satisfies 0 <ν h (t) 1 when x h (t) = 1, otherwise ν h (t) = 0. Denote ν(t) [ν h (t)],h as the bandwidth allocation matrix, which is chosen from the feasible set B { B = ν(t) R U H + ν } h(t)x h (t) 1. (3) ν h = 0ifx h = 0 h H (2) Let μ (t) denote the amont of backlog that can be served for UT in time slot t with maximm vale μ max, which is called service rate hereafter. Define μ T (t) [μ 1 (t),...,μ U (t)]. Note that each UT can associate with at most one FAN in a time slot, ths μ (t) can be expressed as μ (t) = h H C h (t)ν h (t)x h (t) U. (4) D. Qeeing In each time slot, the arrived reqests of all UTs will be qeed in the reqest bffers at the FCN. We assme that FCN has F reqest bffers for each UT U. Denote the qee length of the amont of reqest for file with type f at the beginning of the tth time slot as Q f (t). Define (t) f F Q f (t) and denote Q T (t) = [ (t),..., U (t)] as the qee length vector. We assme that all qees are initially empty, i.e., Q f (0) = 0, U, f F. Let μ f (t) denote the service rate for the reqested file f schedled by the FCN according to a certain qeeing discipline [25], sch as first-inpt, first-opt (FIFO), LIFO, or random discipline. We adopt flly efficient schedling policy given in [25] for qees, which means μ f (t) = μ (t) (5) f F where μ (t) is defined in (4) and μ f (t) = 0ify hf = 0. The qee length Q f (t) is pdated in every time slot t according to the following rles: Q f (t + 1) = [ Q f (t) μ f (t) ] + + A (t) I f (t) (6) where [x] + = max{x, 0}. The qeeing process Q f (t) is stable if the following condition holds [26]: 1 t 1 = lim t t E [ (τ) ] <. (7) III. PROBLEM FORMULATION AND ANALYTICAL FRAMEWORK In this section, we will first introdce the performance metrics in Section III-A, namely, the time-averaged throghpt of the network and the average delay per UT experiences. An average throghpt maximization problem with file

5 ZHAO et al.: FEMOS IN DYNAMIC WIRELESS NETWORKS 1173 reqest qees stability constraints will then be formlated in Section III-B. In Section III-C, we will give the analytical framework based on Lyapnov optimization techniqes. A. Performance Metrics We focs on the total throghpt of the network. Therefore, we adopt the time-averaged sm service rate of different UTs in the network as the performance metric, which is defined as follows: φ av φ(μ) = μ (8) where μ (t) is the averaged expected service rate of UT, i.e., μ = lim t (1/t) t 1 E[μ (τ)]. To spport for transmission latency sensitive applications, i.e., online video, service delay is a key metric needs to be considered [27]. According to Little s Law [28], the average service delay experienced by each UT is proportional to the averaged amont of its nserved reqests waiting at the FCN, which is sm of qee length for different files. Ths, the average delay per UT can be compted as the ratio between the average qee length and the mean traffic arrival rate, which is shown as follows: av Qsm λ. (9) B. Average Network Throghpt Maximization Problem The system objective is to find a feasible FAN assignment X(t) and bandwidth allocation ν(t) to maximize the average network throghpt while maintaining the stability of all the qees in the network. The average network throghpt maximization problem can be formlated in P 1 P1: max X(t),ν(t) s.t. φ(μ) < U X(t) A, ν(t) B t (10) where the reqirement of finite corresponds to the strong stability condition for all the qees [26]. Qeeing stability implies that the bffered file reqests are processed with finite delay. We will show that or proposed algorithms garantee pper bonds for Q f and ths achieve the bonded service delay. Remark 1: It is not difficlt to identify that P1 is a highly challenging stochastic optimization problem with a large amont of stochastic information to be handled (inclding channel conditions and reqest bffer state information) and two optimization variables to be determined, which reqires to design an online operation and schedling scheme for sch a network. Besides, to maximize the network throghpt, it is essential to jointly optimize the FAN-UT association and the resorce allocation, which is always a complicated mixed integer programming problem. Frther, the optimal decisions are temporally correlated de to the random arrival traffic demands. Frthermore, the FCN needs to redce the delay per UT while maintaining the average network throghpt, which reqires the FCN to maintain a good balance between network throghpt and average delay. C. Lyapnov Optimization-Based Analytical Framework In the following, we focs on solving this challenging problem P 1 by sing the Lyapnov optimization techniqe [26], with which we can transfer the challenging stochastic optimization problem P 1 to be a deterministic per-slot problem in each time slot. We first define a qadratic Lyapnov fnction as follows: L(Q(t)) 1 2 QT (t)q(t) = 1 ( Q sm (t) ) 2. (11) 2 We then define a one-slot conditional Lyapnov drift (Q(t)) as follows: (Q(t)) E{L(Q(t + 1)) L(Q(t)) Q(t)}. (12) Accordingly, the one-slot conditional Lyapnov drift-plspenalty fnction is shown as follows: V (Q(t)) = (Q(t)) VE{φ(μ(t)) Q(t)} (13) where V > 0 is the policy control parameter. We first establish the pper bond of V (Q(t)) nder any feasible policy X(t) and ν(t), as specified in Lemma 1. Lemma 1: For any feasible control decision X(t) and ν(t) for P1 sch that X(t) A, ν(t) B, V (Q(t)) is pper bond by { [ V (Q(t)) K E V + Q sm (t) ] μ (t) } Q(t) + E { (t)a (t) Q(t) } (14) where K is a constant. Proof: Please refer to Appendix A. Lemma 1 provides the pper bond for the conditional Lyapnov drift-pls-penalty fnction V (Q(t)), which plays a significant role in the FEMOS. IV. FOG-ENABLED MULTITIER OPERATIONS SCHEDULING ALGORITHM In this section, we introdce the FEMOS algorithm, which mainly consists of two stage operations in per time slot: CAAN and DOBA. We present the DOBA in Section IV-A and describe the CAAN in Section IV-B. In Section IV-C, we propose the FEMOS algorithm and provide the corresponding comptation complexity analysis. A. Dynamic Online Bandwidth Allocation To solve Problem P 1 based on the Lyapnov optimization method [26], it needs to design an algorithm to minimize the pper bond of the Lyapnov drift-pls-penalty term in each time slot. Ignoring the constant components in the pper bond of V (Q(t)) and rearranging them, the pper bond minimization problem is converted to min [ V + Q sm (t) ] μ (t). (15) X(t) A,ν(t) B

6 1174 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018 Note that (t) is observed at the beginning of each time slot, which can be viewed as constant per time-slot. Therefore the pper bond minimization only depends on μ (t), which involves the FAN assignment and bandwidth allocation. For convenience, we define W h (t) [V + (t)]c h (t), which is constant per time-slot. From the definition of μ (t) in (4), the pper bond minimization of V (Q(t)) is transferred to the following eqivalent problem: P AR : max W h (t)ν h (t)x h (t). (16) X(t) A,ν(t) B h H P AR is a joint optimization problem of access node assignment and bandwidth allocation. The main idea of the proposed FEMOS algorithm is to solve the deterministic optimization problem P AR in each time slot. By doing so, the amont of reqest waiting in the qees can be maintained at a small level and the network throghpt can be maximized at the same time. Note that P AR is a nonlinear integer programming problem, for which the comptational complexity of the brte-force search is prohibitive. By exploiting the strctre information of P AR, we transfer problem P AR to the following problem P AR, which is proven eqivalent with P AR: P AR :max X(t) s.t. W h (t)x h (t) h H x h (t) 1 h H x h (t) 1 H, x h (t) {0, 1} (17) h H and the corresponding bandwidth allocation for each UT U can be expressed as follows: { 1 xh = 1 h H ν h (t) = (18) 0 otherwise. Lemma 2: The FAN assignment soltion obtained by the transformed problem P AR along with the bandwidth allocation given by (18) coincides with the soltion for P AR. Proof: See Appendix B. Throgh the eqivalent problem transformation described above, the soltion to joint optimization of FAN assignment and bandwidth allocation are decopled in two stage operations: 1) CAAN and 2) DOBA. Specifically, the CAAN will be exected at FCN with powerfl comptation ability by solving P AR. Once the CAAN is established, the FANs will exected DOBA as (18) independently. B. Centralized Assignment of Access Node The dynamic assignment of FANs in each time slot can be obtained by solving the problem P AR. Note that the constraint of maximm connectable UTs for each FAN in P AR is M, while it redces to 1 in P AR. To frther nderstand the dynamic FAN assignment scheme nder the constraints of P AR,wetakeFig.3 as an example, where the colored dash lines represent the UT-FAN association in time slot t = 1, 2, 3 and the shadowed qees represent the responding UTs did not access the network in those time slots. In time slot t = 1, Fig. 3. Illstration of dynamic FAN assignment scheme nder the constraints in P AR. UT 2, UT 3, and UT 5 are associated with FAN I, III, and II, respectively, and all of them obtained the total bandwidth. The arrival reqests of not associated UTs (UT 1 and UT 3) will be qeed in reqest bffers at the FCN, waiting for being served in the next time slot. To solve P AR efficiently, we demonstrate that the problem P AR can be formlated as the maximization of a normalized modlar fnction sbject to intersection of two partition matroid constraints in the following. This strctre can be exploited to design comptationally efficient algorithms for problem P AR with provable approximation gaps. The definitions of matroid and sbmodlar fnction are given in Appendix G. First, we define E as grond set E ={(, h) U, h H}, that consists of all possible tples and where each tple (, h) denotes an association of UT to FAN h. Frther, we define set E ={(, h) h H}, which consists of all tples for each UT U, along with set E h ={(, h) U}, which consists of all tples for each FAN h H. Next, we proceed to show that P AR can be solved by a simple greedy algorithm. Toward this end, we define W E (,h) E W h(t), where E E. We also define a family of sets I, as the one that incldes each sbset E of E sch that the sbset meets the UT-FAN limits in P AR. Specially { I = E E E E E 1 ; E h 1 h }. (19) Then, we offer the following reslt. Lemma 3: The problem P AR can be formlated as the maximization of a normalized modlar fnction sbject to two partition matriod constraints on grond set E. Proof: See Appendix C. As a conseqence of Lemma 3, we can leverage the famos reslt in [29], which shows that a simple effective greedy algorithm yields a constant factor approximation for the problem of maximizing a normalized non-negative modlar set fnction sbject to matroid constraints. Specializing the approximation garantee of [29] to or case with two matroid constraints, the problem P AR can be approximately solved sing Greedy FAN assignment algorithm with a constant-factor (1/2) approximation garantee.

7 ZHAO et al.: FEMOS IN DYNAMIC WIRELESS NETWORKS 1175 Algorithm 1 Greedy FAN Assignment Algorithm 1: Initialize Ê =, E = E 2: Repeat Determine (, h ) E \ Ê as the soltion to W h (t) max Ê (,h ) I 3: Update Ê = Ê (, h ) 4: Update E = E \(, h ) 5: Until E = 6: Otpt Ê Algorithm 2 FEMOS Algorithm 1: Set t =0,Q(0) = 0; 2: While t < t end, do 3: At beginning of the tth time slot, observe A (t), g h (t) and Q f (t); 4: CAAN: X (t) is obtained by rn Algorithm 1; 5: DOBA: { νh (t) = 1 xh (t) = 1 0 otherwise; 6: Schedle the service rates μ f (t) to the qees Q f (t) according to (5) with any pre-specified qeeing discipline; 7: Update {Q f (t)} according to (6) for each UT based on X (t), ν (t) and μ f (t); 8: t t : end While The Greedy FAN assignment algorithm is described in Algorithm 1. It starts with an empty set Ê = and E = E. It iteratively adds the element (, h ) with highest marginal vale while satisfying the matroid constraints in each step. The marginal vale of an element (, h ) is defined as the gain of adding (, h ) into the UT-FAN association set Ê, given by W h (t), when Ê (, h ) I. In or Greedy FAN assignment algorithm, once the element (, h ) is added to Ê, it shold be deleted from set E. The Greedy FAN assignment algorithm stops ntil E becomes dring the iteration. C. Fog-Enabled Mltitier Operations Schedling Algorithm The proposed FEMOS algorithm is smmarized in Algorithm 2, which will be implemented at both FCN tier and FAN tier in practice. Specifically, in each time slot, within the global network information and qee length of each reqest bffer, the FCN will execte CAAN by rn the Greedy FAN assignment algorithm. The assignment decision is then transmitted to the FAN tier. Next, each FAN performs the DOBA independently and schedle the service rate for the associated UT nder fll efficient schedling policy. Finally, the FCN pdates the reqest bffers for each UT, the length of which will inflence the operations schedling in next slot. Remark 2: 1) Interestingly, a closer inspection of P AR reveals that in each time slot each FAN can associate with at most only one UT and the nmber of UTs that can dynamically associate with FANs depends on the nmber of FANs, which is H. What makes or Greedy FAN assignment algorithm otstanding from the common assignment method which directly select FAN with the best channel condition for each UT or each FAN associates with the UT with the longest qee length is that, we provide a qantitative analysis and selection method for UT-FAN association. 2) In each time slot, the association between UTs and FANs depends on corresponding UT-FAN pair gain W h (t) = [V + (t)]c h (t). The Greedy FAN assignment algorithm greedily selects the UT-FAN pair with the largest W h (t) within the feasible set. For each FAN h, it will associate with one UT with either large qee backlog or good channel condition. If V is small, i.e., V (t), both the qee backlog and the channel condition will determine the decision on UT-FAN association. The FAN h will associate with UT with largest (t)c h (t). Conversely, if V is large enogh, i.e., V (t),thefanh will more effectively invoke the willingness to associate with a UT with a good channel condition C h (t). Under large V, the UT with weak channel conditions cannot access the network for a long time, leading to large accmlated qee length and inflence on the UT-FAN decision in trn. 3) Therefore, the parameter V actally controls the FANs willingness to serve UTs, i.e., performing UT-FAN association. In other words, it controls the tradeoff between network throghpt and transmission delay. In each time slot, the comptational complexity for FEMOS primarily remains in the greedy-ua algorithm. According to Algorithm 1, the comptational complexity of the Greedy FAN assignment algorithm is O( U H Ɣ ), where Ɣ denotes the comptational complexity of searching for the element (, h ). The greedy algorithm starts with an empty set and at each step, it adds one element with the highest marginal vale to the set while maintaining the feasibility of the soltion. Since the objective fnction is modlar, the marginal vale of the elements decreases as we add more elements to the set Ê. Ths, in one iteration, if the largest marginal vale is zero, then the algorithm shold stop. In or case, there wold be at most U H iterations. Each iteration wold involve evalating the marginal vale of at most U H elements. Then, the comptational complexity of searching for the element (, h ) is O(Ɣ ) = O( U H ). Ths the overall comptational complexity for the Greedy FAN assignment algorithm is O( U 2 H 2 ). V. PERFORMANCE ANALYSIS In this section, we provide the main theoretical reslts for FEMOS, which characterize the lower bonds for average network throghpt as well as the pper bonds for the average sm qee length of the reqests of all the UTs. Additionally, the tradeoff between the network average throghpt and average delay will also be revealed. Note that the Greedy FAN assignment algorithm is efficient and garantees a tight (1/2)-approximation for P AR, i.e., the

8 1176 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018 worst case is at least 50% of the optimal soltion. The FAN assignment X(t) and bandwidth allocation ν(t) is then a sboptimal soltion to the problem P AR, which we refer to as imperfect schedling [30], [31]. Therefore, in each time slot, the CAAN and DOBA policy for P AR yields a transmission rate μ(t) R that satisfies [ V + Q sm (t) ] μ (t) β max μ(t) R { [ V + Q sm (t) ] μ (t) } (20) where β is constant with β = (1/2). The parameter β in (20) can be viewed as a tning parameter indicating the degree of precision of imperfect schedling. Notice that when β = 1 it redces to the case with perfect schedling of P AR. Define φ opt as the optimal average network throghpt associated with the problem P 1, agmented with the rectangle constraint R, and μ R, where R is chosen large enogh to contain the optimal time average service rate vector μ. Letμ,0 (t) denote the transmission rate nder the optimal soltion to P 1. The following β-redced problem trns ot to be a good reference to stdy imperfect schedling. β-redced Problem: max X(t),ν(t) s.t. φ(μ) μ(t) βr < U X(t) A, ν(t) B t. (21) Let μ,β (t) denote the transmission rate nder the optimal soltion to the β-redced problem above and denote φ opt β as the corresponding optimal throghpt. In the following Lemma 4, we will demonstrate the relationship between P 1 and β-redced problem in terms of their optimal vales. Lemma 4: Let μ,0 (t) be the optimal transmission rate for the P1. Then the transmission rate to the β-redced problem is μ,β (t) = βμ,0 (t). Proof: See Appendix D. Lemma 4 also implies the relationship between the optimal average throghpt of P 1 and β-redced problem, which is φ opt β = βφ opt. Next, we characterize the performance of FEMOS nder i.i.d. system randomness and assme there exists constants ɛ and φ ɛ sch that the following slater-type conditions holds λ E{μ (t)} ɛ U (22) E{φ(μ(t)} =φ ɛ. (23) Note that the assmptions (22) and (23) ensre the strong stability of the qees in the system, which are generally assmed in the network stability problems [26] and garantee that there exists a stationary and randomized FAN assignment and bandwidth allocation policy. Lemma 5: For any alternative policy inclding FAN assignment X(t) A and bandwidth allocation ν(t) B, we have V (Q(t)) K Vφ ɛ βɛ (t). (24) Proof: See Appendix E. Based on Lemmas 4 and 5, the performance bonds of FEMOS are derived in the following theorem. The term is defined as the long term expected average network φav FEMOS throghpt of FEMOS and FEMOS av is defined as the long term expected average service delay per UT. Theorem 1: For the network defined in Section II, the CAAN and DOBA policy obtained throgh FEMOS algorithm achieves the following performance: φ FEMOS av lim inf t φ ( 1 t ) t 1 E{φ(μ(τ))} βφ opt K V FEMOS av 1 1 t 1 λ lim sp t t E { (t) } (25) K + V( φ opt ) φ ɛ βɛ λ. (26) Proof: See Appendix F. Remark 3: Theorem 1 shows that nder the proposed fogenabled mltitier operation schedling algorithm, the lower bond of average network throghpt increases inversely proportional to V, while the pper bond of average service delay per UT experienced increases linearly with V. If a larger V is sed to prse the better network throghpt performance, it will introdce severe service delay. Hence, there exists an [O(1/V), O(V)] tradeoff between these two objects. Throgh adjsting V, we can balance the network throghpt and service delay. VI. SIMULATION RESULTS We consider a network with H = 9 fixed FANs and U = U randomly deployed UTs. Each FAN can associated with at most M = 12 UTs. The size of the file types in the file library is set to F =1000 and the cached file types of each FAN is set to N =500. The system region hasasizeof20 20 m 2. Each UT reqests files with type f F according to the Zipf distribtion with parameter η r, i.e., p f = (f η r/ i F i η r) [32] and η r = 0.56 in or simlations. An FIFO qeeing discipline is applied in the simlations. The simlation reslts are averaged over 3000 constant time slots with T = 100 milliseconds intervals. We assme that each FAN operates on a B h = 18 MHz bandwidth (100 resorce blocks with 180 khz for each resorce block) and transmits at a fixed power level P = 20 W. Besides, the noise power σ 2 isassmedtobe W. Based on the WINNER II channel model in small-cell scenarios [33], the pathloss coefficients between the FAN h and UT is defined by g h (t) = 10 (PL(dh(t))/10), where d h (t) is the distance from FAN h to UT in time slot t, and PL(d) = Alog 10 (d) + B + Clog 10 (f 0 /5) + X db, where f 0 is the carrier freqency, X db is a shadowing log-normal variable with variance σdb 2 ; A = 18.7, B = 46.8, C = 20, and σ db 2 = 9 in line-of-sight (LOS) condition; A = 36.8, B = 43.8, C = 20, and σdb 2 = 16 in non-los (NLOS) condition. Each link is in

9 ZHAO et al.: FEMOS IN DYNAMIC WIRELESS NETWORKS 1177 (a) (b) Fig. 4. Comparison of FEMOS and other FAN assignment schemes, A max = 100 kbits, U = 100. (a) Network throghpt verss parameter V. (b) Service delay per UT verss parameter V. LOS or NLOS independently and randomly, with probabilities of p l (d) and 1 p l (d), respectively, where { 1, d 2.5 m p l (d) = ( 1 ( log(d)) 3) 1/3, otherwise. (27) A. Performance Comparison With Other Assignment Scheme In this section, we evalate the performance of the proposed FEMOS algorithm and compare its Greedy FAN assignment with the other two dynamic schemes: 1) select best channel (SBC) and 2) select longest qee (SLQ). In SBC scheme, each FAN associates with the UT with the best channel condition between them in each time slot. In SLQ scheme, in each time slot, the UTs with top- H qee length access the network and each of them associates with the FAN with the best transmission condition. Fig. 4 first validates the theoretical reslts for the proposed FEMOS algorithm derived in Theorem 1. The average network throghpt performance is shown in Fig. 4(a), while the average service delay per UT is shown in Fig. 4(b). It can be observed from Fig. 4(a) that the average network throghpt obtained by the Greedy FAN assignment in FEMOS increases as V increases and converges to the maximm vale when V is sfficiently large. Meanwhile, as shown in Fig. 4(b), the average service delay experienced by per UT of FEMOS increases linearly with the control parameter V. Thisisin accordance with the analysis in Remark 2 that with V increasing, the importance of reqest qee backlogs decreases, which makes the dynamic assignment bias toward good channel conditions. Those observations verify the [O(1/V), O(V)] tradeoff between average network throghpt and average qee backlog as demonstrated in Theorem 1 and Remark 3. Fig. 4 also compares the Greedy FAN assignment in FEMOS with SBC and SLQ assignment schemes. We observe that the average network throghpt performance of SBC stabilizes arond kbits/slot, which approaches to the maximm vale of FEMOS. However, the average Fig. 5. Network throghpt-delay tradeoff for FEMOS nder different caching strategies. network throghpt performance deteriorates severely nder SLQ assignment scheme for any V. The average service delay in both SBC and SLQ assignment schemes are mch larger than that of Greedy assignment in FEMOS when V < Specifically, when V = 1, the average delay in FEMOS approaches to zero, bt they are 5 s and 45 s for SLQ and SBC schemes, respectively. Those comparisons demonstrate the advantages of the Greedy in FEMOS. B. FEMOS s Performance Under Different Caching Strategies In Fig. 5, we investigate the impact of different caching strategies on the network throghpt-delay tradeoff performance of the proposed FEMOS algorithm. We compare the following two caching strategies. 1) Random Caching: The random caching strategy, where each FAN caches N = 500 files randomly from the file library F. 2) Poplar Caching: The poplar caching strategy is a heristic caching strategy, where each FAN caches the most N = 500 poplar files [34].

10 1178 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018 (a) (b) Fig. 6. Network throghpt-delay tradeoff for FEMOS nder different parameters. (a) Throghpt-delay tradeoff nder different workload, U = 100. (b) Throghpt-delay tradeoff nder different UT amont, A max = 100 kbits. In general, we can see that nder both the random and the poplar caching strategies, the service delay increases as network throghpt increases, which indicates that a proper V shold be chosen to balance the two objects. In addition, we can observe that the FEMOS algorithm with random caching strategy otperforms that with the poplar caching strategy, in terms of the average delay performance. Specifically, the FEMOS algorithm with random caching can redce service delay roghly 16% over the poplar caching for the V vale of C. FEMOS s Performance Under Different Parameters In Fig. 6, we frther explore the relationship between network throghpt and average service delay of FEMOS algorithm regarding to different workloads (A max ) and different UT amont (U). Specifically, in Fig. 6(a) we explore the average network throghpt verss average service delay in terms of different amont of workloads (A max = 60 kbits, 80 kbits, 100 kbits, and 120 kbits, respectively). The random caching strategy is adopted here. It can be observed that for a given control parameter V, the proposed FEMOS algorithm with lower workload obtains better network throghpt performance, and experiences shorter service delay. For example, when V = 2000, the average delay per UT experienced in FEMOS with A max = 120 kbits is 3.2 s, which is abot twice larger than that with A max = 60 kbits. The average network throghpt obtained by FEMOS with A max = 60 kbits is kbits/slot when V = 2000, while it is kbits/slot obtained by FEMOS with A max = 120 kbits. The reason behind these observations is, according to (25) and (26) in Theorem 1, both the lower bond of the average network throghpt φav FEMOS and the pper bond of average service delay FEMOS av are determined by parameters V and K, where K = ( U /2)(μ 2 max +A2 max ). Larger workload leads to larger K, and ths, a smaller average network throghpt and a longer average service delay are obtained. In Fig. 6(b), we show the impacts of UT amont U on the network throghpt-delay tradeoff performance of FEMOS algorithm. Similar phenomenon exists for the crves with different workload. Network throghpt-delay tradeoff also exists nder different UT amont. Besides, it can be observed that by increasing U, the average service delay increases and the average network throghpt decreases for a given V. This is also consistent with Theorem 1 that the UT amont U inflences the vale of parameter K, and ths impacts both throghpt and service delay performance. Intitively, in this case along with different workload scenarios above, the FCN has to flly tilize the network resorces to serve UTs traffic demand, and either high workload or large UT amont certainly will deteriorate both network throghpt and delay performance. D. Fairness Comparison for FEMOS Each UT in the network expects to gain the resorce as well as the QoE fairly. Ths fairness among the UTs is also a significant indicator to characterize a network performance [35]. In this part, we reveal the fairness among the UTs in FEMOS algorithm. As to UTs, the major fairness concern is the experienced average service delay for each of them. We adopt the fairness index of average delay defined in [36], which is given by ( U ) 2 =1 F d = U (28) U =1 2 where is the average service delay experienced by UT, = /λ. Fig. 7(a) depicts the fairness index of average delay verss the control parameter V. On the whole, the fairness index slightly increases as V increases, which reveals that the delay fairness among the UTs can be improved by increasing V, at expense of large delay per UT experienced. For instance, with A max = 100 kbits and U = 100, the fairness index increases from 0.62 to In addition, with a given control parameter V and the UT amont U = 100, it can also be observed that the service delay fairness index as A max increases, which fits the intition that nder high workload, the FCN needs

11 ZHAO et al.: FEMOS IN DYNAMIC WIRELESS NETWORKS 1179 (a) (b) Fig. 7. Fairness index of average delay for FEMOS. (a) Fairness index verss V, U = 100. (b) Fairness index verss UT amont, V = (a) (b) Fig. 8. Evoltion of sm qee length of reqest bffers nder different parameters. (a) Qee backlog verss t, V = (b) Qee backlog verss t, A max = 120 kbits, U = 100. to make more efforts to balance the qee length among the UTs throgh dynamically adjstment of UT-FAN association. Fig. 7(b) investigates the impacts of UT amont on service delay fairness index. It can be observed that the fairness index increases with the increase of UT amont U. To be more specific, the fairness index increases from 0.59 to 0.66 when U increases from 70 to 100, with the given workload A max = 120 kbits and control parameter V = E. Convergence Time Evalation of FEMOS Finally, we condct nmerical experiments regarding to the convergence time nder the proposed FEMOS algorithm by tracing the sm qee length of reqest bffers with different workloads, UT amont and V. Fig. 8(a) depicts the evoltion of sm qee length of reqest bffers nder different workload A max and UT amont U. We can see that all qee backlogs increase at the beginning and are stable eventally for the simlated three cases. Besides, we see that either large UT amont U or high workload A max leads to a longer convergence time. Despite that, qee backlogs for all three cases are able to be stabilized within approximately 1000 time slots, which reveals that the proposed FEMOS algorithm is adaptable to the scenarios with either reasonable amont of UTs or workload. In Fig. 8(b), we illstrate the evoltion of network qee backlogs nder different control parameter V. We observe that the stable qee backlogs increase as V increases, which again confirms that V indeed affects the service delay. Besides, network with large V reqires more time slots to be stabilized. Ths, a proper V needs to be chosen in practice. VII. CONCLUSION In this paper, we investigated the online mltitier operations schedling in fog-enabled network architectre with heterogeneos node capabilities and dynamic wireless network conditions. A low-complexity online algorithm based on Lyapnov optimization was proposed, which achieves at least (1/2) of the optimal vale. Performance analysis, as well as simlations, explicitly characterized the tradeoffs between average network throghpt and service delay, and confirmed the benefit of CAAN and DOBA. For ftre work, we wold like to investigate the proactive FAN assignment and resorce

12 1180 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018 management problem given the availability of predictive information abot UT behaviors, content reqests, traffic distribtions, fading channel statistics, and network dynamics. APPENDIX A PROOF OF LEMMA 1 Note that for any Q 0,μ 0, A 0, we have ( [Q μ] + + A ) 2 Q 2 + A 2 + μ 2 + 2Q(A μ). (29) Plgging the ineqations (29) into the Lyapnov drift fnction, we have L(Q(t + 1)) L(Q(t)) 1 A 2 2 (t) + 1 μ 2 2 (t) + (t)[a (t) μ (t)]. (30) By adding the penalty term Vφ(μ(t)) on both sides of (30) and rearranging the terms, we have L(Q(t + 1)) L(Q(t)) Vφ(μ(t)) K [ V + Q sm (t) ] μ (t) + (t)a (t) (31) where K = ( U /2)[A 2 max + μ2 max ]. Finally, taking the expectation on both sides of (31) onq(t), we can prove the Lemma 1. APPENDIX B PROOF OF LEMMA 2 According to the constraints that 0 < ν h (t) 1 when x h (t) = 1 and ν h (t) = 0ifx h (t) = 0, we know that the UT will obtain the allocated bandwidth from the FAN h (ν h (t) >0 ) if and only if it is associated with the FAN h (x h = 1), or else it wold be ν h (t) = 0. Throgh proof by contradiction, we prove that nder the optimal UT-FAN association X (t) and bandwidth allocation ν (t), each FAN h H associates with only one UT in time slot t and allocates the whole bandwidth to the associated UT. We first assme that nder the optimal UT-FAN association X (t), thefanh associates with more than one UT, i.e., x 1 h = 1, x 2 h = 1, and x h = 0 where U\{ 1, 2 }. Ths we have ν 1 h (t) >0, ν 2 h (t) >0, ν 1 h (t) + ν 2 h (t) = 1, and νh (t) = 0for\{ 1, 2 }. We frther assme that W 1 h(t) W 2 h(t). Letx 1 h (t) = 1, x 2 h (t) = 0 and ν 1 h (t) = ν 1 h (t) + ν 2 h (t) = 1, ν 2 h (t) = 0, then we have W 1 h(t) = W 1 h(t)ν 1 h (t)x 1 h (t) W 1 h(t)ν 1 h (t)x 1 h (t) + W 2 h(t)ν 2 h (t)x 1 h (t) (32) which shows that x 1 h (t) = 1, x 2 h (t) = 0, and ν 1 h (t) = 1,ν 2 h (t) = 0 wold be better soltion for P AR. In this optimal case, the FAN h associates with UT with the maximm W h (t) and allocates the whole bandwidth to it, which is contradictory to the prior assmption. Extend this to all the FANs h H, in which case the optimal UT-FAN association can be obtained throgh solving the optimization problem P AR, where the constraint x h(t) M is redced to x h(t) 1forh H. Then the optimal bandwidth allocation ν h (t) = 1ifx h (t), otherwise, ν h (t) = 0. APPENDIX C PROOF OF LEMMA 3 Using the definitions in Appendix G, we see that I is the intersection of two partition matroids. Ths P AR can be reformlated as max E I { } WE max E I (,h) E W h (t). (33) Invoking the definition abot sbmodlar fnction given in Appendix G, for all E E E, we can dedce that W E (,h) W E = W E (,h) W E (, h) E\E (34) and W = 0, which proves the desired reslt that P AR can be formlated as a normalized modlar fnction with two partition matroid constraints. APPENDIX D PROOF OF LEMMA 4 In the β-redced problem, perform a change of variables μ (t) = μ(t)/β. Using the fact that φ(μ(t)) = μ (t), we have φ(μ (t)) = βμ (t). Ths we prove the reslt that the β-redced problem becomes eqivalent to the problem P 1. Hence, μ,β (t) = βμ,0 (t). APPENDIX E PROOF OF LEMMA 5 Becase the FEMOS algorithm comes from the minimization of the pper bond of V (Q(t)), for any alternative (possibly randomized) imperfect schedling policy X(t) A and ν(t) B, wehave V (Q(t)) K Vφ ( μ,β (t) ) + (t)e { A (t) μ,β (t) Q(t) } (35) where μ,β (t) = [μ,β 1 (t),..., μ,β U (t)] is from the imperfect schedling policy. Invoking Lemma 4 and rearranging right-hand-side of (35), we have V (Q(t)) K Vφ ( μ,β (t) ) + β { (t)e A (t) μ,0 }. (t) Q(t) (36) Plgging in the slater-type conditions into the right-handside of (36), we have (Q(t)) VE{φ(μ(t)) Q(t)} K Vφ ɛ βɛ which proves the desired reslt. (t) (37)

13 ZHAO et al.: FEMOS IN DYNAMIC WIRELESS NETWORKS 1181 APPENDIX F PROOF OF THEOREM 1 Taking the expectation of (37) and sing the law of iterated expectations yields E{L(Q(τ + 1))} E{L(Q(τ))} VE{φ(μ(τ))} K Vφ ɛ βɛ E { (τ) }. (38) Smming the above over τ {0, 1,...,t 1} for some slot t > 0 and sing the law of telescoping sms yields t 1 E{L(Q(t))} E{L(Q(0))} V E{φ(μ(τ))} t 1 K t Vtφ ɛ βɛ E { (τ) }. (39) Note that ɛ>0, then, dividing (39) bytɛ, rearranging terms, and sing the fact E{L(Q(t))} > 0 yields 1 t t 1 E { (τ) } K + V[ φ(μ(t)) φ ɛ ] βɛ + E{L(Q(0))} βɛt (40) where φ(μ(t)) = (1/t) t 1 E{φ(μ(τ))}. Note that φ(μ(t)) φ opt. By taking a lim sp and divide λ on both sides of (40), we have for all t 1 1 t 1 λ lim sp t t E { (τ) } K + V( φ opt φ ɛ ) βɛ λ (41) ths prove the pper bond of FEMOS av in Theorem 1. Now we consider the policy X(t) A and ν(t) B which achieve the optimal soltion φ opt β to the β-redced problem P1. Then we have E{L(Q(τ + 1))} E{L(Q(τ))} VE{φ(μ(τ))} K Vφ opt β. (42) Smming the above over τ {0, 1,...,t 1} for some slot t > 0 and sing the law of telescoping sms yields t 1 E{L(Q(t))} E{L(Q(0))} V E{φ(μ(τ))} K t Vφ opt β t. Dividing by tv, and rearranging terms, we have 1 t t 1 E{φ(μ(τ))} φ opt β (43) K V E{L(Q(0))}. (44) Vt Taking the lim inf as t of both side of (44) proves the lower bond of φ FEMOS av in Theorem 1. APPENDIX G BASIC DEFINITIONS Sbmodlar Fnctions: Let E be a finite grond set. We define its power set (the set containing all the sbsets of E) as 2 E. Then a real-valed fnction f defined on the sbsets of E, f :2 E R is called a sbmodlar fnction if and only if f (E a) f (E) f (E a) f (E ) E E E&a E\E. (45) A real-valed fnction f :2 E R is called a modlar fnction if and only if the eqality holds in (45). The expression f :2 E R is also called normalized fnction if f ( ) = 0, where denotes an empty set. Matroids: A matroid S is a tple S = (E, I), I 2 E sch that: 1) I is nonempty, in particlar, I; 2) I is downward closed: i.e., if Y I and X Y, then X I; 3) If X, Y I, and X < Y, then y Y\X sch that X {y} I. Partition Matroid: The matroid S = (E, I) is also a partition matroid when the grond set E is partitioned into (disjoint) sets E 1, E 2,...,E l and I ={X E : X E i k i for all i = 1,...,l, for some given parameters k 1, k 2,...,k l. REFERENCES [1] S. Zhao, Z. Shao, H. Qian, and Y. Yang, Online ser-ap association with predictive schedling in wireless caching networks, in Proc. IEEE Glob. Commn. Conf. (GLOBECOM), Singapore, 2017, pp [2] Cisco Visal Networking Index: Global Mobile Data Traffic Forecast Update, , Cisco, Feb [3] X. Ge, S. T, G. Mao, C.-X. Wang, and T. Han, 5G ltra-dense celllar networks, IEEE Wireless Commn., vol. 23, no. 1, pp , Feb [4] Y. Shi, J. Zhang, and K. B. Letaief, Grop sparse beamforming for green clod-ran, IEEE Trans. Wireless Commn., vol. 13, no. 5, pp , May [5] R. Wang, X. Peng, J. Zhang, and K. B. Letaief, Mobility-aware caching for content-centric wireless networks: Modeling and methodology, IEEE Commn. Mag., vol. 54, no. 8, pp , Ag [6] K. Yang et al., An efficient and fine-grained big data access control scheme with privacy-preserving policy, IEEE Internet Things J., vol. 4, no. 2, pp , Apr [7] Y. Yang, J. X, G. Shi, and C.-X. Wang, 5G Wireless Systems: Simlation and Evalation Techniqes. New York, NY: USA: Springer, [8] F. Bonomi, R. Milito, J. Zh, and S. Addepalli, Fog compting and its role in the Internet of Things, in Proc. 1st Ed. MCC Workshop Mobile Clod Compt., Helsinki, Finland, 2012, pp [9] L. M. Vaqero and L. Rodero-Merino, Finding yor way in the fog: Towards a comprehensive definition of fog compting, ACM SIGCOMM Compt. Commn. Rev., vol. 44, no. 5, pp , Oct [10] M. Chiang and T. Zhang, Fog and IoT: An overview of research opportnities, IEEE Internet Things J., vol. 3, no. 6, pp , Dec [11] N. Chen et al., FA 2 ST: Fog as a service technology, in Proc. IEEE 41st Ann. Compt. Softw. Appl. Conf. (COMPSAC), Trin, Italy, Jl. 2017, p [12] P. Yang et al., Identifying the most valable workers in fogassisted spatial crowdsorcing, IEEE Internet Things J., vol. 4, no. 5, pp , Oct [13] E. Baştǧ, M. Bennis, M. Kontoris, and M. Debbah, Cache-enabled small cell networks: Modeling and tradeoffs, EURASIP J. Wireless Commn. Netw., vol. 2015, no. 1, p. 41, 2015.

14 1182 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018 [14] J. Li et al., Distribted caching for data dissemination in the downlink of heterogeneos networks, IEEE Trans. Commn., vol. 63, no. 10, pp , Oct [15] J. Li, B. Bai, J. Zhang, and K. B. Letaief, Cache placement in fog- RANs: From centralized to distribted algorithms, IEEE Trans. Wireless Commn., vol. 16, no. 11, pp , Nov [16] Y. Mao, C. Yo, J. Zhang, K. Hang, and K. B. Letaief, A srvey on mobile edge compting: The commnication perspective, IEEE Commn. Srveys Tts., vol. 19, no. 4, pp , 4th Qart., [17] Y. Mao, J. Zhang, S. H. Song, and K. B. Letaief, Stochastic joint radio and comptational resorce management for mlti-ser mobileedge compting systems, IEEE Trans. Wireless Commn.,vol.16,no.9, pp , Sep [18] K. Shanmgam, N. Golrezaei, A. G. Dimakis, A. F. Molisch, and G. Caire, FemtoCaching: Wireless content delivery throgh distribted caching helpers, IEEE Trans. Inf. Theory, vol. 59, no. 12, pp , Dec [19] Y.-Y. Shih, W.-H. Chng, A.-C. Pang, T.-C. Chi, and H.-Y. Wei, Enabling low-latency applications in fog-radio access networks, IEEE Netw., vol. 31, no. 1, pp , Jan./Feb [20] A.-C. Pang, W.-H. Chng, T.-C. Chi, and J. Zhang, Latency-driven cooperative task compting in mlti-ser fog-radio access networks, in Proc. IEEE Int. Conf. Distrib. Compt. Syst. (ICDCS), Atlanta, GA, USA, 2017, pp [21] M. Zink, K. Sh, Y. G, and J. Krose, Characteristics of YoTbe network traffic at a camps network Measrements, models, and implications, Compt. Netw., vol. 53, no. 4, pp , [22] D. Tse and P. Viswanath, Fndamentals of Wireless Commnication. Cambridge, U.K.: Cambridge Univ. Press, [23] E. Biglieri, J. Proakis, and S. Shamai, Fading channels: Informationtheoretic and commnications aspects, IEEE Trans. Inf. Theory, vol. 44, no. 6, pp , Oct [24] E. H. Ong et al., IEEE ac: Enhancements for very high throghpt WLANs, in Proc. 22nd IEEE Int. Symp. Pers. Indoor Mobile Radio Commn. (PIMRC), Toronto, ON, Canada, 2011, pp [25] L. Hang, S. Zhang, M. Chen, and X. Li, When backpressre meets predictive schedling, IEEE/ACM Trans. Netw., vol. 24, no. 4, pp , Ag [26] M. J. Neely, Stochastic Network Optimization With Application to Commnication and Qeeing Systems (Synthesis Lectres on Commnication Networks), vol. 3. San Rafael, CA, USA: Morgan & Claypool, 2010, pp [27] H. Ahlehagh and S. Dey, Video caching in radio access network: Impact on delay and capacity, in Proc. IEEE Wireless Commn. Netw. Conf. (WCNC), Shanghai, China, 2012, pp [28] S. M. Ross, Introdction to Probability Models. Amsterdam, The Netherlands: Academic Press, [29] J. Edmonds, Matroids and the greedy algorithm, Math. Program., vol. 1, no. 1, pp , [30] X. Lin and N. B. Shroff, The impact of imperfect schedling on crosslayer congestion control in wireless networks, IEEE/ACM Trans. Netw., vol. 14, no. 2, pp , Apr [31] L. Georgiadis, M. J. Neely, and L. Tassilas, Resorce allocation and cross-layer control in wireless networks, Fond. Trends R Netw., vol. 1, no. 1, pp , [32] N. Golrezaei, A. G. Dimakis, and A. F. Molisch, Scaling behavior for device-to-device commnications with distribted caching, IEEE Trans. Inf. Theory, vol. 60, no. 7, pp , Jl [33] N. Khan and C. Oestges, Impact of transmit antenna beamwidth for fixed relay links sing ray-tracing and winner ii channel models, in Proc. 5th Er. Conf. Antennas Propag. (EUCAP), Rome, Italy, 2011, pp [34] H. Ahlehagh and S. Dey, Video-aware schedling and caching in the radio access network, IEEE/ACM Trans. Netw., vol. 22, no. 5, pp , Oct [35] H. Shi, R. V. Prasad, E. Onr, and I. G. M. M. Niemegeers, Fairness in wireless networks: Isses, measres and challenges, IEEE Commn. Srveys Tts., vol. 16, no. 1, pp. 5 24, 1st Qart., [36] Y. Yang and T.-S. P. Ym, Mlticode mltirate compact assignment of OVSF codes for QoS differentiated terminals, IEEE Trans. Veh. Technol., vol. 54, no. 6, pp , Nov Shang Zhao received the B.S. degree from the Whan University of Technology, Whan, China, in 2013, and the M.E. degree from the Shanghai Institte of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China, in 2016, where she is crrently prsing the Ph.D. degree at the Key Laboratory of Wireless Sensor Network and Commnication. Her crrent research interests inclde applications of artificial intelligence in wireless commnication, intelligent clod-fog compting networks, and data analytics and algorithm design in fog compting. Yang Yang (S 99 M 02 SM 10 F 18) received the B.Eng. and M.Eng. degrees in radio engineering from Sotheast University, Nanjing, China, in 1996 and 1999, respectively, and the Ph.D. degree in information engineering from the Chinese University of Hong Kong, Hong Kong, in He is crrently a Professor with the Shanghai Institte of Microsystem and Information Technology (SIMIT), Chinese Academy of Sciences (CAS), Shanghai, China, serving as the Director of the CAS Key Laboratory of Wireless Sensor Network and Commnication, the Director of Shanghai Research Center for Wireless Commnications, and the Co-Director of Shanghai Institte of Fog Compting Technology, Shanghai. He is also a Distingished Adjnct Professor with the School of Information Science and Technology, ShanghaiTech University, Shanghai. He has held faclty positions with the Chinese University of Hong Kong, Hong Kong, Brnel University, Uxbridge, U.K., and University College London, London, U.K. He is a member of the Chief Technical Committee of the National Science and Technology Major Project New Generation Mobile Wireless Broadband Commnication Networks which is fnded by the Ministry of Indstry and Information Technology of China. He is on the Chief Technical Committee for the National 863 Hi-Tech Research and Development Program 5G System R&D Major Projects, which is fnded by the Ministry of Science and Technology of China. Since 2017, he has been serving the OpenFog Consortim as a Board member and the Director of Greater China Region. He has athored or co-athored over 160 papers and filed over 80 technical patents in wireless commnications. His crrent research interests inclde wireless sensor networks, Internet of Things, fog compting, open 5G, and advanced wireless testbeds. Ziy Shao (S 06 M 11) received the B.S. and M.S. degrees from Peking University, Beijing, China, and the Ph.D. degree from the Chinese University of Hong Kong, Hong Kong. He is an Assistant Professor with the School of Information Science and Technology, ShanghaiTech University, Shanghai, China. He was a Visiting Research Fellow with Princeton University, Princeton, NJ, USA, in 2012, and a Visiting Professor with the University of California at Berkeley, Berkeley, CA, USA, in His crrent research interests inclde fog compting and clod compting, network intelligence and learning, and optimization and data science.

15 ZHAO et al.: FEMOS IN DYNAMIC WIRELESS NETWORKS 1183 Ximei Yang received the B.S. and M.S. degrees in commnication and information systems from Shandong University, Jinan, China, in 2001 and 2004, respectively, and the Ph.D. degree from the Shanghai Institte of Microsystem and Information Technology (SIMIT), Chinese Academy of Sciences (CAS), Shanghai, China, in She is crrently a Professor with the SIMIT, CAS. She has athored or co-athored over 30 research papers and has (co)-invented over 30 technical patents in wireless commnications. Her crrent research interests inclde heterogeneos networks, fog compting, and Internet of Things. Ha Qian (S 00 M 05 SM 12) received the B.S. and M.S. degrees from the Department of Electrical Engineering, Tsingha University, Beijing, China, in 1998 and 2000, respectively, and the Ph.D. degree from the School of Electrical and Compter Engineering, Georgia Institte of Technology, Atlanta, GA, USA, in He was with indstry, inclding Marvell Semicondctor, Inc., Santa Clara, CA, USA, from 2005 to 2010, as a System Design Engineer. Since 2010, he has been honored with the 100 Talent Program and joined the Shanghai Research Center for Wireless Commnications, Shanghai Institte of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China. He is crrently with the Shanghai Advanced Research Institte, Chinese Academy of Sciences, Shanghai, as a Fll Professor. He is also an Adjnct Professor with ShanghaiTech University, Shanghai. He has over 15 years of research and development experience in signal processing and wireless commnications. He has extensive experience in wireless baseband algorithms and systems. Many designs have already been in prodction. He has co-athored 2 book chapters, over 80 peer-reviewed papers, and holds over 60 patents. His crrent research interests inclde distribted signal processing, signal processing for big data, and signal processing for wireless commnications. Cheng-Xiang Wang (S 01 M 05 SM 08 F 17) received the B.Sc. and M.Eng. degrees in commnication and information systems from Shandong University, Jinan, China, in 1997 and 2000, respectively, and the Ph.D. degree in wireless commnications from Aalborg University, Aalborg, Denmark, in He has been with Heriot-Watt University, Edinbrgh, U.K., since 2005, where he became a Professor of wireless commnications in He is also an Honorary Fellow with the University of Edinbrgh, Edinbrgh, U.K., a Chair Professor with Shandong University, and a Gest Professor with Sotheast University, Nanjing, China. He was a Research Fellow with the University of Agder, Grimstad, Norway, from 2001 to 2005, a Visiting Researcher with Siemens AG-Mobile Phones, Mnich, Germany, in 2004, and a Research Assistant with the Hambrg University of Technology, Hambrg, Germany, from 2000 to He has co-athored 2 books, 1 book chapter, and over 320 papers in refereed jornals and conference proceedings. His crrent research interests focs on wireless channel measrements/modeling and (B)5G wireless commnication networks, inclding green commnications, cognitive radio networks, high mobility commnication networks, massive MIMO, millimeter wave commnications, and visible light commnications. Prof. Wang was a recipient of nine Best Paper Awards of IEEE Globecom 2010, IEEE ICCT 2011, ITST 2012, IEEE VTC 2013-Spring, IWCMC 2015, IWCMC 2016, IEEE/CIC ICCC 2016, and WPMC 2016, and a Highly Cited Researcher Award by the Web of Science in He has served as an Editor for nine international jornals, inclding the IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY since 2011, the IEEE TRANSACTIONS ON COMMUNICATIONS since 2015, and the IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS from 2007 to He was the lead Gest Editor for the IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS Special Isse on Vehiclar Commnications and Networks. He was also a Gest Editor for the IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS Special Isse on Spectrm and Energy Efficient Design of Wireless Commnication Networks and Special Isse on Airborne Commnication Networks, and a Gest Editor for the IEEE TRANSACTIONS ON BIG DATA S Special Isse on Wireless Big Data. He has served as a Technical Program Committee (TPC) member, the TPC Chair, and the General Chair for over 80 international conferences. He is a Fellow of the IET and HEA.