Cognitive Base Stations in LTE/3GPP Femtocells: ACorrelatedEquilibrium Game-Theoretic Approach

Transcription

1 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 59, NO. 12, DECEMBER Cogniive Base Saions in LTE/3GPP Femocells: ACorrelaedEquilibrium Game-Theoreic Approach Jane Wei Huang, Suden Member, IEEE, andviramkrishnamurhy,fellow, IEEE Absrac This paper considers downlin specrum allocaion in a long erm evoluion (LTE) sysem macrocell which conains muliple femocells. By incorporaing cogniive capabiliies ino femocell base saions, he Home evolved Node Bs (HeNBs) can be formulaed as secondary base saions seeing o maximize he specrum uiliy while minimizing inerference o primary base saions (evolved Node-Bs). The compeiion amongs cogniive HeNBs for specrum resources is formulaed as a non-cooperaive game-heoreic learning problem where each agen (HeNB) sees o adap is sraegy in real ime. We formulae he resource bloc (RB) allocaion among HeNBs in he downlin of a LTE sysem using a game-heoreic framewor, where he correlaed equilibrium soluions of he formulaed game are being invesigaed. AdisribuedRBaccessalgorihmisproposedocompuehe correlaed equilibrium RB allocaion policy. Absrac LTE/3GPP sysem, cogniive base saions, femocells, self-organized newor, correlaed equilibrium, gameheoreic learning. 3GPP 4G AWGN CCI DFP DSL enb HeNB LTE Mbps OFDMA QoS RB SON UMTS WLAN GLOSSARY 3rd Generaion Parnership Projec 4h Generaion Addiive Whie Gaussian Noise Co-Channel Inerference Dynamic Frequency Planning Digial Subscriber Line Evolved Node-B Home Evolved Node-B Long Term Evoluion Megabi Per Second Orhogonal Frequency-Division Muliple Access Qualiy of Service Resource Bloc Self-Organized Newor Universal Mobile Telecommunicaion Sysem Wireless Local Area Newor I. INTRODUCTION A NimporanfeaureofLongTermEvoluion(LTE)/3rd Generaion Parnership Projec (3GPP) sysems [1] is ha i allows disribued implemenaion of femocells o mee a variey of service requiremens. The femocell access poins, denoed as Home evolved Node-B (HeNB) in 3GPP, are lowcos, low-power, plug-and-play cellular base saions. In order Paper approved by E. G. Larsson, he Edior for Game Theory and Communicaions Sysems Opimizaion ofheieeecommunicaionssociey. Manuscrip received November 23, 2010; revised June 6, The auhors are wih he Deparmen of Elecrical and Compuer Engineering, The Universiy of Briish Columbia, Main Mall, Vancouver, BC V6T 1Z4, Canada ( {janeh, viram}@ece.ubc.ca). Digial Objec Ideni er /TCOMM /11$25.00 c 2011 IEEE o provide broadband conneciviy, hese HeNBs will need o possess adapive/cogniive faciliies. In he Ocober 2010 release of 3GPP, HeNBs are described as self-opimized nodes in a Self-Organized Newor (SON) which need o mainain qualiy of service (QoS) wih minimal inervenion from he service operaor [2]. HeNBs are equipped wih cogniive funcionaliies for load balancing, inerference managemen, random access channel opimizaion, capaciy and coverage opimizaion, and handover parameeropimizaion. Wih he above moivaion, his paper considers specrum resource allocaion in an orhogonal frequency division muliple access (OFDMA) LTE downlin sysem which consiss of a macrocell base saion (evolved Node-B (enb)) and muliple femocell base saions (HeNBs). By incorporaing cogniive capabiliies ino hese self-opimized femocell base saions, he cogniive HeNBs aim o maximize he specrum uiliy by uilizing he unoccupied frequencies while minimizing inerference o he enb (primary base saion) in a specrum overlay LTE sysem. The uni of specrum resource o be allocaed in a LTE sysem is called a resource bloc (RB) and i is comprised of 12 subcarriers a a 15 Hz spacing. Given he RB occupancy of he enb, he compeiion for he specrum resources among HeNBs can be formulaed in a game-heoreic seing [3]. Insead of compuing he Nash equilibrium policy of he formulaed game, we see o characerize and compue he correlaed equilibrium policy se [4], [5]. The se of correlaed equilibria is a convex polyope. I includes he se of Nash equilibria indeed he Nash equilibria are isolaed poins a he exrema of his se [6], [7]. The se of correlaed equilibria [5] is arguably he mos naural aracive se for a decenralized adapive algorihm such as he one considered here, and describes a condiion of compeiive opimaliy beween agens (cogniive femocell base saions). I is more preferable han Nash equilibria since i direcly considers he abiliy of agens o coordinae heir acions. This coordinaion can lead o higher performance han if each agen was required o ac in isolaion. Indeed, Har and Mas-Colell observe in [8] ha, for mos simple adapive procedures,... here is a naural coordinaion device: he common hisory, observed by all players. I is hus no reasonable o expec ha, a he end, independence among players will obain. Since he se of correlaed equilibria is convex, fairness beween players can be addressed in his domain. Finally, decenralized, online adapive procedures naurally converge o he correlaed equilibria, whereas he same is no rue for Nash equilibria (he so-called law of conservaion of coordinaion [9]).

2 3486 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 59, NO. 12, DECEMBER 2011 Relaed wor: There are several imporan issues being addressed in recen lieraure regarding he deploymen of HeNB femocells in a LTE sysem. One area of ineres is how o miigae he inerferences among HeNBs so as o improve he sysem performance. In [10] and [11], Lopez-Perez e al. used Dynamic Frequency Planning (DFP), an inerference avoidance echnique, o decrease he iner-cell inerference and increase he newor capaciy by dynamically adaping he radio frequency parameers o he speci c scenario.they also veri ed he performance of he DFP echnique in a OFDMA WiMAX macrocells and femocells sysem. Choi e al. invesigaed in [12] how o minimize he inerference caused by femocells in an open access (specrum underlay) newor. They showed adapive open access will maximize he value of a femocell boh o is owner and o he newor using numerical resuls. Aar e al. sudied he bene s of developing cogniive base-saions in a UMTS LTE newor [13]. Radio resource managemen proocols are no speci ed by sandards, such as 3GPP s UMTS LTE. Thus, here is considerable exibiliy in heir design. The insuf ciency of radiional coexisence soluions in LTE conex is shown in [13]. I is argued ha cogniive base-saions are crucial o an ef cien and disribued radio resource managemen of LTE given is disribued archiecure. One moivaion for such argumen, is he lessons learn from wide-spread deploymen of wireless local area newor (WLAN) access poins. The simple plugand-play naure of WLAN rouers, along wih he unlicensed naure of WLAN specrum access, alleviaed he need for ime and cos-inensive newor plannings. This in urn helped a rapid proliferaion of WLAN hospos. However, as he number of coexising WLAN newors increases, so does heir muual inerfering effec, rendering such simple, sel sh coexisence sraegies problemaic. By incorporaing he main hree cogniive radio capabiliies ino he LTE base-saions, which are 1) radio scene analysis 2) online learning based on he feedbac from RF environmen 3) and agile/dynamic resource access schemes asuccessfulcoexisencesraegyamongenbsandhenbscan be achieved. Main Resuls: We formulae he RB allocaion among HeNBs in he downlin LTE sysem as a game in Secion II. Given he RB usage of he enb in a LTE macrocell, he cogniive HeNBs are modelled as sel sh players compeing o acquire he common frequency resources. This framewor borrows he idea of cogniive radio sysems [14], [15], where we formulae he cogniive HeNBs as secondary users and he enb as he primary user in he shared specrum sysem. A global uiliy funcion is de ned o evaluae he overall LTE newor performance. To achieve he opimal global uiliy value in a disribued se-up, we also de ne a local uiliy funcion for each cogniive HeNB. This localuiliycomprisesof componens ha incorporae self-ineres, fairness and power consumpion of each HeNB. Secion III-A de nes he correlaed equilibrium of a game. A RB access algorihm (Algorihm 1) is proposed which converges o he correlaed equilibrium soluion. This RB Fig. 1. Sysem schema of a single enb macrocell which conains muliple HeNB femocells in a LTE/3GPP newor. access algorihm is based on he regre maching algorihm [8], [16], [17] and i has a disribued naure as each cogniive HeNB does no require he informaion of oher HeNBs. We also prove ha his proposed algorihm will converge o he correlaed equilibrium se of he formulaed game in Secion III-C. Finally, numerical examples are give in Secion IV. II. RESOURCE ALLOCATION AMONG HENBS: PROBLEM FORMULATION We consider a macrocell area wih a number of femocells randomly deployed by home and of ces wihin a OFDMA LTE newor (Fig. 1). By incorporaing he cogniive capaciies ino femocell base saions, he specrum occupancy behaviour of he macrocell base saion (enb) can be formulaed as a primary base saion and ha of he femocell base saions (HeNBs) can be formulaed as secondary base saions (cogniive base saions) in a specrum overlay newor 1.Due o he sel sh naure of each base saion, he compeiion for he common specrum resource among cogniive base saions can be formulaed using a game-heoreic framewor. A. Sysem Descripion The resource allocaion process in LTE newors follows a ime sloed sysem model where each ime slo lengh equals o ha of a RB (0.5 ms), wih RB being he smalles uni of resource ha can be allocaed. Le {1, 2,...,T} denoe he ime slo index. T de nes he ime horizon of he formulaed game. Le N denoe he oal number of available RBs in he sysem, Nhenb denoe he number of RBs occupied by HeNBs a ime and NeNB denoes he number of RBs occupied by enb a ime. Asweconsideraspecrumoverlaysysem, Nhenb (N N enb ). Le K denoe he oal number of HeNBs coexising in he newor, K= {1, 2,...,K} denoe he user index and f {1, 2,...Nhenb } denoe he RB index occupied by 1 As described in [13], i is also possible o sudy an underlay cogniive femocell sraegy in which RBs accessed by enb and HeNBs are no orhogonal. However, as he objecive of our analysis is modelling he compeiion among sel sh HeNBs, he proposed soluions can be readily exended o he aforemenioned specrum underlay LTE sysems.

3 HUANG and KRISHNAMURTHY: COGNITIVE BASE STATIONS IN LTE/3GPP FEMTOCELLS: A CORRELATED EQUILIBRIUM GAME-THEORETIC APPROACH 3487 cogniive base saions (HeNBs). We use p (f) {0, 1} o denoe he acion of he h HeNB on he fh RB a ime, where 0 represens no ransmi and 1 represens ransmi. Le p = {p (1),...,p (N henb )} P denoe he acion of he h HeNB over all he available RBs o HeNBs a ime, wihp denoing he acion space of he h HeNB. p = {p 1,...,p K } Pis used o denoe he composiion of he acions from all he HeNBs a ime. P is he join acion space of all HeNBs. Use s (f) {1, 2,...,Q s} o denoe channel qualiy sae of he h HeNB on he fh RB a ime afer quanizaion. For example, he channel qualiy can be obained by quanizing a coninuous valued channel model comprising of circularly symmeric complex Gaussian random variables ino Q s differen saes. Le s = {s (1),...,s (N henb )} denoe he channel sae composiion of he h HeNB over all he available RBs and s = {s 1,...,s K } denoe he channel sae composiion over all he HeNBs. Le I (f) denoe he inerference inroduced o he h HeNB a ime on he f h RB. The inerference comprises wo pars, namely, noise and co-channel inerference (CCI). The noise n (f) is assumed o be addiive whie Gaussian noise (AWGN) wih a noise covariance of σ 2 (f) and he CCI is inroduced by having differen HeNBs sharing he same RB. An inerference marix w (f) is inroduced o model he CCI among all HeNBs on he fh RB a ime. Theelemens of his inerference marix, i.e., wi,j (f) where i, j K, denoe he cross channel qualiy beween he ih and jh HeNBs. Assuming channel reciprociy, w (f) is a symmeric marix, i.e., wi,j (f) =w j,i (f). Iisclearhaheinerference marix has zero-valued diagonal elemens, i.e., wi,i (f) =0 for i K.Thevalueofw (f) depends on he locaion of all he HeNBs. We assume he h ( K) HeNB has he following informaion a he beginning of a ime slo. 1) Nhenb :henumberofavailablerbsforhenbs, 2) s,i.e.,hechannelsaevecorofheh HeNB, 3) I (f) which is he received inerference a HeNB on RB f a ime, 4) he curren demand level of he h HeNB which is denoed by d. Nhenb can be obained hrough a xed broadband access newor (e.g., DSL, Cable) as described in [18]. s and I (f) can be obained by channel sensing mechanism during he guard inerval a he beginning of each ime slo. d is used o denoe he sysem resource demand level of HeNB a ime, iisofhesameuniashaofsysemcapaciy.d is deermined by he user requiremen wih in a HeNB cell a ime. Inhecasehaherearemoredevicesransmi daa using frequency bandwidh wihin HeNB a ime, d is of higher value. An imporan characerisic of his model is ha radio-speci c quaniyd needs only o be nown o he h HeNB cell, hus allowing decenralized resource allocaion algorihms. Based on he above informaion, HeNB chooses is acion vecor p,sel shly, o maximize is local uiliy funcion. The de niion of uiliy will be presened in Secion II-B. Noe ha in he case ha HeNBs are owned by differen agens and hey are so sophisicaed o behave maliciously, hey can op no o reveal heir rue demand levels d ( =1,...,K)oopimizeheirownuiliiesahecosof reducing he overall sysem performance. I requires mechanism design heory in order o preven his from happening. Similar problems has been sudies in [19] where he auhors applied pricing mechanism o ensure each raional sel sh user maximize is own uiliy funcion, a he same ime opimizing he overall sysem uiliy. Repuaion based mechanism design is anoher area of research, where sysem uses repuaion as aoolomoivaecooperaionbeweenusersandindicaea good behaviour wihin he newor. If a user does no pay aenion o is repuaion and eep acing maliciously, i will be isolaed and discarded. Such repuaion based mechanism has been applied in ad hoc newors and sensor newors [20], [21]. However, his paper assumes all he malicious behaviours have been eliminaed in he sysem and each HeNB uses is rue demand level d o compue is uiliy funcion. The disribued decision maing process amongs HeNBs de nes he acion se p,whichinurnleadsoadifferen realizaion of inerference level I (f), K and f = {1,...,Nhenb }.Therefore,heacionofonefemocellbase saion (HeNB) affecs he uiliies of oher femocells, which moivaes he use of game-heoreic approaches o analyze and compue he RB allocaion policies among all he HeNBs. In he following subsecion, we de ne he global sysem uiliy funcion and he local uiliy funcions for femocell base saions. B. Uiliy Funcion The goal of his paper is o opimize he global resource allocaion problem using a decenralized approach. We should demonsrae a connecion global uiliy funcion and he local uiliy funcion ha will guide he allocaion decision of each HeNB. This connecion is presened hrough he derivaion of global and local performance measures. This subsecion de nes a global uiliy funcion o evaluae he overall sysem performance, based on which a local uiliy funcion is de ned for each cogniive HeNB. Each HeNB sel shly maximizes is local uiliy funcion which does no guaranee he global sysem performance. We aim o design local uiliy funcions which ensure global sysem performance qualiy a he correlaed equilibrium of he formulaed game. Le C denoe he capaciy of HeNB a ime. If a capaciy-achieving code such as urbo code or low-densiy pariy-chec code (LDPC) code is applied for error correcion, C can be expressed as follows using Shannon-Harley s heorem [22]. C = N henb f=1 I (f) = σ 2 (f)+ [ ω log 2 1+ p (f) s (f) ] I (f) K w,i(f) p i(f), (1) i=1 where ω denoes he bandwidh of a RB. We assume ha all he HeNBs rea he inerferences as Gaussian noises. Noe ha w, =0for =1,...,K.

4 3488 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 59, NO. 12, DECEMBER ) Global Uiliy Funcion: Since all cogniive HeNBs have equal prioriy in accessing sysem resources, a global objecive is chosen o maximize he performance of he worsoff HeNB such ha he available resources are fairly allocaed among HeNBs. Therefore, given he acion vecor p of all he HeNBs, he global uiliy funcion a ime is de ned as follows. U G (p ) = min K [ C min( d, 1) ]. (2) Here, he erm min( C, 1) represens he saisfacion level d of he h HeNB and i is a funcion of he insananeous capaciy of he h HeNB a ime divided by is curren demand level. Noe ha mahemaically (2) is equivalen o U G (p C ) = min K,operaionmin( C d, 1) is applied d because he saisfacion level is wihin he range of [0, 1] The acion vecor among all he HeNBs a ime p is chosen o maximize he minimum saisfacion level among all he K HeNBs. Tha is, p =argmax U G(p) =max min [ C min( p P p P K d, 1) ]. (3) This global uiliy opimizaion problem (3) aims o maximize he minimum saisfacion level among all he K HeNBs. Noe he global uiliy funcion can be of oher forms, e.g., aiming o maximize he average saisfacion level among all he K HeNBs, in which case he global uiliy funcion can be speci ed as follows (4). However, his paper focuses on he scenario where he global uiliy is chosen o maximize he wors-off user (2). U G (p )= 1 [ C min( K d, 1) ]. (4) K The sysem objecive is o nd he acion vecor p so as o maximize he saisfacion level of he wors-off HeNB U G (p ). The proposed approach o achieve he correlaed equilibrium policy in a decenralized way, is o allow each cogniive HeNB choosing is acion p ( K)basedonheopimizaion of is local uiliy funcion. The quesion ha remains o be answered is how o selec a proper local uiliy funcion which also ensures a good overall sysem performance. In he nex subsecion, we will derive such a uiliy funcion and deermine is relaion o he global uiliy objecive. 2) Local HeNB Uiliy Funcion: If each cogniive HeNB has a reasonable esimae of he global uiliy funcion, U G, hen he decenralized resource allocaion policy can be direcly realized by each cogniive HeNB choosing an acion which maximizes is esimae of U G.However,asheglobal uiliy funcion also depends on he privae informaion of oher players, i.e., he demand levels d i and acions p i (i K and i = ) of oher HeNBs, such an assumpion is no pracical. Below, we consruc a local uiliy funcion U for HeNB ( K)consisingofhreepars,whereeachparaddresses acerainaspecofheproblemahand. The rs par of he local uiliy funcion re ecs he self ineres componen of a cogniive HeNB, given by, U [1](p ) = min ( C d, 1 ). (5) Maximizing U [1](p ) is equivalen o maximizing HeNB s porion of he global uiliy U G (2). However, a game wih (5) as he only componen of he localuiliyfuncionwould resemble a congesion game, which may no be solvable in closed form. Moreover, (5) only shows he self ineresed par of he global objecive (2) and i neglecs he iner-relaion of decisions of each player on he achieved uiliy of oher players. Since each HeNB nown only is own demand leven d and acion p, we induce such ineracion hrough he following wo principles. 1) A each ime, he capaciy of he h HeNB C should no exceeds is demand level d,asileavesless resource o oher HeNBs. 2) Each HeNB should minimize is ransmission power, as higher ransmission power decreases he performance of oher HeNBs by inroducing higher inerferences. The rs principle is sais ed by inroducing he second componen of he local uiliy funcion where a penaly is inroduced if he choice of resources of a HeNB exceeds is demand level. The deails of he second componen of he local uiliy funcion is shown as follows: U [2](p )= 1 d ( C d ) +, (6) where (x) + denoes he operaion max(x, 0). The second local uiliy componen suppresses greedy HeNB behaviour, and i brings C closer o is demand d.thislocaluiliy componen also helps o mainain he fairness among all he HeNBs. The hird componen of he local uiliy funcion is used o implemen he second principle by considering power as par of he cos of a HeNB, hus, encouraging each HeNB o minimize is power consumpion. The deails of he hird componen is shown as follow: Nhenb U [3](p )= p (f). (7) f=1 Noe ha we assume uni power ransmission of all he HeNBs on each of he RB in our sysem model. In his sysem model, each HeNB is assumed o be a sel sh user aiming o maximize is own uiliy funcion wih he minimum cos. By including he power consumpion cos as par of he local uiliy funcion, i helps he local uiliy o represen he global uiliy. Based on he above de niions, he local uiliy funcion of ahenb can be de ned as follows. U (p )=U [1](p )+α 2 U [2](p )+α 3 U [3](p ), (8) where α 2 and α 3 are he weighing facors inroduced o combine he hree uiliy componens assuming U [1] has a uni weighing facor, i.e., α 1 =1.Theseweighingfacors are necessary because he acual effec of each componen is unnown. By carefully adjusing he values of (α 2,α 3 ),we can change he effec of each of he hree uiliy componens, which hen enable he local uiliy o mimic he behaviour of he global uiliy (2) in he bes way. Thus, he quesion remains o be how o choose (α 2,α 3 ) which lead o he bes overall sysem performance (2). This paper does no provide aclosedformsoluionohisquesion,insead,wechoosehe

5 HUANG and KRISHNAMURTHY: COGNITIVE BASE STATIONS IN LTE/3GPP FEMTOCELLS: A CORRELATED EQUILIBRIUM GAME-THEORETIC APPROACH 3489 weigh facors according o he numerical sudies in (IV). The seleced weighing facors (α 2,α 3 ) does no ensure he maximizaion of he insananeous global uiliy funcion a ime, insead, i maximizes he expeced sysem performance { under differen channel realizaions, i.e., E s 1,s UG (P ) }. 2,...s K Recall ha a cogniive HeNB, K,riesomaximize is uiliy funcion U (p ) sel shly by choosing he acion vecor p a he beginning of each ime slo. In he following secions, we will show he exisence of he correlaed equilibrium soluion, given he disribued decision maing process of HeNBs in a saic environmen. III. CORRELATED EQUILIBRIUM SOLUTIONS WITH A GAME-THEORETIC APPROACH This secion uses game-heoreic approach o formulae he resource allocaion among cogniive femocell base saions (HeNBs) in a saic environmen, each of HeNB is formulaed as a sel sh game player aiming o maximize is local uiliy funcion (8). A saic environmen is where he sysem parameers (e.g., he channel saisics s ( K), he primary base saion behaviour and he number of HeNBs K) are consans or slowly evolve wih ime. We invesigae he correlaed equilibrium soluion of his saic game, which can be obained via a disribued RB access algorihm. The algorihm is an adapive varian of he regre maching procedure of [8]. I dynamically adaps he behaviour of HeNBs o ime varying environmen condiions. We also prove he RB access algorihm converges o he correlaed equilibrium se of he de ned game. A. De niion of Correlaed Equilibrium In a K-player (HeNB) game se-up, each HeNB ( K) is a sel sh game player aiming o devise a rule for selecing an acion vecor p a each ime slo o maximize (he expeced value of) is uiliy funcion U (p ).Sinceeachplayeronly has conrol over is own acion p, he opimal acion policy depends on he raional consideraion of he acion policies from oher users. We focus on he correlaed equilibrium soluion of he considered game [4], [5], his soluion is an imporan generalizaion of he Nash equilibrium and is de ned as follows. De niion 3.1: De ne a join policy π o be a probabiliy disribuion on he join acion space P = P 1 P 2...P K. Given acions of oher players p,hepolicyπ is a correlaed equilibrium, if for any alernaive policy ˆp P,iholdsha, π(p, p )U (p ) π(p, ˆp )U (ˆp ). p P p P Correlaed equilibrium can be inuiively inerpreed as if π provides he K players a sraegy recommendaion from he rused hird-pary. The implici assumpion is ha he K 1 oher players follow his recommendaion, and player as iself wheher i is of is bes ineres o follow he recommendaion as well. The equilibrium condiion saes ha here is no deviaion rule ha could award player abeerexpeceduiliyhanπ. AnyNashequilibriumcan be represened as a correlaed equilibrium when users can generae heir recommendaions independenly. One of he advanages of using correlaed equilibrium is ha i permis coordinaion among users, generally hrough observaion of a common signal, which leads o an improvedperformanceover anashequilibrium[5]. B. Decenralized RB Access Algorihm The RB access algorihm is an adapive exension of he regre maching procedure [8], i enables HeNBs o adap heir policies o ime varying sysem environmen. Le H (p ) denoe he regre marix of he h cogniive HeNB a ime, i is of size P P wih is ( i, j )h enry (i, j P ) speci ed as, H ( i, j ) (p )=1 (p =i) [U (j, p ) U (i, p )], (9) where 1 ( ) is an indicaor funcion. Furhermore, de ne θ o be he overall regre marix of HeNB and i is also of size P P.Theregrevalueθ ( j, i ) measures he average gain of user a ime if had chosen acion i in he pas (from ime 0 o ) inseadofj. Ifhegainisposiive, is more liely o swich o acion i in he fuure, oherwise, is more liely o say wih j. Speci cally, policy ransiion marix is a funcion of he curren regre marix θ 1 as speci ed in (10). Noe his regre based scheme requires users o now he reward for each acion, even if ha acion is no aen. Users updaes is policy based on he calculaed regre marix (10). ARBaccessalgorihmisproposedocompuehecorrelaed equilibrium resource allocaion policy, he deails of which are lised in Algorihm 1. Algorihm 1 LTE Cogniive HeNB RB Access Algorihm Sep 1 Se =0;Iniializep 0 and se θ 0 = H (p 0 ) for K. Sep 2 for =1, 2, 3,... do Acion Updae: For K,choosep = i wih probabiliy = P(p = i p 1 1 i =j = j, θ 1 ) max(θ 1 ( j, i ),0) μ max(θ 1 ( j, i ),0) μ if i = j if i = j (10) Regre Marices Updae: Based on he new acion, he overall regre marices are updaed according o he following sochasic approximaion algorihm wih sep size ε. end for θ = θ 1 + ε (H (p ) θ 1 ), K. (11) Algorihm 1 can be summarized as follows. Sep 1 iniializes he sysem by seing ime index =0and he iniial values of p 0 and θ 0. Sep 2 is he main ieraion of he algorihm which is composed of wo pars: acions updae and regre marices updae. A HeNB chooses is acion for ime slo according o he curren acion p 1 and he regre marix θ 1.In(10),μis a consan parameer which is chosen o be μ i =j max(θ 1 ( j, i ), 0) o ensure he probabiliies are non-negaive. The choice of sep size ε used in (11), can be eiher a decreasing sep size ε =1/ or a small consan sep size ε = ε (0 <ε 1). If sysem parameers do no evolve wih ime, using decreasing sep size convergences he algorihm o he correlaed equilibrium se wih probabiliy

6 3490 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 59, NO. 12, DECEMBER 2011 one. Using a consan sep size enables he algorihm o rac of he correlaed equilibrium se if sysem parameers slowly evolve wih ime. Depending on he selecion of he sep size ε,heregre marix θ can be rewrien as follows. θ ( j, i ) = 1 l,p l =j ( U (i, p l ) U (j, p l ) ), if ε = 1 ; θ ( j, i ) = ε(1 ε) l( U (i, p l ) U (j, p l ) ), l,p l =j if ε = ε. (12) The above RB access algorihm is a modi caion of he regre maching procedure in [8]. In he regre maching approach, he decisions of each HeNB is based on he average hisory of all pas observed performance resuls. This choice, however, is no desirable in our scenario since he sysem parameers may vary over ime. Insead, our algorihm adaps he regre marices θ ( K) according o he updaed sysem parameers which capures he ime-varying naure of he sysem. The decenralized feaure of his RB access algorihm permis is implemenaion among he disribued femocells in a LTE/3GPP newor. Regre marix θ is one of he ey parameers in Algorihm 1, based on which a cogniive HeNB adjuss is fuure acion. The inerpreaion of he ( i, j )h enry of θ is ha measures he average gain ha a cogniive HeNB would have, had i chosen acion j in he pas (i.e, in he ( 1)h ime slo) insead of i. C. Convergence of RB Access Algorihm By using he resul from [23], we inroduce Theorem 3.2 which proves ha he RB access algorihm (Algorihm 1) converges o he correlaed equilibrium under cerain condiions. Le θ denoe he regre marix of he h HeNB when.useγ Ω (θ ) o denoe he projecion of parameer θ on Ω, whereω is he closed negaive orhan of R p p. Le <x,y>denoe he inner produc of x and y. Theorem 3.2: The RB access algorihm (Algorihm 1) is ensured o converge o he correlaed equilibrium se of he formulaed game. Proof: By using Proposiion 3.8 and Corollary 5.7 in [23], we now if every HeNB follows he sraegy in Algorihm 1, i is enough o prove ha he following inequaliy, given by (13), holds in order o prove ha Algorihm 1 converges o he se of correlaed equilibria of he de ned game. < θ Γ Ω (θ ), θ Γ Ω (θ ) > 0. (13) Condiion (13) is originaed from he Blacwell s suf cien condiion for approachabiliy [24]. Therefore, we only need o demonsrae ha (13) holds in order o prove he convergence of RB access algorihm. Noe ha he negaive orhan Ω is a convex se. The lef hand side of (13) can be expressed as, < θ Γ Ω (θ ), θ Γ Ω (θ ) > = < θ Γ Ω (θ ), θ > < θ Γ Ω (θ ), Γ Ω (θ ) >, where < θ Γ Ω (θ ), Γ Ω (θ ) >= 0due o he de niion of projecion. Thus, in order o esablish (13) we need o prove < θ Γ Ω (θ ), θ > 0. Le us consruc a Marov chain wih he ransiion probabiliy speci ed in (10) and use τ i (i P )odenoehe saionary disribuion of such a Marov chain. τ i can be speci ed as follows. τ i = j = i τ j θ + ( j, i ) μ + τ i ( 1 i = j θ + ( i, j ) ), (14) μ where μ is a consan chosen o be μ> j = i θ+ ( j, i ). Then, (14) is equivalen o he following equaion. τ j θ + ( j, i ) =τ i ( i, j ). (15) j = i j = i Since he projecion is on he negaive orhan Ω, θ Γ Ω (θ )=θ +. < θ Γ Ω (θ ), θ > can hen be wrien as < θ Γ Ω (θ ), θ >=< θ +, θ > = θ + ( j, i )[ U (i, p ) U (j, p ) ] τ j j i =j = θ + ( i, j )U (j, p )τ i i =j j θ + ( j, i )U (j, p )τ j j i =j = [ θ + ( i, j )τ i ] θ + ( j, i )τ j U (j, p ) j i =j i =j = 0. (16) Therefore, he condiion saed in (13) is proved o hold. This concludes he proof of Theorem 3.2. D. Correlaed Equilibrium under Dynamic Environmens and Curse of Dimensionaliy In he case ha sysem conains large number of acive mobile users, i causes high dynamic macro base saion behaviour. In which scenario, he problem can be described as: resource allocaion among femocells (HeNBs) in a OFDMA LTE downlin sysem under a dynamic environmen where he resource occupancy behaviour of he primary base saion (sysem sae) is varying quicly while oher sysem parameers (e.g., number of HeNBs) are consans or evolving slowing. By formulaing he dynamic of sysem sae as a Marov chain, i requires Marov game-heoreic approach o formulae he resource allocaion problem among femocell base saions (HeNBs) as a Marov game. Differen from saic game, sysem sae and sae ransiion probabiliies are imporan elemens in dynamic games as hey absrac he imevarying naure of a dynamic environmen. A reasonable choice of he sysem sae is [Nhenb, s 1,...,s K ] which is composed of he number of available RBs for HeNBs and he channel θ +

7 HUANG and KRISHNAMURTHY: COGNITIVE BASE STATIONS IN LTE/3GPP FEMTOCELLS: A CORRELATED EQUILIBRIUM GAME-THEORETIC APPROACH 3491 saes of HeNBs. By de ning he correlaed equilibrium or Nash equilibrium of such a dynamic sochasic game, differen opimizaion algorihms can be used o compue he equilibrium ransmission polices. E.g., [25] proposed ieraive value opimizaion algorihm and sochasic approximaion algorihm o compue he Nash equilibrium policies in he formulaed Marovian game. Poenial applicaions nowihsanding, here remains subsanial hurdles in he applicaion of dynamic sochasic games as a modelling ools in pracice. Discree-ime sochasic games wih nie number of saes are cenral o he analysis of sraegic ineracions among sel sh HeNBs in dynamic environmen. The usefulness of discree-ime games, however, is limied by heir compuaional burden; in paricular, here is curse of dimensionaliy. In a discree-ime dynamic sochasic game, each game player (HeNB) is disinguished by an individual sae a each ime slo. The sysem sae is a vecor encoding he number of players wih each possible value of he individual sae variable [N henb, s 1,...,s K ].Thesysem sae is exponenial o he number of he HeNBs in he sysem. How o ef cienly reduce he sae space is ye an issue o solve before he implemenaion of sochasic games in LTE sysems. One direcion of research is o consider coninuousime sochasic game models. E.g., [26] aims o reduce he dimensionaliy by exploring he alernaive coninuous-ime sochasic games wih a nie number of saes and show ha he coninuous ime has subsanial advanages. IV. NUMERICAL EXAMPLES Algorihm 1 is designed o compue he correlaed equilibrium policies for cogniive HeNBs in he downlin of a OFDMA LTE/3GPP sysem. This secion illusraes he performance of he RB access algorihm (Algorihm 1) in a game se-up. For demonsraion purpose, we consider K =6 HeNBs. For he h HeNB ( K), is channel qualiy a any RB f (f F)iss (f) {1, 2, 3}, isdemandlevelbelongs o he se d {10, 20, 30, 40}.Theacionofheh HeNB a he fh RB is speci ed as p (f) {0, 1}, where0 represens no ransmission and 1 represens ransmi. In he simulaion se-up, we specify he noise covariance o be σ 2 (f) =0.1 ( Kand f F). In his simulaion model, he number of available RB for HeNBs is xed o be N henb =10. Noe ha a LTE enb macrocell usually consiss large number of HeNB femocells. The reason we specify K =6HeNBs and N henb =10in our simulaion is because he policy space grows exponenially wih he number of available RBs for HeNBs under he curren problem formulaion. Thus, i is necessary o reduce he acion space dimensionaliy before he widely applicaion of he proposed algorihm. In he presence of large number of HeNBs, HeNBs can be lumped ino a single oher player (named a composie player). The acion space of ha composie player can be chosen appropriaely and he RB access algorihm can hen be applied. In rs example, we are going o invesigae he impac of he pricing parameers (α 2,α 3 ) in he local uiliy funcion (8) on differen global uiliies (2) and (4). The pricing parameers is chosen o ensure he global sysem performance. This is an off-line calculaion procedure. The simulaion resuls in Sysem Performance (Wors Player) The Effec of Differen Selecions of ( 2, 3 ) on he Sysem Performance Fig. 2. The effec of differen values of (α 2,α 3 )de ned in (8) on he global sysem performance speci ed in (2). Fig. 2 and Fig. 3 are averaged over 1000 ieraions, where 20 differen scenarios wih differen noise n (f), channelsaes s are being considered in each ieraion. In boh gures, x- axis and y-axis denoe α 2 and α 3,respecively.Z-axisdenoes he global sysem performance speci ed in (2) in Fig. 2, while i denoes he sysem average performance speci ed in (4) in Fig. 3. Based on Fig. 2, we noice he compleely sel sh behaviours from HeNBs (α 2 =0,α 3 =0)donoensureheopimumof global sysem performance; while α 2 =5, α 3 =0.25 lead he leas saisfacion level among HeNBs o 0.9, i.e.,u G (p )= 0.9. Wewillspecifyα 2 =5and α 3 =0.25 in he simulaion for Fig. 4. Fig. 3 shows he sysem average performance (4) is less sensiive o he change of pricing parameers (α 2,α 3 ).Sysem average performance is of similar level when he pricing parameers are of he range 0 α 2 7, 0.05 α Thus, in he case we choose (4) as he globaluiliyfuncion,he selecion of (α 2,α 3 ) is no unique, hey can be of any values wihin he above range. I can also be noiced from Fig. 3 ha he sysem average performance decrease dramaically when α 2 > 7. Thiscanbeexplainedasfollows:ifhepower consumpion cos is greaer han a cerain hreshold, a HeNB will choose no o ransmi as he payoff is less han he cos. Thus, a very high power consumpion cos weighing facor (α 3 )canhaveanegaiveeffeconhenbperformances. The nex example (Fig. 4) compares he performance of he proposed RB access algorihm wih he exising Bes Response algorihm wih a global uiliy funcion speci ed in (2). In he simulaion, we use small consan sep size and i is speci ed as ε = ε =0.05. Theresulsareaveragedover 50 scenarios and here are 2000 ieraions. Bes Response is a simple case where each HeNB chooses is acion p a each ime slo solely maximizing is local uiliy (8) and each HeNB assumes ha he acions of oher HeNBs are xed. Thus, Bes Response is a special case of he proposed RB access algorihm wih he sep size chosen o be ε =1.TheacionupdaeinheBesResponseisno afuncionofhepreviousregreθ 1 bu only a funcion of he curren insananeous regre marix H (p ).FromFig

8 3492 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 59, NO. 12, DECEMBER 2011 Sysem Average Performance The Effec of ( 2, 3 ) on Sysem Average Performance Fig. 3. The effec of differen values of (α 2,α 3 )de ned in (8) on he global sysem average performance speci ed in (4). Performance (Wors Player) Ieraion Index Sysem Performance in Saic Enviromen Bes Response 6 8 RB Access Algorihm Fig. 4. Performance comparison beween RB access algorihm (Algorihm 1) and he Bes Response algorihm. we can see ha he sysem performance using he RB access algorihm reaches 0.9 afer 1400 ieraions and he resul from he Bes Response algorihm says a around 0.6. I can be seen ha he RB access algorihm (Algorihm 1) improves he sysem performance grealy compared o he Bes Response algorihm. We can also observe from Fig. 4 ha boh RB access algorihm and Bes Response algorihm do no converge o consan values, i is because ransmission policy p converges o a correlaed equilibrium se when and he correlaed equilibrium se has more han one correlaed equilibrium policy. V. CONCLUSIONS We have proposed implemenaion of cogniive femocell base saions for resource bloc allocaion in he downlin of a enb macrocell LTE/3GPP sysem. By considering he enb as he primary base saion, he HeNBs are formulaed as muliple secondary base saions compeing for specrum resources. The RB allocaion problem is formulaed in a saic environmens, using saic game framewor. A RB access algorihm is proposed o compue he correlaed equilibrium 10 policy in such a environmen. We also prove ha he RB access algorihm converges o he correlaed equilibrium se of he formulaed game. Numerical examples are used o verify he performances of he proposed algorihm. VI. ACKNOWLEDGEMENTS The auhors would lie o han Dr. Alireza Aar for his valuable suggesions and help. REFERENCES [1] 3GPP, Technical Speci caion and Technical Repors for a UTRAN-based 3GPP sysem, vol. TR v0.0.8, Available: hp:// FeaureOrSudyIemFile hm. [2], 3GPP wor iems on Self-Organizing Newors, Oc Available: hp:// [3] B. Wang, Y. Wu, and K. J. R. Liu, Game heory for cogniive radio newors: an overview, Inernaional J. Compu. Telecommun. New., vol. 54, no. 14, pp , Oc [4] R. J. Aumann, Subjeciviy and correlaion in randomized sraegies, J. Mahemaical Economics, vol.1,pp.67 96,Mar [5], Correlaed equilibrium as an expression of Bayesian raionaliy, Economerica, vol.55,no.1,pp.1 18,1987. [6] O. Morgensern and J. v. Neumann, The Theory of Games and Economic Behavior. PrinceonUniversiyPress,1947. [7] J. Nash, Equilibrium poins in n-person games, in Proc. Naional Academy Sciences, 1950,pp [8] S. Har and A. Mas-Colell, A simple adapive procedure leading o correlaed equilibrium, Economerica, vol.68,no.5,pp , Sep [9], Uncoupled dynamics do no lead o Nash equilibrium, American Economic Review, vol.93,no.5,pp ,Dec [10] D. Lopez-Perez, G. d. l. Roche, A. Valcarce, A. Juner, and J. Zhang, Inerference avoidance and dynamic frequency planning for WiMAX femocells newor, in Proc. IEEE Singapore Inernaional Conf., 2008, pp [11] D. Lopez-Perez, A. Juner, and J. Zhang, Opimisaion mehods for dynamic frequency planning in OFDMA newors, in Proc. New., Sep. 2008, pp [12] D. Choi, P. Monajemi, S. Kang, and J. Villasenor, Dealing wih loud neighbors: he bene s and radeoffs of adapive femocell access, in Proc. IEEE GLOBECOM, 2008,p.15. [13] A. Aar, V. Krishnamurhy, and O. Namvar, Inerference managemen using cogniive base-saions for UMTS LTE, o appear in IEEE Commun. Mag., [14] J. Miola III, Cogniive radio for exible mobile mulimedia communicaions, in Proc. IEEE Inernaional Worshop Mobile Mulimedia Commun., 1999,pp [15] S. Hayin, Cogniive radio: brain-empowered wireless communicaions, IEEE J. Sel. Areas Commun., vol.23,no.2,pp ,Feb [16] V. Krishnamurhy, M. Masery, and G. Yin, Decenralized adapive lering algorihms for sensor acivaion in an unaended ground sensor newor: a correlaed equilibrium game heoreic analysis, IEEE Trans. Signal Process., vol.56,no.12,pp ,Dec [17] M. Masery, V. Krishnamurhy, and Q. Zhao, Decenralized dynamic specrum access for cogniive radios: cooperaive design of a noncooperaive game, IEEE Trans. Commun., vol.57,no.2,pp , Feb [18] 3GPP, Overview of 3GPP, vol. Release 9 V0.1.1, Sep Available: hp:// [19] J. W. Huang and V. Krishnamurhy, Truh revealing opporunisic scheduling in cogniive radio sysems, in Proc. SPAWC, June2009, pp [20] K. Ren, T. Li, Z. Wan, R. H. Deng, and K. Kim, Highly reliable rus esablishmen scheme in ad hoc newors, J. Compu. Telecommun. New., vol.45,no.6,pp ,Aug [21] H. Chen, Tas-based rus managemen for wireless sensor newors, Inernaional J. Securiy Appl., vol.3,no.2,pp.21 26,Apr [22] I. C. Wong and B. L. Evans, Opimal downlin of OFDMA resource allocaion wih linear complexiy o maximize ergodic raes, IEEE Trans. Wireless Commun., vol.7,no.3,pp ,Mar.2008.

9 HUANG and KRISHNAMURTHY: COGNITIVE BASE STATIONS IN LTE/3GPP FEMTOCELLS: A CORRELATED EQUILIBRIUM GAME-THEORETIC APPROACH 3493 [23] M. Benam, J. Hofbauer, and S. Sorin, Sochasic approximaions and differenial inclusions par II: applicaions, Mahemaics Operaion Research, vol.31,no.4,pp ,Nov [24] D. Blacwell, An analog of he minmax heorem for vecor payoffs, Paci c J.Mahemaics,vol.6,pp.1 8,1956. [25] J. W. Huang and V. Krishnamurhy, Transmission conrol in cogniive radio as a Marovian dynamic game srucural resul on randomized hreshold policies, IEEE Trans. Commun., vol.58,no.2,pp , Feb [26] U. Doraszelsi and K. L. Judd, Avoiding he curse of dimensionaliy in dynamic sochasic games, under revision a Rand J. Economics, Jane Wei Huang (S 07) received her bachelor s degree from Zhejiang Universiy, China in 2005, and M.Phil. from Hong Kong Universiy of Science and Technology, Hong Kong in She is currenly a Ph.D. suden in he Universiy of Briish Columbia. Her research ineress include game heory, Marov decision process, cogniive radio, and sensor newors. Viram Krishnamurhy (S 90-M 91-SM 99-F 05) was born in He received his bachelor s degree from he Universiy of Aucland, New Zealand in 1988, and Ph.D. from he Ausralian Naional Universiy, Canberra, in He is currenly a professor and holds he Canada Research Chair a he Deparmen of Elecrical Engineering, Universiy of Briish Columbia, Vancouver, Canada. Prior o 2002, he was a chaired professor a he Deparmen of Elecrical and Elecronic Engineering, Universiy of Melbourne, Ausralia, where he also served as depuy head of deparmen. His curren research ineress include compuaional game heory, sochasic dynamical sysems for modeling of biological ion channels and sochasic opimizaion and scheduling. Dr. Krishnamurhy has served as associae ediorforseveraljournalsincluding IEEE TRANSACTIONS AUTOMATIC CONTROL, IEEETRANSACTIONS ON SIGNAL PROCESSING, IEEETRANSACTIONS AEROSPACE AND ELEC- TRONIC SYSTEMS, IEEETRANSACTIONS NANOBIOSCIENCE, andsysems and Conrol Leers. From heservesasdisinguishedlecurerfor he IEEE signal processing sociey. From 2010, he serves as edior in chief of IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING.