Regret of Multi-Channel Bandit Game in Cognitive Radio Networks

Transcription

1 MAEC Web of Cofereces 56, DOI: / maeccof/ Regre of Muli-Chael Badi Game i Cogiive Radio Neworks Ju Ma ad Yoghog Zhag School of Elecroic Egieerig, Uiversiy of Elecroic Sciece ad echology of Chia (UESC), Chegdu, Chia maupaper@foxmail.com; zhagyhh@uesc.edu.c Absrac. he problem of how o evaluae he rae of covergece o Nash equilibrium soluios i he process of chael selecio uder icomplee iformaio is sudied. I his paper, he defiiio of regre is used o reflec he covergece raes of olie algorihms. he process of selecig a idle chael for each secodary user is modeled as a muli-chael badi game. he defiiio of he maximal averaged regre is give. wo exisig olie learig algorihms are used o obai he Nash equilibrium for each SU. he maximal averaged regres are used o evaluae he performaces of olie algorihms. Whe here is a pure sraegy Nash equilibrium i he muli-chael badi game, he maximal averaged regres are fiie. A cooperaio mechaism is also eeded i he process of calculaig he maximal averaged regres. Simulaio resuls show he maximal averaged regres are fiie ad he olie algorihm wih greaer covergece rae has less maximal averaged regres. Keywords-cogiive radio eworks; adversarial badi problem; cogesio game; olie learig; dyamic specrum access. 1 Iroducio Wih he emergeces of ew wireless services ad applicaios, he demad for specrum icreases. However, some sudies show ha may licesed specrum bads have o bee uilized efficiely [1]. I order o alleviae his coradicio, cogiive radio users called secodary users (SUs) are proposed ad allowed o access specrum bads belog o primary users (PUs) as log as hose bads are sesed o be idle. How o selec a proper chael o sese ad access will affec he idle specrum usage. herefore, he chael selecio problem is impora o each SU. Firsly, some works abou chael selecio uder icomplee iformaio are iroduced. I [2], he problem of how o selec a chael se for a SU o access a a ime has bee ivesigaed uder he codiio ha he saisical iformaio of PUs raffic is assumed as a saioary ad simple disribuio which is ukow o SUs i advace. his work is maily based o classical badi models [3]. Due o he sochasic aure of cogiive radio eworks (CRNs), he real primary raffic disribuio is o always saioary. Uder his sceario, he chael se selecio problem is aalyzed i [4]. Furhermore, a similar problem of muliple SUs who eed o selec ad access a chael a a ime is sudied i [5] cosiderig he effec of he ieracios amog SUs o idividual beefis. Previous works have sudied he chael selecio problem wih icomplee iformaio by badi models. From he perspecive of game heory, he chael selecio problem ca be also modeled as a cogesio game. Cogesio game is a kid of o-cooperaive game wih he uiliy of each player by usig a cerai resource depedig o he oal umber of players who are usig he same resource [6], [7]. Some works i he field of game heory have show ha cogesio games are a kid of poeial game i fac [8]. Poeial games are used o sudy he chael selecio problem i [9] ad he ework selecio problem [10] while some olie learig algorihms are applied o obaiig he Nash equilibrium (NE) soluios. Poeial games always have poeial fucios which ca guaraee a leas a pure sraegy NE [8]. Alhough hese works claim ha NE ca be foud uder icomplee iformaio, hey eed more iformaio ha our model because he poeial fucios will be give before searchig he NE. hese works do o cosider he covergece rae of he olie algorihm. he olie algorihm wih greaer covergece rae will reduce he search ime while icrease he rasmissio daa ime. herefore, he problem of how o evaluae he covergece rae of olie algorihm uder icomplee iformaio should be sudied. I his paper, he chael selecio problem wih icomplee iformaio is modeled as a muli-chael badi game from he view of badi models he MAR is used o evaluae he covergece raes of olie algorihms. he Auhors, published by EDP Scieces. his is a ope access aricle disribued uder he erms of he Creaive Commos Aribuio Licese 4.0 (hp://creaivecommos.org/liceses/by/4.0/).

2 MAEC Web of Cofereces 56, DOI: / maeccof/ Sysem Model ad Problem Formulaio Cosider a CRN wih N SUs as well as S SU base saios ad M (M=S<N) primary chaels which belog o PUs. All SUs ad PUs operae i he slo rasmissio srucure. Each PU has oly oe chael. If ay oe of SU odes was o commuicae wih a SU base saio, i mus use he idle slos of primary chaels. A primary chael us serves a SU base saio. For example, here are N=3, M=S=2 i a CRN. Whe SU =1 commuicaes wih SU base saio A, SU ode =1 ca oly uilize he idle slos of chael m=1 which belogs o PU 1. Similarly, if SU =2 commuicaes wih SU base saio B, SU =1 ca oly uilize he idle slos of chael 2 which belogs o PU 2. Here, we assume coecig differe SU base saio do o affec he furher commuicaio wih he desiaio of each SU. he specrum sesig is assumed o be perfec for each SU ad all primary chaels are idle durig he oal simulaio ime for ease of researchig. his assumpio is raioal. he idle ime of some primary chaels are much loger ha he daa rasmissio duraios of secodary users who have lile daa o rasmi (like he emperaure sesors). he effec of primary raffics o he performaces of covergece raes of olie algorihms will be sudied i our fuure works. Each SU selecs a chael from M chaels a a ime o sese ad access if he chael is sesed o be idle ad he umber of usig his chael is o oo much. Each SU is raioal ad selfish wih he goal of maximizig is ow averaged rasmissio rae durig he process of chael selecio. A each slo, each SU selecs a chael a a ime ad receives a reward if he chael is accessed, which is aalogous o he process of a gambler selecig he arm of a slo machie ad receivig a reward. If a SU rasmis successfully, he reward is he chael rasmissio rae which is give by (1) ad 0 for he rasmissio failure. he chael rasmissio rae of SU a slo i chael m ca be wrie i (1), where W is he chael badwidh, P is he rasmi power ad σ 2 is he hermal oise level, which are same o all SUs for he simplificaio of research. Noe ha g,m deoes he chael gai a slo ad chages over he ime bu keeps uchaged durig each slo. Pg m, m, Wlog21 2 r Whe more ha oe SUs access he same chael simulaeously, he idividual rasmissio success probabiliy decreases because of he muual ierferece. herefore, he rewards obaied by SUs are affeced by he umber of SUs usig he same chael a he same ime. We assume s = (s 1,, s N) is a pure sraegy profile a slo for all SUs, where s deoes he selecio sraegy of SU a slo. c (s ) = ( c 1,, c M) deoes he oal umber of SUs i each chael correspodig o he sraegy profile s a slo. he oal umber c m affecs he successful rasmissio probabiliy p(c m) of SUs who use he chael m a slo. Whe c m icreases, p(c m) of SUs decreases. I his paper, we adop a commo MAC proocol (he sloed (1) Aloha) [7] ad he specific expressio of p(c m) is give i (2). Whe oly oe SU is i he chael m, c m=1, i is cerai o rasmi is daa successfully wih probabiliy 1. As meioed before, we model he chael selecio problem as a badi problem. However, he classic badi model is o fi for our siuaio because he rewards of each accessig a chael for each SU are o idepedely draw from a fixed ad ukow disribuio. 0 cm 0 (2) 1 cm 1 p(c m) 1 1 c 1c 1 ( )(1 ) m m cm cm Here, we uilize a varia of he classic badi model called adversarial badi [11] which is a o-sochasic badi problem o model our sceario. he adversarial badi is also used i [4] o fid he opimal chael. We assume ha all SUs use he same olie algorihm o fid he equilibrium chael for hemselves a he same ime. I our model, we use o deoe he slo a which each user makes a selecio usig olie learig algorihms. his srucure is illusraed i Fig.1. A slo, each SU will selec a chael o access based o a specific disribuio over M chaels. his specific disribuio is decided by he updaig rule of a olie learig algorihm. Afer all SUs have compleed he process of chael selecios a slo, he pure sraegy profile ca be represeed by s. Figure 1. A illusraio of he process of calculaig regres uder icomplee iformaio i our model he followig M-1slos will be used o calculae he regre for each chael which has o seleced a slo. I order o express our idea clearly, (, s -) deoes (s 1,, s -1,, s +1,,s N ), which is s. Whe SU does o selec a chael a slo, i may care abou how much reward SU has los for o playig he pure sraegy a slo uder he codiio ha all he oher SUs keep heir selecio sraegies ( s - ) a slo uchaged. If SU kows is ow payoff fucio ad he sraegy se s a slo, SU ca calculae he regre of o selecig he chael ad he regre R (,) is give as follows [11], R(, ) U, su( s ) (3) Where U deoes he SU s reward fucio, U = r,m if SU rasmis successfully ad oherwise U =0. However, i our model, each SU does o kow is ow payoff fucio. Hece, each SU eeds he cooperaio 2

3 MAEC Web of Cofereces 56, DOI: / maeccof/ wih oher SUs. he cooperaio amog SUs will help each SU o fid he equilibrium chael, which has bee showed i Fig.1. herefore, each SU has a iceive o cooperae wih ohers. We use a example o explai he cooperaio process. For example, if SU does o selec he chael a slo, i may selec chael a slo +1 while oher SUs should selec he chaels which have bee chose a slo, amely, s. I oher words, he chael selecio process will be repeaed a slo +1 ad SU will selec chael wihou doub while oher SUs keep s uchaged. herefore, SU will repea his process for M-1 slos because here are M-1 chaels o be chose a slo. Hece, he ex M-1slos are used o calculae he regres which are produced by o selecig he chael ( M 1) a slo. he selecig order for M-1 chaels is decided by each SU. All SUs kow M ad Whe SU complees he regre calculaios for M-1 chaels, SU +1 will be iformed ad coiue o repea he same process wih SU for M-1 slos uil SU N complees his process. he ex ime for all SUs usig he olie algorihm o selec a chael is a slo +N(M-1). For example, i he Fig.1, he secodary selecio ime decided by olie algorihms is =2. I our model, if he sraegy se (, s -) is he same wih s, he sraegy is o eed o ry agai. he oal umber of selecig he chael decided by olie algorihms is a. If he olie algorihm selecs oal =a imes, he averaged regre of SU for chael ( M) afer a imes is as follows, a 1 R( a) [ U(, s ) U ( )] a s 1 a a = 1 (4) U(, s ) U( ) a s 1 1 max Here, we defie he MAR as R max( R( a)) which is he maximal averaged regre ad use he MAR o evaluae he performaces of olie learig algorihms. 3 Olie Learig Scheme I his secio, we use wo exisig olie learig algorihms o fid he NE soluio ad calculae he MAR for all SUs usig wo exisig olie learig algorihms. he firs oe is based o he algorihm Exp3 [11]. he secod oe is based o he sochasic learig algorihm [12]. 3.1 Olie Learig Algorihm Based o Exp3 Each SU calculaes he selecio probabiliy disribuio P ( ) p,1 ( ),, p,m ( ) over M chaels based o (5) accordig o he correspodig weigh value for each chael. he weigh value is updaed by he ormalized reward based o (8). Here, he ormalized reward is he raio bewee he acual reward ad maximal probabiliy reward which is cosraied by he hardware of SUs. he maximal probabiliy reward i our simulaio par is produced by all radom commuicaio codiios. Algorihm 1: Exp3 based olie learig algorihm 1: Iiialize oal imes a, parameer 0<b<1, R (0)=0, 1 R, m (0)=0, ad he weigh value m, 1 of each chael mm for each SU N. 2: Each SU radomly selecs a chael accordig o he probabiliy pm, ad is expressio is as follows, m, b (5) pm, (1 b) M M m1 m, 3: Each SU receives a reward r, if r 0 ad he R( ) R( -1) r (6) 1 4: Each SU updaes he weigh value of each m, chael based o he ormalized reward updaig rule for each SU is as follows, r (7) m, p m, r. he 1 m, m, m, exp b (8) M 5: Cooperaio begi From =1 o N, each SU selecs chael m from he se of chaels M accordig o he predeermied order ad he selecio sraegy of oher SUs s is always uchaged. SU ca obai he rewards for each chael from he se of chaels M, If he chael m has bee seleced a slo, he reward is U( m, s) r, else R, m ()=R, m (-1)+ U (m, s ) (9) Whe SU complees is process of calculaig he R, m (), he SU +1 is iformed ad coiues uil SU N. 6: Cooperaio ed 7: =+1 ad back o Sep 2. 8: Calculae he MAR based o (4) ad fid he MAR 3.2 Sochasic Learig Algorihm Each SU selecs a chael a radom accordig o a dyamic disribuio P ( ) p,1 ( ),, p,m ( ) over M chaels ad uses he ormalized reward o updae he selecio probabiliy disribuio for he ex roud based o (11). A he firs slo (=1), he probabiliy disribuio P ( ) of selecio chael is assumed as he uiform disribuio over M chaels. Algorihm 2: sochasic learig algorihm 1: Iiialize oal imes a, parameer 0<b<1, R (0)=0, R, m (0)=0 for each chael mm for each SU N, ad P ( 1) is he uiform disribuio over M chaels. 2: Le each SU selec a chael radomly accordig o he probabiliy disribuio P ( ). 3

4 MAEC Web of Cofereces 56, DOI: / maeccof/ : Each SU receives a reward r, if r 0 ad he R ( ) R ( 1) r (10) 4: Each SU updaes P 1 as follows, P ( 1) P ( ) b r ( I P ( )) (11) m where b is he learig rae, r is he ormalized reward, I m is he idicaor fucio. 5: Cooperaio begi From =1 o N, each SU selecs chael m from he se of chaels M ad accordig o he predeermied order ad he selecio sraegy of oher SUs s is always uchaged. SU ca obai he rewards for each chael from he se of chaels M, If he chael m has bee seleced a slo, he reward is U( m, s) r, else R, m ()=R, m (-1)+U (m, s ) (12) Whe SU complees is process of calculaig he R, m (), he SU +1 coiues uil SU N 6: Cooperaio ed 7: =+1 ad back o Sep 2. 8: Calculae he MAR based o (4) ad fid he MAR Durig he cooperaio, if SU has seleced chael m a slo, SU has o eed o ry agai. herefore, we hik U ( m, s ) r. 3.3 he Relaioship bewee NE ad MAR I his subsecio, we use he defiiio of regre o evaluae he covergece raes of olie algorihms. We ca fid based o he heorem 1 ha he MAR will be reduced if he selecio sraegy coverges o he equilibrium chael quickly. heorem 1: Whe he muli-chael badi game has a pure sraegy NE, he MAR of olie algorihms for each SU is fiie. Proof: he oal selecio imes which is decided by olie algorihms is a. We assume chael J is he equilibrium sraegy for SU. A a special selecio ime ', each SU will selec opimal chael J ad does o chage is selecio afer slo '. I oher words, whe he oal a is large eough, a 1 R( ) ' ( U J, s U( s )) 0afer each SU passes ' a slo '. his is because he (J, s -) is he same wih s a slo = ' ad he reward of each accessig he opimal chael J is radom for SU. Whe all SUs do o chage heir selecios, he resul of searchig opimal chael will coverge o a NE SU. herefore, a [ U( J, s) U( s )] ad max R max( R( a)). 1 4 Simulaio Resuls he performaces of olie learig algorihms are evaluaed i his secio. here are N=4 SUs ad M=2 primary chaels i a CRN coverig a 500m 500m area. S=2 SU base saios locae i (0,250), (500,250) which are accessible for all SUs. Whe he 4 SUs are seleced radomly, he locaios of SUs are fixed durig he simulaios. All SUs adop he chael model of Fla/Ligh ree desiy proposed i [13]. he rasmissio power is 10-3 W ad he oise power level for all SUs is assumed o be W. he badwidh for all SUs is assumed as 1Hz ad he learig rae b=0.01 for he wo algorihms. I figures, a is showed i abscissa axis, which is from 1000 o Each a rus 100 simulaios. (a) he equilibrium chael for SU 1 is chael 1 (b) he equilibrium chael for SU 2 is chael 2 (c) he equilibrium chael for SU 3 is chael 2 (d) he equilibrium chael for SU4 is chael 1 Figure 2. Idividual profis obaied from differe chaels usig wo olie learig algorihms. 4

5 MAEC Web of Cofereces 56, DOI: / maeccof/ We ca obai NE based o (2) which is ha SU1 ad SU4 selecs chael 1 as well as chael 2 for SU2 ad SU3. he NE sraegy ca be wrie as (1, 2, 2, 1). Firs, we are ieresed i he behaviors of SUs i differe chaels. I Fig. 2, we show he idividual accumulaed rasmissio raes of each SU obaiig from chael 1 ad chael 2 usig algorihm 1 (A1) ad algorihm 2 (A2). From he Fig. 2, we ca see ha each SU has a domia sraegy which is cosise wih he pure sraegy NE (1,2,2,1). From Fig. 2(a), we ca fid SU 1 has obaied more idividual profis from he chael 1 regardless of A1 ad A2. he same resul is also showed i Fig. 2(d). herefore, he chael 1 is he equilibrium soluios for boh SU 1 ad SU 4. I Fig. 2(b), SU 2 has received more idividual profis from selecig he chael 2 ad so dose SU 3 which is showed i Fig.2(c). I Fig.3, we ca oe ha all MARs are fiie. I order o evaluae he covergece raes of wo olie algorihms, we compare he MARs of wo olie algorihms for all SUs i Fig. 3. From Fig.3, we ca see he MAR of he equilibrium sraegy for each SU by A1 is more ha A2. Due o he lower covergece rae, each SU usig A1 will selec o-equilibrium chael more frequely, which will icrease he MAR. he differeces of MARs bewee SUs usig he same olie algorihm are depede o he differe locaios of SUs. Figure 3. Compariso of 4 SUs MAR of wo selecio sraegies 5 Coclusio I his paper, we have modeled he chael selecio problem uder icomplee iformaio from he perspecive of coecio bewee he badi model ad he game model. Whe he chael selecio game has a pure sraegy NE, he MAR of selecio sraegies is fiie. wo exisig olie learig algorihms are used o obai he equilibrium chael for all SUs. he sochasic learig algorihm ouperforms he Exp3 based olie learig algorihm i erms of covergece rae o he Nash equilibrium soluio, which is because he MAR of sochasic learig algorihm is less ha Exp3. his work provides he coecio bewee o-cooperaive games ad badi problems, which will help us o illusrae he covergece rae of differe olie algorihm wihou kowig he payoff fucios ad oher SUs selecio sraegies. Refereces 1. B. Wag ad K.J.R. Liu, Advaces i cogiive radio eworks: A survey, IEEE J. Sel. opics Sigal Process., vol. 5, o. 1, pp. 5 23, Feb Z. Zhou, J. Hai,. Peg, ad J. Slevisky, Chael exploraio ad exploiaio wih imperfec specrum sesig i cogiive radio eworks, IEEE J. Sel. Areas Commu., vol. 31, o. 3, pp , Mar Auer P, Cesa-Biachi N, Fischer P. Fiie-ime aalysis of he muliarmed badi problem, Mach. Lear., vol.47, o.2-3, pp , X. Fag, D. Yag, ad G. Xue, amig wheel of forue i he air: a algorihmic framework for chael selecio sraegy i cogiive radio eworks, IEEE ras. Veh. echol., vol. 62, o. 2, pp , Feb K. Liu ad Q. Zhao, Disribued learig i muliarmed badi wih muliple players, IEEE ras. Sigal Process., vol. 58, o. 11, pp , Nov L. Blumrose ad S. Dobziski, Welfare maximizaio i cogesio games, IEEE J. Sel. Areas Commu., vol. 25, o. 6, pp , Aug L. M. Law, J. Huag, ad M. Liu, Price of aarchy for cogesio games i cogiive radio eworks, IEEE ras. Wireless Commu., vol. 11, o. 10, pp , Oc D.Moderer ad L.S.Shapley, Poeial games, Games Eco. Behav., vol. 14, pp , Y. Xu, J. Wag, Q. Wu, A. Apalaga, ad Y.-D. Yao, Opporuisic specrum access i ukow dyamic evirome: a game-heoreic sochasic learig soluio, IEEE ras. Wireless Commu., vol. 11, o. 4, pp , Apr Li-Chua, C. Feg-su, Z. Daqiag, e al., Nework selecio i cogiive heerogeeous eworks usig sochasic learig, IEEE Commu. Le., vol. 17, o. 12, pp , Dec P. Auer, N. Cesa-Biachi, Y. Freud, ad R. E. Schapire, he o-sochasic muliarmed badi problem, SIAM Joural o Compuig,vol. 32, o. 1, pp , Nov P. Sasry, V. Phasalkar, ad M. hahachar, Deceralized learig of ash equilibria i muliperso sochasic games wih icomplee iformaio, IEEE ras. Sys., Ma, Cyberm., vol. 24, o. 5, pp , May V. Erceg, L. J. Greesei, S. Y. adra, e al., A empirically based pah loss model for wireless chaels i suburba eviromes, IEEE J. Sel. Areas Commu., vol. 17, o. 7, pp , Jul