Channel Selection in Cognitive Radio Networks with Opportunistic RF Energy Harvesting

1 Channel Selection in Cognitive Radio Networks with Opportunistic RF Energy Harvesting Dusit Niyato 1, Ping Wang 1, and Dong In Kim 2 1 School of Computer Engineering, Nanyang Technological University (NTU), Singapore 2 School of Information and Communication Engineering, Sungkyunkwan University (SKKU), Korea Abstract Radio frequency (RF) energy harvesting is a promising technique to sustain an operation of wireless networks. In a cognitive radio network, a secondary user can be equipped with RF energy harvesting capability. We consider such a network where the secondary user can select one of the channels to transmit data when it is not occupied by a primary user, and to harvest RF energy when the primary user transmits data. Specifically, we formulate an optimization problem to determine an optimal channel selection policy for the secondary user. The secondary user selects a channel based on the energy level in its battery (i.e., energy queue) and the number of packets in its data queue. The optimization considers complete information and incomplete information cases, where the secondary user has and does not have the knowledge about channel states, respectively. The performance obtained in the complete information case can serve as an upper bound for the secondary user. Index Terms RF energy transfer, cognitive radio, Markov decision process I. INTRODUCTION Recently, radio frequency (RF) energy harvesting technique with high efficiency has been introduced. Such a technique allows a wireless node to collect and convert electromagnetic wave from ambient sources (e.g., TV, radio towers and cellular base stations) into energy which can be used for data transmission. A few studies have shown the feasibility of the RF energy harvesting (e.g., [1], [2], [3]). For example, the study in [1] showed that with the transmit power of 0.5W by a mobile phone, 40mW, 1.6mW, and 0.4mW of power can be harvested at the distance of 1, 5, and 10 meters, respectively. With RF energy harvesting capability, the wireless especially mobile node can perpetuate its operation without physically changing or recharging its battery. A few works considered and addressed different issues of adopting RF energy harvesting in wireless networks. For example, in sensor networks, power sources could be mobile and move to sensor nodes not only to collect sensed data, but also to transfer energy wirelessly [4], [5]. A MAC protocol was modified and developed to accommodate RF energy transfers [6]. Radio resource allocation was optimized (e.g., capacity maximization given RF energy supply constraint) since the spectrum now can be used for both data and energy transfer [7]. Similarly, in cooperative and multihop networks, relays are not just to forward data, but also transfer energy [8], [9], thus the data and energy routing can be jointly optimized [10]. In traditional cognitive radio network, secondary users can opportunistically access unoccupied channels licensed to primary users. Such an approach can significantly improve the spectrum utilization and transmission performance of the secondary users. However, the secondary users can be constrained by their energy supply for data transmission, and in this case, the RF energy harvesting can be adopted. The secondary users with RF energy harvesting capability not only access the spectrum, but also harvest RF energy opportunistically from the transmission of primary users. Such a scenario was first considered in [11]. The main idea is that a mobile secondary user can opportunistically harvest energy if it moves close to a primary transmitter. Alternatively, the secondary user can opportunistically access a channel and transmit data if it moves away from a primary receiver to avoid interference. However, the channel selection issue was ignored in [11]. In this paper, we consider the cognitive radio network with multiple primary users transmitting on different channels and a secondary user equipped with RF energy harvesting capability. We address the channel selection problem of the secondary user aiming to maximize its throughput. An optimization problem based on Markov decision process (MDP) is formulated and solved taking the states of an energy storage and a data queue of the secondary user into account. In particular, we consider two cases for the availability of channel state information for channel selection. 1 Complete channel state information: The secondary user knows the states of all channels (i.e., idle or busy due to a primary user s transmission). This can be achieved if external entities (e.g., spectrum sensors) facilitate spectrum sensing. The secondary user uses this information to select a channel accordingly. Incomplete channel state information: The secondary user does not know the state of any channel. Therefore, it has to select a channel to sense. The secondary user transmits a packet or harvests RF energy if the selected channel is sensed to be idle or busy, respectively. The spectrum sensing error can happen and affect the decision of the secondary user. The performance evaluation reveals some interesting results. For example, it is not always beneficial for the secondary user if a channel becomes more idle. This negative effect is due 1 By channel state we mean the channel has idle or busy status. 978-1-4799-2003-7/14/$31.00 2014 IEEE 1555

2 to the fact that the secondary user cannot harvest enough RF energy from the primary user for its own data transmission, leading to a decrement of throughput. The rest of this paper is organized as follows. Section II describes the system model and assumptions used in this paper. Sections III and IV present the optimization formulations for the incomplete information and complete information cases, respectively. Section V presents the numerical performance evaluation results. Section VI provides summary of the paper. II. SYSTEM MODEL Fig. 1. System model. We consider a cognitive radio network with N primary users and a secondary user (Fig. 1). The primary user n is allocated with the non-overlapping channel n. The primary user uses this channel to transmit data in a time slot basis and all the primary users align to the same time slot. Therefore, in one time slot, the channel can be idle or busy (i.e., occupied by the primary user for data transmission). We consider the secondary user with RF energy harvesting capability. Specifically, the secondary user can select one of the channels. If the selected channel is busy, the secondary user can harvest energy from the primary transmitter. Let γ n denote the probability that the secondary user harvests one unit of RF energy successfully from channel n. The harvested energy is stored in an energy queue, whose size is E units of energy. On the other hand, if the selected channel is idle, the secondary user can transmit a packet retrieved from its data queue. The secondary user requires W units of energy for data transmission in a time slot. The probability of successful packet transmission on channel n is denoted by σ n. The probability of a packet arrival for the secondary user in a time slot is denoted by α. The arriving packet is buffered in the data queue of the secondary user. The maximum capacity of the data queue is Q packets. Note that the batch packet arrival and transmission can be extended straightforwardly. Here, single packet arrival and transmission are considered for presentation simplicity only. The secondary user has to perform channel selection for RF energy harvesting or packet transmission. In this paper, we consider two cases. Complete information: In this case, we assume that the secondary user has a complete information about states of all channels. This can be realized if there is a secondary base station or spectrum sensors to perform comprehensive spectrum sensing and communicate the sensing result with the secondary user (e.g., using a common control channel). The secondary user can select a channel to harvest RF energy or to transmit a packet based on this channel state information. Incomplete information: The secondary user does not have the information about states of channels. Therefore, the secondary user selects a channel based on statistical information (e.g., a probability of a channel to be idle). After selecting the channel, the secondary user performs spectrum sensing to observe the channel state. If the channel state is idle or busy, the secondary user will transmit a packet or harvest RF energy, respectively. When the secondary user makes a channel selection decision with complete information, an upper-bound performance (e.g., throughput) can be achieved. However, with incomplete information, the performance will be deteriorated. The difference of these performance can quantify the value of (channel state) information. The value-of-information analysis can be performed using the proposed performance analysis and optimization models to be presented later in the paper. Consider spectrum sensing error of the secondary user on a selected channel. The miss detection probability and falsealarm probability are denoted by m n and f n for channel n, respectively. The miss detection happens when the actual channel state is busy, but the secondary user senses it to be idle. In contrast, the false alarm happens when the actual channel state is idle, but the secondary user senses it to be busy. We assume that a channel is modeled as a two-state Markov chain. The transition probability matrix of channel n is denoted by [ ] C0,0 (n) C C n = 0,1 (n) idle (1) C 1,0 (n) C 1,1 (n) busy where 0 and 1 correspond to idle and busy states, respectively. The probability of the channel n to be idle is denoted by η n = 1 C 1,1 (n) C 0,1 (n) C 1,1 (n)+1. III. OPTIMIZATION FORMULATION: INCOMPLETE INFORMATION In this section, we consider a Markov decision process (MDP) for the incomplete information case. Firstly, we define the state and action spaces. Then, we derive the transition probability matrix and obtain an optimal channel selection policy. A. State Space and Action Space We define the state space of the secondary user as follows: } Θ incom = (E, Q); E 0, 1,...,E}, Q 0, 1,...,Q} (2) where E and Q represent the energy level of the energy queue and the number of packets in the data queue of the secondary 1556

3 user, respectively. The action space of the secondary user is defined as follows: Δ=1,...,n,...,N}, where the action δ Δ is the channel to select for sensing and harvesting RF energy or transmitting a packet. B. Transition Probability Matrix We denote the transition probability matrix given action δ Δ of the secondary user by B 0,0 (δ) B 0,1 (δ) B 1,0 (δ) B 1,1 (δ) B 1,2 (δ) P I (δ) =......... B Q,Q 1 (δ) B Q,Q (δ) (3) where each row of matrix P I (δ) corresponds to the number of packets in the data queue (i.e., the data queue state). Matrix B q,q (δ) represents the data queue state transition from q in the current time slot to q in the next time slot. Each row of matrix B q,q (δ) corresponds to the energy level in the energy queue. There are two major cases for deriving matrix B q,q (δ), i.e., q =0and q>0. Forq =0, there is no packet transmission, since the data queue is empty. As a result, the energy level will never decrease. Let B 0 (δ) denote a common matrix for q =0.WehaveB 0 (δ) = 1 ηδ m δ γ δ ηδ m δ γ δ...... 1 ηδ m δ γ δ ηδ m δ γ (4) δ 1 where ηδ =1 η δ and m δ =1 m δ. Each row of matrix B 0 (δ) corresponds to the energy level e. In this matrix, the energy level of the energy queue increases only when the selected channel δ is busy, there is no miss detection, and the secondary user successfully harvests RF energy. Then, B 0,0 (δ) = B 0 (δ)α and B 0,1 (δ) = B 0 (δ)α, where α = 1 α, for when there is no and there is a packet arrival, respectively. For q>0, we have three sub-cases, i.e, when the number of packets decreases, remains the same, and increases. The number of packets decreases, when the selected channel is idle (with probability η δ ), there is no false alarm (with probability fδ =1 f δ) and no packet arrival (with probability 1 α), the packet transmission is successful (with probability σ δ ), and there is enough energy in the energy queue, i.e., e W. The corresponding matrix is defined as follows: B q,q 1 (δ) = 0... η δ fδ α σ δ η δ fδ α σ δ. (5)... η δ fδ α σ δ 0 The first W rows of the matrix B q,q 1 (δ) correspond to the energy level e =0,...,W 1, which is not sufficient for the secondary user to transmit a packet. As a result, there is no change of the number of packets in the data queue. Accordingly, all elements in these rows are zero. The first η δ fδ α σ δ appears at row W +1 which corresponds to the energy level of W, which is sufficient for packet transmission. Therefore, the number of packets in the data queue can decrease together with the energy level decreasing by W units. The number of packets in the data queue can remain the same. The transition matrix is expressed as follows: B q,q (δ) = α b (δ) α b (δ)...... α b (δ) α b (δ) b W,0 (δ) b W,W (δ) b W,W +1 (δ).......... b E,E W (δ) b E,E (δ) (6) Again, the first W rows correspond to the case of not enough energy for packet transmission without packet arrival. Therefore, the energy level can remain the same with probability b (δ) or can increase with probability b (δ), but cannot decrease. The energy level increases if the channel is busy, there is no miss detection and energy is successfully harvested, i.e., b (δ) =ηδ m δ γ δ. Accordingly, the energy level remains the same with probability b (δ) =1 ηδ m δ γ δ. The rows W +1 to E +1 of matrix B q,q (δ) correspond to the case of having enough energy for packet transmission. When the number of packets in queue remains the same, we have the following cases. Firstly, we derive the probability that the energy level decreases by W units, denoted by b e,e W (δ). This case happens when the channel is idle with no false alarm, no packet arrival, and unsuccessful packet transmission or the channel is idle with no false alarm, a packet arrival and successful packet transmission, or the channel is busy with miss detection (i.e., the secondary user transmits and collides with a primary user), and no packet arrival. Therefore, we have b e,e W (δ) =η δfδ (σ δ α + σ δ α)+ ηδ m δα, where σδ =1 σ δ. Secondly, we derive the probability that the energy level remains the same, denoted by b e,e(δ). This case happens when the channel is busy with no miss detection, no energy successfully harvested and no packet arrival, or the channel is idle with false alarm (i.e., the secondary user defers transmission), and no packet arrival. Therefore, we have b e,e(δ) =ηδ m δ γ δ α + η δ f δ α. Thirdly, we derive the probability that the energy level increases by one unit, denoted by b e,e+1(δ). This case 1557

4 happens when the channel is busy with no miss detection, energy successfully harvested, and no packet arrival. Therefore, we have b e,e+1(δ) =ηδ m δ γ δα. Note that for e = E, the energy level cannot increase more than the capacity of the energy queue, and hence b E,E (δ) = b E 1,E 1 (δ)+b E 1,E (δ). The number of packets in the data queue can increase. The transition matrix is expressed as follows: B q,q+1 (δ) = αb (δ) αb (δ)...... αb (δ) αb (δ) b + W,0 (δ) b+ W,W (δ) b+ W,W +1 (δ).......... b + E,E W (δ) b+ E,E (δ) (7) The first W rows (i.e., not enough energy to transmit a packet) is similar to that of B q,q (δ), but with a packet arrival. Similarly, there are three cases for rows W +1 to E +1, when the number of packets in the data queue increases. Firstly, we derive the probability that the energy level decreases by W units, denoted by b + e,e W (δ). This case happens when the channel is idle with no false alarm, unsuccessful packet transmission, and a packet arrival or the channel is busy with miss detection, and a packet arrival. Therefore, we have b + e,e W (δ) =η δfδ σ δ α + η δ m δα. Secondly, we derive the probability that the energy level remains the same, denoted by b + e,e(δ). This case happens when the channel is busy with no miss detection, no energy successfully harvested and a packet arrival, or the channel is idle with false alarm, and a packet arrival. Therefore, we have b + e,e(δ) =ηδ m δ γ δ α + η δf δ α. Thirdly, we derive the probability that the energy level increases by one unit, denoted by b e,e+1(δ). This case happens when the channel is busy with no miss detection, energy successfully harvested, and a packet arrival. Therefore, we have b e,e+1(δ) =ηδ m δ γ δα. Again, for e = E, the energy level cannot increase more than the capacity of the energy queue, and hence b + E,E (δ) = b + E 1,E 1 (δ)+b+ E 1,E (δ). For the case when the data queue is full, the transition matrix is obtained as follows: B Q,Q (δ) =B Q 1,Q 1 (δ) + B Q 1,Q (δ). C. Optimal Policy Then we formulate an optimization problem based on an MDP. Specifically, we will obtain an optimal channel selection policy denoted by π to maximize the throughput of the secondary user. The policy is a mapping from a state to an action to be taken by the secondary user. In other words, given the data queue and energy states, the policy will determine which channel to select. The optimization problem is expressed as follows: max π J T (π) = lim t inf 1 t t E (T (θ t,δ t )) (8) t =1 where J T (π) is the throughput of the secondary user and T (θ t,δ t ) is an immediate throughput function given state θ t Θ incom and action δ t Δ at time t. Let the state variable be defined as θ =(e, q) where e and q are the energy level of the energy queue and the number of packets in the data queue, respectively. The immediate throughput function is defined as follows: ηδ f T (θ, δ) = δ σ δ, (e W ) and (q >0) (9) 0 otherwise. The secondary user successfully transmits a packet if there is enough energy, the queue is not empty, the selected channel is idle, and there is no false alarm. Then, we obtain the optimal policy of the MDP by formulating and solving an equivalent linear programming (LP) problem. The LP problem is expressed as follows: max φ(θ,δ) s.t. θ Θ incom δ Δ φ(θ,δ)= δ Δ φ(θ, δ)t (θ, δ) (10) θ Θ incom δ Δ φ(θ, δ)p θ,θ (δ), θ Θ incom φ(θ, δ) =1, φ(θ, δ) 0 θ Θ incom δ Δ where P θ,θ (δ) denotes the element of matrix P C (δ) as defined in (3) where θ =(e, q) and θ =(e,q ). Let the solution of the LP problem be denoted by φ (θ, δ). The randomized policy of the secondary user can be obtained as follows: π (θ, δ) = φ (θ, δ) δ Δ φ (θ, δ ), for θ Θ incom. (11) IV. OPTIMIZATION FORMULATION: COMPLETE INFORMATION In this section, we consider the complete information case. Here, the secondary user has a complete knowledge about the states of all N channels. The derivations are similar to those in the incomplete information case. Therefore, we explain only the difference for brevity of the paper. A. State Space and Action Space We define the state space of the secondary user as follows: } Θ com = (C, E, Q); E 0, 1,...,E}, Q 0, 1,...,Q} (12) where C is the composite channel state defined as follows: } C = (C 1,...,C N ); C n 0, 1} (13) where C n is the state of channel n. The action space of the secondary user is again Δ. 1558

5 B. Transition Probability Matrix We denote the transition probability matrix given action δ Δ of the secondary user by P C (δ) whose element is obtained from C (c1,...,c N ),(c 1,...,c N ) Q cδ. C (c1,...,c N ),(c 1,...,c N ) is the probability that the states of all channels are (c 1,...,c N ) in the current time slot and change to (c 1,...,c N ) in the next time slot. This probability is the element of the matrix C = C 1 C n C N, where is a Kronecker product and C n is defined as in (1). Q cδ is the transition matrix of data queue and energy queue states when the state of the selected channel δ is c δ 0, 1} for idle and busy, respectively. Matrix Q cδ has a similar structure to that of (3), whose element is B q,q (δ). The selected channel is idle c δ = 0: The element of matrix B q,q (δ) is obtained given that η δ =1. The selected channel is busy c δ = 1: The element of matrix B q,q (δ) is obtained given that η δ =0. For brevity of the paper, we omit the detailed derivation of the transition matrix which is straightforward from the incomplete information case. C. Optimal Policy The MDP can be formulated similar to that in (8). In this case, the state is defined as θ t Θ com, where θ = ((c 1,...,c n,...,c N ),e,q) where c n is the state of channel n, e and q are the energy level of the energy queue and the number of packets in the data queue, respectively. The immediate throughput function in the complete information case is defined as follows: f T (θ, δ) = δ σ δ, (e W ) and (q >0) and (c δ =0) 0 otherwise. (14) The similar method of transforming and solving the equivalent LP problem is applied to obtain the optimal channel section policy for the complete information case. V. PERFORMANCE EVALUATION A. Parameter Setting We consider a secondary user whose data queue and energy queue sizes are 10 packets and 10 units of energy, respectively. The secondary user requires 1 unit of energy for packet transmission. The packet arrival probability is 0.5. There are two channels licensed to primary users 1 and 2. Unless otherwise stated, the probabilities that the channels 1 and 2 will be idle are 0.1 and 0.9, respectively. The probability of successful packet transmission on both channels is 0.95. The probabilities of successful RF energy harvesting with one unit of energy on channels 1 and 2 are 0.95 and 0.70, respectively. We consider three channel selection policies, i.e., complete information, incomplete information, and random policy. In the random policy, the secondary user selects channels 1 and 2 randomly with the probability of 0.5. Throughput (packets/time slot) 0.38 0.36 0.34 0.32 0.3 0.28 Complete information 0.26 Incomplete information Random policy 0.24 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 Packet arrival probability Fig. 2. Throughput under different packet arrival probability. B. Numerical Results Fig. 2 shows the throughput of the secondary user when the packet arrival probability is varied. Clearly, when the packet arrival probability increases, the throughput increases. However, after a certain point, the throughput becomes saturated due to the limited energy harvested from the channels. When the packet arrival probability is small, all three policies yield the same throughput since there is always enough energy to transmit packets. However, when the packet arrival probability is large, the secondary user in the complete information case can fully exploit the channel state information and achieve the largest throughput. However, if the complete channel state information is not available, the secondary user is still able to optimize its channel selection, which yields the higher throughput than that of the random policy. Here we can quantify the performance improvement from having complete channel state information. Throughput (packets/time slot) 0.4 0.35 0.3 0.25 0.2 Complete information Incomplete information Random policy 0.15 0.4 0.5 0.6 0.7 0.8 0.9 Probability of successful RF energy harvesting from channel 1 Fig. 3. Throughput under different harvesting probability for channel 1. Fig. 3 shows the throughput under different probability of successful RF energy harvesting from channel 1. When the secondary user has higher chance to successfully harvest RF energy from one of the channels, the performance improves. Again, the secondary user with complete channel state information achieves the highest throughput, and the channel selection policy with incomplete information still yields the throughput higher than that of the random policy. We investigate the case when the idle probability of channel 1 is varied (Fig. 4). As the idle probability of channel 1 in- 1559

6 Throughput (packets/time slot) 0.5 0.45 0.4 0.35 0.3 0.25 Complete information 0.2 Incomplete information Random policy 0.15 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Probability of channel 1 to be idle Fig. 4. Throughput under different idle probability for channel 1. creases (i.e., becomes less busy), the throughput first increases, since the secondary user has more chance to transmit its packets. However, at a certain point, the throughput decreases. This decrease is due to the fact that channel 1 is mostly idle and the secondary user cannot harvest much RF energy. Therefore, there is not enough energy in the energy queue to transmit packets, thus the throughput decreases. VI. SUMMARY We have considered a cognitive radio network with a secondary user and multiple primary users. The secondary user can select a channel to transmit a packet when the channel is not occupied by the primary user. Alternatively, if the channel is occupied, the secondary user can harvest RF energy, which can be stored and used for packet transmission. We have formulated and solved optimization problems based of Markov decision process to obtain the channel selection policy, such that the throughput of the secondary user is maximized. We have considered two cases of optimization, i.e., complete and incomplete channel state information. The performance gap between these two cases can quantify the benefit of having complete information, i.e., value of (channel state) information. [4] K. Li, H. Luan, and C.-C. Shen, Qi-Ferry: Energy-constrained wireless charging in wireless sensor networks, in Proceedings of IEEE Wireless Communications and Networking Conference (WCNC), pp. 2515-2520, April 2012. [5] A. H. Coarasa, P. Nintanavongsa, S. Sanyal, and K. R. Chowdhury, Impact of mobile transmitter sources on radio frequency wireless energy harvesting, in Proceedings of International Conference on Computing, Networking and Communications (ICNC), pp.573-577, Jan. 2013. [6] J. Kim and J.-W. Lee, Energy adaptive MAC protocol for wireless sensor networks with RF energy transfer, in Proceedings of International Conference on Ubiquitous and Future Networks (ICUFN), pp. 89-94, June 2011. [7] D. W. K. Ng. S. L. Ernest, and R. Schober, Energy-efficient resource allocation in multiuser OFDM systems with wireless information and power transfer, in Proceedings of IEEE Wireless Communications and Networking Conference (WCNC), April 2013. [8] B. Gurakan, O. Ozel, J. Yang, and S. Ulukus, Two-way and multipleaccess energy harvesting systems with energy cooperation, in Conference Record of Asilomar Conference on Signals, Systems and Computers (ASILOMAR), pp.58-62, Nov. 2012. [9] I. Krikidis, S. Timotheou, and S. Sasaki, RF energy transfer for cooperative networks: Data relaying or energy harvesting?, IEEE Communications Letters, vol. 16, no. 11, pp. 1772-1775, Nov. 2012. [10] M. K. Watfa, H. Al-Hassanieh, and S. Selman, Multi-hop wireless energy transfer in WSNs, IEEE Communications Letters, vol. 15, no. 12, pp. 1275-1277, December 2011. [11] S. H. Lee, R. Zhang, and K. B. Huang, Opportunistic wireless energy harvesting in cognitive radio networks, IEEE Transactions on Wireless Communications, accepted. ACKNOWLEDGEMENTS This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (NRF-2013R1A2A2A01067195). REFERENCES [1] H. Ostaffe, Power Out of Thin Air: Ambient RF Energy Harvesting for Wireless Sensors, 2010 [Online] http://powercastco.com/pdf/power- Out-of-Thin-Air.pdf. [2] C. Mikeka, H. Arai, A. Georgiadis, and A. Collado, DTV band micropower RF energy-harvesting circuit architecture and performance analysis, in IEEE International Conference on Proceedings of RFID- Technologies and Applications (RFID-TA), pp. 561-567, Sept. 2011. [3] H. J. Visser and R. J. M. Vullers, RF Energy Harvesting and Transport for Wireless Sensor Network Applications: Principles and Requirements, Proceedings of the IEEE, vol. 101, no. 6, pp. 1410-1423, June 2013. 1560