ON ADAPTIVE TRANSMISSION, SIGNAL DETECTION AND CHANNEL ESTIMATION FOR MULTIPLE ANTENNA SYSTEMS. A Dissertation YONGZHE XIE

ON ADAPTIVE TRANSMISSION, SIGNAL DETECTION AND CHANNEL ESTIMATION FOR MULTIPLE ANTENNA SYSTEMS A Dissertation by YONGZHE XIE Submitted to the Office of Graduate Studies of Texas A&M University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY August 2004 Major Subject: Electrical Engineering

ON ADAPTIVE TRANSMISSION, SIGNAL DETECTION AND CHANNEL ESTIMATION FOR MULTIPLE ANTENNA SYSTEMS A Dissertation by YONGZHE XIE Submitted to Texas A&M University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Approved as to style and content by: Costas N. Georghiades (Chair of Committee) Krishna R. Narayanan (Member) Garng M. Huang (Member) Graham D. Allen (Member) Chanan Singh (Head of Department) August 2004 Major Subject: Electrical Engineering

iii ABSTRACT On Adaptive Transmission, Signal Detection and Channel Estimation for Multiple Antenna Systems. (August 2004) Yongzhe Xie, B. S., Shanghai Jiaotong University; M. Eng, The National University of Singapore Chair of Advisory Committee: Dr. Costas N. Georghiades This research concerns analysis of system capacity, development of adaptive transmission schemes with known channel state information at the transmitter (CSIT) and design of new signal detection and channel estimation schemes with low complexity in some multiple antenna systems. We first analyze the sum-rate capacity of the downlink of a cellular system with multiple transmit antennas and multiple receive antennas assuming perfect CSIT. We evaluate the ergodic sum-rate capacity and show how the sum-rate capacity increases as the number of users and the number of receive antennas increases. We develop upper and lower bounds on the sum-rate capacity and study various adaptive MIMO schemes to achieve, or approach, the sum-rate capacity. Next, we study the minimum outage probability transmission schemes in a multiple-input-single-output (MISO) flat fading channel assuming partial CSIT. Considering two special cases: the mean feedback and the covariance feedback, we derive the optimum spatial transmission directions and show that the associated optimum power allocation scheme, which minimizes the outage probability, is closely related to the target rate and the accuracy of the CSIT. Since CSIT is obtained at the cost of feedback bandwidth, we also consider optimal allocation of bandwidth between the data channel and the feedback channel in order to maximize the average throughput of the data channel in MISO, flat fading, frequency division duplex (FDD) systems.

iv We show that beamforming based on feedback CSI can achieve an average rate larger than the capacity without CSIT under a wide range of mobility conditions. We next study a SAGE-aided List-BLAST detection scheme for MIMO systems which can achieve performance close to that of the maximum-likelihood detector with low complexity. Finally, we apply the EM and SAGE algorithms in channel estimation for OFDM systems with multiple transmit antennas and compare them with a recently proposed least-squares based estimation algorithm. The EM and SAGE algorithms partition the problem of estimating a multi-input channel into independent channel estimation for each transmit-receive antenna pair, therefore avoiding the matrix inversion encountered in the joint least-squares estimation.

To Mom, Dad and Lan v

vi ACKNOWLEDGMENTS I am deeply indebted to my advisor, Prof. Costas N. Georghiades, who has provided technical guidance, enlightening ideas and insights throughout my Ph.D program. He also provided generous financial support during the last four year. His encouragement, trust and understanding helped me endure many hard times. I am very thankful to Dr. Kamyar Rohani of Motorola Labs, who unfortunately passed away in the summer of 2003. Kamyar initiated and supervised our Motorola UPR project on adaptive transmission for multiple antenna systems, which forms most of this dissertation. He provided many useful technical references and insightful suggestions. His dedication to work will always inspire me. I am also grateful to Eric Schorman who succeeded Kamyar in supervising the UPR project. I would like to thank Motorola for funding me during the last three years of my Ph.D. studies. I am very thankful to Prof. Krisha R. Narayanan, Prof. Xiaodong Wang and Prof. Scott Miller for all the excellent courses I took from them. Prof. Don Allen s course on Linear Algebra and Matrix Theory also helped me in my study. I would like to thank Prof. A. Araposthathis for help in proving an important Lemma in Chapter IV. I would like to thank Mr. Qiang Li for very helpful discussions regarding Chapter VI. Dr. Wei Yu also helped me much by answering some of my questions regarding his papers on MIMO Broadcast channels. Discussions with many of my other colleagues, Jing (Tiffany) Li, Wurong Yu, Angelos D. Liveris, Beminihennadige Janath Peiris, Jun Zheng, Wenyan He, Samul Chen, Doan Ngoc Dung, Vivek Gulati, Zigang Yang, Ben Lu, Yan Liu, Yan Wang, Yu (Frank) Zhang, have also been helpful. I am indebted to my wife, Lan Qiang, for all her support and understanding during my Ph.D study. I dedicate this work to Lan, my parents, Ruihe Xie and

vii Qiuhua Gao, as well as my parents-in-law, Wenyi Qiang and Enhui Dong. Without their support, this dissertation would have been impossible. Finally, I would like to thank Bailan, my lovely daughter, who brought so much joy to my life during the last year.

viii TABLE OF CONTENTS CHAPTER Page I INTRODUCTION.......................... 1 A. Dissertation Outline..................... 1 B. A Note on Notation...................... 4 II A BRIEF OVERVIEW OF MULTIPLE ANTENNA SYSTEMS 6 A. Channel Models and Capacity Analysis........... 6 1. Single User MIMO, MISO, SIMO Channels...... 6 a. Ergodic Capacity.................. 7 b. Outage Capacity.................. 8 2. Sum-Capacity of Multiple User Systems........ 8 a. Gaussian MIMO MAC............... 10 b. Gaussian MIMO BC................ 10 B. Transmission and Detection Schemes for Multiple Antenna Systems......................... 11 1. Space-time Coding.................... 11 2. ML Detection and Sphere Decoding.......... 12 3. ZF and MMSE Detector................. 13 4. BLAST.......................... 13 5. Beamforming for MISO Channels............ 14 6. Transmitter Side Pre-filtering: Zero-Forcing Beamforming and Ranked Known Interference........ 14 III IV ON THE SUM-RATE CAPACITY OF MIMO FADING BROAD- CAST CHANNELS......................... 17 A. Introduction.......................... 17 B. Sum-Rate Capacity of Fading MIMO-BC.......... 20 C. Preprocessing to Achieve Multi-user Diversity....... 26 1. Multiuser Diversity in a Fading MIMO-BC...... 27 2. A Lower Bound on the Sum-Rate Capacity...... 28 D. Simulation Results...................... 29 E. Conclusion........................... 34 MINIMUM OUTAGE PROBABILITY TRANSMISSION WITH IMPERFECT FEEDBACK FOR MISO FADING CHANNELS 37

ix CHAPTER Page A. Introduction.......................... 37 B. Mean Feedback........................ 39 C. Covariance Feedback..................... 43 D. Numerical Results....................... 47 E. Conclusion........................... 52 V VI VII OPTIMAL BANDWIDTH ALLOCATION FOR THE DATA AND FEEDBACK CHANNELS IN MISO-FDD SYSTEMS... 53 A. Introduction.......................... 53 B. The Channel Model...................... 54 1. Data Channel...................... 55 2. Channel Prediction and CSI Feedback......... 56 C. Optimal Bandwidth Allocation............... 58 1. Noisy Side Information................. 59 2. Quantized Side Information............... 61 D. Simulation Results...................... 63 E. Conclusion........................... 71 SAGE-AIDED DETECTION OF MULTIPLE TRANSMIT ANTENNA SYSTEMS....................... 72 A. Introduction.......................... 72 B. EM and SAGE for Detecting Superimposed Signals.... 73 1. Initialization....................... 76 2. Implementation..................... 78 3. Complexity........................ 78 4. Soft-output Detection.................. 80 C. Simulation Results...................... 82 D. Conclusion........................... 90 CHANNEL ESTIMATION FOR OFDM SYSTEMS USING EM ALGORITHMS........................ 91 A. Introduction.......................... 91 B. ST-OFDM Systems and Channel Model.......... 93 1. ST-OFDM Systems................... 93 2. The Channel Model................... 94 C. EM-type Channel Estimation Algorithms.......... 95 1. Method of Least Squares................ 95 2. The EM-Based Algorithm................ 96

x CHAPTER Page D. Remarks............................ 101 1. Convergence....................... 101 2. A Message Passing Interpretation.......... 103 3. Implementation Complexity for STC-EM and STC- SAGE........................... 105 E. Simulation Results...................... 106 F. Conclusion........................... 116 VIII CONCLUSION........................... 119 REFERENCES................................... 121 APPENDIX A................................... 131 APPENDIX B................................... 133 APPENDIX C................................... 136 APPENDIX D................................... 139 VITA........................................ 141

xi LIST OF FIGURES FIGURE Page 1 Upper-bounds on sum-rate capacity for N t = 2. ( Bxtyr, ABxtyr and Cxtyr denote capacity bound (equation (3.13)), asymptotic bound (equation (3.20)), and sum-rate capacity with N t = x and N r = y, respectively; sl and dl denote solid line and dotted line, respectively.............................. 31 2 Achievable sum-rate per transmit antenna (N t = 2).......... 32 3 Achievable sum-rate per transmit antenna (N t = 4).......... 34 4 Achievable sum-rate per transmit antenna for different user selection criteria (N t = 2)........................... 35 5 Achievable minimum outage probability vs target rate of meanfeedback for N t = 2. Channel received SNR, SNR rv = P (ξ+σ h 2), is σn 2 fixed at 8dB. SNR fb = ξ......................... 48 σh 2 6 Effect of SNR fb on achievable minimum outage probability of mean-feedback for N t = 2. Channel received SNR, SNR rv = P (ξ+σh 2), is fixed at 8dB. SNR σn 2 fb = ξ.................. 49 σh 2 7 Optimal power allocation over different transmission directions of mean-feedback for N t = 2. Channel received SNR, SNR rv = P (ξ+σh 2), is fixed at 8dB. SNR σn 2 fb = ξ.................. 50 σh 2 8 Minimum outage probability achievable by optimal and equal power allocation over the directions indicated by the two eigenvectors of covariance matrix Σ. Note that the two eigenvalues, λ 1 and λ 2 satisfy λ 1 + λ 2 = 2. Channel received SNR, SNR rv = P, σn 2 is fixed at 8dB............................... 51 9 The random quantization lower bound to the achievable rate for the beamforming scheme vs B N t for different N t. Channel SNR, defined as P σ2 h, is fixed at 10 db..................... 64 σn 2

xii FIGURE Page 10 Achievable rates of practical quantization schemes for N t = 4 and SNR=10dB................................. 66 11 Maximum achievable rate Cd, normalized by the capacity with perfect CSIT obtained using the random quantization lower-bound for different η. N t = 4........................... 67 12 Optimal fraction of the total bandwidth allocated to the feedback channel for different η in order to achieve C d. N t = 4.......... 68 13 Comparison of achievable rates assuming two different partial CSIT models: N t = 32, η = 0.005 and SNR = 10 db.......... 69 14 Symbol Error Rate of different detectors for a 4 4 MIMO system with uncoded 8-PSK modulation..................... 82 15 Bit Error Rate of different detectors for a 4 4 MIMO system with uncoded 8-PSK modulation..................... 83 16 Symbol Error Rate of different detectors for a 4 4 MIMO system with uncoded 16-QAM modulation.................... 84 17 Bit Error Rate of different detectors for a 4 4 MIMO system with uncoded 16-QAM modulation.................... 85 18 Bit Error Rate of different detectors for a 8 8 MIMO system with uncoded 16-QAM modulation.................... 87 19 Bit Error Rate of turbo-coded 4 4 MIMO systems with SAGEaided List-Shifted-BLAST decoding and the simple soft-output BLAST decoding............................. 88 20 ST-OFDM system............................. 93 21 A Message Passing explanation of the EM-type algorithms...... 104 22 (a) The uniform power and delay profile; (b) the hilly terrain profile. 108

xiii FIGURE Page 23 The convergence of MSE with respect to number of iterations of the EM-type estimators compared with the MSE of a 9-tap-STC estimator and the ML estimator. An initial channel estimate is obtained using (7.25). The HT profile as shown in (b) of Fig. 22 is assumed................................. 109 24 The convergence of MSE with respect to number of iterations of the EM-type estimators compared with the MSE of a 9-tap-STC estimator and the ML estimator. The EM algorithm is initialized using the last channel estimate. The HT profile as shown in (b) of Fig. 22 is assumed........................... 110 25 The convergence of MSE with respect to number of iterations of the EM-type estimators compared with the MSE of a 9-tap-STC, 13-tap-STC estimator and the ML estimator. An initial channel estimate is obtained using (7.25). The uniform profile as shown in (a) of Fig. 22 is assumed........................ 111 26 CDF of convergence factor of EM-type estimators with two transmit antennas used............................. 113 27 CDF of convergence factor of EM-type estimators with four transmit antennas used............................. 114 28 BER and WER of ST-OFDM system for channels with the HT profile as shown in (b) of Fig. 22.................... 115 29 BER and WER of ST-OFDM system for channels with the uniform profile as shown in (a) of Fig. 22................. 117

1 CHAPTER I INTRODUCTION The use of multiple transmit/receive antennas has emerged as a promising solution for high data rate communication over wireless channels. The resulting multiple antenna system can provide crucial spatial diversity and additional degrees of freedom which, if appropriately exploited, can yield significant capacity gains [1, 2]. This work sets two goals in the research of multiple antenna wireless systems. One is to analyze the capacity of some multiple antenna systems and develop adaptive transmission schemes with emphasis on exploring channel side information at the transmitter. The other is to introduce new signal detection and channel estimation schemes with low complexity. More specifically, the dissertation has studied the ergodic sum-rate capacity of a multiple input and multiple output (MIMO) broadcast system and some candidate adaptive transmission schemes assuming perfect channel state information at the transmitter side (CSIT), minimum outage probability transmission in a multiple input and single output (MISO) fading channel with partial CSIT, optimal bandwidth allocation in a FDD system, and efficient EM-type signal detection and channel estimation algorithms for multiple antenna fading channels with application to cellular systems. A. Dissertation Outline Chapter II introduces some background knowledge on capacity analysis, and transmission and detection schemes for multiple antenna systems. Chapter III analyzes the sum-rate capacity of the downlink of a cellular sys- This dissertation follows the style of IEEE Transactions on Automatic Control.

2 tem with multiple transmit antennas and multiple receive antennas assuming perfect CSIT. Modelling the downlink as a flat fading multiple-input-multiple-output (MIMO) broadcast channel (MIMO-BC), we evaluate the ergodic sum-rate capacity using the duality between a MIMO multiple access channel (MIMO-MAC) and a MIMO-BC. We show how the sum-rate capacity increases as the number of users and the number of receive antennas increase. We also develop upper and lower bounds on the sum-rate capacity and study various adaptive MIMO schemes to achieve or approach the sum-rate capacity. Sub-optimal transmission schemes, such as ranked known interference cancellation based on channel matrix triangulation and zero-forcing beamforming based on channel matrix inversion are shown to be able to achieve close to capacity performance. In Chapter IV, we consider transmission schemes assuming partial CSIT, since perfect channel state information can be too optimistic in practice. We derive the minimum outage probability transmission schemes in a multiple-input-single-output (MISO) flat fading channel for two special cases: the mean feedback case where the CSIT and the actual channel state are jointly Gaussian, and the covariance feedback case where only the spatial covariance matrix of the channel states is known at the transmitter. In the case of mean feedback, the optimal transmission strategy is proven to be transmitting several independent data streams in the direction of the channel mean vector and its orthogonal directions. In contrast to the case of maximizing the ergodic capacity, the optimum power allocation scheme which minimizes outage probability is closely related to the target rate. For both mean and covariance feedback, we show that it is more desirable to spread the power over all transmission directions than beamforming to a single direction for sufficiently small target rates. In Chapter V, we study the joint optimization of the forward data channel and the feedback channel in terms of bandwidth allocation in order to maximize the average

3 throughput of the data channel in a MISO frequency division duplex (FDD) system. In FDD systems, CSI is usually estimated by the receiver and then fed back to the transmitter through a reliable link, which inevitably requires additional bandwidth and power. If one views bandwidth and power as common resources that can be shared by the data and feedback channels, the question is whether the increased capacity is worth the penalty paid for it. We consider two models of the partial CSIT: the noisy CSIT which is jointly Gaussian distributed with the actual channel state, and the quantized CSIT. In the first model, we use distortion rate theory to relate the CSIT accuracy to the feedback bandwidth. In the second model, we derive a lower bound on the achievable rate of the data channel based on the ensemble of a set of random quantization codebooks. We show that in the MISO flat fading channel case, beamforming based on feedback CSI can achieve an average rate larger than the capacity without CSIT, under a wide range of mobility conditions. Chapter VI proposes a Space Alternating Generalized Expectation-Maximization (SAGE) aided List-BLAST detection scheme, which can achieve performance close to that of the maximum-likelihood detector with low computational complexity. The SAGE algorithm searches for the ML solution iteratively by resolving the interference among signals from different transmit antennas. To improve the probability of convergence to the ML solution, multiple initial points are used. The List-BLAST algorithm, which exhausts the constellation points in the first layers of a BLAST detection scheme, is shown to be an excellent way to generate initial points. The complexity of the proposed detection scheme is compared with that of the sphere detection scheme, and it is shown to have a number of implementation advantages. In Chapter VII, we study channel estimation for an orthogonal frequency division multiplexing (OFDM) system with multiple transmit antennas in a frequency selective fading channel. We propose the EM and the SAGE iterative channel esti-

4 mation algorithms and compare them with a recently proposed least-squares based estimation algorithm. We study the convergence properties of the proposed schemes, the overall system performance and implementation issues through both theoretical analysis and simulation. At each iteration and for every OFDM link, the EM-type algorithms partition the problem of estimating a multi-input channel into independent channel estimation for each transmit-receive antenna pair, therefore avoiding the matrix inversion encountered in the joint least-squares estimation. We also show that the convergence rate for both algorithms is unrelated to the channel delay profile and decreases when the length of the channel or the number of transmit antennas increases. Finally, we conclude the dissertation with a summary on the major contributions in Chapter VIII. B. A Note on Notation Throughout the dissertation, if not otherwise specified in each chapter, we use the following general rules in notation. We use boldface and lower case letters to denote vectors and boldface and uppercase letters to denote matrices. Superscripts T, and H denote transpose, conjugate and transpose conjugate of a matrix or a vector, respectively; A 1, tr(a) and A denote the inverse, trace and determinant of matrix A, respectively; I n denotes the identity matrix of dimension n; when there is no ambiguity on the dimension, I is used to denote the identity matrix; A[i, j] denotes the [i, j] th entry of matrix A; a i denotes the i th entry of vector a. E( ) is the expectation operator; ā will also be used to denote the mean of a. Symbol = is used for definition. Both the scalar or the vector Gaussian distribution

5 is denoted as N (α, Σ) with α denoting the scalar or vector mean and Σ denoting the variance or the covariance matrix. f(a b) or p(a b) is used to denote the conditional PDF of the random variable a given b. E(a b) is used to denote conditional mean. Since we do not need to distinguish between a random variable and its value by using different notations in this dissertation, the variable on the right side of the conditional symbol always denotes the actual value of the corresponding random variable if not otherwise specified. Given a sequence a 1, a 2,, a n of positive numbers, we say that a positive number b n is of the order of O(a n ) as n if an b n is bounded by some constant.

6 CHAPTER II A BRIEF OVERVIEW OF MULTIPLE ANTENNA SYSTEMS Multiple antenna systems were first used at the receiver side to provide multiple independent spatial copies of the received signal to combat fading in wireless communication systems. The recent interest is mainly in the use of multiple transmit antennas because some important applications limit the use of multiple receiver antennas. For example, it is hard to implement two independent antennas on a small mobile device. If multiple antennas are used on both the transmitter and receiver side in a rich scattering wireless channel, the capacity of such a system with channel known at the receive side can increase linearly as the minimum of the number of transmit and receive antennas increases [1][2]. This discovery has triggered enormous research interests in multiple antenna systems in recent years. In this chapter, we will briefly introduce some capacity results, well known transmission and detection schemes of multiple antenna systems, which are closely related to the rest of the chapters. A. Channel Models and Capacity Analysis 1. Single User MIMO, MISO, SIMO Channels Consider the point-to-point communication over a rich-scattering frequency nonselective wireless channel with N t transmit antennas and N r receive antennas. The system in each channel use can be modelled as follows: y = Hx + w, (2.1) where H is a N r N t matrix denoting the channel, with each element of the matrix modelled as i.i.d. zero mean, circularly symmetric, complex Gaussian with normalized

7 variance. If N r = 1, the channel is usually referred to as multiple input single output (MISO) channel. Similarly, we can define a single input and multiple out channel (SIMO) when N t = 1, and a multiple input and multiple out (MIMO) when N t 1 and N r 1. w N (0, σni) 2 is a N r 1 vector denoting the circularly symmetric complex Gaussian noise corrupting the different receivers. y is the received signal vector of dimension N r 1. x denotes the N t 1 complex transmit signal (column) vector. Let S = E[xx H ]. The transmitter is constrained in total power as: tr(s) = P. (2.2) a. Ergodic Capacity The Ergodic capacity is the maximum average achievable rate of a channel with zero error probability. The ergodic capacity of multiple antenna systems with two different assumptions is summarized below: CSI perfectly known only at the receiver. In this case, the average mutual information I(x; y H) between the input and output given H is maximized when x is complex Gaussian distributed and can be computed as I(x; y H) = log I Nr + HSH H (2.3) The ergodic capacity is maximized when S = P N t I Nt [1]. CSI perfectly known at both the transmitter and receiver. Let the singular value decomposition (SVD) of H be H = UDV H, where U and V are unitary matrices and D is a diagonal matrix. Since U and V are available at both the transmitter and receiver, the channel as shown in equation (2.1) can be diagonalized by pre-filtering (multiply) x by matrix V, and post-filtering (multiply) the received vector y by U H. Since orthogonal transformation does not change

8 the distribution of x and W, the MIMO channel is transformed into a parallel set of N = min{n t, N r } Gaussian scalar channels, whose capacity can be achieved by the water-filling power allocation scheme over space and time [3]. b. Outage Capacity In some scenarios of wireless communications, due to delay limits, channel cannot be assumed to be ergodic during the transmission of a code word. For example, the well known quasi-static fading channel model assumes channel remains constant within a transmission block, but changes independently from block to block. In this case, the mutual information expression can be treated as random entities, giving rise to capacity-versus-outage considerations [4]. The channel outage probability is simply defined as ɛ o = Prob(I(x; y) < R t ), (2.4) where R t is the target rate. We can also define the outage capacity C ɛ as the maximum achievable rate at the given target outage probability. For example, C 1% = 3 (bit) means that three bits per channel-use can be achieved with a probability of 99%. 2. Sum-Capacity of Multiple User Systems We consider here two kinds of multiple user systems: the multiple access channel, where multiple transmitters (or users) communicate to a single receiver and the broadcast channel where a single transmitter communicates to multiple receivers (or users). In a cellular system, the multiple access channel corresponds to the uplink (from mobile to base) and the broadcast channel corresponds to the downlink.

9 A K user MIMO-MAC channel can be modelled as y = K H k x k + w, (2.5) k=1 where H k denotes the matrix channel between the k th transmitter and the receiver. The transmitted signal vector x k of the k th user usually has an individual power constraint as tr(s k ) P k (2.6) where P k is the available transmission power of transmitter k. The MIMO-BC channel can be modelled as y k = H k x + w k, 1 k K, (2.7) where y k is the received signal of the k th user; H k denotes the matrix channel between the transmitter and the k th receiver; w k denotes the white Gaussian noise at the k th receiver. The transmit power constraint can be expressed as K tr(s k ) P, (2.8) k=1 where P is the available total transmission power. In a multiple user system, one usually defines the capacity region to be the closure of the set of achievable rate vectors (R 1, R 2,... R K ), where R k denotes the rate of the k th user [5]. Besides the capacity region, sum-rate capacity, defined as the maximum achievable sum-rate, K k=1 R k, is often used to measure the total throughput of a multiple user system.

10 a. Gaussian MIMO MAC Let X c k denote the set of all user s transmit vectors except x k. The capacity region of a MAC channel is the closure of the convex hull of all rate vectors (R 1, R 2,... R K ) satisfying R k I(x k ; y X c k), for any k, (2.9) K R k I(x 1, x 2,..., x K ; y) (2.10) k=1 for some input distribution satisfying the power constraints. For the case of a MIMO- MAC channel, the capacity region is shown to be [6] R = B(S 1, S 2,..., S K ), (2.11) tr(s k ) P k,s k 0 where S k 0 means that S k should be positive semi-definite. B(S 1, S 2,..., S K ) is defined as the set of (R 1, R 2,... R K ) achieved by a given choice of power allocation scheme (S 1, S 2,..., S K ), which can be expressed as R k log 2 ( H k S k H H k + I ); for any k, (2.12) ( K K ) R k log 2 H k S k H H k + I. (2.13) k=1 k=1 The last equation also shows the sum-rate for the given power allocation scheme, which can be maximized over all possible choices of power allocation schemes to obtain the sum-rate capacity. b. Gaussian MIMO BC Compared to the multiple access channel, the broadcast channel is less understood. Only the capacity region of a small class of broadcast channels, called degraded broadcast channels, is known [5]. The most simple type of degraded broadcast channel

11 is formed by two-user scalar AWGN channels, where one receiver (corresponding to the good user) experiences a Gaussian noise with less variance than that of the other user s receiver (the bad user). The border of the capacity region in this case can be achieved by cancellation at the receivers; the bad user always treats the encoded information for the good user as Gaussian interference; the good user can always decode the bad user s information first and then cancel its effect and decode its own information. However, a Gaussian MIMO-BC channel is usually not degraded; thus, its capacity region is unknown. The capacity region and the sum-rate capacity of MIMO-BC and MIMO-MAC have been shown to be closely related. A more detailed treatment of this topic will be given in Chapter III, where the ergodic sum-rate capacity of a fading MIMO-BC is derived and analyzed based on this relation. B. Transmission and Detection Schemes for Multiple Antenna Systems 1. Space-time Coding The concept of space-time coding was first proposed by Tarokh et al. to improve data rate and reliability of communications over fading channels using multiple transmit antennas [7]. By carefully designing the codewords, potential spatial diversity provided by multiple transmit antennas can be achieved. For example, in a slow and frequency non-selective Rayleigh fading channel, performance is shown to be determined by matrices constructed from pairs of distinct code sequences. The minimum rank among these matrices quantifies the diversity gain, while the minimum determinant of these matrices quantifies the coding gain. Based on these criteria, space-time trellis codes have been designed to achieve 2-3 db away from the outage capacity. The decoding of space-time trellis codes requires a maximum-likelihood (ML) sequence de-

12 tection scheme, whose complexity increases exponentially as the the number of trellis states increases. Another well known type of space-time codes is the orthogonal space-time block codes which have a very simple ML decoding scheme such as the Alamouti s scheme for a system with two transmit antennas [8][9]. Although the orthogonality can simplify detection, it usually results in capacity loss except in some special cases [10]. Some recently proposed block codes can achieve close to capacity rate while maintaining a relatively simple decoding structure [10]. Space time coding techniques usually assume no CSIT. Reliable communication is achieved by careful design of the structure of the code sequences. In this thesis, we are mainly focused on techniques utilizing either full or partial CSIT. 2. ML Detection and Sphere Decoding Assume that transmit signal x in the channel model of equation (2.1) is composed of uncoded QAM or QPSK signals. We assume perfectly known H at the receiver side. The maximum-likelihood (ML) detector can be expressed as ˆx = arg min y Hx 2 (2.14) x Ω N t where Ω Nt denotes the set of constellation points in the complex N t -dimensional space. Since exhaustive search for the ML solution over the whole set of Ω Nt is too complex to be implementable, sphere decoding can be used to reduce complexity. Equation (2.14) can be shown to be equivalent to the following: ˆx = arg min (x x ls ) H R H R(x x ls ), (2.15) x Ω N t where x ls = (H H H) 1 H H y is the least-square or zero-forcing estimate of x assuming x is continuous; R is the upper triangular matrix in the QR decomposition of H = QR.

13 To solve (2.15), the sphere decoder avoids the exhaustive search by considering only those points satisfying (x x ls ) H R H R(x x ls ) r 2. This search can be implemented efficiently by exploiting the triangular structure of R as shown in [11, 12]. 3. ZF and MMSE Detector Assume Nr N t. Both the zero-forcing (ZF) and the minimum mean square error (MMSE) detectors perform linear transformation over the received signal y as y = B H y = B H Hx + B H w (2.16) ZF uses B = H(H H H) 1 ; MMSE uses B = H(H H H + detection is then performed on y I SNR ) 1. Symbol-by-symbol to detect each element of x. Note that since noise becomes correlated after the transformation, symbol-by-symbol detection, although very simple, is not optimal. 4. BLAST Different from linear detectors such as ZF and MMSE, the Bell Lab Layered Space- Time (BLAST) scheme [13] is based on nulling and cancelling as introduced below. Denoting the QR decomposition of H = QR, we can perform a linear transformation on the received signal as y = Q H y; the system can be expressed as y = Rx + w, (2.17) where w = Q H w has the same distribution as w since Q is unitary. In the triangulized model above, each row denotes a different encoding/decoding layer with the k th layer interfered only by layers with indexes larger than k. Considering the N th t row (layer) of (2.17), which denotes an underlying scalar channel, one can first detect x Nt ; assuming ˆx Nt is correct, the interference of R[N t 1, N t ]ˆx Nt can be subtracted

14 from layer N t 1 and ˆx Nt 1 can be detected as in a scalar channel. Similarly, layer N t 2, N t 3,, 1 can be detected in order. In practice, nulling and cancelling is conducted in a certain order. One usually hopes to first detect the strongest channel in order to minimize error propagation [14]. 5. Beamforming for MISO Channels The MISO channel is a very important type of channel in wireless communication systems, and in particular cellular systems due to the fact that multiple receive antennas are hard to implement in a mobile device due to the limited space constraint. Two types of transmit diversity schemes are standardized in the current third generation cellular systems [15]. The closed loop diversity or beamforming, requires CSIT; the open loop diversity, including selection diversity and space time block codes, etc. does not require CSIT. Consider the channel model of (2.1), where H = h is a 1 N t vector. Assuming perfect CSIT at the transmitter, the beamforming scheme simply transmits a single data stream, which is weighted by a vector h and then transmitted over different h N t antennas. It can be easily shown that this schemes achieves the capacity of the MISO channel. 6. Transmitter Side Pre-filtering: Zero-Forcing Beamforming and Ranked Known Interference In a single user system, if both the transmitter and the receiver have perfect CSI, singular value decomposition suggests a natural adaptive transmission scheme that can achieve capacity. Both transmitter and receiver antennas need to co-operate in order to implement the multiplications of V and U H for diagonalizing the H [3]. However, in a multi-user broadcast channel, since receivers belonging to different users

15 cannot co-operate, only transmitter side pre-filtering can be used. We introduce below two pre-filtering techniques, namely Zero-Forcing Beamforming (ZFB) and Ranked Known Interference (RKI) 1 which can be viewed as the dual of ZF and BLAST MIMO detectors, respectively [16]. Consider a broadcast channel model similar to (3.1) with N t transmit antennas at the base and K users, each with a single receive antenna (N t = 1). In the case of N r 1, we can view K = N r K, and still apply the same technique. Let X = Bv, where B denotes the pre-coding filter and v is a K 1 vector with the k th element denoting the information signal intended for User k. In ZFB, B = H H (HH H ) 1, so that the system is reduced to K independent parallel Gaussian channels whose power gain can be shown to be [16] b k = 1/(HH H ) 1 [k, k ]. (2.18) Note that ZFB requires K N t for the pseudo-inverse to be available. Let m = rank(h). Consider a QR-type decomposition H = GQ, where G C K m is a lower triangular matrix and Q C m Nt has orthonormal rows. In RKI, B = Q H and the channel becomes a set of m scalar sub-channels with interference as follows: y k = G[k, k ]v k + j<k G[k, j]v j + w k, k = 1, 2,..., m. (2.19) We denote d k = g k,k 2 as the power gain of the k th sub-channel to be used later. Since v and G are known at the transmitter, the interference in each channel is non-causally known at the transmitter; therefore, it can be pre-subtracted before transmission using the dirty paper type coding schemes [17, 18]. Since the ordering 1 RKI is renamed as zero-forcing dirty paper coding in [16]

16 of the users affects the total achievable rate, the scheme is referred to as ranked known interference. Note that in this scheme, the base can at most communicate with N t mobiles at a given instant, as in ZFB.

17 CHAPTER III ON THE SUM-RATE CAPACITY OF MIMO FADING BROADCAST A. Introduction CHANNELS A challenge in the design of cellular systems originates from the sharing of a common transmission medium by multiple users. On one hand, the system capacity of current generation cellular systems is limited by intra-cell and inter-cell interference, motivating techniques aimed at mitigating or suppressing multi-user interference. On the other hand, if some knowledge of user channels is available at the transmitter, adaptive transmission techniques, such as optimal resource allocation and interference pre-subtraction schemes, can be employed to exploit multi-user diversity and avoid multi-user interference, which can greatly improve overall system capacity. Consider a single cell and assume interference from other cells is modelled as Gaussian noise for mathematical convenience. Additionally, it is assumed the fading states of all the mobiles are known at the transmitter and all the receivers. Then, due to the presence of multi-user interference, the optimal power control scheme that maximizes the sum-rate of all the users in the cell for both uplink and downlink should consider the fading states of all the user channels. For the case of a single antenna at both the base and the mobile, the authors in [19] show that the maximum ergodic sum-rate capacity for uplink is achieved by a water-filling scheme across the mobile users. In other words, at any instance, the base need communicate only with the mobile enjoying the best received signal-to-noise-ratio (SNR). Similar results hold true for the downlink channel. If multiple transmit antennas are available at the base, adaptive antenna array techniques [15] can be employed to maximize the received

18 effective SNR of a mobile. The base can still communicate with the single user with the best effective SNR. If a mobile also has multiple receive antennas, a multipleinput-multiple-output (MIMO) data link can be established between the mobile and the base. At a given time, the base can communicate with the user whose MIMO link has the largest potential rate. This scheme is referred to as single-user-mimo (SU-MIMO) in later discussions. Recently, the authors in [6] showed that the optimal power control scheme that achieves the ergodic sum-rate capacity of a fading MIMO multiple-access-channel (MIMO-MAC) is one that may allow multiple mobile users to communicate with the base. In particular, up to 1 N(N + 1) mobile users can communicate with the base at 2 a given instant, where N denotes the number of receive antennas at the base. Can this result be directly extended to the downlink MIMO-broadcast channel (MIMO- BC)? To answer this question, the sum-rate capacity of a non-degraded Gaussian MIMO-BC needs to be evaluated first, whose capacity region is not known [5]. In [16], an interference pre-subtraction strategy using dirty paper type coding [17, 18] was proposed and shown to achieve the sum-rate capacity in the case of two transmit antennas and two users each with one receive antenna. Ref. [20] extended the work of [16] to the more general case of arbitrary number of users and antennas, and showed that the optimal precoding structure corresponds to a decision feedback equalizer that decomposes the broadcast channel into a series of single-user channels with interference pre-subtracted at the transmitter. Ref. [21] established a duality between the dirty paper achievable region for the MIMO-BC and the capacity region of the MIMO-MAC channel. The authors also showed that the sum-rate capacity of a Gaussian MIMO-MAC is the same as that of the Gaussian MIMO-BC with equal total power constraint, which greatly simplifies the evaluation of the sum-capacity of the MIMO-BC. The duality concept and sum-rate capacity of the MIMO-BC was

19 also independently studied in [22]. Applying the duality theory, Ref. [6] s results could be extended to the fading MIMO-BC, i.e., the optimal power control scheme should allow the base to communicate with more than one mobiles simultaneously. Therefore, SU-MIMO may not be optimal. In this chapter, we use convex optimization techniques to solve the optimal power allocation problem, evaluate the sum-rate capacity, and derive upper and lower bounds on the sum-rate capacity of the fading MIMO-BC. We show that the sumrate capacity of the fading MIMO-BC with perfect channel state information at the transmitter (CSIT) increases with the number of users K, but at an asymptotically very low rate [23]. In practice, the optimal solution requires large computation and can be hard to implement. Therefore, we study three sub-optimal multiple transmit antenna schemes, SU-MIMO, ranked known interference (RKI) [16] and zero-forcing beamforming (ZFB) [16, 24], in terms of achievable sum-rate and rate-loss compared to the optimal scheme [23]. Independent work on the topic has also appeared recently in [25, 26]. Note that all results in this chapter are based on the assumption that the number of transmit antennas N t satisfies N t K, which is practically reasonable for cellular systems. For wireless LAN applications, a recent paper [25] studied the sum-rate capacity when the number of receive antennas is N r = 1 and K grows to infinity in a fixed ratio with N t ( Nt K = β > 1), and evaluated the rate of linear growth of the sum-rate capacity. Finally, we note again that perfect CSIT is a key assumption in our model. In practice, a time division duplex (TDD) system under slow mobility conditions could be a good approximation of the model assumed in this correspondence, because in this case the channel states for the downlink could be estimated accurately from the uplink.

20 This chapter is organized as follows: In Section B we discuss the sum-rate capacity of the fading MIMO-BC and derive an upper-bound to it. In Section C we introduce three transmit pre-processing schemes based on RKI, ZFB and SU-MIMO to exploit multi-user diversity. We also derive the performance of a sub-optimal RKI scheme, which can serve as a lower-bound on the sum-rate capacity. Section D includes simulation results and Section E concludes. B. Sum-Rate Capacity of Fading MIMO-BC Consider a discrete-time fading MIMO-BC with N t transmit antennas at the base and K mobile users each with N r receive antennas. Let x C Nt 1 denote the transmitted vector, H k [j, i] the i.i.d. zero-mean flat fading channel gain between transmit antenna i (1 i N t ) and receive antenna j (1 j N r ) for User k (1 k K) and w k C Nr 1 the white Gaussian noise vector with w k N(0, I). Let y k C Nr 1 denote the received vector of the k th user. We have y = Hx + w, (3.1) where y = [y1 T y2 T yk T ]T, H = [H T 1 HT 2 H T K ]T and w = [w1 T w2 T wk T ]T, respectively. As shown in [21], the sum-rate capacity of a Gaussian MIMO-BC is equal to the sum-rate capacity of its dual Gaussian MIMO-MAC channel under the same total power constraint at the transmitter side. This dual MIMO-MAC channel has N t receive antennas at the base-station and K users each with N r transmit antennas, with the channel gain between transmit antenna j of User k and receive antenna i equal to H k [j, i]. Given all users CSI available at both the transmitter and the receiver side, the sum-rate capacity of the dual Gaussian MIMO-MAC based on fixed

21 channel state H is [27] C MAC sum (H) = max log S k ( ) K H H k S k H k + I, (3.2) where S k is the covariance matrix of the transmitted complex Gaussian signal vector of User k, subject to the sum power constraint K k=1 tr(s k) P. According to the duality result of [21], we can evaluate the ergodic sum-rate-capacity of the fading MIMO-BC as k=1 Csum BC = E { H C MAC sum (H)}, (3.3) where the expectation is with respect to the joint channel distribution of H. Since the value of H is known at the transmitter, Csum BC is achieved by choosing the optimal S k for each channel state. The problem of maximizing the ergodic sum-rate capacity of the fading MIMO-BC can be formulated as [ ( )] K Csum BC = max E H log H H k S k (H) S k(h)h k + I k=1 (3.4) subject to: K tr(s k (H)) P (3.5) k=1 S k (H) 0, for k = 1, 2,..., K. (3.6) where P is the total available power at the base-station. Note that A 0 means A is a positive semi-definite matrix. Here, instead of using a long-term average power constraint (E[ K k=1 tr(s k(h))] P ) [28], we use a short-term power constraint, which is a more practical assumption in the cellular downlink. Due to the sum power constraint, this problem is different from the one encountered in computing the sum-rate

22 capacity of the fading MIMO-MAC [6, 27], where each user has an individual power constraint (tr(s k (H)) P k for all k). A recent paper [29] extended the iterative water-filling algorithm of [27] to be used in solving the sum power constrained optimization. Although the proposed algorithm is shown to converge in the simulations, a rigorous proof on the convergence and the efficiency of the algorithm in the general case is yet not available. Note that the constrained optimization problem ((3.4) - (3.6)) is convex with the objective function containing the determinant of a complex Hermitian matrix. This type of problem can be numerically solved by the interior point method of [30]. However, we need to transform the complex matrix optimization problem into an equivalent real format before using the method in [30], which can only deal with real matrices. This process is shown in Appendix A. Here, we develop an upper bound which is more informative. To simplify notation, we denote S k (H) by S k. Let h i k denote the ith column of H k and ξ i = K k=1 hi H k S k h i k, for i = 1, 2,, N t. Letting Ψ = K k=1 HH k S kh k + I, we have ξ 1 + 1............ ξ 2 + 1...... Ψ =. (3.7)............... ξ Nt + 1 Let λ j k denote the jth eigenvalue of S k and λ max k we have = max j λ j k. Letting hmax i = max k h i k 2, ξ i K k=1 h max i h i k 2 λ max k, for i = 1, 2,, N t, (3.8) K k=1 λ max k, for i = 1, 2,, N t, (3.9) where denotes Euclidean norm and the first inequality above is in view of the

23 Rayleigh-Ritz theorem [31]. h i k 2 is actually the effective channel power gain if maximum ratio combining is employed at the receiver side for the link between User k and base antenna i. The power constraint is equivalent to K Nr k=1 j=1 λj k = P (note that λ j k 0 for any k and j), which suggests K k=1 We have the following series of bounds: [ Nt ]} Csum {log BC E (ξ i + 1) E { log i=1 [ Nt λ max k P. (3.10) ( i=1 h max i K k=1 λ max k + 1 )]} (3.11) (3.12) N t E [log (1 + h max 1 P )], (3.13) where the first inequality is in view of Ψ being positive definite and Hadamard s inequality, the second is in view of (3.9) and the third because the h max i are identically distributed random variables and (3.10). We note that the bound in equation (3.11) without expectation is true for every fading state, and therefore also true when the expectation is taken. Remarks: h i k 2 (k = 1, 2,..., K) are i.i.d. random variables having central Chi-square distribution with 2N r degrees of freedom, denoted as χ 2 2N r. The corresponding probability density and cumulative density functions are f(z) = z Nr 1 e z /(N r 1)! (3.14)

24 and N r 1 F (z) = 1 e z i=0 z i i!, (3.15) respectively. The asymptotic cumulative distribution function of h max 1 (K ), which is the maximum of K i.i.d. χ 2 2N r distributed random variables, can be evaluated according to [32, 33] as F (z) = exp [ e (z l K) ], (3.16) where l K can be computed by solving the following equation ( Nr 1 ) e l l i K K = 1 i! K. (3.17) Since l K > 0, we have N r 1 i=0 l i K i! i=0 1 with equality iff N r = 1. Therefore, equation (3.17) suggests that l K ln(k), i.e., the channel gain h max 1 grows, on average, at least as ln(k). Moreover, a larger N r results in a larger l K. However, for any fixed N r, we always have lim K ln(k) l K = lim K [1 ln( N r 1 i=0 Therefore, as K, l K increases as ln(k) independent of N r. To see how C BC sum bound the right-hand side of (3.13): l i K i! )/l K ] = 1. can be affected by K, we use Jensen s inequality to further C BC sum N t log [1 + E (h max 1 ) P ] (3.18) = N t log [ 1 + P [γ + l K + E i (1, e l K )] ] (3.19) N t log [1 + P (γ + l K )], (3.20) where E i (n, x) = 1 e xt /t n dt (3.21)

25 is the exponential integral and γ = 0.577215... is Euler s constant. The approximation in (3.20) is quite good since E i (1, e l K ) < 1 1 e el K t dt (3.22) e Kt dt = e K K, (using l K ln(k)) (3.23) even for a moderate value of K = 5, e K K = 0.0013 γ. Therefore, for large K, we can ignore E i (1, e l K ). We also note that the RHS of (3.18) is also a good approximation of the RHS of (3.13) when K is large as shown in the simulation results. This is because the asymptotic distribution of h max 1 is highly concentrated around l K, so Jensen s inequality is fairly tight. The upper bound of (3.18) suggests that the sum-rate capacity of a fading MIMO-BC with perfect CSI available at the transmitter is mainly limited by N t when N t K. The upper bound increases log-likely with N r, which can be roughly concluded by the fact that E[ h i k 2 ] = N r. This result is not surprising since the sum-rate-capacity of the MIMO-BC is bounded by the capacity of the N t (N r K) single user MIMO channel where receivers can cooperate. Then, according to [1], the ergodic capacity of the single user MIMO channel can increase linearly only with min{n t, N r K} = N t due to N t K. However, we note that as K (3.18) is an asymptotically tighter bound than the capacity of the cooperative MIMO system at high SNR. This is because the capacity of the single user MIMO system increases log-likely at high SNR as K increases due to receiver cooperation. In contrast, the proposed upper bound of (3.20) increases log-likely with l k at high SNR, which in turn increases as ln(k) as K. It is well known that the sum-rate capacity increases with increasing number of users due to multi-user diversity when perfect CSIT is available