Low-Complexity Decoding via Reduced Dimension Maximum-Likelihood Search Jun Won Choi, Byonghyo Shim, Andrew C. Singer, and Nam Ik Cho

Transcription

1 1780 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 3, MARCH 2010 Low-Complexity Decoding via Reduced Dimension Maximum-Likelihood Search Jun Won Choi, Byonghyo Shim, Andrew C. Singer, Nam Ik Cho Abstract In this paper, we consider a low-complexity detection technique referred to as a reduced dimension maximum-likelihood search (RD-MLS). RD-MLS is based on a partitioned search which approximates the maximum-likelihood (ML) estimate of symbols by searching a partitioned symbol vector space rather than that spanned by the whole symbol vector. The inevitable performance loss due to a reduction in the search space is compensated by 1) the use of a list tree search, which is an extension of a single best searching algorithm called sphere decoding, 2) the recomputation of a set of weak symbols, i.e., those ignored in the reduced dimension search, for each strong symbol cidate found during the list tree search. Through simulations on -quadrature amplitude modulation (QAM) transmission in frequency nonselective multi-input-multioutput (MIMO) channels, we demonstrate that the RD-MLS algorithm shows near constant complexity over a wide range of bit error rate (BER) ( ), while limiting performance loss to within 1 db from ML detection. Index Terms Dimension reduction, list tree search, maximumlikelihood (ML) decoding, minimum mean square error (MMSE), multiple input multiple output (MIMO), sphere decoding, stack algorithm. I. INTRODUCTION T HE complex-domain relationship between the transmitted symbol received signal vector in many communication systems can be expressed as where is the transmitted vector whose entries are chosen from a finite symbol alphabet, are the received signal noise vectors, respectively, is a channel matrix. Multiple-input-multiple-output (MIMO) links are a typical example described by this model. In order to achieve the diversity multiplexing gains promised by MIMO technologies [1], [2], a powerful MIMO detection scheme for recovering the transmitted symbol with minimal error is indispensable. In particular, a maximum-likelihood (ML) tree search algorithm referred to as sphere decoding (SD) has received much attention in recent Manuscript received May 05, 2009; accepted October 10, First published November 10, 2009; current version published February 10, Part of this paper was presented in the IEEE Sensor Array Multichannel Signal Processing (SAM) Workshop. This work was supported in part by the second BK 21 project. The associate editor coordinating the review of this manuscript approving it for publication was Prof. Ta-Sung Lee. J. W. Choi A. C. Singer are with the Department of Electrical Computer Engineering, University of Illinois at Urbana-Champaign, Urbana, IL USA. B. Shim is with the School of Information Communication, Korea University, Seoul, Korea ( bshim@korea.ac.kr). N. I. Cho is with the Electrical Computer Engineering Department, Seoul National University, Seoul, Korea. Digital Object Identifier /TSP (1) years [3] [7]. The ML search algorithm searches over the lattice points spanned by noiseless channel outputs to find the one with minimum value of, where a Gaussian noise assumption has been made. Instead of enumerating all lattice points, the SD algorithm restricts the search space to within a sphere centered at the received vector, thereby achieving a considerable reduction in computational complexity. In spite of this benefit, the computational burden of the SD algorithm is still a major concern, since its expected complexity remains exponential with respect to problem size for a fixed signal-to-noise ratio (SNR) [11]. Considering the growing dem for high data rate services in next-generation wireless systems, it remains a challenge to apply the SD algorithm to MIMO systems of large dimension high-order constellations. There have been a number of approaches to reduce the complexity of the SD algorithm, such as the Schnorr-Euchner enumeration [8] [10], descending probabilistic ordering [14], increasing radius sphere decoding [20], the parallel competing branch algorithm [12]. Other approaches trading performance for complexity include the radius scheduling method [13], -best sphere decoder [15], probabilistic tree pruning algorithm [16], [17], sequential Fano decoders [24], -algorithm [21], -algorithm [22], semidefinite relaxation [23]. In this paper, we introduce a near-ml detection technique, referred to as a reduced-dimension ML search (RD-MLS) that provides significant complexity reduction, yet maintains a near-ml performance. By reducing the dimension of the search space from to, the RD-MLS directly achieves a significant reduction in the number of lattice points that must be searched from to. Owing to the direct benefit on complexity, there have been a number of studies [25] [33] on partitioned search techniques where a subset of the symbol vector are estimated by suboptimal methods the remaining symbols are more carefully searched. In [25] [26], a symbol estimate is obtained by concatenation of the elements obtained by an exhaustive search those obtained via zero-forcing or decision feedback estimation. However, due to imperfect decisions made by linear detectors, these schemes suffer a performance loss. In order to mitigate the performance loss caused by dimension reduction, techniques generating multiple cidates for a portion of the symbols choosing a solution among those concatenated with linear estimates have been proposed. It is shown in [27] [28] that enumeration of full cidates achieves the diversity gain of the exact ML detector. Similar methods employing more refined postdetection schemes are found in [29] [30]. In [31], an approach selecting symbol cidates that are close to the minimum mean square error (MMSE) estimate X/$ IEEE

2 CHOI et al.: LOW-COMPLEXITY DECODING VIA REDUCED DIMENSION ML SEARCH 1781 Fig. 1. Illustration of the previous partitioned search schemes the RD-MLS detection. (a) Fixed-complexity sphere decoder [28]. (b) B-Chase detector (l) [31]. (c) RD-MLS detector. The RD-MLS detector finds the cidates using the LTS after reducing the problem size via the MMSE dimension reduction operator while the schemes in [28] [31] fully or partially enumerate the cidates. Refer to Section III for the details of the RD-MLS algorithm. was proposed. Such a partitioned search idea was extended to soft-output maximum a posteriori (MAP) detection in [32] [33]. These schemes allow for a fixed complexity [27] [30] or reduced worst-case complexity [31] but often require a large number of symbol cidates to achieve near-ml performance, resulting in considerable complexity. Our RD-MLS technique is distinct from these approaches in two respects. First, rather than performing an exhaustive or ad-hoc enumeration of cidates as in [27] [31], we employ a list tree search (LTS) method [34], [35] to find promising symbol cidates. The LTS is employed after applying an MMSE dimension reduction operator that performs soft cancellation of interference (see the illustration in Fig. 1). While the LTS has been used to perform soft output decoding [34], [35], its application to partitioned search has not yet been explored to our knowledge. In fact, owing to the LTS, the number of cidates generated to achieve near-ml performance reduces significantly compared to previous schemes [27] [31]. Second, we introduce an efficient LTS algorithm, called a closest-k list stack algorithm ( -LSA), which finds a flexible, but limited number of closest lattice points. Contrary to previous LTS algorithms visiting a large number of lattice points to obtain accurate a posteriori probabilities [34], [35], the -LSA reduces the number of lattice points to visit by employing a stopping criterion which terminates the cidate search adaptively as well as a probabilistic bias for pruning additional unnecessary branches. As a result, the RD-MLS can maintain modest complexity for various channel noise conditions. Through an asymptotic performance analysis, we observe that the diversity gain of the RD-MLS is at most, compared to of the full-dimensional ML search. We show that the -LSA can bring an improvement in the effective SNR by a factor proportional to the size of cidate list, thereby compensating the diversity reduction. We observe through simulation that the performance loss due to the diversity gain reduction is partially mitigated by the additional algorithmic gains offered by the -LSA, leading to performance that appears close to that of the ML detector. In addition, it is shown that the RD-MLS achieves significant complexity reduction over the SD algorithm as well as previous near-ml approaches [31]. The rest of this paper is organized as follows. After describing the system model in Section II, we briefly review the SD algorithm its computational complexity in Section III. We present an asymptotic performance analysis in Section IV. Simulation results are provided in Section V, we conclude in Section VI. We briefly summarize the notation used in this paper. Uppercase lowercase letters written in boldface denote matrices vectors, respectively. The superscripts denote transpose conjugate transpose, respectively. denotes an -norm square of a vector, is a diagonal matrix which has input elements on the main diagonal. denote the real imaginary part of, respectively. denotes a Chi-square distribution with degrees of freedom (DOF). are the cumulative

3 1782 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 3, MARCH 2010 density function (CDF) the inverse CDF of the -rom variable with DOF, respectively. denotes the -function defined as. denotes. denotes a complex Gaussian with the mean variance. II. SPHERE DECODING ALGORITHM A. Sphere Tree Search After the real conversion of complex matrices vectors, the ML detection problem can be written by entry of. Emphasizing that each term in the summation in (6) is a function of, (6) becomes where denotes a set of variables. The SD algorithm can be interpreted as a tree search, where each node is associated with the variables (see [5] [6] for details). A path metric assigned for each node of the tree is defined as (7) where is the channel matrix is the vector comprising elements of the -quadrature amplitude modulation (QAM) set defined as (3), shown at the bottom of the page, where is chosen to satisfy the normalization condition. For example, for 16-QAM for 64-QAM modulation, respectively. The SD algorithm searches the lattice points inside a hypersphere of radius, centered at the received vector [3], [6]. The sphere constraint is expressed as where,,. In the sequel, we let. In order to perform a systematic tree search, following [6], we perform the QR decomposition of the channel matrix, i.e., where are matrices is an upper triangular matrix whose diagonal elements are nonnegative. Since a norm operation is invariant to orthogonal transform, the sphere constraint of (4) can be rewritten where,,,, is the th (2) (4) (5) (6) A complete path starting at the root ending at the bottom of the tree represents a realization of symbol vector, the path having minimum path metric among all complete paths becomes the ML solution of the tree search. In order to find the ML path, the following relationship between parent-child pair nodes is employed: By additivity, we have that for, hence, the path metric monotonically increases with tree depth. Hence, for a node whose path metric violates the sphere condition, i.e.,, all leaf nodes of its subtree violate the condition as well, so that the node its subtree are removed from the tree without loss of optimality, so long as at least one leaf satisfies the sphere codition. Two popular methods for searching the nodes in a branch are Pohst enumeration [3] SE enumeration [8]. In Pohst enumeration, natural spanning from the minimal to the maximal value is used between the interval where (8) (9) (10) (11) (12) where. In contrast, the SE method enumerates the admissible points in a zig-zag order from the midpoint. That is, the SE enumeration spans, when (3)

4 CHOI et al.: LOW-COMPLEXITY DECODING VIA REDUCED DIMENSION ML SEARCH 1783 Fig. 2. Block diagram of the RD-MLS detector., otherwise. By traversing the tree with this branch ordering mechanism, all lattice points inside the sphere are visited the final lattice point having the minimum path metric becomes the ML point. B. Complexity of SD Algorithm Due to the data-driven nature of the search, computational complexity of the SD algorithm is nondeterministic. Expected complexity has been widely considered for assessing the relative computational complexity of various approaches to the SD algorithm [6], [11]. Assuming a uniform distribution of computational cost across the nodes, a lower bound on the expected number of nodes visited by the search algorithm becomes [11] (13) where is the number of the visited nodes, is a modulation order, is the complexity exponent given by SNR (14) where is defined in (3). Since increases exponentially with, search dimension reduction poses a clear strategy for reducing complexity. However, simple reduction of the search dimension might cause significant performance loss, so that a careful mechanism for mitigating such performance loss is needed. In the following section, we propose a dimension reduction algorithm that attempts to mitigate such performance loss, while enabling substantial complexity reduction. III. REDUCED-DIMENSION ML SEARCH ALGORITHM The structure of the RD-MLS system is depicted in Fig. 2. The dimension reduction operator reduces the system dimension by suppressing interference, i.e., the contribution of symbols not participating in the LTS operation. New observation system matrix obtained from the dimension reduction operator are delivered to the -LSA block, which performs the closest lattice point search over the reduced search space. Since the closest point of the reduced-dimension system is not necessarily equal to the ML solution of the original system, we find multiple cidates via LTS. Then, each cidate of the list (denoted as ) found by the -LSA is extended to the full symbol dimension via MMSE-decision feedback (MMSE-DF) estimation. Among the extended list, a final estimate is chosen based on -norm criterion. A. Dimension-Reduced ML Problem As the first step for dimension reduction, the symbol vector is divided into two vectors. With knowledge of the received data channel, the ML solution becomes (15) where are the submatrices constructed by columns of, respectively. Denoting, can be expressed as (16) (17) Insertion of (17) into (16) will return to (15), no dimension reduction is therefore achieved. In order to restrict the search space within, we use a linear estimate of in place of. Employing a linear estimate of, i.e.,, for a given, we obtain the approximate ML estimate is given, the linear LMMSE (LMMSE) esti- Assuming that mate of is (18) (19) (20) Using (18) (20), the approximate ML solution becomes (21), shown at the bottom of the page. Defining the projection operator then, (21) is written as (22) (23) (21)

5 1784 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 3, MARCH 2010 Further, by denoting, we obtain the integer least squares problem (24) B. MMSE Dimension Reduction Operator The preprocessing operation consists of 1) an application of the linear operator 2) a tree search over the transformed system. From the relation, we can express as (25) where. As discussed, is an interference term to detect, the contribution of this term is minimized by the preprocessing. In fact, from the definition of, one can show that (22) can be written as (26) which is the LMMSE estimate of, i.e.,. Hence, (25) becomes (27) where is the LMMSE estimation error. In Section IV-A, we will show that the performance of the RD-MLS detector is limited by the detection performance of. A potentially useful choice for would be that which maximizes the receiver SNR for detecting in (27), i.e., where SNR (28) SNR (29) (30) To obtain an optimal partition from (28), choices would need to be examined, which is clearly burdensome for a large. Thus, a simple scheme such as V-BLAST symbol ordering [18] or probabilistic symbol ordering [14] can be alternative, where the symbols associated with are detected first. C. Closest- List Stack Algorithm ( -LSA) As mentioned, due to the reduced dimensionality of the search, is not guaranteed to be the true ML solution, thus performance loss is unavoidable. In order to mitigate the loss, a list tree search generating multiple cidates for is employed. In [34], the list SD (LSD) algorithm to find best lattice points was proposed. Contrary to the SD algorithm where the radius of the sphere is updated dynamically for each cidate found, the LSD algorithm maintains a fixed radius until it finds the best points. The radius is updated only when the list is full a new cidate replacing an existing one is found. In many cases, therefore, excessive numbers of lattice points are visited, which can easily reduce the benefits of dimension reduction. To maintain the complexity gains of the reduced dimension search while pursuing the performance gain of the LTS, we employ a closest- list stack algorithm. As a best-first tree search technique, the stack algorithm (SA) [19], [20] extends the node in a tree with the minimum-cost metric. For every node extension, node information is stored in the stack, the best node is chosen based on a cost metric of the nodes in the stack. In the reduced system in (24) described in Section III-A, the cost metric for a tree node after the conversion to the real domain is defined by [24] (31) where is the path metric of the node (defined in Section II-A), is the set of child nodes of not generated, is a bias term penalizing short paths [19], [24], [35]. Notice that. While the search of the SA is finished once it arrives at the first leaf node corresponding to the ML point, the -LSA continues the search until it finds additional closest points. However, due to the multiple-latticepoint search, large numbers of back-tracking operations occur. In order to alleviate the complexity increase, the -LSA employs two measures: a stopping criterion probabilistic bias. The number of points collected in the cidate list directly impacts the complexity of the RD-MLS detector. In order to adjust the cidate list size effectively, we terminate the search before filling the list via a stopping criterion. After finding the first closest point, the stopping criterion checks if the path metric of the subsequently found closest points satisfies the condition, i.e.,. Then, the search is stopped if the stopping condition is satisfied. In order to choose, we consider the path metric of the actual transmitted symbols denoted as, i.e., (32) where was defined in Section III-B. The parameter can be chosen such that the probability of the path metric for the transmitted symbols being less than equals some probability, i.e., (33) Since it is hard to find analytically for non-gaussian, we introduce a heuristic way for choosing. Using the path metric of first found closest point, we let, where is called a stopping parameter. Using this condition, the -LSA collects the points whose path metric is less than times of that first found. The rationale behind this choice is that for the case of benign channel noise conditions (e.g., small ), the path metric of the transmitted symbol is significantly smaller than that of other lattice points, so that it is highly possible to find with only a small number of cidates. In the opposite conditions, many paths have a similar path

6 CHOI et al.: LOW-COMPLEXITY DECODING VIA REDUCED DIMENSION ML SEARCH 1785 in our method is chosen by taking into account the contributions of rom noise in the unvisited paths. Specifically, we model to represent the noise contributions from the unvisited levels of the tree,, then assign the probabilistic condition, where is the pruning probability. For a specific, is given by (34) In general, decreases with the tree depth assessing larger bias to short paths. By using an appropriate value for (such as through empirical simulations), we can achieve substantial reduction in back-tracking operations with negligible performance loss. A similar approach has been applied to the SD algorithm in [17] to SA in [36]. Our approach differs from that of [36] in that rather than using the expected noise power to obtain a bias term, we employ a probabilistic condition (34) to choose. Hence, our bias term can be controlled flexibly through the parameter. Fig. 3. Illustration of the stopping criterion for the 2 2 x system G. The dark circle O corresponds to the transmitted symbol vector Hx. The open circles are the observed signal vectors for two scenarios of (a) bad (b)good noise conditions, i.e., k ^w k > k ^w k. As shown in the figure, for bad noise realizations, the points near the observed vector z are likely to have a similar distance metric so that many lattice points are collected. On the other h, for good noise realizations, only a few lattice points have small distance metric. With high probability, the ML point will be found among them. The stopping criterion with m = 1:5 selects the points inside the gray circle as cidate, hence it collects fa; B; C; Og for the case (a) only fog for the case (b). metric so that the number of cidates should increase in efforts to keep the true solution in the list. Refer to the illustration in Fig. 3. Due to the stopping criterion, the cidate list size is adjusted to the channel noise condition lattice points with dominant metric are stored in the list. In Section IV-A, we will show that with this stopping criterion, the LTS offers performance improvement over detection without LTS. Next, we introduce a method to choose the bias term in (31). To compensate the path metric of short paths such that the most likely path chosen to extend appropriately accounts for the differing path lengths of the nodes visited, a proportionate bias term has been used in the traditional stack algorithm [19], [24]. While the bias term in these approaches can be approximate, D. Postprocessing The postprocessing operates in two steps. First, for each of the cidates for obtained by the LTS, we generate the symbol vectors of full dimension by concatenating them with the MMSE-DF estimates of. Then, among those vectors, we choose one minimizing the Euclidean distance metric as the final output. We assume that the columns of entries of are arranged based on the detection ordering provided in [14] or [18]. We let be the th element in the cidate list be the corresponding MMSE-DF estimate of given. Further, we let be the th column of. To obtain the MMSE-DF estimate of, for each cidate of, we first subtract the effect of from, i.e.,, then obtain the estimates successively. The MMSE-DF detection steps can be summarized as follows. STEP 1: Compute for all. STEP 2: (Iteration) for all for, compute end (35) slicer (36) (37) where slicer denotes the function of mapping the complexvalued to the nearest transmitted constellation point. The MMSE-DF filter coefficient is given by [37], [38] (38)

7 1786 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 3, MARCH 2010 Once is obtained by (35) (37), are concatenated for the final list search as (39) the element of minimizing the cost function (40) becomes the final output of the RD-MLS (40) The whole detection procedure of the RD-MLS algorithm is summarized in Appendix A. IV. DISCUSSION The two key parameters affecting the complexity performance of the RD-MLS are the dimension stopping parameter. In this section, we present the performance analysis for the RD-MLS show how these parameters affect the error performance. A. Performance Analysis The aim of this subsection is to derive the upper bound of a detection error probability for the RD-MLS. To make the analysis tractable, we consider the case where the system matrix is partitioned without any column ordering so that the elements of are assumed to be rom i.i.d. complex Gaussian. That is, we disregard the partitioning criteria described in Section III-B for the simplicity of analysis. The detection error probability of the RD-MLS detector,, is defined as (41) where refers to the conditional probability of event under the condition that was transmitted. Let be an exact ML solution. We denote, which is the ML detection error probability given was sent. Then, we have (44) (46), shown at the bottom of the page, where (44) follows from the fact that the RD-MLS chooses a closest point from while the ML detector does from all cidate points. From (46) the fact that, (43) becomes (47) The second term in the right-h side of (47) illustrates the primary source of suboptimality of the RD-MLS. Following symbol partitioning, let be divided into. Let the corresponding partitioned cidate sets be. We can show that (47) becomes (48) (49) where (49) follows from. The second third terms on the right-h side in (49) are the probability that the cidate list found by the LTS does not contain the conditional probability that the MMSE-DF estimation makes a decision error given the LTS yielded in, respectively. We denote these terms as, respectively. From (41) (49), an upper bound of is written as in where is the a priori probability that was transmitted is the conditional error rate (CER) that is not detected by the RD-MLS given that was transmitted. Recalling that the final output chosen from the list is denoted as, becomes (42) (50) (51) (43) (44) (45) (46)

8 CHOI et al.: LOW-COMPLEXITY DECODING VIA REDUCED DIMENSION ML SEARCH 1787 where,, are defined similarly as. To investigate the asymptotic behavior of, we consider the input to the LTS in (27),. We remind the reader that,,. Using these, we can write by (52) Though is discrete in, we approximate the LMMSE estimation error plus noise, as a complex Gaussian vector with zero mean the covariance matrix. For large SNR, we note that becomes a zero matrix while does not vanish as (see Appendix B). From (52) also using this result, approximates to a Gaussian distribution as SNR increases. Asymptotic normality of this type of interference plus noise term for MMSE detection was also validated in [40] [41]. In the LTS with the stopping parameter using the Gaussian approximation, we show in Appendix C that is upper-bounded by (53) where is the normalization factor of the QAM modulation are the eigenvalues of,. The notation implies an expectation taken over. We consider the high SNR regime of (53) as. If we let be the rank of, it follows that for, where are the (nonzero) eigenvalues of. Then, we have the upper bound is a decreasing function of the list stopping parameter. This follows our intuition as, when increases, the number of cidates in the list also increases, the effect of additional loss decreases. Since multiplies the ratio of the minimum squared distance to the noise power, it provides additional effective SNR gains over the single-cidate search. Second, we observe that the diversity gain of the term related to is larger than or equal to. We next take a look at the probability. This is the probability that the MMSE-DF estimation makes a decision error for the system resulting from perfect cancellation of in (35). It is well known that the MMSE-DF detector for the -by- system achieves a diversity of both with without the ordering based on postdetection SNR [18], [42]. Based on (51), at high SNR, the diversity gain would be dominated by, which is the minimum of those for,,, i.e.,. In fact, this diversity gain is the same as that achieved by the scheme employing a single-cidate search [26]. However, owing to the performance gains offered by the LTS, performance loss from the ML search becomes moderate over a wide range of operational regimes. This will be shown in the simulation results in Section V. B. Comments on Complexity The overall complexity of the RD-MLS algorithm comprises those for: 1) the dimension reduction operator; 2) the tree search; 3) the postprocessing operations. In our analysis, the complexity is evaluated by the number of floating-point operations (FLOPS) counted per channel use. The number of FLOPS needed for the tree search operation is expressed as (59) (54) (55) (56) (57) (58) where the inequality in (55) gets tighter as since (56) holds because regardless of realizations of. Since this upper bound does not depend on, it becomes an upper bound of. From (58), we can provide some important observations on the asymptotic behavior of the RD-MLS. First, we observe that where evaluates to 1 if the path is visited, 0 otherwise. is the number of FLOPS per node at tree level. The total number of FLOPS depends on the number of nodes visited. Table I shows the number of FLOPS in the dimension reduction postprocessing steps per node computations of the -LSA. The FLOPS required for the preprocesssing steps are derived from the calculation of while those for postprocessing step is derived from the calculation of in (35) (37). The derivation of is obtained by inspecting the computation of the residual signal path metric update as in (9). From this the node visitation information, averaged numbers of FLOPS (per channel use) for the RD-MLS detector can be measured. V. SIMULATIONS In this section, we observe the complexity performance of the proposed RD-MLS as compared to that of the full-dimensional SD other near-ml detectors. The simulation setup is

9 1788 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 3, MARCH 2010 TABLE I COMPLEXITY OF RD-MLS ALGORITHM based on -QAM transmission over MIMO systems in quasistatic Rayleigh fading channels where the elements of are modeled by independent complex Gaussian rom variables. As a measure of the performance complexity, bit error rate (BER) the average number of FLOPS are used. To obtain the average number of FLOPS through simulations, the average number of real additions is counted per signal transmission. In the simulation, following algorithms are compared. 1) The state-of-the-art full-dimensional SD (FD-SD) algorithm: Algorithm II in [5], which guarantees exact ML performance. 2) RD-MLS : -LSA with the pruning parameter. indicates the parameter sets of RD-MLS; is the reduced dimension is the stopping parameter defined in Section III-C. 3) B-Chase detector of [31]: is the parameter specifying the number of first layer symbols that are picked up to generate the cidate list. 4) Probabilistic tree pruning (PTP) sphere decoder [17]: indicates the pruning probability that controls the radius of sphere search. 5) -best sphere decoder [15]: indicates the maximum number of cidates picked up every tree level. We set the parameter to 5 when SNR db to 10 when SNR db as suggested in [15]. We do not consider the dimension reduction methods in [25] [26] in our comparison due to their performance gap from ML detection. We instead consider the B-Chase detector, which contains several partitioned search detectors [28], [29] as a special case. In addition, we include two low-complexity sphere decoding algorithms [15], [17]. Assuming that the block fading length is large, the complexity of preprocessing such as QR decomposition or STEP 2 in [31] are ignored since they can be shared in the block. In the RD-MLS setup, we set the maximum cidate size of the list to 16 based on intensive simulations. First, we consider 16-QAM transmission for 8 8 MIMO systems, where parameters of the RD-MLS are set to. In Fig. 4, we provide the BER results average FLOPS over the SNR range between db. As shown in Fig. 4, the RD-MLS algorithms exhibits far lower complexity than the full-dimensional SD while maintaining performance within 1 db for most of the SNR range of interest. In particular, at 19 db of SNR, the RD-MLS achieves up to 25% 55% complexity reduction for over the FD-SD. It is also clear that the RD-MLS has best performance/complexity tradeoff among all low-complexity detection schemes considered. In particular, RD-MLS shows a clear complexity benefit in mid high SNR regime since the stopping criterion prevents the LTS from finding unnecessarily many cidates. In Fig. 5, we provide the results of systems with two configurations of RD-MLS. Overall, we observe that the complexity gain of RD-MLS over the Fig. 4. Plots of average BER FLOPS of the QAM system. FD-SD improves dramatically compared to the 8 8 case. The complexity of RD-MLS is lower than those of the PTP-SD -best SD algorithms. Although the B-Chase achieves the lower complexity than RD-MLS in this scenario, it suffers substantial performance loss from the ML detector. On the contrary, the performance loss of the RD-MLS is maintained within 1 db. In Fig. 6, the performance complexity curves for a 6 6 MIMO system with 64-QAM transmission are provided. In the RD-MLS, two configurations are considered. In general, we observe that the performance loss of the RD-MLS over exact ML performance is within 1 db in the BER range of to. For this high-order modulation format,

10 CHOI et al.: LOW-COMPLEXITY DECODING VIA REDUCED DIMENSION ML SEARCH 1789 Fig. 5. Plots of average BER FLOPS of the QAM system. Fig. 6. Plots of average BER FLOPS of the QAM system. we observe that the RD-MLS algorithm achieves a substantial reduction in complexity over the full-dimensional SD while performing close to the ML detector over the SNR range of interest. The increased complexity reduction as compared to the 16-QAM case matches our intuition that the search complexity would be scaled by a factor of due to the reduction in the search space dimension. In Fig. 7, we test how the stopping criterion adjusts the size of the cidate list for different SNRs. In order to observe this, the average size of the cidate list collected by the -LSA for each transmission is measured. A QAM MIMO system is employed with the parameters of the RD-MLS set to (4, 4). The stopping criterion exploits the property that only a small fraction of cidates are necessary to maintain good performance at high SNR. In fact, we observe that the average size of the cidate list decreases with SNR. Fig. 8 shows how complexity performance are traded off through the parameter set. We consider several combinations of in a 6 6 system with 64-QAM modulation. From the BER FLOPS curves in Fig. 8, we observe a clear tradeoff between complexity performance. As or increases, the performance gap from the full-dimensional SD Fig. 7. Average size of the cidate list versus SNR of the K-LSA. The QAM case is considered. decreases at the expense of complexity increase. This demonstrates how the RD-MLS can bridge the gap between the high complexity ML detector a linear suboptimal detector.

11 1790 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 3, MARCH 2010 APPENDIX A SUMMARY OF THE RD-MLS ALGORITHM The following definitions are used. STACK: memory storing the generated nodes. XLIST: memory storing the cidate points found. : current best node (called top node) that has a minimum cost in the stack. : best child node of not yet generated. In addition, we assume that the symbol is ordered according to the V-BLAST detection ordering the dimension reduction operator the matrix are computed prior to the routine that follows. Fig. 8. Complexity performance of RD-MLS for several configurations of (n ;m). The QAM case is considered. VI. CONCLUSIONS A low-complexity near-ml detection technique referred to as RD-MLS detection is presented in this paper. In addition to dimensionality reduction, which directly impacts complexity, two main ideas are proposed for mitigating the performance loss incurred. First, detection in the reduced-dimension system is modified from an ML search to a list tree search. Therefore, instead of detecting the one best symbol, multiple cidates are found in the LTS stage, enabling errors introduced by the dimension reduction step to be mitigated. Second, for each of the symbol cidates found by the LTS, the rest of the symbols are estimated via an MMSE-DF algorithm. After concatenating two symbol estimates, the final output is chosen as the unique minimizer of the ML cost function. We have found, from asymptotic performance analysis, that performance gains that increase the effective SNR can be achieved by the LTS. We observe from simulations that the BER performance of the RD-MLS represents a good balance between the performance of the full-dimensional SD the complexity of linear receivers. Input:,,, Output: STEP 1: (Preprocessing) Compute STEP 2: (Initialization for -LSA) STACK root node STEP 3: (Main routine of -LSA) Let be the current top node. (Node extension) if XLIST else STACK For a new extension, update. end (Parent node update) if all child nodes of have been generated, remove from STACK. else update,. (extend to the next sibling node ) end STEP 4: (Stack sorting) Compare the path metrics of all STACK elements nominate one with minimum path metric as a top node. STEP 5: (Check the stopping criterion) If satisfies the stopping criterion, go to STEP 6, else, go to STEP 3. end STEP 6: (Postprocessing) Using (35) (37), extend all elements of XLIST to full dimension. Output the best cidate by comparing the norm distance of them.

12 CHOI et al.: LOW-COMPLEXITY DECODING VIA REDUCED DIMENSION ML SEARCH 1791 APPENDIX B ASYMPTOTIC NORMALITY OF We assume that the partitioned matrices are given for now. Via singular value decomposition (SVD), we can decompose to, where. Then, can be expressed as its real part is given by, the variance of (60) (61) where is a square root of, i.e.,. Recall that for real Gaussian variable with zero mean variance. Then is expressed as... (62) If we let be the rank of, it follows that. Hence, we have for. On the other h, since for,wehave as. Therefore, vanishes as. In the similar manner, we can express as From the definition of a matrix norm i.e.,,wehave (67)... (63) (68) Note that for,. Since ( ), the projection operator never vanishes. APPENDIX C PROOF OF (53) The cidate search is stopped when a lattice point whose distance is larger than is found. In addition, to simplify the analysis, we include the final lattice point in the list, which is the first lattice point violating the stopping criterion. With this assumption, the list can be expressed as where. Due to the stopping criterion, we have Let be the elements of that correspond to the maximizer minimizer of the cost function, respectively. Given a cidate list, the probability is expressed as (64) (66), shown at the bottom of the page, where. An affine transform of a proper Gaussian variable remains proper Gaussian [43] so that is also proper Gaussian. Recalling By taking an expectation of (69) with respect to,wehave (69) plugging (70) (64) (65) (66)

13 1792 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 3, MARCH 2010 After some manipulation, (70) becomes Since the minimum distance between two constellation points is from (3), it holds that. Finally, the union bound of is given by (71) (72) for given, where (72) follows from that is positive semidefinite. Using from (68) (72), we have (77) (78) (79) (73) Let be decomposed into, where is a unitary matrix is the diagonal matrix whose diagonal entries are the eigenvalues of,. From (22), is expressed as. In addition, since, equals hence (74) where. To remove conditioning, needs to be averaged over. Employing the property, we have (75) where is the th row vector of. With the assumption that the elements of (equivalently ) are i.i.d. complex Gaussian,, follows. Hence, we can show that (75) can be expressed as [45] (76) REFERENCES [1] G. J. Foschini M. J. Gans, On limits of wireless communication a fading environment when using multiple antennas, Wireless Personal Commun., vol. 6, pp , Mar [2] E. Telatar, Capacity of multi-antenna Gaussian channels, Eur. Trans. Telecommun., vol. 10, pp , Nov [3] U. Fincke M. Pohst, Improved methods for calculating vectors of short length in a lattice, including a complexity analysis, Math. Comput., vol. 44, pp , Apr [4] E. Viterbo E. Giglieri, A universal lattice decoder, in GRESTSI 14-eme Colloque, Juan-les-Pins, France, Sep [5] M. O. Damen, H. E. Gamel, G. Caire, On maximum-likelihood detection the search for the closest lattice point, IEEE Trans. Inf. Theory, vol. 49, no. 10, pp , Oct [6] B. Hassibi H. Vikalo, On the sphere-decoding algorithm I. Expected complexity, IEEE Trans. Signal Process., vol. 53, no. 8, pp , Aug [7] B. Hassibi H. Vikalo, On the sphere-decoding algorithm II. Generalizations, second-order statistics, applications to communications, IEEE Trans. Signal Process., vol. 53, no. 8, pp , Aug [8] C. P. Schnorr M. Euchner, Lattice basis reduction: Improved practical algorithms solving subset sum problems, Math. Program., vol. 66, pp , [9] E. Agrell, T. Eriksson, A. Vardy, K. Zegar, Closet point search in lattices, IEEE Trans. Inf. Theory, vol. 48, no. 8, pp , Aug [10] A. M. Chan I. Lee, A new reduced-complexity sphere decoder for multiple antenna systems, in Proc. Int. Conf. Commun., Apr. 2002, pp [11] J. Jalden B. Ottersten, On the complexity of sphere decoding in digital communication, IEEE Trans. Signal Process., vol. 53, no. 4, pp , Apr [12] A. Li, W. Xu, Y. wang, Z. Zhou, J. Wang, A faster ML sphere decoder with competing branches, in Proc. IEEE VTC Conf., Jun. 2005, pp [13] W. Zhao G. B. Giannakis, Sphere decoding algorithms with improved radius search, IEEE Trans. Commun., vol. 53, no. 7, pp , Jul [14] W. Zhao G. B. Giannakis, Reduced complexity closest point decoding algorithms for rom lattices, IEEE Trans. Wireless Commun., vol. 5, no. 1, pp , Jan [15] Z. Guo P. Nilsson, Algorithm implementation of the K-best sphere decoding for MIMO detection, IEEE J. Sel. Areas Commun., vol. 24, no. 2, pp , Mar [16] R. Gowaikar B. Hassibi, Statistical pruning for near-maximum likelihood decoding, IEEE Trans. Signal Process., vol. 55, pp , Jun [17] B. Shim I. Kang, Sphere decoding with a probabilistic tree pruning, IEEE Trans. Signal Process., vol. 56, no. 10, pp , Oct [18] P. W. Wolniansky, G. J. Foschini, G. D. Golden, R. A. Valenzuela, V-BLAST: An architecture for realizaing very high data rates over the rich-scattering wireless channel, in Proc. URSI Int. Symp. Signals, Syst., Electron., Sep. 1998, pp [19] F. Jelinek, A fast sequential decoding algorithm using a stack, IBM J. Res. Develop., vol. 13, pp , Nov

14 CHOI et al.: LOW-COMPLEXITY DECODING VIA REDUCED DIMENSION ML SEARCH 1793 [20] W. Xu, Y. Wang, Z. Zhou, J. Wang, A computationally efficient exact ML sphere decoder, in Proc. IEEE Global Telecommun. Conf., Nov. 2004, pp [21] J. B. Anderson S. Mohan, Sequential coding algorithms: A survey cost analysis, IEEE Trans. Commun., vol. COM-32, no. 2, pp , Feb [22] M. Kokkonen K. Kalliojarvi, Soft-decision decoding of binary linear codes using the t-algorithm, in Proc. IEEE 8th Int. Symp. Personal, Indoor Mobile Radio Commun. (PIMRC), Finl, Sep. 1997, pp [23] W. K. Ma, T. N. Davidson, K. Wong, Z. Q. Luo, P. C. Ching, Quasi-maximum-likelihood multiuser detection using semi-definite relaxation with application to synchronous CDMA, IEEE Trans. Signal Process., vol. 50, no. 4, pp , Apr [24] A. D. Murugan, H. E. Gamal, M. O. Damen, G. Caire, A unified framework for tree search decoding: Rediscovering the sequential decoder, IEEE Trans. Inf. Theory, vol. 52, no. 3, pp , Mar [25] X. Li, H. C. Huang, A. Lozano, G. J. Foschini, Reduced-complexity detection algorithms for systems using multi-element arrays, in Proc. IEEE Global Telecommun. Conf., Nov. 2000, pp [26] W. Choi, R. Negi, J. Cioffi, Combined ML DFE decoding for the V-BLAST system, in Proc. IEEE ICC, Nov. 2000, pp [27] L. G. Barbero J. S. Thompson, Performance analysis of a fixedcomplexity sphere decoder in high-dimensional MIMO systems, in Proc. IEEE ICASSP, May 2006, pp [28] J. Jalden, L. G. Barbero, B. Ottersten, J. S. Thompson, Full diversity detection in MIMO systems with a fixed-complexity sphere decoder, in Proc. IEEE ICASSP, Apr. 2007, pp. III-49 III-52. [29] Y. Li Z. Luo, Parallel detection for V-BLAST system, in Proc. IEEE ICC, May 2002, pp [30] Z. Luo, M. Zhao, S. Liu, Y. Liu, Generalized parallel interference cancellation with near-optimal detection performance, IEEE Trans. Signal Process., vol. 56, no. 1, pp , Jan [31] D. W. Water J. R. Barry, The chase family of detection algorithms for multiple-input multiple-output channels, IEEE Trans. Signal Process., vol. 56, no. 2, pp , Feb [32] A. Elkhazin, K. N. Plataniotis, S. Pasupathy, Reduced-dimension MAP turbo-blast detection, IEEE Trans. Commun., vol. 54, no. 1, pp , Jan [33] E. G. Larsson J. Jalden, Fixed-complexity soft MIMO detection via partial marginalization, IEEE Trans. Signal Process., vol. 56, no. 8, pp , Aug [34] B. Hochwald S. T. Brink, Achieving near-capacity on a multipleantenna channel, IEEE Trans. Commun., vol. 51, no. 3, pp , Mar [35] J. Hagenauer C. Kuhn, The list-sequential (LISS) algorithm its application, IEEE Trans. Commun., vol. 55, no. 5, pp , May [36] S. Bittner, E. Zimmermann, W. Rave, G. Fettweis, List sequential MIMO detection: Noise bias term partial path augmentation, in Proc. IEEE ICC, Istanbul, Turkey, Jun [37] E. Biglieri, G. Taricco, A. Tulino, Decoding space-time codes with BLAST architectures, IEEE Trans. Signal Process., vol. 50, no. 10, pp , Oct [38] B. Hassibi, An efficient square-root algorithm for BLAST, in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Jun. 2000, pp [39] J. W. Choi, B. Shim, A. C. Singer, N. I. Cho, A low-complexity near-ml decoding technique via reduced dimension list stack algorithm, Proc. IEEE SAM 2008, pp , Jul [40] H. V. Poor S. Verdu, Probability of error in MMSE multiuser detection, IEEE Trans. Inf. Theory, vol. 43, no. 3, pp , May [41] P. Li, D. Paul, R. Narasimhan, J. Cioffi, On the distribution of SINR for the MMSE MIMO receiver performance analysis, IEEE Trans. Inf. Theory, vol. 52, no. 1, pp , Jan [42] Y. Jiang, X. Zheng, J. Li, Asymptotic performance analysis of V-BLAST, in Proc. IEEE Globecom 2005, Nov. 2005, pp [43] F. D. Neeser J. L. Massey, Proper complex rom processes with application to information theory, IEEE Trans. Inf. Theory, vol. 39, no. 4, pp , Jul [44] J. G. Proakis, Digital Communications, 4th ed. New York: McGraw Hill, [45] D. Tse P. Viswanath, Fundamentals of Wireless Communication. Cambridge, U.K.: Cambridge Univ. Press, Jun Won Choi received the B.S. M.S. degrees in electrical computer engineering from the Seoul National University (SNU), Seoul, Korea, in , respectively. He is currently pursuing the Ph.D. degree in electrical computer engineering at the University of Illinois at Urbana-Champaign, Urbana. His research interests include signal processing for wireless/underwater communications, statistical signal processing, low-power digital signal processing system design. Byonghyo Shim received the B.S. M.S. degrees in control instrumentation engineering (currently electrical engineering) from Seoul National University, Seoul, Korea, in , respectively, the M.S. degree in mathematics the Ph.D. degree in electrical computer engineering from the University of Illinois at Urbana-Champaign, Urbana, in , respectively. From , he was with the Department of Electronics Engineering at the Korean Air Force Academy as an Officer (First Lieutenant) an Academic Full-time Instructor. From 2005 to 2007, he was with the Qualcomm Inc., San Diego, CA as a Senior/Staff Engineer working on CDMA systems with the emphasis on the next-generation UMTS receiver design. In September 2007, he joined the School of Information Communication, Korea University, Seoul, Korea, as an Assistant Professor. His research interests include signal processing for communications, statistical signal processing, estimation detection, applied linear algebra, information theory. Andrew C. Singer received the S.B., S.M., Ph.D. degrees from the Massachusetts Institute of Technology, Cambridge, all in electrical engineering computer science, in 1990, 1992, 1996, respectively. From 1996 to 1998, he was a Research Scientist at Sers, a Lockheed Martin Company, Manchester, NH. Since 1998, he has been on the faculty at the University of Illinois at Urbana Champaign, Urbana, where he is currently a Willet Faculty Scholar Professor in the Department of Electrical Computer Engineering. He is also a Research Professor, affiliated with the Signal Processing Communications Groups in the Coordinated Science Laboratory. He is also the Director of the Technology Entrepreneur Center in the College of Engineering. In 2000, he cofounded Intersymbol Communications, Inc. (acquired by Finisar (FNSR) in 2007), a communications integrated circuit (IC) company specializing in ICs for the optical communications industry. His current research interests include statistical signal processing; wireless underwater acoustic communications; LIDAR signal processing; adaptive filtering machine learning; universal prediction, data compression, universal methods; nonlinear dynamics solitons; financial modeling. Nam Ik Cho received the B.S., M.S., Ph.D. degrees in control instrumentation engineering from Seoul National University, Seoul, Korea, in 1986, 1988, 1992, respectively. From 1991 to 1993, he was a Research Associate of the Engineering Research Center for Advanced Control Instrumentation, Seoul National University. From 1994 to 1998, he was with the University of Seoul, as an Assistant Professor of electrical engineering. He joined the School of Electrical Engineering, Seoul National University, in 1999, where he is currently a Professor. His research interests include speech, image, video signal processing, adaptive filtering.