ADVANCES IN MULTIUSER DETECTION

Transcription

1 ADVANCES IN MULTIUSER DETECTION Michael Honig Northwestern University A JOHN WILEY & SONS, INC., PUBLICATION

2

3 CHAPTER 3 ITERATIVE TECHNIQUES ALEX GRANT AND LARS RASMUSSEN

4 4 ITERATIVE TECHNIQUES 3.1 INTRODUCTION Theoretical analysis promises performance improvements for multiple access communications through the use of joint detection (in the case of uncoded transmission) or joint decoding (for encoded data). These gains in bit error probability and/or achievable reliable information transmission rate can be significant in non-orthogonal multiple-access channels. Except in a few special circumstances [1 3] however, the optimal joint detection/decoding problem is prohibitively complex [4, 5]. Typically the implementation complexity scales exponentially with the number of independent transmitters. This adds an additional layer of complexity, on top of that required for the optimal detection or decoding of single-user transmissions. The ensuing engineering challenge is therefore to find practical encoders and decoders yielding performance approaching the theoretical limits. Since the 1993 turbo revolution, modern coding practice has been dominated by iteratively decoded codes such as parallel [6] and serial [7] turbo codes, low density parity chec codes [8] and repeat accumulate codes [9]. The success of these codes is largely due to their iterative decoding algorithms, which approximate optimal decoding with manageable computational complexity. As a result, iterative processing has emerged as the framewor of choice for the design of near-optimal systems in a variety of communications scenarios. This chapter describes iterative processing techniques for both uncoded and coded multiple-access channels. For completeness and to introduce notation, we establish a system model in Section Linear multiuser filters are an important component of the methods to be discussed in this chapter. For reference, we give an overview of some well-nown linear filters in Section 3.1.2, with generalizations to accommodate non-uniform prior probabilities. The latter becomes important when the linear filters are used as components of iterative multiuser decoders. Section 3.2 describes iterative methods for the implementation of multiuser detectors. Both linear and non-linear iterations are considered. Linear iterative detectors are developed within the framewor of well-nown iterative methods for solving linear systems (whose exact solutions correspond to given linear filters, such as the decorrelator). This general setup reveals the commonality between many engineering approaches reported in the literature. It also allows the system designer to specify an arbitrary linear filter (e.g. decorrelator, or linear minimum mean squared error filter) as the goal of the iteration. Convergence analysis is reduced to determination of eigenvalues of certain iteration matrices. The gradient method

5 INTRODUCTION 5 for solution of linear systems emerges as the common underlying iterative method, and in particular the conjugate gradient method results in fast convergence, finite termination and a convenient interference cancellation structure. Section 3.3 describes iterative multiuser decoders, based on the turbo principle. We describe a general framewor, which iterates between linear multiuser filtering and singleuser decoding. Variance transfer methods are used for convergence analysis. The following notation will be used. Column vectors will be represented as lowercase bold symbols, x R n with real elements x i R, i = 1, 2,..., n. Similarly, matrices will be uppercase bold symbols, X R n m, with elements X ij R, i = 1, 2,..., n, j = 1, 2,..., m. The superscript t denotes matrix transpose, 1 denotes matrix inverse when applied to matrices, while X = diag(x 1, x 2,..., x n ) is a diagonal matrix with diagonal entries X ii = x i. Similarly, diag(x) is the diagonal matrix with the same principal diagonal as X. The standard inner product between two vectors is denoted (x, y) = x t y. For a function f : R n R, f/ x = ( f/ x 1,..., f/ x n ) t. The expectation operator is denoted E { }, and Cov {x, y} = E {(x E {x}) (y E {y}) t} denotes the covariance matrix of random vectors x and y. N (m, Σ) denotes a multivariate normal distribution with mean vector m and covariance matrix Σ System model In this chapter we consider a K user symbol-synchronous multiple access system with error control coding, transmitting over a common additive white Gaussian noise (AWGN) channel. We adopt the well-nown chip/symbol synchronous discrete time model [10], for notational and conceptual simplicity. We further assume real, binary modulation. Extentions of this model to asynchronous multipath fading channels with multiple antennae are relatively straightforward but notationally cumbersome. More detailed models, including the development from continuous time to discrete time can be found in [11, 12]. A bloc diagram of the system including the transmission side and the receiver front-end is shown in Figure 3.1. The source for user = 1, 2,..., K produces a sequence of R L information bits, b [i], i = 1, 2,..., R L. These sequences are mutually independent. Each user s sequence is encoded by a binary code, C, of rate R, passed through an interleaver, π, and modulated onto an antipodal, e.g. binary phase shift eying (BPSK)

6 6 ITERATIVE TECHNIQUES s 1 [j] a 1 Data Source b 1 [i] C 1 π 1 x 1 [j] s 1 [j] y 1 [i] s 2 [j] a 2 Data Source b 2 [i] C 2 π 2 x 2 [j] r[j] s 2 [j] y 2 [i] Multiuser Receiver n s K [j] a K Data Source b K [i] C K π K x K [j] Noise s K [j] y K [i] Figure 3.1. Coded multiple access system model. signal constellation to produce the interleaved and modulated code bit sequence x [j], j = 1, 2,..., L. With the assumption of chip synchronism, the output of the chip match filtered multiple access channel is characterized by a set of length N modulation, or spreading vectors s [i] = (s,1 [i], s,2 [i],..., s,n [i]) t, for user = 1, 2,..., K, at symbol interval i = 1, 2,..., L. The s,n [i] are referred to as chip amplitudes. With the further assumption of symbol synchronism, the length N received vector r[i] at symbol i can be written r[i] = K s [i]a x [i] + n[i] (3.1) =1 = S[i]Ax[i] + n[i] (3.2) where S[i] = (s 1 [i], s 2 [i],..., s K [i]) is a N K matrix which has as column the modulation vector s [i] for user, symbol i. The scalars a are the received user amplitudes, combining the effect of different transmit levels and any channel attenuation (for simplicity we assume that these amplitudes are fixed for the duration of the transmission). These are collected into a diagonal matrix A = diag(a 1, a 2,..., a K ). The vector x[i] = (x 1 [i], x 2 [i],..., x K [i]) t consists of the transmitted symbols at interval i, and n is a vector of zero-mean additive white Gaussian noise samples with E {n[i]n t [i]} = σ 2 I. For this real-valued channel, the noise variance is related to the underlying thermal noise density via σ 2 = N 0 /2. { We will assume binary unit norm modulation vectors, s [i] 1/ N, +1/ N and real amplitudes a R. This models direct-sequence code division multiple access, and we will use CDMA terminology throughout the chapter. Extension to complex values } N

7 INTRODUCTION 7 accommodates a wider range of linear multiple access channel including TDMA, FDMA, MC-CDMA, and OFDM. One useful analytical model for the modulation vectors is to let the s,m [i] be random, chosen uniformly i.i.d. from { 1/ N, +1/ N}. We shall refer to this model as random spreading. Otherwise, in certain circumstances, the modulation vectors for each user may be fixed for all time, s [i] = s. We shall refer to this as fixed spreading. Equation (3.2) is a chip-level discrete-time model. A symbol-level model is obtained via matched filtering of the modulating spreading sequences, s [i], for all and i. Specifically for symbol interval i, we get y[i] = S t [i]r[i] (3.3) = S t [i]s[i]ax[i] + S t [i]n[i] (3.4) = R[i]Ax[i] + z[i] (3.5) where R[i] = S t [i]s[i] is the correlation matrix 1 at interval i, and z[i] is a vector of colored Gaussian noise samples with E {z[i]z t [i]} = σ 2 R[i]. Note that according to our assumption of unit norm modulation vectors, R ii = 1 and R ij 1 for i j. The vector y[i] is referred to as the symbol matched filter output. The single user matched filter detector outputs ˆx MF [i] = sgn (y[i]), (3.6) i.e. independent hard decisions for each user, based on the symbol matched filter output for each user. The discrete-time models are developed for a given arbitrary time interval i. Throughout the chapter, the symbol interval index will only be included when conceptually required. For example, the symbol index is omitted for cases where an arbitrary symbol interval is considered Multiuser Detectors The minimum probability of error detector for linear multiple access channels is the maximum lielihood (ML) detector, considering all users jointly [13]. Optimal multiuser detection is however an NP-complete problem [4], where the brute-force implementation of the ML detector is in effect a Viterbi algorithm applied to the received matched-filtered signal. Such a system has an inherent complexity of the order O ( 2 K) for binary modulation. 1 R is also commonly denoted the symbol-level channel matrix.

8 8 ITERATIVE TECHNIQUES As alternatives to the ML detector, an abundance of sub-optimal reduced-complexity receiver structures exist in the literature. For an introduction, see [10, 12, 14]. Most of these structures are sub-optimal approximations to classic design criteria. Some approaches, however, satisfy relaxed optimality criteria. The decorrelator [15] and the linear minimum mean-squared error (LMMSE) detector [16 18] are two examples of linear detectors that are optimal with respect to modified criteria. Such linear detection schemes are however fundamentally limited in the spectral efficiency that they can offer in conjunction with independent decoding of each user s data [19, 20]. LMMSE filters, incorporating a-priori information on the data bits, have been described for iterative multiuser decoding in [21 24]. Similar filters have been derived based on probabilistic data association [25], the nonlinear MMSE criterion [26], and neural networs [27]. In the following subsections, the optimal multiuser detector, the decorrelator and a set of MMSE detectors are briefly described. Iterative implementations of the decorrelator and the linear MMSE detectors will subsequently be considered in Section 3.2 for uncoded transmission, and in Section 3.3 for coded transmission Optimal Multiuser Detectors The optimal multiuser detector first appeared in [28] and was subsequently developed in [13, 29, 30]. Minimum error probability optimal multiuser detector results from maximum a posteriori probability (MAP) decisions, ˆx MAP = arg max Pr(x r) (3.7) x { 1,1} K = arg max x { 1,1} K p(r x)pr(x). (3.8) Assuming that Pr(x) = 2 K for all data symbol vectors, we get the maximum-lielihood (ML) multiuser detector, ˆx ML = arg max p(r x) (3.9) x { 1,1} K = arg max exp ( 12 ) r SAx 2 (3.10) x { 1,1} K = arg max x { 1,1} K exp ( 12 (y RAx) R 1 (y RAx) t ). (3.11) In general, MAP and ML multiuser detectors suffer from complexity of the order O ( 2 K), rendering such approaches impractical for large K.

9 INTRODUCTION 9 Linear multiuser detectors such as the decorrelator or linear minimum mean-squared error filter can significantly reduce the complexity, while still improving the bit error rate of uncoded transmission as compared to the single-user matched filter (3.6) Decorrelator Detector The decorrelator approach to multiuser detection introduced in [15] is closely related to the zero-forcing equalizer for intersymbol interference (ISI) channels [31]. Suppose the correlation matrix R = S t S is invertible, and let ˆx DEC = ( S t S ) 1 S t r (3.12) = R 1 y (3.13) = Ax + R 1 z, (3.14) where E {R 1 zz t R 1 } = σ 2 R 1. Whereas the matched filter output y (3.6) consists of correlated data RAx and noise z, ˆx DEC consists of independent data Ax and noise R 1 z. The inverse correlation matrix R 1 can increase the effect of the noise when R is badly conditioned, leading to a trade-off between multiple access interference elimination and noise enhancement. The decorrelator is the maximum lielihood detector for two different modifications of the original detection problem (3.9). Firstly, it is the ML detector resulting from relaxing the integer constraints x { 1, 1} K to x R K. Secondly, the decorrelator does not require nowledge of the user amplitudes A, which is one implementation advantage of this detector. The decorrelator is the ML detector in the absence of nowledge of A. It provides an estimate of Ax rather than of x. Note that the amplitudes A in (3.14) do not affect hard decisions Linear Minimum Mean-Squared Error Detectors The LMMSE filter for multiuser detection [16] is closely related to the LMMSE equalizer for ISI channels [31]. Adaptive methods for LMMSE filtering were first suggested in [17, 18]. For jointly random vectors x and y, with x = E {x} ȳ = E {y} G = (Cov {y, y}) 1 Cov {y, x} the LMMSE estimate of x given y is [32, V.C.19] x + G t xy (y ȳ).

10 10 ITERATIVE TECHNIQUES For jointly Gaussian x, y this linear estimate in fact minimises mean squared error. The LMMSE estimate can be found for either the chip-level model (3.2) considering x, r or the symbol level model (3.5), considering x, y. ˆx r LMMSE = E {x} + G t xr (r r) (3.15) ˆx y LMMSE = E {x} + Gt xy (y ȳ). (3.16) Suppose the data symbols x { 1, 1} K are independent with and it can be shown that The corresponding LMMSE estimates are then E {x} = x, (3.17) Cov {x, x} = I diag ( x x t) = V. (3.18) ˆx r LMMSE = x + VAS t ( SAVAS t + σ 2 I ) 1 (r SA x) (3.19) ˆx y LMMSE = x + VAR ( RAVAR + σ 2 R ) 1 (y RA x) (3.20) = x + A ( 1 R + σ 2 (AVA) 1) 1 (y RA x). (3.21) In case E {x} = 0 we have Cov {x, x} = I, (3.19) and (3.21) simplify into the well-nown LMMSE estimates [16] 2 ˆx r LMMSE = AS t ( SA 2 S t + σ 2 I ) 1 r (3.22) ˆx y LMMSE = A 1 ( R + σ 2 A 2) 1 y. (3.23) Also if σ 2 is set to zero in (3.22) and (3.23), the filters simplify into the decorrelator filters in (3.12) and (3.13), respectively. Since for binary random variables, the mean and probability mass function are uniquely related, x represents the a-priori information available to the detector. Interference cancellation and iterative multiuser decoding are two examples where such prior information can be easily exploited. The case E {x} = 0 corresponds to uniform prior probabilities Per-User Linear Minimum Mean-Squared Error Detectors In some applications where prior information is available, it may still be desirable to ignore information 2 For the symbol-level estimate in (3.21) and (3.23), the scaling by A 1 can be ignored as subsequent hard decisions are not affected by scaling.

11 INTRODUCTION 11 pertaining to the user of interest. This is the case for joint iterative multiuser decoding, discussed in Section 3.3. In such cases, a different filter needs to be derived for each user. The filters presented here were first proposed in [21, 22] for iterative multiuser decoding. Assuming user is the user of interest, we set E {x } = 0 in (3.17) for derivation purposes, even though prior information for user may be available. The corresponding covariance matrix can be expressed as a function of x and V in (3.17) and (3.18), respectively, Cov {x, x} = V = V e e t x 2, (3.24) where e is a length K column vector with a one in position and zeros elsewhere. The per-user (PU) LMMSE estimates for user are now ˆx r PULMMSE, = a s t ( SAV AS t + σ 2 I ) 1 r s j a j x j (3.25) j ˆx y PULMMSE, = a s t S ( RAV AR + σ 2 R ) K 1 y e l R lj a j x j. (3.26) l=1 j Per-User Approximate Nonlinear MMSE Detector Suppose the multiple access interference (MAI) experienced by user is a multivariate Gaussian random vector, i.e., K r = s j a j x j + n (3.27) j=1 where = s a x + s j a j x j +n, (3.28) j } {{ } MAI s j a j x j N s j a j x j, s j s t ja 2 j(1 x 2 j) (3.29) j j j = N s j a j x j, SAW AS t, (3.30) j with W = V e e t ( ) 1 x 2. (3.31) This approach was suggested in [25] as an application of probabilistic data association to multiuser detection, and later derived based on approximations to the nonlinear MMSE

12 12 ITERATIVE TECHNIQUES (NMMSE) criterion in [26]. The principle has previously been proposed for feedbac equalization of ISI channels in [33]. From (3.30) it follows that r given x is a multivariate Gaussian random vector with distribution r N s a x + s j a j x j, SAW AS t + σ 2 I, (3.32) j and the corresponding conditional probability distribution function for r given x is thus proportional to p (r x ) exp x a s t ( SAW AS t + σ 2 I ) 1 r s j a j x j. (3.33) j Given these Gaussian assumptions, we can now find the approximate nonlinear MMSE (ANMMSE) )estimate as ˆx r ANMMSE, = tanh a s t ( SAW AS t + σ 2 I ) 1 r s j a j x j. (3.34) j The corresponding symbol-level estimate is determined as ˆx y ANMMSE, = tanh a s t S ( RAW AR + σ 2 R ) 1 y s t s j a j x j. (3.35) As the number of users K grows large, the matrix SAW AS t +σ 2 I in (3.34) converges to a diagonal matrix, SAW AS t + σ 2 I = j s j s t ja 2 j(1 x 2 j) + σ 2 I (3.36) j 1 a 2 N j(1 x 2 j) + σ 2 I. (3.37) A large-system (LS) approximation to (3.34), avoiding the matrix inversion is therefore determined as [26], ˆx r LSNMMSE, = tanh a s t 1 N j (r ) j s ja j x j j a2 j (1 x2 j ) + σ2. (3.38) Closely related schemes are found in [34] based on direct approximation of the NMMSE detector and in [27] based on neural networs arguments.

13 INTRODUCTION 13 The linear filter applied within the argument to the hyperbolic tangent function in (3.34) has been proposed in [23, 24] as the nominal LMMSE detector. The per-user nominal LMMSE (PUNLMMSE) estimate is found as ˆx PUNLMMSE, = a s t ( SAW AS t + σ 2 I ) 1 r s j a j x j, (3.39) j and the corresponding large-system PUNLMMSE (LSPUNLMMSE) approximation is found as a s t ˆx LSPUNLMMSE, = 1 N (r j s ja j x j ) j a2 j (1. (3.40) x2 j ) + σ2 The detector in (3.40) is in fact the matched filter detector proposed for iterative multiuser decoding in [35], exploiting prior information.

14 14 ITERATIVE TECHNIQUES 3.2 ITERATIVE JOINT DETECTION FOR UNCODED DATA Optimal multiuser detection is prohibitively complex for a large number of active users. Lower complexity solutions can be provided by iterative or multi-stage detection structures corresponding to serial or parallel interference cancellation. This class of detectors forms new decisions for each user by subtracting estimates of other user interference obtained from previous iterations. The original multistage detector [36, 37] is a parallel cancellation device while the first serial canceller was suggested in [38]. In both cases, data estimates of the interfering users are based on hard tentative decisions. As an alternative, linear multistage cancellation was suggested in [39, 40] for parallel and serial scheduling. Linear cancellation schemes were subsequently recognized as iterative methods for solving linear systems [41, 42]. Nonlinearities such as the hyperbolic tangent [27, 43, 44] and the hard limiter (also now as the clipper) [45 48] were later introduced at the output of each cancellation stage, as a compromise between hard decision (sign test) and linear soft decisions. In fact, the clipped soft decision interference canceller was shown in [47] to be an iterative solution to the original optimization problem with modified constraints. We shall pursue this idea further in Section For an overview of linear and nonlinear interference cancellation techniques, see [49]. In Section below the concepts of interference cancellation are introduced based on linear parallel processing. The principles are subsequently extended to arbitrary cancellation schedules and nonlinear tentative decision functions. In Section linear methods are discussed in detail, while nonlinear approaches are investigated as solutions to constrained optimization problems in Section Interference Cancellation In a communications system where performance is interference limited, interference cancellation is appealing as a strategy for improving performance. Consider the symbol level multiple access channel model (3.5). The matched filter output for user at an arbitrary

15 ITERATIVE JOINT DETECTION FOR UNCODED DATA 15 symbol interval is y = s t r = a x + s t s j a j x j + n j The term = a x + j R j a j x j }{{} multiple access interference j s t s j a j x j = j R j a x j is the multiple access interference (MAI) experienced by user and z = s t n is Gaussian noise, colored across users. Suppose that user possesses estimates of the interfering users symbols. Then an obvious approach for reducing multiple access interference is to subtract an estimate of the MAI from the received signal. Typically, this estimate will not be perfect, and there will be some residual MAI. This motivates an iterative cancellation approach for recursively improving symbol estimates and removing left-over interference. At each iteration, the cancelled signal is used to generate a (hopefully improved) estimate for user, which can +z then be used to cancel interference from other user s observations. Iterations will be denoted by an index n. Let the symbol estimate of user = 1, 2,..., K at iteration n = 0, 1, 2,... be ˆx (n). Given the estimates from iteration n, the updated estimate for user at iteration n + 1 is ˆx (n+1) = a 1 = a 1 st r j s j a j ˆx (n) j y j R j a j ˆx (n) j (3.41) (3.42) Equation (3.41) describes an interference canceller woring at the chip rate, while (3.42) operates at the symbol rate. ˆx (1) One possible choice for the initial estimate for each user is ˆx (0) = 0. This leads to = a 1 y, i.e. is equivalent to starting with an initial estimate a 1 y. Typically, the matched filter output is used as the initial estimate. Note that if ˆx (n) = x the cancellation is perfect, and ˆx (n+1) = x + z a

16 16 ITERATIVE TECHNIQUES which is an interference-free AWGN channel with signal-to-noise ratio a 2 /σ Schedules for Iterative Cancellation The iteration (3.41) describes a parallel update of each user s estimate. At each new iteration n + 1, all users cancel the constributions of other user estimates from the previous iteration n. Writing (3.42) for all users in vector form, ˆx (n+1) = A 1 (y (R I) Aˆx (n)). (3.43) This is shown schematically, from the perspective of user in Figure 3.2. Another possibility is that the user s estimates are updated serially, for example in order of user number = 1, 2,.... In this case, at iteration n + 1 a particular user has available the estimates of lower index users from the same iteration n + 1 (these have already been computed) and higher index users from the previous iteration n. The resulting serial update, in both chip-level (3.44) and symbol-level (3.45) form is 1 K = a 1 r s j a j ˆx (n+1) j ˆx (n+1) = a 1 st j=1 1 y j=1 R j a j ˆx (n+1) j j=+1 K j=+1 s j a j ˆx (n) j R j a j ˆx (n) j (3.44). (3.45) Since this approach uses updated estimates as soon as they are available, one might expect serial cancellation to perform better than parallel cancellation. Let L be the strictly lower triangular part of R = L + L t + I. Then the vector form of (3.45) is This is shown schematically in Figure 3.3. ˆx (n+1) = A 1 (y LAˆx (n+1) L t Aˆx (n)). (3.46) Both the parallel and serial cancellation structures pass two signals from iteration to iteration, namely the user estimates ˆx (n), and the received vector y Implementation via Residual Error Update The output of the chip-level parallel interference canceller (3.41) can be re-written as K ˆx (n+1) = a 1 st r s j a j ˆx (n) j + s a ˆx (n) = ˆx (n) = ˆx (n) + a 1 st j=1 r K j=1 s j a j ˆx (n) j. (3.47) + a 1 st η (n) (3.48)

17 ITERATIVE JOINT DETECTION FOR UNCODED DATA 17 y 1 R 1 a 1 ˆx (1) 1 R 1 a 1 y j R j a j ˆx (1) j R j a j y a 1 ˆx (1) a 1 ˆx (2) y y K R K a K ˆx (1) K R K a K Stage 0 Stage 1 Figure 3.2. Parallel interference cancellation. R 1 a 1 ˆx (1) 1 R 1 a 1 ˆx (2) 1 R j a j ˆx (1) j R j a j ˆx (2) j y a 1 ˆx (1) a 1 ˆx (2) y y K R K a K ˆx (1) K R K a K Stage 0 Stage 1 Figure 3.3. Serial interference cancellation.

18 18 ITERATIVE TECHNIQUES where η (n) = r K j=1 s j a j ˆx (n) j (3.49) is the residual error, or noise hypothesis remaining after iteration n. With perfect cancellation, it consists only of the thermal noise, i.e. η (n) = n. Similarly, the output of the chip-level serial canceller (3.44) can we re-written as where η (n) ˆx (n+1) = ˆx (n) 1 = r s j a j ˆx (n+1) j j=1 is the residual error seen by user after iteration n. Let η (n) = s a ( ˆx (n 1) + a 1 st η (n), (3.50) K j= s j a j ˆx (n) j (3.51) ) ˆx (n). (3.52) Then, the residual error for the parallel canceller can be written recursively as η (n) = η (n 1) + K j=1 η (n) j, (3.53) while the residual error for user in the serial canceller can be written recursively as η (n) η (n 1) K + η (n) K = 1 = (3.54) η (n) 1 + η(n+1) 1 > 1. Rather than passing the received vector from stage to stage, as was described in the previous section, iterative interference cancellation can be accomplished by passing the continually updated residual error. Summarizing this development, parallel interference cancellation is implemented via (3.48), (3.52) and (3.53) as shown in Algorithm 1 below. Serial cancellation differs only in the frequency with which the residual error is updated. The serial cancellation scheme is implemented via (3.48), (3.52) and (3.54), which is summarized in Algorithm 2. The residual error approach has some desirable features for implementation. Cancellation schemes can be constructed based on a basic interference cancellation module, shown in Figure 3.4. The module for user taes as inputs the previous estimate for user, ˆx (n 1), and the most recent residual error, η (n 1). Note that in the parallel case, the user index can be ignored as the residual error at iteration n is the same for all users η (n 1) = η (n 1).

19 ITERATIVE JOINT DETECTION FOR UNCODED DATA 19 Initialize ˆx (0) = 0, η (0) = r for n = 1, 2,... do for = 1, 2,..., K do ˆx (n) = ˆx (n 1) + a 1 ( η (n) = s a ˆx (n 1) st η (n 1) ˆx (n) ) end for end for η (n) = η (n 1) + K j=1 η (n) j. Algorithm 1: Parallel Cancellation. Initialise ˆx (0) = 0, η (0) 1 = r for n = 1, 2,... do for = 1, 2,..., K do ˆx (n) = ˆx (n 1) + a 1 η (n) = s a ( ˆx (n 1) st η (n 1) ˆx (n) ) end for end for η (n) = η (n 1) K + η (n) K = 1 η (n) 1 + η(n+1) 1 > 1. Algorithm 2: Serial Cancellation. (3.55)

20 20 ITERATIVE TECHNIQUES From these inputs the module determines as outputs an updated estimate for user, ˆx (n), according to (3.48) or (3.50) and a corresponding update, η (n), to the residual error according to (3.52). ˆx (n 1) ˆx (n) η (n 1) η (n) s t a a s Figure 3.4. Interference cancellation module. Using the module as a basic building bloc, arbitrary cancellation schemes can be systematically constructed. In Figure 3.5. a parallel cancellation scheme is shown, while a serial cancellation scheme is depicted in Figure η (n 1) ˆx (n 1) 1 ˆx (n) 1 ˆx (n+1) 1 η (n) 1 η (n) η (n+1) 1 η (n+1) ˆx (n 1) 2 ˆx (n) 2 η (n) 2 ˆx (n+1) 2 η (n+1) 2 ˆx (n 1) 3 ˆx (n) 3 η (n) 3 ˆx (n+1) 3 η (n+1) 3 ˆx (n 1) 4 ˆx (n) 4 η (n) 4 ˆx (n+1) 4 η (n+1) 4 Figure 3.5. Parallel cancellation Tentative Decision Functions The cancellation strategies so far have been linear. Both the symbol estimates ˆx (n) and noise hypotheses η (n) undergo recursive linear updates. The transmitted symbols belong to a discrete set, assumed to be { 1, +1} in

21 21 ITERATIVE JOINT DETECTION FOR UNCODED DATA (n 1) (n) η1 (n+1) η1 (n 1) η1 (n) x 1 (n+1) x 1 x 1 (n) (n+1) η 1 (n 1) η 1 (n) x 2 (n+1) x 2 x 2 (n) (n 1) η2 (n+1) η 2 (n 1) (n) η2 (n) x 3 (n+1) x 3 x 3 (n) (n 1) η3 (n+1) η 3 (n 1) (n) η3 x 4 x 4 (n) (n 1) η4 η 3 (n+1) (n) x 4 η 2 (n+1) η 4 (n) η4 η 4 Figure 3.6. Serial cancellation. (n) this chapter, yet the estimates x could be any real number. This could have a negative (n) effect on the cancellation process. For example, the situation could arise where x 1, resulting in cancellation of an impossibly large interference estimate. Conversely, in certain circumstances, the cancellation process might be accelerated by maing hard decisions. This motivates the use of a non-linear tentative decision function ζ : R 7 [ 1, +1], which aims to limit the output of the canceler to lie in the interval [ 1, 1], (n+1) x (n) (n) 1 t = ζ x + a. s η (3.56) The corresponding non-linear interference cancellation module is shown in Figure 3.7. (n 1) x ζ (n 1) (n) x (n) η η st a a s Figure 3.7. Interference cancellation module with tentative decision function. D R A F T September 15, 2006, 12:14pm D R A F T

22 22 ITERATIVE TECHNIQUES A wide range of tentative decision functions have been described in the literature. A linear decision function ζ(x) = x results in linear cancellation which is discussed in detail in Section Non-linear choices of the decision function obviously result in non-linear cancellation, discussed in detail in Section Three prominent examples of non-linear tentative decision functions are the hard limiter, or clipper, [47, 48], 1 x 1 ζ(x) = clip(x) x x < 1 (3.57) 1 x 1 the soft limiter [25, 27, 34, 43, 44], ζ(x) = tanh(x) (3.58) and the hard decision [36, 38], ζ = sgn(x). (3.59) Figure 3.8. shows each of these non-linear decision functions. The effect of each of these non-linearities is to restrict the output of the canceller to be in the range [ 1, 1]. This is investigated in more detail in Section tanh(x) -0.5 clip(x) -1 sgn(x) Figure 3.8. Tentative decision functions Linear Methods Implementation of linear detectors such as the decorrelator and the LMMSE filter require matrix inversion, which has cubic complexity in the number of users. This may be a problem if the modulating waveform set changes for every symbol interval. A variety of iterative techniques are available for matrix inversion 3, reducing the required complexity. 3 See [50 54] for comprehensive treatments of such techniques.

23 ITERATIVE JOINT DETECTION FOR UNCODED DATA 23 Techniques such as series expansion (polynomial detectors) [55 58], iterations (interference cancellation) [39 42, 56, 57, 59 66] and gradient decent algorithms [56, 57, 59, 61, 66] have been widely applied for linear multiuser detection. Here, we investigate these techniques and show they can all be implemented as interference cancellation. Taylor series expansion techniques for multiuser detection was first proposed in [55] as the polynomial expansion detector, leading to a multistage implementation. The concept was further developed in [56, 66] where series expansion was related to parallel interference cancellation. Detectors with a finite number stages were designed based on minimizing the output MSE. A more comprehensive presentation of polynomial expansion detectors is found in [57, 58], where the Cayley Hamilton theorem is applied to obtain the exact matrix inverse in a finite number of stages. The Taylor and Cayley Hamilton series expansions are presented in detail in Section below. In [41] interference cancellation schemes were recognized as iterative solution methods for solving linear systems. This relationship was further developed in [56, 60 62], where weighted (or partial) cancellation suggested in [67] was recognized as relaxation techniques for the well-nown Jacobi and Gauss-Seidel iterations. A comprehensive study of iterative solution methods as interference cancellation schemes is found in [42] where convergence aspects were investigated in detail. Optimization of relaxation factors have been investigated in [64, 65] based on large system analysis bounds on extreme eigenvalues of the correlation matrix. Iterative solution methods are defined in Section , with separate subsections dedicated to the Jacobi and Gauss-Seidel iterations in Sections and , respectively. Relaxed iterations are also presented and shown to improve convergence behavior. Gradient decent schemes, such as the steepest decent and conjugate gradient algorithms, were first applied for multiuser detection in [59]. Independently, partial cancellation with changing relaxation factors was recognized as the steepest decent algorithm in [56, 66]. A method for determining the relaxation factors (step sizes) for reaching the minimum MSE possible in a given finite number of steps was developed. Cancellation schemes based on the steepest decent algorithm and the conjugate gradient algorithm were considered in [61], and similar to [59] the conjugate gradient approach was found to provide superior performance. Section includes a comprehensive presentation of gradient decent algorithms applied for multiuser detection. In particular, the conjugate gradient algorithm is found to compare favorably with parallel cancellation in terms of performance versus complexity.

24 24 ITERATIVE TECHNIQUES Solutions to Linear Systems The output of either the decorrelator (3.13) or the LMMSE (3.23) filter (represented as ˆx when the filter is left unspecified) can be written as the solution to a least-squares optimization problem, namely ˆx = arg min x R K Mx y 2 2 (3.60) where M = R for the decorrelator, M = RA for the normalized decorrelator, and M = R + σ 2 A 2 for the LMMSE. In Section 3.2.3, several useful non-linear receiver structures will result by introducing additional constraints to the optimization problem (3.60). The solution to the unconstrained optimization problem (3.60) is however obtained via solution of a system of linear equations, Mˆx = y. (3.61) Efficient solution of (possibly large) linear systems such as (3.61) has been the subject of much study in numerical linear algebra. This viewpoint will prove useful in the design of low-complexity approximations to the decorrelator and LMMSE filters Direct Solution Direct solution of a system of linear equations can be accomplished using Gaussian elimination followed by bac-substitution. In the case of symmetric M (which is true for both the decorrelator and LMMSE), this is equivalent to Cholesy factorization of the matrix M into a lower triangular matrix F, M = FF t (3.62) followed by forward and bacward substitution, 4 Fz = y F tˆx = z forward substitution bacward substitution In the general case, Cholesy factorization is O ( K 3 /3 ), and each substitution step is O ( K 2 /2 ). Useful references for these methods are [50, 51]. If the matrix M is band-diagonal, i.e. M ij = 0 for i j > b, where the integer b K is the semibandwidth, then the Cholesy decomposition is O ( K(b 2 + 3b) ). Liewise, the complexity of the substitution steps decreases to O (Kb) Thus the direct solution approach involves one relatively complex matrix decomposition, followed by two substitution steps. 4 Consider FF tˆx = y with z = F tˆx.

25 ITERATIVE JOINT DETECTION FOR UNCODED DATA Series Expansions One strategy for the design of multistage receiver structures is to develop series expansions for M 1, i.e. find coefficients c n such that M 1 = n c n M n. (3.63) Series expansions motivate a multistage structure shown in Figure 3.9. Such series expansion y M M M c 0 c 1 c 2 Figure 3.9. Multistage receiver structure motivated by series expansion. detectors were first proposed in [55] and developed further in [56 58]. A series with K terms involves calculation of M K, which is O ( K 3). For this approach to be attractive from an implementation point of view, truncation of the series to n K must yield a sufficiently accurate approximation, or the desired level of performance. Let p M (λ) = K ( 1) K n c K n λ n = det(m λi) = 0 (3.64) n=0 be the characteristic equation of the matrix M. The coefficients c n, n = 0, 1,..., K in (3.64) are the elementary symmetric functions of the eigenvalues λ 1 λ 2 λ K of M [52, p. 41], namely c n (λ 1,..., λ K ) = 1 i 1 i n j=1 n λ ij. (3.65) The c n are also given as the sum of the ( K n) different n n principal minors of M [52, Theorem ]. In particular, c 0 = 1, c 1 = tr M and c K = det M. The Cayley Hamilton theorem [52, p. 86] says that a matrix satisfies its own characteristic equation. Theorem 1 (Cayley Hamilton). p M (M) = K ( 1) K n c K n M n = 0 (3.66) n=0

26 26 ITERATIVE TECHNIQUES The Cayley Hamilton theorem provides a way to write any power of M as a linear combination of at most K + 1 terms from M n, n = 0, 1,..., K. In particular, M 1 = 1 ( 1) K det(m) K ( 1) K n c K n M n 1 (3.67) n=1 The series (3.67) describes a K-stage multistage implementation. Computation of all the coefficients c n however is just as complex as matrix inversion in the first place. important point is that there does exist a particular choice of coefficients c n such that the finite power series (3.67) implements matrix inversion exactly. An alternative expansion is provided by the following Taylor series, (I + X) 1 = The ( X) n, (3.68) which is convergent if the spectral radius of X satisfies ρ (X) < 1 (recall that the spectral radius of a matrix is the absolute value of its largest eigenvalue). Setting X = M I in (3.68) results in M 1 = which is convergent if ρ (M) < 2. n=0 ( 1) n (M I) n, (3.69) n=0 The first order truncation of this series for M = R results in the approximate decorrelator [10, Section 5.4] and [68], ˆx (1) DEC = (2I R)y (3.70) = y (R I)y }{{} Interference Estimate (3.71) Equation (3.71) reveals a parallel interference cancellation structure (see Equation (3.74) below). The second term in (3.71) can be regarded as an estimate of the multiple-access interference. In the absence of noise, y = Rx = x + (R I)x and the approximate decorrelator can be seen to be approximating the MAI term (R I)x by (R I)y, which corresponds to using the matched filter outputs as direct estimates of the interfering symbols. Higher order truncations of the Taylor series result in the following n-th order approximation to the decorrelator [55]. ˆx (n) DEC = y + (I R)y + (I R)2 y + + (I R) n y (3.72) = y (R I)ˆx (n 1) DEC (3.73)

27 ITERATIVE JOINT DETECTION FOR UNCODED DATA 27 which is nothing more than parallel interference cancellation. This is apparent, by writing out the approximation from the perspective of user as (omitting the subscript DEC for clarity) ˆx (n) = y }{{} User matched filter output R ˆx (n 1) }{{} Interference estimate from previous stage. (3.74) Figure 3.10.(a) shows stage n of (3.73). This coincides with structure shown in Figure 3.2. for A = I. Note that the decorrelator (which does not require nowledge of A) actually estimates Ax, as compared to the normalized decorrelator, which estimates x. Later we shall see how to obtain the structure of Figure 3.2. via iterative implementation of the normalized decorrelator. In Section , the same result will be derived from the viewpoint of iterative solution methods. In particular, (3.72) is nown as the Jacobi method for iterative solution of a linear system. The per-iteration performance and convergence characteristics of (3.73) will be evaluated in Section Similar to (3.71), a n-th order Taylor series approximation may be obtained from the LMMSE filter, ˆx (n) MMSE = y (R + A 2 σ 2 I)ˆx (n 1) MMSE (3.75) = y (R I)ˆx (n 1) MMSE A 2 (n 1) σ 2ˆx MMSE (3.76) which retains the same parallel interference cancellation structure as (3.71), with the extra cancellation of A 2 σ 2ˆx (n 1) MMSE. Since A is diagonal, computation of A 2 is easy. Stage n of the Taylor series implementation of the LMMSE is shown in Figure 3.10.(b). Truncated Taylor series approximation of the LMMSE filter is therefore only marginally more complex than for the decorrelator filter, although it does require nowledge of the signal-to-noise ratio A 2 /σ 2. Alternatively, (3.73) and (3.75) can be implemented using the chip-level residual error approach with the interference cancellation module shown in Figure In contrast to the Cayley Hamilton approach (3.67), the Taylor series does not require the computation of any series coefficients (all coefficients are 1). However this comes at a cost. Firstly, the Taylor series requires an infinite number of terms for exact solution. Secondly, the series is only convergent for ρ (M) < Iterative Solution Methods Motivated by (3.71) and (3.75), it is interesting to consider other iterative approaches for the approximation of solutions to linear systems.

28 28 ITERATIVE TECHNIQUES ˆx (n 1) 1 R1 ˆx (n 1) 1 σ 2 a R1 ˆx (n 1) j Rj ˆx (n 1) j σ 2 a 2 j + Rj ˆx (n) ˆx (n 1) σ 2 a 2 ˆx (n) y y ˆx (n 1) K RK ˆx (n 1) K σ 2 a 2 K + RK (a) Decorrelator. (b) LMMSE. Figure Taylor series implementation of the decorrelator and LMMSE filters. ˆx (n 1) α ˆx (n) η (n 1) η (n) s t s Figure Interference cancellation module for decorrelator, α = 1 and LMMSE, α = 1 σ 2 /a 2.

29 ITERATIVE JOINT DETECTION FOR UNCODED DATA 29 This is a well-trodden area of linear algebra and indeed, it is possible to devise many different iterative approaches for solution of linear systems, [53]. The main idea is to devise an easily computable iteration that converges rapidly to the required solution. A wide class of iterations can be defined based on a linear splitting of the matrix M into two according to M = M 1 M 2. With such a splitting, (3.61) can be re-written as (M 1 M 2 )ˆx = y, resulting in the following fixed point equation, M 1ˆx = y + M 2ˆx. (3.77) This motivates an iteration of the form M 1 x (n+1) = y + M 2 x (n). (3.78) This is a stationary iteration, meaning that the same operation is performed each iteration (M 1 and M 2 do not depend on the iteration number n). There are two main requirements. 1. The matrix M 1 should be chosen such that it is easy to solve systems of the form M 1 x = z. Obvious candidates for M 1 would be triangular or diagonal matrices. 2. The matrices M 1 and M 2 should be chosen such that (3.78) converges to the solution of the original system. Furthermore, (3.78) should converge as quicly as possible, preferably in much less than K iterations. In order to investigate the convergence properties of (3.78), it is necessary to define an appropriate notion of convergence. Preferably, the convergence would be analyzed in terms of the bit error rate, however this is not tractable. Instead, one possibility is to consider a norm of the error at each iteration. Let ˆx = M 1 y be the desired solution, and let x (n) be the output of the n-th iteration of (3.78), where x (0) is defined as the initial input to the iteration. Let e (n) = ˆx x (n) be the associated error vector. Substitution of x (n) = ˆx e (n) into (3.78) results in e (n) = M 1 1 M 2e (n 1) = (M 1 1 M 2) n e (0). (3.79) Suppose is a vector norm (with associated induced matrix norm). According to the defining properties of vector and matrix norms, e (n) = (M 1 1 M 2) n e (0) (M 1 1 M 2) n e (0). (3.80)

30 30 ITERATIVE TECHNIQUES Now according to [52, Corollary ] matrix norms are in the limit dominated by the spectral radius, and this yields the following result. lim n Xn = ρ (X) n (3.81) Theorem 2. A necessary and sufficient condition for convergence of (3.78) in any norm is ρ (M 1 1 M 2) < 1. (3.82) Typically, it is the Euclidean norm that is of interest. In this case, since M 1 1 M 2 is symmetric, (M 1 1 M 2) n 1/n 2 = ρ (M 1 1 M 2) (3.83) which means that the total squared error decreases geometrically with rate ρ (M 1 1 M 2). For this reason, mae the following definitions. Definition 1 (Iteration Matrix and Convergence Factor). For a given linear decomposition M = M 1 M 2, define the iteration matrix B = M 1 1 M 2. The quantity B n is the convergence factor for n iterations and ρ (B) is the asymptotic convergence factor. There are several design goals for accelerating convergence of iterative methods. It is advantageous to choose an initial guess x (0) close to the desired solution (yet with low implementation complexity). For a fixed number of iterations, one could minimize the convergence factor. It remains to find suitable choices of M 1 and M Jacobi Iteration The iteration (3.78) relies on being able to solve systems of the form M 1 x = z. This can be performed with very low complexity if M 1 is a diagonal matrix, since inversion of a diagonal matrix requires only element-wise inversion of the diagonal elements. One possible convenient choice is M 1 = I and M 2 = I M, in which case the iteration (3.78) becomes x (n+1) = y (M I)x (n) (3.84) which is simply the n-th order Taylor series (3.69). The corresponding asymptotic convergence factor is ρ (M I), which yields the same convergence criterion given above, namely ρ (M) < 2. Another possibility for diagonal M 1 is M 1 = D = diag(m), the diagonal matrix with the same diagonal elements as M. More generally, a design parameter ω could be

31 ITERATIVE JOINT DETECTION FOR UNCODED DATA 31 introduced, M 1 = ωd (3.85) M 2 = ωd M (3.86) Let x (0) = y then (3.78) becomes x (n+1) = D 1 ω ( y (M ωd)x (n)). (3.87) This is in fact identical to a series expansion of M 1 obtained via the n-th order Taylor series expansion of (ωd + M) 1, rather than (I + M) 1 as described above. In either case, the resulting receiver structure is multistage parallel interference cancellation. With ω = 1 and unit energy modulation sequences, diag R = I the resulting Jacobi iteration for the decorrelator is the same as (3.73). Theorem 3. The Taylor series iterative implementation of the decorrelator is convergent if and only if ρ (R) < 2. (3.88) Since the spectral radius of R depends upon the choice of spreading sequences, the convergence of the Taylor series expansion is not assured. However, by choosing an appropriate ω, this situation can be rectified. Theorem 4. The Jacobi implementation of the decorrelator with M 1 = ωi is convergent for any ω > 0 such that ρ (R) < 2ω. For ω = 1, the LMMSE filter (with unit energy modulation sequences), has diag(r + A 2 /σ 2 ) = I + A 2 σ 2 and the resulting Jacobi iteration (with x (0) = y) is x (n+1) = ( I + A 2 σ 2) 1 ( y (R I)x (n)). (3.89) Comparing to (3.73), the only difference is a per-user signal-to-noise ratio scaling each iteration. Theorem 5. The Jacobi iterative implementation of the LMMSE filter (3.89) is convergent if and only if ( (I ρ J,MMSE = ρ + A 2 σ 2) ) 1 (I R) < 1 (3.90) The following theorem gives simpler bounds on convergence for the Jacobi LMMSE.

32 32 ITERATIVE TECHNIQUES Theorem 6. The Jacobi iterative implementation of the LMMSE filter (3.89) is convergent if ρ (R I) < 1 + γ 1 max (3.91) where γ max = max A 2 /σ2. The iteration (3.89) is not convergent if ρ (R I) > 1 + γ 1 min (3.92) where γ min = min A 2 /σ2. For users with equal powers, (3.91) is also necessary for convergence. Proof. Convergence occurs if ρ J,MMSE < 1. Now since both I + A 2 σ 2 and I R are symmetric, ρ J,MMSE = ( I + A 2 σ 2) 1 (I R) 2 (3.93) ρ ( (I + A 2 σ 2) 1 ) 2 I R 2 (3.94) = ρ ( (I + A 2 σ 2) 1 ) ρ (I R) (3.95) = max A 2 A 2 The result (3.91) follows from the definition of γ max. + σ2 ρ (R I). (3.96) The negative result (3.92) follows using similar steps, noting that for diagonal D and symmetric R, ρ (D 1 (I R)) > min D 1 ρ (R I). Comparing Theorems 3 and 5, the Jacobi implementation of the LMMSE admits a wider radius of convergence, increasing that of the decorrelator by at least the inverse of the best user s SNR (Theorem 6). Results from random maxtrix theory can be used to gain insight for large systems with random spreading. Of particular interest is the behaviour of the extremal eigenvalues. Lemma 1. Let λ min and λ max be the smallest and largest eigenvalue of R = S t S where the elements of S are chosen i.i.d. zero mean, variance 1/N and finite higher moments. Then as K, N such that K/N α < 1, λ min ( α 1 ) 2 λ max ( α + 1 ) 2 1. Henceforth, by large systems, we mean systems satisfying the assumptions of Lemma

33 ITERATIVE JOINT DETECTION FOR UNCODED DATA 33 Using Lemma 1 and Theorem 3 it is straighforward to show that for large systems, parallel interference cancellation converges for only lightly loaded systems [69]. Theorem 7. For large systems, the Taylor series implementation of the decorrelator is convergent only for ( ) 2 α < The corresponding asymptotic convergence factor is α + 2 α Gauss-Seidel Iteration Another choice of splitting for which the iteration (3.78) can be performed with low complexity is if M 1 is a triangular matrix. In which case can be solved for x (n+1) by bac-substitution. M 1 x (n+1) = y + M 2 x (n) (3.97) One possible choice for a triangular matrix is M 1 = D + L, where D = diag(m) and L is the strictly lower-triangular part of M = D + L + L t. More generally, one could consider M 1 = D/ω + L, where ω is a design parameter. Since ω = 1 recovers the former choice, it is preferable to proceed with an arbitrary ω. These choices M 1 = 1 ω D + L (3.98) M 2 = 1 ω ω D Lt (3.99) result in the following iteration (with x (0) = y) ( ) ( ) 1 1 ω ω D + L x (n+1) = y + ω D Lt x (n). (3.100) Re-arranging to get x (n+1) on the left hand side gives ( x (n+1) = D 1 ω y L t x (n) Lx (n+1)) + (1 ω)x (n). (3.101) Writing out the iteration from the perspective of user reveals a successive cancellation structure (noting L t ij = L ij for symmetric M), x (n+1) = ω D y i> L i x (n) i }{{} Cancel higher index users from previous stage j< L j x (n+1) i }{{} Cancel lower index users from current stage + (1 ω)x (n). (3.102)

34 34 ITERATIVE TECHNIQUES This is the structure of Figure The triangular structure of L allows cancellation of interference estimates for each user as soon as they are available. This is in contrast to the Jacobi iteration (3.74) where each users interference estimate is not cancelled until the next iteration. For ω = 1, (3.102) is nown as the Gauss-Seidel iterative method. The parameter ω is a relaxation parameter, and in general the method is nown as successive relaxation. Equation (3.102) shows how for ω 1, each successive estimate is a weighted sum of the previous estimate, and the most up-to-date interference cancelled estimate. Since successive relaxation uses interference estimates as soon as they are available, it could be expected that this method has superior convergence properties compared to the Jacobi iteration. Indeed, this is the case. Theorem 8. Successive relaxation (3.101) is convergent for symmetric positive definite M and ω (0, 2). Serial cancellation is convergent for any choice of 0 < ω < 2, while parallel cancellation can also be made convergent with appropriate choice of ω. Thus the design problem is to select the relaxation parameter ω to accelerate convergence Descent Algorithms For symmetric positive definite M, define the vector norm x M 2 1 = M 1 2 x 2 = x t M 1 x, (3.103) i.e. the norm derived from the inner product (, ) M 1. Define f (x) = 1 2 Mx y M 1 2 (3.104) Then an alternative to the least-squares minimization (3.60), which results in the same solution is ˆx = arg min x R K f(x). (3.105) The equivalence of solutions can be seen by noting that Mx = y is the unique stationary point for both problems, since the gradient of f(x) is equal to ( f(x) (x), f(x),..., f(x) ) t = Mx y. (3.106) x 1 x 2 x K This reveals that the gradient at each step is equal to the error vector e = Mx y, and the corresponding unique stationary point is Mx = y.