Bayesian ML Sequence Detection for ISI Channels

Bayesian ML Sequence Detection for ISI Cannels Jill K. Nelson Department of Electrical and Computer Engineering George Mason University Fairfax, VA 030 Email: jnelson@gmu.edu Andrew C. Singer Department of Electrical and Computer Engineering University of Illinois Urbana, IL 680 Email: acsinger@uiuc.edu Abstract We propose a Bayesian tecnique for blind detection of coded data transmitted over a dispersive cannel. Te Bayesian maximum likeliood sequence detector views te cannel taps as stocastic quantities drawn from a known distribution and computes te probability of any transmitted sequence by averaging over te tap values. Te resulting pat metric requires memory of all previous symbols, and ence a tree-based algoritm is employed to find te most likely transmitted sequence. Simulation results sow tat te Bayesian detector can acieve bit error rates witin /4 db of te conventional known-cannel maximum likeliood (ML) sequence detector. I. INTRODUCTION A variety of communication cannels suffer from intersymbol interference, or ISI. Interference among transmitted symbols may be introduced by transmit or receive filtering, bandwidt limitations at te transmitter or receiver, or troug multi-pat propagation. Significant researc as been devoted to developing effective equalization metods for dispersive cannels. Common tecniques range from te bit-error rate (BER) optimal maximum a posteriori probability (MAP) detector to te low-complexity minimum mean-square error (MMSE) linear equalizer. In many communication systems, te parameters of te dispersive cannel may be affected by a variety of pysical and environmental conditions tat are not known a priori. As a result, receivers are often faced wit te callenge of detecting data transmitted over cannels tat are partially, or peraps entirely, unknown. Te beavior of unknown cannels may also vary wit time, tereby presenting an even more difficult problem to te receiver. One approac to equalization of an unknown cannel is to first generate an estimate of te cannel, often by transmitting training symbols, and use te cannel estimate to detect transmitted information []. Suc an approac is less desirable for time-varying cannels, since te estimate must be updated or regenerated as te cannel evolves. Alteatively, te cannel and data may be estimated concurrently to identify te most likely combination of cannel and associated data [] [5]. Since joint estimation of te cannel and data can be proibitively complex, approximate solutions to tis problem are often implemented in practice. Sesadri [4] proposes suc a tecnique based on te Viterbi algoritm. Rater tan anding te trellis on wic te Viterbi algoritm is performed, Sesadri s sceme allows an increase in te number of surviving sequences retained in te Viterbi trellis. Because te unknown cannel parameters used in te likeliood computation are estimated based on te data along te pat in question, Sesadri s algoritm falls in a class of algoritms known as per-survivor processing (PSP) tecniques [6]. Researcers ave also considered blind detection approaces, i.e., recovering te transmitted data witout generating an licit estimate of te cannel. Blind detection scemes, suc as te well-known constant modulus (CM) algoritm [7], typically employ estimates of iger order statistics (HOS) of te cannel to generate data estimates [8]. We propose an alteative blind detection sceme in wic, rater tan assuming a deterministic cannel response, we view te cannel taps as stocastic quantities drawn from a known probability distribution. Using Bayesian tecniques, we average over te unknown quantities to compute te probability of a sequence of transmitted symbols. A stacklike tree searc algoritm is used to identify te most likely sequence based on te Bayesian probability metric. Wile Bayesian approaces to blind detection ave been proposed in te past, tey ave been used witin different detection structures, suc as particle filtering and iterative Viterbi-based scemes [9], [0]. Reader and Cowley considered an approac similar to ours, employing Bayesian tecniques to generate sequence likelioods []. Rater tan using a tree-based algoritm, owever, te autors used te Viterbi algoritm, wic requires eiter a continually anding trellis or te sacrifice of information in prior data []. In te following section, we describe te system model under consideration. Te tree-based algoritm used for Bayesian ML sequence detection is described in Section 3. We ten derive te Bayesian pat metric in Section 4 and lore its empirical performance in Section 5. Conclusions are drawn in Section 6. II. SYSTEM MODEL We consider a system model, pictured in Figure, in wic te information bits are encoded using a rate R = r error-control code prior to transmission over te cannel. We assume te use of a random binary tree code in wic te bits on eac branc are independent of eac oter and of tose on oter brances. In practice, a convolutional code wit a large constraint lengt provides a reasonable approximation. Information bits are transmitted in blocks of lengt N, denoted

b N, yielding blocks of coded bits of lengt rn, denoted x rn, at te output of te encoder. Te encoded bits are transmitted over a lengt-l ISI cannel [n] wit additive wite Gaussian noise (AWGN) w[n] of variance σ = N 0 /. Te effect of te cannel can be modeled as L z n = k x n k + w n, () k=0 were z n denotes te cannel output at time n, and = [ 0... L ] denote te taps of te ISI cannel. For simplicity, we consider transmission of only binary pase sift keying (BPSK) encoded data. Transmission of data from larger symbol constellations is a straigtforward extension of te work presented ere. Eac block of received samples z rn serves as input to te detection and decoding block, wic uses an algoritm based on te stack decoder to estimate te data sequence b N by navigating te tree generated by te combined code and cannel. We assume tat te receiver as knowledge of te error-control code used at te transmitter, te lengt of te cannel, te variance of te AWGN, and te initial state of te encoder. b[n] Rate-R Encoder x[n] ISI Cannel w[n] z[n] Bayesian Detector and Decoder ˆb[n] Fig.. System model for Bayesian ML sequence detection. BPSK-encoded data is passed troug a rate R = error-control encoder, transmitted over r a cannel wit unknown taps, and processed by a detection and decoding block tat employs a stack-like tree searc algoritm. III. ML SEQUENCE DETECTION VIA TREE SEARCH Wen te cannel (or an estimate of te cannel) is known, ML sequence detection can be performed efficiently via te Viterbi algoritm, wic navigates a trellis and stores only te most likely pat to eac state [3]. Wen te cannel is unknown, owever, te conditional likeliood of any new branc in a pat, i.e., p ( b n b n ), is a function of all previous bits in te pat. Computation of te most likely sequence, ten, must be represented as navigation troug a tree rater tan troug a trellis and typically requires significantly iger computation per bit estimated. We use a sequential algoritm, adopted from te decoding of convolutional codes, to determine an estimate of te most likely sequence. Te stack algoritm, wic was originally developed as a metod for decoding tree codes and convolutional codes [4], navigates a tree (defined by te code) in searc of te pat wit te largest likeliood, or metric. A set of possible pats and teir associated metrics are stored in a list (or stack), and at eac iteration, te pat wit te largest metric is extended. Te stack algoritm terminates wen te most likely pat in te stack reaces a leaf of te tree, i.e., wen a complete pat containing N information bits as te largest metric of any pat in te stack. Note tat te various pats contained in te stack at a particular time are not all of te same lengt. Te algoritm may extend a pat of lengt k in one iteration and in te next iteration find te most likely pat to be one of lengt j for some j k. As a result, te pat metric must include a bias term to compensate for te differences in likeliood tat accompany a difference in pat lengt. As will be discussed, te bias term as a significant impact on te performance of te proposed detector. Te stack-like algoritm employed in te Bayesian sequential detection sceme searces troug a tree in a manner similar to tat used by te conventional stack decoder. Te tree navigated, owever, is generated by te combination of te code and te response of te cannel. Additionally, te metric used to order te pats is derived from te proposed Bayesian structure and assumes a lack of cannel knowledge. IV. DERIVATION OF THE BAYESIAN PATH METRIC To implement a stack-like algoritm we must derive an ression for te probability of a lengt-n sequence of bits, i.e., we must compute p(b n z rn, C (k) ). Here, C (k) denotes te code known at te current iteration k, i.e., te bits along all brances of te code tree tat ave been lored by te detector tus far. Since we assume tat eac block of data bits as been encoded using a random binary tree code, knowledge of te code bits on lored pats give no information about unknown code bits in unlored sections of te tree and ence simplifies derivation of te Bayesian pat metric. Note tat te cannel taps do not licitly appear in te probability ression p(b n z rn, C (k) ), as tey are viewed as stocastic quantities. We assume a known distribution for te parameters and average over te distribution to generate te probability of any sequence. A general form for te probability of a lengt-n sequence b n is derived as p(b n z rn, C (k) ) = p(b n, z rn, C (k) )d () = bn,, C(k) )p(b n,, C(k) ), d C(k) ) P(b n = ) C(k) b n ),, C (k) )p()d, were te tird equality follows from te joint independence of te information bits b n, te cannel taps, and te code known so far C (k). Eliminating C (k) ) since it is equal for all pats, we can simplify te Bayesian metric, denoted by m B (b n ), to te form m B (b n ) = P(b n ) b n,, C (k) )p()d. (3) We would like to find an integrable form for b n,, C (k) )p(), te integrand in (3). Because te cannel is unknown, and because te ISI cannel introduces memory in te received data, received samples beyond te pat of interest, i.e. z rn +, are dependent upon tose before. However, in order to develop a closed-form metric of practical complexity, we assume te two subsets of

te received sequence are nearly independent and approximate te overall likeliood by bn,, C(k) ) (4) p(z b n,, C (k) ) + b n,, C (k) ) = p(z bn,, C(k) ) + ). A similar approximation is used in te derivation of te pat metric for bot stack decoding of convolutional codes [5] and stack-based equalization of known ISI cannels [6]. Te form of p(z b n,, C (k) ) is simply te pat metric used for te conventional Viterbi algoritm and is given by were and p(z b n,, C (k) ) = i= p(z i b n,, C (k) ) (5) = σ (R zz [0] + T r zx T R )} (πσ ) /, R k zz[0] = r k zx = k zi, (6) i= k z i x i i L+, (7) i= k R k = ( ) ( ) x i i L+ x i T i L+. (8) i= To compute + ), we average over all possible lengtr(n n) sequences of code bits x rn +, yielding ( ) r(n n) + ) = +,x). x,} r(n n) (9) Suc a sum over r(n n) terms is impractically complex to include in te pat metric. To simplify te ression, we assume independence among te future received samples and approximate te joint likeliood by + ) rn i=+ p(z i ). (0) Suc an approximation can be justified using Massey s approac of appending a random tail to te lengt-n codeword [7]. Given te cannel tap values and te L code bits x i i L+, received sample z i as a Gaussian distribution wit mean E[z i,x i i L+ ] = T x i i L+ () and variance σ. Averaging over all lengt-l binary sequences, we obtain rn i=+ rn p(z i ) = () i=+ L πσ x,} L } σ (z i T x), wic appears in te Bayesian metric for any pat segment of lengt-n information bits. To generate an ression of reasonable complexity for practical implementation, we approximate te Gaussian mixture in () by a single Gaussian, i.e. were and p(z i ) πσ z } σz (z i µ z ), (3) µ z = E[z i ] = 0, (4) σ z = E[z i ] = σ + T. (5) We assume te cannel energy E = T, or an estimate tereof, is known at te receiver. Substituting te approximations in (4), (0), and (3) into te general Bayesian metric ression in (3) yields te result ( ) n ( ) / ( ) r(n n)/ m B (b n ) = πσ π(σ (6) + E) σ (R zz[0] + T r zx T R) } rn (σ zi p()d. + E) i=+ Given te quadratic onential form of (6), we coose te conjugate Gaussian prior over to allow for closed-form integration. To tis end, we assume te cannel taps are drawn from a pdf of te form f() = (π) L K ( ( µ ) T K ( µ ) )}, (7) were µ denotes te vector mean of te cannel taps, K denotes te covariance matrix of te cannel taps, and K denotes te determinant of K. Substituting (7) into (6) and integrating yields ) n (σ ( ) + E) r(n N) / M (8) ( m B (b n ) = were R zz [0] σ (σ ) µt K µ ( r zx σ + K µ ) T M M = K rn i=+ z i } (σ + E) ( r )} zx σ + K µ, ( ) R σ + K. (9) For te simulations and discussion presented ere, we consider scenarios in wic te cannel taps ave mean zero, equal variance σ, and are jointly independent. Under tese assumptions, te distribution over te cannel taps simplifies to } p() = (πσ T )L/ σ. (0)

Tese assumptions on te prior over te cannel taps can be interpreted as a minimum cannel knowledge scenario, since no prior information about (nonzero) tap values nor knowledge of cannel tap correlations is used. In cases in wic te system designer as some prior knowledge of te cannel tap values and/or correlation between te taps, tis information could be incorporated into te pdf over via a nonzero mean vector and non-diagonal correlation matrix. Using te prior over given in (0), te Bayesian pat metric for an ISI cannel simplifies to m B (b n ( ) = ) ( ) () σ L (σ + E) r(n N) R n σ σ + I / σ R zz[0] σ + ( R σ 4rT zx σ + I ) } σ r zx } rn (σ zi. + E) i=+ Te simplified pat metric takes a quadratic onential form. Note tat, wile te metric is a function of all received samples, future samples (tose associated wit information bits beyond te pat segment of interest) are normalized by te larger variance σ + E, indicating tat less information about tese samples is available since te associated transmitted bits are unknown. As simulation results will sow, te approximation yielding tis bias term underestimates te likeliood of future symbols, tereby favoring longer pats. Peraps te most interesting term in te simplified metric ( R σ + I σ ) r zx term in te onential. Tis is te r T zx term takes a form very similar to least squares estimation of te cannel taps and is te element of te metric ( in wic ) we see implicit leaing of te cannel. Te r T R zx σ + I σ r zx term is also te element of te metric tat proibits implementation of te algoritm via trellis searc. Because te algoritm does not generate licit cannel estimates but instead uses information about eac pat and te received data to generate a metric, te number of states increases by te size of te symbol alpabet (a factor of for BPSK) for eac increase in pat lengt. (If te cannel were known, te number of states would remain constant at L after L stages.) Te stack-like algoritm avoids an exaustive searc of te onentially increasing number of states by extending only te pat tat appears most likely rater tan every possible pat. V. SIMULATED PERFORMANCE OF THE BAYESIAN DETECTOR To evaluate te performance of te proposed Bayesian ML detection sceme, we simulate te detector for a tree-tap cannel wit impulse response [n] = 0.407δ[n + ] + 0.85δ[n] + 0.407δ[n ]. () Te performance of te Bayesian detector is simulated for block lengts of N = 0, 00, and 000 data bits. For comparison, we also simulate te performance of conventional ML sequence detection (via te Viterbi algoritm) over te cannel [n], using blocks of lengt 000. For simulation of bot te conventional ML detector and te Bayesian detection sceme, te information bits are encoded using a rate R = / (r = ) convolutional code wit generator matrix G(x) = [x + x + x + ]. (3) Figure sows te BER of te Bayesian and standard ML detectors over a range of SNR values. Te results reveal tat, for SNR between 4 and 9 db, te Bayesian ML detection sceme wit N = 000 acieves performance witin approximately / db of tat of te conventional known-cannel ML detector. It is also clear from Figure tat te BER performance of Bayesian ML sequence detection improves significantly as block size increases. Tis beavior can be lained by noting tat te Bayesian algoritm is using te received data, as well as te assumed transmitted data along eac pat, to lea te values of te cannel taps. Larger block lengts allow te detector more information from wic to lea te response of te cannel. Tis beavior is akin to tat of least-squares estimation, wic generates improved parameter estimates as te amount of data available increases. Note, owever, tat te performance for N = 00 is nearly equivalent to tat for N = 000, indicating tat te Bayesian detector can acieve low BER wit relatively small block lengt. Bit Error Rate 0 0 0 3 0 4 MLSD Known Cannel Bayesian Detector, N=000 Bayesian Detector, N=00 Bayesian Detector, N=0 0 5 4 5 6 7 8 9 E b /N 0 (db) Fig.. Simulated performance of Bayesian ML sequence detector and conventional (Viterbi) ML sequence detector. Te cannel response is [n] = 0.407δ[n + ] + 0.85δ[n] + 0.407δ[n ], and te convolutional code as generator matrix G(x) = [x + x + x + ]. Additional simulations ave been conducted to lore te performance of te Bayesian sequential detector wen te cannel is essentially known. To generate suc a scenario, we let µ = and σ. Results reveal tat, even wen te cannel is known, te Bayesian detector suffers a performance loss of nearly / db wit respect to te Viterbi algoritm,

indicating tat te loss is likely due to a suboptimal bias term rater tan a lack of cannel knowledge. To address tis problem, we consider an alteative pat metric derivation tat avoids te assumption of independence among future received elements and te Gaussian approximation of p(z i ). Rater tan computing te pat likeliood conditioned on te entire received sequence, we consider te probability of te current pat based only on te received sequence tus far, i.e., p(b n z, C(k) ) (4) = P(bn ) p(z p(z C(k) ) bn,, C(k) )p()d (/) n σ L R σ + I = σ / p(z C(k) )(πσ ) / R zz [0] σ + ( R σ 4rT zx σ + I ) } σ r zx, were te prior distribution p() is given by (0). Te quantity p(z C (k) ) can be computed using an exteal stack wic, rater tan extending only one pat at eac stage, extends all pats in te stack. We can ten iteratively generate te conditional likelioods for all n according to p(z C(k) ) (5) = (/) n p(z x, )p()d. x C(k) Note tat te integral in (5) takes te same form as tat in (4). Akin to te work of Kun and Hagenauer [8], we can limit te size of te exteal stack to S entries (S N) to maintain a resonable complexity level. At te end of eac iteration, only te S largest stack entries are retained. Simulation results using te alteative metric given in (4) are presented in Figure 3. In addition to te conventional ML and Bayesian detectors, we also consider te performance of te PSP tecnique presented in [4], wic navigates a trellis rater tan a tree and uses LMS to update a cannel estimate for eac surviving pat. For all detectors, information blocks of lengt N = 500 were encoded using a convolutional code wit generator matrix G(x) = [ + x + x ]. Te exteal stack for te Bayesian detector was limited to 6 entries, and one pat to eac state was retained in te LMS-Viterbi sceme. At low SNR, all tree detectors sow similar bit error rates. As SNR increases, owever, te Viterbi-based sceme reaces an error floor, wile te Bayesian detector closely tracks te performance of te conventional ML detector. Te error floor results from pat eliminations made early in te data block wen te LMS cannel estimate as not converged. Because te Bayesian algoritm navigates a tree rater tan a trellis, it is able to postpone eliminating pat segments until it as more cannel information, tereby significantly improving bit error rate for te early bits in eac block. Bit Error Rate 0 0 3 0 4 MLSD Known Cannel LMS Viterbi Algoritm Modified Bayesian Detector 4 5 6 7 8 9 E b /N 0 (db) Fig. 3. Simulated performance of te conventional ML sequence detector, te Bayesian detector wit modified metric, and a Viterbi-based PSP algoritm employing LMS. Te cannel as impulse response [n] = 0.407δ[n+]+ 0.85δ[n] + 0.407δ[n ], and te convolutional code as generator matrix G(x) = [ + x + x ]. VI. CONCLUSION We ave proposed a novel Bayesian tecnique for detecting data transmitted over an unknown ISI cannel. To allow implicit leaing of te cannel as te detector progresses, te most likely transmitted sequence is identified via a tree searc algoritm. Witout requiring training data, te Bayesian detector can acieve BER witin /4 db of ML sequence detection for a known cannel. Tese promising results warrant furter study of te Bayesian detector, peraps caracterizing its computational complexity (a Pareto random variable for standard stack decoders) and loring extensions to equalization of time-varying cannels. REFERENCES [] S. Haykin, Adaptive Filter Teory. Upper Saddle River, NJ: Prentice- Hall, 996. [] R. Amara and S. Marcos, A blind network of extended Kalman filters for nonstationary cannel equalization, in Proc. IEEE Int. Conf. on Acoustics, Speec, and Signal Processing, May 00, pp. 7 0. [3] L. M. Davis, I. B. Collings, and P. Hoeer, Joint MAP equalization and cannel estimation for frequency-selective and frequency-flat fast-fading cannels, IEEE Transactions on Communications, vol. 49, no., pp. 06 4, December 00. [4] N. Sesadri, Joint data and cannel estimation using blind trellis searc tecniques, IEEE Transactions on Communications, vol. 4, no. -4, pp. 000 0, February-April 994. [5] R. Iltis, J. Synk, and K. Giridar, Bayesian algoritms for blind equalization using parallel adaptive filtering, IEEE Transactions on Communications, vol. 4, no. -4, pp. 07 03, February-April 994. [6] R. Raeli, A. Polydoros, and T. Cing-Kae, Per-survivor processing: A general approac to MLSE in uncertain environments, IEEE Transactions on Communications, vol. 43, pp. 354 364, February-April 995. [7] C. R. Jonson, P. Scniter, T. J. Endres, J. D. Bem, D. R. Brown, and R. A. Casas, Blind equalization using te constant modulus criterion: A review, Proceedings of te IEEE, vol. 86, no. 0, pp. 97 950, October 998. [8] Z. Ding and Y. Li, Blind Equalization and Identification. New York, NY: Marcel Dekker, 00.

[9] J. Miguez and P. M. Djuric, Blind equalization by sequential importance sampling, in Proc. IEEE Int. Symposium on Circuits and Systems, May 00, pp. I 845 I 848. [0] X. Wang and R. Cen, Blind turbo equalization in Gaussian and impulsive noise, IEEE Transactions on Veicular Tecnology, vol. 50, no. 4, pp. 09 05, July 00. [] D. Reader and W. Cowley, Blind maximum likeliood sequence detection, in Proceedings of te IEEE Inteational Conference on Acoustics, Speec, and Signal Processing, vol. 5, 996, pp. 694 697. [] D. Reader, Blind maximum likeliood sequence detection over fast fading communication cannels, P.D. dissertation, University of Sout Australia, August 996. [3] G. D. Foey, Maximum-likeliood sequence estimation of digital sequences in te presence of intersymbol interference, IEEE Transactions on Information Teory, vol. 8, pp. 363 378, May 97. [4] R. E. Blaut, Algebraic Codes for Data Transmission. New York, NY: Cambridge University Press, 003. [5] A. J. Viterbi and J. K. Omura, Principles of Digital Communication and Coding. New York, NY: McGraw-Hill, 979. [6] F. Xiong, A. Zerik, and E. Swedyk, Sequential sequence estimation for cannels wit intersymbol interference of finite or infinite lengt, IEEE Transactions on Communications, vol. 38, no. 6, pp. 795 804, June 990. [7] J. L. Massey, Variable-lengt codes and te fano metric, IEEE Transactions on Information Teory, vol. 8, no., pp. 96 98, January 97. [8] C. Kun and J. Hagenauer, 8-PSK turbo equalization wit te listsequential (LISS) algoritm, in Inteational Symposium on Information Teory, June 004, p. 555.