On a Problem of Massey

Transcription

1 On a Problem of Massey Jon Hamkins Jet Propulsion Laboratory California Institute of Technology Pasadena, CA 909, USA Jon.Hamkins@jpl.nasa.gov Abstract In 976, Massey introduced a method to compute the confidence interval for the frame error rate of a coded communications system based on the simulation of just a few frame errors []. He commented that his approach did not apply to bit error rate confidence intervals, because bit errors are not independent in a coded system. In this paper, we show how to overcome the limitation Massey recognized, and present a method to compute a confidence interval of the bit error rate of a coded communications system from a simulation of bit and frame error events. The proposed interval may be easily computed from the first and second sample moments of the number of bits errors per frame. I. ITRODUCTIO In 976, Massey had a contract with ASA [] to help find a channel code appropriate for the International Ultraviolet Explorer (IUE), a joint space mission between ASA, the European Space Agency, and the UK Space Research Council. The mission did in fact use a convolutional code and went on to great success, returning over 00,000 images that formed a heavily used astronomy database and spawned nearly 4,000 peer-reviewed astronomy papers. IUE decided it wanted a constraint-length 24, rate /2 convolutional code for use with sequential decoding. See, e.g., [2] for a discussion of sequential decoding of convolutional codes. This mission was just prior to the era in which much shorter constraint length codes decoded with the Viterbi algorithm began to become the norm. Indeed, it stands out that at the time Massey referred to constraint length 24 as rather short []! Massey proceeded to analyze and simulate virtually every binary (24,/2) convolutional code that had been proposed up to that time. He considered ten codes in all, including ones designed by himself, Costello, Johannesson, Bahl, Jelenik, Bussgang, Lin, and Lyne see the references of [] for full details on the codes. With the large constraint length of 24 and short frame length of 256 bits, terminating the trellis reduced the code rate noticeably unfortunately, this was before the invention of the tail-biting method which would have neatly avoided the problem (see, e.g., [3]) and so Massey also considered various partial termination schemes. Given the computers of the day, Massey s simulations were limited to decoding, for each code, 0,000 frames at one SR. Four top candidate codes were each simulated an additional This work was performed at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the ational Aeronautics and Space Administration. 40,000 frames. His simulations produced 0 to 5 frame errors for each code. With this hard-fought-for but meager amount of data, Massey used confidence intervals to make conclusions about the relative merit of the various codes. In Massey s words [], The very small number of frame errors, between 0 and 5 inclusive for the best and the worst codes, makes it essential to consider the statistical significance of the frame error probabilities. II. COFIDECE ITERVAL FOR THE FRAME ERROR RATE For an excellent modern discussion of the computation of confidence intervals, including a computation based on using the exact (binomial) distribution for the number of simulated frame errors, as well as Gaussian and Poisson approximations, see [4]. In the following, suppose that X frame errors are observed in a simulation of decoded frames on a memoryless channel, so that ˆp X/ is the observed frame error rate (FER). Let p denote the true FER. A. ormal approximation One simple, popular approach to confidence intervals is to note that by the Central Limit Theorem (CLT), as, the probability density function of ˆp approaches that of a Gaussian random variable with mean p and variance p( p)/ [5]. The mean and variance may be estimated by ˆp and ˆp( ˆp)/, respectively, which leads to the ( α)-confidence interval where (ˆp a, ˆp + a) () ˆp( ˆp) a = Φ ( α/2) (2) and Φ(x) = 2π x e t2 /2 dt is the cumulative distribution function of a zero-mean, unit-variance Gaussian random variable. For a 95% confidence interval, Φ (0.975).96. Massey points out that when X is small, ˆp( ˆp)/ is not a good estimate of the variance of X []. In fact, no matter how large is, if X < 4 the interval includes a negative range!

2 B. Massey s approach: Poisson approximation Massey introduced what is now a standard method to overcome some of the limitations of the Gaussian approximation above, by noting that when and p, X is approximately Poisson distributed, with mean λ and variance λ, where λ p. Using the Poisson probability mass function f X (i) = λi i! e λ leads to the ( α)-confidence interval (λ L, λ H ) when X = x, given by the solution to i=x x i=0 λ i L i! e λ L = α 2 λ i H i! e λ H = α 2 The confidence interval for ˆp = x/, then, is (λ L /, λ H /). This solution works adequately for x < 5 [], and for x > 0 it is nearly identical to the Gaussian approximation discussed above [4]. III. COFIDECE ITERVALS FOR THE BIT ERROR RATE OF A CODED SYSTEM We turn now to the main topic and novel contribution of this paper. We desire to determine the confidence interval for the bit error rate (BER) of a coded communications system, based on simulations of the decoder. A. Massey s identification of the problem Massey recognized that while the procedure in section II-B is useful for computing FER confidence intervals, it cannot be used in the same way to compute BER confidence intervals []: It probably should be pointed out that, although 256 information bits are decoded in each frame so that there are 256 times as many bit decoding decisions as frame decoding decisions, one cannot assert greater statistical confidence in the observed decoding bit error probability than in the observed frame error probability. The reason of course is that the decodings of bits within a frame are highly dependent so that one has no more independent bit decoding decisions from which to infer probabilities than one has independent frame decoding decisions. In other words, counting x bit errors in total simulated bits and applying the formulas (3) and (4) would result in an inappropriately narrow confidence interval for the BER. Much of the literature on confidence intervals for BER treats the case in which bit errors are independent events (e.g., [6], [7]) and MATLAB s function to compute the confidence interval for BER, BERCOFIT(), also assumes this [8]. While there has been some work on BER confidence intervals on channels with memory [9], to our knowledge the computation of confidence intervals for a block-coded system has not been presented. (3) (4) B. BER confidence interval for a coded system Suppose frames of a binary (n, k) code are simulated, and for the sake of analysis, suppose that the decoder is required to output an estimate of the information bits in each frame, whether it successfully completes decoding the frame or not. Let B i be a random variable representing the number of bit errors, B i k, in the i th information block at the output of the decoder. Then B,..., B is a set of i.i.d. random variables. Let µ E[B i ] (5) σ 2 var[b i ] = E[B 2 i ] µ 2 (6) Typically, the distribution of B i is unknown. For modern iteratively decoded channel codes such as low-density paritycheck codes, the distribution of B i may depend on SR of the simulation, or details of the decoder, even when conditioned on the event that a frame error has occurred. For example, at low SR when frame errors are dominated by the decoder s failure to converge, many bit errors may occur in each frame error, while at high SR where the decoder performance is limited by the code s minimum distance or trapping sets, only a handful of bit errors might typically occur in each frame error (and of course, the FER itself is lower). Let p b denote the true BER. The number of bits simulated is k, so the observed BER is given by ˆp b = k B i As, by the CLT we have ( µ ˆp b k, σ 2 ) k 2 For large we may estimate the fist and second moments of B i by their sample first and second moments: µ ˆµ B i = kˆp b (7) ( ) σ 2 ˆσ 2 Bi 2 ˆµ 2 (8) Thus, a ( α)-confidence interval for the BER can be given by (ˆp b a, ˆp b + a ), with a = ˆσ k Φ ( α/2) (9) = ( ) ( ) 2 k Bi 2 B i Φ ( α/2) (0) A simulation would normally record only i B i; by also recording one extra quantity, i B2 i, the confidence interval in (0) may be computed. These two partial sums may be augmented with each new simulated frame, so that the entire sequence B, B 2,... need not be stored. Thus, the confidence interval remains easy to compute.

3 C. BER confidence interval when few frame errors are simulated As with the Gaussian-approximation for the FER confidence interval, the accuracy of the interval in (0) depends on the accuracy of the approximation in (8). Even with very large ensuring the accuracy of the CLT approximation for ˆp b if only a few B i are greater than zero, we won t have an accurate estimate of the variance of ˆp b. For the FER, X > 0 is sufficient; for the BER, even more are needed. What can be said when only a few frame errors have been collected? Since bit errors occur in bunches, not singly, neither the individual bit errors nor the bit errors per frame, B i, are binomial or Poisson distributed, and Massey s approach cannot be directly applied. When frame i is in error, B i k, so that ˆp k ˆp b ˆp () which loosely bounds the confidence interval for BER as (λ L /(k), λ H /), where λ L and λ H are given in (3) and (4). IV. GUIDELIE FOR SIMULATIO LEGTH A simulation which collects X 385 frame errors will be able to estimate the FER to within an error of 0%, with 95% confidence, because from (2), a ˆp = Φ (0.975) ˆp ˆp( ˆp) < Φ (0.975) ˆp < < 0. (2) where we have used ˆp <, Φ (0.975) <.96 and ˆp 385/. In general, when the length of the ( α)- confidence interval for the FER is desired to be shorter than plus or minus 00β% from the estimate, the simulation should be run until X (Φ ( α/2)/β) 2. Such a guideline is useful for the communications engineer to know how long to run a simulation to get a desired accuracy for the FER. We present now an analogous guideline for how long a simulation of a coded system should be run to get a good estimate of the BER. Let ( ) 2 / X B i Bi 2 (3) Theorem : If X > (Φ ( α/2)/β) 2, then the error for the BER estimate, with ( α)-confidence, is less than 00β% of the observed BER, ˆp b. Proof: From (0), we have a < ( ) k Bi 2 Φ ( α/2) (4) β kφ ( α/2) B i Φ ( α/2) (5) = β ˆp b (6) TABLE I MIIMUM X (FER) OR X (BER) FOR A SIMULATIO TO ACHIEVE A GIVE LEVEL OF ACCURACY AT A GIVE LEVEL OF COFIDECE. Error in X or X, at Confidence: FER or BER 90% 95% 99% % % % % % % 6 27 X = umber of frame errors X = Given by (3) Thus, with 95% confidence the BER is within plus or minus 0% of of the simulated BER when X 385. Table I summarizes the minimum X or X a simulation must reach in order to achieve a given accuracy at a given confidence level, for the FER or BER, respectively, using the Gaussian approximation to the interval discussed in the preceding sections. V. EXAMPLES A. Constant number of bit errors per frame Suppose the coded system is such that whenever a frame error is made, the decoded frame contains exactly b bit errors, where b is a constant. In this case, ˆp b = k B i = Bi 2 = bi {frame i in error} = bˆp k (7) bi {frame i in error} = bˆp (8) b 2 I {frame i in error} = b 2 ˆp (9) where I is the indicator function and, as before, ˆp is the observed FER. Plugging into (0), we have a = k b2 ˆp b 2 ˆp 2 Φ ( α/2) (20) = b ˆp( ˆp) Φ ( α/2) (2) k = b k a (22) where a is given in (2), and so the BER confidence interval is (ˆp b a, ˆp b + a ) = b k (ˆp a, ˆp + a) (23) That is, the BER confidence interval is exactly b/k times the FER confidence interval in (2), as expected. This means that on a log plot of BER and FER, the length of the confidence intervals will be the same. This is an extreme case. Typically, there is some variation in the number of bits-in-error in the simulated frames-in-error. In those cases, the length of the BER confidence interval

4 Error Rate FER BER E b / 0, db Fig.. Performance of the CCSDS k = 784, r = /2 turbo code. Error Rate nd simulated frame error Simulated FER Simulated BER 5 th 4 th 3 rd 95% confidence interval FER 95% confidence interval BER 95% confidence interval, assuming independent bit errors 0 9 Probability E b / 0 =.7 db E b / 0 = db E b / 0 = 0 db umber of simulated frames Fig % confidence intervals for FER and BER at E b / 0 =.7 db, as a simulation progressed umber of bit errors in frame Fig. 2. Observed distribution of B i > 0 at various E b / 0. is strictly greater than that of the FER confidence interval, reflecting the uncertainty both in the number of frame errors and in the number of bits in error within frames containing errors. B. A turbo code Among the turbo codes which have been standardized for space communications [0], we consider the one with input length k = 784 and code rate r = /2. The performance of the code is shown in Fig., where it can be seen that an error floor begins at just below FER= 0 5. A decoder was simulated at E b / 0 = 0 db, db, and.7 db, and the observed distribution of B i > 0 is shown in dark gray, light gray, and black, respectively, in Fig. 2. At E b / 0 = 0 db and db, the code is operating in the waterfall region, where codewords that fail to decode correctly have a number of errors in them that is often indistinguishable from random errors which would occur in uncoded transmission at that SR. At E b / 0 =.7 db, however, the code is operating just inside the error floor region. The code has two codewords with input weight 3 and output weight 7, which is the minimum distance of the code. This explains why B i = 3 was observed for about /4 of the frames in error. The code has a total of a few dozen codewords of weight 8, 9,..., 28, and at least 836 codewords with input weight 9 and output weight 29 consistent with B i = 9 being observed in more than 0% of the frames in error. Despite the fact that B i 9 in more than 82% of the frames in error, the average value of B i is higher, approximately 0.0. Thus, most frame errors do not contribute a representative amount to the BER, making it necessary to simulate longer, and check that the guideline in Table I holds. A simulation of about 0 0 frames was run at E b / 0 =.7 db. Fig. 3 illustrates the 95% confidence intervals for the FER and BER as the simulation progressed. The confidence intervals for the FER and BER were computed from (2) and (0), respectively. Since the FER at this SR is less than 0 5, more than 0 6 simulated frames were necessary to collect even ten frames in error, when the Gaussian-approximation confidence intervals begin to be appropriate. The first few frame errors are identified in Fig. 3. After frames were simulated, the FER confidence interval is less than one decade thick, while the BER confidence interval is about.5 decades. At every point in the simulation, the confidence interval for BER is wider than that of the FER, as expected. As the simulation of independent frames progresses, the FER confidence interval continually shrinks; the BER confidence interval also usually shrinks, except that occasionally frames with large numbers of bit errors are observed, which can temporarily increase the uncertainty. For

5 example, the first 3 frame errors observed contained a total of 59 bit errors, but the 4th frame error alone contained 44 bit errors. This had a large impact on the average BER to that point, and an enormous impact on the confidence interval, which became vacuous at the lower end. In this example simulation, the condition of X > 385 was met when 460 million frames, and X = 2048 frame errors, had been simulated. This was more than five times the simulation length required to achieve the same 0% uncertainty in the FER. This is consistent with the higher observed variation in the simulated BER, compared to the simulated FER, as the simulation progressed. Also shown in Fig. 3 is a portion of the 95% confidence interval that would be computed if we incorrectly assumed that bit errors were independent. It can be immediately seen that the interval is inappropriately narrow, because the true BER is outside of the confidence interval over wide ranges of the number of simulated frames. VI. COCLUSIOS Massey derived a method to compute confidence intervals for the FER of a block-coded transmission, and recognized that his method did not apply to confidence intervals of the BER. We presented a method to compute confidence intervals for the BER from the first and second sample moments of simulated bit errors per frame, which is information easily recorded in a software simulation. The BER confidence interval is wider than the FER confidence interval, except when every frame error causes a constant number of bit errors, in which case the interval lengths are the same. We also presented a rule of thumb, analogous to that used for FER, for determining when a simulation has been run long enough to get an accurate estimate of the BER. REFERECES [] J. L. Massey, Comparison of rate one-half, equivalent constraint length 24, binary convolutional codes for use with sequential decoding on the deep-space channel, University of otre Dame, Department of Electrical Engineering, Tech. Rep., Apr [2] S. Lin and D. J. Costello Jr., Error Control Coding: Fundamentals and Applications. ew Jersey: Prentice-Hall, 983. [3] H. Ma and J. Wolf, On tail biting convolutional codes, Communications, IEEE Transactions on, vol. 34, no. 2, pp. 04, 986. [4] M. C. Jeruchim, P. Balaban, and K. S. Shanmugan, Simulation of Communication Systems, 2nd ed. ew York: Kluwer Academic/Plenum, [5] H. Stark and J. W. Woods, Probability, Random Processes, and Estimation Theory for Engineers, 2nd ed. Englewood Cliffs, J: Prentice Hall, 994. [6] C. Jeruchim, Techniques for estimating the bit error rate in the simulation of digital communication systems, Selected Areas in Communications, IEEE Journal on, vol. 2, no., pp , 984. [7] S. Berber, Bit error rate measurement with predetermined confidence, Electronics Letters, vol. 27, no. 3, pp , 99. [8] MATLAB, version (R20a). atick, Massachusetts: The MathWorks Inc., 20. [9] M. Knowles and A. Drukarev, Bit error rate estimation for channels with memory, Communications, IEEE Transactions on, vol. 36, no. 6, pp , 988. [0] CCSDS 3.0-B-2. TM synchronization and channel coding. Blue Book. Issue 2, Aug. 20. [Online]. Available: publications/archive/3x0b2ec.pdf