Lecture 7. Union bound for reducing M-ary to binary hypothesis testing

Size: px

Start display at page:

Download "Lecture 7. Union bound for reducing M-ary to binary hypothesis testing"

Gyles Watts
5 years ago
Views:

1 Lecture 7 Agenda for the lecture M-ary hypothesis testing and the MAP rule Union bound for reducing M-ary to binary hypothesis testing Introduction of the channel coding problem 7.1 M-ary hypothesis testing We now consider an M-ary extension of the binary hypothesis testing problem we have seen so far. Specfically, we now have M hypotheses where the hypothesis ith hypothesis represents our belief that the observation X X was generated by the distribution P i, 1 i M. A (deterministic) decision rule is now given by a partition D = {A 1,..., A M } of X we declare the hypothesis i when we observe X A i. Note that for the binary case this rule was given by {A, A c }. As before, we now have multiple probabilities of error, one corresponding to each hypothesis. Specifically, the probability of error under hypothesis i is given by P ei = 1 P i (A i ), 1 i M. c Himanshu Tyagi. Feel free to use with acknowledgement. 1

2 The decision region is now given by the (convex closure of the) set of all tupples (P e1,..., P e ), or equivalently, the set of tupples (P 1 (A 1 ),..., P M (A M )). Under the Bayesian framework, we assume a prior (q 1,..., q M ) on the hypotheses and seek the test that minimized the average probability of error 1 q i P i (A i ). We now derive a lower bound for this average probability of error, or equivalently, an upper bound on the probability of correct decision. Denote B i = {x : i = argmax q j P j (x) and q j P j (x) < q i P i (x) for all j < i}. j Note that B 1,..., B M constitutes a partition of X. Then, the average probability of correctness is bounded as q i P i (A i ) = q i P i (A i B i ) + q i P i (A i B j ) j i q j P j (A i B j ) = = = j=1 q i P i (A i B i ) + j i q j P j (A i B j ) M q j j=1 q j P j (B j ). j=1 P j (A i B j ) Thus, the decision rule D = (B 1,..., B M ) maximizes the probability of correctness. Note that this decision rule assigns to each observation x the hypothesis i which maximizes q j P j (x), or equivalently, the aposteriori probability P(I = j X = x), is called the maximum

3 Figure 1: Illustration of MAP rule aposteriori probability rule or the MAP rule for short. We can now clarify how the decision maker of Figure 4. can be implemented. Each detection element corresponding to the ith hypothesis simply computes the weighted likelihood q i P i (x) and the decision maker chooses the message with the highest weighted likelihood. This is illustrated in Figure 1. Note that for uniform prior, namely for q i = 1/M for every 1 i M, the MAP rule reduces to the maximum likelihood (ML) rule which chooses the hypothesis i that assigns the maximum probability P i (x) to the observation x. 7. Union bound M-ary to binary In principle we have identified a closed form expression for minimum probability of error. However, it is often impossible to evaluate this expression. In order to gain further insight into the problem, we now present an upper bound for the probability of error derived using the union bound, namely the bound P(A B) P(A) + P(B). From now on, we restrict to the uniform prior in our discussion. 3

4 Let (A 1,..., A M ) be the ML decision region. Furthermore, for every i j, let B ij = {x : P i (x) P j (x)}. Note that B ij denotes the ML test for the binary HT problem between P i and P j. Then, for every i, the error event A c i satisfies A c i = j i A j j i B ji. Thus, the average probability of error is bounded above as P e = 1 M 1 M P i (A c i) P i (B ji ) j i (M 1) max i j P i(b ji ), where we have used the union bound in the second inequality. The simple bound above is very powerful and allows us to bound probability of error for M-ary HT using bounds for binary HT. We illustrate its utility in the following example. Example 7.1. Let X = R and P i be a Gaussian distribution with mean s i and variance 1, 1 i M. In order to derive an upper bound for average probability of error, we first derive a bound for error of the binary HT of P i vs P j with s i < s j. This probability of error is given by P i (B ji ) = P i (X > s ( i + s j ) = P N s ) j s i =: Q ( si s j The Q-function on the right denotes the tail-probability of a standard Gaussian rv, namely Q(x) is the probability that a standard Gaussian rv is greater than x. It is well known that ). 4

5 Q(x) e x /. Thus, by union bound, average probability of error for the M-ary problem is bounded above by s i s j (M 1) max i j e 8 e log M min s i s j i j 8, which is small if log M is much greater than the square of the minimum distance between the means. In particular, if we can tolerate an error probability of ɛ = 0.001, it suffices to have e log M min s i s j i j or equivalently log M < min i j s i s j Back to transmission: Memoryless channel model and message rate We now have a good understanding of how a receiver would work, at least at an abstract level. It will treat each message as a different hypothesis and apply a decision rule for the M-ary hypothesis testing problem. Two views emerged from our treatment above: One which allowed us to compute the error that we get for a fixed number of messages, and the second which characterized the number of messages that we can send with desired error probability (see the Example of the last section). It is this latter view that Shannon proposed and suggested that error can be traded-off for probability of error. We now present a simple abstract model for the transmission channel. A channel (X, W, Y) consists of an input alphabet X, an output alphabet Y, and a transition matrix W which describes the probability with which a particular output occurs corresponding to an input. Specifically, for an input x, the output of the channel is a random variable Y taking values in Y with probability distribution W ( x). For instance, if both X and Y 5

6 are discrete then W (y x) denotes the probability that we will see y at the output of the channel when x is sent. Note that we will not use this channel once, but repeatedly several times. Therefore, we need to describe the joint statistics for different uses of the channel. In this course we assume that different uses of the channel result in independent outputs. Specifically, suppose the channel (X, W, Y) is used n-times and an input sequence x 1,..., x n is transmitted. Then the probability of seeing y 1,..., y n at the output is n W (y i x i ). Such a channel is called a stationary memoryless channel. The central question that we want to answer is For a given channel, what is the maximum number of bits of messages that we can send reliably per channel use? A modulation scheme in communication is designed to send a fixed number of messages, say N, with probability of error less than p. Suppose now you have many more messages, say N 106, to send. For instance, you may wish to send a 1 Mb file using a BPSK scheme. Each use of the modulation scheme now corresponds to one channel use. One simple approach for sending your file is repeatedly using the BPSK scheme for each bit and you will be done with 10 6 uses of the channel. However, in this case you will make an error even if one of the bits flipped. Since each use of channel was independent, this can happen with probability 1 (1 p) 106. Even if p was 10 5, this probability is more than Thus, we make an error with large probability even though our original scheme was well designed. Something more sophisticated than a simple re-use of the original shceme must be done. Note that this problem has two parts: First, the encoding problem, namely to decide what should be sent over the channel each long message to be sent, and second, the decoding problem, namely to identify which message was sent from the channel output. This second 6

7 part is an M-ary hypothesis testing problem, and we already have a good handle over it. We need to answer the first problem. But clearly the two problems are coupled our decoding rule must depend on what input was chosen for each message. In the remainder of this discussion, we will present a heuristic decoding rule and identify a simple property that our chosen inputs for each message must satisfy. Instead of using the MAP rule where the individual detectors were sending an estimate of their beliefs about each message, we will use a sub-optimal hard-thresholding rule where each detector sends a 1 if it believes its message was sent and 0 otherwise. This sub-optimal choice is just for convenience and will allow us to use our bounds for binary HT for bounding the error for the M-ary problem. Formally, consider a scheme for sending M messages over n uses of the channel where when message m is sent the sequence x(m) = (x 1m,..., x nm ) is sent over the channel. The output of the channel in this case is distributed as P m given by n P m (y 1,..., y n ) = W (y i x im ). The receiver looks at the sequence Y 1,..., Y n and resolves the M-ary hypothesis testing problem corresponding to P 1,..., P M. Our specific decision rule, to be described formally in the next class, is depicted in Figure. Figure : Illustration of the hard-thresholding rule 7

Shannon s noisy-channel theorem

Shannon s noisy-channel theorem Information theory Amon Elders Korteweg de Vries Institute for Mathematics University of Amsterdam. Tuesday, 26th of Januari Amon Elders (Korteweg de Vries Institute for