2. What are the tradeoffs among different measures of error (e.g. probability of false alarm, probability of miss, etc.)?

Transcription

1 ECE 830 / CS 76 Spring 06 Instructors: R. Willett & R. Nowak Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics Executive summary In the last lecture we saw that the likelihood ratio statistic was optimal for testing between two simple hypotheses. The test simply compares the likelihood ratio to a threshold. The optimal threshold is a function of the prior probabilities and the costs assigned to different errors. The choice of costs is subjective and depends on the nature of the problem, but the prior probabilities must be known. In practice, we face several questions:. Unfortunately, often the prior probabilities are not known precisely, and thus the correct setting for the threshold is unclear. How should we proceed?. What are the tradeoffs among different measures of error (e.g. probability of false alarm, probability of miss, etc.)? 3. Is the LRT still optimal for different error criteria? 4. Do we really need to store all the observed data, or can we get by with some summary statistics? We will address these questions in these notes. To explore these questions, we will look at a simple example. Example: Detecting ET We observe a sampled radio signal from outer space. Is this just cosmic radiation / noise, or are we receiving a message from extra-terrestrial intelligence (ETI)? null hypothesis (no ETI) : x i N (0, σ ), i =,..., n () alternative hypothesis (ETI) H : x i N (µ, σ ), µ > 0, i =,..., n. () Assume that σ > 0 is known. The first hypothesis is simple. It involves a fixed and known distribution. The second hypothesis is simple if µ is known. However, if all we know is that µ > 0, then the second hypothesis is the composite of many alternative distributions, i.e., the collection {N(µ, σ )} µ>0. In this case, H is called a composite hypothesis.

2 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics The likelihood ratio test takes the form n i= n i= πσ e σ (x i µ) πσ e σ x i = e n (πσ ) n/ σ i= (x i µ) e n (πσ ) n/ σ i= x i H γ The inequalities are preserved if we apply a monotonic transformation to both sides, so we can simplify the expression by taking the logarithm, giving us the log-likelihood ratio test ( ) n µ x σ i + nµ log(γ) i= Assuming µ > 0, this is equivalent to with ν = σ µ ln γ + nµ. Sufficient Statistics n i= x i H ν, Our test statistic t := n i= x i is called the sufficient statistic for the mean of a normal distribution. Let s rewrite our hypotheses in terms of the sufficient statistic: H :t N (0, nσ ), (3) H :t N (nµ, nσ ), µ > 0 (4) We call t a sufficient statistic because t is sufficient for performing our likelihood ratio test. More formally, a sufficient statistic is defined as follows: Definition: Sufficient statistic Let X be an n-dimensional random vector and let θ denote a p-dimensional parameter of the distribution of X. The statistic t := T (x) is a sufficient statistic for θ if and only if the conditional distribution of X given T (X) is independent of θ. More details are available at SuffStats.pdf. If our data is drawn from an exponential family probability model parameterized by θ with the form p θ (x) = exp[a(θ)b(x) + c(x) + d(θ)] then t(x) = n i= b(x i) is a sufficient statistic. Thus if we are faced with a hypothesis test of the form :x i p θ0 (x) (5) H :x i p θ (x) (6) then our test statistic is a simple function of the data, and we do not need to know the functions a, c, and d to compute it (though we may need these functions to choose an appropriate threshold).

3 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics3 3 Choosing thresholds Recall that our ideal threshold is ν = σ nµ ln γ +. However, in our ETI setting we have µ no way of knowing prior probabilities π 0, π to set γ, AND we have no way of knowing a good value for µ. To deal with this, consider an alternative design specification. Let s design a test that minimizes one type of error subject to a constraint on the other type of error. This constrained optimization criterion does not require knowledge of prior probabilities nor cost assignments. It only requires a specification of the maximum allowable value for one type of error, which is sometimes even more natural than assigning costs to the different errors. A classic result due to Neyman and Pearson shows that the solution to this type of optimization is again a likelihood ratio test. 4 Neyman-Pearson Lemma Assume that we observe a random variable distributed according to one of two distributions. : X p 0 H : X p In many problems, is consider to be a sort of baseline or default model and is called the null hypothesis. H is a different model and is called the alternative hypothesis. If a test chooses H when in fact the data were generated by the error is called a false-positive or false-alarm, since we mistakenly accepted the alternative hypothesis. The error of deciding when H was the correct model is called a false-negative or miss. Let T denote a testing procedure based on an observation of X, and let R T denote the subset of the range of X where the test chooses H. The probability of a false-positive is denoted by P 0 (R T ) := p 0 (x) dx. R T The probability of a false-negative is P (R T ), where P (R T ) := p (x) dx, R T is the probability of correctly deciding H, often called the probability of detection. Consider likelihood ratio tests of the form p (x) p o (x) H λ. The subset of the range of X where this test decides H is denoted R LR (λ) := {x : p (x) > λ p 0 (x)}, and therefore the probability of a false-positive decision is P 0 (R LR (λ)) := p 0 (x) dx = R LR (λ) {x:p (x)>λp 0 (x)} p 0 (x) dx

4 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics4 This probability is a function of the threshold λ; the set R LR (λ) shrinks/grows as λ increases/decreases. We can select λ to achieve a desired probability of error. Neyman-Pearson Lemma Consider the likelihood ratio test p (x) p o (x) H λ with λ > 0 chosen so that P 0 (R LR (λ)) = α. There does not exist another test T with P 0 (R T ) α and P (R T ) > P (R LR (λ)). That is, the LRT is the most powerful test with probability of false-negative less than or equal to α. Proof. Let T be any test with P 0 (R T ) = α and let NP denote the LRT with λ chosen so that P 0 (R LR (λ)) = α. To simplify the notation we will denote use R NP to denote the region R LR (λ). For any subset R of the range of X define P i (R) := p i (x) dx, This is simply the probability of X R under hypothesis H i. Note that P i (R NP ) = P i (R NP R T ) + P i (R NP R c T ) R P i (R T ) = P i (R NP R T ) + P i (R c NP R T ) where the superscript c indicates the complement of the set. By assumption P 0 (R NP ) = P 0 (R T ) = α, therefore P 0 (R NP R c T ) = P 0 (R c NP R T ). Now, we want to show which holds if To see that this is indeed the case, P (R NP ) P (R T ) P (R NP R c T ) P (R c NP R T ). P (R NP R c T ) = R NP R c T λ R NP R c T p (x) dx p o (x) dx = λ P o (R NP R c T ) = λ P o (RNP c R T ) = λ p o (x) dx R c NP R T R c NP R T p (x) dx = P (R c NP R T ).

5 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics5 The probability of a false-positive is also called the probability of false-alarm, which we will denote by P F A in the following examples. We will also denote the probability of detection ( probability of a false-negative) by P D. The NP test maximizes P D subject to a constraint on P F A. Example: Detecting a DC Signal in Additive White Gaussian Noise Return to our ETI example from earlier. Assuming µ > 0, our LRT amounts to n i= x i H ν, with ν = σ nµ ln γ +, and since γ was ours to choose, we can equivalently choose ν µ to trade-off between the two types of error. Let s now determine P F A and P D for the log-likelihood ratio test. ( ) P F A = t ν nπσ e nσ dt = Q, nσ π e u / du, the tail probability of the standard normal distri- where Q(z) = u z bution. Similarly, ν P D = ν ( ) (t nµ) ν nµ e nσ dt = Q nπσ nσ In both cases the expression in terms of the Q function is the result of a simple change of variables in the integration. The Q function is invertible, so we can solve for the value of ν in terms of P F A, that is ν = nσ Q (P F A ). Using this we can express P D as ( ) nµ P D = Q Q (P F A ), nµ σ where is simply the square root of the signal-to-noise ratio ( SNR). Since Q(z) as z, it is easy to see that the probability of detection increases as µ and/or n increase. σ. Example: Detecting a Change in Variance Consider the binary hypotheses : X,..., X n H : X,..., X n N (0, σ 0) N (0, σ ), σ > σ 0

6 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics6 The log-likelihood ratio test is n log ( ) ( σ 0 + σ σ 0 ) n x σ i i= H ln(γ). Some simple algebra shows ( ) σ with ν = σo (log(γ) + n ln( σ σ σ 0 σ o )). statistic for variance of a zero-mean normal distribution. n i= x i H ν Note that t := n i= x i is the sufficient Now recall that if X,...,, X n N(0, ), then n i= X i χ n (chi-square distributed with n degrees of freedom). Let s rewrite our null hypothesis test using the sufficient statistic: n x i : t = χ n i= The probability of false alarm is just the probability that a χ n random variable exceeds ν/σ0. This can be easily computed numerically. For example, if we have n = 0 and set P F A = 0.0, then the correct threshold is ν = 37.57σ0. 5 Receiver Operating Characteristic (ROC) curves The binary hypothesis test σ 0 : X = W H : X = S + W where W N(0, σ I n n ) and S = [s, s,..., s n ] T is the known signal. ( P 0 (X) = exp ) (πσ ) n σ XT X [ P (X) = exp ] (πσ ) n σ (X S)T (X S) Apply the likelihood ratio test (LRT): After simplification, we have log Λ(x) = log P (X) P 0 (X) = σ [ XT S + S T S] H X T S H σ γ + ST S = γ γ The test statistic X T S is usually called a matched filter. The LR detector filters data by projecting them onto the signal subspace.

7 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics7 5. Quality of the classifier Question: How can we assess the quality of a detector? Example: : X N(0, ) H : X N(, ) p 0(x) p (x) x P F A and P D characterize the performance of the detector. As γ increases, P F A decreases (good) and P D decreases (bad). Definition: Receiver Operating Characteristic (ROC) Curve An ROC curve is a plot that illustrates the performance of a detector (binary classifier) by plotting its P D vs. P F A at various threshold settings. First use: In World War II. The ROC curve was first developed by electrical engineers and radar engineers during for detecting aircrafts from radar signals after the attack on the Pearl Harbor. To compute the ROC curve, vary the threshold level γ and compute P F A and P D p 0(x) p (x) x The ROC curve PD P F A

8 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics8 Starts from (0, 0) and ends at (, ) (unless p i (± ) > 0). The diagonal line from (0, 0) to (, ) corresponds to random guesses. Depends on signal strength, noise strength, noise type, etc. Example: ROC and SNR The likelihood ratio test gives : X N(0, σ I) H : X N(S, σ I) X T S H γ X T S is also Gaussian distributed. Recall if X N(µ, Σ), then Y = AX N(Aµ, AΣA T ). So we can get We can also compute : X T S N(0 T S, S T σ IS) = N(0, σ S ) H : X T S N(S T S, S T σ IS) = N( S, σ S ) Since Q function is invertible, we can get where S σ P F A = Q( γ 0 ) σ S P D = Q( γ S γ ) = Q( S σ S σ S σ ) γ σ S P D = Q(Q (P F A ) S σ ), = Q (P F A ). Therefore, is the square root of Signal-to-Noise Ratio( SNR).

9 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics PD S /σ = 0.5 S /σ = 0. S /σ = S /σ = P F A 5. The AWGN Assumption AWGN is gaussian distributed as W N(0, σ I) Is real-world noise really additive, white and Gaussian? Noise in many applications (e.g. communication and radar) arise from several independent sources, all adding together at sensors and combining additively to the measurement. Central Limit Theorem If x,..., x n are independent random variables with means µ i and variances σi <,then Z n = n x i µ i n i= σ i N(0, ) in distribution as n. Thus, it is quite reasonable to model noise as additive and Gaussian in many applications. However, whiteness is not always a good assumption. 5.3 Colored Gaussian Noise Example: Correlated noise W = S + S + + S k, where S, S,... S k are interferring signals that are not of interest, and each of them is structured/correlated in time. W N(0, Σ) is called correlated or colored noise, where Σ is a structured covariance matrix. Consider the binary hypothesis test in this case. : X = S 0 + W H : X = S + W

10 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics0 where W N(0, Σ) and S 0 and S are know signal. So we can rewrite the hypothesis as : X N(S 0, Σ) H : X N(S, Σ) The probability density of each hypothesis is [ P i (X) = exp ] (π) n (Σ) (X S i) T Σ (X S i ), i = 0, The log likelihood ratio is ( ) P (X) log = [ (X S ) T Σ(X S ) (X S 0 ) T Σ (X S 0 ) ] P (X) Equivalently, = X T Σ (S S 0 ) ST Σ S + ST 0 Σ S 0 H γ (S S 0 ) T Σ X H γ + ST Σ S ST 0 Σ S 0 Let t(x) = (S S 0 ) T Σ X, we can get = γ : t N((S S 0 ) T Σ S 0, (S S 0 ) T Σ (S S 0 )) H : t N((S S 0 ) T Σ S, (S S 0 ) T Σ (S S 0 )) The probability of false alarm is ( P F A = Q In this case it is natural to define γ (S S 0 ) T Σ S 0 [(S S 0 ) T Σ (S S 0 )] SNR = (S S 0 ) T Σ (S S 0 ) ) Example: ROC with colored Gaussian noise S = [, ], S 0 = [, ], [ ] ρ Σ = ρ, Σ = ρ [ ρ ρ ].

11 Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics The test statistics is The testing problem is equivalent to y = (S S 0 ) T Σ X [ ] [ ] ρ x = [, ] ρ ρ x = + ρ (x + x ) : y N( + ρ, + ρ ) H : y N(+ + ρ, + ρ ) The probabilities of false alarm and detection are +ρ P F A = Q( γ + +ρ +ρ P D = Q( γ +ρ ) ) PD ρ = ρ = 0 ρ = P F A