Topic 3: Hypothesis Testing
|
|
- Pearl Prudence Watson
- 5 years ago
- Views:
Transcription
1 CS 8850: Advanced Machine Learning Fall 07 Topic 3: Hypothesis Testing Instructor: Daniel L. Pimentel-Alarcón c Copyright Introduction One of the simplest inference problems is that of deciding between two options (hypotheses). Example 3. (Healthy vs. Diabetic). The blood glucose level (in mg/dl) of a healthy person can be modeled as N (95, σ ), while that of a diabetic can be modeled as N (40, σ ). Given a new patient with glucose level x, you want to decide between two hypotheses: : x N (95, σ ) H : x N (40, σ ) healthy, diabetic. and H are often called null and alternative hypotheses. Example 3. (Radar). A radar is constantly emitting a signal and monitoring to see if it bounces back (see Figure 3.). The signal x that the radar receives can be modeled as N (0, σ ) if there is nothing (hence the signal doesn t bounce back) and N (µ, σ ) for some µ > 0 if an object is present (hence signal bounces back). Thus it needs to decide between: : x N (0, σ ) nothing there, H : x N (µ, σ ), µ > 0 something there. Example 3.3 (Astrophysics). The NASA wants you to determine whether two meteorites one that fell in Roswell, New Mexico, and one that fell in Chelyabinsk, Russia came from the same asteroid in space. With help from the materials expert in your interdisciplinary team, you are able to determine that if two meteorites come from the same asteroid, the difference x of their magnesium composition Figure 3.: A radar is constantly receiving a signal, and needs to decide whether an object is present or not. See Example
2 Topic 3: Hypothesis Testing 3- Figure 3.: Gene microarrays are data matrices indicating gene activation levels. Each row corresponds to one gene, and each column corresponds to one individual. We want to know which genes are related to a disease. See Example 3.4. can be modeled as N (0, σ ), and N (µ, σ ) otherwise for some unknown µ 0. Hence you need to decide between: : x N (0, σ ) same asteroid, H : x N (µ, σ ), µ 0 different asteroids. Example 3.4 (Genetics). Have you wondered how geneticists determine which genes are associated to which diseases? Essentially, they compare the average activation levels of a gene in healthy and sick individuals (see Figure 3.). The difference x between these activation levels can be modeled as N (0, σ ) if the gene is unrelated to the disease, and N (µ, σ ) for some unknown µ 0 if the gene is related to the disease. We thus have to decide between: : x N (0, σ ) gene unrelated to disease, H : x N (µ, σ ), µ 0 gene related to disease. Example 3.5 (Treatment design). Scientists often want to design a treatment (e.g., a drug or procedure) for a disease (e.g., diabetes or cancer). To this end they measure the disease presence (e.g., glucose level or tumor size) before and after treatment in N patients. The differences x i can be modeled as independent and identically distributed (i.i.d.) N (0, σ ) if the treatment is ineffective, and N (µ, σ ) for some µ < 0 if the treatment is effective. Hence we have : x,..., x N H : x,..., x N iid N (0, σ ) treatment is ineffective, iid N (µ, σ ), µ < 0 treatment is effective. Example 3.6 (Neural activity). Scientists want to determine which regions of the brain are related to certain tasks using functional magnetic resonance imaging (fmri), which essentially creates a video of the brain using magnetic fields that map hydrogen density. For example, say they want to know which region of the brain controls the thumb. Then they take an individual, ask her to move her thumb
3 Topic 3: Hypothesis Testing Figure 3.3: Left: Signal of the thumb µ R D, usually a sinusoid with the periodicity of the thumb movement. Right: Each pixel produces one signal vector x ij R D containing the brain measurements in that pixel over time. Some pixels may show neural activity correlated with the signal of the thumb. We want to find such pixels. See Example 3.6. periodically, and take an fmri video of her brain. Then scientists analyze one pixel at a time. The (i, j) th pixel will produce a signal vector x ij R D containing the brain measurements in that pixel over time. Some pixels will show neural activity correlated with the signal of the thumb µ R D, usually a sinusoid with the periodicity of the thumb movement (see Figure 3.3). Then x ij can be modeled as N (0, σ I) if the pixel is uncorrelated to the thumb movement, and N (µ, σ I) if the pixel is correlated (I denotes the identity matrix of compatible size, in this case D D). Hence for each pixel (i, j) they have to decide: : x ij H : x ij iid N (0, σ I) iid N (µ, σ I) (i, j) th pixel is uncorrelated, (i, j) th pixel is correlated. Definition 3. (Hypothesis test). A hypothesis test is a function t : Ω {, H }. 3. The Likelihood Ratio Test In general, hypothesis testing is all about deciding between two options. We observe a random variable x, and want to decide whether : x p 0 (x), H : x p (x). If your hunch is to simply pick whichever is larger between p 0 (x) and p (x), your intuition is correct. That is essentially the likelihood ratio test (LRT) in its most elemental form: p (x) p 0 (x) H, which in words means: let s make a test: if the likelihood ratio Λ(x) := p(x) p 0(x) is larger than (meaning p is larger), then pick H. Similarly, if Λ(x) < (meaning p 0 is larger), then pick.
4 Topic 3: Hypothesis Testing 3-4 Figure 3.4: Likelihood ratio test x H µ in Example 3.7. Remark 3. (Likelihood). The term likelihood is often a source of confusion. To be more precise, in hypothesis testing we observe an instance of a random variable, i.e., we observe data x = x, and we want to decide which of two distributions (p 0 or p ) is more likely to have generated this data. The likelihood that p 0 generated x is essentially p 0 (x) [evaluated at x = x], and similarly for p. Example 3.7 (Radar). Consider the hypothesis problem in Example 3.. Then Λ(x) = p (x) p 0 (x) = πσ e ( x µ σ ) = e πσ e ( x 0 σ ) Since both sides are positive, taking log we obtain: x xµ+µ σ e x σ = e µ(x µ) σ H. µ(x µ) σ H 0. and since µ > 0, this further simplifies into the following test: x H µ. In words, this LRT tells us to decide H if our observed data x is larger than µ /, and otherwise (see Figure 3.4). 3.3 Outcomes and Decision Regions A test has four possible outcomes, depending on what we decide and the truth: (0 0), (0 ), ( 0), ( ). See Table 3.. Sometimes it is desirable to reduce the probability of one particular kind of error.
5 Topic 3: Hypothesis Testing 3-5 Example 3.8. In Example 3.5, scientists want to avoid a ( 0) error, which would mean that they believe their treatment cures a disease, when it really doesn t. The probabilities of the four outcomes are determined by the regions of the test. Definition 3. (Decision regions). The decision regions R 0 and R of a test t : Ω {, H } are the inverse images of, and H, i.e., R 0 := {x Ω : t(x) = }, R := {x Ω : t(x) = H }. In words, R 0 is the region of the domain of x where we will decide, and similarly for R. Example 3.9. The decision regions of the test x H µ/ are R 0 = (, µ /) and R = ( µ /, ). Decision regions determine the probability of each outcome as follows: p gh := p h (x)dx, for g, h {0, }. R g Example 3.0. The test x H µ/ has the following outcomes probabilities (see Figure 3.5): p 00 = Q 0,σ ( µ /), p 0 = Q µ,σ ( µ /), p 0 = Q 0,σ ( µ /), p = Q µ,σ ( µ /), Decision (0 0) True Negative No-alarm Accept ( 0) False positive H False alarm Type error Truth H (0 ) False negative Miss Type error ( ) True Positive Detect Reject Table 3.: Four possible outcomes of a test. Depending on the field, they might come under different names. We will use (0 0), (0 ), ( 0), ( ) for simplicity.
6 Topic 3: Hypothesis Testing 3-6 Figure 3.5: Outcomes probabilities p gh. Left: test x H µ/ from Example 3.7. Right: test x H τ from Example 3.0; we can pick τ such that p 0 < α. Figure 3.6: Q µ,σ (τ, ) = Q(τ µ/σ, ), where Q is shorthand for Q 0,. where Q µ,σ (τ) is the tail probability of the N (µ, σ ) distribution, i.e., Q µ,σ (τ) := If we want to bound p 0, we could modify our test to τ e ( x µ σ ) dx. πσ x H τ, where τ is a threshold selected to make p 0 smaller than the desired probability of error α (see Figure 3.5). This new test has p 0 = Q 0,σ (τ). Furthermore, a simple change of variable shows that ( τ µ ) Q µ,σ (τ) = Q, (3.) σ where we use Q as shorthand for Q 0, (see Figure 3.6 to build some intuition), so p 0 = Q ( τ /σ). Finally, since Q is invertible, if we want p 0 α, we can pick τ = σq (α). 3.4 Neyman-Pearson Lemma As mentioned in Example 3.8, there are some cases where we want to bound a certain probability of error, say p 0. One way to do this is by increasing R 0. However, as R 0 grows, our accuracy p decreases (see
7 Topic 3: Hypothesis Testing 3-7 Figure 3.5 to build some intuition). Neyman-Pearson s Lemma tells us that the LRT is optimal in the sense that there exists no other test that has lower probability of error p 0 and higher accuracy p. Lemma 3. (Neyman-Pearson). Consider the likelihood ratio test t given by p (x) p 0 (x) H τ, with τ chosen such that p 0 = α. Then there exists no other test t with p 0 α and p > p. Proof. For any region R Ω, let P h (R) be the cumulative probability of p h (x) over R, i.e., P h (R) = p h (x)dx. Then for h {0, }, let p h = P h (R R ) + P h (R R 0), p h = P h (R R ) + P h (R 0 R ). (3.) Now suppose p 0 α. We need to show that p p. From (3.), this is equivalent to showing that P (R R 0) P (R 0 R ), so write P (R R 0) = p (x)dx τp 0 (x)dx = τp 0 (R R 0), (3.3) R R R 0 R R 0 where the inequality follows because in R, p (x) τp 0 (x). By assumption, p 0 = α p 0. This, together with (3.) imply P 0 (R R 0) P 0 (R 0 R ), so (3.3) τp 0 (R 0 R ) = τp 0 (x)dx p (x)dx = P (R 0 R ), R 0 R R 0 R where the last inequality follows because in R 0, p (x) τp 0 (x). Example 3.. Consider : x N (0, σ0), H : x N (0, σ), where σ 0 < σ are known. The likelihood ratio test is Λ(x) = p (x) p 0 (x) = e x σ πσ e x σ 0 πσ 0 = σ 0 e σ ( x σ 0 x σ ) H τ.
8 Topic 3: Hypothesis Testing 3-8 Figure 3.7: Left: Threshold τ selected to achieve probability of error p 0 = α in test x H τ of Example 3., where : x /σ0 χ. This is equivalent to the test x H τ with : x N (0, σ0 ), as in the Right. Or equivalently, e σ σ 0 σ 0 σ σ σ0 σ0 x σ x H σ τ, σ 0 x H log ( ) σ τ, σ 0 ( σ H σ 0σ ) σ log τ. σ 0 σ 0 }{{} τ Now recall that if y N (0, ), then y χ. So we can rewrite our hypotheses as : ( x /σ 0) χ, H : ( x /σ ) χ. Then p h (with h {0, }) is simply the probability that a σ h -scaled χ random variable is larger than τ (see Figure 3.7), i.e., p h = Q χ ( τ /σ h), where Q χ is the tail probability of the χ distribution. Since Q χ is invertible, if we want p 0 α, we can pick τ = σ0q χ (α), and then p = Q χ ( σ 0/σ Q (α)). Neyman-Pearson s Lemma tells us that χ 0 there exists no other test that has lower probability of error p 0 and higher accuracy p. 3.5 Multiple Observations We now study what happens when we have several observations instead of just one. Example 3.. Consider the hypotheses in Example 3.5, or equivalently, in vector form: : x R N N (0, σ I), H : x R N N (µ, σ I), µ < 0,
9 Topic 3: Hypothesis Testing 3-9 where denotes the all ones vector of compatible size, in this case N. The likelihood ratio is given by Λ(x) = p (x) p 0 (x) = ( πσ) N e σ (x µ)t (x µ) ( πσ) N e σ xt x Taking log we obtain the log-likelihood ratio test: = e ( σ x T x µx T +µ T ) e σ x T x = e µ σ (x T Nµ). µ σ ( x T Nµ ) H log τ, x T H σ µ log τ + Nµ } {{ } τ Notice that the direction of the inequalities in the test was inverted because µ < 0. Next observe that m = x T = N i= x i, and since sums of gaussians are gaussians, we can rewrite our hypotheses as : m N (0, Nσ ), H : m N (Nµ, Nσ ), µ < 0, Then our log-likelihood ratio test becomes m H τ, and since τ < 0, p 0 = Φ 0,Nσ (τ ), p = Φ Nµ,Nσ (τ ). where Φ µ,σ is the cumulative distribution function (CDF) of a N (µ, σ ) random variable (see Figure 3.8). Equivalently, with a similar transformation as (3.), we can write this in terms of the CDF Φ of the standard normal N (0, ),. p 0 p ( τ = Φ ), Nσ ( τ Nµ = Φ ). (3.4) Nσ Since Φ is invertible, if we want p 0 α, we can pick τ = NσΦ (α). Plugging this in (3.4), we obtain ( ) ( ) NσΦ (α) Nµ Nµ p = Φ = Φ Φ (α). Nσ σ Nµ σ is often known as the signal to noise ratio. Since Φ(τ) as τ, and since µ < 0 by assumption, it is easy to see that p increases with N and µ, but decreases with σ. 3.6 Multiple Testing In many applications we run multiple tests, and we want to bound the probability of making one or more mistakes.
10 Topic 3: Hypothesis Testing 3-0 Figure 3.8: Outcomes probabilities p gh of the test m H τ. See Example 3.. Example 3.3. In Example 3.6 we have a family of K tests (where K is the number of pixels), and we want to be confident that all the identified pixels are truly correlated to the thumb s movement. Definition 3.3 (Family-wise error rate (FWER)). The family-wise error rate is the probability of making one or more ( 0) errors. More precisely, for a family of K tests, ( K ) F W ER = P k= { k 0 k }, where { k 0 k } denotes the event of deciding H in the k th test given that is true. Lemma 3. (Bonferroni Correction). Consider a family of K tests. Setting p 0 = α /K for each test achieves FWER < α. Proof. As a simple consequence of the union bound, we have: ( K ) K F W ER = P { k 0 k } P ({ k 0 k }) = k= k= K p 0 = K α K = α. k= 3.7 Composite Hypotheses So far we have studied simple hypotheses where all distributions and their parameters are known, as in Examples 3., 3.6 and 3., where the distributions and their parameters are known. However, in many practical situations this is not the case. For instance, in H of Example 3., all we know is that µ > 0.
11 Topic 3: Hypothesis Testing 3- In this case, H is composed of the collection of distributions {N (µ, σ )} µ>0. More generally, a composite problem has the form: : x p 0 (x θ 0 ), θ 0 Θ 0, H : x p (x θ ), θ Θ, where the notation p h (x θ h ) means that θ h is a parameter of the distribution p h, and Θ h is a set of all possible values of the parameter θ h. In general, p 0 and p may be entirely different distributions, and the sets Θ 0, Θ may be entirely different. Even though Examples 3. and 3.5 are composite problems, since we know µ > 0, we were still able to derive its LRT in Examples 3.7 and 3.. This is not always the case. There are some more complicated cases, like Examples 3.3 and 3.4, where µ is completely unknown, and this can complicate things. Example 3.4 (Wald s test). Consider Examples 3.3 and 3.4. The likelihood ratio is Λ(x) = p (x) p 0 (x) = πσ e ( x µ σ ) = e πσ e ( x 0 σ ) Taking log and with some minor algebra we obtain: x xµ+µ σ e x σ = e µ(x µ) σ H. xµ H µ. However, since we don t know the sign of µ, we cannot continue as in Example 3.7, as dividing by µ could reverse the direction of the inequalities in the test. Hence this test is uncomputable, or undetermined. So how can we proceed? For example, let s say we decide to use the test from Example 3.0: x H τ. It could happen that we are lucky and µ > 0. Then our test will be optimal (as shown by Neyman- Pearson s Lemma) with p 0 = Q( τ /σ) and p = Q( τ µ /σ) (see Example 3.0 and Figure 3.5). However, if we are unlucky and µ < 0, our test would be doing something terribly insensible, and would have terrible accuracy p = Q( τ+ µ /σ); see Figure 3.9 to build some intuition. One good compromise is to use Wald s test: which has p 0 p x H τ, = Q( τ /σ), ( τ µ ) = Q + Q σ ( τ + µ Wald s test is not optimal, but is sensible. It has higher probability of error p 0 than if we are lucky and guess µ correctly, but also has higher accuracy p than if we are unlucky and guess µ incorrectly. σ ).
12 Topic 3: Hypothesis Testing 3- Figure 3.9: Composite hypothesis test where µ 0 is unknown. Left: Probabilities p 0 and p of the test x H τ. If we are lucky and µ > 0, this test test will be optimal, but if µ < 0, our test will be terrible. Right: Wald s test x H τ. Wald s test is not optimal, but is sensible. It has higher probability of error p 0 than if we are lucky and guess µ correctly, but also has higher accuracy p than if we are unlucky and guess µ incorrectly. See Example 3.4 and Figure 3.0. Figure 3.0: Left: ROC curves of the test x H τ for different values of µ. Consistent with our analysis from Example 3., we can see that p grows with µ. Right: ROC curves for the test x H τ when µ > 0 (optimal), when µ < 0 (terrible), and for Wald s test. This shows that Wald s test is suboptimal but sensible. See Example ROC Curves As shown in Example 3.4, it is not always possible to device an optimal test. It is thus reasonable to ask how good a test is. For example, how good is Wald s test? One way to do this is with Receiver Operating Characteristic (ROC) curves, which measure a test s performance by plotting its p as a function of its p 0. ROC curves are widely used in laboratories to measure a test s ability to discriminate diseased cases from normal cases, and also to compare the performance of two or more tests. 3.9 Generalized Likelihood Ratio Test Wald s test was an intuitive solution to the simplest composite problem. However, Wald s test has a solid statistical foundation. In fact, Wald s test is the result of estimating µ and then using this estimate in a likelihood ratio test. We will come back to Wald s test and its generalization, introducing the generalized likelihood ratio test (GLRT), but first we will need to learn about estimation, which is our next topic.
Topic 5: Generalized Likelihood Ratio Test
CS 885: Advanced Machine Learning Fall 7 Topic 5: Generalized Likelihood Ratio Test Instructor: Daniel L. Pimentel-Alarcón Copyright 7 5. Introduction We already started studying composite hypothesis problems
More informationComposite Hypotheses and Generalized Likelihood Ratio Tests
Composite Hypotheses and Generalized Likelihood Ratio Tests Rebecca Willett, 06 In many real world problems, it is difficult to precisely specify probability distributions. Our models for data may involve
More informationLecture 5: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics. 1 Executive summary
ECE 830 Spring 207 Instructor: R. Willett Lecture 5: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics Executive summary In the last lecture we saw that the likelihood
More information10. Composite Hypothesis Testing. ECE 830, Spring 2014
10. Composite Hypothesis Testing ECE 830, Spring 2014 1 / 25 In many real world problems, it is difficult to precisely specify probability distributions. Our models for data may involve unknown parameters
More information2. What are the tradeoffs among different measures of error (e.g. probability of false alarm, probability of miss, etc.)?
ECE 830 / CS 76 Spring 06 Instructors: R. Willett & R. Nowak Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics Executive summary In the last lecture we
More informationDetection theory. H 0 : x[n] = w[n]
Detection Theory Detection theory A the last topic of the course, we will briefly consider detection theory. The methods are based on estimation theory and attempt to answer questions such as Is a signal
More informationPATTERN RECOGNITION AND MACHINE LEARNING
PATTERN RECOGNITION AND MACHINE LEARNING Slide Set 3: Detection Theory January 2018 Heikki Huttunen heikki.huttunen@tut.fi Department of Signal Processing Tampere University of Technology Detection theory
More informationAssociation studies and regression
Association studies and regression CM226: Machine Learning for Bioinformatics. Fall 2016 Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar Association studies and regression 1 / 104 Administration
More informationLecture 8: Signal Detection and Noise Assumption
ECE 830 Fall 0 Statistical Signal Processing instructor: R. Nowak Lecture 8: Signal Detection and Noise Assumption Signal Detection : X = W H : X = S + W where W N(0, σ I n n and S = [s, s,..., s n ] T
More informationTopic 2: Logistic Regression
CS 4850/6850: Introduction to Machine Learning Fall 208 Topic 2: Logistic Regression Instructor: Daniel L. Pimentel-Alarcón c Copyright 208 2. Introduction Arguably the simplest task that we can teach
More informationLecture 21: October 19
36-705: Intermediate Statistics Fall 2017 Lecturer: Siva Balakrishnan Lecture 21: October 19 21.1 Likelihood Ratio Test (LRT) To test composite versus composite hypotheses the general method is to use
More informationORF 245 Fundamentals of Statistics Chapter 9 Hypothesis Testing
ORF 245 Fundamentals of Statistics Chapter 9 Hypothesis Testing Robert Vanderbei Fall 2014 Slides last edited on November 24, 2014 http://www.princeton.edu/ rvdb Coin Tossing Example Consider two coins.
More informationECE531 Lecture 6: Detection of Discrete-Time Signals with Random Parameters
ECE531 Lecture 6: Detection of Discrete-Time Signals with Random Parameters D. Richard Brown III Worcester Polytechnic Institute 26-February-2009 Worcester Polytechnic Institute D. Richard Brown III 26-February-2009
More informationIf there exists a threshold k 0 such that. then we can take k = k 0 γ =0 and achieve a test of size α. c 2004 by Mark R. Bell,
Recall The Neyman-Pearson Lemma Neyman-Pearson Lemma: Let Θ = {θ 0, θ }, and let F θ0 (x) be the cdf of the random vector X under hypothesis and F θ (x) be its cdf under hypothesis. Assume that the cdfs
More informationDetection theory 101 ELEC-E5410 Signal Processing for Communications
Detection theory 101 ELEC-E5410 Signal Processing for Communications Binary hypothesis testing Null hypothesis H 0 : e.g. noise only Alternative hypothesis H 1 : signal + noise p(x;h 0 ) γ p(x;h 1 ) Trade-off
More informationStatistical testing. Samantha Kleinberg. October 20, 2009
October 20, 2009 Intro to significance testing Significance testing and bioinformatics Gene expression: Frequently have microarray data for some group of subjects with/without the disease. Want to find
More information14.30 Introduction to Statistical Methods in Economics Spring 2009
MIT OpenCourseWare http://ocw.mit.edu 4.0 Introduction to Statistical Methods in Economics Spring 009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
More informationIntelligent Systems Statistical Machine Learning
Intelligent Systems Statistical Machine Learning Carsten Rother, Dmitrij Schlesinger WS2014/2015, Our tasks (recap) The model: two variables are usually present: - the first one is typically discrete k
More informationSTAT 135 Lab 5 Bootstrapping and Hypothesis Testing
STAT 135 Lab 5 Bootstrapping and Hypothesis Testing Rebecca Barter March 2, 2015 The Bootstrap Bootstrap Suppose that we are interested in estimating a parameter θ from some population with members x 1,...,
More informationSTAT 135 Lab 6 Duality of Hypothesis Testing and Confidence Intervals, GLRT, Pearson χ 2 Tests and Q-Q plots. March 8, 2015
STAT 135 Lab 6 Duality of Hypothesis Testing and Confidence Intervals, GLRT, Pearson χ 2 Tests and Q-Q plots March 8, 2015 The duality between CI and hypothesis testing The duality between CI and hypothesis
More informationEconomics 520. Lecture Note 19: Hypothesis Testing via the Neyman-Pearson Lemma CB 8.1,
Economics 520 Lecture Note 9: Hypothesis Testing via the Neyman-Pearson Lemma CB 8., 8.3.-8.3.3 Uniformly Most Powerful Tests and the Neyman-Pearson Lemma Let s return to the hypothesis testing problem
More informationIntroductory Econometrics
Session 4 - Testing hypotheses Roland Sciences Po July 2011 Motivation After estimation, delivering information involves testing hypotheses Did this drug had any effect on the survival rate? Is this drug
More informationIntelligent Systems Statistical Machine Learning
Intelligent Systems Statistical Machine Learning Carsten Rother, Dmitrij Schlesinger WS2015/2016, Our model and tasks The model: two variables are usually present: - the first one is typically discrete
More informationStatistical Methods for Particle Physics (I)
Statistical Methods for Particle Physics (I) https://agenda.infn.it/conferencedisplay.py?confid=14407 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationLecture Testing Hypotheses: The Neyman-Pearson Paradigm
Math 408 - Mathematical Statistics Lecture 29-30. Testing Hypotheses: The Neyman-Pearson Paradigm April 12-15, 2013 Konstantin Zuev (USC) Math 408, Lecture 29-30 April 12-15, 2013 1 / 12 Agenda Example:
More informationImportance Sampling and. Radon-Nikodym Derivatives. Steven R. Dunbar. Sampling with respect to 2 distributions. Rare Event Simulation
1 / 33 Outline 1 2 3 4 5 2 / 33 More than one way to evaluate a statistic A statistic for X with pdf u(x) is A = E u [F (X)] = F (x)u(x) dx 3 / 33 Suppose v(x) is another probability density such that
More informationHypothesis testing (cont d)
Hypothesis testing (cont d) Ulrich Heintz Brown University 4/12/2016 Ulrich Heintz - PHYS 1560 Lecture 11 1 Hypothesis testing Is our hypothesis about the fundamental physics correct? We will not be able
More informationSTAT 830 Hypothesis Testing
STAT 830 Hypothesis Testing Richard Lockhart Simon Fraser University STAT 830 Fall 2018 Richard Lockhart (Simon Fraser University) STAT 830 Hypothesis Testing STAT 830 Fall 2018 1 / 30 Purposes of These
More informationLecture 10: Generalized likelihood ratio test
Stat 200: Introduction to Statistical Inference Autumn 2018/19 Lecture 10: Generalized likelihood ratio test Lecturer: Art B. Owen October 25 Disclaimer: These notes have not been subjected to the usual
More informationHYPOTHESIS TESTING: FREQUENTIST APPROACH.
HYPOTHESIS TESTING: FREQUENTIST APPROACH. These notes summarize the lectures on (the frequentist approach to) hypothesis testing. You should be familiar with the standard hypothesis testing from previous
More informationBayesian Decision Theory
Introduction to Pattern Recognition [ Part 4 ] Mahdi Vasighi Remarks It is quite common to assume that the data in each class are adequately described by a Gaussian distribution. Bayesian classifier is
More informationMathematical Statistics
Mathematical Statistics MAS 713 Chapter 8 Previous lecture: 1 Bayesian Inference 2 Decision theory 3 Bayesian Vs. Frequentist 4 Loss functions 5 Conjugate priors Any questions? Mathematical Statistics
More informationLecture 2: Review of Basic Probability Theory
ECE 830 Fall 2010 Statistical Signal Processing instructor: R. Nowak, scribe: R. Nowak Lecture 2: Review of Basic Probability Theory Probabilistic models will be used throughout the course to represent
More information8: Hypothesis Testing
Some definitions 8: Hypothesis Testing. Simple, compound, null and alternative hypotheses In test theory one distinguishes between simple hypotheses and compound hypotheses. A simple hypothesis Examples:
More informationLecture 7 Introduction to Statistical Decision Theory
Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7
More informationPartitioning the Parameter Space. Topic 18 Composite Hypotheses
Topic 18 Composite Hypotheses Partitioning the Parameter Space 1 / 10 Outline Partitioning the Parameter Space 2 / 10 Partitioning the Parameter Space Simple hypotheses limit us to a decision between one
More information4 Hypothesis testing. 4.1 Types of hypothesis and types of error 4 HYPOTHESIS TESTING 49
4 HYPOTHESIS TESTING 49 4 Hypothesis testing In sections 2 and 3 we considered the problem of estimating a single parameter of interest, θ. In this section we consider the related problem of testing whether
More informationSummary and discussion of: Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Summary and discussion of: Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing Statistics Journal Club, 36-825 Beau Dabbs and Philipp Burckhardt 9-19-2014 1 Paper
More informationLecture 13: p-values and union intersection tests
Lecture 13: p-values and union intersection tests p-values After a hypothesis test is done, one method of reporting the result is to report the size α of the test used to reject H 0 or accept H 0. If α
More informationIntroduction 1. STA442/2101 Fall See last slide for copyright information. 1 / 33
Introduction 1 STA442/2101 Fall 2016 1 See last slide for copyright information. 1 / 33 Background Reading Optional Chapter 1 of Linear models with R Chapter 1 of Davison s Statistical models: Data, and
More informationBayes Rule for Minimizing Risk
Bayes Rule for Minimizing Risk Dennis Lee April 1, 014 Introduction In class we discussed Bayes rule for minimizing the probability of error. Our goal is to generalize this rule to minimize risk instead
More informationChapter 5: HYPOTHESIS TESTING
MATH411: Applied Statistics Dr. YU, Chi Wai Chapter 5: HYPOTHESIS TESTING 1 WHAT IS HYPOTHESIS TESTING? As its name indicates, it is about a test of hypothesis. To be more precise, we would first translate
More informationStatistical Inference. Hypothesis Testing
Statistical Inference Hypothesis Testing Previously, we introduced the point and interval estimation of an unknown parameter(s), say µ and σ 2. However, in practice, the problem confronting the scientist
More informationLecture 12 November 3
STATS 300A: Theory of Statistics Fall 2015 Lecture 12 November 3 Lecturer: Lester Mackey Scribe: Jae Hyuck Park, Christian Fong Warning: These notes may contain factual and/or typographic errors. 12.1
More informationChris Piech CS109 CS109 Final Exam. Fall Quarter Dec 14 th, 2017
Chris Piech CS109 CS109 Final Exam Fall Quarter Dec 14 th, 2017 This is a closed calculator/computer exam. You are, however, allowed to use notes in the exam. The last page of the exam is a Standard Normal
More informationFrequentist Statistics and Hypothesis Testing Spring
Frequentist Statistics and Hypothesis Testing 18.05 Spring 2018 http://xkcd.com/539/ Agenda Introduction to the frequentist way of life. What is a statistic? NHST ingredients; rejection regions Simple
More informationSTAT 830 Hypothesis Testing
STAT 830 Hypothesis Testing Hypothesis testing is a statistical problem where you must choose, on the basis of data X, between two alternatives. We formalize this as the problem of choosing between two
More informationConfidence Intervals and Hypothesis Tests
Confidence Intervals and Hypothesis Tests STA 281 Fall 2011 1 Background The central limit theorem provides a very powerful tool for determining the distribution of sample means for large sample sizes.
More informationHypothesis Test. The opposite of the null hypothesis, called an alternative hypothesis, becomes
Neyman-Pearson paradigm. Suppose that a researcher is interested in whether the new drug works. The process of determining whether the outcome of the experiment points to yes or no is called hypothesis
More informationNotes on the Multivariate Normal and Related Topics
Version: July 10, 2013 Notes on the Multivariate Normal and Related Topics Let me refresh your memory about the distinctions between population and sample; parameters and statistics; population distributions
More informationChapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1)
Chapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1) Detection problems can usually be casted as binary or M-ary hypothesis testing problems. Applications: This chapter: Simple hypothesis
More informationData Mining. CS57300 Purdue University. March 22, 2018
Data Mining CS57300 Purdue University March 22, 2018 1 Hypothesis Testing Select 50% users to see headline A Unlimited Clean Energy: Cold Fusion has Arrived Select 50% users to see headline B Wedding War
More informationTesting Hypothesis. Maura Mezzetti. Department of Economics and Finance Università Tor Vergata
Maura Department of Economics and Finance Università Tor Vergata Hypothesis Testing Outline It is a mistake to confound strangeness with mystery Sherlock Holmes A Study in Scarlet Outline 1 The Power Function
More informationProbability and Statistics. Terms and concepts
Probability and Statistics Joyeeta Dutta Moscato June 30, 2014 Terms and concepts Sample vs population Central tendency: Mean, median, mode Variance, standard deviation Normal distribution Cumulative distribution
More informationexp{ (x i) 2 i=1 n i=1 (x i a) 2 (x i ) 2 = exp{ i=1 n i=1 n 2ax i a 2 i=1
4 Hypothesis testing 4. Simple hypotheses A computer tries to distinguish between two sources of signals. Both sources emit independent signals with normally distributed intensity, the signals of the first
More informationAnnouncements. Proposals graded
Announcements Proposals graded Kevin Jamieson 2018 1 Hypothesis testing Machine Learning CSE546 Kevin Jamieson University of Washington October 30, 2018 2018 Kevin Jamieson 2 Anomaly detection You are
More informationDecision-making, inference, and learning theory. ECE 830 & CS 761, Spring 2016
Decision-making, inference, and learning theory ECE 830 & CS 761, Spring 2016 1 / 22 What do we have here? Given measurements or observations of some physical process, we ask the simple question what do
More informationLecture 22: Error exponents in hypothesis testing, GLRT
10-704: Information Processing and Learning Spring 2012 Lecture 22: Error exponents in hypothesis testing, GLRT Lecturer: Aarti Singh Scribe: Aarti Singh Disclaimer: These notes have not been subjected
More informationDETECTION theory deals primarily with techniques for
ADVANCED SIGNAL PROCESSING SE Optimum Detection of Deterministic and Random Signals Stefan Tertinek Graz University of Technology turtle@sbox.tugraz.at Abstract This paper introduces various methods for
More informationStatistical Inference: Estimation and Confidence Intervals Hypothesis Testing
Statistical Inference: Estimation and Confidence Intervals Hypothesis Testing 1 In most statistics problems, we assume that the data have been generated from some unknown probability distribution. We desire
More informationMath Review Sheet, Fall 2008
1 Descriptive Statistics Math 3070-5 Review Sheet, Fall 2008 First we need to know about the relationship among Population Samples Objects The distribution of the population can be given in one of the
More informationReview. December 4 th, Review
December 4 th, 2017 Att. Final exam: Course evaluation Friday, 12/14/2018, 10:30am 12:30pm Gore Hall 115 Overview Week 2 Week 4 Week 7 Week 10 Week 12 Chapter 6: Statistics and Sampling Distributions Chapter
More informationBinary Logistic Regression
The coefficients of the multiple regression model are estimated using sample data with k independent variables Estimated (or predicted) value of Y Estimated intercept Estimated slope coefficients Ŷ = b
More informationBayesian Decision Theory
Bayesian Decision Theory Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Fall 2017 CS 551, Fall 2017 c 2017, Selim Aksoy (Bilkent University) 1 / 46 Bayesian
More informationSome General Types of Tests
Some General Types of Tests We may not be able to find a UMP or UMPU test in a given situation. In that case, we may use test of some general class of tests that often have good asymptotic properties.
More informationPolitical Science 236 Hypothesis Testing: Review and Bootstrapping
Political Science 236 Hypothesis Testing: Review and Bootstrapping Rocío Titiunik Fall 2007 1 Hypothesis Testing Definition 1.1 Hypothesis. A hypothesis is a statement about a population parameter The
More informationF2E5216/TS1002 Adaptive Filtering and Change Detection. Course Organization. Lecture plan. The Books. Lecture 1
Adaptive Filtering and Change Detection Bo Wahlberg (KTH and Fredrik Gustafsson (LiTH Course Organization Lectures and compendium: Theory, Algorithms, Applications, Evaluation Toolbox and manual: Algorithms,
More informationLECTURE NOTE #3 PROF. ALAN YUILLE
LECTURE NOTE #3 PROF. ALAN YUILLE 1. Three Topics (1) Precision and Recall Curves. Receiver Operating Characteristic Curves (ROC). What to do if we do not fix the loss function? (2) The Curse of Dimensionality.
More informationTopic 2: Review of Probability Theory
CS 8850: Advanced Machine Learning Fall 2017 Topic 2: Review of Probability Theory Instructor: Daniel L. Pimentel-Alarcón c Copyright 2017 2.1 Why Probability? Many (if not all) applications of machine
More informationMethods for Statistical Prediction Financial Time Series I. Topic 1: Review on Hypothesis Testing
Methods for Statistical Prediction Financial Time Series I Topic 1: Review on Hypothesis Testing Hung Chen Department of Mathematics National Taiwan University 9/26/2002 OUTLINE 1. Fundamental Concepts
More informationStatistics 3858 : Contingency Tables
Statistics 3858 : Contingency Tables 1 Introduction Before proceeding with this topic the student should review generalized likelihood ratios ΛX) for multinomial distributions, its relation to Pearson
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationProbability and Statistics. Joyeeta Dutta-Moscato June 29, 2015
Probability and Statistics Joyeeta Dutta-Moscato June 29, 2015 Terms and concepts Sample vs population Central tendency: Mean, median, mode Variance, standard deviation Normal distribution Cumulative distribution
More informationStat 5421 Lecture Notes Fuzzy P-Values and Confidence Intervals Charles J. Geyer March 12, Discreteness versus Hypothesis Tests
Stat 5421 Lecture Notes Fuzzy P-Values and Confidence Intervals Charles J. Geyer March 12, 2016 1 Discreteness versus Hypothesis Tests You cannot do an exact level α test for any α when the data are discrete.
More informationTopic 17: Simple Hypotheses
Topic 17: November, 2011 1 Overview and Terminology Statistical hypothesis testing is designed to address the question: Do the data provide sufficient evidence to conclude that we must depart from our
More informationProblem Set 2. MAS 622J/1.126J: Pattern Recognition and Analysis. Due: 5:00 p.m. on September 30
Problem Set 2 MAS 622J/1.126J: Pattern Recognition and Analysis Due: 5:00 p.m. on September 30 [Note: All instructions to plot data or write a program should be carried out using Matlab. In order to maintain
More information1 Proof techniques. CS 224W Linear Algebra, Probability, and Proof Techniques
1 Proof techniques Here we will learn to prove universal mathematical statements, like the square of any odd number is odd. It s easy enough to show that this is true in specific cases for example, 3 2
More informationHypothesis testing I. - In particular, we are talking about statistical hypotheses. [get everyone s finger length!] n =
Hypothesis testing I I. What is hypothesis testing? [Note we re temporarily bouncing around in the book a lot! Things will settle down again in a week or so] - Exactly what it says. We develop a hypothesis,
More informationExamples and Limits of the GLM
Examples and Limits of the GLM Chapter 1 1.1 Motivation 1 1.2 A Review of Basic Statistical Ideas 2 1.3 GLM Definition 4 1.4 GLM Examples 4 1.5 Student Goals 5 1.6 Homework Exercises 5 1.1 Motivation In
More informationLecture 8: Information Theory and Statistics
Lecture 8: Information Theory and Statistics Part II: Hypothesis Testing and I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 23, 2015 1 / 50 I-Hsiang
More informationECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
More informationIntroduction to Statistical Inference
Structural Health Monitoring Using Statistical Pattern Recognition Introduction to Statistical Inference Presented by Charles R. Farrar, Ph.D., P.E. Outline Introduce statistical decision making for Structural
More informationReview of Probability Theory
Review of Probability Theory Arian Maleki and Tom Do Stanford University Probability theory is the study of uncertainty Through this class, we will be relying on concepts from probability theory for deriving
More informationChapter 9: Hypothesis Testing Sections
Chapter 9: Hypothesis Testing Sections 9.1 Problems of Testing Hypotheses 9.2 Testing Simple Hypotheses 9.3 Uniformly Most Powerful Tests Skip: 9.4 Two-Sided Alternatives 9.6 Comparing the Means of Two
More informationCherry Blossom run (1) The credit union Cherry Blossom Run is a 10 mile race that takes place every year in D.C. In 2009 there were participants
18.650 Statistics for Applications Chapter 5: Parametric hypothesis testing 1/37 Cherry Blossom run (1) The credit union Cherry Blossom Run is a 10 mile race that takes place every year in D.C. In 2009
More informationLet us first identify some classes of hypotheses. simple versus simple. H 0 : θ = θ 0 versus H 1 : θ = θ 1. (1) one-sided
Let us first identify some classes of hypotheses. simple versus simple H 0 : θ = θ 0 versus H 1 : θ = θ 1. (1) one-sided H 0 : θ θ 0 versus H 1 : θ > θ 0. (2) two-sided; null on extremes H 0 : θ θ 1 or
More informationBias Variance Trade-off
Bias Variance Trade-off The mean squared error of an estimator MSE(ˆθ) = E([ˆθ θ] 2 ) Can be re-expressed MSE(ˆθ) = Var(ˆθ) + (B(ˆθ) 2 ) MSE = VAR + BIAS 2 Proof MSE(ˆθ) = E((ˆθ θ) 2 ) = E(([ˆθ E(ˆθ)]
More informationTopic 10: Hypothesis Testing
Topic 10: Hypothesis Testing Course 003, 2016 Page 0 The Problem of Hypothesis Testing A statistical hypothesis is an assertion or conjecture about the probability distribution of one or more random variables.
More information14.30 Introduction to Statistical Methods in Economics Spring 2009
MIT OpenCourseWare http://ocw.mit.edu.30 Introduction to Statistical Methods in Economics Spring 009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. .30
More informationCONTENTS OF DAY 2. II. Why Random Sampling is Important 10 A myth, an urban legend, and the real reason NOTES FOR SUMMER STATISTICS INSTITUTE COURSE
1 2 CONTENTS OF DAY 2 I. More Precise Definition of Simple Random Sample 3 Connection with independent random variables 4 Problems with small populations 9 II. Why Random Sampling is Important 10 A myth,
More informationHypothesis Testing. BS2 Statistical Inference, Lecture 11 Michaelmas Term Steffen Lauritzen, University of Oxford; November 15, 2004
Hypothesis Testing BS2 Statistical Inference, Lecture 11 Michaelmas Term 2004 Steffen Lauritzen, University of Oxford; November 15, 2004 Hypothesis testing We consider a family of densities F = {f(x; θ),
More information401 Review. 6. Power analysis for one/two-sample hypothesis tests and for correlation analysis.
401 Review Major topics of the course 1. Univariate analysis 2. Bivariate analysis 3. Simple linear regression 4. Linear algebra 5. Multiple regression analysis Major analysis methods 1. Graphical analysis
More informationSTAT 801: Mathematical Statistics. Hypothesis Testing
STAT 801: Mathematical Statistics Hypothesis Testing Hypothesis testing: a statistical problem where you must choose, on the basis o data X, between two alternatives. We ormalize this as the problem o
More informationsimple if it completely specifies the density of x
3. Hypothesis Testing Pure significance tests Data x = (x 1,..., x n ) from f(x, θ) Hypothesis H 0 : restricts f(x, θ) Are the data consistent with H 0? H 0 is called the null hypothesis simple if it completely
More informationHypothesis Testing. 1 Definitions of test statistics. CB: chapter 8; section 10.3
Hypothesis Testing CB: chapter 8; section 0.3 Hypothesis: statement about an unknown population parameter Examples: The average age of males in Sweden is 7. (statement about population mean) The lowest
More informationSolving with Absolute Value
Solving with Absolute Value Who knew two little lines could cause so much trouble? Ask someone to solve the equation 3x 2 = 7 and they ll say No problem! Add just two little lines, and ask them to solve
More informationThis does not cover everything on the final. Look at the posted practice problems for other topics.
Class 7: Review Problems for Final Exam 8.5 Spring 7 This does not cover everything on the final. Look at the posted practice problems for other topics. To save time in class: set up, but do not carry
More informationMasters Comprehensive Examination Department of Statistics, University of Florida
Masters Comprehensive Examination Department of Statistics, University of Florida May 6, 003, 8:00 am - :00 noon Instructions: You have four hours to answer questions in this examination You must show
More information1 Probabilities. 1.1 Basics 1 PROBABILITIES
1 PROBABILITIES 1 Probabilities Probability is a tricky word usually meaning the likelyhood of something occuring or how frequent something is. Obviously, if something happens frequently, then its probability
More informationStat 206: Estimation and testing for a mean vector,
Stat 206: Estimation and testing for a mean vector, Part II James Johndrow 2016-12-03 Comparing components of the mean vector In the last part, we talked about testing the hypothesis H 0 : µ 1 = µ 2 where
More information