Lecture 8: Information Theory and Statistics

Size: px
Start display at page:

Download "Lecture 8: Information Theory and Statistics"

Transcription

1 Lecture 8: Information Theory and Statistics Part II: Hypothesis Testing and Estimation I-Hsiang Wang Department of Electrical Engineering National Taiwan University December 22, / 30 I-Hsiang Wang IT Lecture 8 Part II

2 1 Hypothesis Testing 2 / 30 I-Hsiang Wang IT Lecture 8 Part II

3 1 Hypothesis Testing 3 / 30 I-Hsiang Wang IT Lecture 8 Part II

4 Basic Setup We begin with the simplest setup binary hypothesis testing: 1 Two hypotheses regarding the observation X, indexed by θ {0, 1}: H 0 : X P 0 (Null Hypothesis, θ = 0) H 1 : X P 1 (Alternative Hypothesis, θ = 1) 2 Goal: design a decision making algorithm ϕ : X {0, 1}, x ˆθ, to choose one of the two hypotheses, based on the observed realization of X, so that a certain cost (or risk) is minimized. 3 A popular measure of the cost is based on probability of errors: Probability of false alarm (false positive; type I error): α ϕ P FA (ϕ) P {H 1 is chosen H 0 }. Probability of miss detection (false negative; type II error): β ϕ P MD (ϕ) P {H 0 is chosen H 1 }. 4 / 30 I-Hsiang Wang IT Lecture 8 Part II

5 Deterministic Testing Algorithm Decision Regions X Observation Space A 1 ( ) Acceptance Region of H 1. A 0 ( ) Acceptance Region of H 0. A test ϕ : X {0, 1} is equivalently characterized by its corresponding acceptance (decision) regions: ) { 1 A θ (ϕ) ϕ (ˆθ x X : ϕ (x) = ˆθ }, ˆθ = 0, 1. Hence, the two types of probability of error can be equivalently represented as α ϕ = β ϕ = x A 1 (ϕ) x A 0(ϕ) P 0 (x) = ϕ (x) P 0 (x), x X P 1 (x) = (1 ϕ (x)) P 1 (x). x X When the context is clear, we often drop the dependency on the test ϕ when dealing with acceptance regions Aˆθ. 5 / 30 I-Hsiang Wang IT Lecture 8 Part II

6 Likelihood Ratio Test Definition 1 (Likelihood Ratio Test) A (deteministic) likelihood ratio test (LRT) is a test ϕ τ, parametrized by constants τ > 0 (called threshold), defined as follows: { 1 if P 1 (x) > τp 0 (x) ϕ τ (x) = 0 if P 1 (x) τp 0 (x). For x supp P0, the likelihood ratio L (x) P 1(x) P 0(x). Hence, LRT is a thresholding algorithm on likelihood ratio L (x). Remark: For computational convenience, often one deals with log likelihood ratio (LLR) log (L(x)) = log (P 1 (x)) log (P 0 (x)). 6 / 30 I-Hsiang Wang IT Lecture 8 Part II

7 Trade-Off Between α (P FA ) and β (P MD ) Theorem 1 (Neyman-Pearson Lemma) For a likelihood ratio test ϕ τ and another deterministic test ϕ, α ϕ α ϕτ = β ϕ β ϕτ. pf: Observe x X, 0 (ϕ τ (x) ϕ (x)) (P 1 (x) τp 0 (x)), because if P 1 (x) τp 0 (x) > 0 = ϕ τ (x) = 1 = (ϕ τ (x) ϕ (x)) 0. if P 1 (x) τp 0 (x) 0 = ϕ τ (x) = 0 = (ϕ τ (x) ϕ (x)) 0. Summing over all x X, we get 0 (1 β ϕτ ) (1 β ϕ ) τ (α ϕτ α ϕ ) = (β ϕ β ϕτ ) + τ (α ϕ α ϕτ ). Since τ > 0, from above we conclude that α ϕ α ϕτ = β ϕ β ϕτ. 7 / 30 I-Hsiang Wang IT Lecture 8 Part II

8 (P MD ) (P MD ) (P FA ) 1 (P FA ) Question: What is the optimal trade-off curve? What is the optimal test achieving the curve? 8 / 30 I-Hsiang Wang IT Lecture 8 Part II

9 Randomized Testing Algorithm Randomized tests include deterministic tests as special cases. Definition 2 (Randomized Test) A randomized test decides ˆθ = 1 with probability ϕ (x) and ˆθ = 0 with probability 1 ϕ (x), where ϕ is a mapping ϕ : X [0, 1]. Note: A randomized test is characterized by ϕ, as in deterministic tests. Definition 3 (Randomized LRT) A randomized likelihood ratio test (LRT) is a test ϕ τ,γ, parametrized by cosntants τ > 0 and γ (0, 1), defined as follows: 1 if P 1 (x) > τp 0 (x) ϕ τ,γ (x) = γ if P 1 (x) = τp 0 (x). 0 if P 1 (x) < τp 0 (x) 9 / 30 I-Hsiang Wang IT Lecture 8 Part II

10 Randomized LRT Achieves the Optimal Trade-Off Consider the following optimization problem: Neyman-Pearson Problem minimize ϕ:x [0,1] subject to β ϕ α ϕ α Theorem 2 (Neyman-Pearson) A randomized LRT ϕ τ,γ with the parameters (τ, γ ) satisfying α = α ϕτ,γ, attains optimality for the Neyman-Pearson Problem. 10 / 30 I-Hsiang Wang IT Lecture 8 Part II

11 pf: First argue that for any α (0, 1), one can find (τ, γ ) such that α = α ϕτ,γ = ϕ τ,γ (x) P 0 (x) x X = P 0 (x) + γ P 0 (x) x: L(x)>τ x: L(x)=τ For any test ϕ, due to a similar argument as in Theorem 1, we have x X, (ϕ τ,γ (x) ϕ (x)) (P 1 (x) τ P 0 (x)) 0. Summing over all x X, similarly we get ( βϕ β ϕτ,γ ) + τ ( α ϕ α ϕτ,γ ) 0 Hence, for any feasible test ϕ with α ϕ α = α ϕτ,γ, its probability of type II error β ϕ β ϕτ,γ. 11 / 30 I-Hsiang Wang IT Lecture 8 Part II

12 Bayesian Setup Sometimes prior probabilities of the two hypotheses are known: π θ P {H θ is true}, θ = 0, 1, π 0 + π 1 = 1. In this sense, one can view the index Θ as a (binary) random variable with (prior) distribution P {Θ = θ} = π θ, for θ = 0, 1. With prior probabilities, it then makes sense to talk about the average probability of error for a test ϕ, or more generally, the average cost (risk): { P e (ϕ) π 0 α ϕ + π 1 β ϕ = E Θ,X [1 Θ ˆΘ }], ] R (ϕ) E Θ,X [r Θ, ˆΘ. The Bayesian hypothesis testing problem is to test the two hypotheses with knowledge of prior probabilities so that the average probability of error (or in general, a risk function) is minimized. 12 / 30 I-Hsiang Wang IT Lecture 8 Part II

13 Minimizing Bayes Risk Consider the following problem of minimizing Bayes risk. minimize ϕ:x [0,1] with known Bayesian Problem R (ϕ) E Θ,X [r Θ, ˆΘ (π 0, π 1 ) and r θ,ˆθ ] Theorem 3 (LRT is an Optimal Bayesian Test) Assume r 0,0 < r 0,1 and r 1,1 < r 1,0. A deterministic LRT ϕ τ threshold τ = (r 0,1 r 0,0 ) π 0 (r 1,0 r 1,1 ) π 1 attains optimality for the Bayesian Problem. with 13 / 30 I-Hsiang Wang IT Lecture 8 Part II

14 pf: R (ϕ) = r 0,0 π 0 P 0 (x) (1 ϕ (x)) + r 0,1 π 0 P 0 (x) ϕ (x) x X x X + r 1,0 π 1 P 1 (x) (1 ϕ (x)) + r 1,1 π 1 P 1 (x) ϕ (x) x X x X = r 0,0 π 0 + (r 0,1 r 0,0 ) π 0 P 0 (x) ϕ (x) x X + r 1,0 π 1 + (r 1,1 r 1,0 ) π 1 P 1 (x) ϕ (x) = x X x X ( ) {}}{ [ (r0,1 r 0,0 ) π 0 P 0 (x) (r 1,1 r 1,0 ) π 1 P 1 (x) ] ϕ (x) + r 0,0 π 0 + r 1,0 π 1. For each x X, we shall choose ϕ (x) [0, 1] such that ( ) is minimized. It is then obvious that we should choose { 1 if (r 0,1 r 0,0 ) π 0 P 0 (x) (r 1,1 r 1,0 ) π 1 P 1 (x) < 0 ϕ (x) = 0 if (r 0,1 r 0,0 ) π 0 P 0 (x) (r 1,1 r 1,0 ) π 1 P 1 (x) / 30 I-Hsiang Wang IT Lecture 8 Part II

15 Discussions For binary hypothesis testing problems, the likelihood ratio L (x) P1(x) P 0 (x) turns out to be a sufficient statistics. Moreover, a likelihood ratio test (LRT) is optimal both in the Bayesian and Neyman-Pearson settings. Extensions include M-ary hypothesis testing Minimax risk optimization (with unknown prior) Composite hypothesis testing, etc. Here we do not pursue these directions further. Instead, we would like to explore the asymptotic behavior of hypothesis testing, and the connection with information theoretic tools. 15 / 30 I-Hsiang Wang IT Lecture 8 Part II

16 1 Hypothesis Testing 16 / 30 I-Hsiang Wang IT Lecture 8 Part II

17 i.i.d. Observations So far we focus on the general setting where the observation space X can be arbitrary alphabets. In the following, we consider product space X n, length-n observation sequence X n drawn i.i.d. from one of the two distributions, and the two hypotheses are H 0 : X i i.i.d. P 0, i = 1, 2,..., n H 1 : X i i.i.d. P 1, i = 1, 2,..., n The corresponding probability of errors are denoted by α (n) P (n) FA P {H 1 is chosen H 0 } β (n) P (n) MD P {H 0 is chosen H 1 } Throughout the lecture we assume X = {a 1, a 2,..., a d } is a finite set. 17 / 30 I-Hsiang Wang IT Lecture 8 Part II

18 LRT under i.i.d. Observation (1) With i.i.d. observation, the likelihood ratio of a sequence x n X n is L (x n ) = n i=1 P 1 (x i ) P = 0(x i) a X ( ) N(a x n ) P1 (a) ( P 0(a) = P1 P a X 0(a)) nπ(a xn ), where N (a x n ) # of a s in x n and π (a x n ) 1 n N (a xn ) is the relative frequency of occurrence of symbol a in sequence x n. Note: From the above manipulation, we see that the collection of relative frequency of occurrence (as a X -dim probabilty vector), Π x n [ π (a 1 x n ) π (a 2 x n ) π (a d x n ) ] T, called the type of sequence x n, is a sufficient statistics for all the previously mentioned hypothesis testing problems. 18 / 30 I-Hsiang Wang IT Lecture 8 Part II

19 LRT under i.i.d. Observation (2) Let us further manipulate the LRT, by taking log likelihood ratio: L (x n ) τ n log (L (x n )) log τ n nπ (a x n ) log a X ( P1(a) P 0 (a) π (a x n ) log a X π (a x n ) log a X ) log τ n ) ( π(a x n ) P 0(a) ( π(a x n ) P 1 (a) ) 1 n log τ n D (Π x n P 0 ) D (Π x n P 1 ) 1 n log τ n 19 / 30 I-Hsiang Wang IT Lecture 8 Part II

20 Hypothesis Testing Probability Simplex Observation Space X P (X ) P1 A1 ( ) Acceptance Region of H1. (n) Acceptance F1 A0 ( ) P0 (n) Acceptance F0 Region of H0. Acceptance Region of H0. {xn Ai = decide Hi } 20 / 30 Region of H1. I-Hsiang Wang { (n) Πxn Fi IT Lecture 8 Part II } = decide Hi.

21 Probability Simplex P (X ) P 1 F (n) 1 P F (n) 0 P 0 By Sanov s Theorem, we know that α (n) = P n 0 ( F (n) 1 ) ( 2 nd(p P 0 ), β (n) = P n 1 F (n) 0 ) 2 nd(p P 1 ). 21 / 30 I-Hsiang Wang IT Lecture 8 Part II

22 Asymptotic Behaviors 1 Neyman-Pearson: β (n, ε) min ϕ n :X n [0,1] β (n) ϕ n, subject to α (n) ϕ n ε. It turns out that for all ε (0, 1), { lim 1 n n log β (n, ε) } = D (P 0 P 1 ) { 2 Bayesian: P e (n) min π 0 α (n) ϕ n:x n ϕ [0,1] n + π 1 β (n) ϕ n }. It turns out that { lim 1 n n log P e (n) } = D (P λ P 0 ) = D (P λ P 1 ) where P λ (a) (P 0(a)) λ (P 1 (a)) 1 λ x X (P 0(x)) λ (P 1 (x)) 1 λ, a X, and λ (0, 1) such that D (P λ P 0 ) = D (P λ P 1 ) 22 / 30 I-Hsiang Wang IT Lecture 8 Part II

23 in Neyman-Pearson Setup Theorem 4 (Chernoff-Stein) { For all ε (0, 1), lim 1 n n log β (n, ε) } = D (P 0 P 1 ). pf: We shall prove the achievability and the converese part separately. Achievability: construct a sequence of tests {ϕ n } with α (n) ϕ n ε for { } n sufficiently large, such that lim inf 1 n n log β(n) ϕ n D (P 0 P 1 ). Converse: for any sequence of tests {ϕ n } with α (n) ϕ n ε for n { } sufficiently large, show that lim sup 1 n log β(n) ϕ n D (P 0 P 1 ). n We use method of types to prove both the achievability and the converse. Alternatively, Chapter 11.8 of Cover&Thomas[1] uses a kind of weak typicality to prove the theorem. 23 / 30 I-Hsiang Wang IT Lecture 8 Part II

24 Achievability: Consider a deterministic test ϕ n (x n ) = 1 {D (Π x n P 0 ) δ n }, δ n 1 n ( log 1 ε + d log(n + 1)). In other words, it determines H 1 if D (Π x n P 0 ) δ n, and H 0 otherwise. Check the probability of Type I error : By Prop. 4 in Part I, we have α (n) ϕ n ( = P i.i.d. {D (Π X n P 0 ) δ n } 2 n Xi P 0 where (a) is due to our construction. δ n d log(n+1) n ) (a) = ε, Analyze the probability of Type II error : ((b) is due to Prop. 3 in Part I) β (n) ϕ n = P n 1 (T n (Q)) (b) Q P n : D(Q P 0 )<δ n P n 2 nd n, where D n Q P n : D(Q P 0 )<δ n 2 nd(q P 1) min {D (Q P 1 )}. Q P n : D(Q P 0 )<δ n Since lim n δ n = 0, we have lim n D n = D (P 0 P 1 ), and achievability is done. 24 / 30 I-Hsiang Wang IT Lecture 8 Part II

25 Converse: We prove the converse for deterministic tests. Extension to randomized tests is left as en exercise (HW6). Let A (n) i {x n ϕ n (x n ) = i}, the acceptance region of H i, for i = 0, 1. Let B (n) {x n D (Π x n P 0) < ε n}, ε n 2d log(n+1). By Prop. 4, we have n ) P n 0 (B (n) = 1 P i.i.d. {D (Π X n P 0 ) ε n } Xi P 0 ( 1 2 n ε n d log(n+1) ) n = 1 2 d log(n+1) 1 as n. ) ( ) Hence, for sufficiently large n, both P n 0 (B (n) and P n 0 A (n) 0 > 1 ε, and ( ) ) ( ) ( ) P n 0 B (n) A (n) 0 = P n 0 (B (n) + P n 0 A (n) 0 P n 0 B (n) A (n) 0 > 2 (1 ε) 1 = 1 2ε. Note B (n) = T n (Q). Hence Q n P n, D (Q n P 0) < ε n Q P n: D(Q P 0 )<ε n such that ( ) P n 0 T n (Q n ) A (n) 0 > (1 2ε) P n 0 (T n (Q n )). (1) 25 / 30 I-Hsiang Wang IT Lecture 8 Part II

26 Key Observation : Note that the probability of each sequence in the same type class is the same, under any product distribution. Hence, (1) is equivalent to T n (Q n ) A (n) > (1 2ε) T n (Q n ), ( ) which implies P n 1 T n (Q n) A (n) 0 > (1 2ε) P n 1 (T n (Q n)). 0 Hence, for sufficiently large n, Q n P n with D (Q n P 0 ) < ε n such that ( ) ( ) P n 1 A (n) 0 P n 1 T n (Q n) A (n) 0 > (1 2ε) P n 1 (T n (Q n)) where (c) is due to Prop. 3. (c) (1 2ε) P n 1 2 nd(q n P 1 ), Finally, as lim εn = 0, we have lim D (Qn P1) = D (P0 P1), and the n n converse proof is done. 26 / 30 I-Hsiang Wang IT Lecture 8 Part II

27 in Bayesian Setup Theorem 5 (Chernoff) where { lim 1 n n log P e (n) } = D (P λ P 0 ) = D (P λ P 1 ) ( P λ (a) = max log λ [0,1] (P 0(x)) λ (P 1(x)) 1 λ x X }{{} Chernoff Information CI (P 0, P 1 ) (P0(a))λ (P 1(a)) 1 λ x X (P 0 (x)) λ (P 1 (x)) 1 λ, a X, and λ (0, 1) such that D (P λ P 0 ) = D (P λ P 1 ). Note: The optimal Bayesian test (for minimizing P e ) is the maximum a posterier (MAP) test: ϕ MAP (x n ) = 1 {π 1 P n 1 (xn ) π 0 P n 0 (xn )}. 1 ) 27 / 30 I-Hsiang Wang IT Lecture 8 Part II

28 pf: The proof is based on application of large deviation in analyzing the optimal test, MAP: ϕ MAP (x n ) = 1 {π 1 P 1 (x n ) π 0 P 0 (x n )}. Analysis of error probabilities of MAP test : ( ) ( α (n) = P n 0, β (n) = P n 1 where F (n) 1 F (n) 0 ), { F (n) 1 Q P (X ) D (Q P 0) D (Q P 1) 1 log π 0 n { F (n) 0 Q P (X ) D (Q P 0 ) D (Q P 1 ) 1 log π 0 n : By Sanov s Theorem, we have π 1 } π 1 } } } lim { 1n log n α(n) = min D (Q P 0), lim { 1n log Q F 1 n β(n) = min D (Q P 1), Q F 0 where F 1 {Q P (X ) D (Q P 0 ) D (Q P 1 ) 0} and F 0 {Q P (X ) D (Q P 0) D (Q P 1) 0}.,. 28 / 30 I-Hsiang Wang IT Lecture 8 Part II

29 Exponents : Characterizing the two exponents is equivalent to solving the two (convex) optimization problems: min D (Q P 0 ) Q F 1 min D (Q P 1 ) Q F 0 minimize (Q 1,...,Q d ) d l=1 Q l log Q l P 0 (a l ) minimize (Q 1,...,Q d ) d l=1 Q l log Q l P 1 (a l ) subject to d l=1 Q l log P 1(a l ) P 0 (a l ) 0 Q l 0, l = 1,..., d d l=1 Q l = 1 subject to d l=1 Q l log P 1(a l ) P 0 (a l ) 0 Q l 0, l = 1,..., d d l=1 Q l = 1 It turns out that both problems have a common optimal solution P λ (a) = (P 0(a)) λ (P 1 (a)) 1 λ x X (P 0 (x)) λ (P 1 (x)) 1 λ, a X, with λ [0, 1] such that D (P λ P 0) = D (P λ P 1). 29 / 30 I-Hsiang Wang IT Lecture 8 Part II

30 Hence, both types of error probabilities have the same exponent, and so does the average error probability. This completes the proof of the first part. Chernoff Information : To show that ( CI (P 0, P 1 ) max log λ [0,1] simply observe that a X D (P λ P 0 ) = D (P λ P 1 ) 1 (P 0 (x)) λ (P 1 (x)) 1 λ x X (P 0 (a)) λ (P 1 (a)) 1 λ (log P 0 (a) log P 1 (a)) = 0 ) = D (P λ P 0 ), 1 D (P λ P 0) = D (P λ P 1) = log (P 0 (x)) λ (P 1 (x)) 1 λ. Proof complete. x X 30 / 30 I-Hsiang Wang IT Lecture 8 Part II

Lecture 8: Information Theory and Statistics

Lecture 8: Information Theory and Statistics Lecture 8: Information Theory and Statistics Part II: Hypothesis Testing and I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 23, 2015 1 / 50 I-Hsiang

More information

Lecture 7 Introduction to Statistical Decision Theory

Lecture 7 Introduction to Statistical Decision Theory Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7

More information

Lecture 22: Error exponents in hypothesis testing, GLRT

Lecture 22: Error exponents in hypothesis testing, GLRT 10-704: Information Processing and Learning Spring 2012 Lecture 22: Error exponents in hypothesis testing, GLRT Lecturer: Aarti Singh Scribe: Aarti Singh Disclaimer: These notes have not been subjected

More information

10-704: Information Processing and Learning Fall Lecture 24: Dec 7

10-704: Information Processing and Learning Fall Lecture 24: Dec 7 0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 24: Dec 7 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy of

More information

Detection theory 101 ELEC-E5410 Signal Processing for Communications

Detection theory 101 ELEC-E5410 Signal Processing for Communications Detection theory 101 ELEC-E5410 Signal Processing for Communications Binary hypothesis testing Null hypothesis H 0 : e.g. noise only Alternative hypothesis H 1 : signal + noise p(x;h 0 ) γ p(x;h 1 ) Trade-off

More information

Information Theory and Hypothesis Testing

Information Theory and Hypothesis Testing Summer School on Game Theory and Telecommunications Campione, 7-12 September, 2014 Information Theory and Hypothesis Testing Mauro Barni University of Siena September 8 Review of some basic results linking

More information

Anonymous Heterogeneous Distributed Detection: Optimal Decision Rules, Error Exponents, and the Price of Anonymity

Anonymous Heterogeneous Distributed Detection: Optimal Decision Rules, Error Exponents, and the Price of Anonymity Anonymous Heterogeneous Distributed Detection: Optimal Decision Rules, Error Exponents, and the Price of Anonymity Wei-Ning Chen and I-Hsiang Wang arxiv:805.03554v2 [cs.it] 29 Jul 208 Abstract We explore

More information

INFORMATION THEORY AND STATISTICS

INFORMATION THEORY AND STATISTICS CHAPTER INFORMATION THEORY AND STATISTICS We now explore the relationship between information theory and statistics. We begin by describing the method of types, which is a powerful technique in large deviation

More information

Chapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1)

Chapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1) Chapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1) Detection problems can usually be casted as binary or M-ary hypothesis testing problems. Applications: This chapter: Simple hypothesis

More information

Lecture 4 Noisy Channel Coding

Lecture 4 Noisy Channel Coding Lecture 4 Noisy Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 9, 2015 1 / 56 I-Hsiang Wang IT Lecture 4 The Channel Coding Problem

More information

Lecture 4 Channel Coding

Lecture 4 Channel Coding Capacity and the Weak Converse Lecture 4 Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 15, 2014 1 / 16 I-Hsiang Wang NIT Lecture 4 Capacity

More information

Uncertainty. Jayakrishnan Unnikrishnan. CSL June PhD Defense ECE Department

Uncertainty. Jayakrishnan Unnikrishnan. CSL June PhD Defense ECE Department Decision-Making under Statistical Uncertainty Jayakrishnan Unnikrishnan PhD Defense ECE Department University of Illinois at Urbana-Champaign CSL 141 12 June 2010 Statistical Decision-Making Relevant in

More information

Detection theory. H 0 : x[n] = w[n]

Detection theory. H 0 : x[n] = w[n] Detection Theory Detection theory A the last topic of the course, we will briefly consider detection theory. The methods are based on estimation theory and attempt to answer questions such as Is a signal

More information

Detection Theory. Chapter 3. Statistical Decision Theory I. Isael Diaz Oct 26th 2010

Detection Theory. Chapter 3. Statistical Decision Theory I. Isael Diaz Oct 26th 2010 Detection Theory Chapter 3. Statistical Decision Theory I. Isael Diaz Oct 26th 2010 Outline Neyman-Pearson Theorem Detector Performance Irrelevant Data Minimum Probability of Error Bayes Risk Multiple

More information

DETECTION theory deals primarily with techniques for

DETECTION theory deals primarily with techniques for ADVANCED SIGNAL PROCESSING SE Optimum Detection of Deterministic and Random Signals Stefan Tertinek Graz University of Technology turtle@sbox.tugraz.at Abstract This paper introduces various methods for

More information

Definition 3.1 A statistical hypothesis is a statement about the unknown values of the parameters of the population distribution.

Definition 3.1 A statistical hypothesis is a statement about the unknown values of the parameters of the population distribution. Hypothesis Testing Definition 3.1 A statistical hypothesis is a statement about the unknown values of the parameters of the population distribution. Suppose the family of population distributions is indexed

More information

Lecture 12 November 3

Lecture 12 November 3 STATS 300A: Theory of Statistics Fall 2015 Lecture 12 November 3 Lecturer: Lester Mackey Scribe: Jae Hyuck Park, Christian Fong Warning: These notes may contain factual and/or typographic errors. 12.1

More information

Detection and Estimation Theory

Detection and Estimation Theory ESE 524 Detection and Estimation Theory Joseph A. O Sullivan Samuel C. Sachs Professor Electronic Systems and Signals Research Laboratory Electrical and Systems Engineering Washington University 2 Urbauer

More information

The University of Hong Kong Department of Statistics and Actuarial Science STAT2802 Statistical Models Tutorial Solutions Solutions to Problems 71-80

The University of Hong Kong Department of Statistics and Actuarial Science STAT2802 Statistical Models Tutorial Solutions Solutions to Problems 71-80 The University of Hong Kong Department of Statistics and Actuarial Science STAT2802 Statistical Models Tutorial Solutions Solutions to Problems 71-80 71. Decide in each case whether the hypothesis is simple

More information

STAT 135 Lab 5 Bootstrapping and Hypothesis Testing

STAT 135 Lab 5 Bootstrapping and Hypothesis Testing STAT 135 Lab 5 Bootstrapping and Hypothesis Testing Rebecca Barter March 2, 2015 The Bootstrap Bootstrap Suppose that we are interested in estimating a parameter θ from some population with members x 1,...,

More information

ECE531 Lecture 6: Detection of Discrete-Time Signals with Random Parameters

ECE531 Lecture 6: Detection of Discrete-Time Signals with Random Parameters ECE531 Lecture 6: Detection of Discrete-Time Signals with Random Parameters D. Richard Brown III Worcester Polytechnic Institute 26-February-2009 Worcester Polytechnic Institute D. Richard Brown III 26-February-2009

More information

Review. December 4 th, Review

Review. December 4 th, Review December 4 th, 2017 Att. Final exam: Course evaluation Friday, 12/14/2018, 10:30am 12:30pm Gore Hall 115 Overview Week 2 Week 4 Week 7 Week 10 Week 12 Chapter 6: Statistics and Sampling Distributions Chapter

More information

Lecture 5 Channel Coding over Continuous Channels

Lecture 5 Channel Coding over Continuous Channels Lecture 5 Channel Coding over Continuous Channels I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw November 14, 2014 1 / 34 I-Hsiang Wang NIT Lecture 5 From

More information

Lecture 21. Hypothesis Testing II

Lecture 21. Hypothesis Testing II Lecture 21. Hypothesis Testing II December 7, 2011 In the previous lecture, we dened a few key concepts of hypothesis testing and introduced the framework for parametric hypothesis testing. In the parametric

More information

16.1 Bounding Capacity with Covering Number

16.1 Bounding Capacity with Covering Number ECE598: Information-theoretic methods in high-dimensional statistics Spring 206 Lecture 6: Upper Bounds for Density Estimation Lecturer: Yihong Wu Scribe: Yang Zhang, Apr, 206 So far we have been mostly

More information

Applications of Information Geometry to Hypothesis Testing and Signal Detection

Applications of Information Geometry to Hypothesis Testing and Signal Detection CMCAA 2016 Applications of Information Geometry to Hypothesis Testing and Signal Detection Yongqiang Cheng National University of Defense Technology July 2016 Outline 1. Principles of Information Geometry

More information

Chapters 10. Hypothesis Testing

Chapters 10. Hypothesis Testing Chapters 10. Hypothesis Testing Some examples of hypothesis testing 1. Toss a coin 100 times and get 62 heads. Is this coin a fair coin? 2. Is the new treatment on blood pressure more effective than the

More information

40.530: Statistics. Professor Chen Zehua. Singapore University of Design and Technology

40.530: Statistics. Professor Chen Zehua. Singapore University of Design and Technology Singapore University of Design and Technology Lecture 9: Hypothesis testing, uniformly most powerful tests. The Neyman-Pearson framework Let P be the family of distributions of concern. The Neyman-Pearson

More information

Ch. 5 Hypothesis Testing

Ch. 5 Hypothesis Testing Ch. 5 Hypothesis Testing The current framework of hypothesis testing is largely due to the work of Neyman and Pearson in the late 1920s, early 30s, complementing Fisher s work on estimation. As in estimation,

More information

10. Composite Hypothesis Testing. ECE 830, Spring 2014

10. Composite Hypothesis Testing. ECE 830, Spring 2014 10. Composite Hypothesis Testing ECE 830, Spring 2014 1 / 25 In many real world problems, it is difficult to precisely specify probability distributions. Our models for data may involve unknown parameters

More information

Topic 15: Simple Hypotheses

Topic 15: Simple Hypotheses Topic 15: November 10, 2009 In the simplest set-up for a statistical hypothesis, we consider two values θ 0, θ 1 in the parameter space. We write the test as H 0 : θ = θ 0 versus H 1 : θ = θ 1. H 0 is

More information

ECE531 Lecture 4b: Composite Hypothesis Testing

ECE531 Lecture 4b: Composite Hypothesis Testing ECE531 Lecture 4b: Composite Hypothesis Testing D. Richard Brown III Worcester Polytechnic Institute 16-February-2011 Worcester Polytechnic Institute D. Richard Brown III 16-February-2011 1 / 44 Introduction

More information

HYPOTHESIS TESTING: FREQUENTIST APPROACH.

HYPOTHESIS TESTING: FREQUENTIST APPROACH. HYPOTHESIS TESTING: FREQUENTIST APPROACH. These notes summarize the lectures on (the frequentist approach to) hypothesis testing. You should be familiar with the standard hypothesis testing from previous

More information

Lecture 5: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics. 1 Executive summary

Lecture 5: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics. 1 Executive summary ECE 830 Spring 207 Instructor: R. Willett Lecture 5: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics Executive summary In the last lecture we saw that the likelihood

More information

Let us first identify some classes of hypotheses. simple versus simple. H 0 : θ = θ 0 versus H 1 : θ = θ 1. (1) one-sided

Let us first identify some classes of hypotheses. simple versus simple. H 0 : θ = θ 0 versus H 1 : θ = θ 1. (1) one-sided Let us first identify some classes of hypotheses. simple versus simple H 0 : θ = θ 0 versus H 1 : θ = θ 1. (1) one-sided H 0 : θ θ 0 versus H 1 : θ > θ 0. (2) two-sided; null on extremes H 0 : θ θ 1 or

More information

If there exists a threshold k 0 such that. then we can take k = k 0 γ =0 and achieve a test of size α. c 2004 by Mark R. Bell,

If there exists a threshold k 0 such that. then we can take k = k 0 γ =0 and achieve a test of size α. c 2004 by Mark R. Bell, Recall The Neyman-Pearson Lemma Neyman-Pearson Lemma: Let Θ = {θ 0, θ }, and let F θ0 (x) be the cdf of the random vector X under hypothesis and F θ (x) be its cdf under hypothesis. Assume that the cdfs

More information

Parameter Estimation, Sampling Distributions & Hypothesis Testing

Parameter Estimation, Sampling Distributions & Hypothesis Testing Parameter Estimation, Sampling Distributions & Hypothesis Testing Parameter Estimation & Hypothesis Testing In doing research, we are usually interested in some feature of a population distribution (which

More information

Lecture notes on statistical decision theory Econ 2110, fall 2013

Lecture notes on statistical decision theory Econ 2110, fall 2013 Lecture notes on statistical decision theory Econ 2110, fall 2013 Maximilian Kasy March 10, 2014 These lecture notes are roughly based on Robert, C. (2007). The Bayesian choice: from decision-theoretic

More information

Sequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process

Sequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process Applied Mathematical Sciences, Vol. 4, 2010, no. 62, 3083-3093 Sequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process Julia Bondarenko Helmut-Schmidt University Hamburg University

More information

Master s Written Examination

Master s Written Examination Master s Written Examination Option: Statistics and Probability Spring 016 Full points may be obtained for correct answers to eight questions. Each numbered question which may have several parts is worth

More information

Detection and Estimation Chapter 1. Hypothesis Testing

Detection and Estimation Chapter 1. Hypothesis Testing Detection and Estimation Chapter 1. Hypothesis Testing Husheng Li Min Kao Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville Spring, 2015 1/20 Syllabus Homework:

More information

Chapter 7. Hypothesis Testing

Chapter 7. Hypothesis Testing Chapter 7. Hypothesis Testing Joonpyo Kim June 24, 2017 Joonpyo Kim Ch7 June 24, 2017 1 / 63 Basic Concepts of Testing Suppose that our interest centers on a random variable X which has density function

More information

Hypothesis Testing - Frequentist

Hypothesis Testing - Frequentist Frequentist Hypothesis Testing - Frequentist Compare two hypotheses to see which one better explains the data. Or, alternatively, what is the best way to separate events into two classes, those originating

More information

Chapter 8. Hypothesis Testing. Po-Ning Chen. Department of Communications Engineering. National Chiao-Tung University. Hsin Chu, Taiwan 30050

Chapter 8. Hypothesis Testing. Po-Ning Chen. Department of Communications Engineering. National Chiao-Tung University. Hsin Chu, Taiwan 30050 Chapter 8 Hypothesis Testing Po-Ning Chen Department of Communications Engineering National Chiao-Tung University Hsin Chu, Taiwan 30050 Error exponent and divergence II:8-1 Definition 8.1 (exponent) Arealnumbera

More information

Fundamentals of Statistical Signal Processing Volume II Detection Theory

Fundamentals of Statistical Signal Processing Volume II Detection Theory Fundamentals of Statistical Signal Processing Volume II Detection Theory Steven M. Kay University of Rhode Island PH PTR Prentice Hall PTR Upper Saddle River, New Jersey 07458 http://www.phptr.com Contents

More information

Partitioning the Parameter Space. Topic 18 Composite Hypotheses

Partitioning the Parameter Space. Topic 18 Composite Hypotheses Topic 18 Composite Hypotheses Partitioning the Parameter Space 1 / 10 Outline Partitioning the Parameter Space 2 / 10 Partitioning the Parameter Space Simple hypotheses limit us to a decision between one

More information

Part III. A Decision-Theoretic Approach and Bayesian testing

Part III. A Decision-Theoretic Approach and Bayesian testing Part III A Decision-Theoretic Approach and Bayesian testing 1 Chapter 10 Bayesian Inference as a Decision Problem The decision-theoretic framework starts with the following situation. We would like to

More information

Some General Types of Tests

Some General Types of Tests Some General Types of Tests We may not be able to find a UMP or UMPU test in a given situation. In that case, we may use test of some general class of tests that often have good asymptotic properties.

More information

F2E5216/TS1002 Adaptive Filtering and Change Detection. Course Organization. Lecture plan. The Books. Lecture 1

F2E5216/TS1002 Adaptive Filtering and Change Detection. Course Organization. Lecture plan. The Books. Lecture 1 Adaptive Filtering and Change Detection Bo Wahlberg (KTH and Fredrik Gustafsson (LiTH Course Organization Lectures and compendium: Theory, Algorithms, Applications, Evaluation Toolbox and manual: Algorithms,

More information

Lecture 13: p-values and union intersection tests

Lecture 13: p-values and union intersection tests Lecture 13: p-values and union intersection tests p-values After a hypothesis test is done, one method of reporting the result is to report the size α of the test used to reject H 0 or accept H 0. If α

More information

Optimum Joint Detection and Estimation

Optimum Joint Detection and Estimation 20 IEEE International Symposium on Information Theory Proceedings Optimum Joint Detection and Estimation George V. Moustakides Department of Electrical and Computer Engineering University of Patras, 26500

More information

Introduction to Statistical Inference

Introduction to Statistical Inference Structural Health Monitoring Using Statistical Pattern Recognition Introduction to Statistical Inference Presented by Charles R. Farrar, Ph.D., P.E. Outline Introduce statistical decision making for Structural

More information

EECS 750. Hypothesis Testing with Communication Constraints

EECS 750. Hypothesis Testing with Communication Constraints EECS 750 Hypothesis Testing with Communication Constraints Name: Dinesh Krithivasan Abstract In this report, we study a modification of the classical statistical problem of bivariate hypothesis testing.

More information

IN HYPOTHESIS testing problems, a decision-maker aims

IN HYPOTHESIS testing problems, a decision-maker aims IEEE SIGNAL PROCESSING LETTERS, VOL. 25, NO. 12, DECEMBER 2018 1845 On the Optimality of Likelihood Ratio Test for Prospect Theory-Based Binary Hypothesis Testing Sinan Gezici, Senior Member, IEEE, and

More information

Detection and Estimation Theory

Detection and Estimation Theory ESE 524 Detection and Estimation Theory Joseph A. O Sullivan Samuel C. Sachs Professor Electronic Systems and Signals Research Laboratory Electrical and Systems Engineering Washington University 2 Urbauer

More information

Chapters 10. Hypothesis Testing

Chapters 10. Hypothesis Testing Chapters 10. Hypothesis Testing Some examples of hypothesis testing 1. Toss a coin 100 times and get 62 heads. Is this coin a fair coin? 2. Is the new treatment more effective than the old one? 3. Quality

More information

Topic 3: Hypothesis Testing

Topic 3: Hypothesis Testing CS 8850: Advanced Machine Learning Fall 07 Topic 3: Hypothesis Testing Instructor: Daniel L. Pimentel-Alarcón c Copyright 07 3. Introduction One of the simplest inference problems is that of deciding between

More information

Lecture Testing Hypotheses: The Neyman-Pearson Paradigm

Lecture Testing Hypotheses: The Neyman-Pearson Paradigm Math 408 - Mathematical Statistics Lecture 29-30. Testing Hypotheses: The Neyman-Pearson Paradigm April 12-15, 2013 Konstantin Zuev (USC) Math 408, Lecture 29-30 April 12-15, 2013 1 / 12 Agenda Example:

More information

On the Optimality of Likelihood Ratio Test for Prospect Theory Based Binary Hypothesis Testing

On the Optimality of Likelihood Ratio Test for Prospect Theory Based Binary Hypothesis Testing 1 On the Optimality of Likelihood Ratio Test for Prospect Theory Based Binary Hypothesis Testing Sinan Gezici, Senior Member, IEEE, and Pramod K. Varshney, Life Fellow, IEEE Abstract In this letter, the

More information

Lecture 2. We now introduce some fundamental tools in martingale theory, which are useful in controlling the fluctuation of martingales.

Lecture 2. We now introduce some fundamental tools in martingale theory, which are useful in controlling the fluctuation of martingales. Lecture 2 1 Martingales We now introduce some fundamental tools in martingale theory, which are useful in controlling the fluctuation of martingales. 1.1 Doob s inequality We have the following maximal

More information

Bayesian Social Learning with Random Decision Making in Sequential Systems

Bayesian Social Learning with Random Decision Making in Sequential Systems Bayesian Social Learning with Random Decision Making in Sequential Systems Yunlong Wang supervised by Petar M. Djurić Department of Electrical and Computer Engineering Stony Brook University Stony Brook,

More information

Hypothesis Testing. BS2 Statistical Inference, Lecture 11 Michaelmas Term Steffen Lauritzen, University of Oxford; November 15, 2004

Hypothesis Testing. BS2 Statistical Inference, Lecture 11 Michaelmas Term Steffen Lauritzen, University of Oxford; November 15, 2004 Hypothesis Testing BS2 Statistical Inference, Lecture 11 Michaelmas Term 2004 Steffen Lauritzen, University of Oxford; November 15, 2004 Hypothesis testing We consider a family of densities F = {f(x; θ),

More information

Hypothesis Test. The opposite of the null hypothesis, called an alternative hypothesis, becomes

Hypothesis Test. The opposite of the null hypothesis, called an alternative hypothesis, becomes Neyman-Pearson paradigm. Suppose that a researcher is interested in whether the new drug works. The process of determining whether the outcome of the experiment points to yes or no is called hypothesis

More information

Composite Hypotheses and Generalized Likelihood Ratio Tests

Composite Hypotheses and Generalized Likelihood Ratio Tests Composite Hypotheses and Generalized Likelihood Ratio Tests Rebecca Willett, 06 In many real world problems, it is difficult to precisely specify probability distributions. Our models for data may involve

More information

Institute of Actuaries of India

Institute of Actuaries of India Institute of Actuaries of India Subject CT3 Probability & Mathematical Statistics May 2011 Examinations INDICATIVE SOLUTION Introduction The indicative solution has been written by the Examiners with the

More information

Chapter 6. Hypothesis Tests Lecture 20: UMP tests and Neyman-Pearson lemma

Chapter 6. Hypothesis Tests Lecture 20: UMP tests and Neyman-Pearson lemma Chapter 6. Hypothesis Tests Lecture 20: UMP tests and Neyman-Pearson lemma Theory of testing hypotheses X: a sample from a population P in P, a family of populations. Based on the observed X, we test a

More information

Strong Converse and Stein s Lemma in the Quantum Hypothesis Testing

Strong Converse and Stein s Lemma in the Quantum Hypothesis Testing Strong Converse and Stein s Lemma in the Quantum Hypothesis Testing arxiv:uant-ph/9906090 v 24 Jun 999 Tomohiro Ogawa and Hiroshi Nagaoka Abstract The hypothesis testing problem of two uantum states is

More information

APPENDIX 1 NEYMAN PEARSON CRITERIA

APPENDIX 1 NEYMAN PEARSON CRITERIA 54 APPENDIX NEYMAN PEARSON CRITERIA The design approaches for detectors directly follow the theory of hypothesis testing. The primary approaches to hypothesis testing problem are the classical approach

More information

CSE 312 Final Review: Section AA

CSE 312 Final Review: Section AA CSE 312 TAs December 8, 2011 General Information General Information Comprehensive Midterm General Information Comprehensive Midterm Heavily weighted toward material after the midterm Pre-Midterm Material

More information

Sequential Detection. Changes: an overview. George V. Moustakides

Sequential Detection. Changes: an overview. George V. Moustakides Sequential Detection of Changes: an overview George V. Moustakides Outline Sequential hypothesis testing and Sequential detection of changes The Sequential Probability Ratio Test (SPRT) for optimum hypothesis

More information

2. What are the tradeoffs among different measures of error (e.g. probability of false alarm, probability of miss, etc.)?

2. What are the tradeoffs among different measures of error (e.g. probability of false alarm, probability of miss, etc.)? ECE 830 / CS 76 Spring 06 Instructors: R. Willett & R. Nowak Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics Executive summary In the last lecture we

More information

Mathematical Statistics

Mathematical Statistics Mathematical Statistics MAS 713 Chapter 8 Previous lecture: 1 Bayesian Inference 2 Decision theory 3 Bayesian Vs. Frequentist 4 Loss functions 5 Conjugate priors Any questions? Mathematical Statistics

More information

Statistics. Statistics

Statistics. Statistics The main aims of statistics 1 1 Choosing a model 2 Estimating its parameter(s) 1 point estimates 2 interval estimates 3 Testing hypotheses Distributions used in statistics: χ 2 n-distribution 2 Let X 1,

More information

Some Asymptotic Bayesian Inference (background to Chapter 2 of Tanner s book)

Some Asymptotic Bayesian Inference (background to Chapter 2 of Tanner s book) Some Asymptotic Bayesian Inference (background to Chapter 2 of Tanner s book) Principal Topics The approach to certainty with increasing evidence. The approach to consensus for several agents, with increasing

More information

Hypothesis Testing. Testing Hypotheses MIT Dr. Kempthorne. Spring MIT Testing Hypotheses

Hypothesis Testing. Testing Hypotheses MIT Dr. Kempthorne. Spring MIT Testing Hypotheses Testing Hypotheses MIT 18.443 Dr. Kempthorne Spring 2015 1 Outline Hypothesis Testing 1 Hypothesis Testing 2 Hypothesis Testing: Statistical Decision Problem Two coins: Coin 0 and Coin 1 P(Head Coin 0)

More information

Information Theory and Statistics, Part I

Information Theory and Statistics, Part I Information Theory and Statistics, Part I Information Theory 2013 Lecture 6 George Mathai May 16, 2013 Outline This lecture will cover Method of Types. Law of Large Numbers. Universal Source Coding. Large

More information

Detection Theory. Composite tests

Detection Theory. Composite tests Composite tests Chapter 5: Correction Thu I claimed that the above, which is the most general case, was captured by the below Thu Chapter 5: Correction Thu I claimed that the above, which is the most general

More information

Lecture 2: Basic Concepts of Statistical Decision Theory

Lecture 2: Basic Concepts of Statistical Decision Theory EE378A Statistical Signal Processing Lecture 2-03/31/2016 Lecture 2: Basic Concepts of Statistical Decision Theory Lecturer: Jiantao Jiao, Tsachy Weissman Scribe: John Miller and Aran Nayebi In this lecture

More information

Generalization bounds

Generalization bounds Advanced Course in Machine Learning pring 200 Generalization bounds Handouts are jointly prepared by hie Mannor and hai halev-hwartz he problem of characterizing learnability is the most basic question

More information

Topic 10: Hypothesis Testing

Topic 10: Hypothesis Testing Topic 10: Hypothesis Testing Course 003, 2016 Page 0 The Problem of Hypothesis Testing A statistical hypothesis is an assertion or conjecture about the probability distribution of one or more random variables.

More information

Statistical hypothesis testing The parametric and nonparametric cases. Madalina Olteanu, Université Paris 1

Statistical hypothesis testing The parametric and nonparametric cases. Madalina Olteanu, Université Paris 1 Statistical hypothesis testing The parametric and nonparametric cases Madalina Olteanu, Université Paris 1 2016-2017 Contents 1 Parametric hypothesis testing 3 1.1 An introduction on statistical hypothesis

More information

ST5215: Advanced Statistical Theory

ST5215: Advanced Statistical Theory Department of Statistics & Applied Probability Wednesday, October 5, 2011 Lecture 13: Basic elements and notions in decision theory Basic elements X : a sample from a population P P Decision: an action

More information

6.1 Variational representation of f-divergences

6.1 Variational representation of f-divergences ECE598: Information-theoretic methods in high-dimensional statistics Spring 2016 Lecture 6: Variational representation, HCR and CR lower bounds Lecturer: Yihong Wu Scribe: Georgios Rovatsos, Feb 11, 2016

More information

Hypothesis testing: theory and methods

Hypothesis testing: theory and methods Statistical Methods Warsaw School of Economics November 3, 2017 Statistical hypothesis is the name of any conjecture about unknown parameters of a population distribution. The hypothesis should be verifiable

More information

Detection and Estimation Theory

Detection and Estimation Theory ESE 524 Detection and Estimation Theory Joseph A. O Sullivan Samuel C. Sachs Professor Electronic Systems and Signals Research Laboratory Electrical and Systems Engineering Washington University 211 Urbauer

More information

Chapter 9: Hypothesis Testing Sections

Chapter 9: Hypothesis Testing Sections Chapter 9: Hypothesis Testing Sections 9.1 Problems of Testing Hypotheses 9.2 Testing Simple Hypotheses 9.3 Uniformly Most Powerful Tests Skip: 9.4 Two-Sided Alternatives 9.6 Comparing the Means of Two

More information

Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016

Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,

More information

Introduction to Bayesian Statistics

Introduction to Bayesian Statistics Bayesian Parameter Estimation Introduction to Bayesian Statistics Harvey Thornburg Center for Computer Research in Music and Acoustics (CCRMA) Department of Music, Stanford University Stanford, California

More information

Testing Statistical Hypotheses

Testing Statistical Hypotheses E.L. Lehmann Joseph P. Romano Testing Statistical Hypotheses Third Edition 4y Springer Preface vii I Small-Sample Theory 1 1 The General Decision Problem 3 1.1 Statistical Inference and Statistical Decisions

More information

Lecture 3: Semantics of Propositional Logic

Lecture 3: Semantics of Propositional Logic Lecture 3: Semantics of Propositional Logic 1 Semantics of Propositional Logic Every language has two aspects: syntax and semantics. While syntax deals with the form or structure of the language, it is

More information

Statistical Inference

Statistical Inference Statistical Inference Classical and Bayesian Methods Revision Class for Midterm Exam AMS-UCSC Th Feb 9, 2012 Winter 2012. Session 1 (Revision Class) AMS-132/206 Th Feb 9, 2012 1 / 23 Topics Topics We will

More information

Lecture 7. Union bound for reducing M-ary to binary hypothesis testing

Lecture 7. Union bound for reducing M-ary to binary hypothesis testing Lecture 7 Agenda for the lecture M-ary hypothesis testing and the MAP rule Union bound for reducing M-ary to binary hypothesis testing Introduction of the channel coding problem 7.1 M-ary hypothesis testing

More information

1 Differential Privacy and Statistical Query Learning

1 Differential Privacy and Statistical Query Learning 10-806 Foundations of Machine Learning and Data Science Lecturer: Maria-Florina Balcan Lecture 5: December 07, 015 1 Differential Privacy and Statistical Query Learning 1.1 Differential Privacy Suppose

More information

Robustness and duality of maximum entropy and exponential family distributions

Robustness and duality of maximum entropy and exponential family distributions Chapter 7 Robustness and duality of maximum entropy and exponential family distributions In this lecture, we continue our study of exponential families, but now we investigate their properties in somewhat

More information

Bandits : optimality in exponential families

Bandits : optimality in exponential families Bandits : optimality in exponential families Odalric-Ambrym Maillard IHES, January 2016 Odalric-Ambrym Maillard Bandits 1 / 40 Introduction 1 Stochastic multi-armed bandits 2 Boundary crossing probabilities

More information

44 CHAPTER 2. BAYESIAN DECISION THEORY

44 CHAPTER 2. BAYESIAN DECISION THEORY 44 CHAPTER 2. BAYESIAN DECISION THEORY Problems Section 2.1 1. In the two-category case, under the Bayes decision rule the conditional error is given by Eq. 7. Even if the posterior densities are continuous,

More information

Information Theory in Intelligent Decision Making

Information Theory in Intelligent Decision Making Information Theory in Intelligent Decision Making Adaptive Systems and Algorithms Research Groups School of Computer Science University of Hertfordshire, United Kingdom June 7, 2015 Information Theory

More information

Testing Algebraic Hypotheses

Testing Algebraic Hypotheses Testing Algebraic Hypotheses Mathias Drton Department of Statistics University of Chicago 1 / 18 Example: Factor analysis Multivariate normal model based on conditional independence given hidden variable:

More information

Homework Assignment #2 for Prob-Stats, Fall 2018 Due date: Monday, October 22, 2018

Homework Assignment #2 for Prob-Stats, Fall 2018 Due date: Monday, October 22, 2018 Homework Assignment #2 for Prob-Stats, Fall 2018 Due date: Monday, October 22, 2018 Topics: consistent estimators; sub-σ-fields and partial observations; Doob s theorem about sub-σ-field measurability;

More information

LECTURE 10: NEYMAN-PEARSON LEMMA AND ASYMPTOTIC TESTING. The last equality is provided so this can look like a more familiar parametric test.

LECTURE 10: NEYMAN-PEARSON LEMMA AND ASYMPTOTIC TESTING. The last equality is provided so this can look like a more familiar parametric test. Economics 52 Econometrics Professor N.M. Kiefer LECTURE 1: NEYMAN-PEARSON LEMMA AND ASYMPTOTIC TESTING NEYMAN-PEARSON LEMMA: Lesson: Good tests are based on the likelihood ratio. The proof is easy in the

More information