Example: An experiment can either result in success or failure with probability θ and (1 θ) respectively. The experiment is performed independently
|
|
- Jason Walsh
- 5 years ago
- Views:
Transcription
1 Chapter 3 Sufficient statistics and variance reduction Let X 1,X 2,...,X n be a random sample from a certain distribution with p.m/d.f fx θ. A function T X 1,X 2,...,X n = T X of these observations is called a statistic. From a statistical point of view taking a statistic of the observations is equivalent to taking into account only part of the information in the sample. Example: An experiment can either result in success or failure with probability θ and 1 θ respectively. The experiment is performed independently n times. Let X i = { 1 if the ith repetition results in success 0 if the ith repetition results in failure Let S m = m X i and S n m = n i=m+1 X i. Consider the bivariate statistic T X = S m,s n m. This statistic gives information on how many successes are obtained in the first m experiments and on how many successes are obtained in the last n mexperiments. The information on which particular experiments the successes were obtained in is not retained; neither is the information about how many successes are obtained in the first r experiments for r m. Consider now the statistic UX = n X i. This statistic gives information on the total number of successes in the n repetitions; all other information in the sample is not retained by UX. Note, in fact, that UX retains even less information than T X. Note also that 35
2 36CHAPTER 3. SUFFICIENT STATISTICS AND VARIANCE REDUCTION UX = S m + S n m i.e. UX is a function i.e. statistic of T X. Consequently we come to the conclusion that every time we take a function of a statistic we drop some of the information. We have argued in the past that the Fisher information Iθ = I X θ = E S 2 X, where S X = d dθ log f X X θ, is a measure of the amount of information in the sample X about the parameter θ. Now, if T X is a statistic then a measure of the amount of information in T about θ can be given by the Fisher information of T defined by I T θ = E S 2 T with ST = d dθ log f T T θ where f T t θ is the p.m/d.f. of the statistic T. If ˆθ T is an unbiased estimator of θ based on the statistic T instead of on the whole sample X then the Cramér-Rao inequality becomes V ar ˆθ T I T θ Now, in view of the remarks we made about a statistic being equivalent to taking into account only part of the information in the sample, we should expect to have that I T θ I X θ 3.2 with equality holding if and only if the statistic has retained all the relevant information about θ and dropped only information which does not relate to θ. A statistic which retains all the relevant information about θ and discards only information which does not relate to θ is said to be sufficient for θ. Unfortunately, tempting as it may be, we can not adopt strict equality in 3.2 as the formal definition of sufficiency of a statistic T as this will only be possible in the cases when there is enough regularity for the Fisher Information to be defined. We need a formal definition of sufficiency which holds in all cases irrespective of whether this regularity is there or not. Formal definition of Sufficiency: A statistic TX of the observations X with p.m/d.f. f X x θ is said to sufficient for the parameter θ if the conditional distribution of X given T = t is free of θ i.e. if the conditional p.m/d.f.
3 37 f X x T=t does not involve θ. From this definition of sufficiency we have the following The factorization theorem A statistic TX, where X has joint p.m/d.f. f X x θ, is sufficient for θ if and only if f X x θ = gt, θhx for all x X n where gt,θ is a function of θ and depends on the observations only through the value t of T and hx is a function which does not involve θ. Proof. We first note that if F X,T x,t θ is the joint p.m/d.f. of X and T then f X,T x,t θ = = { fx x θ if t = Tx 0 if t Tx { fx x θ if x A t if x / A t where the set A t = {x : Tx = t} = set of all sample results for which T = t. We can understand better the result in 3.3 in terms of an example. Suppose an experiment which can result in either success or failure is repeated independently three times and on the ith repetition we record X i = 1 if we get a success and X i = 0 if we get a failure i = 1, 2, 3. Let the statistic T = 3 X i be the number of successes in the three repetitions. The possible outcomes of the sample X = X 1,X 2,X 3 and of the statistic T are shown below.
4 38CHAPTER 3. SUFFICIENT STATISTICS AND VARIANCE REDUCTION Partition sets X 1,X 2,X 3 T A 0 = 0,0,0 0 A 1 = 1,0,0 0,1,0 0,0,1 1 A 2 = 1,1,0 1,0,1 0,1,1 2 A 3 = 1,1,1 3 Clearly f X,T 0, 1, 0, 2 θ = PrX 1,X 2, X 3 = 0, 1, 0, = 0 3 X i = 2 since clearly we cannot have the result X 1, X 2,X 3 = 0, 1, 0 and at the same have 3 X i = 2. On the other hand f X,T 0, 1, 0, 1 θ = PrX 1,X 2, X 3 = 0, 1, 0, = PrX 1,X 2, X 3 = 0, 1, 0 = f X 0, 1, 0 θ 3 X i = 1. We now turn our attention to the proof of the factorization theorem. Assume first that T is sufficient for θ i.e. that f X x T=t is free of the parameter θ. Since for t = Tx f X x θ = f X,T x,t θ = f X x T=tf T t θ see 3.3 the factorization follows by taking f X x T=t hx and f T t θ gt, θ.
5 Assume now that the factorization f X x θ = gt,θhx holds for all x X n with t = Tx. It follows that f T t θ = x A t f X x θ = x A t gt,θhx = gt,θ x A t hx = gt, θht 3.4 where the set A t = {x : T x = t} = set of all sample results for which T = t. In calculating 3.4 we have assumed the observations to be discrete; if they are continuous replace summations by integrals. Further in 3.3 we have seen that f X x θ if x A f X x T=t = t f T t θ 0 if x / A t and from 3.4 and the factorization we get gt, θhx f X x T=t = gt,θht = hx if x A t Ht 0 if x / A t i.e. f X x T=t is free of θ. This completes the proof of the factorization theorem. Remark What are the implications of having the conditional p.m/d.f. f X x T=t free of θ? Given that we know that T x = t it follows that x must be situated in the set A t ; if further f X x T=t is free of θ we can conclude that once we know that x is in the set A t the probability of it being in any particular position within A t is not dependent on θ i.e. once we know that x is in the set A t information on its exact position within A t does not relate to θ. Put in another way, all the information in x relating to θ is contained in the value of T x, the information in x which is not retained by the statistic T does not relate to θ. But we have seen that a statistic T which retains all the relevant information about θ and discards only information that is not relevant to θ is what we call a sufficient statistic for θ. Result: Let TX be a statistic of the sample X whose joint distribution depends on a parameter θ. Then under certain regularity conditions on the joint p.d/m.f f X x θ of X and on the p.d/m.f f T t θ of T 39 I T θ I X θ θ Θ
6 40CHAPTER 3. SUFFICIENT STATISTICS AND VARIANCE REDUCTION with equality if and only if TX is sufficient for θ. Here [ ] 2 I T θ = E θ log f TT θ = E 2 θ log f TT θ 2 and I X θ = E [ ] 2 θ log f XX θ = E 2 θ log f XX θ. 2 Proof. The inequality I T θ I X θ will be assumed valid as a consequence of our understanding of what a statistic does and of what the Fisher information represents - although it can be rigorously proved using mathematics. That strict equality holds if and only if T is sufficient for θ follows from the factorization theorem and is left as an exercise. Remark: 1. Notice that the factorization theorem not only gives us necessary and sufficient conditions for the existence of a sufficient statistic it also identifies for us the sufficient statistic. 2. Sufficiency implies that basing inferences about θ on procedures involving sufficient statistics rather than the whole sample, will be more preferable since such procedures discard, outright, unnecessary information which does not relate to θ. In particular, in estimating θ, the best unbiased estimators based on sufficient statistics are not going to be any less efficient in the formal sense than the best unbiased estimators based on the whole sample since for T sufficient I T θ = I X θ i.e. the CRLB for unbiased estimators based on Tis the same as the CRLB for unbiased estimators based on X. Example Let X 1,X 2,...,X n be a random sample from the Bernoulli distribution i.e. { 1 with probability θ X i = 0 with probability 1 θ Hence f Xi x i θ = θ x i 1 θ 1 x i
7 for all i, Use the factorization theorem to find a sufficient statistic for θ and then confirm it is sufficient for θ with the use of the formal definition of sufficiency Solution: The joint mass function of the observations X = X 1,X 2,...,X n is f X x θ = f Xi x i θ = θ x i 1 θ 1 x i = θ P n x i 1 θ n P n x i = g x i, θ h x with h x 1. Hence by the factorization theorem n X i is a sufficient statistic for θ. Notice that since the factorization is not unique, there may be more than one sufficient statistic. For example we could have written f X x θ = θ Sm+S n m 1 θ n Sm S n m = gs m,s n m,θh x with, once again, h x 1 and S m = m x i, S n m = n i=m+1 x i. Hence by the factorization theorem m X i, n i=m+1 X i is a bivariate sufficient statistic for θ. We now show, using the formal definition, that n X i is indeed a sufficient statistic. The conditional p.m.f. of X given that TX = n X i = t is 41 f X x T = t = f X,Tx,t θ f T t θ = f Xx θ f T t θ when Tx = t and zero otherwise. The last equality was obtained using 3.3. However, the statistic T = n X i has the Binomialn,θ distribution. Hence f X x T = t == f Xx θ f T t θ = θ P n x i 1 θ n P n x i n = 1/ n t θ t t 1 θ n t which is independent of θ confirming that n X i is sufficient for θ.
8 42CHAPTER 3. SUFFICIENT STATISTICS AND VARIANCE REDUCTION Example Let X 1,X 2,...,X n be a random sample from the N µ,σ 2 distribution. Then f X x θ = 1 exp 1 2πσ 2σ x 2 i µ 2 = 2πσ 2 n/2 exp = 2πσ 2 n/2 exp 1 1 x i µ 2 x 2 i + µ σ 2 x i nµ2 Suppose that both µ and σ 2 are unknown so that θ = µ,σ 2 T. Then f X x θ = g x i, x 2 i,θ h x with h x 1 and g x i, x 2 i,θ = 2πσ 2 n/2 exp 1 x 2 i + µ σ 2 x i nµ2 Hence the bivariate statistic n X i, n X2 i is sufficient for µ,σ 2. This should NOT be interpreted as saying that n X i is sufficient for µ and n X2 i is sufficient for σ 2. All it says is that all the information contained in the sample about µ and σ 2 is also contained in the statistic n X i, n X2 i. Suppose now that µ is unknown but that σ 2 so that we now have θ = µ and f X x θ = 2πσ 2 n/2 µ exp x σ 2 i nµ2 exp 1 x 2 i }{{}}{{} = g x i,θ h x By the factorization theorem we conclude that n X i is sufficient for θ = µ.
9 Suppose now that µ is known but σ 2 is unknown so that now θ = σ 2 and f X x θ = 2πθ n/2 exp 1 x 2 i + µ x i nµ2 2θ θ 2θ }{{} 1 }{{} = g x i, x 2 i,θ h x By the factorization theorem we conclude that the bivariate statistic n X i, n is sufficient for θ = σ 2. Note that n X2 i by itself is not sufficient for σ 2 unless µ = 0. Example Let X 1,X 2,...,X n be a random sample from the U0, θ distribution i.e. { 1 if 0 < x f Xi x i θ = i < θ θ 0 otherwise Note that θ is involved in the range of the distribution. Hence it is better if we write the p.d.f. of X i as f Xi x i θ = 1 θ I 0,θ x i where I 0,θ is the identity function of the interval 0,θ. For any set A the identity function of A is defined as { 1 if x A I A x = 0 if x / A 43 X2 i Hence f X x θ = 1 θ I 0,θ x i = 1 θ n I 0,θ x i = 1 θ ni 0,θ maxx i. }{{} 1 }{{} = gmaxx i,θ.h x = max 1 i n X i is sufficient for θ by the factorization theorem.
Chapter 8.8.1: A factorization theorem
LECTURE 14 Chapter 8.8.1: A factorization theorem The characterization of a sufficient statistic in terms of the conditional distribution of the data given the statistic can be difficult to work with.
More informationMathematical statistics
October 4 th, 2018 Lecture 12: Information Where are we? Week 1 Week 2 Week 4 Week 7 Week 10 Week 14 Probability reviews Chapter 6: Statistics and Sampling Distributions Chapter 7: Point Estimation Chapter
More information1 Probability Model. 1.1 Types of models to be discussed in the course
Sufficiency January 11, 2016 Debdeep Pati 1 Probability Model Model: A family of distributions {P θ : θ Θ}. P θ (B) is the probability of the event B when the parameter takes the value θ. P θ is described
More informationSTATISTICAL METHODS FOR SIGNAL PROCESSING c Alfred Hero
STATISTICAL METHODS FOR SIGNAL PROCESSING c Alfred Hero 1999 32 Statistic used Meaning in plain english Reduction ratio T (X) [X 1,..., X n ] T, entire data sample RR 1 T (X) [X (1),..., X (n) ] T, rank
More informationChapters 9. Properties of Point Estimators
Chapters 9. Properties of Point Estimators Recap Target parameter, or population parameter θ. Population distribution f(x; θ). { probability function, discrete case f(x; θ) = density, continuous case The
More informationMathematical statistics
October 1 st, 2018 Lecture 11: Sufficient statistic Where are we? Week 1 Week 2 Week 4 Week 7 Week 10 Week 14 Probability reviews Chapter 6: Statistics and Sampling Distributions Chapter 7: Point Estimation
More information1. Fisher Information
1. Fisher Information Let f(x θ) be a density function with the property that log f(x θ) is differentiable in θ throughout the open p-dimensional parameter set Θ R p ; then the score statistic (or score
More informationCompleteness. On the other hand, the distribution of an ancillary statistic doesn t depend on θ at all.
Completeness A minimal sufficient statistic achieves the maximum amount of data reduction while retaining all the information the sample has concerning θ. On the other hand, the distribution of an ancillary
More informationLast Lecture - Key Questions. Biostatistics Statistical Inference Lecture 03. Minimal Sufficient Statistics
Last Lecture - Key Questions Biostatistics 602 - Statistical Inference Lecture 03 Hyun Min Kang January 17th, 2013 1 How do we show that a statistic is sufficient for θ? 2 What is a necessary and sufficient
More informationECE534, Spring 2018: Solutions for Problem Set #3
ECE534, Spring 08: Solutions for Problem Set #3 Jointly Gaussian Random Variables and MMSE Estimation Suppose that X, Y are jointly Gaussian random variables with µ X = µ Y = 0 and σ X = σ Y = Let their
More information1 Probability Model. 1.1 Types of models to be discussed in the course
Sufficiency January 18, 016 Debdeep Pati 1 Probability Model Model: A family of distributions P θ : θ Θ}. P θ (B) is the probability of the event B when the parameter takes the value θ. P θ is described
More informationMethods of evaluating estimators and best unbiased estimators Hamid R. Rabiee
Stochastic Processes Methods of evaluating estimators and best unbiased estimators Hamid R. Rabiee 1 Outline Methods of Mean Squared Error Bias and Unbiasedness Best Unbiased Estimators CR-Bound for variance
More informationStatistics GIDP Ph.D. Qualifying Exam Theory Jan 11, 2016, 9:00am-1:00pm
Statistics GIDP Ph.D. Qualifying Exam Theory Jan, 06, 9:00am-:00pm Instructions: Provide answers on the supplied pads of paper; write on only one side of each sheet. Complete exactly 5 of the 6 problems.
More informationUnbiased Estimation. Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others.
Unbiased Estimation Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others. To compare ˆθ and θ, two estimators of θ: Say ˆθ is better than θ if it
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2008 Prof. Gesine Reinert 1 Data x = x 1, x 2,..., x n, realisations of random variables X 1, X 2,..., X n with distribution (model)
More informationSTAT 730 Chapter 4: Estimation
STAT 730 Chapter 4: Estimation Timothy Hanson Department of Statistics, University of South Carolina Stat 730: Multivariate Analysis 1 / 23 The likelihood We have iid data, at least initially. Each datum
More informationMathematical statistics
October 18 th, 2018 Lecture 16: Midterm review Countdown to mid-term exam: 7 days Week 1 Chapter 1: Probability review Week 2 Week 4 Week 7 Chapter 6: Statistics Chapter 7: Point Estimation Chapter 8:
More informationLecture 11. Multivariate Normal theory
10. Lecture 11. Multivariate Normal theory Lecture 11. Multivariate Normal theory 1 (1 1) 11. Multivariate Normal theory 11.1. Properties of means and covariances of vectors Properties of means and covariances
More informationFinal Exam. 1. (6 points) True/False. Please read the statements carefully, as no partial credit will be given.
1. (6 points) True/False. Please read the statements carefully, as no partial credit will be given. (a) If X and Y are independent, Corr(X, Y ) = 0. (b) (c) (d) (e) A consistent estimator must be asymptotically
More informationUnbiased Estimation. Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others.
Unbiased Estimation Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others. To compare ˆθ and θ, two estimators of θ: Say ˆθ is better than θ if it
More informationCourse: ESO-209 Home Work: 1 Instructor: Debasis Kundu
Home Work: 1 1. Describe the sample space when a coin is tossed (a) once, (b) three times, (c) n times, (d) an infinite number of times. 2. A coin is tossed until for the first time the same result appear
More information557: MATHEMATICAL STATISTICS II BIAS AND VARIANCE
557: MATHEMATICAL STATISTICS II BIAS AND VARIANCE An estimator, T (X), of θ can be evaluated via its statistical properties. Typically, two aspects are considered: Expectation Variance either in terms
More informationSUFFICIENT STATISTICS
SUFFICIENT STATISTICS. Introduction Let X (X,..., X n ) be a random sample from f θ, where θ Θ is unknown. We are interested using X to estimate θ. In the simple case where X i Bern(p), we found that the
More informationModule 2. Random Processes. Version 2, ECE IIT, Kharagpur
Module Random Processes Version, ECE IIT, Kharagpur Lesson 9 Introduction to Statistical Signal Processing Version, ECE IIT, Kharagpur After reading this lesson, you will learn about Hypotheses testing
More informationReview. December 4 th, Review
December 4 th, 2017 Att. Final exam: Course evaluation Friday, 12/14/2018, 10:30am 12:30pm Gore Hall 115 Overview Week 2 Week 4 Week 7 Week 10 Week 12 Chapter 6: Statistics and Sampling Distributions Chapter
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2009 Prof. Gesine Reinert Our standard situation is that we have data x = x 1, x 2,..., x n, which we view as realisations of random
More informationMA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems
MA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems Review of Basic Probability The fundamentals, random variables, probability distributions Probability mass/density functions
More informationDistributions of Functions of Random Variables. 5.1 Functions of One Random Variable
Distributions of Functions of Random Variables 5.1 Functions of One Random Variable 5.2 Transformations of Two Random Variables 5.3 Several Random Variables 5.4 The Moment-Generating Function Technique
More informationPart IB Statistics. Theorems with proof. Based on lectures by D. Spiegelhalter Notes taken by Dexter Chua. Lent 2015
Part IB Statistics Theorems with proof Based on lectures by D. Spiegelhalter Notes taken by Dexter Chua Lent 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly)
More informationBrief Review on Estimation Theory
Brief Review on Estimation Theory K. Abed-Meraim ENST PARIS, Signal and Image Processing Dept. abed@tsi.enst.fr This presentation is essentially based on the course BASTA by E. Moulines Brief review on
More informationSTAT 512 sp 2018 Summary Sheet
STAT 5 sp 08 Summary Sheet Karl B. Gregory Spring 08. Transformations of a random variable Let X be a rv with support X and let g be a function mapping X to Y with inverse mapping g (A = {x X : g(x A}
More informationHT Introduction. P(X i = x i ) = e λ λ x i
MODS STATISTICS Introduction. HT 2012 Simon Myers, Department of Statistics (and The Wellcome Trust Centre for Human Genetics) myers@stats.ox.ac.uk We will be concerned with the mathematical framework
More informationCentral Limit Theorem ( 5.3)
Central Limit Theorem ( 5.3) Let X 1, X 2,... be a sequence of independent random variables, each having n mean µ and variance σ 2. Then the distribution of the partial sum S n = X i i=1 becomes approximately
More informationHypothesis Test. The opposite of the null hypothesis, called an alternative hypothesis, becomes
Neyman-Pearson paradigm. Suppose that a researcher is interested in whether the new drug works. The process of determining whether the outcome of the experiment points to yes or no is called hypothesis
More informationMcGill University. Faculty of Science. Department of Mathematics and Statistics. Part A Examination. Statistics: Theory Paper
McGill University Faculty of Science Department of Mathematics and Statistics Part A Examination Statistics: Theory Paper Date: 10th May 2015 Instructions Time: 1pm-5pm Answer only two questions from Section
More informationAll other items including (and especially) CELL PHONES must be left at the front of the room.
TEST #2 / STA 5327 (Inference) / Spring 2017 (April 24, 2017) Name: Directions This exam is closed book and closed notes. You will be supplied with scratch paper, and a copy of the Table of Common Distributions
More information1 Complete Statistics
Complete Statistics February 4, 2016 Debdeep Pati 1 Complete Statistics Suppose X P θ, θ Θ. Let (X (1),..., X (n) ) denote the order statistics. Definition 1. A statistic T = T (X) is complete if E θ g(t
More informationDA Freedman Notes on the MLE Fall 2003
DA Freedman Notes on the MLE Fall 2003 The object here is to provide a sketch of the theory of the MLE. Rigorous presentations can be found in the references cited below. Calculus. Let f be a smooth, scalar
More informationTwo hours. To be supplied by the Examinations Office: Mathematical Formula Tables THE UNIVERSITY OF MANCHESTER. 21 June :45 11:45
Two hours MATH20802 To be supplied by the Examinations Office: Mathematical Formula Tables THE UNIVERSITY OF MANCHESTER STATISTICAL METHODS 21 June 2010 9:45 11:45 Answer any FOUR of the questions. University-approved
More informationMaster s Written Examination
Master s Written Examination Option: Statistics and Probability Spring 05 Full points may be obtained for correct answers to eight questions Each numbered question (which may have several parts) is worth
More informationSTAT215: Solutions for Homework 2
STAT25: Solutions for Homework 2 Due: Wednesday, Feb 4. (0 pt) Suppose we take one observation, X, from the discrete distribution, x 2 0 2 Pr(X x θ) ( θ)/4 θ/2 /2 (3 θ)/2 θ/4, 0 θ Find an unbiased estimator
More informationFirst Year Examination Department of Statistics, University of Florida
First Year Examination Department of Statistics, University of Florida August 19, 010, 8:00 am - 1:00 noon Instructions: 1. You have four hours to answer questions in this examination.. You must show your
More informationSTAT/MATH 395 A - PROBABILITY II UW Winter Quarter Moment functions. x r p X (x) (1) E[X r ] = x r f X (x) dx (2) (x E[X]) r p X (x) (3)
STAT/MATH 395 A - PROBABILITY II UW Winter Quarter 07 Néhémy Lim Moment functions Moments of a random variable Definition.. Let X be a rrv on probability space (Ω, A, P). For a given r N, E[X r ], if it
More informationHypothesis Testing. 1 Definitions of test statistics. CB: chapter 8; section 10.3
Hypothesis Testing CB: chapter 8; section 0.3 Hypothesis: statement about an unknown population parameter Examples: The average age of males in Sweden is 7. (statement about population mean) The lowest
More informationProof In the CR proof. and
Question Under what conditions will we be able to attain the Cramér-Rao bound and find a MVUE? Lecture 4 - Consequences of the Cramér-Rao Lower Bound. Searching for a MVUE. Rao-Blackwell Theorem, Lehmann-Scheffé
More informationStatistics 3858 : Maximum Likelihood Estimators
Statistics 3858 : Maximum Likelihood Estimators 1 Method of Maximum Likelihood In this method we construct the so called likelihood function, that is L(θ) = L(θ; X 1, X 2,..., X n ) = f n (X 1, X 2,...,
More informationAlgorithms for Uncertainty Quantification
Algorithms for Uncertainty Quantification Tobias Neckel, Ionuț-Gabriel Farcaș Lehrstuhl Informatik V Summer Semester 2017 Lecture 2: Repetition of probability theory and statistics Example: coin flip Example
More information4.5.1 The use of 2 log Λ when θ is scalar
4.5. ASYMPTOTIC FORM OF THE G.L.R.T. 97 4.5.1 The use of 2 log Λ when θ is scalar Suppose we wish to test the hypothesis NH : θ = θ where θ is a given value against the alternative AH : θ θ on the basis
More informationTwo hours. To be supplied by the Examinations Office: Mathematical Formula Tables and Statistical Tables THE UNIVERSITY OF MANCHESTER.
Two hours MATH38181 To be supplied by the Examinations Office: Mathematical Formula Tables and Statistical Tables THE UNIVERSITY OF MANCHESTER EXTREME VALUES AND FINANCIAL RISK Examiner: Answer any FOUR
More informationChapter 3. Point Estimation. 3.1 Introduction
Chapter 3 Point Estimation Let (Ω, A, P θ ), P θ P = {P θ θ Θ}be probability space, X 1, X 2,..., X n : (Ω, A) (IR k, B k ) random variables (X, B X ) sample space γ : Θ IR k measurable function, i.e.
More informationProbability and Distributions
Probability and Distributions What is a statistical model? A statistical model is a set of assumptions by which the hypothetical population distribution of data is inferred. It is typically postulated
More information2.2.2 Comparing estimators
2.2. POINT ESTIMATION 87 2.2.2 Comparing estimators Estimators can be compared through their mean square errors. If they are unbiased, this is equivalent to comparing their variances. In many applications,
More information6.1 Variational representation of f-divergences
ECE598: Information-theoretic methods in high-dimensional statistics Spring 2016 Lecture 6: Variational representation, HCR and CR lower bounds Lecturer: Yihong Wu Scribe: Georgios Rovatsos, Feb 11, 2016
More informationProblem Selected Scores
Statistics Ph.D. Qualifying Exam: Part II November 20, 2010 Student Name: 1. Answer 8 out of 12 problems. Mark the problems you selected in the following table. Problem 1 2 3 4 5 6 7 8 9 10 11 12 Selected
More informationTopic 10: Hypothesis Testing
Topic 10: Hypothesis Testing Course 003, 2016 Page 0 The Problem of Hypothesis Testing A statistical hypothesis is an assertion or conjecture about the probability distribution of one or more random variables.
More informationThe Delta Method and Applications
Chapter 5 The Delta Method and Applications 5.1 Local linear approximations Suppose that a particular random sequence converges in distribution to a particular constant. The idea of using a first-order
More informationMarch 10, 2017 THE EXPONENTIAL CLASS OF DISTRIBUTIONS
March 10, 2017 THE EXPONENTIAL CLASS OF DISTRIBUTIONS Abstract. We will introduce a class of distributions that will contain many of the discrete and continuous we are familiar with. This class will help
More informationRandom vectors X 1 X 2. Recall that a random vector X = is made up of, say, k. X k. random variables.
Random vectors Recall that a random vector X = X X 2 is made up of, say, k random variables X k A random vector has a joint distribution, eg a density f(x), that gives probabilities P(X A) = f(x)dx Just
More information1. (Regular) Exponential Family
1. (Regular) Exponential Family The density function of a regular exponential family is: [ ] Example. Poisson(θ) [ ] Example. Normal. (both unknown). ) [ ] [ ] [ ] [ ] 2. Theorem (Exponential family &
More information4.1 The Expectation of a Random Variable
STAT 42 Lecture Notes 93 4. The Expectation of a Random Variable This chapter begins the discussion of properties of random variables. The focus of this chapter is on expectations of random variables.
More information5.2 Fisher information and the Cramer-Rao bound
Stat 200: Introduction to Statistical Inference Autumn 208/9 Lecture 5: Maximum likelihood theory Lecturer: Art B. Owen October 9 Disclaimer: These notes have not been subjected to the usual scrutiny reserved
More informationChapter 8: Least squares (beginning of chapter)
Chapter 8: Least squares (beginning of chapter) Least Squares So far, we have been trying to determine an estimator which was unbiased and had minimum variance. Next we ll consider a class of estimators
More informationECE 275B Homework # 1 Solutions Version Winter 2015
ECE 275B Homework # 1 Solutions Version Winter 2015 1. (a) Because x i are assumed to be independent realizations of a continuous random variable, it is almost surely (a.s.) 1 the case that x 1 < x 2
More informationStatistics 1B. Statistics 1B 1 (1 1)
0. Statistics 1B Statistics 1B 1 (1 1) 0. Lecture 1. Introduction and probability review Lecture 1. Introduction and probability review 2 (1 1) 1. Introduction and probability review 1.1. What is Statistics?
More informationparameter space Θ, depending only on X, such that Note: it is not θ that is random, but the set C(X).
4. Interval estimation The goal for interval estimation is to specify the accurary of an estimate. A 1 α confidence set for a parameter θ is a set C(X) in the parameter space Θ, depending only on X, such
More informationChapter 1. Statistical Spaces
Chapter 1 Statistical Spaces Mathematical statistics is a science that studies the statistical regularity of random phenomena, essentially by some observation values of random variable (r.v.) X. Sometimes
More informationLecture 25: Review. Statistics 104. April 23, Colin Rundel
Lecture 25: Review Statistics 104 Colin Rundel April 23, 2012 Joint CDF F (x, y) = P [X x, Y y] = P [(X, Y ) lies south-west of the point (x, y)] Y (x,y) X Statistics 104 (Colin Rundel) Lecture 25 April
More informationMathematical Statistics
Mathematical Statistics Chapter Three. Point Estimation 3.4 Uniformly Minimum Variance Unbiased Estimator(UMVUE) Criteria for Best Estimators MSE Criterion Let F = {p(x; θ) : θ Θ} be a parametric distribution
More informationContents 1. Contents
Contents 1 Contents 6 Distributions of Functions of Random Variables 2 6.1 Transformation of Discrete r.v.s............. 3 6.2 Method of Distribution Functions............. 6 6.3 Method of Transformations................
More informationStat 5102 Lecture Slides Deck 3. Charles J. Geyer School of Statistics University of Minnesota
Stat 5102 Lecture Slides Deck 3 Charles J. Geyer School of Statistics University of Minnesota 1 Likelihood Inference We have learned one very general method of estimation: method of moments. the Now we
More informationActuarial Science Exam 1/P
Actuarial Science Exam /P Ville A. Satopää December 5, 2009 Contents Review of Algebra and Calculus 2 2 Basic Probability Concepts 3 3 Conditional Probability and Independence 4 4 Combinatorial Principles,
More informationA Few Notes on Fisher Information (WIP)
A Few Notes on Fisher Information (WIP) David Meyer dmm@{-4-5.net,uoregon.edu} Last update: April 30, 208 Definitions There are so many interesting things about Fisher Information and its theoretical properties
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationQuick Tour of Basic Probability Theory and Linear Algebra
Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra CS224w: Social and Information Network Analysis Fall 2011 Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra Outline Definitions
More informationECE 275B Homework # 1 Solutions Winter 2018
ECE 275B Homework # 1 Solutions Winter 2018 1. (a) Because x i are assumed to be independent realizations of a continuous random variable, it is almost surely (a.s.) 1 the case that x 1 < x 2 < < x n Thus,
More informationEIE6207: Estimation Theory
EIE6207: Estimation Theory Man-Wai MAK Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University enmwmak@polyu.edu.hk http://www.eie.polyu.edu.hk/ mwmak References: Steven M.
More informationChapter 4 HOMEWORK ASSIGNMENTS. 4.1 Homework #1
Chapter 4 HOMEWORK ASSIGNMENTS These homeworks may be modified as the semester progresses. It is your responsibility to keep up to date with the correctly assigned homeworks. There may be some errors in
More informationMultiple Random Variables
Multiple Random Variables This Version: July 30, 2015 Multiple Random Variables 2 Now we consider models with more than one r.v. These are called multivariate models For instance: height and weight An
More informationSpring 2012 Math 541B Exam 1
Spring 2012 Math 541B Exam 1 1. A sample of size n is drawn without replacement from an urn containing N balls, m of which are red and N m are black; the balls are otherwise indistinguishable. Let X denote
More informationLast Lecture. Biostatistics Statistical Inference Lecture 14 Obtaining Best Unbiased Estimator. Related Theorems. Rao-Blackwell Theorem
Last Lecture Biostatistics 62 - Statistical Inference Lecture 14 Obtaining Best Unbiased Estimator Hyun Min Kang February 28th, 213 For single-parameter exponential family, is Cramer-Rao bound always attainable?
More informationPart IA Probability. Definitions. Based on lectures by R. Weber Notes taken by Dexter Chua. Lent 2015
Part IA Probability Definitions Based on lectures by R. Weber Notes taken by Dexter Chua Lent 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after lectures.
More information15 Discrete Distributions
Lecture Note 6 Special Distributions (Discrete and Continuous) MIT 4.30 Spring 006 Herman Bennett 5 Discrete Distributions We have already seen the binomial distribution and the uniform distribution. 5.
More informationELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE)
1 ELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE) Jingxian Wu Department of Electrical Engineering University of Arkansas Outline Minimum Variance Unbiased Estimators (MVUE)
More informationBASICS OF PROBABILITY
October 10, 2018 BASICS OF PROBABILITY Randomness, sample space and probability Probability is concerned with random experiments. That is, an experiment, the outcome of which cannot be predicted with certainty,
More informationDirection: This test is worth 250 points and each problem worth points. DO ANY SIX
Term Test 3 December 5, 2003 Name Math 52 Student Number Direction: This test is worth 250 points and each problem worth 4 points DO ANY SIX PROBLEMS You are required to complete this test within 50 minutes
More informationWhy study probability? Set theory. ECE 6010 Lecture 1 Introduction; Review of Random Variables
ECE 6010 Lecture 1 Introduction; Review of Random Variables Readings from G&S: Chapter 1. Section 2.1, Section 2.3, Section 2.4, Section 3.1, Section 3.2, Section 3.5, Section 4.1, Section 4.2, Section
More informationconditional cdf, conditional pdf, total probability theorem?
6 Multiple Random Variables 6.0 INTRODUCTION scalar vs. random variable cdf, pdf transformation of a random variable conditional cdf, conditional pdf, total probability theorem expectation of a random
More informationAccounting for Baseline Observations in Randomized Clinical Trials
Accounting for Baseline Observations in Randomized Clinical Trials Scott S Emerson, MD, PhD Department of Biostatistics, University of Washington, Seattle, WA 9895, USA October 6, 0 Abstract In clinical
More informationStatistics and Econometrics I
Statistics and Econometrics I Point Estimation Shiu-Sheng Chen Department of Economics National Taiwan University September 13, 2016 Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13,
More informationEstimation Theory. as Θ = (Θ 1,Θ 2,...,Θ m ) T. An estimator
Estimation Theory Estimation theory deals with finding numerical values of interesting parameters from given set of data. We start with formulating a family of models that could describe how the data were
More informationProbability Background
Probability Background Namrata Vaswani, Iowa State University August 24, 2015 Probability recap 1: EE 322 notes Quick test of concepts: Given random variables X 1, X 2,... X n. Compute the PDF of the second
More informationReview Quiz. 1. Prove that in a one-dimensional canonical exponential family, the complete and sufficient statistic achieves the
Review Quiz 1. Prove that in a one-dimensional canonical exponential family, the complete and sufficient statistic achieves the Cramér Rao lower bound (CRLB). That is, if where { } and are scalars, then
More informationMaximum Likelihood Estimation
Chapter 8 Maximum Likelihood Estimation 8. Consistency If X is a random variable (or vector) with density or mass function f θ (x) that depends on a parameter θ, then the function f θ (X) viewed as a function
More informationMachine learning - HT Maximum Likelihood
Machine learning - HT 2016 3. Maximum Likelihood Varun Kanade University of Oxford January 27, 2016 Outline Probabilistic Framework Formulate linear regression in the language of probability Introduce
More informationBivariate distributions
Bivariate distributions 3 th October 017 lecture based on Hogg Tanis Zimmerman: Probability and Statistical Inference (9th ed.) Bivariate Distributions of the Discrete Type The Correlation Coefficient
More informationLikelihoods. P (Y = y) = f(y). For example, suppose Y has a geometric distribution on 1, 2,... with parameter p. Then the pmf is
Likelihoods The distribution of a random variable Y with a discrete sample space (e.g. a finite sample space or the integers) can be characterized by its probability mass function (pmf): P (Y = y) = f(y).
More informationGraduate Econometrics I: Unbiased Estimation
Graduate Econometrics I: Unbiased Estimation Yves Dominicy Université libre de Bruxelles Solvay Brussels School of Economics and Management ECARES Yves Dominicy Graduate Econometrics I: Unbiased Estimation
More informationSpace Telescope Science Institute statistics mini-course. October Inference I: Estimation, Confidence Intervals, and Tests of Hypotheses
Space Telescope Science Institute statistics mini-course October 2011 Inference I: Estimation, Confidence Intervals, and Tests of Hypotheses James L Rosenberger Acknowledgements: Donald Richards, William
More informationTheory of Statistical Tests
Ch 9. Theory of Statistical Tests 9.1 Certain Best Tests How to construct good testing. For simple hypothesis H 0 : θ = θ, H 1 : θ = θ, Page 1 of 100 where Θ = {θ, θ } 1. Define the best test for H 0 H
More informationSummary. Ancillary Statistics What is an ancillary statistic for θ? .2 Can an ancillary statistic be a sufficient statistic?
Biostatistics 62 - Statistical Inference Lecture 5 Hyun Min Kang 1 What is an ancillary statistic for θ? 2 Can an ancillary statistic be a sufficient statistic? 3 What are the location parameter and the
More information