ECE534, Spring 2018: Solutions for Problem Set #3
|
|
- Daniel Chandler
- 5 years ago
- Views:
Transcription
1 ECE534, Spring 08: Solutions for Problem Set #3 Jointly Gaussian Random Variables and MMSE Estimation Suppose that X, Y are jointly Gaussian random variables with µ X = µ Y = 0 and σ X = σ Y = Let their correlation coefficient be ρ with ρ < Based on X, Y, we define the following random variables: W = + ρ + X + ρ + ρ Y ρ Z = + ρ X + ρ + ρ + Y ρ a Are W, Z jointly Gaussian? Justify your answer b Calculate f W Z w, z c Find the MMSE estimator of Z given W d Find the linear MMSE estimator of X given W Solution a Let α = +ρ + ρ and β = +ρ ρ Note that any linear combination of W, Z corresponds to a linear combination of X, Y : aw + bz = aα + bβx + aβ + bαy, which is Gaussian since X, Y are jointly Gaussian Therefore, W and Z are jointly Gaussian b From a, W and Z have a bi-variate Gaussian density, determined by the mean vector and covariance matrix Clearly, µ W = µ Z = 0 For the variances and the correlation coefficient, we have: σz = σw ρ = CovW, W = + ρ ρ =, ρ W,Z = CovW, Z = ρ + ρ + ρ ρ = 0 Therefore, W Z are uncorrelated hence independent, being jointly Gaussian This shows that for w, z R: f W Z w, z = f W wf Z z = e w e z = π π π e w +z c The MMSE estimator of Z given W is the conditional mean E[Z W ] Since Z, W are independent, we have E[Z W ] = E[Z] = µ Z = 0
2 d A straightforward computation gives CovX, W = + ρ + ρ We now have: Ê[X W ] = µ X + CovX, W σw W µ W = + ρ + ρ W Interplay between Information Theory and Estimation Let P, Q be two distributions defined on R k with densities f P, f Q, respectively Assume that the support of the densities coincides with R k, ie, f P x > 0 and f Q x > 0 for any x R k Their Kullback-Leibler divergence is then defined as follows: DP Q = f P x log f P x R k f Q x dx DP Q corresponds to a measure of dissimilarity between P and Q Consider now two continuous random variables X and Y with joint distribution P XY and marginals P X, P Y, respectively Suppose that the support of P XY is R and also the support of the marginals is R Then, the mutual information IX; Y between X, Y is defined as: IX; Y = DP XY P X P Y Assume that two continuous random variables X and Y are related by the following relationship: Y = ax + N, where a 0 is a deterministic parameter, X N 0, and N N 0, is independent of X We already know that ˆX = ˆX a = E[X Y ] is the MMSE estimator of X given Y, while [ MMSE = MMSEa = E X ˆX ] is the achievable mean square error Note that we have explicitly shown the dependence of the estimator and the MMSE on the parameter a Show that by proving the following steps: a IX; Y = log + a b ˆX = a +a Y c MMSEa = +a Combine the above steps to conclude that holds d da IX; Y = MMSEa Note: All logarithms in this exercise are natural base e
3 Solution: a Note that the mutual information can be written in expectation form: { IX; Y = E X,Y log f XY X, Y f X Xf Y Y For the involved densities, we note that Y N 0, + a and therefore f Y y = π + a e y +a For the joint density we have f Y X=x y = f N y ax, hence f X,Y x, y = f Y X=x yf X x = f N y axf X x indepedence of X,N and f X,Y x, y f X xf Y y = f Ny ax f X x f X xf Y y = e y ax / π + a π = + ae y +a y ax e y +a Using the previous expressions in IX; Y, we obtain: IX; Y = { E X,Y Y ax + + a Y + log + a = log + a + [ ] E Y + a Y a E X [X ] + a E X,Y [XY ] Moreover, E X,Y [XY ] = E[XN + ax] = a, E X [X ] =, and E Y [Y ] = + a, thus IX; Y = log + a + b The MMSE estimator is linear and it is given by c ˆX = µ X + a a + a a = log + a a {{ CovX, Y Y VarY µ Y = +a a a + Y a {{ MMSE = E X ˆX CovX, Y = VarX 3 VarY +a = + a
4 Combining the above steps, follows by differentiating the mutual information: d da IX; Y = d log + a = da + a = MMSEa 3 Data Reduction, Sufficient Statistics and the MVUE Assume that we want to estimate an unknown parameter θ Treating θ as a deterministic variable, a good estimator ˆθ of θ is one which takes values close to the true parameter θ when attempting to estimate it If ˆθ is an unbiased estimator of θ, then MSE = E[ˆθ θ ] = Varˆθ as we have proved in class If the bias of ˆθ, bθ = E[ˆθ] θ, is nonzero, the MSE is still a good measure to evaluate the performance of a biased estimator ˆθ Given a set of observations X = {X, X,, X n containing information about the unknown parameter θ, a sufficient statistic is a function of the data: T X = T X, X,, X n containing all information that the data brings about θ any measurable function of the data tx is a statistic, but not necessarily sufficient More rigorously and assuming for simplicity that the data has a joint pdf, T X is a sufficient static for θ if the conditional pdf px, X,, X n T X ; θ = px T X ; θ does not depend on θ Clearly, when this holds, T X provides all information hidden in the data about θ and therefore, the initial set of data X can be discarded and only T X is stored hence, the name sufficient statistic Moreover, sufficient statistics are not unique Relevant to the notion of a sufficient statistic, the following two theorems are important: Theorem Fisher-Neyman factorization theorem: T X is a sufficient statistic if and only if the joint pdf of the data can be factored as px ; θ = hx gt X, θ That is, the joint pdf is factored into two parts: one part that depends only on the statistic and the parameters θ and a second part that is independent of the parameters θ Theorem Rao-Blackwell theorem: Let ˆθ be an estimator of θ with E[ˆθ ] < for all θ Suppose that T X is a sufficient statistic for θ and let ˆθ = E[ˆθ T X ] Then for all θ, [ ] [ ] E ˆθ θ E ˆθ θ The inequality is strict unless ˆθ is a function of the sufficient statistic T X 4
5 As it is clear from the theorem, ˆθ is an estimator of θ and if ˆθ is unbiased, then so is ˆθ since [ ] E[ˆθ ] = E E[ˆθ T X ] = E[ˆθ] = θ Moreover, if there is a unique function qt X which is an unbiased estimator of θ, then qt X is the MVUE To finish the above introductory material, we note that a statistic tx is complete if for any measurable function γ, the condition E[γtX ] = 0 θ implies that γ 0 almost surely If the sufficient statistic T X is also complete, then there will be at most one function qt X that would correspond to an unbiased estimator for θ With the above in mind, an approach to identify the MVUE is the following: i Find a sufficient statistic T X ii Argue that this statistic is complete iii Find an unbiased estimator qt X of θ This can be done via Rao- Blackwellization by considering any unbiased estimator ˆθ of θ and by setting qt X = E[ˆθ T X ] iv The MVUE is ˆθ = qt X Moreover, the MVUE is unique The above approach is based on the Lehmann-Scheffé Theorem, which is a direct consequence of the previous two theorems and the completeness of the statistic T X Let X be a set of n iid N µ, σ observations a Assume that the unknown parameter θ is the mean value µ Use Theorem to show that S n = n X i is a sufficient statistic for µ b Assume that µ is known but θ = σ is the unknown parameter Use Theorem to show that n X i µ is a sufficient statistic for σ c Assume that both µ and σ are unknown, ie, θ = µ, σ Use Theorem to show that the pair X i, is a sufficient statistic in this case d Given the set of data X = {X, X,, X n with θ = µ, a possible statistic is T X = {X, X Argue that T X is not complete by using e Assume that the unknown parameter is µ and note that ˆθ = X is an unbiased estimator for µ Argue that the MVUE which is unique for µ is Sn n X i 5
6 Solution a Consider an arbitrary realization x, x,, x n of X The joint density of X all random variables are iid can be factored are follows: { n px,, x n ; µ = f Xi x i = πσ exp x i µ n σ { = πσ n exp x σ i x i µ + µ { = πσ n exp σ x i µs n + nµ { = πσ n { exp σ x i exp µsn σ + nµ hx gt X,µ Therefore, by Theorem, T X = S n is a sufficient statistic Note: For notational conservatism, here, we use S n to also denote the realized value of the statistic S n b Let Q n = n X i µ Again, we consider the joint density, but this time with θ = σ : { px,, x n ; σ = πσ exp x i µ n σ = hx πσ { n Qn exp σ {{ gt X,σ c Let R n := n X i so that T X = {S n, R n This gives rise to { px,, x n ; µ, σ = πσ exp x i µ n σ θ { = πσ n exp x σ i x i µ + µ { = πσ n exp σ R n µs n + nµ hx gt X,θ d We seek some nonzero function γ, for which EγX, X = 0 Consider a nonzero linear function of the form γx, X = ax + bx with a, b 0 Then, EγX, X = a + bµ 6
7 Clearly, whenever a + b = 0 we get EγX, X = 0 whereas PγX, X = 0 = PX = X = 0, µ Therefore, γt X 0 almost surely and hence T X is not complete e We will follow the Lehmann-Scheffé steps described above i We already established that S n is a sufficient statistic for µ ii [bonus part] To argue that S n is complete, note that S n N nµ, nσ Suppose that for all µ R, we have: EγS n = s nµ γs e nσ ds = 0 3 πnσ We need to show that γ 0 almost surely Observe that the integral in 3 is a bilateral Laplace transform: EγS n = γse sµ σ ds = B{ γ µ σ = 0, µ R 4 Here, γs := e s nµ nσ γs Assuming that γ is continuous, we πnσ deduce that γ is continuous By the properties of the bilateral Laplace transform, 4 implies that γ 0 almost surely and, thus, γ 0 almost surely iii Using Rao-Blackwellization, we begin with some unbiased estimator of µ, which in this case is ˆµ = X, and seek some q given by qs n = Eˆµ S n = EX S n Note that qs n = EX i S n for all i =,, n by the iid assumption on X, X,, X n By summing, we obtain: Sn nqs n = EX i S n = E X i = ES n S n = S n qs n Therefore, qs n = n S n 4 Maximum of a finite set of sub-gaussian Random variables A random variable X R is called sub-gaussian with variance proxy σ if E[X] = 0 and its moment generating function satisfies: We write X subgσ m X u = E [ e ux] e u σ, u R a Use the Chernoff bound to show that when X subg σ PX > t e t σ, t > 0 7
8 b Let X, X,, X n be subgσ random variables, not necessarily independent Show that the expectation of the maximum can be bounded as E[ max i n X i] σ log n Hint: Start your derivation by noting that E[ max X i] = [ ] i n λ E log e λ max i n X i, λ > 0 c With the same assumptions as in the previous part, show that P max X i > t ne t σ, t > 0 i n Solution: a By applying the Chernoff bound, we obtain: PX > t = P e ux > e ut which holds for any u > 0 By subgaussianity, Plugging 6 into 5 gives m X u {{ E e ux e ut, 5 m X u e σ u / 6 PX > t e σ u ut = e φu, 7 where φu := σ u ut Choose u = t σ, which minimizes φu to obtain φu = t σ Therefore, as required PX > t e φu = e t σ b Using the hint, Emax X i = log e i λ E λ max i X i E e λ log λ max i X i Jensen, concavity of log = E λ log max e λx i Monotonicity of e x i E λ log e λx i max X i X i i i λ log ne λ σ subgaussianity = log n λ + λσ {{ gλ 8
9 Minimizing gλ by setting its derivative to zero, we obtain λ = which we have log n σ for E[max X i ] σ log n i c Invoking the union bound and part a we obtain: 5 Erdős-Rényi graphs n P max X i > t = P {X i > t i n PX i > t e t σ ne t σ The distance between any two nodes in a graph is the length of the shortest path connecting them The diameter of a graph is the maximum distance between any two nodes of the graph Consider again the ensemble Gn, p of Erdős-Rényi graphs in Problem 3 of HW# Suppose that p is fixed and let D n be the diameter of a graph drawn from this ensemble Show that P D n as n Note: If p is allowed to scale with n, then p = log n n is a sharp threshold for the aforementioned property, meaning that if p < log n n, then P D n >, as n Hint: Define the random variable X n, which is the number of node pairs u, v in the graph G Gn, p with no common neighbors Show that P X n = 0 as n Solution: Using the provided hint, we note that X n = 0 D n, ie, the diameter of a graph is if and only if all nodes are separated by up to one neighbor, or if they are connected directly Equivalently, X n D n > By employing Markov inequality, we have: PD n > = PX n EX n We need to show that EX n 0 as n, which in turn will guarantee that PD n 9
10 Let u, v be a pair of nodes with u v Define the event A u,v : A u,v := {du, v > = {u, v are neither neigbhors nor share neighbors Here, du, v is the distance shortest path between u and v The nodes u, v are not adjacent ie, neighbors with probability p; the rest of the n nodes are not adjacent to both u and v with probability p n Hence, PA u,v = p n = p p n Clearly A u,v = A v,u since the graph is undirected Therefore, we consider only the nodes S n := {u, v u < v n We further note that the cardinality of S n is n We now have: X n = and thus, EX n = u,v S n PA u,v = S n p n u,v S n Au,v n pe n p 0 as n, where the elementary inequality e x + x, x R has been used 0
Mathematical statistics
October 4 th, 2018 Lecture 12: Information Where are we? Week 1 Week 2 Week 4 Week 7 Week 10 Week 14 Probability reviews Chapter 6: Statistics and Sampling Distributions Chapter 7: Point Estimation Chapter
More informationEIE6207: Estimation Theory
EIE6207: Estimation Theory Man-Wai MAK Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University enmwmak@polyu.edu.hk http://www.eie.polyu.edu.hk/ mwmak References: Steven M.
More informationChapter 8: Least squares (beginning of chapter)
Chapter 8: Least squares (beginning of chapter) Least Squares So far, we have been trying to determine an estimator which was unbiased and had minimum variance. Next we ll consider a class of estimators
More informationProbability Theory and Statistics. Peter Jochumzen
Probability Theory and Statistics Peter Jochumzen April 18, 2016 Contents 1 Probability Theory And Statistics 3 1.1 Experiment, Outcome and Event................................ 3 1.2 Probability............................................
More informationELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE)
1 ELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE) Jingxian Wu Department of Electrical Engineering University of Arkansas Outline Minimum Variance Unbiased Estimators (MVUE)
More informationAdvanced Signal Processing Introduction to Estimation Theory
Advanced Signal Processing Introduction to Estimation Theory Danilo Mandic, room 813, ext: 46271 Department of Electrical and Electronic Engineering Imperial College London, UK d.mandic@imperial.ac.uk,
More informationExercises with solutions (Set D)
Exercises with solutions Set D. A fair die is rolled at the same time as a fair coin is tossed. Let A be the number on the upper surface of the die and let B describe the outcome of the coin toss, where
More informationBrief Review on Estimation Theory
Brief Review on Estimation Theory K. Abed-Meraim ENST PARIS, Signal and Image Processing Dept. abed@tsi.enst.fr This presentation is essentially based on the course BASTA by E. Moulines Brief review on
More informationSTAT 730 Chapter 4: Estimation
STAT 730 Chapter 4: Estimation Timothy Hanson Department of Statistics, University of South Carolina Stat 730: Multivariate Analysis 1 / 23 The likelihood We have iid data, at least initially. Each datum
More informationLecture Notes 5 Convergence and Limit Theorems. Convergence with Probability 1. Convergence in Mean Square. Convergence in Probability, WLLN
Lecture Notes 5 Convergence and Limit Theorems Motivation Convergence with Probability Convergence in Mean Square Convergence in Probability, WLLN Convergence in Distribution, CLT EE 278: Convergence and
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationECE534, Spring 2018: Solutions for Problem Set #4 Due Friday April 6, 2018
ECE534, Spring 2018: s for Problem Set #4 Due Friday April 6, 2018 1. MMSE Estimation, Data Processing and Innovations The random variables X, Y, Z on a common probability space (Ω, F, P ) are said to
More informationExercises and Answers to Chapter 1
Exercises and Answers to Chapter The continuous type of random variable X has the following density function: a x, if < x < a, f (x), otherwise. Answer the following questions. () Find a. () Obtain mean
More informationProbability and Distributions
Probability and Distributions What is a statistical model? A statistical model is a set of assumptions by which the hypothetical population distribution of data is inferred. It is typically postulated
More informationExample: An experiment can either result in success or failure with probability θ and (1 θ) respectively. The experiment is performed independently
Chapter 3 Sufficient statistics and variance reduction Let X 1,X 2,...,X n be a random sample from a certain distribution with p.m/d.f fx θ. A function T X 1,X 2,...,X n = T X of these observations is
More informationChapters 9. Properties of Point Estimators
Chapters 9. Properties of Point Estimators Recap Target parameter, or population parameter θ. Population distribution f(x; θ). { probability function, discrete case f(x; θ) = density, continuous case The
More informationContinuous Random Variables
1 / 24 Continuous Random Variables Saravanan Vijayakumaran sarva@ee.iitb.ac.in Department of Electrical Engineering Indian Institute of Technology Bombay February 27, 2013 2 / 24 Continuous Random Variables
More informationMAS223 Statistical Inference and Modelling Exercises
MAS223 Statistical Inference and Modelling Exercises The exercises are grouped into sections, corresponding to chapters of the lecture notes Within each section exercises are divided into warm-up questions,
More informationAppendix A : Introduction to Probability and stochastic processes
A-1 Mathematical methods in communication July 5th, 2009 Appendix A : Introduction to Probability and stochastic processes Lecturer: Haim Permuter Scribe: Shai Shapira and Uri Livnat The probability of
More informationx log x, which is strictly convex, and use Jensen s Inequality:
2. Information measures: mutual information 2.1 Divergence: main inequality Theorem 2.1 (Information Inequality). D(P Q) 0 ; D(P Q) = 0 iff P = Q Proof. Let ϕ(x) x log x, which is strictly convex, and
More informationMathematical statistics
October 18 th, 2018 Lecture 16: Midterm review Countdown to mid-term exam: 7 days Week 1 Chapter 1: Probability review Week 2 Week 4 Week 7 Chapter 6: Statistics Chapter 7: Point Estimation Chapter 8:
More informationProbability Background
Probability Background Namrata Vaswani, Iowa State University August 24, 2015 Probability recap 1: EE 322 notes Quick test of concepts: Given random variables X 1, X 2,... X n. Compute the PDF of the second
More informationChapter 4 : Expectation and Moments
ECE5: Analysis of Random Signals Fall 06 Chapter 4 : Expectation and Moments Dr. Salim El Rouayheb Scribe: Serge Kas Hanna, Lu Liu Expected Value of a Random Variable Definition. The expected or average
More informationMultiple Random Variables
Multiple Random Variables Joint Probability Density Let X and Y be two random variables. Their joint distribution function is F ( XY x, y) P X x Y y. F XY ( ) 1, < x
More information5 Operations on Multiple Random Variables
EE360 Random Signal analysis Chapter 5: Operations on Multiple Random Variables 5 Operations on Multiple Random Variables Expected value of a function of r.v. s Two r.v. s: ḡ = E[g(X, Y )] = g(x, y)f X,Y
More informationUCSD ECE250 Handout #20 Prof. Young-Han Kim Monday, February 26, Solutions to Exercise Set #7
UCSD ECE50 Handout #0 Prof. Young-Han Kim Monday, February 6, 07 Solutions to Exercise Set #7. Minimum waiting time. Let X,X,... be i.i.d. exponentially distributed random variables with parameter λ, i.e.,
More informationEEL 5544 Noise in Linear Systems Lecture 30. X (s) = E [ e sx] f X (x)e sx dx. Moments can be found from the Laplace transform as
L30-1 EEL 5544 Noise in Linear Systems Lecture 30 OTHER TRANSFORMS For a continuous, nonnegative RV X, the Laplace transform of X is X (s) = E [ e sx] = 0 f X (x)e sx dx. For a nonnegative RV, the Laplace
More informationFormulas for probability theory and linear models SF2941
Formulas for probability theory and linear models SF2941 These pages + Appendix 2 of Gut) are permitted as assistance at the exam. 11 maj 2008 Selected formulae of probability Bivariate probability Transforms
More informationUnbiased Estimation. Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others.
Unbiased Estimation Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others. To compare ˆθ and θ, two estimators of θ: Say ˆθ is better than θ if it
More informationTwo hours. Statistical Tables to be provided THE UNIVERSITY OF MANCHESTER. 14 January :45 11:45
Two hours Statistical Tables to be provided THE UNIVERSITY OF MANCHESTER PROBABILITY 2 14 January 2015 09:45 11:45 Answer ALL four questions in Section A (40 marks in total) and TWO of the THREE questions
More informationSTAT 512 sp 2018 Summary Sheet
STAT 5 sp 08 Summary Sheet Karl B. Gregory Spring 08. Transformations of a random variable Let X be a rv with support X and let g be a function mapping X to Y with inverse mapping g (A = {x X : g(x A}
More informationVariations. ECE 6540, Lecture 10 Maximum Likelihood Estimation
Variations ECE 6540, Lecture 10 Last Time BLUE (Best Linear Unbiased Estimator) Formulation Advantages Disadvantages 2 The BLUE A simplification Assume the estimator is a linear system For a single parameter
More informationSolutions to Homework Set #6 (Prepared by Lele Wang)
Solutions to Homework Set #6 (Prepared by Lele Wang) Gaussian random vector Given a Gaussian random vector X N (µ, Σ), where µ ( 5 ) T and 0 Σ 4 0 0 0 9 (a) Find the pdfs of i X, ii X + X 3, iii X + X
More informationt x 1 e t dt, and simplify the answer when possible (for example, when r is a positive even number). In particular, confirm that EX 4 = 3.
Mathematical Statistics: Homewor problems General guideline. While woring outside the classroom, use any help you want, including people, computer algebra systems, Internet, and solution manuals, but mae
More informationST5215: Advanced Statistical Theory
Department of Statistics & Applied Probability Wednesday, October 19, 2011 Lecture 17: UMVUE and the first method of derivation Estimable parameters Let ϑ be a parameter in the family P. If there exists
More informationMath 362, Problem set 1
Math 6, roblem set Due //. (4..8) Determine the mean variance of the mean X of a rom sample of size 9 from a distribution having pdf f(x) = 4x, < x
More informationUCSD ECE153 Handout #34 Prof. Young-Han Kim Tuesday, May 27, Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE53 Handout #34 Prof Young-Han Kim Tuesday, May 7, 04 Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei) Linear estimator Consider a channel with the observation Y XZ, where the
More informationHT Introduction. P(X i = x i ) = e λ λ x i
MODS STATISTICS Introduction. HT 2012 Simon Myers, Department of Statistics (and The Wellcome Trust Centre for Human Genetics) myers@stats.ox.ac.uk We will be concerned with the mathematical framework
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationChapter 3: Unbiased Estimation Lecture 22: UMVUE and the method of using a sufficient and complete statistic
Chapter 3: Unbiased Estimation Lecture 22: UMVUE and the method of using a sufficient and complete statistic Unbiased estimation Unbiased or asymptotically unbiased estimation plays an important role in
More informationECE 275A Homework 7 Solutions
ECE 275A Homework 7 Solutions Solutions 1. For the same specification as in Homework Problem 6.11 we want to determine an estimator for θ using the Method of Moments (MOM). In general, the MOM estimator
More informationMultiple Random Variables
Multiple Random Variables This Version: July 30, 2015 Multiple Random Variables 2 Now we consider models with more than one r.v. These are called multivariate models For instance: height and weight An
More informationStatistics and Econometrics I
Statistics and Econometrics I Point Estimation Shiu-Sheng Chen Department of Economics National Taiwan University September 13, 2016 Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13,
More informationSpring 2012 Math 541B Exam 1
Spring 2012 Math 541B Exam 1 1. A sample of size n is drawn without replacement from an urn containing N balls, m of which are red and N m are black; the balls are otherwise indistinguishable. Let X denote
More informationMath 494: Mathematical Statistics
Math 494: Mathematical Statistics Instructor: Jimin Ding jmding@wustl.edu Department of Mathematics Washington University in St. Louis Class materials are available on course website (www.math.wustl.edu/
More informationRandom Variables. Random variables. A numerically valued map X of an outcome ω from a sample space Ω to the real line R
In probabilistic models, a random variable is a variable whose possible values are numerical outcomes of a random phenomenon. As a function or a map, it maps from an element (or an outcome) of a sample
More informationMethods of evaluating estimators and best unbiased estimators Hamid R. Rabiee
Stochastic Processes Methods of evaluating estimators and best unbiased estimators Hamid R. Rabiee 1 Outline Methods of Mean Squared Error Bias and Unbiasedness Best Unbiased Estimators CR-Bound for variance
More informationQuick Tour of Basic Probability Theory and Linear Algebra
Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra CS224w: Social and Information Network Analysis Fall 2011 Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra Outline Definitions
More information557: MATHEMATICAL STATISTICS II BIAS AND VARIANCE
557: MATHEMATICAL STATISTICS II BIAS AND VARIANCE An estimator, T (X), of θ can be evaluated via its statistical properties. Typically, two aspects are considered: Expectation Variance either in terms
More informationMathematical statistics
October 1 st, 2018 Lecture 11: Sufficient statistic Where are we? Week 1 Week 2 Week 4 Week 7 Week 10 Week 14 Probability reviews Chapter 6: Statistics and Sampling Distributions Chapter 7: Point Estimation
More informationPart IB Statistics. Theorems with proof. Based on lectures by D. Spiegelhalter Notes taken by Dexter Chua. Lent 2015
Part IB Statistics Theorems with proof Based on lectures by D. Spiegelhalter Notes taken by Dexter Chua Lent 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly)
More informationLecture 8: Information Theory and Statistics
Lecture 8: Information Theory and Statistics Part II: Hypothesis Testing and I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 23, 2015 1 / 50 I-Hsiang
More informationUnbiased Estimation. Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others.
Unbiased Estimation Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others. To compare ˆθ and θ, two estimators of θ: Say ˆθ is better than θ if it
More informationLecture 7 Introduction to Statistical Decision Theory
Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7
More information2. Variance and Covariance: We will now derive some classic properties of variance and covariance. Assume real-valued random variables X and Y.
CS450 Final Review Problems Fall 08 Solutions or worked answers provided Problems -6 are based on the midterm review Identical problems are marked recap] Please consult previous recitations and textbook
More informationUCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, Homework Set #6 Due: Thursday, May 22, 2011
UCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, 2014 Homework Set #6 Due: Thursday, May 22, 2011 1. Linear estimator. Consider a channel with the observation Y = XZ, where the signal X and
More informationMathematical Statistics
Mathematical Statistics Chapter Three. Point Estimation 3.4 Uniformly Minimum Variance Unbiased Estimator(UMVUE) Criteria for Best Estimators MSE Criterion Let F = {p(x; θ) : θ Θ} be a parametric distribution
More informationChapter 8.8.1: A factorization theorem
LECTURE 14 Chapter 8.8.1: A factorization theorem The characterization of a sufficient statistic in terms of the conditional distribution of the data given the statistic can be difficult to work with.
More informationEC212: Introduction to Econometrics Review Materials (Wooldridge, Appendix)
1 EC212: Introduction to Econometrics Review Materials (Wooldridge, Appendix) Taisuke Otsu London School of Economics Summer 2018 A.1. Summation operator (Wooldridge, App. A.1) 2 3 Summation operator For
More information18 Bivariate normal distribution I
8 Bivariate normal distribution I 8 Example Imagine firing arrows at a target Hopefully they will fall close to the target centre As we fire more arrows we find a high density near the centre and fewer
More informationBASICS OF PROBABILITY
October 10, 2018 BASICS OF PROBABILITY Randomness, sample space and probability Probability is concerned with random experiments. That is, an experiment, the outcome of which cannot be predicted with certainty,
More information10-704: Information Processing and Learning Fall Lecture 24: Dec 7
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 24: Dec 7 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy of
More informationUC Berkeley Department of Electrical Engineering and Computer Sciences. EECS 126: Probability and Random Processes
UC Berkeley Department of Electrical Engineering and Computer Sciences EECS 6: Probability and Random Processes Problem Set 3 Spring 9 Self-Graded Scores Due: February 8, 9 Submit your self-graded scores
More informationFor a stochastic process {Y t : t = 0, ±1, ±2, ±3, }, the mean function is defined by (2.2.1) ± 2..., γ t,
CHAPTER 2 FUNDAMENTAL CONCEPTS This chapter describes the fundamental concepts in the theory of time series models. In particular, we introduce the concepts of stochastic processes, mean and covariance
More informationExpectation. DS GA 1002 Probability and Statistics for Data Science. Carlos Fernandez-Granda
Expectation DS GA 1002 Probability and Statistics for Data Science http://www.cims.nyu.edu/~cfgranda/pages/dsga1002_fall17 Carlos Fernandez-Granda Aim Describe random variables with a few numbers: mean,
More informationECE302 Exam 2 Version A April 21, You must show ALL of your work for full credit. Please leave fractions as fractions, but simplify them, etc.
ECE32 Exam 2 Version A April 21, 214 1 Name: Solution Score: /1 This exam is closed-book. You must show ALL of your work for full credit. Please read the questions carefully. Please check your answers
More informationPart IA Probability. Definitions. Based on lectures by R. Weber Notes taken by Dexter Chua. Lent 2015
Part IA Probability Definitions Based on lectures by R. Weber Notes taken by Dexter Chua Lent 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after lectures.
More informationJoint Probability Distributions and Random Samples (Devore Chapter Five)
Joint Probability Distributions and Random Samples (Devore Chapter Five) 1016-345-01: Probability and Statistics for Engineers Spring 2013 Contents 1 Joint Probability Distributions 2 1.1 Two Discrete
More informationFinal Examination Solutions (Total: 100 points)
Final Examination Solutions (Total: points) There are 4 problems, each problem with multiple parts, each worth 5 points. Make sure you answer all questions. Your answer should be as clear and readable
More informationBivariate distributions
Bivariate distributions 3 th October 017 lecture based on Hogg Tanis Zimmerman: Probability and Statistical Inference (9th ed.) Bivariate Distributions of the Discrete Type The Correlation Coefficient
More informationTheory of Statistics.
Theory of Statistics. Homework V February 5, 00. MT 8.7.c When σ is known, ˆµ = X is an unbiased estimator for µ. If you can show that its variance attains the Cramer-Rao lower bound, then no other unbiased
More informationNotes, March 4, 2013, R. Dudley Maximum likelihood estimation: actual or supposed
18.466 Notes, March 4, 2013, R. Dudley Maximum likelihood estimation: actual or supposed 1. MLEs in exponential families Let f(x,θ) for x X and θ Θ be a likelihood function, that is, for present purposes,
More informationconditional cdf, conditional pdf, total probability theorem?
6 Multiple Random Variables 6.0 INTRODUCTION scalar vs. random variable cdf, pdf transformation of a random variable conditional cdf, conditional pdf, total probability theorem expectation of a random
More information6 The normal distribution, the central limit theorem and random samples
6 The normal distribution, the central limit theorem and random samples 6.1 The normal distribution We mentioned the normal (or Gaussian) distribution in Chapter 4. It has density f X (x) = 1 σ 1 2π e
More informationLecture 7: Chapter 7. Sums of Random Variables and Long-Term Averages
Lecture 7: Chapter 7. Sums of Random Variables and Long-Term Averages ELEC206 Probability and Random Processes, Fall 2014 Gil-Jin Jang gjang@knu.ac.kr School of EE, KNU page 1 / 15 Chapter 7. Sums of Random
More informationChapter 7. Basic Probability Theory
Chapter 7. Basic Probability Theory I-Liang Chern October 20, 2016 1 / 49 What s kind of matrices satisfying RIP Random matrices with iid Gaussian entries iid Bernoulli entries (+/ 1) iid subgaussian entries
More informationEstimation Theory. as Θ = (Θ 1,Θ 2,...,Θ m ) T. An estimator
Estimation Theory Estimation theory deals with finding numerical values of interesting parameters from given set of data. We start with formulating a family of models that could describe how the data were
More informationESTIMATION THEORY. Chapter Estimation of Random Variables
Chapter ESTIMATION THEORY. Estimation of Random Variables Suppose X,Y,Y 2,...,Y n are random variables defined on the same probability space (Ω, S,P). We consider Y,...,Y n to be the observed random variables
More informationCourse: ESO-209 Home Work: 1 Instructor: Debasis Kundu
Home Work: 1 1. Describe the sample space when a coin is tossed (a) once, (b) three times, (c) n times, (d) an infinite number of times. 2. A coin is tossed until for the first time the same result appear
More informationSolutions to Homework Set #5 (Prepared by Lele Wang) MSE = E [ (sgn(x) g(y)) 2],, where f X (x) = 1 2 2π e. e (x y)2 2 dx 2π
Solutions to Homework Set #5 (Prepared by Lele Wang). Neural net. Let Y X + Z, where the signal X U[,] and noise Z N(,) are independent. (a) Find the function g(y) that minimizes MSE E [ (sgn(x) g(y))
More informationECE 275A Homework 6 Solutions
ECE 275A Homework 6 Solutions. The notation used in the solutions for the concentration (hyper) ellipsoid problems is defined in the lecture supplement on concentration ellipsoids. Note that θ T Σ θ =
More informationECON 3150/4150, Spring term Lecture 6
ECON 3150/4150, Spring term 2013. Lecture 6 Review of theoretical statistics for econometric modelling (II) Ragnar Nymoen University of Oslo 31 January 2013 1 / 25 References to Lecture 3 and 6 Lecture
More informationPerhaps the simplest way of modeling two (discrete) random variables is by means of a joint PMF, defined as follows.
Chapter 5 Two Random Variables In a practical engineering problem, there is almost always causal relationship between different events. Some relationships are determined by physical laws, e.g., voltage
More informationStatistics: Learning models from data
DS-GA 1002 Lecture notes 5 October 19, 2015 Statistics: Learning models from data Learning models from data that are assumed to be generated probabilistically from a certain unknown distribution is a crucial
More informationPhysics 403. Segev BenZvi. Parameter Estimation, Correlations, and Error Bars. Department of Physics and Astronomy University of Rochester
Physics 403 Parameter Estimation, Correlations, and Error Bars Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Review of Last Class Best Estimates and Reliability
More informationStatistics 351 Probability I Fall 2006 (200630) Final Exam Solutions. θ α β Γ(α)Γ(β) (uv)α 1 (v uv) β 1 exp v }
Statistics 35 Probability I Fall 6 (63 Final Exam Solutions Instructor: Michael Kozdron (a Solving for X and Y gives X UV and Y V UV, so that the Jacobian of this transformation is x x u v J y y v u v
More informationClassical Estimation Topics
Classical Estimation Topics Namrata Vaswani, Iowa State University February 25, 2014 This note fills in the gaps in the notes already provided (l0.pdf, l1.pdf, l2.pdf, l3.pdf, LeastSquares.pdf). 1 Min
More informationSTA 6857 Autocorrelation and Cross-Correlation & Stationary Time Series ( 1.4, 1.5)
STA 6857 Autocorrelation and Cross-Correlation & Stationary Time Series ( 1.4, 1.5) Outline 1 Announcements 2 Autocorrelation and Cross-Correlation 3 Stationary Time Series 4 Homework 1c Arthur Berg STA
More informationNotes 6 : First and second moment methods
Notes 6 : First and second moment methods Math 733-734: Theory of Probability Lecturer: Sebastien Roch References: [Roc, Sections 2.1-2.3]. Recall: THM 6.1 (Markov s inequality) Let X be a non-negative
More information4 Derivations of the Discrete-Time Kalman Filter
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof N Shimkin 4 Derivations of the Discrete-Time
More informationREVIEW OF MAIN CONCEPTS AND FORMULAS A B = Ā B. Pr(A B C) = Pr(A) Pr(A B C) =Pr(A) Pr(B A) Pr(C A B)
REVIEW OF MAIN CONCEPTS AND FORMULAS Boolean algebra of events (subsets of a sample space) DeMorgan s formula: A B = Ā B A B = Ā B The notion of conditional probability, and of mutual independence of two
More informationChp 4. Expectation and Variance
Chp 4. Expectation and Variance 1 Expectation In this chapter, we will introduce two objectives to directly reflect the properties of a random variable or vector, which are the Expectation and Variance.
More informationLecture 6: Gaussian Channels. Copyright G. Caire (Sample Lectures) 157
Lecture 6: Gaussian Channels Copyright G. Caire (Sample Lectures) 157 Differential entropy (1) Definition 18. The (joint) differential entropy of a continuous random vector X n p X n(x) over R is: Z h(x
More informationCHAPTER 5. Jointly Probability Mass Function for Two Discrete Distributed Random Variables:
CHAPTER 5 Jointl Distributed Random Variable There are some situations that experiment contains more than one variable and researcher interested in to stud joint behavior of several variables at the same
More informationWe introduce methods that are useful in:
Instructor: Shengyu Zhang Content Derived Distributions Covariance and Correlation Conditional Expectation and Variance Revisited Transforms Sum of a Random Number of Independent Random Variables more
More informationIntroduction to Normal Distribution
Introduction to Normal Distribution Nathaniel E. Helwig Assistant Professor of Psychology and Statistics University of Minnesota (Twin Cities) Updated 17-Jan-2017 Nathaniel E. Helwig (U of Minnesota) Introduction
More informationFebruary 26, 2017 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM
February 26, 2017 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM Abstract. The Rao-Blacwell theorem told us how to improve an estimator. We will discuss conditions on when the Rao-Blacwellization of an estimator
More informationENGG2430A-Homework 2
ENGG3A-Homework Due on Feb 9th,. Independence vs correlation a For each of the following cases, compute the marginal pmfs from the joint pmfs. Explain whether the random variables X and Y are independent,
More informationThe Delta Method and Applications
Chapter 5 The Delta Method and Applications 5.1 Local linear approximations Suppose that a particular random sequence converges in distribution to a particular constant. The idea of using a first-order
More informationProbability and Measure
Part II Year 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 2018 84 Paper 4, Section II 26J Let (X, A) be a measurable space. Let T : X X be a measurable map, and µ a probability
More information