A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2011
|
|
- Rafe Horn
- 5 years ago
- Views:
Transcription
1 A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2011 Reading Chapter 5 (continued) Lecture 8 Key points in probability CLT CLT examples
2 Prior vs Likelihood Box & Tiao
3 Learning in Bayesian Estimation Box & Tiao
4
5
6
7
8
9 3 I. Mutually exclusive events: If a occurs then b cannot have occurred. Let c = a + b + or (same as a b) P (c) =P {a or b occurred} = P (a)+p(b) Let d = a b and (same as a b) P (d) =P {a and b occurred} =0 ifmutuallyexclusive II. Non-mutually exclusive events: P (c) =P {a or b} = P (a)+p (b) P (ab) }{{} III. Independent events: P (ab) P (a)p (b) Examples I. Mutually exclusive events toss a coin once: 2possibleoutcomesH&T H&Taremutuallyexclusive H&Tarenotindependent because P (HT)=P{heads & tails} =0soP (HT) P (H)P (T ).
10 4 II. Independent events toss a coin twice = experiment The outcomes of the experiment are events might be defined as: H 1 H 2 =eventthathon1sttoss,hon2nd H 1 T 2 =eventthathon1sttoss,ton2nd T 1 H 2 =eventthatton1sttoss,hon2nd T 1 T 2 =eventthatton1sttoss,ton2nd 1st toss 2nd toss H 1 H 2 H 1 T 2 T 1 H 2 T 1 T 2 note P (H 1 H 2 )=P (H 1 )P (H 2 )[aslongascoinnotalteredbetweentosses]
11 5 Random Variables Of interest to us is the distribution of probability along the realnumberaxis: Random variables assign numbers to events or, more precisely, map the event space into a set of numbers: a X(a) event number The definition of probability translates directly over to thenumbersthatareassignedbyrandomvariables. The following properties are true for a real random variable. 1. Let {X x} =eventthatther.v.x is less than the number x; definedforallx [this defines all intervals on the real number line to be events] 2. the events {X =+ } and {X = } have zero probability. (Otherwise, moments would not be finite, generally.) Distribution function: (CDF = Cumulative Distribution Function) properties: F X (x) =P {X x} P {all events A : X(A) x} 1. F X (x) isamonotonicallyincreasingfunctionofx. 2. F ( ) =0,F (+ ) =1 3. P {x 1 X x 2 } = F (x 2 ) F (x 1 ) Probability Density Function (pdf) properties: f X (x) = df X(x) dx 1. f X (x) dx = P {x X x + dx} 2. dx f X(x) =F X ( ) F X ( ) =1 0=1
12 All three measures are localization measures Other quantities are needed to measure the width and asymmetry of the PDF, etc.
13 6 Continuous r.v. s: derivativeoff X (x) exists x Discrete random variables: use delta functions to write the pdf in pseudo continuous form. e.g. coin flipping 1 heads Let X = 1 tails then f X (x) = 1 [δ(x +1)+δ(x 1)] 2 F X (x) = 1 [U(x +1)+U(x 1)] 2 Functions of a random variable: The function Y = g(x) isarandomvariablethatisamappingfrom some event A to a number Y according to: Y (A) =g[x(a)] Theorem, ify = g(x), then the pdf of Y is f Y (y) = n j=1 f X (x j ) dg(x)/dx x=xj, where x j,j =1,n are the solutions of x = g 1 (y). Note the normalization property is conserved (unit area). This is one of the most important equations! Example * Y = g(x) =ax + b dg dx = a g 1 (y) = x 1 = y b a f X (x 1 ) f Y (y) = dg(x 1 )/dx = a 1 f X ( y b a ).
14 Comment about natural random number generators
15 7 To check: show that dy f Y (y) =1 Example Suppose we want to transform from a uniform distribution to an exponential distribution: We want ant f Y (y) =exp( y). A typical random number generator gives f X (x) with { 1, 0 x<1; f X (x) = 0, otherwise. Choose y = g(x) = ln(x). Then: Moments dg dx = 1 x x 1 = g 1 (y) =e y f Y (y) = f X[exp( y)] 1/x 1 = x 1 =e y. Factoid: Poission events in time have spacings that are exponentially distributed We will always use angular brackets < > to denote average over an ensemble (integrating over an ensemble); time averages and other sample averages will be denoted differently. Expected value of a random variable: E(X) X = dx xf X (x) Arbitrary power: denotes expectation w.r.t. the PDF of x X n = dx x n f X (x) Variance: σ 2 x = X2 X 2 Function of a random variable: If y = g(x) and Y dy y f Y (y) then it is easy to show that Y = dx g(x)f X (x). Proof: y dy f Y (y) = dy n j=1 f X [x j (y)] dg[x j (y)]/dx
16 8 A change of variable: dy = dg dx yields the result. dx Central Moments: µ n = (X X ) n Moment Tests: Moments are useful for testing hypotheses such as whether a given PDF is consistent with data: E.g. Consistency with Gaussian PDF: kurtosis k = µ 4 3=0 µ 3/2 2 skewness parameter γ = µ 3 =0 µ 3/2 2 k > 0 4th moment proportionately larger larger amplitude tail than Gaussian and less probable values near the mean.
17 9 Uses of Moments: Often one wants to infer the underlying PDF of an observable, e.g. perhaps because determination of the PDF is tantamount to understanding the underlying physics of some process. Two approaches are: 1. construct a histogram and compare the shape with a theoretical shape. 2. determine some of the moments (usually low-order) and compare. Suppose the data are {x j,j =1,N} 1. One could form bins of size x and count how many x j fall into each bin. If N is large enough so that n k = # points in the k-th bin is also large, then a reasonably good estimate of the PDF can be made. (But beware of dependence of results on choice of binning.) 2. However, often times N is too small or one would like to determine only basic information about the shape of the distribution (is it symmetric?), or determine the mean and variance of the PDF or test whether the data are consistent with a given PDF (hypothesis testing). Some of the typical situations are: i) assume the data were drawn from a Gaussian parent PDF; estimate themeanandσ of the Gaussian [parameter estimation] ii) test whether the data are consistent with a Gaussian PDF [moment test] note that if the r.v. is zero mean then the PDF is determined solely by one parameter: σ 1 /2σ f X (x) = 2 2πσ 2 e x2 The moments are (n 1)σ n (n 1)!! σ n n even x n = 0 n odd Therefore, the n = 2 moment = 1st non-zero moment all other moments. This statement remains for more multi-dimensional Gaussian processes: Any moment of order higher than 3 is redundant... gaussianity. or can be used as a test for
18 10 Characteristic Function: Of considerable use is the characteristic function Φ X (ω) e iωx dx f X (x) e iωx. If we know Φ X (ω) then we know all there is to know about the PDF because f X (x) = 1 dω Φ X (ω) e iωx 2π is the inversion formula. If we know all the moments of f X (x), then we also can completely characterize f X (x). Similarly, the characteristic function is a moment-generating function: Φ X (ω) = e iωx n=0 (iωx) n = n! because the expectation of the sum = sum of the expectations. By taking derivatives we can show that or Φ ω ω=0 = i X 2 Φ ω 2 ω=0 = i2 X 2 k Φ ω k ω=0 = in X n n=0 (iω) n n! X n X n = i n n Φ ω n ω=0 =( i) n n Φ ω n ω=0 Price stheorem Characteristic functions are useful for deriving PDFs of combinations of r.v. s as well as for deriving particular moments.
19 11 Joint Random Variables Let X and Y be two random variables with their associated sample spaces. The actual events associated with X and Y may or may not be independent (e.g. throwing a die may map into X; choosing colored marbles from a hat may map into Y ). The relationship of the events will be described by the joint distribution function of X and Y : and the joint probability density function is F XY (x, y) P {X x, Y y} f XY (x, y) 2 F xy (x, y) x y (a two dimensional PDF) Note that the one dimensional PDF of X, for example, is obtained by integrating the joint PDF over all y: f X (x) = dy f XY (x, y) which corresponds to asking what the PFf of X is given that the certain event for Y occurs. Example: flip two coins a and b. Let heads =1; tails =0. Define 2 r.v. s: X = a + b; Y = a. With these definitions X + Y are statistically dependent. Characteristic function of joint r.v. s: Φ XY (ω 1, ω 2 ) = e i(ω1x+ω2y ) = dx dy e i(ω1x+ω2y) f XY (x, y). For x, y independent [ ][ ] Φ XY (ω 1, ω 2 )= dx f X (x) e iω1x dy f Y (y) e iω2y Φ X (ω 1 ) Φ Y (ω 2 ). Example for independent r.v. s: flip two coins a and b. As before, heads = 1 and tails = 0, let x = a, y = b (x and y are independent). Independent random variables Two random variables are said to be independent if the events mapping into one r.v. are independent of those mapping into the other.
20 12 In this case, joint probabilities are factorable so that F XY (x, y) = F X (x) F Y (y) f XY (x, y) = f X (x) f Y (y). Such factorization is plausible if one considers moments of independent r.v. s: X n Y m = X n Y m which follows from X n Y m dx dy x n y m f XY (x, y) =[ [ dx x n f X (x)] ] dy y m f Y (y).
21 13 Convolution theorem for sums of independent RVs If Z = X +Y where X, Y are independent random variables, then the PDF of Z is the convolution of the PDFs of X and Y : f Z (z) =f X (x) f Y (y) = dx f X (x) f Y (z x) = dx f X (z x) f Y (x). proof: By definition, Consider Now, as before, this is f Z (z) = d dz F Z(z) F z (z) =P {Z z} F Z (z) = P {X + Y z} = P {Y z X}. To evaluate this, first evaluate the probability P {Y z x} where x is just a number. Now P {Y z x} F Y (z x) z x dy f Y (y) but P {Y z X} is the probability that Y z x for all values of x so we need to integrate over x and weight by the probability of x: P {Y z X} = dx f X (x) z x dy F Y (y) that is, P {Y z X} is the expected value of F Y (z x). By the Leibniz integration formula d db we obtain the convolution results. g(b) a dω h(ω) h(g(b)) dg(b) db
22 14 Characteristic function of Z = X + Y For X, Y independent we have f Z = f X f Y Φ Z (ω) = e iωz = Φ X (ω) Φ Y (ω) Variance of Z: if variance of X and Y are σ 2 X, σ2 Y, then variance of Z is σ2 Z = σ2 X + σ2 Y. Assume X and Y and hence Z are zero mean r.v. s, thenwehave σ 2 X = x2 = i 2 2 φ x ω 2 (ω =0) = 2 φ x ω 2 (ω =0) σ 2 Y = y2 = 2 φ y ω 2 (ω =0) Using Price s theorem: σz 2 = Z2 = 2 φ Z (ω =0) ω2 = 2 ω 2 [φ X(ω) φ Y (ω)] ω=0 = ω [ = φ X [ φ X 2 φ Y ω 2 φ Y ω + φ Y + φ Y 2 φ x ω 2 φ ] X ω ω=0 +2 φ X ω φ ] Y. ω ω=0 We have discovered that variances add (independent variables only): σ 2 Z = σ 2 X + σ 2 Y.
23 Multivariate random variables: N dimensional The results for the bivariate case are easily extrapolated. If where the X j are all independent r.v. s, then Z = X 1 + X X N = N j=1 X j f Z (z) =f X1 f X2... f XN and and N Φ Z = Φ Xj (ω) j=1 N σz 2 σx 2 j. j=1
24 16 Central Limit Theorem: Let Z N = 1 N N X j j=1 where the X j are independent r.v. s with means and variances µ j X j σ 2 j = X 2 j X j 2. and the PDFs of the X j s are almost arbitrary. Restrictions on the distributions of each X j are that i) σ 2 j >m>0 m =constant ii) X n <M=constantforn>2 In the limit N,Z N becomes a Gaussian random variable with mean Z N = 1 N N µ j j=1 and variance σ 2 Z = 1 N N σj 2. j=1 Example: supposethex j are all uniformly distributed between ± 1 2,so f X (x) =Π(x) sin πf πf = sin ω 2 ω/2
25 17 Thus the characteristic function is Φ j (ω) = e iωx j = sin ω/2 ω/2 Graphically: Gaussian N =2 N =3 N = e x2 ( sin ω 2 ω/2 )2 sin ω/2 ( ω/2 )3 e ω2 From the convolution results we have ( sin ω/2 φ NZ N (ω) = ω/2 From the transformation of random variables we have that ) N f ZN (x) = Nf NZ N ( Nx) and by the scaling theorem for Fourier transforms φ ZN (ω) =φ NZ N ( ω ) ( sin ω/2 N ) N. N = ω/2 N
26 Now or if the CLT holds: lim N φ Z N (ω) =e 1 2 ω2 σ 2 Z f ZN (x) = 1 2πσ 2 Z e x2 /2σ 2 Z. 18 Consistency with this limiting form can be seen by expanding φ ZN for small ω ( ω/2 N 1 3! φ ZN (ω) (ω/2 N) 3 ) N ω/2 ω 2 1 N 24 that is identical to the expansion of exp ( ω 2 σ 2 Z /2).
27 CLT Comments A sum of Gaussian RVs is automatically a Gaussian RV (can show using characteristic functions) Convergence to a Gaussian form depends on the actual PDFs of the terms in the sum and their relative variances Exceptions exist!
28 19 CLT: Example of a PDF that does not work The Cauchy distribution and its characteristic function are f X (x) = α π Φ(w) = e α ω 1 α 2 + x 2 Now has a characteristic function Z N = 1 N N x j j=1 Φ N (ω) =e Nα ω / N By inspection the exponential will not converge to a Gaussian. Instead, the sum of N Cauchy RVs is a Cauchy RV. Is the Cauchy distribution a legitimate PDF? No! The variance diverges: X 2 = dx x 2 α π 1 α 2 + x 2.
29 A CLT Problem Consider a set of N quantities that are i.i.d. (independently and identically distributed) with zero mean {a i, i =1,...,N} a i = 0 a i a j = σ 2 aδ ij We are interested in the cross correlation between all unique pairs C N = 1 N X i<j N X = N(N 1)/2 a i a j = 1 N X N 1 i=1 What do you expect <C N > to be? What do you expect the PDF of C N to be? N j=i+1 a i a j
30 A CLT Problem (2) Note: The number of independent quantities (random variables) is N The sum C N has terms that are products of i.i.d. variables Any given term in the sum is s.i. of some of the other terms The PDF of products is different from the PDF of individual factors In the limit N >> 1 there should be many independent terms in the sum N=2: Can show that PDF is symmetric (odd order moments = 0) N>2: Can show that the third moment 0 What gives?
31
32 20 Conditional Probabilities & Bayes Theroem We have considered P (ζ), the probability of an event ζ. Also obeying axioms of probability are conditional probabilities: P (ψ ζ), the probability of the event ψ given that the event ζ has occurred. Recast the axioms as P (ψ ζ) P (ψζ) P (ζ) I. P (ψ ζ) 0 II. P (ψ ζ)+p( ψ ζ) =1 III. P (ψζ η) = P (ψ η)p (ζ ψη) = P (ζ η)p (ψ ζη) How does this relate to experiments? Use the product rule: P (ζ ψη) = P (ζ η)p (ψ ζη) P (ψ η) or, letting M = model (or hypothesis), D = data and I = background information (assumptions), Terms: P (M DI) =P (M I) P (D MI) P (D I) prior: P (M I) sampling distribution for D: P (D MI) (also called likelihood for M) prior predictive for D: P (D I) (also called global likelihood for M or evidence for M)
33 21 Particular strengths of Bayesian method include: 1. One must often be explicit about what is assumed about I, the background information. 2. In assessing models, we get a PDF for parameters rather than just point estimates. 3. Occam s razor (simpler models win, all else being equal) is easily invoked when comparing models. We may have many different models, M i that we wish to compare. Form the odds ratio: fromtheposterior PDFs: P (M i DI): O i,j P (M i DI) P (M j DI) = P (M i I) P (D M i I) P (M j I) P (D M j I).
34 22 Example Data: {k i },i=1,...,n, drawn from Poisson process Poisson PDF: P k = λk e λ k! Want: mean of process Frequentist approach: We need an estimator for the mean; consider the likelihood f(λ) = n 1 n P (k i )= n i=1 k i! λ i=1 ki e nλ. i=1 Maximizing, [ df dλ =0=f(λ) n + λ 1 ] n k i i=1 we obtain an estimator for the mean is k = 1 n n k i. i=1
35 23 Bayesian approach: Likelihood (as before): P (D MI) = n 1 n P (k i )= n ı=1 k i! λ i=1 k i e nλ. i=1 Prior: Assume P (M I) =P (λ I) P (λ I)λ λ U(λ) Prior Predictive: P (D I) dλ U(λ)P (D MI) = n n x n ı=1 k i! Γ(n x). Combining all the above, we find P (λ {k i }I) = nn x Γ(n x) λn x e nλ U(λ) Note that rather than getting a point estimate for the mean, we get a PDF for its value. For hypothesis testing, this is much more useful than a point estimate.
Probability, CLT, CLT counterexamples, Bayes. The PDF file of this lecture contains a full reference document on probability and random variables.
Lecture 5 A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2015 http://www.astro.cornell.edu/~cordes/a6523 Probability, CLT, CLT counterexamples, Bayes The PDF file of
More informationA6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2011
A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2011 Reading Chapter 5 of Gregory (Frequentist Statistical Inference) Lecture 7 Examples of FT applications Simulating
More informationA6523 Modeling, Inference, and Mining Jim Cordes, Cornell University. Motivations: Detection & Characterization. Lecture 2.
A6523 Modeling, Inference, and Mining Jim Cordes, Cornell University Lecture 2 Probability basics Fourier transform basics Typical problems Overall mantra: Discovery and cri@cal thinking with data + The
More informationLecture 2: Repetition of probability theory and statistics
Algorithms for Uncertainty Quantification SS8, IN2345 Tobias Neckel Scientific Computing in Computer Science TUM Lecture 2: Repetition of probability theory and statistics Concept of Building Block: Prerequisites:
More informationconditional cdf, conditional pdf, total probability theorem?
6 Multiple Random Variables 6.0 INTRODUCTION scalar vs. random variable cdf, pdf transformation of a random variable conditional cdf, conditional pdf, total probability theorem expectation of a random
More informationDeep Learning for Computer Vision
Deep Learning for Computer Vision Lecture 3: Probability, Bayes Theorem, and Bayes Classification Peter Belhumeur Computer Science Columbia University Probability Should you play this game? Game: A fair
More informationPart IA Probability. Definitions. Based on lectures by R. Weber Notes taken by Dexter Chua. Lent 2015
Part IA Probability Definitions Based on lectures by R. Weber Notes taken by Dexter Chua Lent 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after lectures.
More informationStatistics for scientists and engineers
Statistics for scientists and engineers February 0, 006 Contents Introduction. Motivation - why study statistics?................................... Examples..................................................3
More informationAlgorithms for Uncertainty Quantification
Algorithms for Uncertainty Quantification Tobias Neckel, Ionuț-Gabriel Farcaș Lehrstuhl Informatik V Summer Semester 2017 Lecture 2: Repetition of probability theory and statistics Example: coin flip Example
More informationCourse: ESO-209 Home Work: 1 Instructor: Debasis Kundu
Home Work: 1 1. Describe the sample space when a coin is tossed (a) once, (b) three times, (c) n times, (d) an infinite number of times. 2. A coin is tossed until for the first time the same result appear
More informationSummary of basic probability theory Math 218, Mathematical Statistics D Joyce, Spring 2016
8. For any two events E and F, P (E) = P (E F ) + P (E F c ). Summary of basic probability theory Math 218, Mathematical Statistics D Joyce, Spring 2016 Sample space. A sample space consists of a underlying
More informationMultiple Random Variables
Multiple Random Variables Joint Probability Density Let X and Y be two random variables. Their joint distribution function is F ( XY x, y) P X x Y y. F XY ( ) 1, < x
More informationRecitation 2: Probability
Recitation 2: Probability Colin White, Kenny Marino January 23, 2018 Outline Facts about sets Definitions and facts about probability Random Variables and Joint Distributions Characteristics of distributions
More informationChapter 2: Random Variables
ECE54: Stochastic Signals and Systems Fall 28 Lecture 2 - September 3, 28 Dr. Salim El Rouayheb Scribe: Peiwen Tian, Lu Liu, Ghadir Ayache Chapter 2: Random Variables Example. Tossing a fair coin twice:
More informationA6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring
Lecture 8 A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2015 http://www.astro.cornell.edu/~cordes/a6523 Applications: Bayesian inference: overview and examples Introduction
More informationL2: Review of probability and statistics
Probability L2: Review of probability and statistics Definition of probability Axioms and properties Conditional probability Bayes theorem Random variables Definition of a random variable Cumulative distribution
More informationPreliminary Statistics Lecture 2: Probability Theory (Outline) prelimsoas.webs.com
1 School of Oriental and African Studies September 2015 Department of Economics Preliminary Statistics Lecture 2: Probability Theory (Outline) prelimsoas.webs.com Gujarati D. Basic Econometrics, Appendix
More informationChapter 3: Random Variables 1
Chapter 3: Random Variables 1 Yunghsiang S. Han Graduate Institute of Communication Engineering, National Taipei University Taiwan E-mail: yshan@mail.ntpu.edu.tw 1 Modified from the lecture notes by Prof.
More informationChapter 2. Random Variable. Define single random variables in terms of their PDF and CDF, and calculate moments such as the mean and variance.
Chapter 2 Random Variable CLO2 Define single random variables in terms of their PDF and CDF, and calculate moments such as the mean and variance. 1 1. Introduction In Chapter 1, we introduced the concept
More informationChapter 2 Random Variables
Stochastic Processes Chapter 2 Random Variables Prof. Jernan Juang Dept. of Engineering Science National Cheng Kung University Prof. Chun-Hung Liu Dept. of Electrical and Computer Eng. National Chiao Tung
More informationFundamentals of Digital Commun. Ch. 4: Random Variables and Random Processes
Fundamentals of Digital Commun. Ch. 4: Random Variables and Random Processes Klaus Witrisal witrisal@tugraz.at Signal Processing and Speech Communication Laboratory www.spsc.tugraz.at Graz University of
More informationReview of Probability Theory
Review of Probability Theory Arian Maleki and Tom Do Stanford University Probability theory is the study of uncertainty Through this class, we will be relying on concepts from probability theory for deriving
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More informationwhere r n = dn+1 x(t)
Random Variables Overview Probability Random variables Transforms of pdfs Moments and cumulants Useful distributions Random vectors Linear transformations of random vectors The multivariate normal distribution
More informationLecture 2. Spring Quarter Statistical Optics. Lecture 2. Characteristic Functions. Transformation of RVs. Sums of RVs
s of Spring Quarter 2018 ECE244a - Spring 2018 1 Function s of The characteristic function is the Fourier transform of the pdf (note Goodman and Papen have different notation) C x(ω) = e iωx = = f x(x)e
More informationMath 416 Lecture 3. The average or mean or expected value of x 1, x 2, x 3,..., x n is
Math 416 Lecture 3 Expected values The average or mean or expected value of x 1, x 2, x 3,..., x n is x 1 x 2... x n n x 1 1 n x 2 1 n... x n 1 n 1 n x i p x i where p x i 1 n is the probability of x i
More informationChapter 3, 4 Random Variables ENCS Probability and Stochastic Processes. Concordia University
Chapter 3, 4 Random Variables ENCS6161 - Probability and Stochastic Processes Concordia University ENCS6161 p.1/47 The Notion of a Random Variable A random variable X is a function that assigns a real
More informationIntroduction to Probability and Stocastic Processes - Part I
Introduction to Probability and Stocastic Processes - Part I Lecture 2 Henrik Vie Christensen vie@control.auc.dk Department of Control Engineering Institute of Electronic Systems Aalborg University Denmark
More informationFourier and Stats / Astro Stats and Measurement : Stats Notes
Fourier and Stats / Astro Stats and Measurement : Stats Notes Andy Lawrence, University of Edinburgh Autumn 2013 1 Probabilities, distributions, and errors Laplace once said Probability theory is nothing
More informationFundamental Tools - Probability Theory II
Fundamental Tools - Probability Theory II MSc Financial Mathematics The University of Warwick September 29, 2015 MSc Financial Mathematics Fundamental Tools - Probability Theory II 1 / 22 Measurable random
More informationWhy study probability? Set theory. ECE 6010 Lecture 1 Introduction; Review of Random Variables
ECE 6010 Lecture 1 Introduction; Review of Random Variables Readings from G&S: Chapter 1. Section 2.1, Section 2.3, Section 2.4, Section 3.1, Section 3.2, Section 3.5, Section 4.1, Section 4.2, Section
More informationQuick Tour of Basic Probability Theory and Linear Algebra
Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra CS224w: Social and Information Network Analysis Fall 2011 Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra Outline Definitions
More informationNorthwestern University Department of Electrical Engineering and Computer Science
Northwestern University Department of Electrical Engineering and Computer Science EECS 454: Modeling and Analysis of Communication Networks Spring 2008 Probability Review As discussed in Lecture 1, probability
More informationBasics on Probability. Jingrui He 09/11/2007
Basics on Probability Jingrui He 09/11/2007 Coin Flips You flip a coin Head with probability 0.5 You flip 100 coins How many heads would you expect Coin Flips cont. You flip a coin Head with probability
More informationGaussian vectors and central limit theorem
Gaussian vectors and central limit theorem Samy Tindel Purdue University Probability Theory 2 - MA 539 Samy T. Gaussian vectors & CLT Probability Theory 1 / 86 Outline 1 Real Gaussian random variables
More information1 Random Variable: Topics
Note: Handouts DO NOT replace the book. In most cases, they only provide a guideline on topics and an intuitive feel. 1 Random Variable: Topics Chap 2, 2.1-2.4 and Chap 3, 3.1-3.3 What is a random variable?
More informationRandom Variables. P(x) = P[X(e)] = P(e). (1)
Random Variables Random variable (discrete or continuous) is used to derive the output statistical properties of a system whose input is a random variable or random in nature. Definition Consider an experiment
More informationPreliminary statistics
1 Preliminary statistics The solution of a geophysical inverse problem can be obtained by a combination of information from observed data, the theoretical relation between data and earth parameters (models),
More informationChapter 3: Random Variables 1
Chapter 3: Random Variables 1 Yunghsiang S. Han Graduate Institute of Communication Engineering, National Taipei University Taiwan E-mail: yshan@mail.ntpu.edu.tw 1 Modified from the lecture notes by Prof.
More informationPerhaps the simplest way of modeling two (discrete) random variables is by means of a joint PMF, defined as follows.
Chapter 5 Two Random Variables In a practical engineering problem, there is almost always causal relationship between different events. Some relationships are determined by physical laws, e.g., voltage
More informationE X A M. Probability Theory and Stochastic Processes Date: December 13, 2016 Duration: 4 hours. Number of pages incl.
E X A M Course code: Course name: Number of pages incl. front page: 6 MA430-G Probability Theory and Stochastic Processes Date: December 13, 2016 Duration: 4 hours Resources allowed: Notes: Pocket calculator,
More informationProbability theory for Networks (Part 1) CS 249B: Science of Networks Week 02: Monday, 02/04/08 Daniel Bilar Wellesley College Spring 2008
Probability theory for Networks (Part 1) CS 249B: Science of Networks Week 02: Monday, 02/04/08 Daniel Bilar Wellesley College Spring 2008 1 Review We saw some basic metrics that helped us characterize
More informationMultivariate random variables
Multivariate random variables DS GA 1002 Statistical and Mathematical Models http://www.cims.nyu.edu/~cfgranda/pages/dsga1002_fall16 Carlos Fernandez-Granda Joint distributions Tool to characterize several
More informationBASICS OF PROBABILITY
October 10, 2018 BASICS OF PROBABILITY Randomness, sample space and probability Probability is concerned with random experiments. That is, an experiment, the outcome of which cannot be predicted with certainty,
More informationMA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems
MA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems Review of Basic Probability The fundamentals, random variables, probability distributions Probability mass/density functions
More information1.1 Review of Probability Theory
1.1 Review of Probability Theory Angela Peace Biomathemtics II MATH 5355 Spring 2017 Lecture notes follow: Allen, Linda JS. An introduction to stochastic processes with applications to biology. CRC Press,
More informationLecture 1: August 28
36-705: Intermediate Statistics Fall 2017 Lecturer: Siva Balakrishnan Lecture 1: August 28 Our broad goal for the first few lectures is to try to understand the behaviour of sums of independent random
More information1: PROBABILITY REVIEW
1: PROBABILITY REVIEW Marek Rutkowski School of Mathematics and Statistics University of Sydney Semester 2, 2016 M. Rutkowski (USydney) Slides 1: Probability Review 1 / 56 Outline We will review the following
More informationReview (probability, linear algebra) CE-717 : Machine Learning Sharif University of Technology
Review (probability, linear algebra) CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Some slides have been adopted from Prof. H.R. Rabiee s and also Prof. R. Gutierrez-Osuna
More informationSection 9.1. Expected Values of Sums
Section 9.1 Expected Values of Sums Theorem 9.1 For any set of random variables X 1,..., X n, the sum W n = X 1 + + X n has expected value E [W n ] = E [X 1 ] + E [X 2 ] + + E [X n ]. Proof: Theorem 9.1
More informationChapter 2: Fundamentals of Statistics Lecture 15: Models and statistics
Chapter 2: Fundamentals of Statistics Lecture 15: Models and statistics Data from one or a series of random experiments are collected. Planning experiments and collecting data (not discussed here). Analysis:
More informationLecture 11. Probability Theory: an Overveiw
Math 408 - Mathematical Statistics Lecture 11. Probability Theory: an Overveiw February 11, 2013 Konstantin Zuev (USC) Math 408, Lecture 11 February 11, 2013 1 / 24 The starting point in developing the
More informationSTAT 414: Introduction to Probability Theory
STAT 414: Introduction to Probability Theory Spring 2016; Homework Assignments Latest updated on April 29, 2016 HW1 (Due on Jan. 21) Chapter 1 Problems 1, 8, 9, 10, 11, 18, 19, 26, 28, 30 Theoretical Exercises
More information1 Probability and Random Variables
1 Probability and Random Variables The models that you have seen thus far are deterministic models. For any time t, there is a unique solution X(t). On the other hand, stochastic models will result in
More information1 Presessional Probability
1 Presessional Probability Probability theory is essential for the development of mathematical models in finance, because of the randomness nature of price fluctuations in the markets. This presessional
More informationfor valid PSD. PART B (Answer all five units, 5 X 10 = 50 Marks) UNIT I
Code: 15A04304 R15 B.Tech II Year I Semester (R15) Regular Examinations November/December 016 PROBABILITY THEY & STOCHASTIC PROCESSES (Electronics and Communication Engineering) Time: 3 hours Max. Marks:
More informationBrief Review of Probability
Brief Review of Probability Nuno Vasconcelos (Ken Kreutz-Delgado) ECE Department, UCSD Probability Probability theory is a mathematical language to deal with processes or experiments that are non-deterministic
More informationUC Berkeley Department of Electrical Engineering and Computer Sciences. EECS 126: Probability and Random Processes
UC Berkeley Department of Electrical Engineering and Computer Sciences EECS 6: Probability and Random Processes Problem Set 3 Spring 9 Self-Graded Scores Due: February 8, 9 Submit your self-graded scores
More informationLecture Notes 5 Convergence and Limit Theorems. Convergence with Probability 1. Convergence in Mean Square. Convergence in Probability, WLLN
Lecture Notes 5 Convergence and Limit Theorems Motivation Convergence with Probability Convergence in Mean Square Convergence in Probability, WLLN Convergence in Distribution, CLT EE 278: Convergence and
More informationReview (Probability & Linear Algebra)
Review (Probability & Linear Algebra) CE-725 : Statistical Pattern Recognition Sharif University of Technology Spring 2013 M. Soleymani Outline Axioms of probability theory Conditional probability, Joint
More informationSUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416)
SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416) D. ARAPURA This is a summary of the essential material covered so far. The final will be cumulative. I ve also included some review problems
More information3F1 Random Processes Examples Paper (for all 6 lectures)
3F Random Processes Examples Paper (for all 6 lectures). Three factories make the same electrical component. Factory A supplies half of the total number of components to the central depot, while factories
More informationRandom Variables. Random variables. A numerically valued map X of an outcome ω from a sample space Ω to the real line R
In probabilistic models, a random variable is a variable whose possible values are numerical outcomes of a random phenomenon. As a function or a map, it maps from an element (or an outcome) of a sample
More informationLecture Notes 2 Random Variables. Discrete Random Variables: Probability mass function (pmf)
Lecture Notes 2 Random Variables Definition Discrete Random Variables: Probability mass function (pmf) Continuous Random Variables: Probability density function (pdf) Mean and Variance Cumulative Distribution
More informationIntroduction to Machine Learning
What does this mean? Outline Contents Introduction to Machine Learning Introduction to Probabilistic Methods Varun Chandola December 26, 2017 1 Introduction to Probability 1 2 Random Variables 3 3 Bayes
More informationContinuous Random Variables
1 / 24 Continuous Random Variables Saravanan Vijayakumaran sarva@ee.iitb.ac.in Department of Electrical Engineering Indian Institute of Technology Bombay February 27, 2013 2 / 24 Continuous Random Variables
More informationRandom Variables and Their Distributions
Chapter 3 Random Variables and Their Distributions A random variable (r.v.) is a function that assigns one and only one numerical value to each simple event in an experiment. We will denote r.vs by capital
More informationOrder Statistics and Distributions
Order Statistics and Distributions 1 Some Preliminary Comments and Ideas In this section we consider a random sample X 1, X 2,..., X n common continuous distribution function F and probability density
More informationFundamentals. CS 281A: Statistical Learning Theory. Yangqing Jia. August, Based on tutorial slides by Lester Mackey and Ariel Kleiner
Fundamentals CS 281A: Statistical Learning Theory Yangqing Jia Based on tutorial slides by Lester Mackey and Ariel Kleiner August, 2011 Outline 1 Probability 2 Statistics 3 Linear Algebra 4 Optimization
More informationLecture 6 Basic Probability
Lecture 6: Basic Probability 1 of 17 Course: Theory of Probability I Term: Fall 2013 Instructor: Gordan Zitkovic Lecture 6 Basic Probability Probability spaces A mathematical setup behind a probabilistic
More information2. Variance and Covariance: We will now derive some classic properties of variance and covariance. Assume real-valued random variables X and Y.
CS450 Final Review Problems Fall 08 Solutions or worked answers provided Problems -6 are based on the midterm review Identical problems are marked recap] Please consult previous recitations and textbook
More information1 Variance of a Random Variable
Indian Institute of Technology Bombay Department of Electrical Engineering Handout 14 EE 325 Probability and Random Processes Lecture Notes 9 August 28, 2014 1 Variance of a Random Variable The expectation
More information3. Probability and Statistics
FE661 - Statistical Methods for Financial Engineering 3. Probability and Statistics Jitkomut Songsiri definitions, probability measures conditional expectations correlation and covariance some important
More informationCME 106: Review Probability theory
: Probability theory Sven Schmit April 3, 2015 1 Overview In the first half of the course, we covered topics from probability theory. The difference between statistics and probability theory is the following:
More informationChapter 4. Chapter 4 sections
Chapter 4 sections 4.1 Expectation 4.2 Properties of Expectations 4.3 Variance 4.4 Moments 4.5 The Mean and the Median 4.6 Covariance and Correlation 4.7 Conditional Expectation SKIP: 4.8 Utility Expectation
More informationProbability Review. Gonzalo Mateos
Probability Review Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ September 11, 2018 Introduction
More informationEE514A Information Theory I Fall 2013
EE514A Information Theory I Fall 2013 K. Mohan, Prof. J. Bilmes University of Washington, Seattle Department of Electrical Engineering Fall Quarter, 2013 http://j.ee.washington.edu/~bilmes/classes/ee514a_fall_2013/
More informationM378K In-Class Assignment #1
The following problems are a review of M6K. M7K In-Class Assignment # Problem.. Complete the definition of mutual exclusivity of events below: Events A, B Ω are said to be mutually exclusive if A B =.
More informationEEL 5544 Noise in Linear Systems Lecture 30. X (s) = E [ e sx] f X (x)e sx dx. Moments can be found from the Laplace transform as
L30-1 EEL 5544 Noise in Linear Systems Lecture 30 OTHER TRANSFORMS For a continuous, nonnegative RV X, the Laplace transform of X is X (s) = E [ e sx] = 0 f X (x)e sx dx. For a nonnegative RV, the Laplace
More informationRandom variables. DS GA 1002 Probability and Statistics for Data Science.
Random variables DS GA 1002 Probability and Statistics for Data Science http://www.cims.nyu.edu/~cfgranda/pages/dsga1002_fall17 Carlos Fernandez-Granda Motivation Random variables model numerical quantities
More informationIntroduction to Information Entropy Adapted from Papoulis (1991)
Introduction to Information Entropy Adapted from Papoulis (1991) Federico Lombardo Papoulis, A., Probability, Random Variables and Stochastic Processes, 3rd edition, McGraw ill, 1991. 1 1. INTRODUCTION
More informationReview of probability
Review of probability Computer Sciences 760 Spring 2014 http://pages.cs.wisc.edu/~dpage/cs760/ Goals for the lecture you should understand the following concepts definition of probability random variables
More informationStatistical techniques for data analysis in Cosmology
Statistical techniques for data analysis in Cosmology arxiv:0712.3028; arxiv:0911.3105 Numerical recipes (the bible ) Licia Verde ICREA & ICC UB-IEEC http://icc.ub.edu/~liciaverde outline Lecture 1: Introduction
More informationSTAT 418: Probability and Stochastic Processes
STAT 418: Probability and Stochastic Processes Spring 2016; Homework Assignments Latest updated on April 29, 2016 HW1 (Due on Jan. 21) Chapter 1 Problems 1, 8, 9, 10, 11, 18, 19, 26, 28, 30 Theoretical
More informationfunctions Poisson distribution Normal distribution Arbitrary functions
Physics 433: Computational Physics Lecture 6 Random number distributions Generation of random numbers of various distribuition functions Normal distribution Poisson distribution Arbitrary functions Random
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More information3. Review of Probability and Statistics
3. Review of Probability and Statistics ECE 830, Spring 2014 Probabilistic models will be used throughout the course to represent noise, errors, and uncertainty in signal processing problems. This lecture
More informationRelationship between probability set function and random variable - 2 -
2.0 Random Variables A rat is selected at random from a cage and its sex is determined. The set of possible outcomes is female and male. Thus outcome space is S = {female, male} = {F, M}. If we let X be
More informationMAS223 Statistical Inference and Modelling Exercises
MAS223 Statistical Inference and Modelling Exercises The exercises are grouped into sections, corresponding to chapters of the lecture notes Within each section exercises are divided into warm-up questions,
More informationMath 151. Rumbos Fall Solutions to Review Problems for Exam 2. Pr(X = 1) = ) = Pr(X = 2) = Pr(X = 3) = p X. (k) =
Math 5. Rumbos Fall 07 Solutions to Review Problems for Exam. A bowl contains 5 chips of the same size and shape. Two chips are red and the other three are blue. Draw three chips from the bowl at random,
More informationProbability Theory Review
Cogsci 118A: Natural Computation I Lecture 2 (01/07/10) Lecturer: Angela Yu Probability Theory Review Scribe: Joseph Schilz Lecture Summary 1. Set theory: terms and operators In this section, we provide
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationWeek 2. Review of Probability, Random Variables and Univariate Distributions
Week 2 Review of Probability, Random Variables and Univariate Distributions Probability Probability Probability Motivation What use is Probability Theory? Probability models Basis for statistical inference
More informationCommunication Theory II
Communication Theory II Lecture 5: Review on Probability Theory Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt Febraury 22 th, 2015 1 Lecture Outlines o Review on probability theory
More informationECE Lecture #10 Overview
ECE 450 - Lecture #0 Overview Introduction to Random Vectors CDF, PDF Mean Vector, Covariance Matrix Jointly Gaussian RV s: vector form of pdf Introduction to Random (or Stochastic) Processes Definitions
More informationProbability Review. Yutian Li. January 18, Stanford University. Yutian Li (Stanford University) Probability Review January 18, / 27
Probability Review Yutian Li Stanford University January 18, 2018 Yutian Li (Stanford University) Probability Review January 18, 2018 1 / 27 Outline 1 Elements of probability 2 Random variables 3 Multiple
More informationLecture 3: Central Limit Theorem
Lecture 3: Central Limit Theorem Scribe: Jacy Bird (Division of Engineering and Applied Sciences, Harvard) February 8, 003 The goal of today s lecture is to investigate the asymptotic behavior of P N (εx)
More informationMTH739U/P: Topics in Scientific Computing Autumn 2016 Week 6
MTH739U/P: Topics in Scientific Computing Autumn 16 Week 6 4.5 Generic algorithms for non-uniform variates We have seen that sampling from a uniform distribution in [, 1] is a relatively straightforward
More informationECE 650 Lecture 4. Intro to Estimation Theory Random Vectors. ECE 650 D. Van Alphen 1
EE 650 Lecture 4 Intro to Estimation Theory Random Vectors EE 650 D. Van Alphen 1 Lecture Overview: Random Variables & Estimation Theory Functions of RV s (5.9) Introduction to Estimation Theory MMSE Estimation
More information