A Probability Review
|
|
- Griffin Flynn
- 5 years ago
- Views:
Transcription
1 A Probability Review Outline: A probability review Shorthand notation: RV stands for random variable EE 527, Detection and Estimation Theory, # 0b 1
2 A Probability Review Reading: Go over handouts 2 5 in EE 420x notes Basic probability rules: (1) Pr{Ω} = 1, Pr{ } = 0, 0 Pr{A} 1; Pr{ i=1 A i} = i=1 Pr{A i} if A i A }{{} j = for all i j; A i and A j disjoint (2) Pr{A B} = Pr{A} + Pr{B} Pr{A B}, Pr{A c } = 1 Pr{A}; (3) If A B, then Pr{A B} = Pr{A} Pr{B}; (4) Pr{A B} = Pr{A B} Pr{B} (conditional probability) or Pr{A B} = Pr{A B} Pr{B} (chain rule); EE 527, Detection and Estimation Theory, # 0b 2
3 (5) Pr{A} = Pr{A B 1 } Pr{B 1 } + + Pr{A B n } Pr{B n } if B 1, B 2,, B n form a partition of the full space Ω; (6) Bayes rule: Pr{A B} = Pr{B A} Pr{A} Pr{B} EE 527, Detection and Estimation Theory, # 0b 3
4 Reminder: Independence, Correlation and Covariance For simplicity, we state all the definitions for pdfs; the corresponding definitions for pmfs are analogous Two random variables X and Y are independent if f X,Y (x, y) = f X (x) f Y (y) Correlation between real-valued random variables X and Y : E X,Y {X Y } = + + x y f X,Y (x, y) dx dy Covariance between real-valued random variables X and Y : cov X,Y (X, Y ) = E X,Y {(X µ X ) (Y µ Y )} = + + (x µ X ) (y µ Y ) f X,Y (x, y) dx dy EE 527, Detection and Estimation Theory, # 0b 4
5 where µ X = E X (X) = µ Y = E Y (Y ) = + + x f X (x) dx y f Y (y) dy Uncorrelated random variables: Random variables X and Y are uncorrelated if c X,Y = cov X,Y (X, Y ) = 0 (1) If X and Y are real-valued RVs, then (1) can be written as E X,Y {X Y } = E X {X} E Y {Y } Mean Vector and Covariance Matrix: Consider a random vector X = X 1 X 2 X N EE 527, Detection and Estimation Theory, # 0b 5
6 The mean of this random vector is defined as µ 1 E µ X = µ 2 = E [X X 1 1] X{X} = E [X X 2 2] µ N E XN [X N ] Denote the covariance between X i and X k, cov Xi,X k (X i, X k ), by c i,k ; hence, the variance of X i is c i,i = cov Xi,X k (X i, X i ) = var Xi (X i ) = σx 2 }{{} i The covariance matrix of X is defined as more notation C X = c 1,1 c 1,2 c 1,N c 2,1 c 2,2 c 2,N c N,1 c N,2 c N,N The above definitions apply to both real and complex vectors X Covariance matrix of a real-valued random vector X: C X = E X {(X E X [X]) (X E X [X]) T } = E X [X X T ] E X [X] (E X [X]) T For real-valued X, c i,k = c k,i and, therefore, C X is a symmetric matrix EE 527, Detection and Estimation Theory, # 0b 6
7 Linear Transform of Random Vectors Linear Transform For real-valued Y, X, A, Y = g(x) = A X Mean Vector: µ Y = E X {A X} = A µ X (2) Covariance Matrix: C Y = E Y {Y Y T } µ Y µ T Y = E X {A X X T A T } A µ X µ T X A T ( ) = A E X {X X T } µ X µ }{{ T X} C X A T = A C X A T (3) EE 527, Detection and Estimation Theory, # 0b 7
8 Reminder: Iterated Expectations In general, we can find E X,Y [g(x, Y )] using iterated expectations: E X,Y [g(x, Y )] = E Y {E X Y [g(x, Y ) Y ]} (4) where E X Y denotes the expectation with respect to f X Y (x y) and E Y denotes the expectation with respect to f Y (y) Proof E Y {E X Y [g(x, Y ) Y ]} = = = = E X Y [g(x, Y ) y] f Y (y) dy ( + ) g(x, y)f X Y (x y) dx f Y (y) dy + + = E X,Y [g(x, Y )] g(x, y)f X Y (x y)f Y (y) dx dy g(x, y) f X,Y (x, y) dx dy EE 527, Detection and Estimation Theory, # 0b 8
9 Reminder: Law of Conditional Variances Define the conditional variance of X given Y = y to be the variance of X with respect to f X Y (x y), ie ] var X Y (X Y = y) = E X Y [(X E X Y [X y]) 2 y = E X Y [X 2 y] (E X Y [X y]) 2 The random variable var X Y (X Y ) is a function of Y only, taking values var(x Y = y) Its expected value with respect to Y is E Y {var X Y (X Y )} = E Y { E X Y [X 2 Y ] (E X Y [X Y ]) 2} iterated exp = E X,Y [X 2 ] E Y {(E X Y [X Y ]) 2 } = E X [X 2 ] E Y {(E X Y [X Y ]) 2 } Since E X Y [X Y ] is a random variable (and a function of Y only), it has variance: var Y {E X Y [X Y ]} = E Y {(E X Y [X Y ]) 2 } (E Y {E X Y [X Y ]}) 2 iterated exp = E Y {(E X Y [X Y ]) 2 } (E X,Y [X]) 2 = E Y {E X Y [X Y ] 2 } (E X [X]) 2 EE 527, Detection and Estimation Theory, # 0b 9
10 Adding the above two expressions yields the law of conditional variances: E Y {var X Y (X Y )} + var Y {E X Y [X Y ]} = var X (X) (5) Note: (4) and (5) hold for both real- and complex-valued random variables EE 527, Detection and Estimation Theory, # 0b 10
11 Useful Expectation and Covariance Identities for Real-valued Random Variables and Vectors E X,Y [a X + b Y + c] = a E X [X] + b E Y [Y ] + c var X,Y (a X + b Y + c) = a 2 var X (X) + b 2 var Y (Y ) +2 a b cov X,Y (X, Y ) where a, b, and c are constants and X and Y are random variables A vector/matrix version of the above identities: E X,Y [A X + B Y + c] = A E X [X] + B E Y [Y ] + c cov X,Y (A X + B Y + c) = A cov X (X) A T + B cov Y (Y ) B T +A cov X,Y (X, Y ) B T + B cov X,Y (Y, X) A T where T denotes a transpose and cov X,Y (X, Y ) = E X,Y {(X E X [X]) (Y E Y [Y ]) T } Useful properties of crosscovariance matrices: cov X,Y,Z (X, Y + Z) = cov X,Y (X, Y ) + cov X,Z (X, Z) EE 527, Detection and Estimation Theory, # 0b 11
12 cov Y,X (Y, X) = [cov X,Y (X, Y )] T cov X (X) = cov X (X, X) var X (X) = cov X (X, X) cov X,Y (A X + b, P Y + q) = A cov X,Y (X, Y ) P T (To refresh memory about covariance and its properties, see p 12 of handout 5 in EE 420x notes For random vectors, see handout 7 in EE 420x notes, particularly pp 1 15) Useful theorems: (1) (handout 5 in EE 420x notes) E X (X) = E Y [E X Y (X Y )] shown on p 8 E X Y [g(x) h(y ) y] = h(y) E X Y [g(x) y] E X,Y [g(x) h(y )] = E Y {h(y ) E X Y [g(x) Y ]} The vector version of (1) is the same just put bold letters EE 527, Detection and Estimation Theory, # 0b 12
13 (2) var X (X) = E Y [var X Y (X Y )] + var Y (E X Y [X Y ]); and the vector/matrix version is cov X (X) }{{} variance/covariance matrix of X = E Y [cov X Y (X Y )] +cov Y (E X Y [X Y ]) shown on p (3) Generalized law of conditional variances: cov X,Y (X, Y ) = E Z [cov X,Y Z (X, Y Z)] +cov Z (E X Z [X Z], E Y Z [Y Z]) (4) Transformation: Y = g(x) one-to-one Y 1 = g 1 (X 1,, X n ) Y n = g n (X 1,, X n ) then f Y (y) = f X (h 1 (y 1 ),, h n (y n )) J EE 527, Detection and Estimation Theory, # 0b 13
14 where h( ) is the unique inverse of g( ) and J = x y T = x 1 x 1 y n y 1 x n y 1 x n y n Print and read the handout Probability distributions from the Course readings section on WebCT Bring it with you to the midterm exams EE 527, Detection and Estimation Theory, # 0b 14
15 Jointly Gaussian Real-valued RVs Scalar Gaussian random variables: f X (x) = 1 2 π σ 2 X exp [ (x µ X) 2 2 σ 2 X ] Definition Two real-valued RVs X and Y are jointly Gaussian if their joint pdf is of the form f X,Y (x, y) = 1 exp { 2πσ X σ Y 1 ρ 2 X,Y 1 2 (1 ρ 2 X,Y ) (x µ X ) (y µ Y ) ] 2 ρ X,Y σ X σ Y [ (x µx ) 2 } σ 2 X + (y µ Y ) 2 σ 2 Y (6) This pdf is parameterized by µ X, µ Y, σ 2 X, σ 2 Y, and ρ X,Y Here, σ X = σ 2 X and σ Y = σ 2 Y Note: We will soon define a more general multivariate Gaussian pdf EE 527, Detection and Estimation Theory, # 0b 15
16 If X and Y are jointly Gaussian, contours of equal joint pdf are ellipses defined by the quadratic equation (x µ X ) 2 σ 2 X + (y µ Y ) 2 (x µ X ) (y µ Y ) σ 2 2 ρ X,Y = const 0 Y σ X σ Y Examples: In the following examples, we plot contours of the joint pdf f X,Y (x, y) for zero-mean jointly Gaussian RVs for various values of σ X, σ Y, and ρ X,Y EE 527, Detection and Estimation Theory, # 0b 16
17 EE 527, Detection and Estimation Theory, # 0b 17
18 If X and Y Gaussian, eg are jointly Gaussian, the conditional pdfs are ( X {Y = y} N ρ X,Y σ X y E Y [Y ] σ Y +E X [X], (1 ρ 2 X,Y ) σ 2 X If X and Y are jointly Gaussian and uncorrelated, ie ρ X,Y = 0, they are also independent (7) ) EE 527, Detection and Estimation Theory, # 0b 18
19 Gaussian Random Vectors Real-valued Gaussian random vectors: f X (x) = 1 (2 π) N/2 C X 1/2 exp [ 1 2 (x µ X) T C 1 X (x µ X ) ] Complex-valued Gaussian random vectors: f Z (z) = 1 π N C Z exp [ (z µ Z ) H C 1 Z (z µ Z ) ] Notation for real- and complex-valued Gaussian random vectors: X N r (µ X, C X ) [or simply N (µ X, C X )] X N c (µ X, C X ) complex real An affine transform of a Gaussian vector is also a Gaussian random vector, ie if then Y = A X + b Y N r (A µ X + b, A C X A T ) Y N c (A µ X + b, A C X A H ) real complex EE 527, Detection and Estimation Theory, # 0b 19
20 The Gaussian random vector W N r (0, σ 2 I n ) (where I n denotes the identity matrix of size n) is called white; pdf contours of a white Gaussian random vector are spheres centered at the origin Suppose that W [n], n = 0, 1,, N 1 are independent, identically distributed (iid) zeromean univariate Gaussian N (0, σ 2 ) Then, for W = [W [0], W [1],, W [N 1]] T, f W (w) = N (w 0, σ 2 I) Suppose now that, for these W [n], Y [n] = θ + W [n] where θ is a constant What is the joint pdf of Y [0], Y [1],, and Y [N 1]? This pdf is the pdf of the vector Y = [Y [0], Y [1],, Y [N 1]] T : Y = 1 θ + W where 1 is an N 1 vector of ones Now, Since θ is a constant, f Y (y) = N (y 1 θ, σ 2 I) f Y (y) = f Y θ (y θ) EE 527, Detection and Estimation Theory, # 0b 20
21 Gaussian Random Vectors A real-valued random vector X = [X 1, X 2,, X n ] T with mean µ and covariance matrix Σ with determinant Σ > 0 (ie Σ is positive definite) is a Gaussian random vector (or X 1, X 2,, X n Gaussian RVs) if and only if its joint pdf is are jointly f X (x) = 1 2πΣ 1/2 exp[ 1 2 (x µ)t Σ 1 (x µ)] (8) Verify that, for n = 2, this joint pdf reduces to the twodimensional pdf in (6) Notation: We use X N (µ, Σ) to denote a Gaussian random vector Since Σ is positive definite, Σ 1 is also positive definite and, for x µ, (x µ) T Σ 1 (x µ) > 0 which means that the contours of the multivariate Gaussian pdf in (8) are ellipsoids The Gaussian random vector X N (0, σ 2 I n ) (where I n denotes the identity matrix of size n) is called white contours of the pdf of a white Gaussian random vector are spheres centered at the origin EE 527, Detection and Estimation Theory, # 0b 21
22 Properties of Real-valued Gaussian Random Vectors Property 1: For a Gaussian random vector, uncorrelation implies independence This is easy to verify by setting Σ i,j = 0 for all i j in the joint pdf, then Σ becomes diagonal and so does Σ 1 ; then, the joint pdf reduces to the product of marginal pdfs f Xi (x i ) = N (µ i, Σ i,i ) = N (µ i, σ 2 ) Clearly, this property Xi holds for blocks of RVs (subvectors) as well Property 2: A linear transform of a Gaussian random vector X N (µ X, Σ X ) yields a Gaussian random vector: Y = A X N (A µ X, A Σ X A T ) It is easy to show that E Y [Y ] = A µ X and cov Y (Y ) = Σ Y = A Σ X A T So E Y [Y ] = E X [A X] = A E X [X] = A µ X and Σ Y = E Y [(Y E Y [Y ]) (Y E Y [Y ]) T ] = E X [(A X A µ X ) (A X A µ X ) T ] = A E X [(X µ X ) (X µ X ) T ] A T = A Σ X A T EE 527, Detection and Estimation Theory, # 0b 22
23 Of course, if we use the definition of a Gaussian random vector in (8), we cannot yet claim that Y is a Gaussian random vector (For a different definition of a Gaussian random vector, we would be done right here) Proof We need to verify that the joint pdf of Y indeed has the right form Here, we decide to take the equivalent (easier) task and verify that the characteristic function of Y has the right form Definition Suppose X f X (X) Then the characteristic function of X is given by Φ X (ω) = E X [exp(j ω T X)] where ω is an n-dimensional real-valued vector and j = 1 Thus Φ X (ω) = + + f X (x) exp(j ω T x) dx proportional to the inverse multi-dimensional Fourier transform of f X (x); therefore, we can find f X (x) by taking the Fourier transform (with the appropriate proportionality factor): f X (x) = 1 (2π) n + + Φ X (ω) exp( j ω T x) dx EE 527, Detection and Estimation Theory, # 0b 23
24 Example: The characteristic function for X N (µ, σ 2 ) is given by Φ X (ω) = exp( 1 2 ω2 σ 2 + j µ ω) (9) and for a Gaussian random vector Z N (µ, Σ), Φ Z (ω) = exp( 1 2 ωt Σω + j ω T µ) (10) Now, go back to our proof: Y = AX is the characteristic function of Φ Y (ω) = E Y [exp(j ω T Y )] = E X [exp(j ω T A X)] = exp( 1 2 ωt A Σ X A T ω + j ω T A µ X ) Thus Y = A X N (A µ X, AΣ X A T ) EE 527, Detection and Estimation Theory, # 0b 24
25 Property 3: Marginals of a Gaussian random vector are Gaussian, ie if X is a Gaussian random vector, then, for any {i 1, i 2,, i k } {1, 2,, n}, Y = X i1 X i2 X ik is a Gaussian random vector To show this, [ we use ] Property 2 X1 Here is an example with n = 3 and Y = We set X 3 Y = [ ] X 1 X 2 X 3 thus Here and Y N ( [ ] [ ] ) µ 1 Σ1,1 Σ, 1,3 µ 3 Σ 3,1 Σ 3,3 E X X 1 X 2 = µ 1 µ 2 X 3 µ 3 cov X X 1 X 2 = Σ 1,1 Σ 1,2 Σ 1,3 Σ 2,1 Σ 2,2 Σ 2,3 X 3 Σ 3,1 Σ 3,2 Σ 3,3 EE 527, Detection and Estimation Theory, # 0b 25
26 and note that [ µ1 µ 3 ] = [ ] µ 1 µ 2 µ 3 and [ ] Σ1,1 Σ 1,3 Σ 3,1 Σ 3,3 = [ ] Σ 1,1 Σ 1,2 Σ 1,3 Σ 2,1 Σ 2,2 Σ 2,3 Σ 3,1 Σ 3,2 Σ 3, The converse of Property 3 does not hold in general; here is a counterexample: Example: Suppose X 1 N (0, 1) and X 2 = { 1, wp 1 2 1, wp 1 2 are independent RVs and consider X 3 = X 1 X 2 Observe that X 3 N (0, 1) and f X 1,X3 (x 1, x 3 ) is not a jointly Gaussian pdf EE 527, Detection and Estimation Theory, # 0b 26
27 Property 4: Conditionals of Gaussian random vectors are Gaussian, ie if then X = [ X1 X 2 ] {X 2 X 1 = x 1 } N and {X 1 X 2 = x 2 } N ( [ ] [ ] ) µ N 1 Σ1,1 Σ, 1,2 µ 2 Σ 2,1 Σ 2,2 ( ) Σ 2,1 Σ1,1 1 (x 1 µ 1 )+µ 2, Σ 2,2 Σ 2,1 Σ1,1 1 Σ 1,2 ( ) Σ 1,2 Σ2,2 1 (x 2 µ 2 )+µ 1, Σ 1,1 Σ 1,2 Σ2,2 1 Σ 2,1 EE 527, Detection and Estimation Theory, # 0b 27
28 Example: Compare this result to the case of n = 2 in (7): {X 2 X 1 = x 1 } N ( Σ2,1 (x 1 µ 1 ) + µ 2, Σ 2,2 Σ 2 ) 1,2 Σ 1,1 Σ 1,1 In particular, having X = X 2 and Y = X 1, y = x 1, this result becomes: {X Y = y} N ( σx,y σ 2 Y (y µ Y ) + µ X, σ 2 X σ2 X,Y σ 2 Y ) where σ X,Y = cov X,Y (X, Y ), σ 2 X = cov X,X (X, X) = var X (X), and σ 2 Y = cov Y,Y (Y, Y ) = var Y (Y ) Now, it is clear that ρ X,Y = σ X,Y σ X σ Y where σ X = σ 2 X > 0 and σ Y = σ 2 Y > 0 Example: Suppose EE 527, Detection and Estimation Theory, # 0b 28
29 Property 5: If X N (µ, Σ) then (x µ) T Σ 1 (x µ) χ 2 d (Chi-square in your distr table) EE 527, Detection and Estimation Theory, # 0b 29
30 Additive Gaussian Noise Channel Consider a communication channel with input X N (µ X, τ 2 X) and noise W N (0, σ 2 ) where X and W are independent and the measurement Y is Y = X + W Since X and W are independent, we have f X,W (x, w) = f X (x) f W (x) and [ X W ] ( [ µ N X 0 What is f Y X (y x)? Since ], [ τ 2 X 0 0 σ 2 ] ) {Y X = x} = x + W N (x, σ 2 ) we have f Y X (y x) = N (y x, σ 2 ) EE 527, Detection and Estimation Theory, # 0b 30
31 [ X Y How about f Y (y)? Construct the joint pdf f X,Y (x, y) of X and Y : since [ ] [ ] [ ] X 1 0 X = Y 1 1 W then ] N ( [ ] [ µx 0 ], [ ] [ ] [ τ 2 X σ ] ) yielding [ X Y ] N ( [ ] [ ] µ X τ 2 X τ, 2 ) X µ X τ 2 X τ 2 X + σ 2 Therefore, f Y (y) = N ( y µx, τ 2 X + σ 2) EE 527, Detection and Estimation Theory, # 0b 31
32 Complex Gaussian Distribution Consider joint pdf of real and imaginary part of an n 1 complex vector Z Z = U + j V Assume X = [ U Y ] The 2 n-variate Gaussian pdf of the (real!) vector X is f X (x) = 1 (2 π) 2 n Σ X exp [ 1 2 (z µ X) T Σ 1 X (z µ X ) ] where µ X = [ µu ], Σ µ X = V [ ] ΣUU Σ UV Σ V U Σ V V Therefore, Pr{x A} = x A f X (x) dx EE 527, Detection and Estimation Theory, # 0b 32
33 Complex Gaussian Distribution (cont) Assume that Σ X has a special structure: Σ UU = Σ V V and Σ UV = Σ V U [Note that Σ UV = Σ T V U has to hold as well] Then, we can define a complex Gaussian pdf of Z as f Z (z) = 1 π n Σ X exp [ (z µ Z ) H Σ 1 Z (z µ Z ) ] where µ Z = µ U + j µ V Σ Z = E Z {(Z µ Z ) (Z µ Z ) H } = 2 (Σ UU + j Σ V U ) 0 = E Z {(Z µ Z ) (Z µ Z ) T } EE 527, Detection and Estimation Theory, # 0b 33
EEL 5544 Noise in Linear Systems Lecture 30. X (s) = E [ e sx] f X (x)e sx dx. Moments can be found from the Laplace transform as
L30-1 EEL 5544 Noise in Linear Systems Lecture 30 OTHER TRANSFORMS For a continuous, nonnegative RV X, the Laplace transform of X is X (s) = E [ e sx] = 0 f X (x)e sx dx. For a nonnegative RV, the Laplace
More informationMultiple Random Variables
Multiple Random Variables This Version: July 30, 2015 Multiple Random Variables 2 Now we consider models with more than one r.v. These are called multivariate models For instance: height and weight An
More information3. Probability and Statistics
FE661 - Statistical Methods for Financial Engineering 3. Probability and Statistics Jitkomut Songsiri definitions, probability measures conditional expectations correlation and covariance some important
More informationProbability Background
Probability Background Namrata Vaswani, Iowa State University August 24, 2015 Probability recap 1: EE 322 notes Quick test of concepts: Given random variables X 1, X 2,... X n. Compute the PDF of the second
More informationx. Figure 1: Examples of univariate Gaussian pdfs N (x; µ, σ 2 ).
.8.6 µ =, σ = 1 µ = 1, σ = 1 / µ =, σ =.. 3 1 1 3 x Figure 1: Examples of univariate Gaussian pdfs N (x; µ, σ ). The Gaussian distribution Probably the most-important distribution in all of statistics
More informationIntroduction to Probability and Stocastic Processes - Part I
Introduction to Probability and Stocastic Processes - Part I Lecture 2 Henrik Vie Christensen vie@control.auc.dk Department of Control Engineering Institute of Electronic Systems Aalborg University Denmark
More informationMultivariate Random Variable
Multivariate Random Variable Author: Author: Andrés Hincapié and Linyi Cao This Version: August 7, 2016 Multivariate Random Variable 3 Now we consider models with more than one r.v. These are called multivariate
More informationMultivariate Statistics
Multivariate Statistics Chapter 2: Multivariate distributions and inference Pedro Galeano Departamento de Estadística Universidad Carlos III de Madrid pedro.galeano@uc3m.es Course 2016/2017 Master in Mathematical
More informationconditional cdf, conditional pdf, total probability theorem?
6 Multiple Random Variables 6.0 INTRODUCTION scalar vs. random variable cdf, pdf transformation of a random variable conditional cdf, conditional pdf, total probability theorem expectation of a random
More information5. Random Vectors. probabilities. characteristic function. cross correlation, cross covariance. Gaussian random vectors. functions of random vectors
EE401 (Semester 1) 5. Random Vectors Jitkomut Songsiri probabilities characteristic function cross correlation, cross covariance Gaussian random vectors functions of random vectors 5-1 Random vectors we
More informationBASICS OF PROBABILITY
October 10, 2018 BASICS OF PROBABILITY Randomness, sample space and probability Probability is concerned with random experiments. That is, an experiment, the outcome of which cannot be predicted with certainty,
More information5 Operations on Multiple Random Variables
EE360 Random Signal analysis Chapter 5: Operations on Multiple Random Variables 5 Operations on Multiple Random Variables Expected value of a function of r.v. s Two r.v. s: ḡ = E[g(X, Y )] = g(x, y)f X,Y
More informationReview: mostly probability and some statistics
Review: mostly probability and some statistics C2 1 Content robability (should know already) Axioms and properties Conditional probability and independence Law of Total probability and Bayes theorem Random
More informationThe Multivariate Gaussian Distribution [DRAFT]
The Multivariate Gaussian Distribution DRAFT David S. Rosenberg Abstract This is a collection of a few key and standard results about multivariate Gaussian distributions. I have not included many proofs,
More informationLet X and Y denote two random variables. The joint distribution of these random
EE385 Class Notes 9/7/0 John Stensby Chapter 3: Multiple Random Variables Let X and Y denote two random variables. The joint distribution of these random variables is defined as F XY(x,y) = [X x,y y] P.
More informationP (x). all other X j =x j. If X is a continuous random vector (see p.172), then the marginal distributions of X i are: f(x)dx 1 dx n
JOINT DENSITIES - RANDOM VECTORS - REVIEW Joint densities describe probability distributions of a random vector X: an n-dimensional vector of random variables, ie, X = (X 1,, X n ), where all X is are
More informationStochastic Processes. M. Sami Fadali Professor of Electrical Engineering University of Nevada, Reno
Stochastic Processes M. Sami Fadali Professor of Electrical Engineering University of Nevada, Reno 1 Outline Stochastic (random) processes. Autocorrelation. Crosscorrelation. Spectral density function.
More informationStat 5101 Notes: Algorithms
Stat 5101 Notes: Algorithms Charles J. Geyer January 22, 2016 Contents 1 Calculating an Expectation or a Probability 3 1.1 From a PMF........................... 3 1.2 From a PDF...........................
More informationLecture 11. Multivariate Normal theory
10. Lecture 11. Multivariate Normal theory Lecture 11. Multivariate Normal theory 1 (1 1) 11. Multivariate Normal theory 11.1. Properties of means and covariances of vectors Properties of means and covariances
More informationEE4601 Communication Systems
EE4601 Communication Systems Week 2 Review of Probability, Important Distributions 0 c 2011, Georgia Institute of Technology (lect2 1) Conditional Probability Consider a sample space that consists of two
More informationUniversity of Cambridge Engineering Part IIB Module 3F3: Signal and Pattern Processing Handout 2:. The Multivariate Gaussian & Decision Boundaries
University of Cambridge Engineering Part IIB Module 3F3: Signal and Pattern Processing Handout :. The Multivariate Gaussian & Decision Boundaries..15.1.5 1 8 6 6 8 1 Mark Gales mjfg@eng.cam.ac.uk Lent
More informationPerhaps the simplest way of modeling two (discrete) random variables is by means of a joint PMF, defined as follows.
Chapter 5 Two Random Variables In a practical engineering problem, there is almost always causal relationship between different events. Some relationships are determined by physical laws, e.g., voltage
More informationLecture Note 1: Probability Theory and Statistics
Univ. of Michigan - NAME 568/EECS 568/ROB 530 Winter 2018 Lecture Note 1: Probability Theory and Statistics Lecturer: Maani Ghaffari Jadidi Date: April 6, 2018 For this and all future notes, if you would
More informationStat 5101 Notes: Algorithms (thru 2nd midterm)
Stat 5101 Notes: Algorithms (thru 2nd midterm) Charles J. Geyer October 18, 2012 Contents 1 Calculating an Expectation or a Probability 2 1.1 From a PMF........................... 2 1.2 From a PDF...........................
More informationContinuous Random Variables
1 / 24 Continuous Random Variables Saravanan Vijayakumaran sarva@ee.iitb.ac.in Department of Electrical Engineering Indian Institute of Technology Bombay February 27, 2013 2 / 24 Continuous Random Variables
More informationChapter 5. The multivariate normal distribution. Probability Theory. Linear transformations. The mean vector and the covariance matrix
Probability Theory Linear transformations A transformation is said to be linear if every single function in the transformation is a linear combination. Chapter 5 The multivariate normal distribution When
More informationL2: Review of probability and statistics
Probability L2: Review of probability and statistics Definition of probability Axioms and properties Conditional probability Bayes theorem Random variables Definition of a random variable Cumulative distribution
More informationACM 116: Lectures 3 4
1 ACM 116: Lectures 3 4 Joint distributions The multivariate normal distribution Conditional distributions Independent random variables Conditional distributions and Monte Carlo: Rejection sampling Variance
More informationMA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems
MA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems Review of Basic Probability The fundamentals, random variables, probability distributions Probability mass/density functions
More informationRandom Variables and Their Distributions
Chapter 3 Random Variables and Their Distributions A random variable (r.v.) is a function that assigns one and only one numerical value to each simple event in an experiment. We will denote r.vs by capital
More informationSection 8.1. Vector Notation
Section 8.1 Vector Notation Definition 8.1 Random Vector A random vector is a column vector X = [ X 1 ]. X n Each Xi is a random variable. Definition 8.2 Vector Sample Value A sample value of a random
More informationElliptically Contoured Distributions
Elliptically Contoured Distributions Recall: if X N p µ, Σ), then { 1 f X x) = exp 1 } det πσ x µ) Σ 1 x µ) So f X x) depends on x only through x µ) Σ 1 x µ), and is therefore constant on the ellipsoidal
More information01 Probability Theory and Statistics Review
NAVARCH/EECS 568, ROB 530 - Winter 2018 01 Probability Theory and Statistics Review Maani Ghaffari January 08, 2018 Last Time: Bayes Filters Given: Stream of observations z 1:t and action data u 1:t Sensor/measurement
More informationMultivariate Gaussians. Sargur Srihari
Multivariate Gaussians Sargur srihari@cedar.buffalo.edu 1 Topics 1. Multivariate Gaussian: Basic Parameterization 2. Covariance and Information Form 3. Operations on Gaussians 4. Independencies in Gaussians
More informationChapter 5 continued. Chapter 5 sections
Chapter 5 sections Discrete univariate distributions: 5.2 Bernoulli and Binomial distributions Just skim 5.3 Hypergeometric distributions 5.4 Poisson distributions Just skim 5.5 Negative Binomial distributions
More informationPhysics 403. Segev BenZvi. Parameter Estimation, Correlations, and Error Bars. Department of Physics and Astronomy University of Rochester
Physics 403 Parameter Estimation, Correlations, and Error Bars Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Review of Last Class Best Estimates and Reliability
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More informationMultiple Random Variables
Multiple Random Variables Joint Probability Density Let X and Y be two random variables. Their joint distribution function is F ( XY x, y) P X x Y y. F XY ( ) 1, < x
More informationREVIEW OF MAIN CONCEPTS AND FORMULAS A B = Ā B. Pr(A B C) = Pr(A) Pr(A B C) =Pr(A) Pr(B A) Pr(C A B)
REVIEW OF MAIN CONCEPTS AND FORMULAS Boolean algebra of events (subsets of a sample space) DeMorgan s formula: A B = Ā B A B = Ā B The notion of conditional probability, and of mutual independence of two
More information18 Bivariate normal distribution I
8 Bivariate normal distribution I 8 Example Imagine firing arrows at a target Hopefully they will fall close to the target centre As we fire more arrows we find a high density near the centre and fewer
More informationLecture 2. Spring Quarter Statistical Optics. Lecture 2. Characteristic Functions. Transformation of RVs. Sums of RVs
s of Spring Quarter 2018 ECE244a - Spring 2018 1 Function s of The characteristic function is the Fourier transform of the pdf (note Goodman and Papen have different notation) C x(ω) = e iωx = = f x(x)e
More informationRecitation 2: Probability
Recitation 2: Probability Colin White, Kenny Marino January 23, 2018 Outline Facts about sets Definitions and facts about probability Random Variables and Joint Distributions Characteristics of distributions
More informationSolutions to Homework Set #5 (Prepared by Lele Wang) MSE = E [ (sgn(x) g(y)) 2],, where f X (x) = 1 2 2π e. e (x y)2 2 dx 2π
Solutions to Homework Set #5 (Prepared by Lele Wang). Neural net. Let Y X + Z, where the signal X U[,] and noise Z N(,) are independent. (a) Find the function g(y) that minimizes MSE E [ (sgn(x) g(y))
More informationBasics on Probability. Jingrui He 09/11/2007
Basics on Probability Jingrui He 09/11/2007 Coin Flips You flip a coin Head with probability 0.5 You flip 100 coins How many heads would you expect Coin Flips cont. You flip a coin Head with probability
More informationMAS223 Statistical Inference and Modelling Exercises
MAS223 Statistical Inference and Modelling Exercises The exercises are grouped into sections, corresponding to chapters of the lecture notes Within each section exercises are divided into warm-up questions,
More informationQuick Tour of Basic Probability Theory and Linear Algebra
Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra CS224w: Social and Information Network Analysis Fall 2011 Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra Outline Definitions
More informationJoint Distributions. (a) Scalar multiplication: k = c d. (b) Product of two matrices: c d. (c) The transpose of a matrix:
Joint Distributions Joint Distributions A bivariate normal distribution generalizes the concept of normal distribution to bivariate random variables It requires a matrix formulation of quadratic forms,
More informationECE 650 Lecture 4. Intro to Estimation Theory Random Vectors. ECE 650 D. Van Alphen 1
EE 650 Lecture 4 Intro to Estimation Theory Random Vectors EE 650 D. Van Alphen 1 Lecture Overview: Random Variables & Estimation Theory Functions of RV s (5.9) Introduction to Estimation Theory MMSE Estimation
More informationME 597: AUTONOMOUS MOBILE ROBOTICS SECTION 2 PROBABILITY. Prof. Steven Waslander
ME 597: AUTONOMOUS MOBILE ROBOTICS SECTION 2 Prof. Steven Waslander p(a): Probability that A is true 0 pa ( ) 1 p( True) 1, p( False) 0 p( A B) p( A) p( B) p( A B) A A B B 2 Discrete Random Variable X
More informationECE 636: Systems identification
ECE 636: Systems identification Lectures 3 4 Random variables/signals (continued) Random/stochastic vectors Random signals and linear systems Random signals in the frequency domain υ ε x S z + y Experimental
More informationRecall that if X 1,...,X n are random variables with finite expectations, then. The X i can be continuous or discrete or of any other type.
Expectations of Sums of Random Variables STAT/MTHE 353: 4 - More on Expectations and Variances T. Linder Queen s University Winter 017 Recall that if X 1,...,X n are random variables with finite expectations,
More informationThe Delta Method and Applications
Chapter 5 The Delta Method and Applications 5.1 Local linear approximations Suppose that a particular random sequence converges in distribution to a particular constant. The idea of using a first-order
More informationLecture 2: Repetition of probability theory and statistics
Algorithms for Uncertainty Quantification SS8, IN2345 Tobias Neckel Scientific Computing in Computer Science TUM Lecture 2: Repetition of probability theory and statistics Concept of Building Block: Prerequisites:
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More informationIntroduction to Computational Finance and Financial Econometrics Probability Review - Part 2
You can t see this text! Introduction to Computational Finance and Financial Econometrics Probability Review - Part 2 Eric Zivot Spring 2015 Eric Zivot (Copyright 2015) Probability Review - Part 2 1 /
More informationThe Multivariate Normal Distribution. In this case according to our theorem
The Multivariate Normal Distribution Defn: Z R 1 N(0, 1) iff f Z (z) = 1 2π e z2 /2. Defn: Z R p MV N p (0, I) if and only if Z = (Z 1,..., Z p ) T with the Z i independent and each Z i N(0, 1). In this
More informationChapter 5,6 Multiple RandomVariables
Chapter 5,6 Multiple RandomVariables ENCS66 - Probabilityand Stochastic Processes Concordia University Vector RandomVariables A vector r.v. is a function where is the sample space of a random experiment.
More informationIntroduction to Probability Theory
Introduction to Probability Theory Ping Yu Department of Economics University of Hong Kong Ping Yu (HKU) Probability 1 / 39 Foundations 1 Foundations 2 Random Variables 3 Expectation 4 Multivariate Random
More informationMultivariate Distributions
IEOR E4602: Quantitative Risk Management Spring 2016 c 2016 by Martin Haugh Multivariate Distributions We will study multivariate distributions in these notes, focusing 1 in particular on multivariate
More informationUCSD ECE153 Handout #34 Prof. Young-Han Kim Tuesday, May 27, Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE53 Handout #34 Prof Young-Han Kim Tuesday, May 7, 04 Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei) Linear estimator Consider a channel with the observation Y XZ, where the
More informationIntroduction to Normal Distribution
Introduction to Normal Distribution Nathaniel E. Helwig Assistant Professor of Psychology and Statistics University of Minnesota (Twin Cities) Updated 17-Jan-2017 Nathaniel E. Helwig (U of Minnesota) Introduction
More informationStatistics, Data Analysis, and Simulation SS 2015
Statistics, Data Analysis, and Simulation SS 2015 08.128.730 Statistik, Datenanalyse und Simulation Dr. Michael O. Distler Mainz, 27. April 2015 Dr. Michael O. Distler
More informationLecture 2: Review of Probability
Lecture 2: Review of Probability Zheng Tian Contents 1 Random Variables and Probability Distributions 2 1.1 Defining probabilities and random variables..................... 2 1.2 Probability distributions................................
More information[POLS 8500] Review of Linear Algebra, Probability and Information Theory
[POLS 8500] Review of Linear Algebra, Probability and Information Theory Professor Jason Anastasopoulos ljanastas@uga.edu January 12, 2017 For today... Basic linear algebra. Basic probability. Programming
More informationFormulas for probability theory and linear models SF2941
Formulas for probability theory and linear models SF2941 These pages + Appendix 2 of Gut) are permitted as assistance at the exam. 11 maj 2008 Selected formulae of probability Bivariate probability Transforms
More information2 Functions of random variables
2 Functions of random variables A basic statistical model for sample data is a collection of random variables X 1,..., X n. The data are summarised in terms of certain sample statistics, calculated as
More informationNotes on Random Vectors and Multivariate Normal
MATH 590 Spring 06 Notes on Random Vectors and Multivariate Normal Properties of Random Vectors If X,, X n are random variables, then X = X,, X n ) is a random vector, with the cumulative distribution
More informationIntroduction to Computational Finance and Financial Econometrics Matrix Algebra Review
You can t see this text! Introduction to Computational Finance and Financial Econometrics Matrix Algebra Review Eric Zivot Spring 2015 Eric Zivot (Copyright 2015) Matrix Algebra Review 1 / 54 Outline 1
More informationExpectation. DS GA 1002 Statistical and Mathematical Models. Carlos Fernandez-Granda
Expectation DS GA 1002 Statistical and Mathematical Models http://www.cims.nyu.edu/~cfgranda/pages/dsga1002_fall16 Carlos Fernandez-Granda Aim Describe random variables with a few numbers: mean, variance,
More informationcomponent risk analysis
273: Urban Systems Modeling Lec. 3 component risk analysis instructor: Matteo Pozzi 273: Urban Systems Modeling Lec. 3 component reliability outline risk analysis for components uncertain demand and uncertain
More informationStat 206: Sampling theory, sample moments, mahalanobis
Stat 206: Sampling theory, sample moments, mahalanobis topology James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Notation My notation is different from the book s. This is partly because
More informationECE 541 Stochastic Signals and Systems Problem Set 9 Solutions
ECE 541 Stochastic Signals and Systems Problem Set 9 Solutions Problem Solutions : Yates and Goodman, 9.5.3 9.1.4 9.2.2 9.2.6 9.3.2 9.4.2 9.4.6 9.4.7 and Problem 9.1.4 Solution The joint PDF of X and Y
More informationwhere r n = dn+1 x(t)
Random Variables Overview Probability Random variables Transforms of pdfs Moments and cumulants Useful distributions Random vectors Linear transformations of random vectors The multivariate normal distribution
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationProbability and Distributions
Probability and Distributions What is a statistical model? A statistical model is a set of assumptions by which the hypothetical population distribution of data is inferred. It is typically postulated
More informationLecture 3: Pattern Classification
EE E6820: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 1 2 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mixtures
More informationWe introduce methods that are useful in:
Instructor: Shengyu Zhang Content Derived Distributions Covariance and Correlation Conditional Expectation and Variance Revisited Transforms Sum of a Random Number of Independent Random Variables more
More informationThe Gaussian distribution
The Gaussian distribution Probability density function: A continuous probability density function, px), satisfies the following properties:. The probability that x is between two points a and b b P a
More informationExpectation. DS GA 1002 Probability and Statistics for Data Science. Carlos Fernandez-Granda
Expectation DS GA 1002 Probability and Statistics for Data Science http://www.cims.nyu.edu/~cfgranda/pages/dsga1002_fall17 Carlos Fernandez-Granda Aim Describe random variables with a few numbers: mean,
More informationReview (Probability & Linear Algebra)
Review (Probability & Linear Algebra) CE-725 : Statistical Pattern Recognition Sharif University of Technology Spring 2013 M. Soleymani Outline Axioms of probability theory Conditional probability, Joint
More informationDependence. MFM Practitioner Module: Risk & Asset Allocation. John Dodson. September 11, Dependence. John Dodson. Outline.
MFM Practitioner Module: Risk & Asset Allocation September 11, 2013 Before we define dependence, it is useful to define Random variables X and Y are independent iff For all x, y. In particular, F (X,Y
More informationBayes Decision Theory
Bayes Decision Theory Minimum-Error-Rate Classification Classifiers, Discriminant Functions and Decision Surfaces The Normal Density 0 Minimum-Error-Rate Classification Actions are decisions on classes
More information4. Distributions of Functions of Random Variables
4. Distributions of Functions of Random Variables Setup: Consider as given the joint distribution of X 1,..., X n (i.e. consider as given f X1,...,X n and F X1,...,X n ) Consider k functions g 1 : R n
More informationCS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 1
CS434a/541a: Pattern Recognition Prof. Olga Veksler Lecture 1 1 Outline of the lecture Syllabus Introduction to Pattern Recognition Review of Probability/Statistics 2 Syllabus Prerequisite Analysis of
More informationLecture 2: Review of Basic Probability Theory
ECE 830 Fall 2010 Statistical Signal Processing instructor: R. Nowak, scribe: R. Nowak Lecture 2: Review of Basic Probability Theory Probabilistic models will be used throughout the course to represent
More informationReview (probability, linear algebra) CE-717 : Machine Learning Sharif University of Technology
Review (probability, linear algebra) CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Some slides have been adopted from Prof. H.R. Rabiee s and also Prof. R. Gutierrez-Osuna
More informationStatistics for scientists and engineers
Statistics for scientists and engineers February 0, 006 Contents Introduction. Motivation - why study statistics?................................... Examples..................................................3
More informationGaussian vectors and central limit theorem
Gaussian vectors and central limit theorem Samy Tindel Purdue University Probability Theory 2 - MA 539 Samy T. Gaussian vectors & CLT Probability Theory 1 / 86 Outline 1 Real Gaussian random variables
More informationVariations. ECE 6540, Lecture 02 Multivariate Random Variables & Linear Algebra
Variations ECE 6540, Lecture 02 Multivariate Random Variables & Linear Algebra Last Time Probability Density Functions Normal Distribution Expectation / Expectation of a function Independence Uncorrelated
More informationStatistical Pattern Recognition
Statistical Pattern Recognition A Brief Mathematical Review Hamid R. Rabiee Jafar Muhammadi, Ali Jalali, Alireza Ghasemi Spring 2012 http://ce.sharif.edu/courses/90-91/2/ce725-1/ Agenda Probability theory
More informationTAMS39 Lecture 2 Multivariate normal distribution
TAMS39 Lecture 2 Multivariate normal distribution Martin Singull Department of Mathematics Mathematical Statistics Linköping University, Sweden Content Lecture Random vectors Multivariate normal distribution
More information4. CONTINUOUS RANDOM VARIABLES
IA Probability Lent Term 4 CONTINUOUS RANDOM VARIABLES 4 Introduction Up to now we have restricted consideration to sample spaces Ω which are finite, or countable; we will now relax that assumption We
More information1 Review of Probability and Distributions
Random variables. A numerically valued function X of an outcome ω from a sample space Ω X : Ω R : ω X(ω) is called a random variable (r.v.), and usually determined by an experiment. We conventionally denote
More informationUC Berkeley Department of Electrical Engineering and Computer Sciences. EECS 126: Probability and Random Processes
UC Berkeley Department of Electrical Engineering and Computer Sciences EECS 6: Probability and Random Processes Problem Set 3 Spring 9 Self-Graded Scores Due: February 8, 9 Submit your self-graded scores
More informationStatistics STAT:5100 (22S:193), Fall Sample Final Exam B
Statistics STAT:5 (22S:93), Fall 25 Sample Final Exam B Please write your answers in the exam books provided.. Let X, Y, and Y 2 be independent random variables with X N(µ X, σ 2 X ) and Y i N(µ Y, σ 2
More information1 Random Variable: Topics
Note: Handouts DO NOT replace the book. In most cases, they only provide a guideline on topics and an intuitive feel. 1 Random Variable: Topics Chap 2, 2.1-2.4 and Chap 3, 3.1-3.3 What is a random variable?
More information1 Introduction. Systems 2: Simulating Errors. Mobile Robot Systems. System Under. Environment
Systems 2: Simulating Errors Introduction Simulating errors is a great way to test you calibration algorithms, your real-time identification algorithms, and your estimation algorithms. Conceptually, the
More informationMath 3215 Intro. Probability & Statistics Summer 14. Homework 5: Due 7/3/14
Math 325 Intro. Probability & Statistics Summer Homework 5: Due 7/3/. Let X and Y be continuous random variables with joint/marginal p.d.f. s f(x, y) 2, x y, f (x) 2( x), x, f 2 (y) 2y, y. Find the conditional
More informationLecture 25: Review. Statistics 104. April 23, Colin Rundel
Lecture 25: Review Statistics 104 Colin Rundel April 23, 2012 Joint CDF F (x, y) = P [X x, Y y] = P [(X, Y ) lies south-west of the point (x, y)] Y (x,y) X Statistics 104 (Colin Rundel) Lecture 25 April
More informationChapter 4. Multivariate Distributions. Obviously, the marginal distributions may be obtained easily from the joint distribution:
4.1 Bivariate Distributions. Chapter 4. Multivariate Distributions For a pair r.v.s (X,Y ), the Joint CDF is defined as F X,Y (x, y ) = P (X x,y y ). Obviously, the marginal distributions may be obtained
More information