where r n = dn+1 x(t)

Transcription

1 Random Variables Overview Probability Random variables Transforms of pdfs Moments and cumulants Useful distributions Random vectors Linear transformations of random vectors The multivariate normal distribution Sums of independent random variables Central limit theorem Taylor Series Expansion Taylor series expansion about t = a, x(t) =x(a)+ dx(t) dt where (t a)+ d2 x(t) (t a) 2 d 2 t 2! r n = dn+1 x(t) dt (t a) n+1 t=τ [t,a] + + dn x(t) (t a) n + r d n n t n! J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Introduction Random signals evolve in time in an unpredictable manner We must assume something doesn t change in order to use them Usually this is their average properties Probability Space Let Ω denote all possible outcomes, ζ, of an experiment Event: A subset of outcomes The event is said to occur if the outcome of the experiment is one of the members of the subset It s an or (union) Not an and (intersection) A collection of subsets with certain properties is called a field and will be denoted as F The probability of each event in the field is denoted Pr { } for k =1, 2,... The collection (Ω, F, Pr { }) is called a probability space J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

2 Ω ζ Random Variables x(ζ) Range Real Line Random variable: a function that assigns a real number, x(ζ), to each outcome ζ in the sample space of a random experiment Conditions {ζ : ζ Ω and <x(ζ) x 0 } F for all x 0 R 1 Pr {x(ζ) = } = 0 Pr {x(ζ) = } = 0 Random Variable Properties The sample space Ω is the domain of the random variable The set of all values that x( ) can have is the range of the random variable For now the range is the real line, R 1 Later we will generalize to allow it to be a vector or a sequence This is a many to one mapping. That is, a set of points, ζ 1,ζ 2,... may take on the same value of the random variable Will sometimes abbreviate RV The RV may be real- or complex-valued Random variables are actually a deterministic function of any event in the field: {ζ 1,ζ 2,...,ζ k } J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Random Variable Properties Continued Notation: x(ζ) =x x(ζ) denotes a random variable (function of the experimental outcome) x denotes its value A random variable may be Discrete-Valued, Continuous-Valued, ormixed Similar to spectral densities If x(t) is nearly periodic, it has discrete spectra Most stationary random signals x(t) have continuous spectra A combination of the two types has mixed spectra Cumulative Distribution Function (cdf) Recall that probability is defined on the field of possible events Some of these events (subsets of outcomes) can be defined as {ζ : x(ζ) x} Cumulative Distribution Function (cdf): The probability of these events, Pr {x(ζ) x} Note the cdf is a function of x R 1 : F x (x) Pr {x(ζ) x} The cdf is continuous from the right J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

3 Properties of the cdf 0 F x (x) 1 lim x + F x (x) =1 lim x F x (x) =0 F x (x) is a nondecreasing function of x Thus, if a<b,thenf x (a) F x (b) F x (x) is continuous from the right That is, for h>0, F x (b) = lim h 0 F x (b + h) =F x (b + ) Pr {a <x(ζ) b} = F x (b) F x (a) Pr {x(ζ) =b} = F x (b) F x (b ) Probability Density Function (pdf) Probability Density Function (pdf): When it exists, is defined as f x (x) df x(x) dx Thus, we also have F x (x) = x f x (u)du For a small interval x, can be thought of as f x (x) x F x (x + x) F x (x) =Pr{x <x(ζ) x + x} J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Properties of the pdf f x (x) 0 Pr {a x(ζ) b} = b a f x(u)du F x (x) = x f x(u)du + f x(u)du =1 A valid pdf can be formed from any nonnegative, piecewise continuous function g(x) that has a finite integral The pdf must be defined for all real values of x If x does not take on some values, this implies f x (x) =0for those values Point Statistics, Averages, and Moments In general, we will often not have a complete description of an RV (the pdf or cdf) Estimating a pdf or cdf is difficult in general, especially if x(ζ) is a sequence or vector However, we can often estimate some properties of the distribution without estimating the distribution itself These are called point statistics We will only discuss a subset of point statistics: statistical averages or moments These are useful because In many cases, we can estimate them accurately from data They give us useful information about the distribution We don t have to know the distribution to estimate them J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

4 Expected Values Defined Expected value: defined for a random variable x(ζ) as E[x(ζ)] μ x = + xf x (x)dx If x(ζ) is discrete-valued, the pdf consists only of impulses and can be alternatively written in terms of the pmf ( ) + E[x(ζ)] = x p k δ(x x k ) dx = k = k k + p k xδ(x x k )dx p k x k Expectation Properties E[x(ζ)] μ x = Also called the mean of x(ζ) + xf x (x)dx The mean can be regarded as the center of gravity of f x (x) If f x (x) is symmetric about a, f(x a) =f(a x), thenμ x = a If f x (x) is even, μ x =0 This is a linear operation E[αx(ζ)+β] =αμ x + β If y(ζ) =g(x(ζ)), a function of the random variable x(ζ), y(ζ) is also a random variable such that E[y(ζ)] E[g(x(ζ))] = g(x)f x (x)dx J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Moments Defined mth-order Moment of x(ζ) is defined as r (m) x E[x m (ζ)] = x m f x (x)dx mth-order Central Moment of x(ζ) is defined as γ (m) x E[(x(ζ) μ x ) m ]= (x μ x ) m f x (x)dx Mean-Squared Value: The second-order moment, r x (2). Note that E [ x 2 (ζ) ] E 2 [x(ζ)] Moments and Definitions r x (m) E[x m (ζ)] = x m f x (x)dx γ x (m) E[(x(ζ) μ x ) m ]= (x μ x ) m f x (x)dx Variance of x(ζ) is defined as var[x(ζ)] σx 2 γ x (2) =E [ (x(ζ) μ x ) 2] Standard deviation of x(ζ) is defined as σ x = var[x(ζ)] Obvious moments r x (0) =1 r x (1) = μ x Trivial central moments γ x (0) =1 γ x (1) =0 γ x (2) = σx 2 J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

5 Relationship of Moments Moments and central moments are related: m ( ) m γ x (m) = ( 1) k μ k k xr x (m k) k=0 It also possible to calculate the first m moments from the first m central moments. Characteristic Functions Characteristic Function of x(ζ) is defined as [ Φ(ξ) E e jξx(ζ)] = f x (x)e jξx dx Similar to the continuous-time Fourier transform However, the exponent is positive (why?) Will not use F x (ξ) to avoid confusion with cdf (same as text) Text claims the independent variable ξ should not be thought of as frequency (why?) This will be useful for handling sums of random variables because the distribution of the sum is the convolution of the pdf s J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Moment Generating Functions Moment Generating Function of x(ζ) is defined as [ Φ(s) E e sx(ζ)] = f x (x)e sx dx Similar to the continuous-time Laplace transform However, the exponent is positive (why?) Using a Taylor series expansion of e sx at x close to zero we have Φ(s) = E = E [e sx(ζ)] [1+sx(ζ)+ [sx(ζ)]2 2! + + [sx(ζ)]m m! = 1+sμ x + s2 2! r(2) x + + sm m! r(m) x +... ] +... Moment Generating Functions Φ(s) = 1+sμ x + s2 2! r(2) x + + sm m! r(m) x +... If all the moments of x(ζ) are known and exist, we can create Φ(s) and solve for f x (x) by inverse Laplace transform Thus the set of all moments (if they exist) completely define the pdf! If Φ(s) is known and analytic (not in the book), we can use it to solve for the moments r (m) x = dm [ Φ(s)] ds m =( j) m dm [ Φ(ξ)] s=0 dξ m ξ=0 J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

6 Cumulants Cumulant Generating Function of x(ζ) is defined as Ψ x (s) ln Φ x (s) =lne [e sx(ζ)] Second Characteristic Function of x(ζ) is defined as Common Random Variables Uniform distribution Normal distribution Cauchy distribution Ψ x (ξ) ln Φ x (ξ) =lne [e jξx(ζ)] mth Cumulant of x(ζ) is defined as κ x (m) dm [ Ψ(s)] ds m =( j) m dm [ Ψ(ξ)] s=0 dξ m ξ=0 Useful for high-order moment analysis (advanced SSP topic) and working with products of characteristic functions J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Uniform Distribution f x (x) = { 1 b a a x b 0 otherwise Useful for situations in which outcomes are equally likely Often denoted as x(ζ) U[a, b] Mean and variance μ x = a + b 2 σ 2 x = (b a)2 12 F(x) Gaussian Distribution Function x f x (x) = Normal Distribution f(x) πσ 2 x exp Also called the Gaussian distribution Often denoted as x(ζ) N(μ x,σx) 2 Arises naturally in many applications Central limit theorem (more later) Gaussian Density Function x ( (x μ x) 2 ) 2σ 2 x J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

7 Normal Distribution Properties f x (x) = 1 e (x μ x) 2σx 2 2πσ 2 x Φ x (ξ) = exp(jμ x ξ 1 2 σ2 xξ 2 ) Defined completely by μ x and σx 2 All higher-order moments can be determined in terms of the first two moments Higher-order moments provide no additional information Due to the central limit theorem, would like to know how the distribution of random variables differs from a normal distribution Cumulants generally provide this information For normally distributed random variables, κ x (m) =0for m>2 2 Cauchy Random Variable f x (x) = β π 1 (x μ) 2 + β 2 F x (x) = π arctan x μ β Φ x (ξ) = exp(jμξ β ξ ) A heavy-tailed distribution (relative to Gaussian) Two parameters, μ and β Mean: μ x = μ Variance does not exist because E[x 2 ] does not exist Moment generating function does not exist for some values of s The sum of M independent Cauchy random variables is also Cauchy! Example of an infinite-variance random variable J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Random Vectors Random Vectors Ω x(ζ) R M Ω x(ζ) R M ζ ζ Much of what we have discussed generalizes to vector random-variables in an obvious manner However, the lower order moments have special properties and are important to signal processing Each outcome of the assumed underlying random experiment produces an entire random vector Each element of the vector is not generated independently from a separate experiment This is an important concept Random M vector: a real-valued vector containing M random variables x =[x 1 (ζ),x 2 (ζ),...,x m (ζ)] T Transpose is denoted by T Maps an abstract probability space to a vector-valued real space R M Thus the range is an M-dimensional space J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

8 Distribution and Density Functions Distribution and Density Functions f x (x) = F x (x 1,...,x M ) Pr {x 1 (ζ) x 1,...x M (ζ) x M } lim x 1 0. x M 0 Pr {x 1 <x 1 (ζ) x 1 + x 1,...,x M <x M (ζ) x M + x M } x 1... x M The commas denote an and condition: it is the probability that all M random variables x i (ζ) are less than the stated values A RV is completely characterized by its joint cdf or pdf Often written as F x (x) =Pr{x(ζ) x} f x (x) F x (x) = = F x (x) x 1 x m x1 x xm f x (u)du f x (u)du 1 du M J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Independence Independent Two random variables x 1 (ζ) and x 2 (ζ) are independent if the events {x 1 (ζ) x 1 } and {x 2 (ζ) x 2 } are jointly independent Pr {x 1 (ζ) x 1,x 2 (ζ) x 2 } =Pr{x 1 (ζ) x 1 } Pr {x 2 (ζ) x 2 } Equivalent conditions are F x1,x 2 (x 1,x 2 ) = F x1 (x 1 )F x2 (x 2 ) f x1,x 2 (x 1,x 2 ) = f x1 (x 1 )f x2 (x 2 ) Vector Statistics In general it is very difficult to estimate vector pdf s and/or cdf s Called the curse of dimensionality Even if they are known, they are difficult to work with in general However, there is a rich statistical theory developed for second order moments (mean and variance) We will on this aspect of SSP for this course Higher-order moments is an advanced topic covered later in the text Worth self-study J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

9 Vector Mean Mean Vector: defined for a random variable x(ζ) as E[x 1 (ζ)] μ 1 μ x =E[x(ζ)] =. =. E[x M (ζ)] μ M The integral of the vector expectation is taken over the entire C M space, if the random vector is complex-valued The mean vector is simply the vector of means Autocorrelation Matrix Autocorrelation matrix: defined for a random vector x(ζ) as R x E [ x(ζ)x H (ζ) ] r r 1M = r M1... r mm where r ii E [ x i (ζ) 2] r ij E[x i (ζ)x j (ζ) ]=rji H denotes conjugate transpose operation This is a completely different definition than the autocorrelation defined for deterministic signals r ii are second-order moments of x i (ζ), denoted earlier as r x (2) i r ij measure correlation between two random variables This will be precisely defined later The autocorrelation matrix R x is Hermitian: R x = R H x J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Autocovariance Matrix Autocovariance matrix: defined for a random variable x(ζ) as Γ x E [ γ γ 1M (x(ζ) μ x )(x(ζ) μ x ) H] =..... γ M1... γ MM where γ ii E [ x i (ζ) μ i 2] = σ 2 x i γ ij E[(x i (ζ) μ i )(x j (ζ) μ j ) ]=E [ x i (ζ)x j (ζ) ] μ i μ j = γ ji Sometimes γ ij is denoted as σ ij γ ii is sometimes called the self variance of x i (ζ) γ ij is called the covariance of x i (ζ) and x j (ζ) Correlation Coefficients Correlation coefficient: is denoted as ρ ij and is defined for random variables x i (ζ) and x j (ζ) as γ ij = ρ ij σ i σ j ρ ij γ ij σ i σ j ρ ii =1and 1 ρ ij 1 Extreme values of ρ ij indicate a linear relationship between x j (ζ) and x i (ζ): x j (ζ) =αx i (ζ)+β Does not tell you what α or β are ρ ij =1implies α>0, ρ ij = 1 implies α<0 If ρ ij =0, x i (ζ) and x j (ζ) are said to be uncorrelated The covariance matrix Γ x is Hermitian: Γ x = Γ H x J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

10 If μ x = 0, Γ x = R x Autocorrelation and Autocovariance Γ x = R x μ x μ H x If ρ ij =0or γ ij =0, x i (ζ) and x j (ζ) are uncorrelated If r ij =0, x i (ζ) and x j (ζ) are orthogonal The degree of correlation is a weaker measure of interaction than independence If x i (ζ) and x j (ζ) are independent, γ ij = ρ ij =0 Converse is not true (sufficient, but not a necessary condition) If x i (ζ) and x j (ζ) have a normal distribution, then γ ij = ρ ij =0implies x i (ζ) and x j (ζ) are independent If one or both random variables have zero mean and are uncorrelated, they are orthogonal Cross-Correlation and Cross-Covariance Cross-correlation matrix: defined for a random vectors x(ζ) C M and y(ζ) C L as R x y E [ x(ζ)y H (ζ) ] E[x 1 (ζ)y1(ζ)]... E[x 1 (ζ)yl (ζ)] =..... E[x M (ζ)y1(ζ)]... E[x M (ζ)yl (ζ)] Cross-covariance matrix: defined for a random vectors x(ζ) C M and y(ζ) C L as Γ xy E [ (x(ζ) μ x )(y(ζ) μ y ) H] = R xy μ x μ H y Uncorrelated if Γ xy = 0 or, equivalently, R xy = μ x μ H y Orthogonal if R xy = 0 J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Linear Transformations y(ζ) = Ax(ζ) μ y = E[y(ζ)]=E[Ax(ζ)] = A E[x(ζ)] = Aμ x R y = E [ y(ζ)y(ζ) H] =E [ Ax(ζ) x(ζ) H A H] = AR x A H Γ y = AΓ x A H R xy = R x A H R yx = AR x Γ xy = Γ x A H Γ yx = AΓ x f x (x) = Normal Random Vectors 1 (2π) M/2 Γ x 1 2 Φ x (ξ) = exp(jξ T μ x 1 2 ξt Γ x ξ) exp ( 1 2 (x μ x) T Γ 1 x (x μ x ) ) Often denoted as x(ζ) N(μ x, Γ x ) The exponent is a positive definite quadratic function of x The pdf is completely specified by μ x and Γ x Both are relatively easy to estimate in practice All higher-order moments can be calculated from these If all pairs uncorrelated, are also independent A linear transformation, y = Ax + b is also normally distributed, y(ζ) N(μ x + b, AΓ x A H ) J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

11 Let Sums of Independent Random Variables y(ζ) = c k x k (ζ) μ y = c k μ xk σy 2 = c k 2 σx 2 k Therefore y(ζ) = x 1 (ζ)+x 2 (ζ) Φ y (ξ) = E [e jξy(ζ)] = E [e ] jξ[x 1(ζ)+x 2 (ζ)] [ ] = E e jξx 1(ζ) E [e ] jξx 2(ζ) = Φ x1 (ξ)φ x2 (ξ) f y (y) = f x1 (y) f x2 (y) Sums of Independent Random Variables Continued This generalizes in the obvious manner y(ζ) = x k (ζ) f y (y) =f x1 (y) f x2 (y) f xm (y) M Φ y (ξ) = Φ xk (ξ) Ψ y (ξ) = κ (m) y = Ψ xk (ξ) κ (m) x k J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver Stable Distributions y(ζ) = x k (ζ) Stable Distributions: distributions that are preserved (self-reproduce) under convolution Examples: Gaussian distribution, Cauchy distribution The only stable distribution with finite variance is the Gaussian distribution The other distributions have infinite variance and possibly infinite mean Examples: some diffusion processes, electrical noise, some random walks, coupled oscillators with friction Useful to model signals with large variability Very good discussion in text (not critical to this class) Central Limit Theorem y(ζ) = x k (ζ) If x i (ζ) are independent identically distributed (IID) random variables and the distribution f x (x) is stable, clearly y(ζ) converges to the same distribution as M Central Limit Theorem: If The mean and variance of each random variable exist (finite) The random variables are mutually independent The random variables are identically distributed then the distribution of y(ζ) approaches a normal distribution as M J. McNames Portland State University ECE 538/638 Random Variables Ver J. McNames Portland State University ECE 538/638 Random Variables Ver

12 Central Limit Theorem Comments y(ζ) = x k (ζ) The convergence often occurs even if the distributions are not identical The convergence is more accurate (rapid) near the mean (center) of the distribution The approximation may be poor in the tails The convergence is in the cdf, not the pdf. Consider discrete RV s Does not apply when the variance of the RV s is infinite J. McNames Portland State University ECE 538/638 Random Variables Ver