Fundamentals of Digital Commun. Ch. 4: Random Variables and Random Processes Klaus Witrisal witrisal@tugraz.at Signal Processing and Speech Communication Laboratory www.spsc.tugraz.at Graz University of Technology November 6, 5 Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. /36 Outline 4- Probability (brief review) 4- Random Variables 4-3 Random Processes 4-4 Noise Example for a Random Process Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. /36
4- Probability Fundamental concept: random experiment with an uncertain outcome Definitions: ω... outcomes: ω Ω Ω... sample space: set of all outcomes Events: subsets of sample space Ω for which propability can be defined Probability: functional mapping of events onto reals: P (E) for all events E B B... collection of (all possible) events (Ω, B,P)... probability space Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 3/36 Probability (cont d) Probability is: relative frequency of an event ocurring in n trials A, B... events P (A) = lim n n A n n A... number of occurancies of A n... number of trials definitions sure event E: P (E) = null event A =: P (A) = Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 4/36
Probability (cont d) Events can be represented as sets (Venn diagram) product of events A and B (intersection) C = A B = AB mutually exclusive events A B =,i.e.p (AB) = sum of events A or B (union) C = A B = A + B P (A + B) =P (A)+P(B) P (AB) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 5/36 Probability (cont d) conditional probability P (A B) = lim n n AB n B = P (AB) P (B) probability that an event A occurs, given that an event B has also occured product theorem P (AB) =P (A B)P (B) =P (B A)P (A) leads to the Bayes-Theorem P (A B) = P (B A)P (A) P (B) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 6/36
Probability (cont d) total probability: event B occurs jointly with one of the mutually exclusive events A,A,..., A K, where K P (A k )= k= Find the probability P (B), givenp (B A k ) and P (A k ): P (B) = P (B A )P (A )+P(B A )P (A )+... +P (B A K )P (A K ) = K K P (B A k )P (A k )= P (BA k ) k= k= Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 7/36 Probability (cont d) Def.: a-priori probabilities: P (A k ) Def.: a-posteriori probability: P (A k B) Assume: event B occurs jointly with one of the mutually exclusive events A,A,..., A K : Find the probability for A k, when B has already occured. (B could be an observation of A k ) can be solved using the Bayes-theorem and the total probability P (A k B) = P (B A k)p (A k ) P (B) = P (B A k)p (A k ) K i= P (B A i)p (A i ) given P (A k ) and P (B A k ) (likelihoods) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 8/36
4- Random Variables sample space Ω (with outcomes ω i,ν j (and events)) is mapped onto the set of reals R the random variable is a function of the outcomes notation: X = X(ω i ),Y = Y (ν j ), short X, Y upper-case symbols are used to distinguish from defined variables discrete random variables (RV) e.g. dice X {,,..., 6} (number of eyes); P (X = x m )= 6 continuous RV e.g. voltage of a battery X = U Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 9/36 Characterization of RVs cumulative distribution function CDF (dt: Verteilungsfunktion) F X (a) =P (X a) = lim n dimensionless probability density function PDF (dt: Wahrscheinlichkeitsdichtefunktion) unit is /[x] f X (x) = df X(x) dx = lim n,δx n (X a) n [ Δx ( nδx n ) ] Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. /36
Characterization of RVs (cont d) properties of the CDF F X (x), nondecreasing F X () = F X ( ) = following the def.: discrete points at X = x are included probability that X lies in some interval: P (x <X x ) = P (X x ) P (X x ) = F X (x ) F X (x ) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. /36 Characterization of RVs (cont d) properties of the PDF PDF is nonnegative f X (x) area under the PDF f X(x)dx = computation of the CDF from the PDF F X (x) = x+ɛ f X (λ)dλ = P (X x) probability that X lies in some interval: P (x <X x )= x +ɛ x +ɛ f X (λ)dλ Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. /36
Ensemble Averages and Moments mean value, expected value, first-order moment, ensemble average E{X} = m X = (center of gravity of the PDF) k-th order moment E{X k } = xf X (x)dx x k f X (x)dx centralized k-th-order moment (moment about mean) E{(X m X ) k } = (x m X ) k f X (x)dx Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 3/36 Ensemble Averages and Moments (cont d) variance (centralized second moment) σ X =var{x} =E{(X m X) } = (x m X ) f X (x)dx standard deviation σ X = var{x} generalized: mean of a function g(x) of an RV E{g(X)} = g(x)f X (x)dx Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 4/36
Ensemble Averages and Moments (cont d) definition through events (for lim n ) E{X} = lim n E{g(X)} = lim n n n n X(A i ) i= n g[x(a i )] i= canbeusedtoestimate moments will be an approximation for a limited sample n< Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 5/36 Extension to two Random Variables bivariate statistics: joint characterization of two RVs for > variables: multivariate statistics CDF F XY (x, y) =P (X x, Y y) PDF f XY (x, y) = F XY (x, y) x y Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 6/36
4 Scatter plot of samples of, Y 3 Scatter plot of samples of, Y 3 samples of Y samples of Y - - - - -3-3 -3 - - 3 4 samples of -4-3 - - 3 4 samples of joint PDF of and Y joint PDF of and Y.5.5.5.5-3 - - 3 - - Y Y - Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 7/36 correlation Extension to two Random Variables (cont d) R XY =E{XY } = uncorrelated RVs xyf XY (x, y)dx dy R XY =E{XY } =E{X}E{Y } = m X m Y orthogonal RVs R XY =E{XY } = covariance K XY =E{(X m X )(Y m Y )} = R XY m X m Y Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 8/36
4-3 Random processes X(A i,t)... function of two variables: A i... event, realization t... time (usually) for an event A i : sample function x i (t) ensemble of sample functions: {x i (t)}, i random process: X(t) (also in lower case) illustration of an ensemble of sample functions: (blackboard) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 9/36 Characterization of random processes ensemble averages mean (at time t) m X (t) =E{X(t)} = xf X(t) (x) dx can be time-variant (e.g. sine-wave plus noise) autocorrelation(function ACF) R X (t,t )=E{X(t )X(t )} = x x f X(t )X(t )(x,x ) dx dx f X(t )X(t )(x,x )... joint PDF for X(t ) and X(t ) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. /36
relizations till 5 of an ensemble of trials 4 Mean value; first moment 3 4 5 6 7 8 9 estimate of E x(t) - -4 3 4 5 6 7 8 9 Second moment (= autocorrelation) 3 4 5 6 7 8 9 3 4 5 6 7 8 9 estimate of E x(5) x(t).8.6.4. -. 4 4 44 46 48 5 5 54 56 58 6 sample number, time t 3 4 5 6 7 8 9 3 4 5 6 7 8 9 sample number, time Power Spectrum Magnitude (db) - -4-6 Fundamentals.of Digital.4Commun.Ch..6 4:.8 Random Variables. and Random.4 Processes.6 p..8 /36. Frequency 4 Scatter plot of two samples of (t) 3 Scatter plot of two samples of (t) 3 (t ) for t = sample 3 - (t ) for t = sample - - - -3-3 -3 - - 3 4 (t ) for t = sample -4-3 - - 3 4 (t ) for t = sample joint PDF of (t ) and (t ) joint PDF of (t ) and (t ).5.5.5.5-3 - - 3 - - x x - x x Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. /36
relizations till 5 of an ensemble of trials 5 GRAZ UNIVERSITY Mean value; OFfirstTECHNOLOGY moment 3 4 5 6 7 8 9 estimate of E x(t) -5 3 4 5 6 7 8 9 3 4 5 6 7 8 9 3 4 5 6 7 8 9 estimate of E x(5) x(t).8.6.4. -. Second moment (= autocorrelation) -.4 3 4 5 6 7 8 sample number, time t 3 4 5 6 7 8 9 3 4 5 6 7 8 9 sample number, time Power Spectrum Magnitude (db) 3 - - -3-4 -5..4.6.8...4.6.8. Frequency Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 3/36 Characterization of random processes (cont d) stationarity (time-invariance) first-order stationarity (stationary w.r.t. mean) E{X(t)} = m X =const. second-order stationarity (stationary w.r.t. ACF) R X (t,t )=R X (τ) =E{X(t)X(t + τ)} where τ = t t i.e.: no variability w.r.t. time shift exact definition: Stationarity (of n-th order): PDF (of n-th order) is invariant w.r.t. time shifts Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 4/36
Characterization of random processes (cont d) Important definitions: Strict-Sense-Stationary (SSS) (Stationär im strengen Sinn) PDFs of any order are stationary Wide-Sense-Stationary (WSS) (Stationär im weiteren Sinn) Mean and autocorrelation function (ACF) are stationary (i.e. first and second-order moments) for Gaussian processes: WSS implies SSS (hence the importance of the first two moments) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 5/36 Characterization (cont d); ACF of WSS Random Processes Properties R X (τ) =R X ( τ)... even symmetry R X () R X (τ) R X () = E{X (t)}... second moment; power Power Spectral Density PSD (Leistungsdichtespektrum) X(t)... power signal, if stationary PSD is Fourier transform of the ACF Def. : R X (τ) F S X (jω) PSD is power content in frequency domain Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 6/36
Power Spectral Density (PSD) (cont d) definition using the Fourier transform of X(t): needs time limitation of the sample functions: X T (A i,jω)= T/ T/ x(a i,t)e jωt dt Wiener-Kinchin theorem S X (jω) = lim = T Proof at blackboard! T E A i { X T (A i,jω} } R X (τ)e jωτ dτ = F[R X (τ)] Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 7/36 relizations till 5 of an ensemble of trials FT x (t) 6 4 3 4 5 6 7 8 9 FT x (t)..4.6.8...4.6.8. 6 4..4.6.8...4.6.8. 6 3 4 5 6 7 8 9 FT x 3 (t) 4 3 4 5 6 7 8 9 3 4 5 6 7 8 9 FT x 4 (t) FT x 5 (t)..4.6.8...4.6.8. 6 4..4.6.8...4.6.8. 6 4..4.6.8...4.6.8. 6 E FT (t) 4 3 4 5 6 7 8 9 sample number, time Fundamentals.of Digital.4Commun.Ch..6 4:.8 Random Variables. and Random.4Processes.6 p..8 8/36. normalized frequency
relizations till 5 of an ensemble of trials 4 Mean value; first moment 3 4 5 6 7 8 9 estimate of E x(t) - -4 3 4 5 6 7 8 9 Second moment (= autocorrelation) 3 4 5 6 7 8 9 3 4 5 6 7 8 9 estimate of E x(5) x(t).8.6.4. -. 4 4 44 46 48 5 5 54 56 58 6 sample number, time t 3 4 5 6 7 8 9 3 4 5 6 7 8 9 sample number, time Power Spectrum Magnitude (db) - -4-6 Fundamentals.of Digital.4Commun.Ch..6 4:.8 Random Variables. and Random.4 Processes.6 p..8 9/36. Frequency relizations till 5 of an ensemble of trials 5 GRAZ UNIVERSITY Mean value; OFfirstTECHNOLOGY moment 3 4 5 6 7 8 9 estimate of E x(t) -5 3 4 5 6 7 8 9 3 4 5 6 7 8 9 3 4 5 6 7 8 9 estimate of E x(5) x(t).8.6.4. -. Second moment (= autocorrelation) -.4 3 4 5 6 7 8 sample number, time t 3 4 5 6 7 8 9 3 4 5 6 7 8 9 sample number, time Power Spectrum Magnitude (db) 3 - - -3-4 -5..4.6.8...4.6.8. Frequency Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 3/36
Characterization (cont d); Ergodic Random Proc. Definition: (all) time and ensemble averages are identical necessary condition: stationarity (otherwise, ensemble average would be time-variant) example: ensemble average and DC-value it must hold: x DC m X, where x DC = x i (t) = lim T m X =E{X(t)} = T T/ T/ x i (t) dt xf X(t) (x) dx = lim n n n x i (t) i= Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 3/36 Characterization (cont d); Ergodic Random Proc. equivalence must hold for all averages hard to proof Example: ACF R X (τ) =E{X (t)x(t + τ)} x i (t)x i (t + τ) consequence of ergodicity: ensemble averages (moments) can be estimated from a single sample function! Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 3/36
4-4 Noise Example for a random process Physics: freely moving electrons e PSD, ACF S X (jω)=s X (f) = N N /... double-sided noise PSD thermal noise N = kt F R X (τ) = N δ(τ) PDF: Gaussian PDF = normal distribution Power: P N = N B = σ ; B... equivalent bandwidth f X(t) (x) = ( ) x exp πσ σ Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 33/36 Why is noise Gaussian distributed? Central Limit Theorem (Zentraler Grenzwertsatz): a sum of (many) random processes has Gaussian PDF illustration: PDF of one dice: uniform (discrete) two dice: triangular PDF many dice: PDF approaches Gaussian PDF generally (without proof): PDF of the sum of two (independent) random variables corresponds to the convolution of their PDFs Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 34/36
. Ein Wuerfel Wahrscheinlichk..5..5 5 5. Zwei Wuerfel Wahrscheinlichk..5..5 Wahrscheinlichk..5.5 5 5.. Drei Wuerfel Normalverteilung 5 5 Augenzahl (discrete PDFs) Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 35/36 Simulation: Ein Wuerfel, Wuerfe x4 Simulation: Ein Wuerfel, Wuerfe Ereignisse.5.5 5 5 Simulation: Zwei Wuerfel, Wuerfe 5 5 x4 Simulation: Zwei Wuerfel, Wuerfe Ereignisse.5.5 5 5 Simulation: Drei Wuerfel, Wuerfe 5 5 x4 Simulation: Drei Wuerfel, Wuerfe Ereignisse.5.5 5 5 Augenzahl 5 5 Augenzahl Fundamentals of Digital Commun.Ch. 4: Random Variables and Random Processes p. 36/36