Signals. Time-domain & Frequency-domain representations. Signals & Noises YCJ

Transcription

1 Signals Deterministic signals & Stochastic (random) signals Continuous time signals & Discrete Time signals Analog signals & Digital signals Energy signals & Power signals Periodic signals & Non-periodic signals Time-domain & Frequency-domain representations 1

2 The Delta Function: δ(t) Definition: δ(t) = 0, for t 0 δ(t) dt = 1 Other definitions: δ(t) = lim n f n (t) δ(t) = lim a Sin(at)/πt f n (t) n/2-1/n 1/n t 2

3 The Delta Function: δ(t) δ(at) = 1/ a δ(t) f(t) δ(t) = f(0) δ(t) - δ(t)f(t) dt = f(0) - δ(t-t 0 )f(t) dt = f(t 0 ) 3

4 The Sinc Function: Sin(x)/x Sin(x)/x Sinc(x) = Sin(x)/x Sin(ωt)/ωt Sin(t)/t dt = π π 2π x lim a Sin(at)/πt = δ(t)

5 The Integral of e jωt - e jωt dt = 2π δ(ω) or - e jωt dω = 2π δ(t) - e jωt dt = lim a -a e jωt dt = lim a 2a Sin(ωa)/(ωa) = lim a 2π Sin(ωa)/(ωπ) = 2π δ(ω) a 5

6 Fourier Fundamental Theorem F(ω) = - f(t)e -jωt dt f(t) = 1/(2π) - F(ω)e jωt dω 6

7 The Fourier Integral F(ω) = - f(t)e -jωt dt = R(ω) + j I(ω) = A(ω) e jφ(ω) An example: f(t) = e -αt u(t) α > 0 F(ω) = 1/(α+jω) = α/(α 2 +ω 2 ) + j (-ω)/(α 2 +ω 2 ) = 1/(α 2 +ω 2 ) 1/2 e -jarctan(ω/α) 7

8 Fourier Integral of a Real Function F(ω) = - f(t)e -jωt dt F*(ω) = - f(t)e jωt dt F*(ω) = F(-ω) F(ω) = A(ω) e jφ(ω) F*(ω) = A(ω) e -jφ(ω) F(-ω) = A(-ω) e jφ( ω) A(ω) = A(-ω) φ(ω) = φ( ω) 8

9 Basic Properties of Fourier Integral Linearity: a 1 f 1 (t)+a 2 f 2 (t) a 1 F 1 (ω)+a 2 F 2 (ω) Symmetry: F(-t) 2πf(ω) Time Scaling: f(at) 1/ a F(ω/a) Time Shifting: f(t-t o ) F(ω)e jω t o Frequency Shifting: f(t)e jω ot F(ω ω o ) Modulation: f(t)cos(ω o t) [F(ω ω o )+F(ω+ω o )]/2 9

10 Convolution & Convolution Theorem g(t) = f(t) * h(t) = - f(τ) h(t-τ) dτ G(ω) = F(ω) H(ω) h(t)f(t) (1/2π) H(ω)*F(ω) 10

11 Parseval Theorem & Energy - f(t)g*(t) dt = (1/2π) - F(ω)G*(ω)dω - f(t)f*(t) dt = (1/2π) - F(ω)F*(ω)dω - f(t) 2 dt = (1/2π) - F(ω) 2 dω Energy Spectral Density: F(ω) 2 11

12 Fourier Transforms of Basic Functions p T (t) -T T t (2T)Sin(Tω)/Tω Sin(at)/πt p a (ω) -a a ω p T (t)cos(ω o t) (T){Sin[T(ω ω o )]/[T(ω ω o )] + Sin[T(ω+ω o )]/[T(ω+ω o )]} q T (t) 4Sin 2 (Tω/2)/(Tω 2 ) -T T t 12

13 Fourier Transforms of Basic Functions δ(t) 1; 1 2πδ(ω) δ(t-t o ) e jω t o; e jω ot 2πδ(ω ω o ) Cos(ω o t) π[δ(ω ω o ) + δ(ω+ω o )] Sin(ω o t) -jπ[δ(ω ω o ) - δ(ω+ω o )] Sgn(t) u(t) 2/jω πδ(ω) + 1/(jω) 13

14 Bandwidth and Duration B ω = - H(ω) dω /H(0) D τ = - h(t) dt / h(t) max B ω D τ 2π 14

15 The Summation of e jkωt k= k= Σ e jkωt = (2π/T) Σ δ[ω k(2π/τ)] k=- k= k= k=- Σ e jkω = (2π) Σ δ(ω k2π) k=- k= k=- Σ e jk(ω-ω)μt = (2π/MT) Σ δ[ω-ω k(2π/μτ)] k=- k= k=- 15

16 The Fourier Series For a periodic function f(t) with period T: f(t) = f(t-t) f(t) = Σ k α k e jkω o t where ω o = 2π/T T/2 α k =(1/T) -T/2 f(t) e -jkω o t dt = (1/T) F o (kω o ) Where F o (ω) is the Fourier transform of f o (t) and f o (t) = { f(t), for t (-T/2,T/2), 0, otherwise 16

17 The Fourier Series f(t) -T/2 T/2 t f o (t) F o (ω) -T/2 T/2 t 3ω o 2ω o ω o ω o 2ω o 3ω o ω 17

18 Parseval Theorem & Average Power Average Power = (1/T) -T/2 f(t) 2 dt k= = Σ α k 2 k=- k= -T/2 = Σ (1/T)F o [k(2π/τ)] 2 k=- If G(ω) is the Fourier transform of a pulse of duration T - G(ω) 2 d ω = (2π/Τ) Σ G[k(2π/Τ)] 2 k= k=- 18

19 Autocorrelation Function & Power Spectral Density T/2 R v (τ) = lim T (1/T) -T/2 v(t)v*(t-τ)dt Average Power = P v = R v (0) The Power Spectral Density: S v (ω) S v (ω) = - R v (τ) e -jωτ dτ P v = R v (0) = 1/(2π) - S v (ω) dω 19

20 Power Spectral Density of Periodic Signals T/2 R v (τ) = (1/T) -T/2 v(t)v*(t-τ)dt k= R v (τ) = Σ α k 2 e jkω oτ k=- k= R v (0) = Σ α k 2 k=- S v (ω) =2π Σ α k 2 δ(ω kω o ) k= k=- 20

21 Linear Time-Invariant System x(t) h(t) H(ω) y(t) Linearity: a x 1 (t)+ b x 2 (t) a y 1 (t)+ b y 2 (t) Time- Invariant: x(t-τ) y(t-τ) Convolution: y(t) = - x(t-τ)h(τ)dτ Y(ω) =X(ω)H(ω) 21

22 Frequency Response of LTI System e jωt H(ω) H(ω) e jωt H(ω) = H(ω) e jarg{h(ω)} A 1 e jω 1 t +A 2 e jω 2 t H(ω) H(ω 1 ) A 1 e jω 1 t + H(ω 2 ) A 1 e jω 2 t x(t) = Σ k α k e jkω o t H(ω) y(t) = Σ k α k H(kω ο ) e jkω o t 22

23 Frequency Response of LTI System X(ω) Y(ω) ω=2πf ω=2πf 23

24 Power Spectral Density & LTI System x(t) h(t) H(ω) S v (ω) = S x (ω) H(ω) 2 y(t) For periodic Signals k= S v (ω) = Σ α k 2 H(ω) 2 δ(ω kω o ) k=- S v (ω) = Σ α k 2 H(kω o ) 2 δ(ω kω o ) k= k=- 24

25 Probability Space ξ : Experimental Outcome {ξ 1, ξ 2, ξ 3 }: Event {ξ}: Elementary Event S : Certain Event φ: Impossible Event 25

26 Two-Face Coin Example: two-face coin {H, T} ξ = H, 1 ξ =T 2 {ξ 1 } ={ H }, {ξ 2 } ={ T } S = {H,T} φ = {} 26

27 Six-Face Die Example: six-face die {f 1, f 2, f 3, f 4, f 5, f 6 } ξ =f k k, k = 1, 2, 3, 4, 5, 6 {ξ k } ={ f k }, k = 1, 2, 3, 4, 5, 6 even Event: { f 2, f 4,, f 6 } smallest two Event: { f 1, f 2 } S = {f 1, f 2, f 3, f 4, f 5, f 6 } φ = {} 27

28 Three Axioms Q: How to specify (characterize) a probability space? A: To find All the probabilities to All events A and B are events, Let AB = A B and A+B = A B Axiom 1: P(A) 0 Axiom 2: P(S) = 1 Axiom 3: If AB = φ Then P(A+B) = P(A) + P(B) 28

29 Basic Results P(φ) = 0 (use Axiom 3, B=φ) P(A) 1 (use Axioms 1, 2 and 3, B=A ) P(A+B) = P(A) + P(B) P(AB) A+B = A + A c B P(A+B) = P(A)+P(A c B) B = (A+A c )B P(B) = P(AB)+P(A c B) P(A+B) = P(A)+P(B) P(AB) 29

30 Real Line : A Probability Space If f(x) 0, for all x, and - f(x)dx =1 We define P{ x x o } = - xo f(x)dx Then the probability space is completely characterized Events: {x 1 < x x 2 }, {x 1 x}, {x x 2 } P{single point} = 0 30

31 Conditional Probability The conditional probability of an event A given M: assuming P(M) 0 P(A M) = P(AM) / P(M) 1. If M A, Then P(A M) = 1 2. If A M, Then P(A M) P(A) 3. If A and M are independent, Then P(A M) = P(A) 4. Conditional Probability is a Probability P(A M) 0 P(S M) = 1 If A B =φ, Then P(A+B M) = P(A M) + P(B M) 31

32 Conditional Probability Example 1: In a fair six-face die experiment A={f 2 }, and M = {even} P(A) = 1/6, P(M) = ½, P(AM) = 1/6 P(A M) = (1/6)/(1/2) = 1/3 Example 2: Example 3: 3 white balls and two red balls t: the age a person dies to P{ t t o } = 0 f(t)dt Prior Probability Posteriori Probability 32

33 Total Probability Theorem { A 1, A 2, A 3, A n } is a partition of S Then P(B) = P(B A 1 )P(A 1 ) + P(B A 2 )P(A 2 ) + + P(B A n )P(A n ) Bayes Theorem: P(A k B) = P(B A k ) P(A k ) /P(B) = P(B A k ) P(A k ) /{P(B A 1 )P(A 1 ) +P(B A 2 )P(A 2 ) + +P(B A n )P(A n )} Example: four boxes of components Independent: A & B are independent if P(AB)=P(A)P(B) 33

34 Random Variables X(ξ): assignment of a number to every experimental outcome, ξ, in the entire probability space,s. Examples: X(f k ) = k; Y(f k ) = k 2 Events generated by random variables: {X x}; {X 3.5}={ f 1, f 2, f 3 } {Y y}; {Y 7} = { f 1, f 2 } 34

35 The Distribution Function F X (x) = P{X x }, x: a real variable Probability Distribution Function, Cumulative Distribution Function (CDF) Examples: coin tossing, die rolling, real line 35

36 Properties of Distribution Function F(x + ) = lim ε 0 F(x+ε); F(x- ) = lim ε 0 F(x-ε) 1. F(+) = 1; F(-) = 0 2. F(x) is non-decreasing 3. P{X > x} = 1-F(x) 4. F(x + ) = F(x), i.e., F is continuous from right 5. P{x 1 < X x 2 } = F(x 2 ) F(x 1 ) 6. P{X = x} = F(x) - F(x - ) 7. P{x 1 x x 2 } = F(x 2 ) F(x 1- ) 36

37 Probability Density Function (PDF) 1. f(x) 0 f(x) = (d/dx)f(x) 2. - x f(u)du = F(x) 3. - f(u)du = 1 4. a b f(u)du = F(b) F(a) 37

38 Discrete Random Variables and PMF Six-face die experiment Bernoulli trials & Binomial Distribution A: event of a successful trial P(A) = p; P(A c )= q; p+q = 1 p n (k) = P { A occurs exactly k times in n trials} p n (k) = ( n k )pk q (n-k) Σ ( n k )pk q (n-k) = 1 n k=0 38

39 Poisson Distribution Poisson Theorem: For n, p 0, np µ ( n k )pk q (n-k) e µ (µ k /k!), k=0,1,.., Random Poisson points: T τ P{k points in τ out of n points in T } = ( n k )pk q (n-k) where p = τ/t Now, let T, n and n/t λ Then, P{k points in τ } e nτ/τ (nτ/τ) k /k! = e λτ (λτ) k /k!, k=0,1,.., 39

40 Continuous Random Variables & PDF Gaussian (Normal) distribution f X (x) = 1/[(2π ) 1/2 σ x ] exp{-(x-m x ) 2 /2σ x2 )} Uniform distribution Rayleigh distribution f R (r) = r/σ 2 exp{-r 2 /2σ 2 )} for r>0 = 0 for r < 0 40

41 Mean, Variance & Moments m x = - xf(x)dx σ x2 = - (x-m x ) 2 f(x)dx m x = Σ kp{x=k} σ x2 = Σ (k- m x ) 2 P{X=k} N-th moment = - x n f(x)dx 41

42 Characteristic Function Φ(ω) = - f(x)e jωx dx Φ(ω) Φ(0) = 1 Moment generation using characteristic function 42

43 Functions of a Random Variable Y = g(x) ξ X real numbers g real numbers Y Y is a random variable F Y (y) = P{Y y }= P{g(X) y } y 1 y=g(x) y 2 x 2 x 2 x 2 x 1 x 43

44 Functions of a Random Variable F Y (y) = P{Y y }= P{g(X) y } y 1 y=g(x) y 2 x 2 x 2 x 2 x 1 x F Y (y 1 ) = F X (x 1 ) F Y (y 2 ) = F X (x 2 ) + F X (x 2 ) - F X (x 2 ) 44

45 Examples 1. Y = ax + b F Y (y) = P{Y y } =P{aX+b y } = P{aX y-b }= P{X (y-b)/a} = F X ((y-b)/a) if a>0 { { P{X (y-b)/a} 1- F X ((y-b)/a) if a<0 2. Y = X 2 For y < 0, F Y (y) = 0 For y > 0. F Y (y) = P{Y y } =P{X 2 y } = P{-y 1/2 < X y 1/2 } = F X (y 1/2 ) - F X (-y 1/2 ) y=x 2 x 45

46 Fundamental Theorem y=g(x) y y+dy f X (x) x 1 x 2 x 3 x x 1 +dx 1 x 2 +dx 2 x 3 +dx 3 f Y (y) = f X (x 1 ) / g (x 1 ) + f X (x 2 ) / g (x 2 ) + + f X (x n ) / g (x n ) where x 1, x 2, x 3, x n are the solutions of y=g(x) 46

47 Fundamental Theorem f Y (y)dy = P{ y < Y y+dy } = P{ y < g(x) y+dy } = P{ x 1 < X x 1 +dx 1 } + P{ x 2 < X x 2 +dx 2 } + P{ x 3 < X x 3 +dx 3 } = f X (x 1 )dx 1 +f X (x 2 )dx 2 + f X (x 3 )dx 3 = f X (x 1 )dy/g (x 1 )+ f X (x 2 )dy/ g (x 2 ) + f X (x 3 )dy/ g (x 3 ) f Y (y) =f X (x 1 )/ g (x 1 ) + f X (x 2 )/ g (x 2 ) + f X (x 3 )/ g (x 3 ) 47

48 Examples 1. Y = ax + b f Y (y) = (1/ a ) f X ([y-b]/a) 2. Y = ax 2, a > 0 3. Y = a sin(x), a > 0 & X uniform 4. Y = tan(x) f Y (y) = (1/(2ay) 1/2 ) {f X ([y/a] 1/2 ) + f X (-[y/a] 1/2 )} f Y (y) = 1/π (a 2 -y 2 ) 1/2, for y < a F Y (y) = ½ + (1/π )sin -1 (y/a) f Y (y) = 1/π (1+y 2 ) F Y (y) = ½ + (1/π )tan 1 (y) 48

49 Conditional Distribution & Density The conditional distribution of a random variable X given the event, M, is F(x M) = P{X x M} = P(X x & M)/P(M) Conditional distribution function is a distribution function The conditional density function f(x M) is given by f(x M) = (d/dx)f(x M) = lim x 0 [F(x+ x M)-F(x M)]/ x = lim x 0 P{x X<x+ x M}/ x 49

50 Conditional Distribution Examples F(x M)) M = {X a} F(x) a x F(x M)) M = {b < X a} F(x) b a x 50

51 Total Probability & Bayes Theorem F(x) = F(x A 1 )P(A 1 ) + F(x A 2 )P(A 2 ) + + F(x A n )P(A n ) f(x) = f(x A 1 )P(A 1 ) + f(x A 2 )P(A 2 ) + + f(x A n )P(A n ) P(A X x) = [F(x A)/F(x)] P(A) P(A x 1 <X x 2 ) = {[F(x 2 A)-F(x 1 A)]/[F(x 2 )-F(x 1 )]} P(A) P(A X = x ) = lim x 0 P{A x < X x+ x} = [f(x A)/f(x)]P(A) P(A) = P(A x)f(x)dx f(x A) = P(A X = x ) f(x) / P(A x)f(x)dx 51

52 Two Random Variables X(ξ), Y(ξ) :assignment of a pair of numbers {(x,y)} to every experimental outcome, ξ, in the entire probability space,s. {(x,y)} y ξ X Y M x 52

53 Joint Distribution y X (x,y) {X x} {Y y} x = {X x, Y y} F XY (x,y) = P{X x, Y y} x F XY (-, y) = 0 F XY (x, -) = 0 F XY (, ) = 1 F XY (, y) = F Y (y) : marginal distribution F XY (x, ) = F X (x) : marginal distribution 53

54 Joint Distribution y y 2 (x 0,y 2 ) y 1 {X x (x 0,y 1 ) 0, y 1 <Y y 2 } x 0 x 1 x 2 x (x 1,y 0 ) (x 2,y 0 ) y 0 {x 1 <X x 2,Y y 0 } P {x 1 <X x 2,Y y} = F XY (x 2,y)- F XY (x 1,y) P {X x, y 1 <Y y 2 } = F XY (x,y 2 )- F XY (x,y 1 ) 54

55 Joint Distribution y (x 1,y 2 ) (x 2,y 2 ) (x 1,y 1 ) ( x 2,y 1 ) x P {x 1 <X x 2, y 1 <Y y 2 } {x 1 <X x 2, y 1 <Y y 2 } = F XY (x 2,y 2 ) - F XY (x 2,y 1 ) - F XY (x 1,y 2 ) + F XY (x 1,y 1 ) 55

56 Joint Density f XY (x,y) =( 2 / x y)f XY (x,y) F XY (x,y) = - x - y fxy (u,v)dudv Marginal density f X (x) = - fxy (x,y)dy f Y (y) = - fxy (x,y)dx 56

57 Independence If X and Y are independent F XY (x,y) = P{X x, Y y} = P{X x} P{Y y} = F X (x) F Y (y) and f XY (x,y) = f X (x) f Y (y) X and Y are independent if and only if f XY (x,y) = f X (x) f Y (y) 57

58 Discrete Distributions Joint distribution P{ X= x k, Y = y m } = p km Σ k Σ m p km = 1 Marginal distribution p k = Σ m p km p m = Σ k p km 58

59 Bivariate Gaussian Distribution Random Variables X and Y are jointly normal (gaussian) if f XY (x,y) = 1/[2πσ x σ y (1-ρ 2 ) 1/2 ] exp{-a(x,y)/2(1-ρ 2 )} where A(x,y) = (x-m x ) 2 /σ 2 x + (y-m y ) 2 /σ 2 y 2ρ(x-m x )(y-m y )/σ x σ y Marginal normal density: f X (x) and f Y (y) f X (x) = 1/[(2π ) 1/2 σ x ] exp{-(x-m x ) 2 /2σ x2 )} f Y (y) = 1/[(2π ) 1/2 σ y ] exp{-(y-m y ) 2 /2σ y2 )} 59

60 Mean, Variance & Covariance m x = E(X) = - - x fxy (x,y)dxdy = - x fx (x)dx m y = E(Y) = - - y fxy (x,y)dxdy = - y fy (y)dy C(x,y)= E{(X- m x )(Y- m y )} = - - (x- mx )(y- m y ) f XY (x,y)dxdy Correlation coefficient: ρ(x,y) = C(x,y)/σ x σ y ρ(x,y) 1 For Joint Gaussian density, un-correlation implies independence. 60

61 Conditional Density F Y {y x 1 <X x 2 } = P{x 1 <X x 2,Y y}/p{x 1 <X x 2 } =[F XY (x 2,y)- F XY (x 1,y)] / [F X (x 2 )- F X (x 1 )] x2 f Y {y x 1 <X x 2 } = x1 fxy (x,y)dx /[F X (x 2 )- F X (x 1 )] x+ x f Y {y x<x x+ x} = x fxy (x,y)dx /[F X (x+ x)- F X (x)] x+ x f Y X {y X=x} = lim x 0 x fxy (x,y)dx /[F X (x+ x)- F X (x)] x+ x f Y X {y X=x} = lim x 0 (1/ x) x fxy (x,y)dx /f X (x)= f XY (x,y)/f X (x) f Y X (y X)= f XY (x,y)/f X (x) and f X Y (x Y)= f XY (x,y)/f Y (y) 61

62 Conditional Gaussian Density IF X and Y are jointly normal, then f XY (x,y) = 1/[2πσ x σ y (1-ρ 2 ) 1/2 ] exp{-a(x,y)/2(1-ρ 2 )} where A(x,y) = (x-m x ) 2 /σ 2 x + (y-m y ) 2 /σ 2 y 2ρ(x-m x )(y-m y )/σ x σ y and f Y X (y X)=1/{σ y [2π (1-ρ 2 )] 1/2 } exp{-b(x,y)/2σ y2 (1-ρ 2 )} where B(x,y) = [(y-m y ) ρσ y /σ x (x-m x )] 2 62

63 Conditional Mean & Variance Conditional Mean: m y x = E(Y X=x) = y f Y X (y x)dy Conditional Variance: σ y x2 = E{(Y-m y x ) 2 X=x) = (y-m y x ) 2 f Y X (y x)dy For Normal density m y x =m y + ρσ y /σ x (x-m x ) σ y x2 = σ y2 (1-ρ 2 ) Conditional mean as a Random Variable E(Y X=x) = φ(x) E(Y X) = φ(x) 63

64 Functions of Two Random Variables Z = g(x,y), P{Z z} = P{g(X,Y) z} F Z (z) = Dz f XY (x,y)dxdy where Dz = { g(x,y) z} Example: Z = X+Y F Z (z) = P{Z z} = P{X+Y z} z-y = - [ - fxy (x,y)dx]dy f Z (z) = (d/dz)f Z (z) = - fxy (z-y,y)dy X and Y independent f Z (z)= - fx (z-y)f Y (y) dy 64

65 Functions of Two Random Variables Example: Z = (X 2 +Y 2 ) 1/2 F Z (z) = P{Z z} = P{(X 2 +Y 2 ) 1/2 z} = Dz f XY (x,y)dxdy Let R = (X 2 +Y 2 ) 1/2 and θ =tan -1 (Y/X) P{(X 2 +Y 2 ) 1/2 2π z z} = P{R z} = 0 0 r g(r,θ)drdθ z = 2π 0 r g(r,θ)drdθ If f XY (x,y) = (1/2πσ 2 )exp[-(x 2 +y 2 )/2σ 2 ] Then f Z (z) = (z/σ 2 )exp[-(z 2 /2σ 2 )] for z > 0 i.e., Z is a Rayleigh distrinution 65

66 Polar Coordinate f(t) = X cos(ωt) + Y sin(ωt) = R cos(ωt θ) R = (X 2 +Y 2 ) 1/2 and θ =tan -1 (Y/X) If f XY (x,y) = (1/2πσ 2 )exp[-(x 2 +y 2 )/2σ 2 ] Then f Rφ (z) = (1/2π)(z/σ 2 )exp[-(z 2 /2σ 2 )] for z > 0 and θ < π 66

67 Joint Moments & Characteristic Function Expected value of Z=g(X,Y), E(Z) = - zf Z (z)dz = - - g(x,y)f XY (x,y)dxdy E(X+Y) = E(x) + E(Y) E(XY) = - - xy f XY (x,y)dxdy E(X)E(Y) unless X and Y are independent Φ(ω 1,ω 2 ) = E (e jω 1 X e jω 2 Y ) = - - e jω 1 x e jω 2 y f XY (x,y)dxdy 67

68 Mean Square Estimation Y is a random variable, C is a constant Find a C such that e=e{(y - C) 2 } is minimized e=e{(y - C) 2 } = E{Y 2 2YC + C 2 } = E{Y 2 } 2C E{Y} + C 2 = E{Y 2 } 2Cm y + C 2 e/ C = -2m y + 2C = 0 C = m y e min = E{(Y - m y ) 2 } = σ y 2 68

69 Mean Square Estimation Y is a random variable, X is another random variable Find a function of X, c(x), such that e=e{[y c(x)] 2 } is minimized e=e{[y-c(x)] 2 } = - - [y-c(x)] 2 f XY (x,y)dxdy = - f X (x){ - [y-c(x)] 2 f Y X (y x)dy} dx Since f X (x) is non-negative, for every fixed x the best choice for c(x) is c(x) = E(Y X=x) i.e., c(x) = E(Y X) = φ(x), the conditional mean e min = E{[Y - φ(x)] 2 } = E{Y 2 } 2E{Yφ(X)]} + E{φ 2 (X)} = E{Y 2 } E{φ 2 (X)} If X and Y are independent, Then c(x) = m y 69

70 Mean Square Estimation Y is a random variable, X is another random variable Find a linear function of X, c(x)= ax+b, such that e=e{[y c(x)] 2 } is minimized e = E{[Y-(aX+b)] 2 } = E{[(Y-aX)-b] 2 } If X and Y are jointly normal, b = m y -am x a = ρ σ y /σ x and e min = σ y 2 (1-ρ 2 ) φ(x) = (ρ σ y /σ x )X + m y -(ρσ y /σ x )m x i.e., the conditional mean, φ(x), is a linear function of X 70

71 Sample Mean & Sample Variance X k, k = 1, 2, 3, 4,,n; X k are i.i.d. with E(X k )=m, and E(X-m) 2 = σ 2 n Sample Mean: X= (1/n)Σ k=1 Xk n Sample Variance: V = [1/(n-1)]Σ k=1 (Xk X) 2 E(X) = m, σ 2 X = σ 2 /n E(V) = σ 2 71

72 Parameter Estimation X k = m + Y k, k = 1, 2, 3, 4,,n; Y k are i.i.d. and zero mean To find a minimum variance unbiased estimator of m n M= Σ k=1 ak X k i.e., E(M) = m and E[(M-m) 2 ] is minimum Σ k=1 n ak = m and Σ k=1 n ak σ k 2 = minimum Lagrange multiplier: V = Σ k=1 n ak σ k 2 λ (Σk=1 n ak 1) V/ a k = 0 a k = λ/2σ k 2 and λ = 2/(Σ k=1 n 1/σk 2 ) 72

73 Stochastic (Random) Processes X(t, ξ): assign a time function for every experimental outcome: ξ Continuous time process X(t, ξ) Discrete-time process X[n, ξ] Continuous-state process Discrete-state process For a fixed outcome ξ 0, we have a time function: x(t, ξ 0 ) A single realization is a sample function: x(t, ξ 0 ) For a fixed time t 0, we have a random variable: x(t 0, ξ) t 0 t 73

74 Random Processes Examples: X(t, ξ) = R(ξ)cos[ωt+θ(ξ)] Random Walk: X[n+1, ξ] = X[n, ξ] + X n Equality of processes 1. Almost everywhere: P{ X(t)-Y(t) 0} = 0 2. Mean-square (M.S.) sense: E{ X(t)-Y(t) 2 } = 0 74

75 Distribution & Density Functions First-order statistics F(x;t) = P{X(t) x} f(x;t) = ( / x)f(x;t) m X (t) = E{X(t)} = - xf(x;t)dx Second-order statistics F(x 1,x 2 ;t 1,t 2 ) = P{X(t 1 ) x 1, X(t 2 ) x 2 } f(x 1,x 2 ;t 1,t 2 ) = ( 2 / x 1 x 2 )F(x 1,x 2 ;t 1,t 2 ) 75

76 Autocorrelation Function R X (t 1,t 2 ) = E{X(t 1 )X(t 2 )} = - - x 1 x 2 f(x 1,x 2 ;t 1,t 2 ) dx 1 dx 2 R X (t,t) = E{X(t)X(t)}: second moment C X (t 1,t 2 ) = R X (t 1,t 2 ) m X (t 1 )m X (t 2 ): covariance Example: X(t) = A cos(ωt+θ) R X (t 1,t 2 ) = (A 2 /2)cos[ω(t 1 -t 2 )] 76

77 Stationary Processes Strictly Stationary For all n, all (t 1,t 2 t 3,t 4.t n-1,t n ) and all f(x 1,x 2. x n-1,x n ;t 1,t 2. t n-1,t n ) = f(x 1,x 2. x n-1,x n ;t 1 +,t 2 +. t n-1 +,t n + ) Wide-sense Stationary (WSS) m X (t) = E{X(t)} is independent of t R X (t 1,t 2 ) dependent only on τ = t 1 -t 2 Cyclo-stationary both m X (t) and R X (t+τ,t) are periodic function of t Y(t) = X(t) cos(ωt); where X(t) is WSS 77

78 Autocorrelation Function of WSS Processes R X (τ) = R X (-τ) R X (τ) R X (0) If for some T we have R X (T) = R X (0), then R X (kt) = R X (0) for all k A stationary process is Ergodic if for all function g(x) and for all ξ S if lim (1/ ) - /2 /2 g[x(t,ξ)]dt = E{g(X(t)} 78

79 Power Spectral Density S(ω) = - R(τ)e -jωτ dτ R(τ) = (1/2π) - S(ω)e jωτ dω R(0) = (1/2π) - S(ω) dω 79

80 LTI Systems X(t) m X (t) R X (τ) h(t) H(ω) Y(t) = - X(τ)h(t-τ)dτ = X(t)*h(t) m Y (t) = m X (t)*h(t) R Y (τ) = R X (τ)*h(t)*h(-t) S(ω) X S(ω) Y = S(ω) X H(ω) 2 80

81 White Process (White Noise) A process X(t) is called a white process if it has a flat power spectral density, i.e. S X (ω) is a constant for all ω. S X (ω) N 0 /2 ω Noise equivalent bandwidth of a filter: BW N BW N = (1/ H(ω) 2 max) 0 H(ω) 2 dω Total Noise Power: N = N 0 BW N H(ω) 2 max 81

82 Counting Process N(t) X 1 X 2 X 3 X 4 X 5 X 6 X k : Inter-arrival time 82

83 Poisson Process P N(T) [n] = (λt) n e -λt /n! [N(t 1 )-N(t 1 )] & [N(t 3 )-N(t 2 )] are independent E{N(t)} = λt, & R(t 1, t 2 ) = λt 1 +λ 2 t 1 t (t 2 1 <t 2 ) Inter-arrival time distribution F X (x) = 1- e -λx for x>0 83

84 Brownian Motion X(t) = 0, and [X(t+τ)-X(t)] is a zero mean Gaussian Random Variable with Variance = ατ Process of independent increment 84

85 Gaussian Process (Gaussian Noise) A process X(t) is called a Gaussian process if for all n and for all (t 1,t 2 t 3,t 4.t n-1,t n ), the random variables {X(t 1 ), X(t 2 ),.., X(t n )} have a jointly Gaussian density. For Gaussian processes, the mean, m X (t), and autocorrelation function, R X (t 1,t 2 ), completely characterize the process. If a Gaussian process X(t) is passed through an LTI system, then the output Process Y(t) is also Gaussian. For Gaussian processes, WSS and strictly stationary are equivalent 85

86 Power Spectrum of A Digital PAM System X(t) = Σ n=- Bn g(t-nt) E{X(t)} = Σ n=- E(Bn )g(t-nt) = m B Σ n=- g(t-nt) R X (t+τ,t) = E{X(t)X(t+τ)} = Σ k=- RB [k] Σ n=- g(t-nt)g(t+τ-nt-kt) X(t) is cyclostationary with period T! T/2 R X (τ) = (1/T) -T/2 R X (t+τ,t)dt S X (ω) = (1/T) {Σ k=- RB [k]e -jkωt } G(ω) 2 = (1/T)S B (ω) G(ω) 2 86