Exam #3 Review Wha is on an exam? Read hrough he homework and class examples 4 mulipar quesions. Poins assigned based on complexiy. (4Q, 40-60 ps. Sp. 06/Fa. 06) Skills #5.3, 4., 4.6, 5. Skills #6.3, 4., 4.6, 5. his exam is likely o be our problems, old quesions (exam maerial) and new quesions. Old Quesions (similar o,, or 4 rom exam): ) join densiy uncion derive marginal densiies, means, variances, correlaion, ideniy i independen and/or correlaed. ) Funcions o oher random variables Y=() deermine he pd, deined prob., condiional prob. 3) Funcions o wo random variables Z=a±bY deermine he pd, mean, variance New Quesions: ) Conidence Inerval Gaussian and Suden s- based inervals. ) Auocorrelaion/Crosscorrelaion in ime and probabiliy. You will be given a random sequence or process. Deermine he mean. Deermine he auocorrelaion. Perorm a cross-correlaion. Answer relaed quesions. Given a power specral densiy, deermine he random process or sequence mean, nd momen (oal power), variance. Noe: No ilering or requency domain power specral densiy quesions (like previous exam 3).. And now or a quick chaper review he imporan inormaion wihou he res! B.J. Bazuin, Spring 08 o 35 ECE 3800
ex Elemens 7 A Coninuous random Variable 7. A Coninuous Random Variable and Is Densiy, Disribuion Funcion, and Expeced Values 7. Example Calculaions or a Single Random Variable 7.3 Seleced Coninuous Disribuions 7.3. he Uniorm Disribuion 7.3. he Exponenial Disribuion 7.4 Condiional Probabiliies or a Coninuous Random Variable 7.5 Discree PMFs and Dela Funcions 7.6 Quanizaion 7.7 A Final Word 8 Muliple Coninuous Random Variables 8. Join Densiies and Disribuion Funcions 8. Expeced Values and Momens 8.3 Independence 8.4 Condiional Probabiliies or Muliple Random Variables 8.5 Exended Example: wo Coninuous Random Variables 8.6 Sums o Independen Random Variables 8.7 Random Sums 8.8 General ransormaions and he Jacobian 8.9 Parameer Esimaion or he Exponenial Disribuion 8.0 Comparison o Discree and Coninuous Disribuions 9 he Gaussian and Relaed Disribuions 9. he Gaussian Disribuion and Densiy 9. Quanile Funcion 9.3 Momens o he Gaussian Disribuion 9.4 he Cenral Limi heorem 9.5 Relaed Disribuions 9.5. he Laplace Disribuion 9.5. he Rayleigh Disribuion 9.5.3 he Chi-Squared and F Disribuions 9.6 Muliple Gaussian Random Variables 9.6. Independen Gaussian Random Variables 9.6. ransormaion o Polar Coordinaes 9.6.3 wo Correlaed Gaussian Random Variables 9.7 Example: Digial Communicaions Using QAM 9.7. Background 9.7. Discree ime Model 9.7.3 Mone Carlo Exercise 9.7.4 QAM Recap B.J. Bazuin, Spring 08 o 35 ECE 3800
0 Elemens o Saisics 0. A Simple Elecion Poll 0. Esimaing he Mean and Variance 0.3 Recursive Calculaion o he Sample Mean 0.4 Exponenial Weighing 0.5 Order Saisics and Robus Esimaes 0.6 Esimaing he Disribuion Funcion 0.7 PMF and Densiy Esimaes 0.8 Conidence Inervals 0.9 Signiicance ess and p-values 0.0 Inroducion o Esimaion heory 0. Minimum Mean Squared Error Esimaion 0. Bayesian Esimaion 3a Random Signals and Noise 3. Inroducion o Random Signals 3. A Simple Random Process 3.3 Fourier ransorms 3.4 WSS Random Processes 3.5 WSS Signals and Linear Filers 3.6 Noise 3.6. Probabilisic Properies o Noise 3.6. Specral Properies o Noise 3.7 Example: Ampliude Modulaion 3.8 Example: Discree ime Wiener Filer 3.9 he Sampling heorem or WSS Random Processes 3.9. Discussion 3.9. Example: Figure 3.4 3.9.3 Proo o he Random Sampling heorem Reserved or uure maerial and inal exam: Fourier ransorm, Power Specral Densiy, Linear Filers, Noise, Wiener Filer, Sampling heorem B.J. Bazuin, Spring 08 3 o 35 ECE 3800
See Exam Review or relaed maerials B.J. Bazuin, Spring 08 4 o 35 ECE 3800
Gaussian Disribuion and Densiy he Gaussian or Normal probabiliy densiy uncion is deined as:, where μ is he mean and σ is he variance he Gaussian Cumulaive Disribuion Funcion (CDF) he CDF can no be represened in a closed orm soluion! 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0. 0. Gaussian PDF and pd 0-8 -6-4 - 0 4 6 8 Normal Disribuion Gaussian wih zero mean and uni variance. he Normal probabiliy densiy uncion is deined as: x x N exp, or x he Normal Cumulaive Disribuion Funcion (CDF) x v N x exp dv v Noe he relaionship beween he Gaussian and Gaussian-Normal is see he MALAB: GaussianDemo.m F x x B.J. Bazuin, Spring 08 5 o 35 ECE 3800
Imporan noes on he Gaussian curve: he pd. here is only one maximum and i occurs a he mean value.. he densiy uncion is symmeric abou he mean value. 3. he widh o he densiy uncion is direcly proporional o he sandard deviaion,. he widh o occurs a he poins where he heigh is 0.607 o he maximum value. hese are also he poins o he maximum slope. Also noe ha: Pr 683 Pr 0. 0. 955 4. he maximum value o he densiy uncion is inversely proporional o he sandard deviaion,. 5. Since he densiy uncion has an area o uniy, i can be used as a represenaion o he impulse or dela uncion by leing approach zero. ha is x lim exp x 0 B.J. Bazuin, Spring 08 6 o 35 ECE 3800
Speciic Values or he Sandard Normal CDF, where μ = 0 is he mean and he variance σ =. Φ B.J. Bazuin, Spring 08 7 o 35 ECE 3800
Gaussian o Normal is a linear scaling Leing he linear relaionship be deined as he inverse mapping he Jocobian or derivaive becomes hereore hen or he normalized orm he R.V., In addiion, we would expec Φ Φ B.J. Bazuin, Spring 08 8 o 35 ECE 3800
wo-sided Gaussian Probabiliy Prμσxμσ 0.687 Prμσxμσ 0.9545 Prμ3σxμ3σ 0.9973 One-Sided Gaussian Probabiliy 0 0.5 0.843 0.977 3 0.9987 hree will be muliple problems and examples where eiher a wo-sided or on-sided Gaussian probabiliy is required. here are dierences in he soluions derived i he wrong one is seleced! B.J. Bazuin, Spring 08 9 o 35 ECE 3800
Equivalen Gaussian probabiliy represenaions, Manipulaions Pra b Pra b Pra b, Pra b Pr a Using normalized probabiliy Pra b Pr a Pra b Φ b b, Z b, Φ a he normalizaion o he Gaussian is oen implemened using Z. he compuaions wih he sandard normalizaion is reerred o as a z-score. Equivalen Probabiliies PrZ b Φb PrZ a Φa Pra Zb Φb Φa Also noe Φz Φz B.J. Bazuin, Spring 08 0 o 35 ECE 3800
Oher relaionships wih normalized Gaussian Pra Z a Φ Φ Φ Φ Pra Z a Φ or in general Pra Z b Φ Φ Φ Φ PraZb Φ Φ Perorming Compuaions he error uncion Φz For muliple bounds P b a b F b F er er a his deiniion is valid or MALAB and ECEL and WIKIPEDIA. here are oher sources ha do no deine i his way, so check beore use! Φz B.J. Bazuin, Spring 08 o 35 ECE 3800
he complemenary error uncion Φz Φ Φz Φ here are also inverse uncions or er and erc! zφ erinv he Q uncion in communicaions is he ail o he Gaussian Qz Φ B.J. Bazuin, Spring 08 o 35 ECE 3800
9.4 Cenral Limi heorem hps://en.wikipedia.org/wiki/cenral_limi_heorem In probabiliy heory, he cenral limi heorem (CL) esablishes ha, in mos siuaions, when independen random variables are added, heir properly normalized sum ends oward a normal disribuion (inormally a "bell curve") even i he original variables hemselves are no normally disribued. he heorem is a key concep in probabiliy heory because i implies ha probabilisic and saisical mehods ha work or normal disribuions can be applicable o many problems involving oher ypes o disribuions. he convoluion o pd o summed R.V. begins o look Gaussian aer a large number o R.V. are summed. Sums o IID R.V. S I n is known, he expeced value o he sum should be expeced S E n S E n I we normalize he summed random variance hen Y S S S Y E S S 0 S Y Var S S.0 S Based on he Cenral Limi heorem, Y will be a Normal R.V. as n becomes very large. B.J. Bazuin, Spring 08 3 o 35 ECE 3800
0. Esimaing he mean and variance Same as presened in he Sark & Woods slides he mos common esimae o he mean is he sample mean: where and he sample mean is an unbiased esimae as well as a consisen esimaor or he mean. his is he basis o he weak law o large numbers. he srong law o large numbers involves he relaionship Change in variance wih higher summed values: lim B.J. Bazuin, Spring 08 4 o 35 ECE 3800
Chebyshev s Inequaliy he inverse Forming an exac compuaion or CL based Gaussian Φ he ypical conidence inerval based on Gaussian saisics becomes 95% B.J. Bazuin, Spring 08 5 o 35 ECE 3800
0.8 Conidence Inervals wo disinc cases or consideraion. ) he conidence inerval or an esimae o he mean when he variance is known. Gaussian disribuion assumpion using z ) he conidence inerval or an esimae o he mean when he variance is unknown. Suden s- disribuion assumpion using A conidence inerval or an esimae o he mean is an inerval (L(),U()) such ha where τ is a hreshold, ypically 0.0 o 0.05 (99% or 95%). he upper and lower limis are ypically symmeric, bu i dieren, he: smalles is oen used as symmeric bounds. For he esimae o he mean wih known variance, he pd o he esimae is based on he Gaussian Normal disribuion ~ 0, he probabiliy is hen Φ For τ = 0.05 Φ 0.050.95 Φ 0.95 0.975.96.96 Rewriing he original inerval.96.96 0.95 For τ = 0.0 B.J. Bazuin, Spring 08 6 o 35 ECE 3800
Φ 0.00.99 Φ 0.99 0.995.58.58 Rewriing he original inerval.58.58 0.995 For he esimae o he mean wih unknown variance, he pd o he esimae is based on he Suden s- disribuion A saisical esimae o he variance mus be compued (or be available). hen, We deine We also require he degree o reedom or he disribuion, deined as v=n-. ables or he Suden s- disribuion are available. For 95%, Noice ha or ininie degrees o reedom, he Suden s disribuion is a Gaussian Normal! B.J. Bazuin, Spring 08 7 o 35 ECE 3800
B.J. Bazuin, Spring 08 8 o 35 ECE 3800 Conidence Inerval and Suden s disribuion hp://en.wikipedia.org/wiki/suden's_-disribuion Suden's disribuion arises when (as in nearly all pracical saisical work) he populaion sandard deviaion is unknown and has o be esimaed rom he daa. Noe ha: he disribuion depends on ν = n-, bu no μ or σ; he lack o dependence on μ and σ is wha makes he -disribuion imporan in boh heory and pracice. -disribuion conidence inerval For a wo sided conidence inerval, we wan.. I C n P n n n (A: exbook seps & ) Find he appropriae value based on he value v=n- or he conidence inerval seleced. (Hin. he ables says x and n, bu you are looking up v=n- (no n) and inding =x based on F ) (B: exbook sep 3 & 4) Based on he known compued variance or he esimaed R.V and he known number o samples. Compue he bounds on he inequaliy. C I n n P. Or he bounds on he rue mean, based on he conidence inerval are C I n n P. c c F F d CI c c 00 or c c, -sided
HW 4-4. Sark & Woods: A very large populaion o bipolar ransisors has a curren gain wih a mean value o 0 and a sandard deviaion o 0. he value o curren gain may be assumed o be independen Gaussian random variables. a) Find he conidence limis or a conidence level o 90% on he sample mean i i is compued rom a sample size o 50. k k n n wo sided es a 90% means ha k =.645. 0 k.645.343 n 50 0.343 0.343 b) Repea par (a) i he sample size is. wo sided es a 90% means ha k =.645. 0 k.645 3.590 n 0 3.590 0 3.590 Exercise 4-4. Sark & Woods A very large populaion o resisor values has a rue mean o 00 ohms and a sample sandard deviaion o 4 ohms. Find he conidence inerval on he sample mean or a conidence level o 95% i i is compued rom: a) a sample size o 00. v = 99 Using v=60 (no 00 given) and F=0.975 ( sided es) on p. G-4, =.00. hereore ~ ~ S S n n ~ 4 S.00 0.8 n 00 00 0.8 00 0.8 99. 00.8 Using v=0 (no 00 given) and F=0.975 ( sided es) on p. G-4, =.98. ~ 4 S.98 0.79 n 00 B.J. Bazuin, Spring 08 9 o 35 ECE 3800
99.08 00.79 b) a sample size o 9. v = 8 Using v=8 and F=0.975 ( sided es) on p. G-4, =.306. hereore ~ ~ S S n n ~ 4 S.306 3.075 n 9 00 3.075 00 3.075 B.J. Bazuin, Spring 08 0 o 35 ECE 3800
3. Inroducion o Random Signals A random process is a collecion o ime uncions and an associaed probabiliy descripion. When a coninuous or discree or mixed process in ime/space can be describe mahemaically as a uncion conaining one or more random variables. A sinusoidal waveorm wih a random ampliude. A sinusoidal waveorm wih a random phase. A sequence o digial symbols, each aking on a random value or a deined ime period (e.g. ampliude, phase, requency). A random walk (-D or 3-D movemen o a paricle) he enire collecion o possible ime uncions is an ensemble, designaed as x, where one paricular member o he ensemble, designaed as x, is a sample uncion o he ensemble. In general only one sample uncion o a random process can be observed! Le () be a random process. I we ake muliple ime samples,,,,, hen each ime sample is a random variable.,,, he random process migh hen have a nh order densiy uncion ha could be described as,,, ;,,, Nominally we migh described he densiy uncion o he elemens sampled rom he random process as ; he mean, nd momen and variance o () would hen be deined as ; ; B.J. Bazuin, Spring 08 o 35 ECE 3800
Signal correlaion As we have muliple imes a which he sample may be aken, we mus be able o compare samples ses o hemselves or dieren samples or sample ose sequences o each oher. Auo-correlaion is deined as Auo-covariance is deined as,,,,, For real R.V. i can also be shown ha,,,,,,, I wo separae random process exis, we describe cross-correlaion and cross covariance as,,,,,, B.J. Bazuin, Spring 08 o 35 ECE 3800
Saionary vs. Nonsaionary Random Processes he probabiliy densiy uncions or random variables in ime have been discussed, bu wha is he dependence o he densiy uncion on he value o ime, or n, when i is aken? I all marginal and join densiy uncions o a process do no depend upon he choice o he ime origin, he process is said o be saionary (ha is i doesn change wih ime). All he mean values and momens are consans and no uncions o ime! For nonsaionary processes, he probabiliy densiy uncions change based on he ime origin or in ime. For hese processes, he mean values and momens are uncions o ime. In general, we always aemp o deal wih saionary processes or approximae saionary by assuming ha he process probabiliy disribuion, means and momens do no change signiicanly during he period o ineres. Examples: Resisor values (noise varies based on he local emperaure) Wind velociy (varies signiicanly rom day o day) Humidiy (hough i can change rapidly during showers) he requiremen ha all marginal and join densiy uncions be independen o he choice o ime origin is requenly more sringen (igher) han is necessary or sysem analysis. A more relaxed requiremen is called saionary in he wide sense: where he mean value o any random variable is independen o he choice o ime,, and ha he correlaion o wo random variables depends only upon he ime dierence beween hem. ha is E E and E 0 R 0 or You will ypically deal wih Wide-Sense Saionary Signals (WSS). For WSS, he auocorrelaion and auocovariance are a uncion o he dierence in ime and no he absolue imes.,, 0, B.J. Bazuin, Spring 08 3 o 35 ECE 3800
For WSS random processes Addiional properies,,,,,. For real random processes he auo-correlaion and auo-covariance are symmeric abou 0.. he zeroeh lag (=0) o he auo-correlaion is he nd momen or power. And i mus be posiive. 0 0 0 0 3. he zeroeh lag (=0) o he auo-correlaion is a maximum or all ime lags. 0 4. I () is a zero mean WSS random process, he sum o he process and a consan will have a consan acor as par o he auocorrelaion and can be described as. he previous can be exended o, or a non-zero mean () For wo independen WSS random processes a b, B.J. Bazuin, Spring 08 4 o 35 ECE 3800
, Now wha is he auo-covariance? B.J. Bazuin, Spring 08 5 o 35 ECE 3800
Ergodic and Nonergodic Random Processes Ergodiciy deals wih he problem o deermining he saisics o an ensemble based on measuremens rom a sample uncion o he ensemble. For ergodic processes, all he saisics can be deermined rom a single uncion o he process. his may also be saed based on he ime averages. For an ergodic process, he ime averages (expeced values) equal he ensemble averages (expeced values). ha is o say, n n x x dx lim n d Noe ha ergodiciy canno exis unless he process is saionary! Ergodiciy is he concep ha ies ime based compuaions wih probabilisic based compuaions! n n n x x dx lim d he ime auocorrelaion lim x x d x x Overall WSS, ergodic processes are preerred as he saring condiions or engineering model, sysems and simulaions! Noes and igures are based on or aken rom maerials in he course exbook: Probabilisic Mehods o Signal and Sysem Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxord Press, 999. ISBN: 0-9-5354-9 B.J. Bazuin, Spring 08 6 o 35 ECE 3800
A Process or Deermining Saionariy and Ergodiciy a) Find he mean and he nd momen based on he probabiliy b) Find he ime sample mean and ime sample nd momen based on ime averaging. c) I he means or nd momens are uncions o ime non-saionary d) I he ime average mean and momens are no equal o he probabilisic mean and momens or i i is no saionary, hen i is non ergodic. Example Compuaions or means and nd momen: x x dx and x x x lim x d and x lim dx x d B.J. Bazuin, Spring 08 7 o 35 ECE 3800
he Auocorrelaion Funcion For a sample uncion deined by samples in ime o a random process, how alike are he dieren samples? Deine: and he auocorrelaion is deined as: R, E dx dx x x x x, he above uncion is valid or all processes, saionary and non-saionary. For WSS processes: R, E R I he process is ergodic, he ime average is equivalen o he probabilisic expecaion, or lim x x d x x and R Deine: For WSS R KK KK x k k and x k l * * k, l E k l x x pm x, x ; k l R, k E k x l x k * * * E k n 0 n x x pm x, x ; k,0 k l 0 k 0 k 0 x x I he process is ergodic, he sample average is equivalen o he probabilisic expecaion, or N * KK k lim n k n N N n N k 0 k l As a noe or hings you ve been compuing, he zero h lag o he auocorrelaion is R R E E dx x x, 0 0 lim x d x B.J. Bazuin, Spring 08 8 o 35 ECE 3800
Properies o Auocorrelaion Funcions ) R 0 E he mean squared value o he random process can be obained by observing he zeroh lag o he auocorrelaion uncion. ) R R or R k R k he auocorrelaion uncion is an even uncion in ime. Only posiive (or negaive) needs o be compued or an ergodic WSS random process. 3) R 0 or R k 0 R R he auocorrelaion uncion is a maximum a 0. For periodic uncions, oher values may equal he zeroh lag, bu never be larger. 4) I has a DC componen, hen Rxx has a consan acor. N R RNN Noe ha he mean value can be compued rom he auocorrelaion uncion consans! 5) I has a periodic componen, hen Rxx will also have a periodic componen o he same period. hink o: A w, 0 where A and w are known consans and hea is a uniorm random variable. A R E w 5b) For signals ha are he sum o independen random variable, he auocorrelaion is he sum o he individual auocorrelaion uncions. W Y RWW R RYY Y For non-zero mean uncions, (le w, x, y be zero mean and W,, Y have a mean) R R R R R hen we have WW WW WW YY Y Rww W Rxx Ryy Y Y R R R R WW ww W xx yy Y Y R R R ww R ww W xx yy Y W Y R R xx yy B.J. Bazuin, Spring 08 9 o 35 ECE 3800
6) I is ergodic and zero mean and has no periodic componen, hen we expec lim R 0 7) Auocorrelaion uncions can no have an arbirary shape. One way o speciying shapes permissible is in erms o he Fourier ransorm o he auocorrelaion uncion. ha is, i hen he resricion saes ha Addiional concep: R R R exp jw R 0 or all w d a N a EN N a R NN B.J. Bazuin, Spring 08 30 o 35 ECE 3800
he Crosscorrelaion Funcion For a wo sample uncion deined by samples in ime o wo random processes, how alike are he dieren samples? Deine: he cross-correlaion is deined as: R Y R Y and Y Y, E Y dx dy x y x y,, EY dy dx y x y x, he above uncion is valid or all processes, joinly saionary and non-saionary. For joinly WSS processes: R R Y Y, E Y RY EY R Noe: he order o he subscrips is imporan or cross-correlaion!, I he processes are joinly ergodic, he ime average is equivalen o he probabilisic expecaion, or and Y Y lim x y d x y lim y x d y x Y Y R R Y Y Y B.J. Bazuin, Spring 08 3 o 35 ECE 3800
Properies o Crosscorrelaion Funcions ) he properies o he zoreh lag have no paricular signiicance and do no represen mean-square values. I is rue ha he ordered crosscorrelaions mus be equal a 0.. R R 0 or 0 Y 0 Y Y 0 Y ) Crosscorrelaion uncions are no generally even uncions. However, here is an anisymmery o he ordered crosscorrelaions: For Subsiue Y Y Y R Y R Y lim x y d x y lim x y d x y lim y x d y x Y 3) he crosscorrelaion does no necessarily have is maximum a he zeroh lag. his makes sense i you are correlaing a signal wih a imed delayed version o isel. he crosscorrelaion should be a maximum when he lag equals he ime delay! I can be shown however ha R R R 0 Y 0 As a noe, he crosscorrelaion may no achieve he maximum anywhere 4) I and Y are saisically independen, hen he ordering is no imporan R Y E Y E EY Y and R Y R Y Y B.J. Bazuin, Spring 08 3 o 35 ECE 3800
rig Ideniies By he way, i is useul o have basic rig ideniies handy when dealing wih his su sin a sinb a b a b a b a b a b sin a b sina b sina b a sinb sina b sina b and as well as sina sina a a a sina a sin a a a a B.J. Bazuin, Spring 08 33 o 35 ECE 3800
B.J. Bazuin, Spring 08 34 o 35 ECE 3800 Generic Example o a Discree Specral Densiy sin C B A where he phase angles are uniormly disribued R.V rom 0 o π. sin sin C B A C B A E E R sin sin sin sin sin sin BC BC C AC AC B AB AB A E R sin sin sin BC C B A E R Wih pracice, we can see ha he above mah becomes E C E B A R which lead o C B A R Forming he PSD And hen aking he Fourier ransorm C B A S 4 4 C B A S
B.J. Bazuin, Spring 08 35 o 35 ECE 3800 We also know rom he beore d S dw w S hereore, he nd momen can be immediaely compued as d C B A 4 4 4 4 C B A C B A We can also see ha he mean value beconmed A C B A E sin So, he variance is C B A C B A A is a DC erm whereas B and C are AC erms as would be expeced rom ().