Statistics Definition: The science of assembling, classifying, tabulating, and analyzing data or facts:

8. Statistics Statistics Defiitio: The sciece of assemblig, classifyig, tabulatig, ad aalyzig data or facts: Descritive statistics The collectig, grouig ad resetig data i a way that ca be easily uderstood or assimilated. Iductive statistics or statistical iferece Use data to draw coclusios about or estimate arameters of, the eviromet from which the data came from. Theoretical Areas: Samlig Theory Estimatio Theory selectig samles from a collectio of data that is too large to be eamied comletely. cocered with makig estimates or redictios based o the data that are available. Cofidece iterval or level Hyothesis testig Curve fittig ad regressio Aalysis of Variace attemts to decide which of two or more hyotheses about the data are true. attemt to fid mathematical eressios that best rereset the data. attemt to assess the sigificace of variatios i the data ad the relatio of these variaces to the hysical situatios from which the data arose. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. of 45

From out r tetbook Probability theory allows us to model situatios that ehibit radomess i terms of radom eerimets ivolvig samle saces, evets, ad robability distributios. The aioms of robability allow us to develo a etesive set of tools for calculatig robabilities ad averages for a wide array of radom eerimets. The field of statistics lays the key role of bridgig robability models to the real world. I alyig robability models to real situatios, we must erform eerimets ad gather data to aswer questios such as: What are the values of arameters, e.g., mea ad variace, of a radom variable of iterest? Are the observed data cosistet with a assumed distributio? Are the observed data cosistet with a give arameter value of a radom variable? Statistics is cocered with the gatherig ad aalysis of data ad with the drawig of coclusios or ifereces from the data. The methods from statistics rovide us with the meas to aswer the above questios. I this chater we first cosider the estimatio of arameters of radom variables. We develo methods for obtaiig oit estimates as well as cofidece itervals for arameters of iterest. We the cosider hyothesis testig ad develo methods that allow us to accet or reect statemets about a radom variable based o observed data. We will aly these methods to determie the goodess of fit of distributios to observed data. The Gaussia radom variable lays a crucial role i statistics. We ote that the Gaussia radom variable is referred to as the ormal radom variable i the statistics literature. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 2 of 45

8. Samles ad Samlig Distributios The origi of the term statistics is i the gatherig of data about the oulatio i a state or locality i order to draw coclusios about roerties of the oulatio, e.g., otetial ta reveue or size of ool of otetial army recruits. Tyically the size of a oulatio was too large to make a ehaustive aalysis, so statistical ifereces about the etire oulatio were draw based o observatios from a samle of idividuals. The term oulatio is still used i statistics, but it ow refers to the collectio of all obects or elemets uder study i a give situatio. We suose that the roerty of iterest is observable ad measurable, ad that it ca be modeled by a radom variable. We gather observatio data by takig samles from the oulatio. 2 as cosistig of ideedet radom variables with the same distributio as. We defie a radom samle,,, Statistical methods ivariably ivolve erformig calculatios o the observed data. Tyically,,, : 2 we calculate a statistic based o the radom samle ˆ g,,, 2 I other words, a statistic is simly a fuctio of the radom vector. Clearly the statistic ˆ is itself a radom variable, ad so is subect to radom variability. Therefore estimates, ifereces ad coclusios based o the statistic must be stated i robabilistic terms. We have already ecoutered statistics to estimate imortat arameters of a radom variable. The samle mea is used to estimate the eected value of a radom variable : The relative frequecy of a evet A is a secial case of a samle mea ad is used to estimate the robability of A: f A The samlig distributio of a statistic ˆ is give by its robability distributio (cdf, df, or mf). The samlig distributio allows us to calculate arameters of ˆ e.g., mea, variace, ad momets, as well as robabilities ivolvig ˆ, Pa b. We will see that the samlig distributio ad its arameters allow us to determie the accuracy ad quality of the statistic ˆ. I Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 3 of 45

Remider from chater 7 S be the sum of iid radom variables with fiite mea Let E ad fiite variace 2 VAR ad let Z be the zero-mea, uit variace radom variable defied by S Z Note that Z is sometimes writte i terms of the samle mea: M Z Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 5 of 45

For eamle, to achieve a robability that the correct result is withi 95% we would comute: f 2Q 4 0. 95 P A Selectig = 0.05, the Q value becomes Q 4 0.05 2 Usig a looku tables (Table 8. i the tetbook. 44) 4.96 ad the umber of samles eeded to achieve the robability ad accuracy is : 2.96 2 4 For = 0.00, we would require = 960400 samles. 0.9604 Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 6 of 45

8.2 Parameter Estimatio I this sectio, we cosider the roblem of estimatig a arameter of a radom variable. We suose that we have obtaied a radom samle, 2,, cosistig of ideedet, idetically distributed versios of. Our estimator is give by a fuctio of : ˆ g,,, 2 After makig our observatios, we have the values, 2,, ad evaluate the estimate for by a sigle value g,, 2,. For this reaso ˆ is called a oit estimator for the arameter. We cosider the followig three questios:. What roerties characterize a good estimator? 2. How do we determie that a estimator is better tha aother? 3. How do we fid good estimators? I addressig the above questios, we also itroduce a variety of useful estimators. 8.2. Proerties of Estimators. Ubiased estimator 2. Small mea square estimatio error 3. Cosistet estimators ted towards the correct value as is icreased Ideally, a good estimator should be equal to the arameter, o the average. We say that the estimator ˆ is a ubiased estimator for if E ˆ The bias of ay estimator is defied by B ˆ E ˆ From Eq. (8.4) i Eamle 8., we see that the samle mea is a ubiased estimator for the mea. However, biased estimators are ot uusual as illustrated by the followig eamle. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 7 of 45

A secod measure of the quality of a estimator is the mea square estimatio error: 2 2 E ˆ Eˆ Eˆ Eˆ 2 ˆ Eˆ Eˆ Bˆ E ˆ 2 VAR ˆ B ˆ 2 E Obviously a good estimator should have a small mea square estimatio error because this imlies that the estimator values are clustered close to. If ˆ is a ubiased estimator of the, B ˆ ad the mea square error is simly the variace of the estimator ˆ. 0 I comarig two ubiased estimators, we clearly refer the oe with the smallest estimator variace. The comariso of biased estimators with ubiased estimators ca be tricky. It is ossible for a biased estimator to have a smaller mea square error tha ay ubiased estimator [Hardy]. I such situatios the biased estimator may be referable. 2 Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 9 of 45

MATLAB Note: std std var, flag imlied as 0 var,,, flag secified as A third measure of quality of a estimator ertais to its behavior as the samle size is icreased. We say that ˆ is a cosistet estimator if ˆ coverges to i robability, that is, as er Eq. (7.2), for every 0, limp ˆ 2 0 The estimator ˆ is said to be a strogly cosistet estimator if ˆ coverges to almost surely, that is, with robability, cf. Eqs. (7.22) ad (7.37). Cosistet estimators, whether biased or ubiased, ted towards the correct value of as is icreased. 2 Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 0 of 45

8.2.2 Fidig Good Estimators Ideally we would like to have estimators that are ubiased, have miimum mea square error, ad are cosistet. Ufortuately, there is o guaratee that ubiased estimators or cosistet estimators eist for all arameters of iterest. There is also o straightforward method for fidig the miimum mea square estimator for arbitrary arameters. Fortuately we do have the class of maimum likelihood estimators which are relatively easy to work with, have a umber of desirable roerties for large, ad ofte rovide estimators that ca be modified to be ubiased ad miimum variace. The et sectio deals with maimum likelihood estimatio. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. of 45

8.3 Maimum Likelihood Estimatio We ow cosider the maimum likelihood method for fidig a oit estimator ˆ for a ukow arameter. I this sectio we first show how the method works. We the reset several roerties that make maimum likelihood estimators very useful i ractice. The maimum likelihood method selects as its estimate the arameter value that maimizes the,,,. 2 robability of the observed data Before itroducig the formal method we use a eamle to demostrate the basic aroach. Let, 2,, be the observed values of a radom samle for the radom variable ad let be the arameter of iterest. The likelihood fuctio of the samle is a fuctio of defied as follows: l l,,, ; ; 2 l; f,, 2 2,,,,,, for discrete r. v. for cotiuousr. v. where the likelihood fuctio is based o the the oit mf or oit df evaluated at the observatio values. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 2 of 45

Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 3 of 45 Sice the samles are iid, we have a simle eressio for the likelihood fuctio: 2 2,,, or f f f f f 2 2,,, The maimum likelihood method selects the estimator value ˆ * where * is the arameter value that maimizes the likelihood fuctio, that is, ;,,, ma ; 2 * l l where the maimum is take over all allowable values of. Usually assumes a cotiuous set of values, so we fid the maimum of the likelihood fuctio over usig stadard methods from calculus. It is usually more coveiet to work with the log likelihood fuctio because we the work with the sum of terms istead of the roduct of terms i Eqs. (8.22) ad (8.23): ; l ; l L For discrete r.v. L l L ; l l ; l ; For coti8uous r.v. L f f l L ; l l ; l ; Maimizig the log likelihood fuctio is equivalet to maimizig the likelihood fuctio sice l() is a icreasig fuctio of. We obtai the maimum likelihood estimate by fidig the value * for which: 0 ; l ; l L 0 ; ; ; L L L Note: d du u u d d l

Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 4 of 45 Eamle 8.0 Estimatio of for a Beroulli radom variable Suose we erform ideedet observatios of a Beroulli radom variable with robability of success. Fid the maimum likelihood estimate for. Let i i i i,, 2, be the observed outcomes of the Beroulli trials. The mf for a idividual outcome ca be writte as follows: 0,, i i i if i if i The log likelihood fuctio is: i i i i i L l l l ; We take the first derivative with resect to ad set the result equal to zero: i i d d L i d d l l ; 0 ; i i L i d d 0 i i i i i * Therefore the maimum likelihood estimator for is the relative frequecy of successes, which is a secial case of the samle mea. From the revious sectio we kow that the samle mea estimator is ubiased ad cosistet.

The maimum likelihood estimator ossesses a imortat ivariace roerty that, i geeral, is ot satisfied by other estimators. Suose that istead of the arameter, we are iterested i estimatig a fuctio of, say h, which we assume is ivertible. It ca be show the that if * * is the maimum likelihood estimate of the h is the maimum likelihood estimate for h. (See Problem 8.34.) 8.3. *Cramer-Rao Boud I geeral, we would like to fid the ubiased estimator ˆ with the smallest ossible variace. This estimator would roduce the most accurate estimates i the sese of beig tightly clustered aroud the true value. The Cramer-Rao iequality addresses this questio i two stes. First, it rovides a lower boud to the miimum ossible variace achievable by ay ubiased estimator. This boud rovides a bechmark for assessig all ubiased estimators of. Secod, if a ubiased estimator achieves the lower boud the it has the smallest ossible variace ad mea square error. Furthermore, this ubiased estimator ca be foud usig the maimum likelihood method. Sice the radom samle ˆ will ehibit some uavoidable radom variatio ad hece will have ozero variace. Is there a lower limit to how small this variace ca be? The aswer is yes ad the lower boud is give by the recirocal of the Fisher iformatio which is defied as follows: 2 2 I E L E lf is a vector radom variable, we eect that the estimator The term iside the braces is called the score fuctio, which is defied as the artial derivative of the log likelihood fuctio with resect to the arameter. Note that the score fuctio is a fuctio of the vector radom variable. We have already see this fuctio whe fidig maimum likelihood estimators. The eected value of the score fuctio is zero sice: Score E l Score f f f f f d f d 0 where we assume that order of the artial derivative ad itegratio ca be echaged. Therefore I is equal to the variace of the score fuctio. d Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 7 of 45

The score fuctio measures the rate at which the log likelihood fuctio chages as varies. If L teds to chage quickly about the value 0 for most observatios of we ca eect that: () The Fisher iformatio will ted to be large sice the argumet iside the eected value i Eq. (8.34) will be large; (2) small deartures from the value 0 will be readily discerible i the observed statistics because the uderlyig df is chagig quickly. O the other had, if the likelihood fuctio chages slowly about 0 the the Fisher iformatio will be small. I additio, sigificatly differet values of 0 may have quite similar likelihood fuctios makig it difficult to distiguish amog arameter values from the observed data. I summary, larger values of I should allow for better erformig estimators that will have smaller variaces. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 8 of 45

We are ow ready to state the Cramer-Rao iequality. The Cramer-Rao lower boud cofirms our coecture that the variace of ubiased estimators must be bouded below by a ozero value. If the Fisher iformatio is high, the the lower boud is small, suggestig that low variace, ad hece accurate, estimators are ossible. The term I serves as a referece oit (boud) for the variace of all ubiased estimators, ad the ratio I VARˆ rovides a measure of efficiecy of a ubiased estimator. We say that a ubiased estimator is efficiet if it achieves the lower boud (also referred to as the Cramer- Rao boud). Based o Eq. 8.38, if a efficiet estimator eists the it ca be foud usig the maimum likelihood method. If a efficiet estimator does ot eist, the the lower boud i Eq. (8.37) is ot achieved by ay ubiased estimator. 8.3.2 Proof of Cramer-Rao Iequality skied Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 9 of 45

8.3.3 *Asymtotic Proerties of Maimum Likelihood Estimators Maimum likelihood estimators satisfy the followig asymtotic roerties that make them very useful whe the umber of samles is large.. Maimum likelihood estimates are cosistet: * lim 0 where 0 is the true arameter value * 2. For large, the maimum likelihood estimate is asymtotically Gaussia distributed, that * is, has a Gaussia distributio with zero mea ad variace 0 3. Maimum likelihood estimates are asymtotically efficiet: ˆ VAR lim I 0. The cosistecy roerty () imlies that maimum likelihood estimates will be close to the true value for large, ad asymtotic efficiecy (3) imlies that the variace becomes as small as ossible. The asymtotic Gaussia distributed roerty (2) is very useful because it allows us to evaluate the robabilities ivolvig the maimum likelihood estimator. I Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 20 of 45

8.4 Cofidece Itervals The samle mea estimator rovides us with a sigle umerical value for the estimate of E, amely, Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 2 of 45 This sigle umber gives o idicatio of the accuracy of the estimate or the cofidece that we ca lace o it. We ca obtai a idicatio of accuracy by comutig the samle variace, which is the average disersio about : ˆ i 2 2 ˆ i 2 If ˆ is small, the the observatios are tightly clustered about ad we ca be cofidet that 2 is close to E[]. O the other had, if ˆ is large, the samles are widely disersed about ad we caot be cofidet that is close to E[]. I this sectio we itroduce the otio of cofidece itervals, which aroach the questio i a differet way. Istead of seekig a sigle value that we desigate to be the estimate of the arameter of iterest (i.e., E ), we attemt to secify a iterval or set of values that is highly likely to cotai the true value of the arameter. I articular, we ca secify some high robability, say a ad ose the followig roblem: Fid a iterval [l(), u()] such that P l u a I other words, we use the observed data to determie a iterval that by desig cotais the true value of the arameter with robability a. We say that such a iterval is a cofidece iterval. This aroach simultaeously hadles the questio of the accuracy ad cofidece of a estimate. The robability a is a measure of the cosistecy, ad hece degree of cofidece, with which the iterval cotais the desired arameter: If we were to comute cofidece itervals a large umber of times, we would fid that aroimately a 00% of the time, the comuted itervals would cotai the true value of the arameter. For this reaso, a is called the cofidece level. The width of a cofidece iterval is a measure of the accuracy with which we ca ioit the estimate of a arameter. The arrower the cofidece iterval, the more accurately we ca secify the estimate for a arameter. The robability i Eq. (8.50) clearly deeds o the df of the sectio, we obtai cofidece itervals i the cases where the variables or ca be aroimated by Gaussia radom variables. ' s. I the remaider of this ' s are Gaussia radom

Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 22 of 45 We will use the equivalece betwee the followig evets: a a a a a a a a a a a a The last evet describes a cofidece iterval i terms of the observed data, ad the first evet will allow us to calculate robabilities from the samlig distributios. 8.4. Case : 's Gaussia; Ukow Mea ad Kow Variace

8.4.2 Case 2: 's Gaussia; Mea ad Variace Ukow Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 23 of 45

For a give -a the cofidece itervals give by Eq. (8.58) should be wider tha those give by Eq. (8.52), sice the former assumes that the variace is ukow. Figure 8.2 comares the Gaussia df ad the Studet s t df. It ca be see that the Studet s t df s are more disersed tha the Gaussia df ad so they ideed lead to wider cofidece itervals. O the other had, sice the accuracy of the samle variace icreases with, we ca eect that the cofidece iterval give by Eq. (8.58) should aroach that give by Eq. (8.52). It ca be see from Fig. 8.2 that the Studet s t df s do aroach the df of a zero-mea, uit-variace Gaussia radom variable with icreasig. This cofirms that Eqs. (8.52) ad (8.58) give the same cofidece itervals for large. Thus the bottom row of Table 8.2 yields the same cofidece itervals as Table 8.. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 24 of 45

8.4.3 Case 3: 's No-Gaussia; Mea ad Variace Ukow Equatio (8.58) ca be misused to comute cofidece itervals i eerimetal measuremets ad i comuter simulatio studies.the use of the method is ustified oly if the samles are iid ad aroimately Gaussia. If the radom variables are ot Gaussia, the above method for comutig cofidece itervals ca be modified usig the method of batch meas. This method ivolves erformig a series of ideedet eerimets i which the samle mea of the radom variable is comuted. If we assume that i each eerimet each samle mea is calculated from a large umber of iid observatios, the the cetral limit theorem imlies that the samle mea i each eerimet is aroimately Gaussia. We ca therefore comute a cofidece iterval from Eq. (8.58) usig the set of samle meas as the s. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 26 of 45

8.4.4 Cofidece Itervals for the Variace of a Gaussia Radom Variable Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 27 of 45

8.4.5 Summary of Cofidece Itervals for Gaussia Radom Variables I this sectio we have develoed cofidece itervals for the mea ad variace of Gaussia radom variables. The choice of cofidece iterval method deeds o which arameters are kow ad o whether the umber of samles is small or large. The cetral limit theorem makes the cofidece itervals reseted here alicable i a broad rage of situatios. Table 8.4 summarizes the cofidece itervals develoed i this sectio. The assumtios for each case ad the corresodig cofidece itervals are listed. 8.4.6 *Samlig Distributios for the Gaussia Radom Variable ot discussed Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 29 of 45

8.5 Hyothesis Testig I some situatios we are iterested i testig a assertio about a oulatio based o a radom samle. This assertio is stated i the form of a hyothesis about the uderlyig distributio of, ad the obective of the test is to accet or reect the hyothesis based o the observed data. Eamles of such assertios are: A give coi is fair. A ew maufacturig rocess roduces ew ad imroved batteries that last loger. Two radom oise sigals have the same mea. We first cosider sigificace testig where the obective is to accet or reect a give ull hyothesis H 0. Net we cosider the testig of agaist a alterative hyothesis H. We develo decisio rules for determiig the outcome of each test ad itroduce metrics for assessig the goodess or quality of these rules. I this sectio we use the traditioal aroach to hyothesis testig where we assume that the arameters of a distributio are ukow but ot radom. I the et sectio we use Bayesia models where the arameters of a distributio are radom variables with kow a riori robabilities. 8.5. Sigificace Testig I the geeral case we wish to test a hyothesis H 0 about a arameter of the radom variable. We call H 0 the ull hyothesis. The obective of a sigificace test is to accet or reect the ull hyothesis based o a radom samle, 2,,. I articular we are iterested i whether the observed data is sigificatly differet from what would be eected if the ull hyothesis is true. To secify a decisio rule we artitio the observatio sace ito a reectio or critical regio R ~ C where we reect the hyothesis ad a accetace regio R ~ where we accet the hyothesis. The decisio rule is the: Two kids of errors ca occur whe eecutig this decisio rule: Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 30 of 45

If the hyothesis is true, the we ca evaluate the robability of a Tye I error: If the ull hyothesis is false, we have o iformatio about the true distributio of the observatios ad hece we caot evaluate the robability of Tye II errors. We call the sigificace level of a test, ad this value reresets our tolerace for Tye I errors, that is, of reectig H 0 whe i fact it is true. The level of sigificace of a test rovides a imortat desig criterio for testig. Secifically, the reectio regio is chose so that the robability of Tye I error is o greater tha a secified level. Tyical values of are % ad 5%. The revious eamle shows reectio regios that are defied i terms of either two tails or oe tail of the distributio. We say that a test is two-tailed or two-sided if it ivolves two tails, that is, the reectio regio cosists of two itervals. Similarly, we refer to oe-tailed or oe-sided regios where the reectio regio cosists of a sigle iterval. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 3 of 45

A alterative aroach to hyothesis testig is to ot set the level ahead of time ad thus ot decide o a reectio regio. Istead, based o the observatio, e.g., we ask the questio, Assumig H 0 is true, what is the robability that the statistic would assume a value as etreme or more etreme tha P is? We call this robability the -value of the test statistic. If close to oe, the there is o reaso to reect the ull hyothesis, but if there is reaso to reect the ull hyothesis. Eamle 8.22 Defiig a -value P is small, the For eamle, i Eamle 8.22, the samle mea of 55 hours for =9 batteries has a -value: Note that a observatio value of 53.0 would yield a -value of 0.0. The -value for 55 is much smaller, so clearly this observatio calls for the ull hyothesis to be reected at % ad eve lower levels. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 32 of 45

8.5.2 Testig Simle Hyotheses A hyothesis test ivolves the testig of two or more hyotheses based o observed data. We will focus o the biary hyothesis case where we test a ull hyothesis H 0 agaist a alterative hyothesis H. The outcome of the test is: accet H 0 or reect H 0 ad accet H. A simle hyothesis secifies the associated distributio comletely. If the distributio is ot secified comletely (e.g., a Gaussia df with mea zero ad ukow variace), the we say that we have a comosite hyothesis. We cosider the testig of two simle hyotheses first. This case aears frequetly i electrical egieerig i the cotet of commuicatios systems. Whe the alterative hyothesis is simle, we ca evaluate the robability of Tye II errors, that is, of accetig H 0 whe His true. The robability of Tye II error rovides us with a secod criterio i the desig of a hyothesis test. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 33 of 45

The followig eamle shows that the umber of observatio samles rovides a additioal degree of freedom i desigig a hyothesis test. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 34 of 45

Differet criteria ca be used to select the reectio regio for reectig the ull hyothesis. A commo aroach is to select so the Tye I error is. This aroach, however, does ot comletely secify the reectio regio, for eamle, we may have a choice betwee oe-sided ad two-sided tests. The Neyma-Pearso criterio idetifies the reectio regio i a simle biary hyothesis test i which the Tye I error is equal to ad where the Tye II error is miimized. The followig result shows how to obtai the Neyma-Pearso test. Note that terms where ca be assiged to either R ~ or C R ~. We rove the theorem at the is called the likelihood ratio fuctio ad is give by the ratio of the ed of the sectio. likelihood of the observatio give H to the likelihood give H. The Neyma-Pearso test reects the ull hyothesis wheever the likelihood ratio is equal or eceeds the threshold. A more comact form of writig the test is: Sice the log fuctio is a icreasig fuctio, we ca equivaletly work with the log likelihood ratio: Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 35 of 45

8.5.3 Testig Comosite Hyotheses 8.5.4 Cofidece Itervals ad Hyothesis Testig 8.5.5 Summary of Hyothesis Tests This sectio has develoed may of the most commo hyothesis tests used i ractice. We develoed the tests i the cotet of secific eamles. Table 8.5 summarizes the basic hyothesis tests that were develoed i this sectio. The table resets the tests with the geeral test statistics ad arameters. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 37 of 45

8.6 Bayesia Decisio Methods I the revious sectios we develoed methods for estimatig ad for drawig ifereces about a arameter assumig that is ukow but ot radom. I this sectio, we elore methods that assume that is a radom variable ad that we have a riori kowledge of its distributio. This ew assumtio leads to ew methods for addressig estimatio ad hyothesis testig roblems. 8.6. Bayes Hyothesis Testig Cosider a simle biary hyothesis roblem where we are to decide betwee two hyotheses,,, : 2 based o a radom samle ad we assume that we kow that H 0 occurs with robability 0 ad H with robability 0. There are four ossible outcomes of the hyothesis test, ad we assig a cost to each outcome as a measure of its relative imortace:. H 0 true ad decide H 0 Cost = C 00 2. H 0 true ad decide H (Tye I error) Cost = C 0 3. Htrue ad decide H 0 (Tye II error) Cost = C 0 4. H true ad decide H Cost = C It is reasoable to assume that the cost of a correct decisio is less tha that of a erroeous decisio, that is C00 C0 ad C C0. Our obective is to fid the decisio rule that miimizes the average cost C: C C00 PH 0 H0 C0 PH H0 0 C PH H C0 PH 0 H Note: i detectio ad estimatio, otimal is tyically defied based o a cost fuctio. The cost fuctios are ofte readily related to the oeratio or arameter (e.g. miimum measquare-error, maimum likelihood, etc.) but they ca iclude arbitrary terms, as log as they ca be readily defied based o the measured values, differeces, statistics, or robability. The followig theorem idetifies the decisio rule that miimizes the average cost. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 38 of 45

8.6.2 Proof of Miimum Cost Theorem 8.6.3 Bayes Estimatio The framework for hyothesis testig that we described above ca also be alied to arameter estimatio. To estimate a arameter we assume the followig situatio. We suose that the arameter is a radom variable with a kow a riori distributio. A radom eerimet is erformed by ature to determie the value of that is reset. We caot observe directly, but we ca observe the radom samle, 2,, accordig to the active value of. Our obective is to obtai a estimator miimizes a cost fuctio that deeds o g ad :, which is distributed g which If the cost fuctio is the squared error, C g, g 2, we have the mea square estimatio roblem. I Chater 6 we showed that the otimum estimator is the coditioal : E. eected value of give Aother cost fuctio of iterest is g g otimum estimator is the media of the a osteriori df f C,, for which it ca be show that the Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 40 of 45.

A third cost fuctio of iterest is: This cost fuctio is aalogous to the cost fuctio i Eamle 8.3 i that the cost is always equal to ecet whe the estimate is withi of the true arameter value. It ca be show that the best estimator for this cost fuctio is the MAP estimator which maimizes the a. We eamie these estimators i the Problems. f osteriori robability We coclude with a estimator discovered by Bayes ad which gave birth to the aroach develoed i this sectio. The aroach was quite cotroversial because the use of a a riori distributio leads to two differet iterretatios of the meaig of robability. See [Bulmer,. 69] for a iterestig discussio o this cotroversy. I ractice, we do ecouter may situatios where we have a riori kowledge of the arameters of iterest. I such cases, Bayes methods have roved to be very useful. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 4 of 45

8.7 Testig the Fit of a Distributio to Data How well does the model fit the data? Suose you have ostulated a robability model for some radom eerimet, ad you are ow iterested i determiig how well the model fits your eerimetal data. How do you test this hyothesis? I this sectio we reset the chi-square test, which is widely used to determie the goodess of fit of a distributio to a set of eerimetal data. The atural first test to carry out is a eyeball comariso of the ostulated mf, df, or cdf ad a eerimetally determied couterart. If the outcome of the eerimet,, is discrete, the we ca comare the relative frequecy of outcomes with the robability secified by the mf, as show i Fig 8.6. If is cotiuous, the we ca artitio the real ais ito K mutually eclusive itervals ad determie the relative frequecy with which outcomes fall ito each iterval. These umbers would be comared to the robability of fallig i the iterval, as show i Fig 8.7. If the relative frequecies ad corresodig robabilities are i good agreemet, the we have established that a good fit eists. We ow show that the aroach outlied above leads to a test ivolvig the multiomial distributio. I have ot erformed this tye of testig i the ast. It is a alterative that I should cosidered i the future. the rest is left to the studet (ad me as well) to aly i ractice. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 43 of 45

Summary A statistic is a fuctio of a radom samle that cosists of iid observatios of a radom variable of iterest. The samlig distributio is the df or mf of the statistic. The critical values of a give statistic are the iterval edoits at which the comlemetary cdf achieves certai robabilities. A oit estimator is ubiased if its eected value equals the true value of the arameter of iterest, ad it is cosistet if it is asymtotically ubiased. The mea square error of a estimator is a measure of its accuracy. The samle mea ad the samle variace are cosistet estimators. Maimum likelihood estimators are obtaied by workig with the likelihood ad log likelihood fuctios. Maimum likelihood estimators are cosistet ad their estimatio error is asymtotically Gaussia ad efficiet. The Cramer-Rao iequality rovides a way of determiig whether a ubiased estimator achieves the miimum mea square error. A estimator that achieves the lower boud is said to be efficiet. Cofidece itervals rovide a iterval that is determied from observed data ad that by desig cotais a arameter iterest with a secified robability level. We develoed cofidece itervals for biomial, Gaussia, Studet s t, ad chi-square samlig distributios. Whe the umber of samles is large, the cetral limit theorem allows us to use estimators ad cofidece itervals for Gaussia radom variables eve if the radom variable of iterest is ot Gaussia. The samle mea ad samle variace for ideedet Gaussia radom variables are ideedet radom variables. The chi-square ad Studet s t-distributio are derived from statistics ivolvig Gaussia radom variables. A sigificace test is used to determie whether observed data are cosistet with a hyothesized distributio. The level of sigificace of a test is the robability that the hyothesis is reected whe it is actually true. A biary hyothesis tests decides betwee a ull hyothesis ad a alterative hyothesis based o observed data. A hyothesis is simle if the associated distributio is secified comletely. A hyothesis is comosite if the associated distributio is ot secified comletely. Simle biary hyothesis tests are assessed i terms of their sigificace level ad their Tye II error robability or, equivaletly, their ower. The Neyma-Pearso test leads to a likelihood ratio test that meets a target Tye I error robability while maimizig the ower of the test. Bayesia models are based o the assumtio of a a riori distributio for the arameters of iterest, ad they rovide a alterative aroach to assessig ad derivig estimators ad hyothesis tests. The chi-square distributio rovides a sigificace test for the fit of observed data to a hyothetical distributio. Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 44 of 45

CHECKLIST OF IMPORTANT TERMS Accetace regio Alterative hyothesis Bayes decisio rule Bayes estimator Chi-square goodess-of-fit test Comosite hyothesis Cofidece iterval Cofidece level Cosistet estimator Cramer-Rao iequality Critical regio Critical value Decisio rule Efficiecy False alarm robability Fisher iformatio Ivariace roerty Likelihood fuctio Likelihood ratio fuctio Log likelihood fuctio Maimum likelihood method Maimum likelihood test Mea square estimatio error Method of batch meas Neyma-Pearso test Normal radom variable Null hyothesis Poit estimator Poulatio Power Probability of detectio Radom samle Reectio regio Samlig distributio Score fuctio Sigificace level Sigificace test Simle hyothesis Statistic Strogly cosistet estimator Tye I error Tye II error Ubiased estimator Notes ad figures are based o or take from materials i the tetbook: Alberto Leo-Garcia, Probability, Statistics, ad Radom Processes For Electrical Egieerig, 3rd ed., Pearso Pretice Hall, 2008, ISBN: 03-4722-8. 45 of 45