Machine Learning. Tutorial on Basic Probability. Lecture 2, September 15, 2006
|
|
- Elaine Bernadette Haynes
- 6 years ago
- Views:
Transcription
1 Mache Learg -7/5 7/5-78, 78, all 6 Tutoral o asc robablty Erc g f Lecture, Setember 5, 6 Readg: Cha. &, C & Cha 5,6, TM What s ths? Classcal AI ad ML research gored ths heomea The roblem a eamle: you wat to catch a flght at :am from tt to S, ca I make t f I leave at 7am ad take a 8 at CMU? artal observablty road state, other drvers' las, etc. osy sesors rado traffc reorts ucertaty acto outcomes flat tre, etc. mmese comlety of modelg ad redctg traffc Reasog uder ucertaty!
2 asc robablty Cocets A samle sace S s the set of all ossble outcomes of a cocetual or hyscal, reeatable eermet. S ca be fte or fte. E.g., S may be the set of all ossble outcomes of a dce roll: S {,,3,4,5,6} E.g., S may be the set of all ossble ucleotdes of a DA ste: S { A,T, C,G} E.g., S may be the set of all ossble ostos tme-sace ostos o of a arcraft o a radar scree: S {, R } {, 36 } {, } A evet A s the ay subset S : ma + Seeg "" or "6" a roll; observg a "G" at a ste; UA7 sace-tme terval A evet sace E s the ossble worlds the outcomes ca hae All dce-rolls, readg a geome, motorg the radar sgal Vsualzg robablty Sace A robablty sace s a samle sace of whch, for every subset s S, there s a assgmet s S such that: s Σ s S s s s called the robablty or robablty mass of s Evet sace of all ossble worlds. Its area s Worlds whch A s true Worlds whch A s false a s the area of the oval
3 Kolmogorov Aoms All robabltes are betwee ad true regardless of the evet, my outcome s true false o evet makes my outcome true The robablty of a dsjucto s gve by A A + A A A? A A Why use robablty? There have bee attemts to develo dfferet methodologes for ucertaty: uzzy logc Qualtatve reasog Qualtatve hyscs robablty theory s othg but commo sese reduced to calculato erre Lalace, 8. I 93, de ett roved that t s rratoal to have belefs that volate these aoms, the followg sese: If you bet accordace wth your belefs, but your belefs volate the aoms, the you ca be guarateed to lose moey to a ooet whose belefs more accurately reflect the true state of the world. ere, bettg ad moey are roes for decso makg ad utltes. What f you refuse to bet? Ths s lke refusg to allow tme to ass: every acto cludg acto s a bet 3
4 Radom Varable A radom varable s a fucto that assocates a uque umercal value a toke wth every outcome of a eermet. The value of the r.v. wll vary from tral to tral as the eermet s reeated ω S Dscrete r.v.: The outcome of a dce-roll The outcome of readg a t at ste : ary evet ad dcator varable: Seeg a "A" at a ste, o/w. Ths descrbes the true or false outcome a radom evet. Ca we descrbe rcher outcomes the same way?.e.,,, 3, 4, for beg A, C, G, T --- thk about what would hae f we take eectato of. Ut-ase Radom vector [ A, T, G, C ] ', [,,,]' seeg a "G" at ste Cotuous r.v.: The outcome of recordg the true locato of a arcraft: The outcome of observg the measured locato of a arcraft true ω obs Dscrete rob. Dstrbuto I the dscrete case, a robablty dstrbuto o S ad hece o the doma of s a assgmet of a o-egatve real umber s to each s S or each vald value of such that Σ s S s. s tutvely, s corresods to the frequecy or the lkelhood of gettg s the eermets, f reeated may tmes call s s the arameters a dscrete robablty dstrbuto A robablty dstrbuto o a samle sace s sometmes called a robablty model, artcular f several dfferet dstrbutos are uder cosderato wrte models as M, M, robabltes as M, M e.g., M may be the arorate rob. dst. f s from "far dce", M s for the "loaded dce". M s usually a two-tule of {dst. famly, dst. arameters} 4
5 5 eroull dstrbuto: er Multomal dstrbuto: Mult, Multomal dcator varable:., w.. ad ],, [ where, [,...,6] [,...,6] j j j j j j j k k T G C A j j k T G C A j j de the dce - face} where, { Dscrete Dstrbutos for for Multomal dstrbuto: Mult, Cout varable: Dscrete Dstrbutos j j K where, M K K K K!!!!!!!! L L L
6 Cotuous rob. Dstrbuto A cotuous radom varable ca assume ay value a terval o the real le or a rego a hgh dmesoal sace usually corresods to a real-valued measuremets of some roerty, e.g., legth, osto, It s ot ossble to talk about the robablty of the radom varable assumg a artcular value --- Istead, we talk about the robablty of the radom varable assumg a value wth a gve terval, or half terval [ ],, < [ ], Arbtrary oolea combato of basc roostos Cotuous rob. Dstrbuto The robablty of the radom varable assumg a value wth some gve terval from to s defed to be the area uder the grah of the robablty desty fucto betwee ad. [ ] d, robablty mass:, ote that + d. Cumulatve dstrbuto fucto CD: < ' d ' robablty desty fucto D: + d d d ; >, Car flow o Lberty rdge cooked u! 6
7 What s the tutve meag of If a ad b, That s: the whe a value s samled from the dstrbuto wth desty, you are a/b tmes as lkely to fd that s very close to tha that s very close to. h < < + h lm h h < < + h + h d h a h b d h + h h Cotuous Dstrbutos Uform robablty Desty ucto / b a for a b elsewhere ormal Gaussa robablty Desty ucto / σ e πσ The dstrbuto s symmetrc, ad s ofte llustrated as a bell-shaed curve. Two arameters, mea ad σ stadard devato, determe the locato ad shae of the dstrbuto. The hghest ot o the ormal curve s at the mea, whch s also the meda ad mode. The mea ca be ay umercal value: egatve, zero, or ostve. Eoetal robablty Dstrbuto / desty : / e, o CD : e f f.4.3 < area Tme etwee Successve Arrvals ms. 7
8 Statstcal Characterzatos Eectato: the cetre of mass, mea value, frst momet: Samle mea: E S d dscrete cotuous Varace: the sreadess: Var Samle varace S [ E ] [ E ] d dscrete cotuous Gaussa ormal desty D If, σ, the robablty desty fucto df of s defed as / σ e πσ We wll ofte use the recso λ /σ stead of the varace σ. ere s how we lot the df matlab s-3:.:3; lots,ormdfs,mu,sgma ote that a desty evaluated at a ot ca be bgger tha! 8
9 Gaussa CD If Z,, the cumulatve desty fucto s defed as Φ z dz / z e dz π Ths has o closed form eresso, but s bult to most software ackages eg. ormcdf matlab stats toolbo. Use of the cdf If, σ, the Z /σ,. ow much mass s cotaed sde the [-.98σ,.98σ] terval? a b b a a < < b σ < Z < σ Φ σ Φ σ Sce Z.96 ormcdf.96 we have σ < < σ.95 9
10 Cetral lmt theorem If,, are..d. we wll come back to ths ot shortly cotuous radom varables The defe f,,..., As fty, Gaussa wth mea E[ ] ad varace Var[ ] Somewhat of a justfcato for assumg Gaussa ose s commo Elemetary maulatos of robabltes Set robablty of mult-valued r.v. {Odd} +3+5 /6+/6+/6 ½,, j j K Mult-varat dstrbuto: Jot robablty: true true j j {,K, } Margal robablty: j j S
11 Codtoal robablty racto of worlds whch s true that also have true "havg a headache" "comg dow wth lu" / /4 / fracto of flu-flcted worlds whch you have a headache / Defto: Corollary: The Cha Rule robablstc Iferece "havg a headache" "comg dow wth lu" / /4 / Oe day you wake u wth a headache. ou come wth the followg reasog: "sce 5% of flues are assocated wth headaches, so I must have a 5-5 chace of comg dow wth flu Is ths reasog correct?
12 robablstc Iferece "havg a headache" "comg dow wth lu" / /4 / The roblem:? The ayes Rule What we have just dd leads to the followg geeral eresso: Ths s ayes Rule
13 3 More Geeral orms of ayes Rule lu eadhead Drakeer + Z Z Z Z Z Z Z Z Z Z + S y y y ror Dstrbuto Suort that our roostos about the ossble has a "causal flow" e.g., ror or ucodtoal robabltes of roostos e.g., lu true ad Drkeer true. corresod to belef ror to arrval of ay ew evdece A robablty dstrbuto gves values for all ossble assgmets: Drkeer [.,.9,.,.8] ormalzed,.e., sums to
14 Jot robablty A jot robablty dstrbuto for a set of RVs gves the robablty of every atomc evet samle ot lu,drkeer a matr of values: lu,drkeer, eadache? Every questo about a doma ca be aswered by the jot dstrbuto, as we wll see later. osteror codtoal robablty Codtoal or osteror see later robabltes e.g., lueadache.78 gve that flu s all I kow OT f flu the 7.8% chace of eadache Reresetato of codtoal dstrbutos: lueadache -elemet vector of -elemet vectors If we kow more, e.g., Drkeer s also gve, the we have lueadache,drkeer.7 Ths effect s kow as ela away! lueadache,lu ote: the less or more certa belef remas vald after more evdece arrves, but s ot always useful ew evdece may be rrelevat, allowg smlfcato, e.g., lueadache,stealerw lueadache Ths kd of ferece, sactoed by doma kowledge, s crucal 4
15 Iferece by eumerato Start wth a Jot Dstrbuto uldg a Jot Dstrbuto of M3 varables rob.4..7 Make a truth table lstg all combatos of values of your varables f there are M oolea varables the the table wll have M rows.. or each combato of values, say how robable t s. ormalzed,.e., sums to Iferece wth the Jot Oe you have the JD you ca.4 ask for the robablty of ay atomc evet cosstet wth you query E row E..7. 5
16 Iferece wth the Jot Comute Margals.4..7 lu eadache. Iferece wth the Jot Comute Margals.4..7 eadache. 6
17 Iferece wth the Jot Comute Codtoals.4..7 E E E E E E IE E row row. Iferece wth the Jot Comute Codtoals.4. lu eadhead lu eadhead eadhead.7. Geeral dea: comute dstrbuto o query varable by fg evdece varables ad summg over hdde varables 7
18 Summary: Iferece by eumerato Let be all the varables. Tycally, we wat the osteror jot dstrbuto of the query varables gve secfc values e for the evdece varables E Let the hdde varables be --E The the requred summato of jot etres s doe by summg out the hdde varables: Eeα,Eeα h,ee, h The terms the summato are jot etres because, E, ad together ehaust the set of radom varables Obvous roblems: Worst-case tme comlety Od where d s the largest arty Sace comlety Od to store the jot dstrbuto ow to fd the umbers for Od etres??? Codtoal deedece Wrte out full jot dstrbuto usg cha rule: eadache;lu;vrus;drkeer eadache lu;vrus;drkeer lu;vrus;drkeer eadache lu;vrus;drkeer lu Vrus;Drkeer Vrus Drkeer Drkeer Assume deedece ad codtoal deedece eadachelu;drkeer luvrus Vrus Drkeer I.e.,? deedet arameters I most cases, the use of codtoal deedece reduces the sze of the reresetato of the jot dstrbuto from eoetal to lear. Codtoal deedece s our most basc ad robust form of kowledge about ucerta evromets. 8
19 Rules of Ideedece --- by eamles Vrus Drkeer Vrus ff Vrus s deedet of Drkeer lu Vrus;Drkeer luvrus ff lu s deedet of Drkeer, gve Vrus eadache lu;vrus;drkeer eadachelu;drkeer ff eadache s deedet of Vrus, gve lu ad Drkeer Margal ad Codtoal Ideedece Recall that for evets E.e. ad say, y, the codtoal robablty of E gve, wrtte as E, s E ad / the robablty of both E ad are true, gve s true E ad are statstcally deedet f E E.e., rob. E s true does't deed o whether s true; or equvaletly E ad E. E ad are codtoally deedet gve f E, E or equvaletly E, E 9
20 Why kowledge of Ideedece s useful Lower comlety tme, sace, search Motvates effcet ferece for all kds of queres Stay tued!! Structured kowledge about the doma easy to learg both from eert ad from data easy to grow Where do robablty dstrbutos come from? Idea Oe: uma, Doma Eerts Idea Two: Smler robablty facts ad some algebra e.g.,,, Idea Three: Lear them from data! A good chuk of ths course s essetally about varous ways of learg varous forms of them!
21 Desty Estmato A Desty Estmator lears a mag from a set of attrbutes to a robablty Ofte kow as arameter estmato f the dstrbuto form s secfed omal, Gaussa Three mortat ssues: ature of the data d, correlated, Objectve fucto MLE, MA, Algorthm smle algebra, gradet methods, EM, Evoluto scheme lkelhood o test data, redctablty, cosstecy, arameter Learg from d data Goal: estmate dstrbuto arameters from a dataset of deedet, detcally dstrbuted d, fully observed, trag cases D {,..., } Mamum lkelhood estmato MLE. Oe of the most commo estmators. Wth d ad full-observablty assumto, wrte L as the lkelhood of the data: L,, K, ; ; ;, K, ; 3. ck the settg of arameters most lkely to have geerated the data we saw: ; * arg ma L arg ma log L
22 Eamle : eroull model Data: We observed d co tossg: D{,,,, } Reresetato: ary r.v: Model: for for ow to wrte the lkelhood of a sgle observato? {, } The lkelhood of datasetd{,, }:,,..., #head #tals MLE Objectve fucto: h t l ; D log D log log + log We eed to mamze ths w.r.t. Take dervatves wrt h h l h h h MLE or MLE Suffcet statstcs h, where k, requecy as samle mea The couts, are suffcet statstcs of data D
23 MLE for dscrete jot dstrbutos More geerally, t s easy to show that # records whch evet s true evet total umber of records Ths s a mortat but sometmes ot so effectve learg algorthm! Eamle : uvarate ormal Data: We observed d real samles: Model: D{-.,,, -5.,, 3} Log lkelhood: / πσ e{ / σ } MLE: take dervatve ad set to zero: l ; D log D log πσ σ l / σ MLE l + 4 σ σ σ σ MLE ML 3
24 Overfttg Recall that for eroull Dstrbuto, we have What f we tossed too few tmes so that we saw zero head? We have head ad we wll redct that the robablty of ML, seeg a head et s zero!!! The rescue: head ML head Where ' s kow as the seudo- magary cout head ML ut ca we make ths more formal? head + tal head + ' head tal + + ' The ayesa Theory The ayesa Theory: e.g., for date D ad model M MD DMM/D the osteror equals to the lkelhood tmes the ror, u to a costat. Ths allows us to cature ucertaty about the model a rcled way 4
25 5 erarchcal ayesa Models are the arameters for the lkelhood α are the arameters for the ror α. We ca have hyer-hyer-arameters, etc. We sto whe the choce of hyer-arameters makes o dfferece to the margal lkelhood; tycally make hyerarameters costats. Where do we get the ror? Itellget guesses Emrcal ayes Tye-II mamum lkelhood comutg ot estmates of α : ma arg α α α v v v v MLE ayesa estmato for eroull eta dstrbuto: osteror dstrbuto of : otce the somorhsm of the osteror to the ror, such a ror s called a cojugate ror + + β α β α t h t h,...,,...,,..., Γ Γ + Γ β α β α β α β α β α β α,, ;
26 ayesa estmato for eroull, co'd osteror dstrbuto of :,..., h t α β h + α,...,,..., Mamum a osteror MA estmato: t + β MA arg ma log osteror mea estmato:,..., ata arameters ca be uderstood as seudo-couts ayes ror stregth: Aα+β + α β h + α t + β h D d C d + α + A ca be teroerated as the sze of a magary data set from whch we obta the seudo-couts Effect of ror Stregth Suose we have a uform ror αβ/, ad we observe v h, 8 t Weak ror A. osteror redcto: v v + h h, t 8, α α'. 5 + Strog ror A. osteror redcto: v v + h h, t 8, α α'. 4 + owever, f we have eough data, t washes away the ror. e.g., v h, t 8. The the estmates uder + + weak ad strog ror are + ad +, resectvely, both of whch are close to. 6
27 7 ayesa estmato for ormal dstrbuto ormal ror: Jot robablty: osteror: { } τ πτ / e / τ σ σ τ σ τ τ σ σ ~ ad, / / / / / / ~ where Samle mea { } τ πτ σ πσ / e e, / / { } σ πσ ~ / ~ e ~ /
CS 2750 Machine Learning Lecture 5. Density estimation. Density estimation
CS 750 Mache Learg Lecture 5 esty estmato Mlos Hausrecht mlos@tt.edu 539 Seott Square esty estmato esty estmato: s a usuervsed learg roblem Goal: Lear a model that rereset the relatos amog attrbutes the
More informationProbability and Statistics. What is probability? What is statistics?
robablt ad Statstcs What s robablt? What s statstcs? robablt ad Statstcs robablt Formall defed usg a set of aoms Seeks to determe the lkelhood that a gve evet or observato or measuremet wll or has haeed
More informationRandom Variables. ECE 313 Probability with Engineering Applications Lecture 8 Professor Ravi K. Iyer University of Illinois
Radom Varables ECE 313 Probablty wth Egeerg Alcatos Lecture 8 Professor Rav K. Iyer Uversty of Illos Iyer - Lecture 8 ECE 313 Fall 013 Today s Tocs Revew o Radom Varables Cumulatve Dstrbuto Fucto (CDF
More informationCHAPTER VI Statistical Analysis of Experimental Data
Chapter VI Statstcal Aalyss of Expermetal Data CHAPTER VI Statstcal Aalyss of Expermetal Data Measuremets do ot lead to a uque value. Ths s a result of the multtude of errors (maly radom errors) that ca
More information2. Independence and Bernoulli Trials
. Ideedece ad Beroull Trals Ideedece: Evets ad B are deedet f B B. - It s easy to show that, B deedet mles, B;, B are all deedet ars. For examle, ad so that B or B B B B B φ,.e., ad B are deedet evets.,
More informationLearning Graphical Models
School of omuter Scece Statstcal learg wth basc grahcal models robablstc Grahcal Models -78 Lecture 7 Oct 8 7 Recetor A Recetor B ase ase D ase E 3 4 5 Erc g Gee G TF F 6 7 Gee H 8 Readg: J-ha. 56 F-ha.
More informationLecture 3 Naïve Bayes, Maximum Entropy and Text Classification COSI 134
Lecture 3 Naïve Baes, Mamum Etro ad Tet Classfcato COSI 34 Codtoal Parameterzato Two RVs: ItellgeceI ad SATS ValI = {Hgh,Low}, ValS={Hgh,Low} A ossble jot dstrbuto Ca descrbe usg cha rule as PI,S PIPS
More informationParameter Estimation
arameter Estmato robabltes Notatoal Coveto Mass dscrete fucto: catal letters Desty cotuous fucto: small letters Vector vs. scalar Scalar: la Vector: bold D: small Hgher dmeso: catal Notes a cotuous state
More informationLecture 9. Some Useful Discrete Distributions. Some Useful Discrete Distributions. The observations generated by different experiments have
NM 7 Lecture 9 Some Useful Dscrete Dstrbutos Some Useful Dscrete Dstrbutos The observatos geerated by dfferet eermets have the same geeral tye of behavor. Cosequetly, radom varables assocated wth these
More informationBayes (Naïve or not) Classifiers: Generative Approach
Logstc regresso Bayes (Naïve or ot) Classfers: Geeratve Approach What do we mea by Geeratve approach: Lear p(y), p(x y) ad the apply bayes rule to compute p(y x) for makg predctos Ths s essetally makg
More informationSummary of the lecture in Biostatistics
Summary of the lecture Bostatstcs Probablty Desty Fucto For a cotuos radom varable, a probablty desty fucto s a fucto such that: 0 dx a b) b a dx A probablty desty fucto provdes a smple descrpto of the
More informationEntropy, Relative Entropy and Mutual Information
Etro Relatve Etro ad Mutual Iformato rof. Ja-Lg Wu Deartmet of Comuter Scece ad Iformato Egeerg Natoal Tawa Uverst Defto: The Etro of a dscrete radom varable s defed b : base : 0 0 0 as bts 0 : addg terms
More informationUNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS Postpoed exam: ECON430 Statstcs Date of exam: Jauary 0, 0 Tme for exam: 09:00 a.m. :00 oo The problem set covers 5 pages Resources allowed: All wrtte ad prted
More informationå 1 13 Practice Final Examination Solutions - = CS109 Dec 5, 2018
Chrs Pech Fal Practce CS09 Dec 5, 08 Practce Fal Examato Solutos. Aswer: 4/5 8/7. There are multle ways to obta ths aswer; here are two: The frst commo method s to sum over all ossbltes for the rak of
More informationChapter 5 Properties of a Random Sample
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Revew for the prevous lecture Cocepts: t-dstrbuto, F-dstrbuto Theorems: Dstrbutos of sample mea ad sample varace, relatoshp betwee sample mea ad sample
More informationLecture 3 Probability review (cont d)
STATS 00: Itroducto to Statstcal Iferece Autum 06 Lecture 3 Probablty revew (cot d) 3. Jot dstrbutos If radom varables X,..., X k are depedet, the ther dstrbuto may be specfed by specfyg the dvdual dstrbuto
More informationElementary manipulations of probabilities
Elemetary maipulatios of probabilities Set probability of multi-valued r.v. {=Odd} = +3+5 = /6+/6+/6 = ½ X X,, X i j X i j Multi-variat distributio: Joit probability: X true true X X,, X X i j i j X X
More informationBayesian belief networks
Lecture 14 ayesa belef etworks los Hauskrecht mlos@cs.ptt.edu 5329 Seott Square Desty estmato Data: D { D1 D2.. D} D x a vector of attrbute values ttrbutes: modeled by radom varables { 1 2 d} wth: otuous
More informationUNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS Exam: ECON430 Statstcs Date of exam: Frday, December 8, 07 Grades are gve: Jauary 4, 08 Tme for exam: 0900 am 00 oo The problem set covers 5 pages Resources allowed:
More informationPoint Estimation: definition of estimators
Pot Estmato: defto of estmators Pot estmator: ay fucto W (X,..., X ) of a data sample. The exercse of pot estmato s to use partcular fuctos of the data order to estmate certa ukow populato parameters.
More informationSimulation Output Analysis
Smulato Output Aalyss Summary Examples Parameter Estmato Sample Mea ad Varace Pot ad Iterval Estmato ermatg ad o-ermatg Smulato Mea Square Errors Example: Sgle Server Queueg System x(t) S 4 S 4 S 3 S 5
More informationLecture Notes Types of economic variables
Lecture Notes 3 1. Types of ecoomc varables () Cotuous varable takes o a cotuum the sample space, such as all pots o a le or all real umbers Example: GDP, Polluto cocetrato, etc. () Dscrete varables fte
More informationENGI 4421 Joint Probability Distributions Page Joint Probability Distributions [Navidi sections 2.5 and 2.6; Devore sections
ENGI 441 Jot Probablty Dstrbutos Page 7-01 Jot Probablty Dstrbutos [Navd sectos.5 ad.6; Devore sectos 5.1-5.] The jot probablty mass fucto of two dscrete radom quattes, s, P ad p x y x y The margal probablty
More informationContinuous Random Variables: Conditioning, Expectation and Independence
Cotuous Radom Varables: Codtog, xectato ad Ideedece Berl Che Deartmet o Comuter cece & Iormato geerg atoal Tawa ormal Uverst Reerece: - D.. Bertsekas, J.. Tstskls, Itroducto to robablt, ectos 3.4-3.5 Codtog
More informationX ε ) = 0, or equivalently, lim
Revew for the prevous lecture Cocepts: order statstcs Theorems: Dstrbutos of order statstcs Examples: How to get the dstrbuto of order statstcs Chapter 5 Propertes of a Radom Sample Secto 55 Covergece
More informationMean is only appropriate for interval or ratio scales, not ordinal or nominal.
Mea Same as ordary average Sum all the data values ad dvde by the sample sze. x = ( x + x +... + x Usg summato otato, we wrte ths as x = x = x = = ) x Mea s oly approprate for terval or rato scales, ot
More informationNonparametric Density Estimation Intro
Noarametrc Desty Estmato Itro Parze Wdows No-Parametrc Methods Nether robablty dstrbuto or dscrmat fucto s kow Haes qute ofte All we have s labeled data a lot s kow easer salmo bass salmo salmo Estmate
More informationBayesian belief networks
Lecture 19 ayesa belef etworks los Hauskrecht mlos@cs.ptt.edu 539 Seott Square Varous ferece tasks: robablstc ferece Dagostc task. from effect to cause eumoa Fever redcto task. from cause to effect Fever
More informationChapter 3 Sampling For Proportions and Percentages
Chapter 3 Samplg For Proportos ad Percetages I may stuatos, the characterstc uder study o whch the observatos are collected are qualtatve ature For example, the resposes of customers may marketg surveys
More informationSTATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. x, where. = y - ˆ " 1
STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS Recall Assumpto E(Y x) η 0 + η x (lear codtoal mea fucto) Data (x, y ), (x 2, y 2 ),, (x, y ) Least squares estmator ˆ E (Y x) ˆ " 0 + ˆ " x, where ˆ
More information( ) = ( ) ( ) Chapter 13 Asymptotic Theory and Stochastic Regressors. Stochastic regressors model
Chapter 3 Asmptotc Theor ad Stochastc Regressors The ature of eplaator varable s assumed to be o-stochastc or fed repeated samples a regresso aalss Such a assumpto s approprate for those epermets whch
More informationEconometric Methods. Review of Estimation
Ecoometrc Methods Revew of Estmato Estmatg the populato mea Radom samplg Pot ad terval estmators Lear estmators Ubased estmators Lear Ubased Estmators (LUEs) Effcecy (mmum varace) ad Best Lear Ubased Estmators
More informationLecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model
Lecture 7. Cofdece Itervals ad Hypothess Tests the Smple CLR Model I lecture 6 we troduced the Classcal Lear Regresso (CLR) model that s the radom expermet of whch the data Y,,, K, are the outcomes. The
More informationRandom Variables and Probability Distributions
Radom Varables ad Probablty Dstrbutos * If X : S R s a dscrete radom varable wth rage {x, x, x 3,. } the r = P (X = xr ) = * Let X : S R be a dscrete radom varable wth rage {x, x, x 3,.}.If x r P(X = x
More informationGenerative classification models
CS 75 Mache Learg Lecture Geeratve classfcato models Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Data: D { d, d,.., d} d, Classfcato represets a dscrete class value Goal: lear f : X Y Bar classfcato
More informationLecture 3. Sampling, sampling distributions, and parameter estimation
Lecture 3 Samplg, samplg dstrbutos, ad parameter estmato Samplg Defto Populato s defed as the collecto of all the possble observatos of terest. The collecto of observatos we take from the populato s called
More informationSTK4011 and STK9011 Autumn 2016
STK4 ad STK9 Autum 6 Pot estmato Covers (most of the followg materal from chapter 7: Secto 7.: pages 3-3 Secto 7..: pages 3-33 Secto 7..: pages 35-3 Secto 7..3: pages 34-35 Secto 7.3.: pages 33-33 Secto
More information2SLS Estimates ECON In this case, begin with the assumption that E[ i
SLS Estmates ECON 3033 Bll Evas Fall 05 Two-Stage Least Squares (SLS Cosder a stadard lear bvarate regresso model y 0 x. I ths case, beg wth the assumto that E[ x] 0 whch meas that OLS estmates of wll
More informationBIOREPS Problem Set #11 The Evolution of DNA Strands
BIOREPS Problem Set #11 The Evoluto of DNA Strads 1 Backgroud I the md 2000s, evolutoary bologsts studyg DNA mutato rates brds ad prmates dscovered somethg surprsg. There were a large umber of mutatos
More informationCHAPTER 6. d. With success = observation greater than 10, x = # of successes = 4, and
CHAPTR 6 Secto 6.. a. We use the samle mea, to estmate the oulato mea µ. Σ 9.80 µ 8.407 7 ~ 7. b. We use the samle meda, 7 (the mddle observato whe arraged ascedg order. c. We use the samle stadard devato,
More informationArtificial Intelligence Learning of decision trees
Artfcal Itellgece Learg of decso trees Peter Atal atal@mt.bme.hu A.I. November 21, 2016 1 Problem: decde whether to wat for a table at a restaurat, based o the followg attrbutes: 1. Alterate: s there a
More informationBayesian Classification. CS690L Data Mining: Classification(2) Bayesian Theorem: Basics. Bayesian Theorem. Training dataset. Naïve Bayes Classifier
Baa Classfcato CS6L Data Mg: Classfcato() Referece: J. Ha ad M. Kamber, Data Mg: Cocepts ad Techques robablstc learg: Calculate explct probabltes for hypothess, amog the most practcal approaches to certa
More informationParametric Density Estimation: Bayesian Estimation. Naïve Bayes Classifier
arametrc Dest Estmato: Baesa Estmato. Naïve Baes Classfer Baesa arameter Estmato Suose we have some dea of the rage where arameters should be Should t we formalze such ror owledge hoes that t wll lead
More informationBASIC PRINCIPLES OF STATISTICS
BASIC PRINCIPLES OF STATISTICS PROBABILITY DENSITY DISTRIBUTIONS DISCRETE VARIABLES BINOMIAL DISTRIBUTION ~ B 0 0 umber of successes trals Pr E [ ] Var[ ] ; BINOMIAL DISTRIBUTION B7 0. B30 0.3 B50 0.5
More informationρ < 1 be five real numbers. The
Lecture o BST 63: Statstcal Theory I Ku Zhag, /0/006 Revew for the prevous lecture Deftos: covarace, correlato Examples: How to calculate covarace ad correlato Theorems: propertes of correlato ad covarace
More informationProbabilistic Graphical Models
School of Comuter Scece Probablstc rahcal Models Parameter Est. fully observed Bs Erc Xg X X X X Lecture 7 February 5 04 X X 3 X X X 3 X 3 Readg: KF-cha 7 X 4 X 4 Erc Xg @ CMU 005-04 Learg rahcal Models
More informationbest estimate (mean) for X uncertainty or error in the measurement (systematic, random or statistical) best
Error Aalyss Preamble Wheever a measuremet s made, the result followg from that measuremet s always subject to ucertaty The ucertaty ca be reduced by makg several measuremets of the same quatty or by mprovg
More information1 Solution to Problem 6.40
1 Soluto to Problem 6.40 (a We wll wrte T τ (X 1,...,X where the X s are..d. wth PDF f(x µ, σ 1 ( x µ σ g, σ where the locato parameter µ s ay real umber ad the scale parameter σ s > 0. Lettg Z X µ σ we
More informationChapter 4 Multiple Random Variables
Revew for the prevous lecture: Theorems ad Examples: How to obta the pmf (pdf) of U = g (, Y) ad V = g (, Y) Chapter 4 Multple Radom Varables Chapter 44 Herarchcal Models ad Mxture Dstrbutos Examples:
More informationDr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur
Aalyss of Varace ad Desg of Exermets-I MODULE II LECTURE - GENERAL LINEAR HYPOTHESIS AND ANALYSIS OF VARIANCE Dr Shalabh Deartmet of Mathematcs ad Statstcs Ida Isttute of Techology Kaur Tukey s rocedure
More informationChannel Models with Memory. Channel Models with Memory. Channel Models with Memory. Channel Models with Memory
Chael Models wth Memory Chael Models wth Memory Hayder radha Electrcal ad Comuter Egeerg Mchga State Uversty I may ractcal etworkg scearos (cludg the Iteret ad wreless etworks), the uderlyg chaels are
More informationX X X E[ ] E X E X. is the ()m n where the ( i,)th. j element is the mean of the ( i,)th., then
Secto 5 Vectors of Radom Varables Whe workg wth several radom varables,,..., to arrage them vector form x, t s ofte coveet We ca the make use of matrx algebra to help us orgaze ad mapulate large umbers
More informationMultivariate Transformation of Variables and Maximum Likelihood Estimation
Marquette Uversty Multvarate Trasformato of Varables ad Maxmum Lkelhood Estmato Dael B. Rowe, Ph.D. Assocate Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Copyrght 03 by Marquette Uversty
More informationMEASURES OF DISPERSION
MEASURES OF DISPERSION Measure of Cetral Tedecy: Measures of Cetral Tedecy ad Dsperso ) Mathematcal Average: a) Arthmetc mea (A.M.) b) Geometrc mea (G.M.) c) Harmoc mea (H.M.) ) Averages of Posto: a) Meda
More informationContinuous Distributions
7//3 Cotuous Dstrbutos Radom Varables of the Cotuous Type Desty Curve Percet Desty fucto, f (x) A smooth curve that ft the dstrbuto 3 4 5 6 7 8 9 Test scores Desty Curve Percet Probablty Desty Fucto, f
More informationSTK3100 and STK4100 Autumn 2017
SK3 ad SK4 Autum 7 Geeralzed lear models Part III Covers the followg materal from chaters 4 ad 5: Sectos 4..5, 4.3.5, 4.3.6, 4.4., 4.4., ad 4.4.3 Sectos 5.., 5.., ad 5.5. Ørulf Borga Deartmet of Mathematcs
More informationChapter 14 Logistic Regression Models
Chapter 4 Logstc Regresso Models I the lear regresso model X β + ε, there are two types of varables explaatory varables X, X,, X k ad study varable y These varables ca be measured o a cotuous scale as
More informationChapter 5 Properties of a Random Sample
Lecture 3 o BST 63: Statstcal Theory I Ku Zhag, /6/006 Revew for the revous lecture Cocets: radom samle, samle mea, samle varace Theorems: roertes of a radom samle, samle mea, samle varace Examles: how
More informationMaximum Likelihood Estimation
Marquette Uverst Maxmum Lkelhood Estmato Dael B. Rowe, Ph.D. Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Coprght 08 b Marquette Uverst Maxmum Lkelhood Estmato We have bee sag that ~
More informationOrdinary Least Squares Regression. Simple Regression. Algebra and Assumptions.
Ordary Least Squares egresso. Smple egresso. Algebra ad Assumptos. I ths part of the course we are gog to study a techque for aalysg the lear relatoshp betwee two varables Y ad X. We have pars of observatos
More informationD KL (P Q) := p i ln p i q i
Cheroff-Bouds 1 The Geeral Boud Let P 1,, m ) ad Q q 1,, q m ) be two dstrbutos o m elemets, e,, q 0, for 1,, m, ad m 1 m 1 q 1 The Kullback-Lebler dvergece or relatve etroy of P ad Q s defed as m D KL
More information1 Onto functions and bijections Applications to Counting
1 Oto fuctos ad bectos Applcatos to Coutg Now we move o to a ew topc. Defto 1.1 (Surecto. A fucto f : A B s sad to be surectve or oto f for each b B there s some a A so that f(a B. What are examples of
More informationQualifying Exam Statistical Theory Problem Solutions August 2005
Qualfyg Exam Statstcal Theory Problem Solutos August 5. Let X, X,..., X be d uform U(,),
More informationSpecial Instructions / Useful Data
JAM 6 Set of all real umbers P A..d. B, p Posso Specal Istructos / Useful Data x,, :,,, x x Probablty of a evet A Idepedetly ad detcally dstrbuted Bomal dstrbuto wth parameters ad p Posso dstrbuto wth
More information22 Nonparametric Methods.
22 oparametrc Methods. I parametrc models oe assumes apror that the dstrbutos have a specfc form wth oe or more ukow parameters ad oe tres to fd the best or atleast reasoably effcet procedures that aswer
More informationSTK3100 and STK4100 Autumn 2018
SK3 ad SK4 Autum 8 Geeralzed lear models Part III Covers the followg materal from chaters 4 ad 5: Cofdece tervals by vertg tests Cosder a model wth a sgle arameter β We may obta a ( α% cofdece terval for
More informationModule 7. Lecture 7: Statistical parameter estimation
Lecture 7: Statstcal parameter estmato Parameter Estmato Methods of Parameter Estmato 1) Method of Matchg Pots ) Method of Momets 3) Mamum Lkelhood method Populato Parameter Sample Parameter Ubased estmato
More informationChain Rules for Entropy
Cha Rules for Etroy The etroy of a collecto of radom varables s the sum of codtoal etroes. Theorem: Let be radom varables havg the mass robablty x x.x. The...... The roof s obtaed by reeatg the alcato
More informationFunctions of Random Variables
Fuctos of Radom Varables Chapter Fve Fuctos of Radom Varables 5. Itroducto A geeral egeerg aalyss model s show Fg. 5.. The model output (respose) cotas the performaces of a system or product, such as weght,
More informationON BIVARIATE GEOMETRIC DISTRIBUTION. K. Jayakumar, D.A. Mundassery 1. INTRODUCTION
STATISTICA, ao LXVII, 4, 007 O BIVARIATE GEOMETRIC DISTRIBUTIO ITRODUCTIO Probablty dstrbutos of radom sums of deedetly ad detcally dstrbuted radom varables are maly aled modelg ractcal roblems that deal
More informationParameter, Statistic and Random Samples
Parameter, Statstc ad Radom Samples A parameter s a umber that descrbes the populato. It s a fxed umber, but practce we do ot kow ts value. A statstc s a fucto of the sample data,.e., t s a quatty whose
More informationChapter 8: Statistical Analysis of Simulated Data
Marquette Uversty MSCS600 Chapter 8: Statstcal Aalyss of Smulated Data Dael B. Rowe, Ph.D. Departmet of Mathematcs, Statstcs, ad Computer Scece Copyrght 08 by Marquette Uversty MSCS600 Ageda 8. The Sample
More informationTHE ROYAL STATISTICAL SOCIETY GRADUATE DIPLOMA
THE ROYAL STATISTICAL SOCIETY EXAMINATIONS SOLUTIONS GRADUATE DIPLOMA PAPER II STATISTICAL THEORY & METHODS The Socety provdes these solutos to assst caddates preparg for the examatos future years ad for
More informationIS 709/809: Computational Methods in IS Research. Simple Markovian Queueing Model
IS 79/89: Comutatoal Methods IS Research Smle Marova Queueg Model Nrmalya Roy Deartmet of Iformato Systems Uversty of Marylad Baltmore Couty www.umbc.edu Queueg Theory Software QtsPlus software The software
More informationDimensionality Reduction and Learning
CMSC 35900 (Sprg 009) Large Scale Learg Lecture: 3 Dmesoalty Reducto ad Learg Istructors: Sham Kakade ad Greg Shakharovch L Supervsed Methods ad Dmesoalty Reducto The theme of these two lectures s that
More informationChapter 4 Multiple Random Variables
Revew o BST 63: Statstcal Theory I Ku Zhag, /0/008 Revew for Chapter 4-5 Notes: Although all deftos ad theorems troduced our lectures ad ths ote are mportat ad you should be famlar wth, but I put those
More informationLECTURE - 4 SIMPLE RANDOM SAMPLING DR. SHALABH DEPARTMENT OF MATHEMATICS AND STATISTICS INDIAN INSTITUTE OF TECHNOLOGY KANPUR
amplg Theory MODULE II LECTURE - 4 IMPLE RADOM AMPLIG DR. HALABH DEPARTMET OF MATHEMATIC AD TATITIC IDIA ITITUTE OF TECHOLOGY KAPUR Estmato of populato mea ad populato varace Oe of the ma objectves after
More informationChapter -2 Simple Random Sampling
Chapter - Smple Radom Samplg Smple radom samplg (SRS) s a method of selecto of a sample comprsg of umber of samplg uts out of the populato havg umber of samplg uts such that every samplg ut has a equal
More informationLaw of Large Numbers
Toss a co tmes. Law of Large Numbers Suppose 0 f f th th toss came up H toss came up T s are Beroull radom varables wth p ½ ad E( ) ½. The proporto of heads s. Itutvely approaches ½ as. week 2 Markov s
More informationChapter -2 Simple Random Sampling
Chapter - Smple Radom Samplg Smple radom samplg (SRS) s a method of selecto of a sample comprsg of umber of samplg uts out of the populato havg umber of samplg uts such that every samplg ut has a equal
More informationTHE ROYAL STATISTICAL SOCIETY GRADUATE DIPLOMA
THE ROYAL STATISTICAL SOCIETY 3 EXAMINATIONS SOLUTIONS GRADUATE DIPLOMA PAPER I STATISTICAL THEORY & METHODS The Socety provdes these solutos to assst caddates preparg for the examatos future years ad
More information8.1 Hashing Algorithms
CS787: Advaced Algorthms Scrbe: Mayak Maheshwar, Chrs Hrchs Lecturer: Shuch Chawla Topc: Hashg ad NP-Completeess Date: September 21 2007 Prevously we looked at applcatos of radomzed algorthms, ad bega
More informationRecall MLR 5 Homskedasticity error u has the same variance given any values of the explanatory variables Var(u x1,...,xk) = 2 or E(UU ) = 2 I
Chapter 8 Heterosedastcty Recall MLR 5 Homsedastcty error u has the same varace gve ay values of the eplaatory varables Varu,..., = or EUU = I Suppose other GM assumptos hold but have heterosedastcty.
More informationChapter 8. Inferences about More Than Two Population Central Values
Chapter 8. Ifereces about More Tha Two Populato Cetral Values Case tudy: Effect of Tmg of the Treatmet of Port-We tas wth Lasers ) To vestgate whether treatmet at a youg age would yeld better results tha
More informationClass 13,14 June 17, 19, 2015
Class 3,4 Jue 7, 9, 05 Pla for Class3,4:. Samplg dstrbuto of sample mea. The Cetral Lmt Theorem (CLT). Cofdece terval for ukow mea.. Samplg Dstrbuto for Sample mea. Methods used are based o CLT ( Cetral
More informationQuantitative analysis requires : sound knowledge of chemistry : possibility of interferences WHY do we need to use STATISTICS in Anal. Chem.?
Ch 4. Statstcs 4.1 Quattatve aalyss requres : soud kowledge of chemstry : possblty of terfereces WHY do we eed to use STATISTICS Aal. Chem.? ucertaty ests. wll we accept ucertaty always? f ot, from how
More informationSTATISTICAL INFERENCE
(STATISTICS) STATISTICAL INFERENCE COMPLEMENTARY COURSE B.Sc. MATHEMATICS III SEMESTER ( Admsso) UNIVERSITY OF CALICUT SCHOOL OF DISTANCE EDUCATION CALICUT UNIVERSITY P.O., MALAPPURAM, KERALA, INDIA -
More informationPart I: Background on the Binomial Distribution
Part I: Bacgroud o the Bomal Dstrbuto A radom varable s sad to have a Beroull dstrbuto f t taes o the value wth probablt "p" ad the value wth probablt " - p". The umber of "successes" "" depedet Beroull
More informationIntroduction to local (nonparametric) density estimation. methods
Itroducto to local (oparametrc) desty estmato methods A slecture by Yu Lu for ECE 66 Sprg 014 1. Itroducto Ths slecture troduces two local desty estmato methods whch are Parze desty estmato ad k-earest
More informationSet Theory and Probability
Set Theory ad Probablty Dr. Bob Baley Set Theory We beg wth a toc of dscusso. A set ca be cosdered to be ay collecto of zero or more obects or ettes covered by the toc of dscusso. Examles: hadwrtte umerals
More informationLogistic regression (continued)
STAT562 page 138 Logstc regresso (cotued) Suppose we ow cosder more complex models to descrbe the relatoshp betwee a categorcal respose varable (Y) that takes o two (2) possble outcomes ad a set of p explaatory
More informationMedian as a Weighted Arithmetic Mean of All Sample Observations
Meda as a Weghted Arthmetc Mea of All Sample Observatos SK Mshra Dept. of Ecoomcs NEHU, Shllog (Ida). Itroducto: Iumerably may textbooks Statstcs explctly meto that oe of the weakesses (or propertes) of
More informationModule 7: Probability and Statistics
Lecture 4: Goodess of ft tests. Itroducto Module 7: Probablty ad Statstcs I the prevous two lectures, the cocepts, steps ad applcatos of Hypotheses testg were dscussed. Hypotheses testg may be used to
More informationd dt d d dt dt Also recall that by Taylor series, / 2 (enables use of sin instead of cos-see p.27 of A&F) dsin
Learzato of the Swg Equato We wll cover sectos.5.-.6 ad begg of Secto 3.3 these otes. 1. Sgle mache-fte bus case Cosder a sgle mache coected to a fte bus, as show Fg. 1 below. E y1 V=1./_ Fg. 1 The admttace
More informationChapter 4 (Part 1): Non-Parametric Classification (Sections ) Pattern Classification 4.3) Announcements
Aoucemets No-Parametrc Desty Estmato Techques HW assged Most of ths lecture was o the blacboard. These sldes cover the same materal as preseted DHS Bometrcs CSE 90-a Lecture 7 CSE90a Fall 06 CSE90a Fall
More informationChapter 2 Supplemental Text Material
-. Models for the Data ad the t-test Chapter upplemetal Text Materal The model preseted the text, equato (-3) s more properl called a meas model. ce the mea s a locato parameter, ths tpe of model s also
More informationSTA 108 Applied Linear Models: Regression Analysis Spring Solution for Homework #1
STA 08 Appled Lear Models: Regresso Aalyss Sprg 0 Soluto for Homework #. Let Y the dollar cost per year, X the umber of vsts per year. The the mathematcal relato betwee X ad Y s: Y 300 + X. Ths s a fuctoal
More informationTraining Sample Model: Given n observations, [[( Yi, x i the sample model can be expressed as (1) where, zero and variance σ
Stat 74 Estmato for Geeral Lear Model Prof. Goel Broad Outle Geeral Lear Model (GLM): Trag Samle Model: Gve observatos, [[( Y, x ), x = ( x,, xr )], =,,, the samle model ca be exressed as Y = µ ( x, x,,
More informationThe expected value of a sum of random variables,, is the sum of the expected values:
Sums of Radom Varables xpected Values ad Varaces of Sums ad Averages of Radom Varables The expected value of a sum of radom varables, say S, s the sum of the expected values: ( ) ( ) S Ths s always true
More information7. Joint Distributions
7. Jot Dstrbutos Chrs Pech ad Mehra Saham Ma 2017 Ofte ou wll work o problems where there are several radom varables (ofte teractg wth oe aother. We are gog to start to formall look at how those teractos
More information