Bayesian Belief Network
|
|
- Erica West
- 5 years ago
- Views:
Transcription
1 Bayesia Belief Network Ucertaity & Probability Baye's rule Choosig Hypotheses- Maximum a posteriori Maximum Likelihood - Baye's cocept learig Maximum Likelihood of real valued fuctio Bayes optimal Classifier Joit distributios Naive Bayes Classifier 1
2 Ucertaity Our mai tool is the probability theory, which assigs to each setece umerical degree of belief betwee 0 ad 1 It provides a way of summarizig the ucertaity Variables Boolea radom variables: cavity might be true or false Discrete radom variables: weather might be suy, raiy, cloudy, sow Weather=suy) Weather=raiy) Weather=cloudy) Weather=sow) Cotiuous radom variables: the temperature has cotiuous values 2
3 Where do probabilities come from? Frequets: From experimets: form ay fiite sample, we ca estimate the true fractio ad also calculate how accurate our estimatio is likely to be Subjective: Aget s believe Objectivist: True ature of the uiverse, that the probability up heads with probability 0.5 is a probability of the coi Before the evidece is obtaied; prior probability a) the prior probability that the propositio is true cavity)=0.1 After the evidece is obtaied; posterior probability a b) The probability of a give that all we kow is b cavity toothache)=0.8 3
4 Axioms of Probability (Kolmogorov s axioms, first published i Germa 1933) All probabilities are betwee 0 ad 1. For ay propositio a 0 a) 1 true)=1, false)=0 The probability of disjuctio is give by a b) = a) + b) a b) Product rule a b) = a b) b) a b) = b a) a) 4
5 Theorem of total probability If evets A 1,..., A are mutually exclusive with the Bayes s rule (Reveret Thomas Bayes ) He set dow his fidigs o probability i "Essay Towards Solvig a Problem i the Doctrie of Chaces" (1763), published posthumously i the Philosophical Trasactios of the Royal Society of Lodo b a) = a b)b) a) 5
6 6 Diagosis What is the probability of meigitis i the patiet with stiff eck? A doctor kows that the disease meigitis causes the patiet to have a stiff eck i 50% of the time -> s m) Prior Probabilities: That the patiet has meigitis is 1/ > m) That the patiet has a stiff eck is 1/20 -> s) m s) = 0.5 * = m s) = s m)m) s) Normalizatio ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( x P y P y x P x y P x P y P y x P x y P = = 0.6, ,0.08 ) ( ), ( ) ( ) ( ) ( ) ( ) ( 1 = = + = α α α x y P x y P Y P Y X P X Y P x y P x y P
7 Bayes Theorem h) = prior probability of hypothesis h D) = prior probability of traiig data D h D) = probability of h give D D h) = probability of D give h Choosig Hypotheses Geerally wat the most probable hypothesis give the traiig data Maximum a posteriori hypothesis h MAP : 7
8 If assume h i )=h j ) for all h i ad h j, the ca further simplify, ad choose the Maximum likelihood (ML) hypothesis 8
9 Example Does patiet have cacer or ot? A patiet takes a lab test ad the result comes back positive. The test returs a correct positive result (+) i oly 98% of the cases i which the disease is actually preset, ad a correct egative result (-) i oly 97% of the cases i which the disease is ot preset Furthermore, of the etire populatio have this cacer Suppose a positive result (+) is retured... 9
10 Normalizatio cacer +) = = cacer +) = = The result of Bayesia iferece depeds strogly o the prior probabilities, which must be available i order to apply the method Joit distributio A joit distributio for toothache, cavity, catch, detist s probe catches i my tooth :-( We eed to kow the coditioal probabilities of the cojuctio of toothache ad cavity What ca a detist coclude if the probe catches i the achig tooth? toothache catch cavity)cavity) cavity toothache catch) = toothache cavity) For possible variables there are 2 possible combiatios 10
11 Coditioal Idepedece Oce we kow that the patiet has cavity we do ot expect the probability of the probe catchig to deped o the presece of toothache catch cavity toothache) = catch cavity) toothache cavity catch) = toothache cavity) Idepedece betwee a ad b a b) = a) b a) = b) a b) = a) b) toothache, catch, cavity, Weather = cloudy) = = Weather = cloudy) toothache, catch, cavity) The decompositio of large probabilistic domais ito weakly coected subsets via coditioal idepedece is oe of the most importat developmets i the recet history of AI This ca work well, eve the assumptio is ot true! 11
12 A sigle cause directly ifluece a umber of effects, all of which are coditioally idepedet cause, effect1, effect2,... effect) = cause) effecti cause) i= 1 Naive Bayes Classifier Assume target fuctio f: X è V, where each istace x described by attributes a 1, a 2.. a Most probable value of f(x) is: 12
13 v NB Naive Bayes assumptio: which gives Naive Bayes Algorithm For each target value v j ç estimate v j ) For each attribute value a i of each attribute a ç estimate a i v j ) 13
14 Traiig dataset Class: C1:buys_computer= yes C2:buys_computer= o Data sample: X = (age<=30, Icome=medium, Studet=yes Credit_ratig=Fair) age icome studet credit_ratig buys_computer <=30 high o fair o <=30 high o excellet o high o fair yes >40 medium o fair yes >40 low yes fair yes >40 low yes excellet o low yes excellet yes <=30 medium o fair o <=30 low yes fair yes >40 medium yes fair yes <=30 medium yes excellet yes medium o excellet yes high yes fair yes >40 medium o excellet o Naïve Bayesia Classifier: Example Compute X C i ) for each class age= <30 buys_computer= yes ) = 2/9=0.222 age= <30 buys_computer= o ) = 3/5 =0.6 icome= medium buys_computer= yes )= 4/9 =0.444 icome= medium buys_computer= o ) = 2/5 = 0.4 studet= yes buys_computer= yes)= 6/9 =0.667 studet= yes buys_computer= o )= 1/5=0.2 credit_ratig= fair buys_computer= yes )=6/9=0.667 credit_ratig= fair buys_computer= o )=2/5=0.4 buys_computer= yes )=9/ 14 buys_computer= o )=5/ 14 X=(age<=30,icome =medium, studet=yes,credit_ratig=fair) X C i ) : X buys_computer= yes )= x x x =0.044 X buys_computer= o )= 0.6 x 0.4 x 0.2 x 0.4 =0.019 X C i )*C i ) : X buys_computer= yes ) * buys_computer= yes )=0.028 X buys_computer= o ) * buys_computer= o )=0.007 X belogs to class buys_computer=yes 14
15 Coditioal idepedece assumptio is ofte violated...but it works surprisigly well ayway Naive Bayes assumptio of coditioal idepedece too restrictive But it's itractable without some such assumptios... Bayesia Belief etworks describe coditioal idepedece amog subsets of variables allows combiig prior kowledge about (i)depedecies amog variables with observed traiig data 15
16 Bayesia etworks A simple, graphical otatio for coditioal idepedece assertios ad hece for compact specificatio of full joit distributios Sytax: a set of odes, oe per variable a directed, acyclic graph (lik "directly iflueces") a coditioal distributio for each ode give its parets: P (X i Parets (X i )) I the simplest case, coditioal distributio represeted as a coditioal probability table (CPT) givig the distributio over X i for each combiatio of paret values Bayesia Networks Bayesia belief etwork allows a subset of the variables coditioally idepedet A graphical model of causal relatioships Represets depedecy amog the variables Gives a specificatio of joit probability distributio X Z Y P q Nodes: radom variables q Liks: depedecy q X,Y are the parets of Z, ad Y is the paret of P q No depedecy betwee Z ad P q Has o loops or cycles 16
17 Coditioal Idepedece Oce we kow that the patiet has cavity we do ot expect the probability of the probe catchig to deped o the presece of toothache catch cavity toothache) = catch cavity) toothache cavity catch) = toothache cavity) Idepedece betwee a ad b a b) = a) b a) = b) Example Topology of etwork ecodes coditioal idepedece assertios: Weather is idepedet of the other variables Toothache ad Catch are coditioally idepedet give Cavity 17
18 Bayesia Belief Network: A Example Family History Smoker (FH, S) (FH, ~S) (~FH, S) (~FH, ~S) LC LugCacer Emphysema ~LC PositiveXRay Dyspea Bayesia Belief Networks The coditioal probability table for the variable LugCacer: Shows the coditioal probability for each possible combiatio of its parets Example I'm at work, eighbor Joh calls to say my alarm is rigig, but eighbor Mary does't call. Sometimes it's set off by mior earthquakes. Is there a burglar? Variables: Burglary, Earthquake, Alarm, JohCalls, MaryCalls Network topology reflects "causal" kowledge: A burglar ca set the alarm off A earthquake ca set the alarm off The alarm ca cause Mary to call The alarm ca cause Joh to call 18
19 Belief Networks Burglary B) Earthquake E) Alarm Burg. Earth. A) t t.95 t f.94 f t.29 f f.001 JohCalls A J) t.90 f.05 MaryCalls A M) t.7 f.01 Full Joit Distributio x,..., x ) = x parets( X )) 1 i i i= 1 j m a b e) = j a) m a) a b e) b) e) = =
20 Compactess A CPT for Boolea X i with k Boolea parets has 2 k rows for the combiatios of paret values Each row requires oe umber p for X i = true (the umber for X i = false is just 1-p) If each variable has o more tha k parets, the complete etwork requires O( 2 k ) umbers I.e., grows liearly with, vs. O(2 ) for the full joit distributio For burglary et, = 10 umbers (vs = 31) Iferece i Bayesia Networks How ca oe ifer the (probabilities of) values of oe or more etwork variables, give observed values of others? Bayes et cotais all iformatio eeded for this iferece If oly oe variable with ukow value, easy to ifer it I geeral case, problem is NP hard 20
21 Example I the burglary etwork, we migth observe the evet i which JohCalls=true ad MarryCalls=true We could ask for the probability that the burglary has occurred Burglary JohCalls=ture,MarryCalls=true) Remember - Joit distributio cavity toothache) = = cavity toothache) = cavity toothache) toothache) = 0.6 = cavity toothache) toothache) =
22 Normalizatio 1 = y x) + y x) Y X ) = α X Y ) Y ) α y x), y x) α 0.12,0.08 = 0.6,0.4 Normalizatio Cavity toothache) = αcavity, toothache) = α[cavity, toothache,catch) + Cavity, toothache, catch)] = α[< 0.108,0.016 > + < 0.012,0.064 >] = α < 0.12,0.08 >=< 0.6,0.4 > X is the query variable E evidece variable Y remaiig uobservable variable X e) = αx,e) = α X,e, y) Summatio over all possible y (all possible values of the uobservable variables Y) y 22
23 Burglary JohCalls=ture,MarryCalls=true) The hidde variables of the query are Earthquake ad Alarm B j,m) = αb, j,m) = α e B,e,a, j,m) For Burglary=true i the Bayesai etwork a b j,m) = α b)e)a b,e) j a)m a) e a To compute we had to add four terms, each computed by multiplyig five umbers I the worst case, where we have to sum out almost all variables, the complexity of the etwork with Boolea variables is O(2 ) 23
24 b) is costat ad ca be moved out, e) term ca be moved outside summatio a b j,m) = αb) e) a b,e) j a)m a) e a JohCalls=true ad MarryCalls=true, the probability that the burglary has occured is aboud 28% B, j,m) = α < , > < 0.284,0.716 > Computatio for Burglary=true 24
25 Variable elimiatio algorithm Elimiate repeated calculatio Dyamic programmig Irrelevat variables (X query variable, E evidece variables) 25
26 Complexity of exact iferece The burglary etwork belogs to a family of etworks i which there is at most oe udirected path betwee tow odes i the etwork These are called sigly coected etworks or polytrees The time ad space complexity of exact iferece i polytrees is liear i the size of etwork Size is defied by the umber of CPT etries If the umber of parets of each ode is bouded by a costat, the the complexity will be also liear i the umber of odes For multiply coected etworks variable elimiatio ca have expoetial time ad space complexity 26
27 Costructig Bayesia Networks A Bayesia etwork is a correct represetatio of the domai oly if each ode is coditioally idepedet of its predecessors i the orderig, give its parets MarryCalls JohCalls,Alarm,Eathquake,Bulgary)=MaryCalls Alarm) Coditioal Idepedece relatios i Bayesia etworks The topological sematics is give either of the specificatios of DESCENDANTS or MARKOV BLANKET 27
28 Local sematics Example JohCalls is idipedet of Burglary ad Earthquake give the value of Alarm 28
29 Example Burglary is idipedet of JohCalls ad MaryCalls give Alarm ad Earthquake 29
30 Costructig Bayesia etworks 1. Choose a orderig of variables X 1,,X 2. For i = 1 to add X i to the etwork select parets from X 1,,X i-1 such that P (X i Parets(X i )) = P (X i X 1,... X i-1 ) This choice of parets guaratees: P (X 1,,X ) = π i =1 P (X i X 1,, X i-1 ) (chai rule) = π i =1 P (X i Parets(X i )) (by costructio) The compactess of Bayesia etworks is a example of locally structured systems Each subcompoet iteracts directly with oly bouded umber of other compoets Costructig Bayesia etworks is difficult Each variable should be directly iflueced by oly a few others The etwork topology reflects thes direct iflueces 30
31 Example Suppose we choose the orderig M, J, A, B, E J M) = J)? Example Suppose we choose the orderig M, J, A, B, E J M) = J)? No A J, M) = A J)? A J, M) = A)? No B A, J, M) = B A)? B A, J, M) = B)? 31
32 Example Suppose we choose the orderig M, J, A, B, E J M) = J)? No A J, M) = A J)? A J, M) = A)? No B A, J, M) = B A)? Yes B A, J, M) = B)? No E B, A,J, M) = E A)? E B, A, J, M) = E A, B)? Example Suppose we choose the orderig M, J, A, B, E J M) = J)? No A J, M) = A J)? A J, M) = A)? No B A, J, M) = B A)? Yes B A, J, M) = B)? No E B, A,J, M) = E A)? No E B, A, J, M) = E A, B)? Yes 32
33 Example cotd. Decidig coditioal idepedece is hard i o causal directios (Causal models ad coditioal idepedece seem hardwired for humas!) Network is less compact: = 13 umbers eeded Some liks represet teuous relatioship that require difficult ad uatural probability judgmet, such the probability of Earthquake give Burglary ad Alarm 33
Bayesian Belief Network
Bayesia Belief Network a b) = a) b) toothache, catch, cavity, Weather = cloudy) = = Weather = cloudy) toothache, catch, cavity) The decompositio of large probabilistic domais ito weakly coected subsets
More informationUncertainty. Variables. assigns to each sentence numerical degree of belief between 0 and 1. uncertainty
Bayes Classificatio Ucertaity & robability Baye's rule Choosig Hypotheses- Maximum a posteriori Maximum Likelihood - Baye's cocept learig Maximum Likelihood of real valued fuctio Bayes optimal Classifier
More informationBayesian Belief Network
Bayesian Belief Network a! b) = a) b) toothache, catch, cavity, Weather = cloudy) = = Weather = cloudy) toothache, catch, cavity) The decomposition of large probabilistic domains into weakly connected
More informationUncertainty. Variables. assigns to each sentence numerical degree of belief between 0 and 1. uncertainty
Bayes Classification n Uncertainty & robability n Baye's rule n Choosing Hypotheses- Maximum a posteriori n Maximum Likelihood - Baye's concept learning n Maximum Likelihood of real valued function n Bayes
More informationMachine Learning 4771
Machie Learig 4771 Istructor: Toy Jebara Topic 14 Structurig Probability Fuctios for Storage Structurig Probability Fuctios for Iferece Basic Graphical Models Graphical Models Parameters as Nodes Structurig
More information15-780: Graduate Artificial Intelligence. Density estimation
5-780: Graduate Artificial Itelligece Desity estimatio Coditioal Probability Tables (CPT) But where do we get them? P(B)=.05 B P(E)=. E P(A B,E) )=.95 P(A B, E) =.85 P(A B,E) )=.5 P(A B, E) =.05 A P(J
More informationElementary manipulations of probabilities
Elemetary maipulatios of probabilities Set probability of multi-valued r.v. {=Odd} = +3+5 = /6+/6+/6 = ½ X X,, X i j X i j Multi-variat distributio: Joit probability: X true true X X,, X X i j i j X X
More informationOutline. L7: Probability Basics. Probability. Probability Theory. Bayes Law for Diagnosis. Which Hypothesis To Prefer? p(a,b) = p(b A) " p(a)
Outlie L7: Probability Basics CS 344R/393R: Robotics Bejami Kuipers. Bayes Law 2. Probability distributios 3. Decisios uder ucertaity Probability For a propositio A, the probability p(a is your degree
More informationAs stated by Laplace, Probability is common sense reduced to calculation.
Note: Hadouts DO NOT replace the book. I most cases, they oly provide a guidelie o topics ad a ituitive feel. The math details will be covered i class, so it is importat to atted class ad also you MUST
More informationCEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering
CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio
More informationProbability and MLE.
10-701 Probability ad MLE http://www.cs.cmu.edu/~pradeepr/701 (brief) itro to probability Basic otatios Radom variable - referrig to a elemet / evet whose status is ukow: A = it will rai tomorrow Domai
More informationRandom Variables, Sampling and Estimation
Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig
More informationMixtures of Gaussians and the EM Algorithm
Mixtures of Gaussias ad the EM Algorithm CSE 6363 Machie Learig Vassilis Athitsos Computer Sciece ad Egieerig Departmet Uiversity of Texas at Arligto 1 Gaussias A popular way to estimate probability desity
More informationChapter 6 Sampling Distributions
Chapter 6 Samplig Distributios 1 I most experimets, we have more tha oe measuremet for ay give variable, each measuremet beig associated with oe radomly selected a member of a populatio. Hece we eed to
More informationProperties and Hypothesis Testing
Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.
More informationEXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY
EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY GRADUATE DIPLOMA, 016 MODULE : Statistical Iferece Time allowed: Three hours Cadidates should aswer FIVE questios. All questios carry equal marks. The umber
More informationUncertainty and Belief Networks. Introduction to Artificial Intelligence CS 151 Lecture 1 continued Ok, Lecture 2!
Uncertainty and Belief Networks Introduction to Artificial Intelligence CS 151 Lecture 1 continued Ok, Lecture 2! This lecture Conditional Independence Bayesian (Belief) Networks: Syntax and semantics
More informationLecture 2. The Lovász Local Lemma
Staford Uiversity Sprig 208 Math 233A: No-costructive methods i combiatorics Istructor: Ja Vodrák Lecture date: Jauary 0, 208 Origial scribe: Apoorva Khare Lecture 2. The Lovász Local Lemma 2. Itroductio
More informationExponential Families and Bayesian Inference
Computer Visio Expoetial Families ad Bayesia Iferece Lecture Expoetial Families A expoetial family of distributios is a d-parameter family f(x; havig the followig form: f(x; = h(xe g(t T (x B(, (. where
More informationIt is always the case that unions, intersections, complements, and set differences are preserved by the inverse image of a function.
MATH 532 Measurable Fuctios Dr. Neal, WKU Throughout, let ( X, F, µ) be a measure space ad let (!, F, P ) deote the special case of a probability space. We shall ow begi to study real-valued fuctios defied
More informationExercises Advanced Data Mining: Solutions
Exercises Advaced Data Miig: Solutios Exercise 1 Cosider the followig directed idepedece graph. 5 8 9 a) Give the factorizatio of P (X 1, X 2,..., X 9 ) correspodig to this idepedece graph. P (X) = 9 P
More informationGoodness-of-Fit Tests and Categorical Data Analysis (Devore Chapter Fourteen)
Goodess-of-Fit Tests ad Categorical Data Aalysis (Devore Chapter Fourtee) MATH-252-01: Probability ad Statistics II Sprig 2019 Cotets 1 Chi-Squared Tests with Kow Probabilities 1 1.1 Chi-Squared Testig................
More informationBayesian networks are graphical models that characterize how variables are independent of each other.
Ali Tomescu, http://people.csail.mit.edu/~aliush 6.867 Machie learig Prof. Tommi Jaakkola Week 12, Tuesday, November 19th, 2013 Lecture 21 Lecture 21: Hidde Markov Models Fial exam: Eveig of December 10
More informationExpectation-Maximization Algorithm.
Expectatio-Maximizatio Algorithm. Petr Pošík Czech Techical Uiversity i Prague Faculty of Electrical Egieerig Dept. of Cyberetics MLE 2 Likelihood.........................................................................................................
More informationThis lecture. Reading. Conditional Independence Bayesian (Belief) Networks: Syntax and semantics. Chapter CS151, Spring 2004
This lecture Conditional Independence Bayesian (Belief) Networks: Syntax and semantics Reading Chapter 14.1-14.2 Propositions and Random Variables Letting A refer to a proposition which may either be true
More information10-701/ Machine Learning Mid-term Exam Solution
0-70/5-78 Machie Learig Mid-term Exam Solutio Your Name: Your Adrew ID: True or False (Give oe setece explaatio) (20%). (F) For a cotiuous radom variable x ad its probability distributio fuctio p(x), it
More informationChapter 8: Estimating with Confidence
Chapter 8: Estimatig with Cofidece Sectio 8.2 The Practice of Statistics, 4 th editio For AP* STARNES, YATES, MOORE Chapter 8 Estimatig with Cofidece 8.1 Cofidece Itervals: The Basics 8.2 8.3 Estimatig
More informationProblem Set 2 Solutions
CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S
More informationCS284A: Representations and Algorithms in Molecular Biology
CS284A: Represetatios ad Algorithms i Molecular Biology Scribe Notes o Lectures 3 & 4: Motif Discovery via Eumeratio & Motif Represetatio Usig Positio Weight Matrix Joshua Gervi Based o presetatios by
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More informationECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015
ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],
More informationStatistical Pattern Recognition
Statistical Patter Recogitio Classificatio: No-Parametric Modelig Hamid R. Rabiee Jafar Muhammadi Sprig 2014 http://ce.sharif.edu/courses/92-93/2/ce725-2/ Ageda Parametric Modelig No-Parametric Modelig
More informationA widely used display of protein shapes is based on the coordinates of the alpha carbons - - C α
Nice plottig of proteis: I A widely used display of protei shapes is based o the coordiates of the alpha carbos - - C α -s. The coordiates of the C α -s are coected by a cotiuous curve that roughly follows
More informationIntroduction to Artificial Intelligence CAP 4601 Summer 2013 Midterm Exam
Itroductio to Artificial Itelligece CAP 601 Summer 013 Midterm Exam 1. Termiology (7 Poits). Give the followig task eviromets, eter their properties/characteristics. The properties/characteristics of the
More informationThe Bayesian Learning Framework. Back to Maximum Likelihood. Naïve Bayes. Simple Example: Coin Tosses. Given a generative model
Back to Maximum Likelihood Give a geerative model f (x, y = k) =π k f k (x) Usig a geerative modellig approach, we assume a parametric form for f k (x) =f (x; k ) ad compute the MLE θ of θ =(π k, k ) k=
More informationProbability and statistics: basic terms
Probability ad statistics: basic terms M. Veeraraghava August 203 A radom variable is a rule that assigs a umerical value to each possible outcome of a experimet. Outcomes of a experimet form the sample
More informationCSE 527, Additional notes on MLE & EM
CSE 57 Lecture Notes: MLE & EM CSE 57, Additioal otes o MLE & EM Based o earlier otes by C. Grat & M. Narasimha Itroductio Last lecture we bega a examiatio of model based clusterig. This lecture will be
More informationCS 330 Discussion - Probability
CS 330 Discussio - Probability March 24 2017 1 Fudametals of Probability 11 Radom Variables ad Evets A radom variable X is oe whose value is o-determiistic For example, suppose we flip a coi ad set X =
More informationQuick Review of Probability
Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter & Teachig Material.
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationSTATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. Comments:
Recall: STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS Commets:. So far we have estimates of the parameters! 0 ad!, but have o idea how good these estimates are. Assumptio: E(Y x)! 0 +! x (liear coditioal
More informationChapter 22. Comparing Two Proportions. Copyright 2010, 2007, 2004 Pearson Education, Inc.
Chapter 22 Comparig Two Proportios Copyright 2010, 2007, 2004 Pearso Educatio, Ic. Comparig Two Proportios Read the first two paragraphs of pg 504. Comparisos betwee two percetages are much more commo
More informationChapter 6 Principles of Data Reduction
Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationn n i=1 Often we also need to estimate the variance. Below are three estimators each of which is optimal in some sense: n 1 i=1 k=1 i=1 k=1 i=1 k=1
MATH88T Maria Camero Cotets Basic cocepts of statistics Estimators, estimates ad samplig distributios 2 Ordiary least squares estimate 3 3 Maximum lielihood estimator 3 4 Bayesia estimatio Refereces 9
More informationDS 100: Principles and Techniques of Data Science Date: April 13, Discussion #10
DS 00: Priciples ad Techiques of Data Sciece Date: April 3, 208 Name: Hypothesis Testig Discussio #0. Defie these terms below as they relate to hypothesis testig. a) Data Geeratio Model: Solutio: A set
More informationBasics of Probability Theory (for Theory of Computation courses)
Basics of Probability Theory (for Theory of Computatio courses) Oded Goldreich Departmet of Computer Sciece Weizma Istitute of Sciece Rehovot, Israel. oded.goldreich@weizma.ac.il November 24, 2008 Preface.
More informationCS 2750 Machine Learning. Lecture 23. Concept learning. CS 2750 Machine Learning. Concept Learning
Lecture 3 Cocept learig Milos Hauskrecht milos@cs.pitt.edu Cocept Learig Outlie: Learig boolea fuctios Most geeral ad most specific cosistet hypothesis. Mitchell s versio space algorithm Probably approximately
More informationCS 2750 Machine Learning. Lecture 22. Concept learning. CS 2750 Machine Learning. Concept Learning
Lecture 22 Cocept learig Milos Hauskrecht milos@cs.pitt.edu 5329 Seott Square Cocept Learig Outlie: Learig boolea fuctios Most geeral ad most specific cosistet hypothesis. Mitchell s versio space algorithm
More informationPROBABILITY LOGIC: Part 2
James L Bec 2 July 2005 PROBABILITY LOGIC: Part 2 Axioms for Probability Logic Based o geeral cosideratios, we derived axioms for: Pb ( a ) = measure of the plausibility of propositio b coditioal o the
More informationApproximations and more PMFs and PDFs
Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.
More informationCS322: Network Analysis. Problem Set 2 - Fall 2009
Due October 9 009 i class CS3: Network Aalysis Problem Set - Fall 009 If you have ay questios regardig the problems set, sed a email to the course assistats: simlac@staford.edu ad peleato@staford.edu.
More informationBIOS 4110: Introduction to Biostatistics. Breheny. Lab #9
BIOS 4110: Itroductio to Biostatistics Brehey Lab #9 The Cetral Limit Theorem is very importat i the realm of statistics, ad today's lab will explore the applicatio of it i both categorical ad cotiuous
More information1.010 Uncertainty in Engineering Fall 2008
MIT OpeCourseWare http://ocw.mit.edu.00 Ucertaity i Egieerig Fall 2008 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu.terms. .00 - Brief Notes # 9 Poit ad Iterval
More information5. Likelihood Ratio Tests
1 of 5 7/29/2009 3:16 PM Virtual Laboratories > 9. Hy pothesis Testig > 1 2 3 4 5 6 7 5. Likelihood Ratio Tests Prelimiaries As usual, our startig poit is a radom experimet with a uderlyig sample space,
More informationQuick Review of Probability
Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter 2 & Teachig
More informationMath 61CM - Solutions to homework 3
Math 6CM - Solutios to homework 3 Cédric De Groote October 2 th, 208 Problem : Let F be a field, m 0 a fixed oegative iteger ad let V = {a 0 + a x + + a m x m a 0,, a m F} be the vector space cosistig
More informationLecture 1 Probability and Statistics
Wikipedia: Lecture 1 Probability ad Statistics Bejami Disraeli, British statesma ad literary figure (1804 1881): There are three kids of lies: lies, damed lies, ad statistics. popularized i US by Mark
More informationLecture 12: November 13, 2018
Mathematical Toolkit Autum 2018 Lecturer: Madhur Tulsiai Lecture 12: November 13, 2018 1 Radomized polyomial idetity testig We will use our kowledge of coditioal probability to prove the followig lemma,
More informationCHAPTER 5. Theory and Solution Using Matrix Techniques
A SERIES OF CLASS NOTES FOR 2005-2006 TO INTRODUCE LINEAR AND NONLINEAR PROBLEMS TO ENGINEERS, SCIENTISTS, AND APPLIED MATHEMATICIANS DE CLASS NOTES 3 A COLLECTION OF HANDOUTS ON SYSTEMS OF ORDINARY DIFFERENTIAL
More information11 Correlation and Regression
11 Correlatio Regressio 11.1 Multivariate Data Ofte we look at data where several variables are recorded for the same idividuals or samplig uits. For example, at a coastal weather statio, we might record
More informationEmpirical Distributions
Empirical Distributios A empirical distributio is oe for which each possible evet is assiged a probability derived from experimetal observatio. It is assumed that the evets are idepedet ad the sum of the
More informationProblem Set 4 Due Oct, 12
EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios
More informationAn Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More informationModule 1 Fundamentals in statistics
Normal Distributio Repeated observatios that differ because of experimetal error ofte vary about some cetral value i a roughly symmetrical distributio i which small deviatios occur much more frequetly
More informationStatistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample.
Statistical Iferece (Chapter 10) Statistical iferece = lear about a populatio based o the iformatio provided by a sample. Populatio: The set of all values of a radom variable X of iterest. Characterized
More informationSome special clique problems
Some special clique problems Reate Witer Istitut für Iformatik Marti-Luther-Uiversität Halle-Witteberg Vo-Seckedorff-Platz, D 0620 Halle Saale Germay Abstract: We cosider graphs with cliques of size k
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationREAL ANALYSIS II: PROBLEM SET 1 - SOLUTIONS
REAL ANALYSIS II: PROBLEM SET 1 - SOLUTIONS 18th Feb, 016 Defiitio (Lipschitz fuctio). A fuctio f : R R is said to be Lipschitz if there exists a positive real umber c such that for ay x, y i the domai
More informationCS161 Handout 05 Summer 2013 July 10, 2013 Mathematical Terms and Identities
CS161 Hadout 05 Summer 2013 July 10, 2013 Mathematical Terms ad Idetities Thaks to Ady Nguye ad Julie Tibshirai for their advice o this hadout. This hadout covers mathematical otatio ad idetities that
More informationStatistics 511 Additional Materials
Cofidece Itervals o mu Statistics 511 Additioal Materials This topic officially moves us from probability to statistics. We begi to discuss makig ifereces about the populatio. Oe way to differetiate probability
More informationMATH 1910 Workshop Solution
MATH 90 Workshop Solutio Fractals Itroductio: Fractals are atural pheomea or mathematical sets which exhibit (amog other properties) self similarity: o matter how much we zoom i, the structure remais the
More informationCS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5
CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio
More informationLecture 2: April 3, 2013
TTIC/CMSC 350 Mathematical Toolkit Sprig 203 Madhur Tulsiai Lecture 2: April 3, 203 Scribe: Shubhedu Trivedi Coi tosses cotiued We retur to the coi tossig example from the last lecture agai: Example. Give,
More informationMOST PEOPLE WOULD RATHER LIVE WITH A PROBLEM THEY CAN'T SOLVE, THAN ACCEPT A SOLUTION THEY CAN'T UNDERSTAND.
XI-1 (1074) MOST PEOPLE WOULD RATHER LIVE WITH A PROBLEM THEY CAN'T SOLVE, THAN ACCEPT A SOLUTION THEY CAN'T UNDERSTAND. R. E. D. WOOLSEY AND H. S. SWANSON XI-2 (1075) STATISTICAL DECISION MAKING Advaced
More informationFrequentist Inference
Frequetist Iferece The topics of the ext three sectios are useful applicatios of the Cetral Limit Theorem. Without kowig aythig about the uderlyig distributio of a sequece of radom variables {X i }, for
More informationAchieving Stationary Distributions in Markov Chains. Monday, November 17, 2008 Rice University
Istructor: Achievig Statioary Distributios i Markov Chais Moday, November 1, 008 Rice Uiversity Dr. Volka Cevher STAT 1 / ELEC 9: Graphical Models Scribe: Rya E. Guerra, Tahira N. Saleem, Terrace D. Savitsky
More informationMachine Learning Brett Bernstein
Machie Learig Brett Berstei Week Lecture: Cocept Check Exercises Starred problems are optioal. Statistical Learig Theory. Suppose A = Y = R ad X is some other set. Furthermore, assume P X Y is a discrete
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationGeneralized Semi- Markov Processes (GSMP)
Geeralized Semi- Markov Processes (GSMP) Summary Some Defiitios Markov ad Semi-Markov Processes The Poisso Process Properties of the Poisso Process Iterarrival times Memoryless property ad the residual
More informationA PROBABILITY PRIMER
CARLETON COLLEGE A ROBABILITY RIMER SCOTT BIERMAN (Do ot quote without permissio) A robability rimer INTRODUCTION The field of probability ad statistics provides a orgaizig framework for systematically
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationProbabilistic Reasoning Systems
Probabilistic Reasoning Systems Dr. Richard J. Povinelli Copyright Richard J. Povinelli rev 1.0, 10/7/2001 Page 1 Objectives You should be able to apply belief networks to model a problem with uncertainty.
More informationStat 421-SP2012 Interval Estimation Section
Stat 41-SP01 Iterval Estimatio Sectio 11.1-11. We ow uderstad (Chapter 10) how to fid poit estimators of a ukow parameter. o However, a poit estimate does ot provide ay iformatio about the ucertaity (possible
More informationSNAP Centre Workshop. Basic Algebraic Manipulation
SNAP Cetre Workshop Basic Algebraic Maipulatio 8 Simplifyig Algebraic Expressios Whe a expressio is writte i the most compact maer possible, it is cosidered to be simplified. Not Simplified: x(x + 4x)
More informationA statistical method to determine sample size to estimate characteristic value of soil parameters
A statistical method to determie sample size to estimate characteristic value of soil parameters Y. Hojo, B. Setiawa 2 ad M. Suzuki 3 Abstract Sample size is a importat factor to be cosidered i determiig
More informationSequences, Mathematical Induction, and Recursion. CSE 2353 Discrete Computational Structures Spring 2018
CSE 353 Discrete Computatioal Structures Sprig 08 Sequeces, Mathematical Iductio, ad Recursio (Chapter 5, Epp) Note: some course slides adopted from publisher-provided material Overview May mathematical
More informationOutline. CSCI-567: Machine Learning (Spring 2019) Outline. Prof. Victor Adamchik. Mar. 26, 2019
Outlie CSCI-567: Machie Learig Sprig 209 Gaussia mixture models Prof. Victor Adamchik 2 Desity estimatio U of Souther Califoria Mar. 26, 209 3 Naive Bayes Revisited March 26, 209 / 57 March 26, 209 2 /
More informationLecture 9: Hierarchy Theorems
IAS/PCMI Summer Sessio 2000 Clay Mathematics Udergraduate Program Basic Course o Computatioal Complexity Lecture 9: Hierarchy Theorems David Mix Barrigto ad Alexis Maciel July 27, 2000 Most of this lecture
More informationClass 23. Daniel B. Rowe, Ph.D. Department of Mathematics, Statistics, and Computer Science. Marquette University MATH 1700
Class 23 Daiel B. Rowe, Ph.D. Departmet of Mathematics, Statistics, ad Computer Sciece Copyright 2017 by D.B. Rowe 1 Ageda: Recap Chapter 9.1 Lecture Chapter 9.2 Review Exam 6 Problem Solvig Sessio. 2
More informationRandomized Algorithms I, Spring 2018, Department of Computer Science, University of Helsinki Homework 1: Solutions (Discussed January 25, 2018)
Radomized Algorithms I, Sprig 08, Departmet of Computer Sciece, Uiversity of Helsiki Homework : Solutios Discussed Jauary 5, 08). Exercise.: Cosider the followig balls-ad-bi game. We start with oe black
More informationPattern Classification, Ch4 (Part 1)
Patter Classificatio All materials i these slides were take from Patter Classificatio (2d ed) by R O Duda, P E Hart ad D G Stork, Joh Wiley & Sos, 2000 with the permissio of the authors ad the publisher
More informationA quick activity - Central Limit Theorem and Proportions. Lecture 21: Testing Proportions. Results from the GSS. Statistics and the General Population
A quick activity - Cetral Limit Theorem ad Proportios Lecture 21: Testig Proportios Statistics 10 Coli Rudel Flip a coi 30 times this is goig to get loud! Record the umber of heads you obtaied ad calculate
More informationIntroductory statistics
CM9S: Machie Learig for Bioiformatics Lecture - 03/3/06 Itroductory statistics Lecturer: Sriram Sakararama Scribe: Sriram Sakararama We will provide a overview of statistical iferece focussig o the key
More informationChapter 22. Comparing Two Proportions. Copyright 2010 Pearson Education, Inc.
Chapter 22 Comparig Two Proportios Copyright 2010 Pearso Educatio, Ic. Comparig Two Proportios Comparisos betwee two percetages are much more commo tha questios about isolated percetages. Ad they are more
More informationWhat is Probability?
Quatificatio of ucertaity. What is Probability? Mathematical model for thigs that occur radomly. Radom ot haphazard, do t kow what will happe o ay oe experimet, but has a log ru order. The cocept of probability
More information6 Integers Modulo n. integer k can be written as k = qn + r, with q,r, 0 r b. So any integer.
6 Itegers Modulo I Example 2.3(e), we have defied the cogruece of two itegers a,b with respect to a modulus. Let us recall that a b (mod ) meas a b. We have proved that cogruece is a equivalece relatio
More informationObjectives. Probabilistic Reasoning Systems. Outline. Independence. Conditional independence. Conditional independence II.
Copyright Richard J. Povinelli rev 1.0, 10/1//2001 Page 1 Probabilistic Reasoning Systems Dr. Richard J. Povinelli Objectives You should be able to apply belief networks to model a problem with uncertainty.
More informationBHW #13 1/ Cooper. ENGR 323 Probabilistic Analysis Beautiful Homework # 13
BHW # /5 ENGR Probabilistic Aalysis Beautiful Homework # Three differet roads feed ito a particular freeway etrace. Suppose that durig a fixed time period, the umber of cars comig from each road oto the
More informationSieve Estimators: Consistency and Rates of Convergence
EECS 598: Statistical Learig Theory, Witer 2014 Topic 6 Sieve Estimators: Cosistecy ad Rates of Covergece Lecturer: Clayto Scott Scribe: Julia Katz-Samuels, Brado Oselio, Pi-Yu Che Disclaimer: These otes
More information