Bayesian Networks. Distinguished Prof. Dr. Panos M. Pardalos
|
|
- Gregory Kelley
- 6 years ago
- Views:
Transcription
1 Distinguished Prof. Dr. Panos M. Pardalos Center for Applied Optimization Department of Industrial & Systems Engineering Computer & Information Science & Engineering Department Biomedical Engineering Program, McKnight Brain Institute University of Florida
2 Lecture Outline Introduction Probability Space Conditional Probability Bayes Rule Bayesian Approach Example Graph Theory Concepts Definition Inferencing
3 Introduction are applied in cases of uncertainty when we know certain [conditional] probabilities and are looking for unknown probabilities given specific conditions. Applications: bioinformatics and medicine, engineering, document classification, image processing, data fusion, and decision support systems, etc Examples: Inference: P(Diagnosis Symptom) Anomaly detection: Is this observation anomalous? Active Data Collection: What is the next diagnostic test given a set of observations?
4 Discrete Random Variables Probability Space Conditional Probability Bayes Rule Let A denote a boolean-valued random variable If A denotes an event, and there is some degree of uncertainty as to whether A occurs. Examples A = Patient has Tuberculosis A = Coin flipping outcome is Head A = France will win World Cup in 2010
5 Probability Space Conditional Probability Bayes Rule Intuition Behind Probability Intuitively probability of event A equals to the proportion of the outcomes where A is true Ω is the set of all possible outcomes. Its area is P(Ω) = 1 The set colored in orange corresponds to the outcomes where A is true P(A) = Area of orange oval. Clearly 0 P(A) 1.
6 Kolmogorov s Probability Axioms Probability Space Conditional Probability Bayes Rule The theory of probability as a mathematical discipline can and should be developed from axioms in exactly the same way as geometry and algebra. Andrey Nikolaevich Kolmogorov. Foundations of the Theory of Probability, P(A) 0, A Ω 2. P(Ω) = 1 3. σ-additivity: Any countable sequence of pairwise disjoint events A 1, A 2,... satisfies ( ) P A i = P(A i ) i i
7 Probability Space Conditional Probability Bayes Rule Other Ways to Deal with Uncertainty Three-valued logic: True / False / Maybe Fuzzy logic (truth values between 0 and 1) Non-monotonic reasoning (especially focused on Penguin informatics) Dempster-Shafer theory (and an extension known as quasi-bayesian theory) Possibabilistic Logic But...
8 Coherence of the Axioms Probability Space Conditional Probability Bayes Rule The Kolmogorov s axioms of probability are the only model with this property: Wagers (probabilities) are assigned in such a way that no matter what set of wagers your opponent chooses you are not exposed to certain loss Bruno de Finetti, Probabilismo, Napoli, Logos 14, 1931, pp Bruno de Finetti. Probabilism: A Critical Essay on the Theory of Probability and on the Value of Science, (translation of 1931 article) in Erkenntnis, volume 31, September 1989, pp
9 Probability Space Conditional Probability Bayes Rule Consequences of the Axioms P(Ā) = 1 P(A), where Ā = Ω\A P( ) = 0
10 Probability Space Conditional Probability Bayes Rule Consequences of the Axioms P(A B) = P(A) + P(B) P(A B) P(A) = P(A B) + P(A B)
11 Conditional Probability Probability Space Conditional Probability Bayes Rule P(A B) = Proportion of the space in which A is true that also have B true Formal definition: P(B A) = P(A B) P(A)
12 Conditional Probability: Example Probability Space Conditional Probability Bayes Rule Let us draw a card from the deck of 52 playing cards. A =the card is a court card. P(A) = 12/52 = 3/13 B =the card is a queen. P(B) = 4/52 = 1/13, P(B A) = P(B) = 1/13 If we apply the definition we obtain very intuitive result: P(B A) = 1/13 3/13 = 1/3 P(A B) = 1/13 1/13 = 1 C =the suit is spade. P(C) = 1/4 Note that P(C A) = P(C) = 1/4. In other words event C is independent from A.
13 Probability Space Conditional Probability Bayes Rule Independent Events Definition Two events A and B are independent if and only if P(A B) = P(A)Pr(B). Let us denote independence of A and B as I (A, B). The independence of A and B implies P(A B) = P(A), if P(B) 0 P(B A) = P(B), if P(A) 0 Why?
14 Probability Space Conditional Probability Bayes Rule Conditional Independence One might observe that people of longer arms tend to have higher levels of reading skills If the age is fixed then this relationship disappears Arm length and reading skills are conditionally independent given the age Definition Two events A and B are conditionally independent given C if and only if P(A B C) = P(A C)Pr(B C). Notation: I (A, B C). P(A B, C) = P(A C), if P(B C) 0 P(B A, C) = P(B C), if P(A C) 0
15 Probability Space Conditional Probability Bayes Rule Bayes Rule The definition of conditional probability P(A B) = P(A B) P(B) implies the chain rule: P(A B) = P(A B)P(B). By symmetry P(A B) = P(B A)P(A) After we equate right hand sides and do some algebra we obtain Bayes Rule P(B A) = P(A B)P(B) P(A)
16 Probability Space Conditional Probability Bayes Rule Monty Hall Problem The treasure is equally probable contained in one of the boxes A, B and C, i.e. P(A) = P(B) = P(C). You are offered to chose one of them. Let us say you choose box A. Then the host of the game opens the box which you did not chose and does not contain the treasure
17 Monty Hall Problem Introduction Probability Space Conditional Probability Bayes Rule For instance, the host has opened box C Then you are offered an option to reconsider your choice. What would you do? In other words what are the probabilities P(A N A,C ) and P(B N A,C )? What does your intuition advise? Now apply Bayes Rule.
18 Bayesian Approach Example Classification based on Bayes Theorem Let Y denote class variable. For example we want to predict if the borrower will default. Let X = (X 1, X 1,..., X k ) denote the attribute set (i.e. home owner, marital status, annual income, etc) We can treat X and Y as random variables and determine P(Y X ) (posterior probability). Knowing the probability P(Y X ) we can relate the relate the record X to the class that maximizes the posterior probability. How can we estimate P(Y X ) from training data?
19 Bayesian Approach Example # Home Owner (binary) Marital Status (categorical) Annual Income (continuous) 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes Table: Historical data for default prediction Defaulted Borrower (class)
20 Bayes approach Introduction Bayesian Approach Example Accurate estimate of posterior probability for every possible combination of attributes and classes requires a very large training set, even for a moderate number of attributes. We can utilize Bayes theorem instead P(Y X ) = P(X Y ) P(Y ) P(X ) P(X ) is a constant and can be calculated as a normalization multiplier P(Y ) can be easily estimated from training set (fraction of training records that belong to each class) P(X Y ) is a more challenging task. Methods: Bayesian Network
21 Bayesian Approach Example Attributes are assumed to be conditionally independent, given the class label y: thus P(X Y = y) = k P(X i Y = y) i=1 P(Y X ) = P(Y ) k i=1 P(X i Y = y) P(X ) Now we need to estimate P(X i Y ) for i = 1,..., k.
22 Estimating Probabilities Bayesian Approach Example P(X i = x Y = y) is estimated according to fraction of training instances in class y that take on a particular attribute value x i. For example P(Home Owner=Yes Y = No) = 3/7 P(Marital Status=Single Y = Yes) = 2/3 What about continuous attributes? One solution is to discretize each continuous attribute and then replace value with its corresponding interval (transform continuous attributes into ordinal attributes). How can we discretize?..
23 Bayesian Approach Example Continuous Attributes Assume a certain type of probability distribution for continuous attribute. For example it can be a Gaussian distribution having p.d.f. f ij (x i ) = 1 e (xi µij ) 2σ ij 2 2πσij 2 parameters for f ij can be estimated based on training records that belongs to class y i
24 Continuous Attributes Bayesian Approach Example Using approximation P(x i < X i x i + ɛ Y = y i ) = xi +ɛ x i f ij (y)dy f ij (x i )ɛ and the fact that ɛ cancels out when we normalize posterior probability for P(Y X ) allows us to assume P(X i = x i Y = y j ) = f ij (x i )
25 Example Introduction Bayesian Approach Example The sample mean for annual income attribute with respect to the class No Variance x = = 110 s 2 = ( )2 + ( ) (75 100) 2 6 s = 2975 = Given a test record with income $120K P(Income = 120 No) = = π54.54 e ( ) =
26 Bayesian Approach Example Example Suppose X =(Home Owner=No, Marital Status=Married, Income = $120K) P( Home Owner = Yes Y = No) = 3/7 P( Home Owner = No Y = No) = 4/7 P( Home Owner = Yes Y = Yes) = 0 P( Home Owner = No Y = Yes) = 1 P( Marital Status = Divorced Y = No) = 2/7 P( Marital Status = Married Y = No) = 1/7 P( Marital Status = Single Y = No) = 4/7 P( Marital Status = Divorced Y = Yes) = 2/3 P( Marital Status = Married Y = Yes) = 1/3 P( Marital Status = Single Y = Yes) = 0
27 Example Introduction Bayesian Approach Example For annual income: class No: x = 110, s 2 = 2975 class Yes: x = 90, s 2 = 25 Class-conditional probabilities: P(X No) = P(Home Owner = No No) P(Status=Married No) P(Annual Income = $120K No) = 4/7 4/ = P(X Yes) = P(Home Owner = No Yes) P(Status=Married Yes) P(Annual Income = $120K Yes) = = 0
28 Example Introduction Bayesian Approach Example Posterior probabilities P(No X ) = α 7/ = α P(Yes X ) = 0 where α = 1/P(X ) Since P(No X ) > (Yes X ) the record is classified as No
29 : Discussion Bayesian Approach Example Robust to isolated noise points because such points are averaged out when estimating conditional probabilities from data Can handle missing values by ignoring the example during model building and classification Robust to irrelevant attributes. If X i is irrelevant then P(X i Y ) is almost uniformly distributed and thus P(X i Y ) has little impact on posterior probability Correlated attributes can degrade the performance because conditional independence does not hold. account dependence between attributes
30 Graph Theory Concepts Definition Inferencing Directed Graph Definition A directed graph or digraph G is an ordered pair G := (V, A) with V is a set, whose elements are called vertices or nodes, A V V is a set of ordered pairs of vertices, called directed edges, arcs, or arrows. V = {V 1, V 2, V 3, V 4, V 5 } E = {(V 1, V 1 ), (V 1, V 4 ), (V 2, V 1 ), (V 4, V 2 ), (V 5, V 5 )} Cycle: V 1 V 4 V 2
31 Directed Acyclic Graph Graph Theory Concepts Definition Inferencing Definition A directed acyclic graph (DAG), is a directed graph with no directed cycles; that is, for any vertex v, there is no nonempty directed path that starts and ends on v.
32 Some Graph Theory Notions Graph Theory Concepts Definition Inferencing V 1 and V 4 are parents of V 2 (V 1, V 2 ) E and (V 4, V 2 ) E V 5, V 3 and V 2 are descendants of V 1 V1 is connected to V 5, V 3 and V 2 with directed paths V 4 and V 2 are ancestors of V 3 There exist directed paths from V4 and V 2 to V 3 V 6 and V 4 are nondescendents of V 1 Directed paths from V1 to V 4 and V 6 do not exist
33 Bayesian Network Definition Graph Theory Concepts Definition Inferencing Elements of Bayesian Network: Directed acyclic graph (DAG) encodes the dependence relationships among a set of variables A probability table associating each node to its immediate parent nodes Each node of the graph represents a variable Each arc asserts the dependence relationship between the pair of variables DAG satisfies Markov condition
34 The Markov Condition Graph Theory Concepts Definition Inferencing Definition Suppose we have a joint probability distribution P of the random variables in some set V and a DAG G = (V, E). We say that (G, P) satisfies the Markov condition if for each variable X V, {X } is conditionally independent of the set of all its nondescendents (ND X ) given the set of all its parents (PA X ). I ({X }, ND X PA X ). The definitin implies that a root node X, which has no parents, is unconditionally independent from its nondescendents.
35 Graph Theory Concepts Definition Inferencing Figure: Bayes network: a case study
36 Markov Condition Example Graph Theory Concepts Definition Inferencing Node Parents Independency E I (E, {D, Hb}) D I (D, E) HD E, D I (HD, Hb {E, D}) Hb?? B?? C?? Note that I (A, B C) implies I (A, D C) whenever D B.
37 Graph Theory Concepts Definition Inferencing Representation Recall that a naïve Bayes classifier assumes conditional independence of attributes X 1, X 1,..., X k, given target class Y This can be represented using a Bayesian Network below
38 Inferencing Introduction Graph Theory Concepts Definition Inferencing We can compute joint probability from a Bayesian Network P(X 1, X 2,..., X n ) = n P(X i parents(x i )) i=1 Thus we can compute any conditional probability P(X k X m ) = P(X entries X P(X) k, X m ) matching X = k,xm P(X m ) P(X) entries X matching X k
39 Example of Inferencing Graph Theory Concepts Definition Inferencing Suppose no prior information about the person is given What is the probability of developing heart disease? α = {Yes, No}, β = {Healthy, Unhealthy} P(HD = Yes) = = α β P(HD = Yes E = α, D = β) P(E = α, D = β) = = α β P(HD = Yes E = α, D = β) P(E = α) P(D = β) = = = 0.49
40 Graph Theory Concepts Definition Inferencing Now let us compute probability of heart disease when the person has high blood pressure γ = {Yes, No} Probability of high blood pressure P(B = High) = = γ P(B = High HD = γ) P(HD = γ) = = = The posterior probability of heart disease given high blood pressure is P(BP = High HD = Yes) P(HD = Yes) P(HD = Yes BP = High) = P(BP = High) = ( )/ = =
41 Complexity Issues Introduction Graph Theory Concepts Definition Inferencing Recall, we can compute any conditional probability: P(X k X m ) = P(X entries X P(X) k, X m ) matching X = k,xm P(X m ) P(X) entries X matching X k Generally it requires exponentially large number of operations We can apply various tricks to reduce complexity But querying of Bayes nets is NP-hard D. M. Chickering, D. Heckerman, C. Meek, Large-Sample Learning of is NP-Hard. Journal of Machine Learning Research, 5 (2004)
42 Graph Theory Concepts Definition Inferencing Discussion Bayes network is an elegant way of encoding casual probabilistic dependencies. The dependency model can be represented graphically Constructing a network requires effort but adding a new variable is quite straightforward Well suited for incomplete data. Due to probabilistic nature of the model the method is robust to model overfitting
43 What we have learned Independence and conditional independence Bayes theorem Naïve Bayes classification The definition of a Bayes net Computing probabilities with a Bayes net
44 Literature Pang-Ning Tan, Michael Steinbach, Vipin Kumar, Introduction to Data Mining, Addison-Wesley, 2005 Finn V. Jensen, Thomas D Nielsen. and Decision Graphs. 2nd Ed., Springer, 2007 Richard E. Neapolitan, Learning, Prentice Hall, 2003 Indea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann, 1988
Bayesian Approaches Data Mining Selected Technique
Bayesian Approaches Data Mining Selected Technique Henry Xiao xiao@cs.queensu.ca School of Computing Queen s University Henry Xiao CISC 873 Data Mining p. 1/17 Probabilistic Bases Review the fundamentals
More informationMotivation. Bayesian Networks in Epistemology and Philosophy of Science Lecture. Overview. Organizational Issues
Bayesian Networks in Epistemology and Philosophy of Science Lecture 1: Bayesian Networks Center for Logic and Philosophy of Science Tilburg University, The Netherlands Formal Epistemology Course Northern
More informationProbabilistic Reasoning. (Mostly using Bayesian Networks)
Probabilistic Reasoning (Mostly using Bayesian Networks) Introduction: Why probabilistic reasoning? The world is not deterministic. (Usually because information is limited.) Ways of coping with uncertainty
More informationCS 484 Data Mining. Classification 7. Some slides are from Professor Padhraic Smyth at UC Irvine
CS 484 Data Mining Classification 7 Some slides are from Professor Padhraic Smyth at UC Irvine Bayesian Belief networks Conditional independence assumption of Naïve Bayes classifier is too strong. Allows
More informationLecture 10: Introduction to reasoning under uncertainty. Uncertainty
Lecture 10: Introduction to reasoning under uncertainty Introduction to reasoning under uncertainty Review of probability Axioms and inference Conditional probability Probability distributions COMP-424,
More informationDirected Graphical Models
CS 2750: Machine Learning Directed Graphical Models Prof. Adriana Kovashka University of Pittsburgh March 28, 2017 Graphical Models If no assumption of independence is made, must estimate an exponential
More informationUncertainty and Bayesian Networks
Uncertainty and Bayesian Networks Tutorial 3 Tutorial 3 1 Outline Uncertainty Probability Syntax and Semantics for Uncertainty Inference Independence and Bayes Rule Syntax and Semantics for Bayesian Networks
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 143 Part IV
More informationConsider an experiment that may have different outcomes. We are interested to know what is the probability of a particular set of outcomes.
CMSC 310 Artificial Intelligence Probabilistic Reasoning and Bayesian Belief Networks Probabilities, Random Variables, Probability Distribution, Conditional Probability, Joint Distributions, Bayes Theorem
More informationCourse Introduction. Probabilistic Modelling and Reasoning. Relationships between courses. Dealing with Uncertainty. Chris Williams.
Course Introduction Probabilistic Modelling and Reasoning Chris Williams School of Informatics, University of Edinburgh September 2008 Welcome Administration Handout Books Assignments Tutorials Course
More informationRecall from last time: Conditional probabilities. Lecture 2: Belief (Bayesian) networks. Bayes ball. Example (continued) Example: Inference problem
Recall from last time: Conditional probabilities Our probabilistic models will compute and manipulate conditional probabilities. Given two random variables X, Y, we denote by Lecture 2: Belief (Bayesian)
More informationNaïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
More informationIntroduction to Artificial Intelligence. Unit # 11
Introduction to Artificial Intelligence Unit # 11 1 Course Outline Overview of Artificial Intelligence State Space Representation Search Techniques Machine Learning Logic Probabilistic Reasoning/Bayesian
More informationBayesian Networks BY: MOHAMAD ALSABBAGH
Bayesian Networks BY: MOHAMAD ALSABBAGH Outlines Introduction Bayes Rule Bayesian Networks (BN) Representation Size of a Bayesian Network Inference via BN BN Learning Dynamic BN Introduction Conditional
More informationData Mining Classification: Basic Concepts and Techniques. Lecture Notes for Chapter 3. Introduction to Data Mining, 2nd Edition
Data Mining Classification: Basic Concepts and Techniques Lecture Notes for Chapter 3 by Tan, Steinbach, Karpatne, Kumar 1 Classification: Definition Given a collection of records (training set ) Each
More informationProbabilistic Graphical Networks: Definitions and Basic Results
This document gives a cursory overview of Probabilistic Graphical Networks. The material has been gleaned from different sources. I make no claim to original authorship of this material. Bayesian Graphical
More informationRecall from last time. Lecture 3: Conditional independence and graph structure. Example: A Bayesian (belief) network.
ecall from last time Lecture 3: onditional independence and graph structure onditional independencies implied by a belief network Independence maps (I-maps) Factorization theorem The Bayes ball algorithm
More informationBayesian Networks. Motivation
Bayesian Networks Computer Sciences 760 Spring 2014 http://pages.cs.wisc.edu/~dpage/cs760/ Motivation Assume we have five Boolean variables,,,, The joint probability is,,,, How many state configurations
More informationIntelligent Systems: Reasoning and Recognition. Reasoning with Bayesian Networks
Intelligent Systems: Reasoning and Recognition James L. Crowley ENSIMAG 2 / MoSIG M1 Second Semester 2016/2017 Lesson 13 24 march 2017 Reasoning with Bayesian Networks Naïve Bayesian Systems...2 Example
More informationProbabilistic Graphical Models (I)
Probabilistic Graphical Models (I) Hongxin Zhang zhx@cad.zju.edu.cn State Key Lab of CAD&CG, ZJU 2015-03-31 Probabilistic Graphical Models Modeling many real-world problems => a large number of random
More informationReasoning with Uncertainty
Reasoning with Uncertainty Representing Uncertainty Manfred Huber 2005 1 Reasoning with Uncertainty The goal of reasoning is usually to: Determine the state of the world Determine what actions to take
More informationCS 2750: Machine Learning. Bayesian Networks. Prof. Adriana Kovashka University of Pittsburgh March 14, 2016
CS 2750: Machine Learning Bayesian Networks Prof. Adriana Kovashka University of Pittsburgh March 14, 2016 Plan for today and next week Today and next time: Bayesian networks (Bishop Sec. 8.1) Conditional
More informationCh.6 Uncertain Knowledge. Logic and Uncertainty. Representation. One problem with logical approaches: Department of Computer Science
Ch.6 Uncertain Knowledge Representation Hantao Zhang http://www.cs.uiowa.edu/ hzhang/c145 The University of Iowa Department of Computer Science Artificial Intelligence p.1/39 Logic and Uncertainty One
More informationObjectives. Probabilistic Reasoning Systems. Outline. Independence. Conditional independence. Conditional independence II.
Copyright Richard J. Povinelli rev 1.0, 10/1//2001 Page 1 Probabilistic Reasoning Systems Dr. Richard J. Povinelli Objectives You should be able to apply belief networks to model a problem with uncertainty.
More informationBayesian Reasoning. Adapted from slides by Tim Finin and Marie desjardins.
Bayesian Reasoning Adapted from slides by Tim Finin and Marie desjardins. 1 Outline Probability theory Bayesian inference From the joint distribution Using independence/factoring From sources of evidence
More informationQuantifying Uncertainty & Probabilistic Reasoning. Abdulla AlKhenji Khaled AlEmadi Mohammed AlAnsari
Quantifying Uncertainty & Probabilistic Reasoning Abdulla AlKhenji Khaled AlEmadi Mohammed AlAnsari Outline Previous Implementations What is Uncertainty? Acting Under Uncertainty Rational Decisions Basic
More information{ p if x = 1 1 p if x = 0
Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =
More informationBayesian Inference. Definitions from Probability: Naive Bayes Classifiers: Advantages and Disadvantages of Naive Bayes Classifiers:
Bayesian Inference The purpose of this document is to review belief networks and naive Bayes classifiers. Definitions from Probability: Belief networks: Naive Bayes Classifiers: Advantages and Disadvantages
More informationCMPSCI 240: Reasoning about Uncertainty
CMPSCI 240: Reasoning about Uncertainty Lecture 17: Representing Joint PMFs and Bayesian Networks Andrew McGregor University of Massachusetts Last Compiled: April 7, 2017 Warm Up: Joint distributions Recall
More informationCOMP9414: Artificial Intelligence Reasoning Under Uncertainty
COMP9414, Monday 16 April, 2012 Reasoning Under Uncertainty 2 COMP9414: Artificial Intelligence Reasoning Under Uncertainty Overview Problems with Logical Approach What Do the Numbers Mean? Wayne Wobcke
More informationProbabilistic Reasoning Systems
Probabilistic Reasoning Systems Dr. Richard J. Povinelli Copyright Richard J. Povinelli rev 1.0, 10/7/2001 Page 1 Objectives You should be able to apply belief networks to model a problem with uncertainty.
More informationDATA MINING LECTURE 10
DATA MINING LECTURE 10 Classification Nearest Neighbor Classification Support Vector Machines Logistic Regression Naïve Bayes Classifier Supervised Learning 10 10 Illustrating Classification Task Tid Attrib1
More informationDiscrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 14
CS 70 Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 14 Introduction One of the key properties of coin flips is independence: if you flip a fair coin ten times and get ten
More informationNaïve Bayes classification. p ij 11/15/16. Probability theory. Probability theory. Probability theory. X P (X = x i )=1 i. Marginal Probability
Probability theory Naïve Bayes classification Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s height, the outcome of a coin toss Distinguish
More informationProbability Theory Review
Probability Theory Review Brendan O Connor 10-601 Recitation Sept 11 & 12, 2012 1 Mathematical Tools for Machine Learning Probability Theory Linear Algebra Calculus Wikipedia is great reference 2 Probability
More informationProbability. CS 3793/5233 Artificial Intelligence Probability 1
CS 3793/5233 Artificial Intelligence 1 Motivation Motivation Random Variables Semantics Dice Example Joint Dist. Ex. Axioms Agents don t have complete knowledge about the world. Agents need to make decisions
More informationBased on slides by Richard Zemel
CSC 412/2506 Winter 2018 Probabilistic Learning and Reasoning Lecture 3: Directed Graphical Models and Latent Variables Based on slides by Richard Zemel Learning outcomes What aspects of a model can we
More informationPROBABILISTIC REASONING SYSTEMS
PROBABILISTIC REASONING SYSTEMS In which we explain how to build reasoning systems that use network models to reason with uncertainty according to the laws of probability theory. Outline Knowledge in uncertain
More informationRepresentation. Stefano Ermon, Aditya Grover. Stanford University. Lecture 2
Representation Stefano Ermon, Aditya Grover Stanford University Lecture 2 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 2 1 / 32 Learning a generative model We are given a training
More informationQuantifying uncertainty & Bayesian networks
Quantifying uncertainty & Bayesian networks CE417: Introduction to Artificial Intelligence Sharif University of Technology Spring 2016 Soleymani Artificial Intelligence: A Modern Approach, 3 rd Edition,
More informationProbability Theory for Machine Learning. Chris Cremer September 2015
Probability Theory for Machine Learning Chris Cremer September 2015 Outline Motivation Probability Definitions and Rules Probability Distributions MLE for Gaussian Parameter Estimation MLE and Least Squares
More informationUncertainty. Russell & Norvig Chapter 13.
Uncertainty Russell & Norvig Chapter 13 http://toonut.com/wp-content/uploads/2011/12/69wp.jpg Uncertainty Let A t be the action of leaving for the airport t minutes before your flight Will A t get you
More informationGraphical Models - Part I
Graphical Models - Part I Oliver Schulte - CMPT 726 Bishop PRML Ch. 8, some slides from Russell and Norvig AIMA2e Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Outline Probabilistic
More informationP (A B) P ((B C) A) P (B A) = P (B A) + P (C A) P (A) = P (B A) + P (C A) = Q(A) + Q(B).
Lectures 7-8 jacques@ucsdedu 41 Conditional Probability Let (Ω, F, P ) be a probability space Suppose that we have prior information which leads us to conclude that an event A F occurs Based on this information,
More informationIntroduction to Probabilistic Graphical Models
Introduction to Probabilistic Graphical Models Kyu-Baek Hwang and Byoung-Tak Zhang Biointelligence Lab School of Computer Science and Engineering Seoul National University Seoul 151-742 Korea E-mail: kbhwang@bi.snu.ac.kr
More informationEE562 ARTIFICIAL INTELLIGENCE FOR ENGINEERS
EE562 ARTIFICIAL INTELLIGENCE FOR ENGINEERS Lecture 16, 6/1/2005 University of Washington, Department of Electrical Engineering Spring 2005 Instructor: Professor Jeff A. Bilmes Uncertainty & Bayesian Networks
More informationProbabilistic Graphical Models and Bayesian Networks. Artificial Intelligence Bert Huang Virginia Tech
Probabilistic Graphical Models and Bayesian Networks Artificial Intelligence Bert Huang Virginia Tech Concept Map for Segment Probabilistic Graphical Models Probabilistic Time Series Models Particle Filters
More information1. what conditional independencies are implied by the graph. 2. whether these independecies correspond to the probability distribution
NETWORK ANALYSIS Lourens Waldorp PROBABILITY AND GRAPHS The objective is to obtain a correspondence between the intuitive pictures (graphs) of variables of interest and the probability distributions of
More informationProbabilistic Classification
Bayesian Networks Probabilistic Classification Goal: Gather Labeled Training Data Build/Learn a Probability Model Use the model to infer class labels for unlabeled data points Example: Spam Filtering...
More informationLearning Bayesian Networks (part 1) Goals for the lecture
Learning Bayesian Networks (part 1) Mark Craven and David Page Computer Scices 760 Spring 2018 www.biostat.wisc.edu/~craven/cs760/ Some ohe slides in these lectures have been adapted/borrowed from materials
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2016
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2016 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University August 30, 2017 Today: Decision trees Overfitting The Big Picture Coming soon Probabilistic learning MLE,
More informationEcon 325: Introduction to Empirical Economics
Econ 325: Introduction to Empirical Economics Lecture 2 Probability Copyright 2010 Pearson Education, Inc. Publishing as Prentice Hall Ch. 3-1 3.1 Definition Random Experiment a process leading to an uncertain
More informationBayesian Networks aka belief networks, probabilistic networks. Bayesian Networks aka belief networks, probabilistic networks. An Example Bayes Net
Bayesian Networks aka belief networks, probabilistic networks A BN over variables {X 1, X 2,, X n } consists of: a DAG whose nodes are the variables a set of PTs (Pr(X i Parents(X i ) ) for each X i P(a)
More informationIntroduction to Bayes Nets. CS 486/686: Introduction to Artificial Intelligence Fall 2013
Introduction to Bayes Nets CS 486/686: Introduction to Artificial Intelligence Fall 2013 1 Introduction Review probabilistic inference, independence and conditional independence Bayesian Networks - - What
More informationMachine Learning Lecture 14
Many slides adapted from B. Schiele, S. Roth, Z. Gharahmani Machine Learning Lecture 14 Undirected Graphical Models & Inference 23.06.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ leibe@vision.rwth-aachen.de
More informationChris Bishop s PRML Ch. 8: Graphical Models
Chris Bishop s PRML Ch. 8: Graphical Models January 24, 2008 Introduction Visualize the structure of a probabilistic model Design and motivate new models Insights into the model s properties, in particular
More informationArtificial Intelligence
Artificial Intelligence Dr Ahmed Rafat Abas Computer Science Dept, Faculty of Computers and Informatics, Zagazig University arabas@zu.edu.eg http://www.arsaliem.faculty.zu.edu.eg/ Uncertainty Chapter 13
More informationWhere are we? Knowledge Engineering Semester 2, Reasoning under Uncertainty. Probabilistic Reasoning
Knowledge Engineering Semester 2, 2004-05 Michael Rovatsos mrovatso@inf.ed.ac.uk Lecture 8 Dealing with Uncertainty 8th ebruary 2005 Where are we? Last time... Model-based reasoning oday... pproaches to
More informationAn AI-ish view of Probability, Conditional Probability & Bayes Theorem
An AI-ish view of Probability, Conditional Probability & Bayes Theorem Review: Uncertainty and Truth Values: a mismatch Let action A t = leave for airport t minutes before flight. Will A 15 get me there
More information10/18/2017. An AI-ish view of Probability, Conditional Probability & Bayes Theorem. Making decisions under uncertainty.
An AI-ish view of Probability, Conditional Probability & Bayes Theorem Review: Uncertainty and Truth Values: a mismatch Let action A t = leave for airport t minutes before flight. Will A 15 get me there
More informationComputational Genomics
Computational Genomics http://www.cs.cmu.edu/~02710 Introduction to probability, statistics and algorithms (brief) intro to probability Basic notations Random variable - referring to an element / event
More informationARTIFICIAL INTELLIGENCE. Uncertainty: probabilistic reasoning
INFOB2KI 2017-2018 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Uncertainty: probabilistic reasoning Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from
More informationGraphical models and causality: Directed acyclic graphs (DAGs) and conditional (in)dependence
Graphical models and causality: Directed acyclic graphs (DAGs) and conditional (in)dependence General overview Introduction Directed acyclic graphs (DAGs) and conditional independence DAGs and causal effects
More informationBayesian Networks Inference with Probabilistic Graphical Models
4190.408 2016-Spring Bayesian Networks Inference with Probabilistic Graphical Models Byoung-Tak Zhang intelligence Lab Seoul National University 4190.408 Artificial (2016-Spring) 1 Machine Learning? Learning
More informationStochastic Methods. 5.0 Introduction 5.1 The Elements of Counting 5.2 Elements of Probability Theory
5 Stochastic Methods 5.0 Introduction 5.1 The Elements of Counting 5.2 Elements of Probability Theory 5.4 The Stochastic Approach to Uncertainty 5.4 Epilogue and References 5.5 Exercises Note: The slides
More informationUncertainty. Introduction to Artificial Intelligence CS 151 Lecture 2 April 1, CS151, Spring 2004
Uncertainty Introduction to Artificial Intelligence CS 151 Lecture 2 April 1, 2004 Administration PA 1 will be handed out today. There will be a MATLAB tutorial tomorrow, Friday, April 2 in AP&M 4882 at
More informationUncertainty. Chapter 13
Uncertainty Chapter 13 Outline Uncertainty Probability Syntax and Semantics Inference Independence and Bayes Rule Uncertainty Let s say you want to get to the airport in time for a flight. Let action A
More informationDirected Graphical Models or Bayesian Networks
Directed Graphical Models or Bayesian Networks Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Bayesian Networks One of the most exciting recent advancements in statistical AI Compact
More informationBe able to define the following terms and answer basic questions about them:
CS440/ECE448 Section Q Fall 2017 Final Review Be able to define the following terms and answer basic questions about them: Probability o Random variables, axioms of probability o Joint, marginal, conditional
More informationIntroduction to Machine Learning
Introduction to Machine Learning CS4375 --- Fall 2018 Bayesian a Learning Reading: Sections 13.1-13.6, 20.1-20.2, R&N Sections 6.1-6.3, 6.7, 6.9, Mitchell 1 Uncertainty Most real-world problems deal with
More informationData Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan, Steinbach, Kumar Adapted by Qiang Yang (2010) Tan,Steinbach,
More informationDirected Graphical Models. William W. Cohen Machine Learning
Directed Graphical Models William W. Cohen Machine Learning 10-601 MOTIVATION FOR GRAPHICAL MODELS Recap: A paradox of induction A black crow seems to support the hypothesis all crows are black. A pink
More informationReview: Bayesian learning and inference
Review: Bayesian learning and inference Suppose the agent has to make decisions about the value of an unobserved query variable X based on the values of an observed evidence variable E Inference problem:
More informationIntroduction to Machine Learning
Uncertainty Introduction to Machine Learning CS4375 --- Fall 2018 a Bayesian Learning Reading: Sections 13.1-13.6, 20.1-20.2, R&N Sections 6.1-6.3, 6.7, 6.9, Mitchell Most real-world problems deal with
More informationImplementing Machine Reasoning using Bayesian Network in Big Data Analytics
Implementing Machine Reasoning using Bayesian Network in Big Data Analytics Steve Cheng, Ph.D. Guest Speaker for EECS 6893 Big Data Analytics Columbia University October 26, 2017 Outline Introduction Probability
More informationUncertainty. Logic and Uncertainty. Russell & Norvig. Readings: Chapter 13. One problem with logical-agent approaches: C:145 Artificial
C:145 Artificial Intelligence@ Uncertainty Readings: Chapter 13 Russell & Norvig. Artificial Intelligence p.1/43 Logic and Uncertainty One problem with logical-agent approaches: Agents almost never have
More informationCS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Matrix Data: Classification: Part 2 Instructor: Yizhou Sun yzsun@ccs.neu.edu September 21, 2014 Methods to Learn Matrix Data Set Data Sequence Data Time Series Graph & Network
More informationLecture 8: Bayesian Networks
Lecture 8: Bayesian Networks Bayesian Networks Inference in Bayesian Networks COMP-652 and ECSE 608, Lecture 8 - January 31, 2017 1 Bayes nets P(E) E=1 E=0 0.005 0.995 E B P(B) B=1 B=0 0.01 0.99 E=0 E=1
More informationLecture 2. Conditional Probability
Math 408 - Mathematical Statistics Lecture 2. Conditional Probability January 18, 2013 Konstantin Zuev (USC) Math 408, Lecture 2 January 18, 2013 1 / 9 Agenda Motivation and Definition Properties of Conditional
More informationn How to represent uncertainty in knowledge? n Which action to choose under uncertainty? q Assume the car does not have a flat tire
Uncertainty Uncertainty Russell & Norvig Chapter 13 Let A t be the action of leaving for the airport t minutes before your flight Will A t get you there on time? A purely logical approach either 1. risks
More informationAdvanced classifica-on methods
Advanced classifica-on methods Instance-based classifica-on Bayesian classifica-on Instance-Based Classifiers Set of Stored Cases Atr1... AtrN Class A B B C A C B Store the training records Use training
More informationCOS402- Artificial Intelligence Fall Lecture 10: Bayesian Networks & Exact Inference
COS402- Artificial Intelligence Fall 2015 Lecture 10: Bayesian Networks & Exact Inference Outline Logical inference and probabilistic inference Independence and conditional independence Bayes Nets Semantics
More informationCS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Chapter 8&9: Classification: Part 3 Instructor: Yizhou Sun yzsun@ccs.neu.edu March 12, 2013 Midterm Report Grade Distribution 90-100 10 80-89 16 70-79 8 60-69 4
More informationProbabilistic Models
Bayes Nets 1 Probabilistic Models Models describe how (a portion of) the world works Models are always simplifications May not account for every variable May not account for all interactions between variables
More informationBuilding Bayesian Networks. Lecture3: Building BN p.1
Building Bayesian Networks Lecture3: Building BN p.1 The focus today... Problem solving by Bayesian networks Designing Bayesian networks Qualitative part (structure) Quantitative part (probability assessment)
More informationDATA MINING: NAÏVE BAYES
DATA MINING: NAÏVE BAYES 1 Naïve Bayes Classifier Thomas Bayes 1702-1761 We will start off with some mathematical background. But first we start with some visual intuition. 2 Grasshoppers Antenna Length
More informationCOMP5211 Lecture Note on Reasoning under Uncertainty
COMP5211 Lecture Note on Reasoning under Uncertainty Fangzhen Lin Department of Computer Science and Engineering Hong Kong University of Science and Technology Fangzhen Lin (HKUST) Uncertainty 1 / 33 Uncertainty
More information1 : Introduction. 1 Course Overview. 2 Notation. 3 Representing Multivariate Distributions : Probabilistic Graphical Models , Spring 2014
10-708: Probabilistic Graphical Models 10-708, Spring 2014 1 : Introduction Lecturer: Eric P. Xing Scribes: Daniel Silva and Calvin McCarter 1 Course Overview In this lecture we introduce the concept of
More informationLecture 1: Bayesian Framework Basics
Lecture 1: Bayesian Framework Basics Melih Kandemir melih.kandemir@iwr.uni-heidelberg.de April 21, 2014 What is this course about? Building Bayesian machine learning models Performing the inference of
More informationBayesian Learning. CSL603 - Fall 2017 Narayanan C Krishnan
Bayesian Learning CSL603 - Fall 2017 Narayanan C Krishnan ckn@iitrpr.ac.in Outline Bayes Theorem MAP Learners Bayes optimal classifier Naïve Bayes classifier Example text classification Bayesian networks
More informationOrigins of Probability Theory
1 16.584: INTRODUCTION Theory and Tools of Probability required to analyze and design systems subject to uncertain outcomes/unpredictability/randomness. Such systems more generally referred to as Experiments.
More informationLecture Notes 1 Basic Probability. Elements of Probability. Conditional probability. Sequential Calculation of Probability
Lecture Notes 1 Basic Probability Set Theory Elements of Probability Conditional probability Sequential Calculation of Probability Total Probability and Bayes Rule Independence Counting EE 178/278A: Basic
More informationProbabilistic representation and reasoning
Probabilistic representation and reasoning Applied artificial intelligence (EDAF70) Lecture 04 2019-02-01 Elin A. Topp Material based on course book, chapter 13, 14.1-3 1 Show time! Two boxes of chocolates,
More informationUncertainty and knowledge. Uncertainty and knowledge. Reasoning with uncertainty. Notes
Approximate reasoning Uncertainty and knowledge Introduction All knowledge representation formalism and problem solving mechanisms that we have seen until now are based on the following assumptions: All
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2014
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2014 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationCMPSCI 240: Reasoning about Uncertainty
CMPSCI 240: Reasoning about Uncertainty Lecture 4: Sequential experiments Andrew McGregor University of Massachusetts Last Compiled: February 2, 2017 Outline 1 Recap 2 Sequential Experiments 3 Total Probability
More informationThe Naïve Bayes Classifier. Machine Learning Fall 2017
The Naïve Bayes Classifier Machine Learning Fall 2017 1 Today s lecture The naïve Bayes Classifier Learning the naïve Bayes Classifier Practical concerns 2 Today s lecture The naïve Bayes Classifier Learning
More informationAdvanced Probabilistic Modeling in R Day 1
Advanced Probabilistic Modeling in R Day 1 Roger Levy University of California, San Diego July 20, 2015 1/24 Today s content Quick review of probability: axioms, joint & conditional probabilities, Bayes
More informationArtificial Intelligence Bayesian Networks
Artificial Intelligence Bayesian Networks Stephan Dreiseitl FH Hagenberg Software Engineering & Interactive Media Stephan Dreiseitl (Hagenberg/SE/IM) Lecture 11: Bayesian Networks Artificial Intelligence
More information