Outline. Introduction. Bayesian Probability Theory Bayes rule Bayes rule applied to learning Bayesian learning and the MDL principle
|
|
|
- John York
- 6 years ago
- Views:
Transcription
1 Outline Introduction Bayesian Probability Theory Bayes rule Bayes rule applied to learning Bayesian learning and the MDL principle Sequence Prediction and Data Compression Bayesian Networks Copyright 2015 Kee Siong Ng. All rights reserved. 2
2 Medical Diagnosis Suppose in a population only 8 in 1000 people have a certain cancer. We have a lab test for the cancer that returns a positive result in 98 of the cases in which the cancer is actually present; and a negative result in 97 of the cases in which the cancer is not present. Suppose we now observe a new patient for whom the lab test returns a positive result. Should we diagnose the patient as having cancer or not? Copyright 2015 Kee Siong Ng. All rights reserved. 3
3 Monty Hall Problem Suppose you re on a game show, and you re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, Do you want to pick door No. 2? Is it to your advantage to switch your choice? Copyright 2015 Kee Siong Ng. All rights reserved. 4
4 Bayesian Probability Theory Probability denotes degree of belief. Prescribes a rational approach to changing one s belief upon seeing new evidence. A central concept is the Bayes Rule, named after Reverend Thomas Bayes ( ). Copyright 2015 Kee Siong Ng. All rights reserved. 5
5 Bayes Rule Let A and B be events/propositions, we have Pr(A B)= Pr(B A)Pr(A). Pr(B) Derivation is simple: By definition of conditional probability, Pr(A B)= Pr(A ^ B) Pr(B) and Pr(B A)= Thus Pr(A B) Pr(B) =Pr(A ^ B) =Pr(B A) Pr(A) Pr(A ^ B). Pr(A) Dividing by Pr(B) yields Pr(A B)= Pr(A ^ B) Pr(B) = Pr(B A)Pr(A). Pr(B) Copyright 2015 Kee Siong Ng. All rights reserved. 6
6 Application to Medical Diagnosis Prior: Pr(cancer)=0.008 Lab test: Pr( cancer)=0.992 Pr(posTest cancer)=0.98 Pr(posTest cancer)=0.03 Pr(negTest cancer)=0.02 Pr(negTest cancer)=0.97 Posterior probabilities: Pr(cancer postest) =Pr(posTest cancer) Pr(cancer)/ Pr(posTest) = /Pr(posTest)= K Copyright 2015 Kee Siong Ng. All rights reserved. 7
7 Application to Medical Diagnosis Prior: Pr(cancer)=0.008 Lab test: Pr( cancer)=0.992 Pr(posTest cancer)=0.98 Pr(negTest cancer)=0.02 Pr(posTest cancer)=0.03 Pr(negTest cancer)=0.97 Posterior probabilities: Pr(cancer postest) =Pr(posTest cancer) Pr(cancer)/ Pr(posTest) = /Pr(posTest)= K Pr( cancer postest) =Pr(posTest cancer) Pr( cancer)/ Pr(posTest) = /Pr(posTest)= K Copyright 2015 Kee Siong Ng. All rights reserved. 7
8 A Second Test New prior after first test: Pr(cancer)=0.207 Second lab test: Pr( cancer)=0.793 Pr(posTest 2 cancer)=0.8 Pr(negTest 2 cancer)=0.2 Pr(posTest 2 cancer)=0.1 Pr(negTest 2 cancer)=0.9 Posterior probabilities after second positive test: Pr(cancer postest 2 )=Pr(posTest 2 cancer)pr(cancer)/pr(postest 2 ) = /Pr(posTest 2 )= K 2 Pr( cancer postest 2 )=Pr(posTest 2 cancer)pr( cancer)/pr(postest 2 ) = /Pr(posTest 2 )=0.079 K 2 Copyright 2015 Kee Siong Ng. All rights reserved. 8
9 Some Lessons That s why doctors order multiple tests. Probabilistic reasoning can appear counter-intuitive (in other words, we are not very good with probabilistic reasoning). Copyright 2015 Kee Siong Ng. All rights reserved. 9
10 Application to Monty Hall Problem Suppose you re on a game show, and you re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, Do you want to pick door No. 2? Is it to your advantage to switch your choice? Copyright 2015 Kee Siong Ng. All rights reserved. 10
11 Application to Monty Hall Problem Let C i denotes car is behind door i. Prior probability: Pr(C 1 )=Pr(C 2 )=Pr(C 3 )= 1 3 Suppose we pick door 1 and the host then reveals a goat behind door 2 (event denoted by H 2 ). Pr(C 1 H 2 )= Pr(H 2 C 1 )Pr(C 1 ) Pr(H 2 ) = 1/2 1/3 Pr(H 2 ) = 1 Pr(H 2 ) 1 6 Copyright 2015 Kee Siong Ng. All rights reserved. 11
12 Application to Monty Hall Problem Let C i denotes car is behind door i. Prior probability: Pr(C 1 )=Pr(C 2 )=Pr(C 3 )= 1 3 Suppose we pick door 1 and the host then reveals a goat behind door 2 (event denoted by H 2 ). Pr(C 1 H 2 )= Pr(H 2 C 1 )Pr(C 1 ) Pr(H 2 ) Pr(C 3 H 2 )= Pr(H 2 C 3 )Pr(C 3 ) Pr(H 2 ) = 1/2 1/3 Pr(H 2 ) = 1 Pr(H 2 ) = 1 1/3 Pr(H 2 ) = 1 Pr(H 2 ) Copyright 2015 Kee Siong Ng. All rights reserved. 11
13 Application to Monty Hall Problem Let C i denotes car is behind door i. Prior probability: Pr(C 1 )=Pr(C 2 )=Pr(C 3 )= 1 3 Suppose we pick door 1 and the host then reveals a goat behind door 2 (event denoted by H 2 ). Pr(C 1 H 2 )= Pr(H 2 C 1 )Pr(C 1 ) Pr(H 2 ) Pr(C 3 H 2 )= Pr(H 2 C 3 )Pr(C 3 ) Pr(H 2 ) = 1/2 1/3 Pr(H 2 ) = 1 Pr(H 2 ) = 1 1/3 Pr(H 2 ) = 1 Pr(H 2 ) Since Pr(C 3 H 2 ) > Pr(C 1 H 2 ), we should switch! Copyright 2015 Kee Siong Ng. All rights reserved. 11
14 Bayes Rule Applied to Learning Let H be a class of functions. Suppose we have seen data D. Given h 2 H, what is Pr(h D), i.e. the posterior probability of h given that we have seen the data D? Copyright 2015 Kee Siong Ng. All rights reserved. 12
15 Bayes Rule Applied to Learning Let H be a class of functions. Suppose we have seen data D. Given h 2 H, what is Pr(h D), i.e. the posterior probability of h given that we have seen the data D? By Bayes rule, Pr(h D)= Pr(D h)pr(h) Pr(D) Copyright 2015 Kee Siong Ng. All rights reserved. 12
16 Bayes Rule Applied to Learning Let H be a class of functions. Suppose we have seen data D. Given h 2 H, what is Pr(h D), i.e. the posterior probability of h given that we have seen the data D? By Bayes rule, Pr(h D)= Pr(D h)pr(h) Pr(D) Pr(h) is the prior probability of h Pr(D h) is the likelihood of h Copyright 2015 Kee Siong Ng. All rights reserved. 12
17 Bayes Rule Applied to Learning By Bayes rule, Pr(h D)= Pr(D h)pr(h) Pr(D) Bayes rule says posterior probability of h is proportional to prior probability of h times the likelihood of h. Pr(h) incorporates our background knowledge Pr(D h) can be simplified by making assumptions on the data-generation process. For example, if D =[(x 1,y 1 ),(x 2,y 2 ),...,(x n,y n )] are generated independently at random, then Pr(D h)= n i=1 Pr((x i,y i ) h)= n i=1 Pr(h(x i )=y i ). Copyright 2015 Kee Siong Ng. All rights reserved. 13
18 MAP Estimator A learning algorithm: Input: H, D, and prior probability Pr(h) 1. For each function h 2 H, calculate the posterior Pr(h D)= Pr(D h)pr(h). Pr(D) 2. Output the function h MAP with the highest posterior h MAP = argmax h2h Pr(h D). MAP = Maximum a posteriori Copyright 2015 Kee Siong Ng. All rights reserved. 14
19 Relationship to MDL Principle MDL = Minimum Description Length A formal version of Occam s razor: choose the shortest explanation for the observed data. We have h MAP = argmax h2h Pr(h)Pr(D h) = argmax log 2 Pr(h)+log 2 Pr(D h) h2h = argmin log 2 Pr(h) log 2 Pr(D h) h2h The last line can be interpreted as saying h MAP is the shortest hypothesis describing the data well. Copyright 2015 Kee Siong Ng. All rights reserved. 15
20 A Short Detour on Information Theory Consider the problem of designing a code to transmit messages drawn at random, where the probability of drawing message i is Pr(i). We want a code that minimises the expected number of bits we need to transmit. This implies assigning shorter codes to messages that are more probable. Shannon and Weaver (1949) showed that the optimal code assigns log 2 Pr(i) bits to encode message i. Copyright 2015 Kee Siong Ng. All rights reserved. 16
21 Relationship to MDL Principle We have h MAP = argmin h2h log 2 Pr(h) log 2 Pr(D h) Information-theoretic interpretation: log 2 Pr(h) is the code length of h under the optimal code (wrt the prior) for hypothesis space H; log 2 Pr(D h) is the code length of the observed data D under the optimal code wrt conditional probability of data given hypothesis h. Instead of transmitting D, we transmit a function h plus information on how to reconstruct D from h. Copyright 2015 Kee Siong Ng. All rights reserved. 17
22 Bayes Optimal Estimator The hypothesis h MAP is not the best estimator! Copyright 2015 Kee Siong Ng. All rights reserved. 18
23 Bayes Optimal Estimator The hypothesis h MAP is not the best estimator! Consider H = {h 1,h 2,h 3 }. Suppose Pr(h 1 D)=0.4 Pr(h 2 D)=0.3 Pr(h 3 D)=0.3. Given x, suppose further that h 1 (x)=pos h 2 (x)=neg h 3 (x)=neg. We have h MAP (x)=h 1 (x)=pos. But taking h 1,h 2,h 3 into account, x is positive with probability 0.4 and negative with probability 0.6. Copyright 2015 Kee Siong Ng. All rights reserved. 18
24 Bayes Optimal Estimator The best estimate of the label of x given data D and function class H is arg max y2y  h i 2H Pr(h i D)I(h i (x)=y) (3) Any system that predicts using (3) is called a Bayes optimal estimator. On average, no other prediction method that uses the same function class H and same prior can outperform a Bayes optimal estimator. A Bayes optimal estimator may not be in H itself. Copyright 2015 Kee Siong Ng. All rights reserved. 19
25 Bayes, Occam, Epicurus The best estimate of the label of x given data D and function class H is arg max y2y  h i 2H Pr(h i D)I(h i (x)=y) (4) Epicurus (342 BC 270 BC) principle of multiple explanations: keep all hypotheses that are consistent with the data. Occam s razor ( ): entities should not be multiplied beyond necessity. Widely understood to mean: Among all hypotheses consistent with the observations, choose the simplest. The Bayes optimal estimator strikes a balance between the two principles. Copyright 2015 Kee Siong Ng. All rights reserved. 20
26 Problems Solved with Bayesian Analysis A good prior allows us to cope with small sample sizes. If the sample size is large, then the prior is overwhelmed by the likelihood, which is determined entirely by the data. A Bayes optimal estimator weighs and combines the prediction of multiple plausible hypotheses. Copyright 2015 Kee Siong Ng. All rights reserved. 21
27 Problems with Bayesian Analysis Prescribed solution is often computationally intractable. Choosing a good prior Pr(h) is not always straightforward. Copyright 2015 Kee Siong Ng. All rights reserved. 22
28 Test Your Knowledge: Exercise I You have been called to jury duty in a town where there are two taxi companies, Green Cabs Ltd. and Blue Taxi Inc. Blue Taxi uses cars painted blue; Green Cabs uses green cars. Green Cabs dominates the market, with 85 of the taxies on the road. On a misty winter night a taxi sideswiped another car and drove off. A witness says it was a blue cab. The witness is tested under conditions like those on the night of the accident, and 80 of the time she correctly reports the colour of the cab that is seen. What is the probability that the taxi that caused the accident is blue? Copyright 2015 Kee Siong Ng. All rights reserved. 23
29 Test Your Knowledge: Exercise II You are a physician. You think it is quite likely (say 60 probability) that one of your patients has strep throat, but you aren t sure. You take some swabs from the throat and send them to a lab for testing. The test is (like nearly all lab tests) not perfect. If the patient has strep throat, then 70 of the time the lab says YES. But 30 of the time it says NO. If, however, the patient does not have strep throat, then 90 of the time the lab says NO. But 10 of the time it says YES. You send five successive swabs to the lab, from the same patient. You get back these results, in order: YES, NO, YES, NO, YES. From that, which of the following can you conclude and why? 1. The results are worthless. 2. It is likely that the patient does not have strep throat. 3. It is more likely than not that the patient does have strep throat. Copyright 2015 Kee Siong Ng. All rights reserved. 24
30 Session Review Bayes rule is a useful tool in probability-type problems. The maximum a posteriori (MAP) estimator can be interpreted as the shortest description of the data. A weighted average of the models (Bayesian optimal estimator) is better than the MAP estimator. Copyright 2015 Kee Siong Ng. All rights reserved. 25
31 Review of Session 2 Bayesian optimal estimators are what we want ideally but they are usually very hard to compute. We next look at a (rare) case where the Bayesian optimal estimator can be computed cheaply. Copyright 2015 Kee Siong Ng. All rights reserved. 1
Discrete Mathematics and Probability Theory Spring 2014 Anant Sahai Note 10
EECS 70 Discrete Mathematics and Probability Theory Spring 2014 Anant Sahai Note 10 Introduction to Basic Discrete Probability In the last note we considered the probabilistic experiment where we flipped
MODULE -4 BAYEIAN LEARNING
MODULE -4 BAYEIAN LEARNING CONTENT Introduction Bayes theorem Bayes theorem and concept learning Maximum likelihood and Least Squared Error Hypothesis Maximum likelihood Hypotheses for predicting probabilities
A Short Introduction to the Theory of Statistical Learning. Kee Siong Ng, PhD
A Short Introduction to the Theory of Statistical Learning Kee Siong Ng, PhD keesiong.ng@gmail.com Outline Introduction Bayesian Probability Theory Sequence Prediction and Data Compression Bayesian Networks
Bayesian Learning Extension
Bayesian Learning Extension This document will go over one of the most useful forms of statistical inference known as Baye s Rule several of the concepts that extend from it. Named after Thomas Bayes this
Lecture 9: Bayesian Learning
Lecture 9: Bayesian Learning Cognitive Systems II - Machine Learning Part II: Special Aspects of Concept Learning Bayes Theorem, MAL / ML hypotheses, Brute-force MAP LEARNING, MDL principle, Bayes Optimal
Formalizing Probability. Choosing the Sample Space. Probability Measures
Formalizing Probability Choosing the Sample Space What do we assign probability to? Intuitively, we assign them to possible events (things that might happen, outcomes of an experiment) Formally, we take
More Information! Conditional Probability and Inverse Methods Peter Zoogman Harvard Atmospheres Journal Club April 12, 2012
More Information! Conditional Probability and Inverse Methods Peter Zoogman Harvard Atmospheres Journal Club April 12, 2012 Bayesian Analysis not very intuitive! Humans are bad at combining a priori information
Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 14
CS 70 Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 14 Introduction One of the key properties of coin flips is independence: if you flip a fair coin ten times and get ten
2.4 Conditional Probability
2.4 Conditional Probability The probabilities assigned to various events depend on what is known about the experimental situation when the assignment is made. Example: Suppose a pair of dice is tossed.
From inductive inference to machine learning
From inductive inference to machine learning ADAPTED FROM AIMA SLIDES Russel&Norvig:Artificial Intelligence: a modern approach AIMA: Inductive inference AIMA: Inductive inference 1 Outline Bayesian inferences
CSCE 478/878 Lecture 6: Bayesian Learning
Bayesian Methods Not all hypotheses are created equal (even if they are all consistent with the training data) Outline CSCE 478/878 Lecture 6: Bayesian Learning Stephen D. Scott (Adapted from Tom Mitchell
Consider an experiment that may have different outcomes. We are interested to know what is the probability of a particular set of outcomes.
CMSC 310 Artificial Intelligence Probabilistic Reasoning and Bayesian Belief Networks Probabilities, Random Variables, Probability Distribution, Conditional Probability, Joint Distributions, Bayes Theorem
Bayesian Learning. Artificial Intelligence Programming. 15-0: Learning vs. Deduction
15-0: Learning vs. Deduction Artificial Intelligence Programming Bayesian Learning Chris Brooks Department of Computer Science University of San Francisco So far, we ve seen two types of reasoning: Deductive
Statistical learning. Chapter 20, Sections 1 4 1
Statistical learning Chapter 20, Sections 1 4 Chapter 20, Sections 1 4 1 Outline Bayesian learning Maximum a posteriori and maximum likelihood learning Bayes net learning ML parameter learning with complete
Bayesian Learning. Reading: Tom Mitchell, Generative and discriminative classifiers: Naive Bayes and logistic regression, Sections 1-2.
Bayesian Learning Reading: Tom Mitchell, Generative and discriminative classifiers: Naive Bayes and logistic regression, Sections 1-2. (Linked from class website) Conditional Probability Probability of
Sampling Based Filters
Statistical Techniques in Robotics (6-83, F4) Lecture#3 (Tuesday September 2) Sampling Based Filters Lecturer: Drew Bagnell Scribe:Haley Dalzell, Matt Barnes, De-An Huang Monty Hall Problem The Monty Hall
Probability Pr(A) 0, for any event A. 2. Pr(S) = 1, for the sample space S. 3. If A and B are mutually exclusive, Pr(A or B) = Pr(A) + Pr(B).
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this
Intermediate Math Circles November 15, 2017 Probability III
Intermediate Math Circles November 5, 07 Probability III Example : You have bins in which there are coloured balls. The balls are identical except for their colours. The contents of the containers are:
Applied Logic. Lecture 4 part 2 Bayesian inductive reasoning. Marcin Szczuka. Institute of Informatics, The University of Warsaw
Applied Logic Lecture 4 part 2 Bayesian inductive reasoning Marcin Szczuka Institute of Informatics, The University of Warsaw Monographic lecture, Spring semester 2017/2018 Marcin Szczuka (MIMUW) Applied
Probability, Statistics, and Bayes Theorem Session 3
Probability, Statistics, and Bayes Theorem Session 3 1 Introduction Now that we know what Bayes Theorem is, we want to explore some of the ways that it can be used in real-life situations. Often the results
Introduction to Bayesian Learning. Machine Learning Fall 2018
Introduction to Bayesian Learning Machine Learning Fall 2018 1 What we have seen so far What does it mean to learn? Mistake-driven learning Learning by counting (and bounding) number of mistakes PAC learnability
Mathematical Foundations of Computer Science Lecture Outline October 18, 2018
Mathematical Foundations of Computer Science Lecture Outline October 18, 2018 The Total Probability Theorem. Consider events E and F. Consider a sample point ω E. Observe that ω belongs to either F or
Statistical learning. Chapter 20, Sections 1 3 1
Statistical learning Chapter 20, Sections 1 3 Chapter 20, Sections 1 3 1 Outline Bayesian learning Maximum a posteriori and maximum likelihood learning Bayes net learning ML parameter learning with complete
Joint, Conditional, & Marginal Probabilities
Joint, Conditional, & Marginal Probabilities The three axioms for probability don t discuss how to create probabilities for combined events such as P [A B] or for the likelihood of an event A given that
Introduction to Machine Learning. Lecture 2
Introduction to Machine Learning Lecturer: Eran Halperin Lecture 2 Fall Semester Scribe: Yishay Mansour Some of the material was not presented in class (and is marked with a side line) and is given for
Topics. Bayesian Learning. What is Bayesian Learning? Objectives for Bayesian Learning
Topics Bayesian Learning Sattiraju Prabhakar CS898O: ML Wichita State University Objectives for Bayesian Learning Bayes Theorem and MAP Bayes Optimal Classifier Naïve Bayes Classifier An Example Classifying
Bayesian Learning. CSL603 - Fall 2017 Narayanan C Krishnan
Bayesian Learning CSL603 - Fall 2017 Narayanan C Krishnan ckn@iitrpr.ac.in Outline Bayes Theorem MAP Learners Bayes optimal classifier Naïve Bayes classifier Example text classification Bayesian networks
The rest of the course
The rest of the course Subtleties involved with maximizing expected utility: Finding the right state space: The wrong state space leads to intuitively incorrect answers when conditioning Taking causality
Probabilistic Machine Learning. Industrial AI Lab.
Probabilistic Machine Learning Industrial AI Lab. Probabilistic Linear Regression Outline Probabilistic Classification Probabilistic Clustering Probabilistic Dimension Reduction 2 Probabilistic Linear
CS1800 Probability 3: Bayes Rule. Professor Kevin Gold
CS1800 Probability 3: Bayes Rule Professor Kevin Gold Brief Review of Probability - Counting Outcomes and Axioms Recall from before that we can calculate probabilities by counting success outcomes and
Conditional Probability P( )
Conditional Probability P( ) 1 conditional probability where P(F) > 0 Conditional probability of E given F: probability that E occurs given that F has occurred. Conditioning on F Written as P(E F) Means
Bayesian Learning. Examples. Conditional Probability. Two Roles for Bayesian Methods. Prior Probability and Random Variables. The Chain Rule P (B)
Examples My mood can take 2 possible values: happy, sad. The weather can take 3 possible vales: sunny, rainy, cloudy My friends know me pretty well and say that: P(Mood=happy Weather=rainy) = 0.25 P(Mood=happy
Bayesian Learning. Two Roles for Bayesian Methods. Bayes Theorem. Choosing Hypotheses
Bayesian Learning Two Roles for Bayesian Methods Probabilistic approach to inference. Quantities of interest are governed by prob. dist. and optimal decisions can be made by reasoning about these prob.
Lecture 2. Conditional Probability
Math 408 - Mathematical Statistics Lecture 2. Conditional Probability January 18, 2013 Konstantin Zuev (USC) Math 408, Lecture 2 January 18, 2013 1 / 9 Agenda Motivation and Definition Properties of Conditional
Elementary Discrete Probability
Elementary Discrete Probability MATH 472 Financial Mathematics J Robert Buchanan 2018 Objectives In this lesson we will learn: the terminology of elementary probability, elementary rules of probability,
Confusion matrix. a = true positives b = false negatives c = false positives d = true negatives 1. F-measure combines Recall and Precision:
Confusion matrix classifier-determined positive label classifier-determined negative label true positive a b label true negative c d label Accuracy = (a+d)/(a+b+c+d) a = true positives b = false negatives
Learning with Probabilities
Learning with Probabilities CS194-10 Fall 2011 Lecture 15 CS194-10 Fall 2011 Lecture 15 1 Outline Bayesian learning eliminates arbitrary loss functions and regularizers facilitates incorporation of prior
Predictive MDL & Bayes
Predictive MDL & Bayes Marcus Hutter Canberra, ACT, 0200, Australia http://www.hutter1.net/ ANU RSISE NICTA Marcus Hutter - 2 - Predictive MDL & Bayes Contents PART I: SETUP AND MAIN RESULTS PART II: FACTS,
MACHINE LEARNING INTRODUCTION: STRING CLASSIFICATION
MACHINE LEARNING INTRODUCTION: STRING CLASSIFICATION THOMAS MAILUND Machine learning means different things to different people, and there is no general agreed upon core set of algorithms that must be
Statistical learning. Chapter 20, Sections 1 3 1
Statistical learning Chapter 20, Sections 1 3 Chapter 20, Sections 1 3 1 Outline Bayesian learning Maximum a posteriori and maximum likelihood learning Bayes net learning ML parameter learning with complete
Philosophy 148 Announcements & Such. Independence, Correlation, and Anti-Correlation 1
Branden Fitelson Philosophy 148 Lecture 1 Branden Fitelson Philosophy 148 Lecture 2 Philosophy 148 Announcements & Such Independence, Correlation, and Anti-Correlation 1 Administrative Stuff Raul s office
Bayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
ECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
2 Chapter 2: Conditional Probability
STAT 421 Lecture Notes 18 2 Chapter 2: Conditional Probability Consider a sample space S and two events A and B. For example, suppose that the equally likely sample space is S = {0, 1, 2,..., 99} and A
Model Averaging (Bayesian Learning)
Model Averaging (Bayesian Learning) We want to predict the output Y of a new case that has input X = x given the training examples e: p(y x e) = m M P(Y m x e) = m M P(Y m x e)p(m x e) = m M P(Y m x)p(m
Probability Based Learning
Probability Based Learning Lecture 7, DD2431 Machine Learning J. Sullivan, A. Maki September 2013 Advantages of Probability Based Methods Work with sparse training data. More powerful than deterministic
Philosophy 148 Announcements & Such. Axiomatic Treatment of Probability Calculus I. Axiomatic Treatment of Probability Calculus II
Branden Fitelson Philosophy 148 Lecture 1 Branden Fitelson Philosophy 148 Lecture 2 Philosophy 148 Announcements & Such Administrative Stuff Raul s office is 5323 Tolman (not 301 Moses). We have a permanent
Philosophy 148 Announcements & Such
Branden Fitelson Philosophy 148 Lecture 1 Philosophy 148 Announcements & Such Administrative Stuff Raul s office is 5323 Tolman (not 301 Moses). We have a permanent location for the Tues. section: 206
Maximizing expected utility
Maximizing expected utility Earlier we looked at many decision rules: maximin minimax regret principle of insufficient reason... The most commonly used rule (and the one taught in business schools!) is
P (A B) P ((B C) A) P (B A) = P (B A) + P (C A) P (A) = P (B A) + P (C A) = Q(A) + Q(B).
Lectures 7-8 jacques@ucsdedu 41 Conditional Probability Let (Ω, F, P ) be a probability space Suppose that we have prior information which leads us to conclude that an event A F occurs Based on this information,
1 A Lower Bound on Sample Complexity
COS 511: Theoretical Machine Learning Lecturer: Rob Schapire Lecture #7 Scribe: Chee Wei Tan February 25, 2008 1 A Lower Bound on Sample Complexity In the last lecture, we stopped at the lower bound on
Independence. Conditional Probability. Conditional Probability. Independence. SCONE Lab. Definition: Two events E and F are independent iff
Independence Definition: Two events E and F are independent iff Seoul National University Example Pr(E F) = Pr(E) Pr(F) Flip a coin twice Event E: First trial is heads Event F: Second trial is tails Pr(E)
Bayesian Learning. Chapter 6: Bayesian Learning. Bayes Theorem. Roles for Bayesian Methods. CS 536: Machine Learning Littman (Wu, TA)
Bayesian Learning Chapter 6: Bayesian Learning CS 536: Machine Learning Littan (Wu, TA) [Read Ch. 6, except 6.3] [Suggested exercises: 6.1, 6.2, 6.6] Bayes Theore MAP, ML hypotheses MAP learners Miniu
Tutorial 1 : Probabilities
Lund University ETSN01 Advanced Telecommunication Tutorial 1 : Probabilities Author: Antonio Franco Emma Fitzgerald Tutor: Farnaz Moradi January 11, 2016 Contents I Before you start 3 II Exercises 3 1
COMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 2. Statistical Schools Adapted from slides by Ben Rubinstein Statistical Schools of Thought Remainder of lecture is to provide
ECE 450 Lecture 3. Overview: Randomly Selected Experiments & Hypothesis Testing. Digital Communications Examples. Many General Probability Examples
ECE 450 Lecture 3 Overview: Randomly Selected Experiments & Hypothesis Testing Digital Communications Examples Many General Probability Examples D. van Alphen Review & Practice: Conditional Probability
Joint, Conditional, & Marginal Probabilities
Joint, Conditional, & Marginal Probabilities Statistics 110 Summer 2006 Copyright c 2006 by Mark E. Irwin Joint, Conditional, & Marginal Probabilities The three axioms for probability don t discuss how
A quantum probability explanation for the inverse fallacy
A quantum probability explanation for the inverse fallacy Riccardo Franco Jerome Busemeyer Indiana University Send Correspondence to Riccardo Franco, riccardo.franco@polito.it Abstract This article presents
Machine Learning. CS Spring 2015 a Bayesian Learning (I) Uncertainty
Machine Learning CS6375 --- Spring 2015 a Bayesian Learning (I) 1 Uncertainty Most real-world problems deal with uncertain information Diagnosis: Likely disease given observed symptoms Equipment repair:
Lecture 3. January 7, () Lecture 3 January 7, / 35
Lecture 3 January 7, 2013 () Lecture 3 January 7, 2013 1 / 35 Outline This week s lecture: Fast review of last week s lecture: Conditional probability. Partition, Partition theorem. Bayes theorem and its
Probabilistic models
Kolmogorov (Andrei Nikolaevich, 1903 1987) put forward an axiomatic system for probability theory. Foundations of the Calculus of Probabilities, published in 1933, immediately became the definitive formulation
Great Theoretical Ideas in Computer Science
15-251 Great Theoretical Ideas in Computer Science Probability Theory: Counting in Terms of Proportions Lecture 10 (September 27, 2007) Some Puzzles Teams A and B are equally good In any one game, each
Bi=[X=i] i=i,z, forma partition. Example
![Bi=[X=i] i=i,z, forma partition. Example](/thumbs/91/107636497.jpg "Bi=[X=i] i=i,z, forma partition. Example")
Example Suppose that a person plays a game in which his score must be one of the 50 numbers 1, 2,,50 and that each of these 50 numbers is equally likely to be his score The first time he plays the game,
Homework 4 solutions
Homework 4 solutions Due Thu, Sept 25, 2003 Problem 2 p.104 Certainly for i < 7 we know that this conditional probability must be zero (there aren t any negative numbers on the dice). i = 7 The first die
ECEN 303: Homework 3 Solutions
Problem 1: ECEN 0: Homework Solutions Let A be the event that the dog was lost in forest A and A c be the event that the dog was lost in forest B. Let D n be the event that the dog dies on the nth day.
Chapter 7 Probability Basics
Making Hard Decisions Chapter 7 Probability Basics Slide 1 of 62 Introduction A A,, 1 An Let be an event with possible outcomes: 1 A = Flipping a coin A = {Heads} A 2 = Ω {Tails} The total event (or sample
CS 361: Probability & Statistics
February 12, 2018 CS 361: Probability & Statistics Random Variables Monty hall problem Recall the setup, there are 3 doors, behind two of them are indistinguishable goats, behind one is a car. You pick
Introduction to Statistical Learning Theory
Introduction to Statistical Learning Theory In the last unit we looked at regularization - adding a w 2 penalty. We add a bias - we prefer classifiers with low norm. How to incorporate more complicated
STAT509: Probability
University of South Carolina August 20, 2014 The Engineering Method and Statistical Thinking The general steps of engineering method are: 1. Develop a clear and concise description of the problem. 2. Identify
Statistical Methods for Astronomy
Statistical Methods for Astronomy Probability (Lecture 1) Statistics (Lecture 2) Why do we need statistics? Useful Statistics Definitions Error Analysis Probability distributions Error Propagation Binomial
CS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Chapter 8&9: Classification: Part 3 Instructor: Yizhou Sun yzsun@ccs.neu.edu March 12, 2013 Midterm Report Grade Distribution 90-100 10 80-89 16 70-79 8 60-69 4
MAE 493G, CpE 493M, Mobile Robotics. 6. Basic Probability
MAE 493G, CpE 493M, Mobile Robotics 6. Basic Probability Instructor: Yu Gu, Fall 2013 Uncertainties in Robotics Robot environments are inherently unpredictable; Sensors and data acquisition systems are
Notes on Machine Learning for and
Notes on Machine Learning for 16.410 and 16.413 (Notes adapted from Tom Mitchell and Andrew Moore.) Choosing Hypotheses Generally want the most probable hypothesis given the training data Maximum a posteriori
Thinking with Probability
Thinking with Probability Thinking fast and slow Thinking by decomposing events formalising probability rules conditional probability Brain Teasers Tijms Problems, Chap 1 2 dice rolled; one is 6. What
9/12/17. Types of learning. Modeling data. Supervised learning: Classification. Supervised learning: Regression. Unsupervised learning: Clustering
Types of learning Modeling data Supervised: we know input and targets Goal is to learn a model that, given input data, accurately predicts target data Unsupervised: we know the input only and want to make
Machine Learning, Midterm Exam: Spring 2008 SOLUTIONS. Q Topic Max. Score Score. 1 Short answer questions 20.
10-601 Machine Learning, Midterm Exam: Spring 2008 Please put your name on this cover sheet If you need more room to work out your answer to a question, use the back of the page and clearly mark on the
Bayesian inference. Justin Chumbley ETH and UZH. (Thanks to Jean Denizeau for slides)
Bayesian inference Justin Chumbley ETH and UZH (Thanks to Jean Denizeau for slides) Overview of the talk Introduction: Bayesian inference Bayesian model comparison Group-level Bayesian model selection
With Question/Answer Animations. Chapter 7
With Question/Answer Animations Chapter 7 Chapter Summary Introduction to Discrete Probability Probability Theory Bayes Theorem Section 7.1 Section Summary Finite Probability Probabilities of Complements
C Examples of Bayes Calculations
There are two buckets containing poker chips. Bucket A has Bucket B has 700 red chips 300 blue chips 300 red chips 700 blue chips Flip a coin to choose a bucket at random. From the chosen bucket, select
Our learning goals for Bayesian Nets
Our learning goals for Bayesian Nets Probability background: probability spaces, conditional probability, Bayes Theorem, subspaces, random variables, joint distribution. The concept of conditional independence
COMP61011! Probabilistic Classifiers! Part 1, Bayes Theorem!
COMP61011 Probabilistic Classifiers Part 1, Bayes Theorem Reverend Thomas Bayes, 1702-1761 p ( T W ) W T ) T ) W ) Bayes Theorem forms the backbone of the past 20 years of ML research into probabilistic
Examples of frequentist probability include games of chance, sample surveys, and randomized experiments. We will focus on frequentist probability sinc
FPPA-Chapters 13,14 and parts of 16,17, and 18 STATISTICS 50 Richard A. Berk Spring, 1997 May 30, 1997 1 Thinking about Chance People talk about \chance" and \probability" all the time. There are many
Probability in Programming. Prakash Panangaden School of Computer Science McGill University
Probability in Programming Prakash Panangaden School of Computer Science McGill University Two crucial issues Two crucial issues Correctness of software: Two crucial issues Correctness of software: with
Probability. A and B are mutually exclusive
Probability The probability of an event is its true relative frequency, the proportion of times the event would occur if we repeated the same process over and over again.! A and B are mutually exclusive
(3) Review of Probability. ST440/540: Applied Bayesian Statistics
Review of probability The crux of Bayesian statistics is to compute the posterior distribution, i.e., the uncertainty distribution of the parameters (θ) after observing the data (Y) This is the conditional
Probabilistic models
Probabilistic models Kolmogorov (Andrei Nikolaevich, 1903 1987) put forward an axiomatic system for probability theory. Foundations of the Calculus of Probabilities, published in 1933, immediately became
Bayesian Uncertainty: Pluses and Minuses
Bayesian Uncertainty: Pluses and Minuses Rod White Thanks to Rob Willink, Blair Hall, Dave LeBlond Measurement Standards Laboratory New Zealand Outline: A simple statistical experiment (coin toss) Confidence
6.080 / Great Ideas in Theoretical Computer Science Spring 2008
MIT OpenCourseWare http://ocw.mit.edu 6.080 / 6.089 Great Ideas in Theoretical Computer Science Spring 2008 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
Latent Variable Models and Expectation Maximization
Latent Variable Models and Expectation Maximization Oliver Schulte - CMPT 726 Bishop PRML Ch. 9 2 4 6 8 1 12 14 16 18 2 4 6 8 1 12 14 16 18 5 1 15 2 25 5 1 15 2 25 2 4 6 8 1 12 14 2 4 6 8 1 12 14 5 1 15
Machine Learning
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 13, 2011 Today: The Big Picture Overfitting Review: probability Readings: Decision trees, overfiting
Generalization bounds
Advanced Course in Machine Learning pring 200 Generalization bounds Handouts are jointly prepared by hie Mannor and hai halev-hwartz he problem of characterizing learnability is the most basic question
Introduction to Bayesian Learning
Course Information Introduction Introduction to Bayesian Learning Davide Bacciu Dipartimento di Informatica Università di Pisa bacciu@di.unipi.it Apprendimento Automatico: Fondamenti - A.A. 2016/2017 Outline
Knowledge Discovery in Data: Overview. Naïve Bayesian Classification. .. Spring 2009 CSC 466: Knowledge Discovery from Data Alexander Dekhtyar..
Spring 2009 CSC 466: Knowledge Discovery from Data Alexander Dekhtyar Knowledge Discovery in Data: Naïve Bayes Overview Naïve Bayes methodology refers to a probabilistic approach to information discovery
Bayesian learning Probably Approximately Correct Learning
Bayesian learning Probably Approximately Correct Learning Peter Antal antal@mit.bme.hu A.I. December 1, 2017 1 Learning paradigms Bayesian learning Falsification hypothesis testing approach Probably Approximately
Quantitative Understanding in Biology 1.7 Bayesian Methods
Quantitative Understanding in Biology 1.7 Bayesian Methods Jason Banfelder October 25th, 2018 1 Introduction So far, most of the methods we ve looked at fall under the heading of classical, or frequentist
Probabilistic Graphical Models
Probabilistic Graphical Models Introduction. Basic Probability and Bayes Volkan Cevher, Matthias Seeger Ecole Polytechnique Fédérale de Lausanne 26/9/2011 (EPFL) Graphical Models 26/9/2011 1 / 28 Outline
CMPT Machine Learning. Bayesian Learning Lecture Scribe for Week 4 Jan 30th & Feb 4th
CMPT 882 - Machine Learning Bayesian Learning Lecture Scribe for Week 4 Jan 30th & Feb 4th Stephen Fagan sfagan@sfu.ca Overview: Introduction - Who was Bayes? - Bayesian Statistics Versus Classical Statistics
An Introduction to Bayesian Reasoning
Tilburg Center for Logic and Philosophy of Science (TiLPS) Tilburg University, The Netherlands EPS Seminar, TiLPS, 9 October 2013 Overview of the Tutorial This tutorial aims at giving you an idea of why
Lecture 3: More on regularization. Bayesian vs maximum likelihood learning
Lecture 3: More on regularization. Bayesian vs maximum likelihood learning L2 and L1 regularization for linear estimators A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting
PubH 7470: STATISTICS FOR TRANSLATIONAL & CLINICAL RESEARCH
PubH 7470: STATISTICS FOR TRANSLATIONAL & CLINICAL RESEARCH The First Step: SAMPLE SIZE DETERMINATION THE ULTIMATE GOAL The most important, ultimate step of any of clinical research is to do draw inferences;