Review of probabilities

Transcription

1 CS 1675 Introduction to Machine Learning Lecture 5 Density estimation Milos Hauskrecht milos@pitt.edu 5329 Sennott Square Review of probabilities 1

2 robability theory Studies and describes random processes and their outcomes Random processes may result in multiple different outcomes Example 1: coin flip Outcome is either head or tail binary outcome Fair coin: outcomes are equally likely Example 2: sum of numbers obtained by rolling 2 dice Outcome number in between 2 to 12 Fair dices: outcome 2 is less likely then 3 robability theory Studies and describes random processes and their outcomes Random processes may have multiple different outcomes Example 3: height of a person Select randomly a person from your school/city and report her height Outcomes can be real numbers nd many others related to measurements, lotteries, etc 2

3 robabilities When the process is repeated many times outcomes occur with certain relative frequencies or probabilities Example 1: coin flip Fair coin: outcomes are equally likely robability of head is 0.5 and tail is 0.5 iased coin robability of head is 0.8 and tail is 0.2 Head outcome is 4 times more likely than tail robabilities When the process is repeated many times outcomes occur with certain relative frequencies or probabilities Example 2: sum of numbers obtained by rolling 2 dice Outcome number in between 2 to 12 Fair dice: outcome 2 is less likely then 3 4 is less likely then 3, etc 3

4 robability distribution function Discrete mutually exclusive outcomes the chance of outcomes is represented by a probability distribution function probability distribution function assigns a number between 0 and 1 to every outcome Example 1: coin flip iased coin robability of head is 0.8 and tail is 0.2 Head outcome is 4 time more likely than tail tail = coin head = What is the condition we need to satisfy? robability distribution function Discrete mutually exclusive outcomes the chance of outcomes is represented by a probability distribution function probability distribution function assigns a number between 0 and 1 to every outcome Example 1: coin flip iased coin robability of head is 0.8 and tail is 0.2 Head outcome is 4 time more likely than tail tail = coin head = What is the condition we need to satisfy? Sum of probabilities for discrete set of outcomes is 1 4

5 robability for real-valued outcomes When the process is repeated many times outcomes occur with certain relative frequencies or probabilities Example 3: height of a person Select randomly a person from your school/city and report her height Outcomes can be real numbers Different outcomes can be more or less likely Normal Gaussian density robability density function Real-valued outcomes the chance of outcomes is represented by a probability density function probability density function px Condition on px and 1? 5

6 robability density function Real-valued outcomes the chance of outcomes is represented by a probability density function probability density function px Conditions on px and 1? p x dx 1 robability density function Real-valued outcomes the chance of outcomes is represented by a probability density function probability density function px Can px values for some x be negatives? 6

7 robability density function Real-valued outcomes the chance of outcomes is represented by a probability density function probability density function px Can px values for some x be negatives? No robability density function Real-valued outcomes the chance of outcomes is represented by a probability density function probability density function px Can px values for some x be > 1? Remember we need p x dx 1 7

8 robability density function Real-valued outcomes the chance of outcomes is represented by a probability density function probability density function px Can px values for some x be > 1? Remember we need: p x dx 1 Yes Random variable Random variable = function that maps observed outcomes quantities to real valued outcomes inary random variables: Two outcomes mapped to 0,1 Example: Coin flip. Tail mapped to 0, Head mapped to 1 Note: Only one value for each outcome: either 0 or 1 probability of tail x 0 probability of head x 1 robability distribution: ssigns a probability to each possible outcome iased coin x = CS 2750 Machine Learning 8

9 Random variable Example: roll of a dice Outcomes =1,2,3,4,5,6 based on the roll of a die trivial map to the same number iased dice Example: x height of a person Real valued outcomes trivial map to the same number CS 2750 Machine Learning robability Let be an outcome event, and its complement. Then? CS 1571 Intro to I 9

10 robability Let be an event, and its complement. Then 1? robability Let be an event, and its complement. Then 1 0 False 0? 10

11 robability Let be an event, and its complement. Then 1 0 False 0 1 True 1 Joint probability Joint probability: Let and be two events. The probability of an event, occurring jointly, We can add more events, say,,,c C,, C 11

12 Independence Independence : Let, be two events. The events are independent if:,? Independence Independence : Let, be two events. The events are independent if:, 12

13 Conditional probability Conditional probability : Let, be two events. The conditional probability of given is defined as:? Conditional probability Conditional probability : Let, be two events. The conditional probability of given is defined as:, roduct rule: rewrite of the conditional probability, 13

14 14 ayes theorem ayes theorem Why?,, Density estimation

15 Density estimation Density estimation: is an unsupervised learning problem Goal: Learn a model that represent the relations among attributes in the data D D, D,.., D } Data: Di x i { 1 2 n a vector of attribute values ttributes: modeled by random variables X { X with 1, X2,, Xd} Continuous or discrete valued variables Density estimation: learn an underlying probability distribution model : p X p X, X,, X from D 1 2 d Data: Density estimation D { D1, D2,.., Dn} D x a vector of attribute values i i Objective: estimate the model of the underlying probability distribution over variables X, px, using examples in D true distribution n samples p X D D, D,.., D } { 1 2 n estimate pˆ X 15

16 Density estimation true distribution n samples p X D D, D,.., D } { 1 2 n estimate pˆ X Standard iid assumptions: Samples are independent of each other come from the same identical distribution fixed px Independently drawn instances from the same fixed distribution Density estimation Types of density estimation: arametric the distribution is modeled using a set of parameters pˆ X p X Example: mean and covariances of a multivariate normal Estimation: find parameters describing data D Non-parametric The model of the distribution utilizes all examples in D s if all examples were parameters of the distribution Examples: Nearest-neighbor 16

17 Learning via parameter estimation In this lecture we consider parametric density estimation asic settings: set of random variables X { X1, X2,, Xd} model of the distribution over variables in X with parameters : pˆ X Example: Gaussian distribution with mean and variance parameters Data D { 1 2 n D, D,.., D } Objective: find parameters such that p X fits data D the best Model ML arameter estimation pˆ X p X Θ Maximum likelihood ML max Find that maximizes likelihood D, D1, D2,.., Dn, D, D2, Dn, n Di, i1 Data D 1 log-likelihood ML arg max { 1 2 n D, D,.., D } p D, p D, log p D, log D, n i1 p D, arg max i Independent examples log p D, 17