Random variables. DS GA 1002 Probability and Statistics for Data Science.

Transcription

1 Random variables DS GA 1002 Probability and Statistics for Data Science Carlos Fernandez-Granda

2 Motivation Random variables model numerical quantities that are uncertain They allow us to structure the information we have about these quantities in a principled way

3 Definition Given a probability space (Ω, F, P), a random variable X is a function from the sample space Ω to the real numbers R We use uppercase letters to denote random variables: X, Y,... Once the outcome ω Ω is revealed, X (ω) is the realization of X We use lowercase letters to denote numerical values: x, y,...

4 Characterization Given a probability space (Ω, F, P), for any set S P (X S) = P ({ω X (ω) S}). We will almost never construct probabilistic models like this!

5

6 Discrete random variables Discrete random variables take values on a finite or countably infinite subset of R such as the integers The probability mass function (pmf) of X is defined as p X (x) := P ({ω X (ω) = x}) In words, p X (x) is the probability that X equals x The pmf completely specifies a random variable

7 Probability mass function If D is the range of X, then ( D, 2 D ), p X is a valid probability space Any pmf satisfies p X (x) 0 for any x D, p X (x) = 1 x D P (X S) = x S p X (x) for any S D

8 Probability mass function px (x) x

9 Example P (X {1, 4}) P (X > 3)

10 Example P (X {1, 4}) = p X (1) + p X (4) = 0.5 P (X > 3)

11 Example P (X {1, 4}) = p X (1) + p X (4) = 0.5 P (X > 3) = p X (4) + p X (5) = 0.6

12 Defining a discrete random variable To define a discrete random variable X we just need A discrete range D A nonnegative function p X satisfying p X (x) = 1 x D

13 Bernoulli random variable Experiment with two possible outcomes (coin flip with bias p) p X (0) = 1 p p X (1) = p Special case: Indicator random variable of an event S { 1, if ω S, 1 S (ω) = 0, otherwise Bernoulli with parameter P (S) Allows to represent an event by a random variable

14 Example: Coin flips You flip a coin with bias p until you obtain heads (flips are independent) If you model the number of flips as a random variable X, what is p X?

15 Example: Coin flips p X (k)

16 Example: Coin flips p X (k) = P (k flips)

17 Example: Coin flips p X (k) = P (k flips) = P (1st flip = tails,..., k 1th flip = tails, kth flip = heads)

18 Example: Coin flips p X (k) = P (k flips) = P (1st flip = tails,..., k 1th flip = tails, kth flip = heads) = P (1st flip = tails) P (k 1th flip = tails) P (kth flip = heads)

19 Example: Coin flips p X (k) = P (k flips) = P (1st flip = tails,..., k 1th flip = tails, kth flip = heads) = P (1st flip = tails) P (k 1th flip = tails) P (kth flip = heads) = (1 p) k 1 p

20 Geometric random variable The pmf of a geometric random variable with parameter p is p X (k) = (1 p) k 1 p k = 1, 2,...

21 Geometric random variable p = px (k) k

22 Geometric random variable p = px (k) k

23 Geometric random variable p = px (k) k

24 Example: Coin flips You flip a coin with bias p n times (flips are independent) If you model the number of heads as a random variable X, what is p X?

25 Example: Coin flips What is the probability of getting k heads and then n k tails? P (k heads, then n k tails)

26 Example: Coin flips What is the probability of getting k heads and then n k tails? P (k heads, then n k tails) = P (1st = heads,..., kth = heads, k + 1th = tails,..., nth = tails)

27 Example: Coin flips What is the probability of getting k heads and then n k tails? P (k heads, then n k tails) = P (1st = heads,..., kth = heads, k + 1th = tails,..., nth = tails) = P (1st = heads) P (kth = heads) P (k + 1th = tails) P (nth = tails)

28 Example: Coin flips What is the probability of getting k heads and then n k tails? P (k heads, then n k tails) = P (1st = heads,..., kth = heads, k + 1th = tails,..., nth = tails) = P (1st = heads) P (kth = heads) P (k + 1th = tails) P (nth = tails) = p k (1 p) n k

29 Example: Coin flips Any fixed order of k heads and n k tails has the same probability

30 Example: Coin flips Any fixed order of k heads and n k tails has the same probability We are interested in the union of these events

31 Example: Coin flips Any fixed order of k heads and n k tails has the same probability We are interested in the union of these events Can we just add their probabilities?

32 Example: Coin flips Any fixed order of k heads and n k tails has the same probability We are interested in the union of these events Can we just add their probabilities? How many possible orders are there?

33 Example: Coin flips Any fixed order of k heads and n k tails has the same probability We are interested in the union of these events Can we just add their probabilities? How many possible orders are there? ( ) n := k n! k! (n k)!

34 Example: Coin flips Any fixed order of k heads and n k tails has the same probability We are interested in the union of these events Can we just add their probabilities? How many possible orders are there? ( ) n := k p X (k) = n! k! (n k)! ( ) n p k (1 p) n k k

35 Binomial random variable The pmf of a binomial random variable with parameters n and p is p X (k) = ( ) n p k (1 p) n k, k k = 0, 1, 2,..., n

36 Binomial random variable n = 20, p = px (k) k

37 Binomial random variable n = 20, p = px (k) k

38 Binomial random variable n = 20, p = px (k) k

39 Example: Call center Model the number of calls received per day Assumptions: 1. Each call occurs independently from every other call 2. A given call has the same probability of occurring at any given time of the day 3. Calls occur at a rate of λ calls per day

40 Example: Call center Discretize day into n slots

41 Example: Call center Discretize day into n slots Probability of receiving m calls in one slot?

42 Example: Call center Discretize day into n slots Probability of receiving m calls in one slot? (λ/n) m

43 Example: Call center Discretize day into n slots Probability of receiving m calls in one slot? (λ/n) m If n is large enough λ/n >> (λ/n) m for all m > 1

44 Example: Call center Discretize day into n slots Probability of receiving m calls in one slot? (λ/n) m If n is large enough λ/n >> (λ/n) m for all m > 1 Assume that in each slot we either receive one call or none at all. What is probability of k calls in a day?

45 Example: Call center Discretize day into n slots Probability of receiving m calls in one slot? (λ/n) m If n is large enough λ/n >> (λ/n) m for all m > 1 Assume that in each slot we either receive one call or none at all. What is probability of k calls in a day? Binomial with parameters n and p := λ/n!

46 Example: Call center P (k calls during the day )

47 Example: Call center P (k calls during the day ) = lim P (k calls in n small intervals) n

48 Example: Call center P (k calls during the day ) = lim P (k calls in n small intervals) n ( ) n = lim p k (1 p) (n k) n k

49 Example: Call center P (k calls during the day ) = lim n = lim n = lim n P (k calls in n small intervals) ( ) n p k (1 p) (n k) k ( n k ) ( λ n ) k ( 1 λ ) (n k) n

50 Example: Call center P (k calls during the day ) = lim n = lim n = lim n = lim n P (k calls in n small intervals) ( ) n p k (1 p) (n k) k ( n k ) ( λ n ) k ( 1 λ ) (n k) n n! λ k k! (n k)! (n λ) k ( 1 λ ) n n

51 Example: Call center P (k calls during the day ) = lim n = lim n = lim n = lim n P (k calls in n small intervals) ( ) n p k (1 p) (n k) k ( n k = λk e λ k! ) ( λ n ) k ( 1 λ ) (n k) n n! λ k k! (n k)! (n λ) k ( 1 λ ) n n Identity proved in the notes lim n n! (n k)! (n λ) k ( 1 λ ) n = e λ n

52 Poisson random variable The pmf of a Poisson random variable with parameter λ is p X (k) = λk e λ k! k = 0, 1, 2,...

53 Poisson random variable λ = px (k) k

54 Poisson random variable λ = px (k) k

55 Poisson random variable λ = px (k) k

56 Example: Call center Pmf of binomial with parameters n and p = λ n Poisson with parameter λ converges to pmf of This is an example of convergence in distribution

57 Binomial random variable n = 40, p = k

58 Binomial random variable n = 80, p = k

59 Binomial random variable n = 400, p = k

60 Poisson random variable λ = k

61 Call-center data Assumptions do not hold over the whole day (why?) They do hold (approximately) for intervals of time Example: Data from a call center in Israel We compare the histogram of the number of calls received in an interval of 4 hours over 2 months and the pmf of a Poisson random variable fitted to the data

62 Call-center data Real data Poisson distribution Number of calls

63

64 Continuous random variables Useful to model continuous quantities without discretizing Assigning nonzero probabilities to events of the form {X = x} for x R doesn t work! Instead, we only consider events of the form {X S} where S is a union of intervals (formally a Borel set) We cannot consider every possible subset of R for technical reasons

65 Cumulative distribution function The cumulative distribution function (cdf) of X is defined as F X (x) := P ({X (ω) x : ω Ω}) = P (X x) In words, F X (x) is the probability of X being smaller than x The cdf can be defined for both continuous and discrete random variables

66 Cumulative distribution function The cdf completely specifies the distribution of the random variable The probability of any interval (a, b] is given by P (a < X b) = P (X b) P (X a) = F X (b) F X (a) To define a continuous random variable we just need a valid cdf! A valid underlying probability space exists, but we don t need to worry about it

67 Properties of the cdf lim F X (x) = 0, x lim F X (x) = 1, x F X (b) F X (a) if b > a, i.e. F X is nondecreasing

68 Example 0 for x < x for 0 x 1 F X (x) := 0.5 ( for 1 x (x 2) 2) for 2 x 3 1 for x > 3

69 Example 1 FX (x) x

70 Example P (0.5 < X 2.5) =

71 Example P (0.5 < X 2.5) = F X (2.5) F X (0.5) = 0.375

72 Example 1 FX (x) 0.5 P(X (0.5, 2.5]) x

73 Probability density function When the cdf is differentiable, its derivative can be interpreted as a density Probability density function f X (x) := df X (x) d x The pdf is not a probability measure! (It can be greater than 1)

74 Probability density function By the fundamental theorem of calculus Intuitively, P (a < X b) = F X (b) F X (a) = b a f X (x) dx lim P (X (x, x + )) = f X (x) 0

75 Properties of the pdf For any union of intervals (any Borel set) S P (X S) = f X (x) dx In particular, S f X (x) dx = 1 From the monotonicity of the cdf f X (x) 0

76 Example 0 for x < x for 0 x 1 F X (x) := 0.5 ( for 1 x (x 2) 2) for 2 x 3 1 for x > 3 f X (x) = for x < 0, for 0 x 1 for 1 x 2 for 2 x 3 for x > 3.

77 Example 0 for x < x for 0 x 1 F X (x) := 0.5 ( for 1 x (x 2) 2) for 2 x 3 1 for x > 3 f X (x) = 0 for x < 0, for 0 x 1 for 1 x 2 for 2 x 3 for x > 3.

78 Example 0 for x < x for 0 x 1 F X (x) := 0.5 ( for 1 x (x 2) 2) for 2 x 3 1 for x > 3 0 for x < 0, 0.5 for 0 x 1 f X (x) = for 1 x 2 for 2 x 3 for x > 3.

79 Example 0 for x < x for 0 x 1 F X (x) := 0.5 ( for 1 x (x 2) 2) for 2 x 3 1 for x > 3 0 for x < 0, 0.5 for 0 x 1 f X (x) = 0 for 1 x 2 for 2 x 3 for x > 3.

80 Example 0 for x < x for 0 x 1 F X (x) := 0.5 ( for 1 x (x 2) 2) for 2 x 3 1 for x > 3 0 for x < 0, 0.5 for 0 x 1 f X (x) = 0 for 1 x 2 x 2 for 2 x 3 for x > 3.

81 Example 0 for x < x for 0 x 1 F X (x) := 0.5 ( for 1 x (x 2) 2) for 2 x 3 1 for x > 3 0 for x < 0, 0.5 for 0 x 1 f X (x) = 0 for 1 x 2 x 2 for 2 x 3 0 for x > 3.

82 Example FX (x) x fx (x) x

83 Example P (0.5 < X 2.5) =

84 Example P (0.5 < X 2.5) = f X (x) dx

85 Example P (0.5 < X 2.5) = = f X (x) dx 0.5 dx x 2 dx = 0.375

86 Example 1 P (X (0.5, 2.5]) fx (x) x

87 Uniform random variable Pdf of a uniform random variable with domain [a, b]: { 1 f X (x) = b a, if a x b, 0, otherwise

88 Uniform random variable in [a, b] 1 b a 1 fx (x) FX (x) 0 0 a x b a x b

89 Exponential random variable Used to model waiting times (time until a certain event occurs) Examples: decay of a radioactive particle, telephone call, mechanical failure of a device Pdf of an exponential random variable with parameter λ: { λe λx, if x 0, f X (x) = 0, otherwise

90 Exponential random variables 1.5 λ = 0.5 λ = 1.0 λ = 1.5 fx (x) x

91 Call-center data Example: Data from a call center in Israel We compare the histogram of the inter-arrival times between calls occurring between 8 pm and midnight over two days and the pdf of an exponential random variable fitted to the data

92 Call center Exponential distribution Real data Interarrival times (s)

93 Gaussian or normal random variable Extremely popular in probabilistic models and statistics Sums of independent random variables converge to Gaussian distributions under certain assumptions Pdf of a Gaussian random variable with mean µ and standard deviation σ f X (x) = 1 e (x µ)2 2σ 2 2πσ

94 Gaussian random variables µ = 2 σ = 1 µ = 0 σ = 2 µ = 0 σ = 4 fx (x) x

95 Height data Example: Data from a population of people We compare the histogram of the heights and the pdf of a Gaussian random variable fitted to the data

96 Height data Gaussian distribution Real data Height (inches)

97 Problem The Gaussian cdf does not have a closed form solution This complicates computing the probability that a Gaussian belongs to a set

98 Standard Gaussian If X is Gaussian with mean µ and standard deviation σ, then U := X µ σ is a standard Gaussian, with mean zero and unit standard deviation ( [ X µ a µ P (X [a, b]) = P σ σ, b µ ]) σ ( ) ( ) b µ a µ = Φ Φ σ σ Φ is the cdf of a standard Gaussian

99 Beta random variable Useful in Bayesian statistics Unimodal continuous distribution in the unit interval The pdf of a beta distribution with parameters a and b is defined as f β (θ; a, b) := { θ a 1 (1 θ) b 1 β(a,b), if 0 θ 1, 0 otherwise β (a, b) := u a 1 (1 u) b 1 du u

100 Beta random variables fx (x) a = 1 b = 1 a = 1 b = 2 a = 3 b = 3 a = 6 b = 2 a = 3 b = x

101

102 Conditioning on an event We usually define random variables using their pmf, cdf or pdf How can we incorporate the information that X S for some set S?

103 Conditional pmf If X has pmf p X, the conditional pmf of X given X S is p X X S (x) := P (X = x X S) { px (x) = s S p X (s) if x S 0 otherwise. Valid pmf in the new probability space restricted to the event {X S}

104 Conditional cdf If X has pdf f X, the conditional cdf of X given X S is F X X S (x) := P (X x X S) P (X x, X S) = P (X S) u x,u S = f X (u) du u S f X (u) du Valid cdf in the new probability space restricted to the event {X S}

105 Example: Geometric random variables are memoryless We flip a coin repeatedly until we obtain heads, but pause after k 0 flips (which were tails) What is the probability of obtaining heads in k more flips?

106 Example: Geometric random variables are memoryless P (k more flips)

107 Example: Geometric random variables are memoryless P (k more flips) = p X X >k0 (k)

108 Example: Geometric random variables are memoryless P (k more flips) = p X X >k0 (k) = p X (k) m=k 0 +1 p X (m)

109 Example: Geometric random variables are memoryless P (k more flips) = p X X >k0 (k) = = p X (k) m=k 0 +1 p X (m) (1 p) k 1 p m=k 0 +1 (1 p)m 1 p

110 Example: Geometric random variables are memoryless P (k more flips) = p X X >k0 (k) = = p X (k) m=k 0 +1 p X (m) (1 p) k 1 p m=k 0 +1 (1 p)m 1 p = (1 p) k k 0 1 p for k > k 0 Geometric series: m=k 0 +1 α m 1 = αk 0 1 α for any α < 1

111 Example: Exponential random variables are memoryless Assume inter-arrival times are exponential with parameter λ You get an , then no for t 0 minutes How is the waiting time until the next distributed now?

112 Example: Exponential random variables are memoryless F T T >t0 (t)

113 Example: Exponential random variables are memoryless F T T >t0 (t) = t t 0 f T (u) du f T (u) du t 0

114 Example: Exponential random variables are memoryless F T T >t0 (t) = = t t 0 f T (u) du f T (u) du t 0 t t 0 λe λu du λe λu du t 0

115 Example: Exponential random variables are memoryless F T T >t0 (t) = = t t 0 f T (u) du f T (u) du t 0 t t 0 λe λu du λe λu du t 0 = e λt e λt 0 e λt 0

116 Example: Exponential random variables are memoryless F T T >t0 (t) = = t t 0 f T (u) du f T (u) du t 0 t t 0 λe λu du λe λu du t 0 = e λt e λt 0 e λt 0 = 1 e λ(t t 0) for t > t 0

117 Example: Exponential random variables are memoryless F T T >t0 (t) = = t t 0 f T (u) du f T (u) du t 0 t t 0 λe λu du λe λu du t 0 = e λt e λt 0 e λt 0 = 1 e λ(t t 0) for t > t 0 Differentiating with respect to t f T T >t0 (t) = λe λ(t t 0) for t > t 0

118

119 Functions of random variables For any deterministic function g and r.v. X, Y := g (X ) is a random variable Formally, X maps elements of Ω to R, so Y does too since Y (ω) = g (X (ω))

120 Discrete random variables If X is discrete p Y (y) = P (Y = y) = P (g (X ) = y) = p X (x) {x g(x)=y}

121 Continuous random variables If X is continuous F Y (y) = P (Y y) = P (g (X ) y) = f X (x) dx, {x g(x) y} Then we can differentiate to obtain the pdf f Y

122 Gaussian random variable If X is a Gaussian random variable with mean µ and standard deviation σ, derive the distribution of U := X µ σ

123 Gaussian random variable F U (u)

124 Gaussian random variable ( ) X µ F U (u) = P u σ

125 Gaussian random variable ( X µ F U (u) = P σ = (x µ)/σ u ) u 1 2πσ e (x µ)2 2σ 2 dx

126 Gaussian random variable ( X µ F U (u) = P σ = = (x µ)/σ u u ) u 1 e (x µ)2 2σ 2 dx 2πσ 1 2π e w2 2 dw by the change of variables w = x µ σ

127 Gaussian random variable ( X µ F U (u) = P σ = = (x µ)/σ u u ) u 1 e (x µ)2 2σ 2 dx 2πσ 1 2π e w2 2 dw by the change of variables w = x µ σ To obtain the pdf we differentiate with respect to u

128 Gaussian random variable ( X µ F U (u) = P σ = = (x µ)/σ u u ) u 1 e (x µ)2 2σ 2 dx 2πσ 1 2π e w2 2 dw by the change of variables w = x µ σ To obtain the pdf we differentiate with respect to u f U (u) = 1 2π e u2 2

129 Gaussian random variable ( X µ F U (u) = P σ = = (x µ)/σ u u ) u 1 e (x µ)2 2σ 2 dx 2πσ 1 2π e w2 2 dw by the change of variables w = x µ σ To obtain the pdf we differentiate with respect to u f U (u) = 1 2π e u2 2 U is a standard Gaussian random variable

130

131 Generating random variables Simulation is crucial to leverage probabilistic models effectively (life is not a homework problem!) It requires being able to sample from arbitrary distributions General approach: 1. Generate samples uniformly from the unit interval [0, 1] 2. Transform the samples so that they have the desired distribution

132 Sampling from a discrete distribution Aim: Generate a discrete random variable X with pmf p X using samples from a uniform random variable U Possible values of X : x 1 < x 2 <... How can we assign samples u 1, u 2,... from U to x 1, x 2,...?

133 Sampling from a discrete distribution x 3 x 2 x 1 0 u 1 u 2 u 3 u 4 u 5 1

134 Sampling from a discrete distribution Idea: Assign samples in an interval of length p X (x i ) to x i

135 Sampling from a discrete distribution x 3 x 2 x 1 0 u 1 u 2 u 3 u 4 u 5 1

136 Sampling from a discrete distribution x 3 x 2 x 1 0 u 1 u 2 u 3 u 4 u 5 1

137 Sampling from a discrete distribution x 1 if 0 U p X (x 1 ) x 2 if p X (x 1 ) U p X (x 1 ) + p X (x 2 ) X =... x i if i 1 j=1 p X (x j ) U i j=1 p X (x j )...

138 Sampling from a discrete distribution x 1 if 0 U F X (x 1 ) x 2 if F X (x 1 ) U F X (x 2 ) X =... x i if F X (x i 1 ) U F X (x i )...

139 Sampling from a discrete distribution x 3 x 2 x 1 0 u 1 F X (x 1 ) u 2 u 3 u 4 F X (x 2 ) u 5 1

140 Inverse-transform sampling Aim: Generate a continuous random variable X with cdf F X using samples from a uniform random variable U Algorithm: 1. Obtain a sample u of U 2. Set x := F 1 X (u)

141 Inverse-transform sampling F Y (y)

142 Inverse-transform sampling F Y (y) = P (Y y)

143 Inverse-transform sampling F Y (y) = P (Y y) = P ( F 1 X (U) y)

144 Inverse-transform sampling F Y (y) = P (Y y) = P ( F 1 X (U) y) = P (U F X (y))

145 Inverse-transform sampling F Y (y) = P (Y y) = P ( F 1 X (U) y) = P (U F X (y)) = FX (y) u=0 du

146 Inverse-transform sampling F Y (y) = P (Y y) = P ( F 1 X (U) y) = P (U F X (y)) = FX (y) u=0 = F X (y) du

147 Generating an exponential random variable Aim: Generate an exponential random variable X with parameter λ F X (x) := 1 e λx F 1 X (u) = 1 ( ) 1 λ log 1 u F 1 X (U) is an exponential random variable with parameter λ

148 Generating an exponential random variable F 1 X (u 5) F 1 X (u 4) F 1 X (u 3) F 1 F 1 X (u 2) X (u 1) 0 u 1 u 2 u 3 u 4 u 51