Chapter 2. Continuous random variables

Transcription

1 Chapter 2 Continuous random variables

2 Outline Review of probability: events and probability Random variable Probability and Cumulative distribution function Review of discrete random variable Introduction to continuous random variable Expected values Variance and standard deviation Variance for 2-dim random variable Quantiles and Cumulative distribution function Standard continuous univariate distributions Uniform Exponential Normal

3 Review of Probability Probability considers an experiment before it is performed Probability is a measure of the chance that an event may occur in the experiment Tossing a coin or conducting an election survey is an example of an experiment An event is a subset of the sample space, the set of all possible outcomes Seeing tail in coin or a positive response from the survey is an event The legitimate questions are then: What is the probability of seeing tail in the experiment of tossing a coin, or what is the probability of getting a positive response in a survey?

4 The Axioms of Probability Mathematically, probability is a function P which assigns to each event A in the sample space Ω a number P(A) in [0,1] such that Axiom 1: P(A) 0 for all A Ω; Axiom 2: P(Ω) = 1; Axiom 3: P(A B) = P(A) + P(B) if A B = for any A,B Ω When an event has an associated probability to occur, this gives rise to uncertainty Uncertainty principle is the fundamental of statistical inference

5 An example of random variable A random variable X is a function from sample space Ω to real numbers R Exercise 221 (a) Experiment 1: In a presential election with two candidates M and O, the possible outcomes are Ω = {Candidate M wins, Candidate O wins} If we define a random variable X that maps from Ω to {0,1} Then the probability of the event {Candidate O wins} is equivalent to P(X = 1) (b) Experiment 2: A national air quality monitoring system automatically collects measurements of ozone level at designated sites The possible outcomes are Ω = {x : x 0} If we define a random variable X to be the numerical measurements, the probability that ozone level falls below a certain level c is given by P(X c)

6 Random variable A random variable X is a function that associates a unique number with each possible outcome of an experiment Associated with each random variable X is a probability distribution function that describes the chance of all possible outcomes of X Often in scientific investigation, X represents the variable of main interest that can be measured or observed All observable events are expressed in terms of a random variable

7 Distribution function In order to describe all possible outcomes of an experiment, we focus on an event of the basic form {X x} for fixed x, where x can take any value How how can one express a general event {a < X b} using the basic form with set operations? {a < X b} = {X b} {X a} c If we have a rule of assigning probability to an event of the basic form, then the probability of any event can be determined

8 Cumulative Distribution function For any univariate random variable X, cumulative distribution function, cdf, F X : R [0,1], is defined by F X (x) F(x) = P(X x) 1 F(b) F(a) Cumulative distribution function Moreover, F( ) = 0 and F( ) = 1 and F is a non-decreasing function An example is drawn in Figure 1 Note that for a b, 0 a b Figure: Example of cumulative distribution function x P(a < X b) = F(b) F(a) Why is it so?

9 Discrete random variable Discrete random variables are probability models describing the outcomes of experiments with countable sample space, often assigning only the integers or the non-negative integers Examples include college membership, exam grades(a,b,c,d,e), number of goals in a match, number of children in a family

10 Discrete random variable Discrete random variables are probability models describing the outcomes of experiments with countable sample space, often assigning only the integers or the non-negative integers Examples include college membership, exam grades(a,b,c,d,e), number of goals in a match, number of children in a family The probability distribution function F X (x) = P(X x) can be obtained by P(X x) = P(X int(x)) = int(x) k= P(X = x), where int(x) denotes the largest integer smaller than or equal to x, eg int(52) = 5, int(3) = 3, int(-21) = -2

11 Probability mass function The probability distribution of a discrete random variable X is characterised by probability mass function, pmf, p(x), where p(x) = P(X = x) The probability mass function p(x) satisfies 0 p(x) 1 for all x; x= p(x) = 1 For any event A, P(X A) = x A p(x) For example, P(a < X b) = P(X = a + 1) + P(X = a + 2) + + P(X = b) = p(a + 1) + p(a + 2) + + p(b)

12 Probability mass function P(a < X b) a x b Figure: Example of probability mass function Shaded area represents P(a < X b)

13 Example of discrete random variable Exercise 241 For a random variable X that takes values {0,1} with probabilities θ,1 θ, obtain P(X x) for all x 0 and plot the cumulative distribution function

14 0 if x < 0 P(X x) = θ if 0 x < 1 1 if x 1

15 Continuous random variable When the outcome of an experiment is a measurement on a continuous scale, such as ozone level measurements in the earlier example, the random variable is called continuous random variable Examples include

16 Continuous random variable When the outcome of an experiment is a measurement on a continuous scale, such as ozone level measurements in the earlier example, the random variable is called continuous random variable Examples include height, weight, direction, waiting times in the hospital, price of stock

17 Continuous random variable When the outcome of an experiment is a measurement on a continuous scale, such as ozone level measurements in the earlier example, the random variable is called continuous random variable Examples include height, weight, direction, waiting times in the hospital, price of stock Again, the cumulative distribution function is defined by F(x) = F X (x) = P(X x) However, if X is continuous random variable and hence P(X = x) = 0 for all x P(a < X < b) = P(a X < b) = P(a < X b) = P(a X b)

18 Probability density function The probability density function, pdf f (x) a continuous random variable X is defined by f (x) = d dx F X(x) so that it satisfies F X (x) = x f (u)du The probability density function f (x) satisfies f (x) 0 for all x; f (x)dx = 1 For any event A, P(X A) = x A f (x)dx

19 Interpretation of probability density function Due to results from calculus, P(a < X b) = F X (b) F X (a) = = b b a f (x)dx f (x)dx a f (x)dx

20 Probability density function f (x) P(a < X b) a b x Figure: Example of probability density function P(a < X b) is the area under the curve between x = a and x = b

21 Exercise 251 For a random variable X with cumulative distribution function { x if 0 x 1 F X (x) = 0 otherwise (a) Find P(03 < X 05) (b) Find the pdf of X

22 (a) P(03 < X 05) = F(05) F(03) = = 02 { 1 if 0 x 1 (b) f (x) = 0 if x < 0 or x > 1

23 Expectation If X is a discrete random variable with probability mass function p(x) on {0,1, }, then the expected value of X is µ X = E[X] = xp(x) x=0 If X is a continuous random variable with probability density function f (x) on (, ), then the expected value of X is µ X = E[X] = xf (x)dx

24 Expectations of functions of random variables Suppose Y = g(x) where g is a fixed function If X is a discrete random variable with probability mass function p(x) on {0,1, }, then the expected value of Y is µ Y = E[Y ] = g(x)p(x) x=0 If X is a continuous random variable with probability density function f (x) on (, ), then the expected value of Y is µ Y = E[Y ] = g(x)f (x)dx

25 Exercise 261 Let f (x) = exp( x) for all x 0 Find (i)e[x] (= µ X ) (ii)e[x 2 ] and (iii)e[(x µ X ) 2 ]

26 (i) µ X = E[X] = = [ x exp( x)] x exp( x)dx 0 exp( x) dx = 0 + [ exp( x)] 0 = 0 ( 1) = 1 (ii) E[X 2 ] = 0 x 2 exp( x)dx = [ x 2 exp( x)] 0 + 2x exp( x) dx = = 2 (iii) E[(X µ X ) 2 ] = = 0 (x 2 2x + 1)exp( x)dx = = (x 1) 2 exp( x)dx

27 Properties of Expected values Theorem If X has expectation E[X] and Y is a linear function of X as Y = ax + b then Y has expectation E[Y ] = ae[x] + b

28 Properties of Expected values Theorem If X has expectation E[X] and Y is a linear function of X as Y = ax + b then Y has expectation E[Y ] = ae[x] + b More generally, the following properties hold: E[g(X) + h(x)] = E[g(X)] + E[h(X)] (1) E[cg(X)] = ce[g(x)] (2) E[aX + b] = ae[x] + b (3) Note that we proved them in MATH 104 for discrete random variables

29 Using linear properties of expectation, we may compute E[(X a) 2 ] by E[(X a) 2 ] = E[X 2 2aX + a 2 ] = E[X 2 ] E[2aX] + E[a 2 ] = E[X 2 ] 2aE[X] + a 2

30 Variance and Standard Deviation If X is a random variable with expected value µ X = E[X], the variance of X is σx 2 = Var[X] = E[(X µ X ) 2 ] { = x=0 (x µ X) 2 p(x) discrete rv (x µ X) 2 f (x)dx continous rv The variance of X can be calculated as σ 2 X = E[X 2 ] µ 2 X The standard deviation of X is σ X = Var[X]

31 Variance and Standard Deviation: example Exercise 271 For f (x) = exp( x) for all x 0, Find σ X

32 From the previous example,σ 2 X = 1 so σ X = 1

33 Properties of Variance and Standard Deviation Theorem If Var[X] exists and Y = a + bx, then Var[Y ] = b 2 Var[X] Hence, the standard deviation of Y is σ Y = b σ X

34 Properties of Variance and Standard Deviation Theorem If Var[X] exists and Y = a + bx, then Var[Y ] = b 2 Var[X] Hence, the standard deviation of Y is σ Y = b σ X Why do you need to take the absolute value in the above expression?

35 Probability mass function Probability mass function y µ = 25 σ = 11 y µ = 083 σ = 083 x x Density Density µ = 25 σ = µ = 083 σ = x x Figure: Expectations and Standard deviations for discrete and continuous random variables

36 Correlation With multivariate data, we need to characterise dependence: The correlation between X and Y, denoted Corr(X,Y ), and defined as: The correlation between two random variables X and Y is Corr(X,Y ) = E[{X E[X]}{Y E[Y ]}] Var(X)Var(Y ) = E[(X µ X)(Y µ Y )] σ X σ Y

37 centre here spread spread Correlation near 0 loose clustering Correlation near 1 tight clustering centre here Figure: Mean and Variance are measures of location and scale; Correlation measures linear association between variables Measure of clustering around a straight line with a slope [ 1,1] Correlation is a scale free measure of linear dependence between two variables

38 Theorem If X and Y are independent, then Corr(X, Y ) = 0

39 Covariance Recall that the variance of a random variable X is given by Var(X) = E [ (X µ X ) 2] The covariance between two random variables X and Y is defined in a similar way: The covariance between random variables X and Y is Cov(X,Y ) = E [(X µ X )(Y µ Y )], so that Var(X) = Cov(X,X) and Cov(X,Y ) is the expected product of the deviations of each variable from its expected value

40 Covariance cont Exercise 281 Show that covariance is equivalent to Cov(X,Y ) = E[XY ] µ X µ Y

41 Cov(X,Y) = E[(X µ X )(Y µ Y )] = E[XY µ X Y µ Y X + µ X µ Y ] = E[XY ] µ X E[Y ] µ Y E[X] + µ X µ Y = E[XY ] µ X µ Y µ Y µ X + µ X µ Y = E[XY ] µ X µ Y

42 Cov(X,Y) = E[(X µ X )(Y µ Y )] = E[XY µ X Y µ Y X + µ X µ Y ] = E[XY ] µ X E[Y ] µ Y E[X] + µ X µ Y = E[XY ] µ X µ Y µ Y µ X + µ X µ Y = E[XY ] µ X µ Y We have the following interpretation: Cov(X,Y ) = ρσ X σ Y, where Corr(X,Y) = ρ, and σ X and σ Y are the square roots of the variances of X and Y respectively

43 Quantiles Often interest is in the values of a continuous random variable which are not exceeded with a given probability, eg income of lower 10% income tax payer or score of top 5% students Let X be a random variable and p any value such that 0 p 1 The the pth quantile of the distribution of X is the value x p that satisfies: P(X x p ) = p When p = 05, the quantitle x 05 is called median

44 pth Quantile See Figure 6 for visualisation 1 Cumulative distribution function p F(x) 0 x p x Figure: x p is the pth quantile obtained from cdf

45 Quartiles The quartiles of a distribution are the values at which we can cut the distribution into four equally likely slices: (x 025,x 05,x 075 ) Figure 7 shows quartiles in cdf and pdf Cumulative distribution function Density F(x) 05 f (x) 025 x(25) x(75) x 0 x(5) x(75) x Figure: Quartiles (x 025, x 05, x 075 ) shown on cdf and pdf respectively

46 Uniform distribution This distribution is used to model variables that can take any value on a fixed interval, when the probability of occurrence does not vary over the interval The pdf of a Uniform random variable X, distributed on the interval (a,b) is given by: 1 if a < x < b; f (x;θ) = b a 0 otherwise, where θ = (a,b) and Θ is the set of (a,b) such that < a < b < This is written as X Uniform(a,b)

47 0 1/(b a) P(a < X x 0 ) a x0 b x Figure: Pdf for Uniform(a, b) random variable Shaded area represents P(a < X x 0 )

48 Cumulative distribution function and quantiles Exercise 2101 For X Uniform(a,b), (i) Find the cdf and sketch its graph (ii) Find the mean µ X and variance σ 2 X (iii) Find the median and compare to the mean

49 (i) F(x) = x a 1 x a b a du = b a for a x b and 1 for x b

50 (i) F(x) = x a 1 x a b a du = b a for a x b and 1 for x b (ii) µ X = b a x 1 a+b b a dx = 2 and E[X 2 ] = b a x2 1 b a dx = a2 +ab+b 2 3 So σx 2 = E[X 2 ] µ 2 X = (a b)2 12

51 (i) F(x) = x a 1 x a b a du = b a for a x b and 1 for x b (ii) µ X = b a x 1 a+b b a dx = 2 and E[X 2 ] = b a x2 1 b a dx = a2 +ab+b 2 3 So σx 2 = E[X 2 ] µ 2 X = (a b)2 12 (iii) F(x) = x a b a = 05 so x 05 = a + 05(b a) = (a + b)/2, same as the mean

52 Exercise 2102 Numerically evaluate the pdf, cdf and the quantile function of a Uniform distribution dunif(05, -2, 2) # pdf Uniform(-2,2) at x=05, f(05 punif(03, 0, 1) # cdf of Uniform(0,1) at x=03, P x = seq(-1, 1, length=101) fx = dunif(x, -1,1) plot(x, fx) lines(x, fx)

53 Exponential distribution This distribution is often used to model variables that are the times until specific events happen when the events occur at random at a given rate over time The pdf of an Exponential random variable X is { θ exp( θx) for x > 0, f (x;θ) = 0 otherwise, where 0 < x and 0 < θ This is written as X Exponential(θ) and θ Θ = (0, )

54 f(x) P(X>1) x Figure: Pdfs for Exponential(θ) random variables for θ = 1

55 Shape of Exponential density function Exercise 2103 Old Q: How is the shape of the function related to the parameter θ? Which one has lower tail probability P(X > 10)? Find P(X > 1) when θ = 1

56 P(X > 1) = 1 exp( x) dx = [ exp( x)] 1 = 0 ( exp( 1)) = exp( 1) = 03678

57 Cumulative distribution function and quantiles The cdf of the Exponential(θ) distribution is { 0 if x 0, F(x) = 1 exp( θx) if x > 0 Exercise 2104 For X Exponential(θ), (i) Derive the cumulative distribution from the pdf (ii) Find the median (iii) The mean µ X of the Exponential(θ) is 1θ Compare the median and the mean Which one is larger? Why does the median differ from the mean?

58 (i) F(x) = x 0 f (u)du = x 0 θ exp( θu) du = 1 exp( θx) (ii) Solving F(x p ) = p gives x p = θ 1 log(1 p) 1 For median, p = 05 so the median is x 05 = 1 θ log 2 (iii) µ X = 1/θ > (1/θ)log 2 = x 05 As the distribution is not symmetric, the mean and the median are not the same The median is always smaller because there is higher concentration of probability on smaller values but with thin probability values are stretching to the far right

59 Exercise 2105 Numerically evaluate the pdf, cdf and the quantile function of an Exponential distribution dexp(3, rate=2) # pdf of Exponential(2) at x=3, f(3)= pexp(3, 5) # cdf of Exponential(5) at x=3, P(X<3)= x = seq(0, 4, length=100) fx = dexp(x, rate=2) plot(x, fx) lines(x, fx)

60 Exercise 2106 Suppose that a goal is scored at random in a fixed time of the cup final and the time until the event can be modelled by an Exponential distribution with rate parameter θ = 2/3 hours If the first goal has been scored just now, what is the probability of waiting time until the next goal is (i) more than 30 minutes (ii) between 30 and 50 minutes

61 Let X be the random variable of the waiting time Then X Exponential(2/3) and F(x) = 1 exp( (2/3)x) (i) P(X > 1/2) = 1 F(1/2) = exp( 2/3 1/2) = (ii) P(1/2 < X < 5/6) = F(5/6) F(1/2) = exp( 2/3 1/2) exp( 2/3 5/6) = 01428

62 Normal distribution: background quoted from gqview weblib/gausshtml The normal distribution was introduced by the French mathematician Abraham De Moivre in 1733 De Moivre used this distribution to approximate probabilities of winning in various games of chance involving coin tossing It was later used by the German mathematician Karl Gauss to prredict the location of astronomical bodies and became known as the Gaussian distribution In the late nineteenth century statisticians started to believe that most data sets would have histograms with the Gaussian bell-shaped form and that all normal data sets would follow this form and so the curve came to be known as the normal curve

63 Normal distribution The pdf of a Normal random variable X is { f (x;θ) = 1 exp 1 ( ) } x µ 2, 2πσ 2 σ where θ = (µ,σ), < µ <, 0 < σ and < x < This is written as X N(µ,σ 2 ) and θ Θ = (, ) (0, ) E[X] = µ, Var(X) = σ 2

64 Shape of Normal density function Exercise 2107 How is the shape of the function related to the parameter θ? Which one has higher probability of P( X > 3)? sigma=05 sigma=1 sigma= x Figure: Pdfs for Normal(µ, σ 2 ) random variables where µ = 0 and σ = 05, 1, 15

65 The larger σ, the more spread So σ = 15 has the largest probability of P( X > 3) and σ = 05 has the smallest

66 Cumulative distribution function and quantiles The normal cdf is F(x) = x f (u)du = x 1 { exp 1 ( u µ ) 2 },du 2πσ 2 2 σ This does not have a closed form expression so numerical evaluation is required, if we want to obtain probabilities of the form P(X x) or quantiles

67 Exercise 2108 Numerically evaluate the pdf, cdf and the quantile function of a Normal distribution pnorm(0, mean=2, sd=sqrt(5)) # P(X<0) when X ~ N(2,5) pnorm(0, 2, sqrt(5)) # P(X<0) when X ~ N(2,5) pnorm(-2, 0, 2) # P(X >-2) when X ~ N(0,4) 0 qnorm(0975, 0, 1) # u such that P(X<u)=0975, 1 Note that the R functions for the Normal distribution use the standard deviation σ, not the variance σ 2

68 Exercise 2109 A normal distribution is proposed to model the variation in height of women with parameters µ = 160 and σ 2 = 25 measured in cm Find the proportion of tall women, defined as over 175cm tall

69 Let X be the random variable of women height then X Normal(160,5 2 ) So P(X > 175) = { exp 1 2π 5 2 ( x ) 2 } dx

70 Let X be the random variable of women height then X Normal(160,5 2 ) So P(X > 175) = { exp 1 2π 5 2 ( x ) 2 } dx In the above example we have expressed the proportion in terms of an integral and as the number of deviations from the mean The integral is impossible to calculate analytically so numerical evaluation is required to obtain probabilities or quantiles

71 Standardardization of the random variable It is useful to express such probabilities in terms of a standardized random variable, with µ = 0 and σ = 1 If X N(µ,σ 2 ) then Z = X µ σ N(0,1), and conversely if Z N(0, 1), then X = µ + σz N(µ,σ 2 )

72 Standardardization of the random variable It is useful to express such probabilities in terms of a standardized random variable, with µ = 0 and σ = 1 If X N(µ,σ 2 ) then Z = X µ σ N(0,1), and conversely if Z N(0, 1), then X = µ + σz N(µ,σ 2 ) The formal proof will be given in M230 and here it is sufficient to note that E[Z] = 0 Var[Z] = 1

73 Standard normal distribution A random variable Z is said to have a standard normal distribution with mean 0 and standard deviation 1 if its pdf is given by f (z) = 1 exp( z 2 /2), 2π where < z < and is denoted by Z N(0,1) The cumulative distribution function, ie the area under the curve, of the standard normal variable Z is given by Φ(z) = P(Z z) = z 1 2π exp( x 2 /2)dx Values of Φ(z) are obtained from a table of standard normal probabilities or from computer software such as R: pnorm(z) z Φ(z)

74 Standard normal distribution: example Exercise We repeat the previous example to illustrate the standardization procedure: P(X > 175) = P( X > ) 5 5 = P(Z > 3) = 1 P(Z 3) = 1 Φ(3) = = 00013

75 Probabilities of Normal distribution µ 3σ µ 2σ µ σ µ µ + σ µ + 2σ µ + 3σ P(µ σ < X < µ + σ) = 0683 P(µ 2σ < X < µ + 2σ) = 0954 P(µ 3σ < X < µ + 3σ) = 0997 Figure: Illustration of coverage probability of Normal(µ, σ 2 ) distribution