Probability, Random Processes and Inference

Transcription

1 INSTITUTO POLITÉCNICO NACIONAL CENTRO DE INVESTIGACION EN COMPUTACION Laboratorio de Ciberseguridad Probability, Random Processes and Inference Dr. Ponciano Jorge Escamilla Ambrosio

2 Course Content 1.4. General Random Variables Continuous Random Variables and PDFs Cumulative Distribution Function Normal Random Variables Joint PDFs of Multiple Random Variables Conditioning The Continuous Bayes Rule The Strong Law of Large Numbers 2

3 General Random Variables Continuous random variables The velocity of a vehicle traveling along the highway Continuous random variables can take on any real value in an interval. possibly of infinite length, such as (0, ) or the entire real line. In this section the concepts and method for discrete r.v.s, such as expectation, PMF, and conditioning, for their continuous counterparts are introduced. 3

4 Probability Density Function Continuous random variable. A random variable is called continuous if there exists a non negative function f X, called the probability density function of X, or PDF, such that: For every subset B of the real line 4

5 Probability Density Function The probability that the value of X falls within an interval is: which can be interpreted as the area under the graph of the PDF. 5

6 Probability Density Function 6

7 Probability Density Function For any single value a, we have: For this reason, including or excluding the endpoints of an interval has no effect on its probability: 7

8 Probability Density Function To qualify as a PDF, a function f X must be: o nonnegative, i.e., f X (x) 0 for every x, o have the normalisation property: Graphically, this means that the entire area under the graph of the PDF must be equal to 1. 8

9 Discrete vs. continuous r.v.s. Recall that for a discrete r.v., the CDF jumps at every point in the support, and is flat everywhere else. In contrast, for a continuous r.v. the CDF increases smoothly. 9

10 Discrete vs. continuous r.v.s. For a continuous r.v. X with CDF, F X (x), the probability density function (PDF) of X is the derivative f X (x) of the CDF, given by f X (x) = F X (x). The support of X, and of its distribution, is the set of all x where f X (x) > 0. The PDF represents the density of probability at the point x. 10

11 Probability Density Function To get from the PDF back to the CDF we apply: Thus, analogous to how we obtained the value of a discrete CDF at x by summing the PMF over all values less than or equal to x; here we integrate the PDF over all values up to x, so the CDF is the accumulated area under the PDF. 11

12 Probability Density Function Since we can freely convert between the PDF and the CDF using the inverse operations of integration and differentiation, both the PDF and CDF carry complete information about the distribution of a continuous r.v. Thus the PDF completely specifies the behavior of continuous random variables. 12

13 Probability Density Function For an interval [x, x+ ] with very small length, we have: So we can view f X (x) as the probability mass per unit length near x. Even though a PDF is used to calculate event probabilities, f X (x) is not the probability of any particular event. In particular, it is not restricted to be less than or equal to one. 13

14 Probability Density Function An important way in which continuous r.v.s differ from discrete r.v.s is that for a continuous r.v. X, P(X = x) = 0 for all x. This is because P(X = x) is the height of a jump in the CDF at x, but the CDF of X has no jumps! Since the PMF of a continuous r.v. would just be 0 everywhere, we work with a PDF instead. 14

15 Probability Density Function The PDF is analogous to the PMF in many ways, but there is a key difference: for a PDF f X, the quantity f X (x) is not a probability, and in fact it is possible to have f X (x) > 1 for some values of x. To obtain a probability, we need to integrate the PDF. In summary: To get a desired probability, integrate the PDF over the appropriate range. 15

16 Examples of PDFs The Logistic distribution has CDF: To get the PDF, we differentiate the CDF, which gives: Example: 16

17 Examples of PDFs 17

18 Examples of PDFs The Rayleigh distribution has CDF: To get the PDF, we differentiate the CDF, which gives: Example: 18

19 Examples of PDFs 19

20 Examples of PDFs A continuous r.v. X is said to have Uniform distribution on the interval (a, b) if its PDF is: The CDF is the accumulated area under the PDF: 20

21 Examples of PDFs We denote this by X Unif(a, b). The Uniform distribution that we will most frequently use is the Unif(0, 1) distribution, also called the standard Uniform. The Unif(0, 1) PDF and CDF are particularly simple: f(x) = 1 and F(x) = x for 0 < x < 1. For a general Unif(a, b) distribution, the PDF is constant on (a, b), and the CDF is ramp-shaped, increasing linearly from 0 to 1 as x ranges from a to b. 21

22 Examples of PDFs For Uniform distributions, probability is proportional to length. 22

23 PDF Properties 23

24 Expected Value and Variance of a Continuous r.v. The expected value or expectation or mean of a continuous r.v. X is defined by: This sis similar to the discrete case except that the PMF is replaced by the PDF, and summation is replaced by integration. Its mathematical properties are similar to the discrete case. 24

25 Expected Value and Variance of a Continuous r.v. If X is a continuous random variable with given PDF, then any real-valued function Y = ɡ(X) of X is also a random variable. Note that Y can be a continuous r.v., but Y can also be discrete, e.g., ɡ(x) = 1 for x 0 and ɡ(x) = 0, otherwise. In either case, the mean of ɡ(X) satisfies the expected value rule: 25

26 Expected Value and Variance of a Continuous r.v. The nth moment of a continuous r.v. X is defined as E[X n ], the expected value of the random variable X n. The variance of X denoted as var(x), is defined as the expected value of the random variable (X - E[X n ]) 2 : 26

27 Expected Value and Variance of a Continuous r.v. Example. Consider a uniform PDF over an interval [a, b], its expectation is given by: 27

28 Expected Value and Variance of a Continuous r.v. Its variance is given as: 28

29 Expected Value and Variance of a Continuous r.v. The exponential continuous random variable has PDF: where is a positive parameter characterising the PDF, with 29

30 Expected Value and Variance of a Continuous r.v. The probability that X exceeds a certain value decreases exponentially. This is, for any a 0, we have: An exponential random variable can be a good model for the amount of time until an incident of interest takes place. a message arriving at a computer, some equipment breaking down, a light bulb burning out, etc. 30

31 Expected Value and Variance of a Continuous r.v. 31

32 Expected Value and Variance of a Continuous r.v. The mean of the exponential r.v. X is calculated by: 32

33 Expected Value and Variance of a Continuous r.v. The variance of the exponential r.v. X is calculated by: 33

34 Cumulative Distribution Functions The cumulative distribution function, CDF, of a random variable X is denoted as F X and provides the probability P(X x). In particular for every x we have: The CDF F X (x) accumulates probability up to the value of x. 34

35 Cumulative Distribution Functions Any random variable associated with a given probability model has CDF, regardless of whether it is discrete or continuous. {X x} is always an event and therefore has well-defined probability. 35

36 Cumulative Distribution Functions 36

37 Cumulative Distribution Functions 37

38 Cumulative Distribution Functions 38

39 Cumulative Distribution Functions 39

40 Normal Random Variables A continuous random variable X is normal or Gaussian or normally distributed if it has PDF of the form: where μ and σ are two scalar parameters characterising the PDF (abbreviated N(μ, σ 2 ), and referred to as normal density function), with σ assumed positive. 40

41 Normal Random Variables It can be verified that the normalisation property holds: N(1,1) 41

42 Normal Random Variables If X is N(μ, σ 2 ), then: E(X) = μ Proof: The PDF is symmetric about x = μ. If X is N(μ, σ 2 ), then: Var(X) = σ 2 Proof: 42

43 Normal Random Variables Its maximum value occurs at the mean value of its argument. It is symmetrical about the mean value. The points of maximum absolute slope occur at one standard deviation above and below the mean. Its maximum value is inversely proportional to its standard deviation. The limit as the standard deviation approaches zero is a unit impulse. 43

44 Normal Random Variables 44

45 Linear Function of a Normal Random Variable If X is a normal r.v. with mean and variance 2, and if a 0, b are scalars, then the random variable: Y = ax + b is also normal, with mean and variance: E[Y] = a + b, var(y) = a

46 Standard Normal Random Variables A normal random variable Y with zero mean and unit variance, N(0, 1), is said to be a standard normal. Its PDF and CDF are denoted by and, respectively: 46

47 Standard Normal Random Variables The PDF of a normal r.v. cannot be integrated in terms of the common elementary functions, and therefore the probabilities of X falling in various intervals are obtained from tables or by computer. Example, the Standard Normal Table. The table only provides the values of (y) for y 0, because the omitted values can be calculated using the symmetry of the PDF. 47

48 Standard Normal Random Variables 48

49 Standard Normal Random Variables 49

50 Standard Normal Random Variables It would be overwhelming to construct tables for all μ and σ values required in application. Standardise the r.v. Let X be a normal (Gaussian) random variable with mean μ and variance σ 2 values. We standardise X by defining a new random variable Y given by: 50

51 Standard Normal Random Variables Since Y is a linear function of X, it is normal, This means: Thus, Y is a standard normal random variable. This allows us to calculate the probability of any event defined in terms of X by redefining the event in terms of Y, and then using the standard normal table. 51

52 Standard Normal Random Variables Example 1: 52

53 Standard Normal Random Variables Example 2: The annual snowfall at a particular geographic location is modelled as a normal random variable with a mean = 60 inches and a standard deviation of = 20. What is the probability that this year s snowfall will be at least 80 inches? 53

54 Standard Normal Random Variables Solution: 54

55 Standard Normal Random Variables Example 3: (Height Distribution of Men). Assume that the height X, in inches, of a randomly selected man in a certain population is normally distributed with μ = 69 and σ = 2.6. Find 1. P(X < 72), 2. P(X > 72), 3. P(X < 66), 4. P( X μ < 3). 55

56 Standard Normal Random Variables The table gives (z) only for z 0, and for z < 0 we need to make use of the symmetry of the normal distribution. This implies that, for any z, P(Z < z) = P(Z > z). Thus, solution: 56

57 Standard Normal Random Variables 57

58 Standard Normal Random Variables Normal r.v.s. are often used in signal processing and communications engineering to model noise and unpredictable distortions of signals. Example: 58

59 Standard Normal Random Variables 59

60 Standard Normal Random Variables Solution: 60

61 Standard Normal Random Variables Three important benchmarks for the Normal distribution are the probabilities of falling within one, two, and three standard deviations of the mean. The % rule tells us that these probabilities are what the name suggests. ( % rule). If X N(μ, 2 ), then: Standardising 61

62 Standard Normal Random Variables Three important benchmarks for the Normal distribution are the probabilities of falling within one, two, and three standard deviations of the mean. The % rule tells us that these probabilities are what the name suggests. ( % rule). If X N(μ, 2 ), then: Standardising 62

63 Joint PDF of Multiple Random Variables Two continuous random variables associated with the same experiment are jointly continuous and can be described in terms of a joint PDF f X,Y if f X,Y is a nonnegative function that satisfies: for every subset B of the two-dimensional plane. The notation means that the integration is carried out over the set B. 63

64 Joint PDF of Multiple Random Variables In the particular case where B is a rectangle of the form B = {(x, y) a x b, c y d}, we have: If B is the entire two-dimensional plane, then we obtain the normalisation property: 64

65 Joint PDF of Multiple Random Variables To interpret the joint PDF, we let be a small positive number and consider the probability of a small rectangle. Then we have: so we can view f X,Y (a, c) as the probability per unit area in the vicinity of (a, c). 65

66 Joint PDF of Multiple Random Variables 66

67 Joint PDF of Multiple Random Variables The joint PDF contains all relevant probabilistic information on the random variables X, Y, and their dependencies. Therefore, the joint PDF allow us to calculate the probability of any event that can be defined in terms of these two random variables. 67

68 Marginals Marginal PDF. For continuous r.v.s X and Y with joint PDF f X,Y, the marginal PDF of X is: Similarly, the marginal PDF of Y is: 68

69 Marginals Marginalisation works analogously with any number of variables. For example, if we have the joint PDF of X, Y, Z,W but want the joint PDF of X,W, we just have to integrate over all possible values of Y and Z: Conceptually this is very easy just integrate over the unwanted variables to get the joint PDF of the wanted variables but computing the integral may or may not be difficult. 69

70 Marginals Example 1. 70

71 Marginals Example 1. 71

72 Joint CDFs If X and Y are two random variables associated with the same experiment, their joint CDF is defined by: If X and Y are described by a joint PDF f X,Y, then: 72

73 Joint PDF of Multiple Random Variables Conversely, if X and Y are continuous with joint CDF F X,Y their joint PDF is the derivative of the joint CDF with respect to x and y: 73

74 Joint CDF of Multiple Random Variables Let X and Y be described by a uniform PDF on the unit square. The joint CDF is given by: It can be verified that: for al (x, y) in the unit square. 74

75 Expectation If X and Y are jointly continuous random variables and ɡ is some function, then Z = ɡ (X, Y) is also a random variable. Thus the expected value rule applies: As an important special case, for any scalars a, b, and c, we have: 75

76 More than Two Random Variables The joint PDF of three random variables X, Y, and Z is defined in analogy with the case of two random variables. For example: For any set B. We have the relations such as: 76

77 More than Two Random Variables The expected value rule takes the form: If ɡ is linear, of the form ax +by + cz, then: 77

78 More than Two Random Variables 78

79 More than Two Random Variables 79

80 Conditioning The conditional PDF of a continuous random variable X, given an event A with P(A) 0, is defined as a nonnegative function f X A that satisfies: for any subset B of the real line. 80

81 Conditioning In particular, by letting B be the entire real line, we obtain the normalisation property: so that f X A is a legitimate PDF. 81

82 Conditioning In the important special case where we condition on an event of form {X A}, with P(X A) 0, the definition of conditional probabilities yields: By comparing with the earlier formula, it gives: 82

83 Conditioning 83

84 Conditioning Example 1. 84

85 Joint Conditional PDF Suppose that X and Y are jointly continuous random variables, with joint PDF f X,Y. If we condition on a positive probability event of the form C = {(X,Y) A}, we have: In this case, the conditional PDF of X, given this event, can be obtained from the formula: 85

86 Joint Conditional PDF A version of the total probability theorem, which involves conditional PDFs is given as: if the events A 1,, A n form a partition of the sample space, then: Using the total probability theorem: 86

87 Joint Conditional PDF Finally, the formula can be written as: We then take the derivative of both sides, with respect to x, and obtain the desired result. 87

88 Joint Conditional PDF Example 2. 88

89 Joint Conditional PDF Example 3. 89

90 Joint Conditional PDF To interpret the conditional PDF, let us fix some small positive numbers 1 and 2, and condition on the event B = {y Y y + 2 }. We have: 90

91 Joint Conditional PDF Therefore, f X Y (x y) 1 provides us with the probability that X belongs to a small interval [x, x + 1 ], given that Y belongs to a small interval [y, y + 2 ]. Since f X Y (x y) 1 does not depend on 2, we can think of the limiting case where 2 decreases to zero and write: And more generally: 91

92 Joint Conditional PDF The conditional probability PDF f X Y (x y) can be seen as a description of the probability law of X, given that the event {Y = y} has occurred. As in the discrete case, the PDF f X Y, together with the marginal PDF f y are sometimes used to calculate the joint PDF. This approach can also be used for modelling: instead of directly specifying f X Y, it is often natural to provide a probability law for Y, in terms of a PDF f Y, and then provide a conditional PDF f X Y (x y) for X, given any possible value y of Y. 92

93 Joint Conditional PDF Example 4. The speed of a typical vehicle that drives past a police radar is modelled as an exponentially distributed random variable X with mean 50 miles per hour. The police radar s measurement Y of the vehicle s speed has an error which is modeled as a normal random variable with zero mean and standard deviation equal to one tenth of the vehicle s speed. What is the joint PDF of X and Y? 93

94 Joint Conditional PDF Solution. We have f X (x) = (1/50)e -x/50, for x 0. Also, conditioned on X = x, the measurement Y has a normal PDF with mean x and variance x 2 /100. Therefore: Thus, for all x 0 and all y: 94

95 Conditional PDF for More Than Two r.v.s. Conditional PDF can be defined for the extension for the case of more than two random variables: The analogue multiplication rule is given as: 95

96 Conditional Expectation For a continuous random variable X, we define the conditional expectation E[X A] given an event A, similar to the unconditional case, except that we now need to use the conditional PDF f X A. Let X and Y be jointly continuous random variables, and let A be an event with P(A) 0, then the conditional expectation of X given the event A is defined by: 96

97 Conditional Expectation The conditional expectation of X given that Y = y is defined by: The expectation rule, for a function ɡ(x): and 97

98 Conditional Expectation Total expectation theorem: Let A 1, A 2,, A n be disjoint events that form a partition of the same space, and assume that P(A i ) 0 for all i. Then: Similarly: 98

99 Conditional Expectation There are natural analogues for the case of functions of several random variables. For example: And: 99

100 Conditional Expectation Example

101 Independence Two continuous random variable X and Y are independent if their joint PDF is the product of the marginal PDFs: Comparing with the formula f X,Y (x, y) = f X Y (x y)f Y (y), we see that independence is the same as the condition: or, symmetrically: 101

102 Independence For the case of more than three random variables, for example, we say that three random variables X, Y, and Z are independent if: 102

103 Independence Example. Independent Normal Random Variables. Let X and Y be independent normal random variables with means x, y, and variances 2 x, 2 y, respectively. Their joint PDF is of the form: This joint PDF has the shape of a bell cantered at ( x, y ), and whose width in the x and y directions is proportional to 2 x and 2 y, respectively. 103

104 Independence Additional insight into the form of the PDF can be get by considering its contours. i.e., sets of points ata which the PDF takes a constant value. These contours are described by an equation of the form: and are ellipses whose two axes are horizontal and vertical. If 2 x = 2 y, then the contours are circles. 104

105 Independence 105

106 Independence 106

107 Independence If X and Y are independent, then any two events of the form {X A} and {Y B} are independent: 107

108 Independence Independence implies that: The property: can be used to provide a general definition of independence between two random variables, e.g., if X is discrete and Y is continuous. 108

109 Independence Similarly than to the discrete case, if X and Y are independent, then: for any two functions ɡ and h. The variance of the sum of independent random variables is equal to the sum of their variances: 109

110 Summary of Independence 110

111 Summary of Independence 111

112 The continuous Bayes rule Inference problem: We have an unobserved random variable X with known PDF, and we obtain a measurement Y according to a conditional PDF f X Y. Given an observed value y of Y, the inference problem is to evaluate the conditional PDF f X Y (x y). 112

113 The continuous Bayes rule Thus, whatever information is provided by the event {Y = y} is captured by the conditional PDF f X Y (x y). It thus suffices to evaluate this PDF. From the formula f X f Y X = f X,Y = f Y f X Y, it follows: 113

114 The continuous Bayes rule Based on the normalisation property an equivalent expression is: 114

115 The continuous Bayes rule 115

116 The continuous Bayes rule 116

117 Sums of Independent Random Variables Convolution Let Z = X + Y, where X and Y are independent integer-valued random variables with PMFs p X and p Y, respectively. Then, for any integer z: The resulting PMF p Z is called the convolution of the PMFs of X and Y. 117

118 Covariance and Correlation The covariance of two random variables X and Y, denoted by cov(x, Y), is defined as: When cov(x, Y) = 0, we say X and Y are uncorrelated. A positive o negative covariance indicates that the values of X E[X] and Y E[Y] obtained in a single experiment tend to have the same or the opposite sign, respectively. 118

119 Covariance and Correlation 119

120 Covariance and Correlation Multiplying this out and using linearity, we have an equivalent expression: Covariance has the following key properties: 1. Cov(X,X) = Var(X). 2. Cov(X, Y ) = Cov(Y,X). 3. Cov(X, c) = 0 for any constant c. 4. Cov(aX, Y ) = acov(x, Y ) for any constant a. 120

121 Covariance and Correlation 5. Cov(X + Y,Z) = Cov(X,Z) + Cov(Y,Z). 6. Cov(X + Y,Z +W) = Cov(X,Z) + Cov(X,W) + Cov(Y,Z) + Cov(Y,W). 7. Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y ). For n r.v.s X 1,...,X n, 121

122 Covariance and Correlation The correlation coefficient (X,Y) of two random variables X and Y that have nonzero variances is defined as: It may be viewed as a normalised version of the covariance cov(x, Y). ranges from -1 to

123 Covariance and Correlation If > 0 (or < 0), then the values of X E[X] and Y E[Y] tend to have the same (or opposite, respectively) sign. The size of provides a normalized measure of the extent to which this is true. Always assuming that X and Y have positive variances, it cab be shown that = 1 (or = 1) if and only if there exists a positive (or negative, respectively) constant c such that: 123

124 Covariance and Correlation 124

125 Covariance and Correlation 125

126 The Weak Law of Large Numbers 126

127 The Central Limit Theorem 127

128 The Strong Law of Large Numbers 128