Probability Theory and Random Variables One of the most noticeable aspects of many computer science related phenomena is the lack of certainty. When a job is submitted to a batch oriented computer system, the exact time the job will be completed is uncertain. The number of jobs submitted tomorrow is probably not known either. Similarly, the exact response time for an interactive inquiry system cannot be predicted. If several computers are attached to a local area network, some of them may try to communicate at almost the same time and thus cause a collision on the network. How often this will happen during a given period of time is a random number. In order to work with such observed, uncertain processes, we need to put them into a mathematical framework. This is the purpose of this chapter. To apply probability theory to the process under study, we view it as a random experiment, that is, as an experiment whose outcome is not known in advance but for which the set of all possible individual outcomes is known. The sample space Ω (Stichprobenraum, espace échantillon) of a random experiment is the set of all possible simple outcomes of the experiment. These possible outcomes are called elementary events (Elementarereignisse, événements élémentaires). Elementary events are mutually exclusive. Example 1 Tossing a fair die: Example 2 Tossing two fair dice, consider the sum of points: Example 3 Tossing a fair coin again and again until the first head appears: Example 4 Measuring the "response time" of a computer program: Sample spaces can be finite or infinite. They are also classified as discrete if the number of sample points is finite or countably infinite. A sample space is continuous if its elementary events consist of all the numbers of some finite or infinite interval of the real line.
Probability Theory and Random Variables 2 An event (Ereignis, événement) is defined as a subset of a sample space. The empty set represents the impossible event, whereas Ω stands for the certain event. Example 5 In Example 1 or Example 2, if A = {2,3,5}, then A is the event of tossing a prime number while B = {1,3,5} is the event of tossing an odd number. Example 6 In Example 4, A = {t 20 < t < 30} is the event that the response time is between 20 and 30 seconds. Note, that in general events are not mutually exclusive. E.g. in Example 5 the events A and B may appear simultaneously. If A and B are two events of a given experiment, then also A B and A B are events. The event A * = ### \ A is the complementary event to A. If A B = then A and B are mutually exclusive or incompatible. The probability (Wahrscheinlichkeit, probabilité) of an event is defined as a real function P[ ] which satisfies the following Axioms 1 (1) 0 P[A] for every event A (2) P[Ω] = 1 (3) P[A B] = P[A] + P[B] if the events A and B are mutually exclusive It is immediate from (3) that for any finite collection A 1, A 2,..., A n of mutually exclusive events P[A 1 A 2... A n ] = P[A 1 ] + P[A 2 ] +... + P[A n ] We assume as Axiom (4) that this remains true if n tends to infinity: (4) P[A 1 A 2 A 3...] = P[A 1 ] + P[A 2 ] + P[A 3 ] +... The above axioms lead to the following important consequences: Theorem 1 (a) P[ ] = 0 (b) (c) P[A * ] = 1 - P[A] for every event A P[A B] = P[A] + P[B] - P[A B] for any events A, B
Probability Theory and Random Variables 3 (d) A B implies P[A] P[B] for any events A, B Example 7 We toss two fair dice and consider the total number of spots as the result of the experiment. We define the following events: A : the result is a prime number B : the result is less than 5 The conditional probability (bedingte Wahrscheinlichkeit, probabilité conditionnelle) P[A B], read as "conditional probability of A, given B" is defined as P[A B] = Two events A and B are independent, if and only if P[A B] = P[A]. Using the above definition of the conditional probability immediately leads us to following equivalent definition: Two events A and B are independent, if and only if P[A B] = P[A] P[B] Check whether the events A and B in Example 7 are independent or not! The concept of two events A and B being independent should not be confused with the concept of their being mutually exclusive. In fact, if A and B are mutually exclusive then P[A B] = P[ ] = 0, however, if A and B are independent and P[A] 0 and P[B] 0 then P[A B] = P[A] P[B] 0.
Probability Theory and Random Variables 4 Hence, mutually exclusive events are not independent. One of the main uses of conditional probability is to assist in the calculation of unconditional probability by the use of the following theorem. Theorem 2 (Law of total probability) Let A 1, A 2, A 3,..., A n be events such that (a) (b) (c) A i A j = if i j (mutually exclusive events) P[Ai] > 0 for i = 1,2,3,...,n A 1 A 2... A n = Ω The family of events A 1, A 2, A 3,..., A n is then called a partition of Ω. For any event A, the probability P[A] can then be calculated as follows: P[A] = P[A 1 ] P[A A 1 ] + P[A 2 ] P[A A 2 ] +... + P[A n ] P[A A n ] Example 8 There are three coins in a box. The first one is fair. The second one is weighted so that heads is twice as likely to appear as tails, and the third one shows tails on both sides. A coin is randomly drawn from the box and tossed. Find the probability of heads. Another direct consequence of the definition of conditional probability is the following theorem: Theorem 3 (Bayes' Theorem) Suppose the events A 1, A 2, A 3,..., A n form a partition of Ω. Then for any event A with P[A] > 0
Probability Theory and Random Variables 5 P[A i A] = Example 9 In a certain college, 25 % of the male students and 15 % of the female students are studying computer science. 70 % of all students are men. A student is chosen at random. (a) Find the probability that the student is studying computer science. (b) If the student is studying computer science, find the probability that the student is female. Example 10 A red die and a white one are tossed simultaneously. Let A, B, and C be events defined as follows: A: the red die turns up odd, B: the white die turns up odd, C: the total number of spots of both dice is odd. The system of events A 1, A 2, A 3,..., A n is called independent if P[A i A j ] = P[A i ] P[A j ] P[A i A j A k ] = P[A i ] P[A j ] P[A k ] P[A 1 A 2... A n ] = P[A 1 ] P[A 2 ]... P[A n ] for all combinations of indices such that 1 i < j <... < k n.
Probability Theory and Random Variables 6 Reconsider the events in Example 10. Exercise 1 Let A and B be events with P[A] = 0.3, P[A B] = 0.4, and P[B] = p. Calculate p if (a) A and B are mutually exclusive, (b) A and B are independent, and (c) A is a subset of B. Exercise 2 A pair of fair dice is thrown. If the numbers appearing are different, find the probability that (a) the sum is 6, (b) the sum is 4 or less, (c) the sum is even. Exercise 3 Let three fair coins be tossed. Let A = "all heads or all tails", B = "at least two heads", and C = "at most two heads". Of the pairs (A,B), (A,C) and (B,C), which are independent? Is the system of the events A, B, C independent? Exercise 4 20 % of all HiQuHard computers are assembled on Mondays, 25 % of them on Tuesdays, Wednesdays and Thursdays, and 5% on Fridays. The company statistician has determined, that 4 % of the computers produced on Monday are "lemons", i.e. they are defective. 1 % of the computers made on Tuesday, Wednesday or Thursday are lemons, and 3 % of the computers manufactured on Friday are lemons. You find that your HiQuHard computer is a lemon. What is the probability it was manufactured on Monday? Exercise 5 Suppose there are n people in a room. (a) What is the probability p(n) that at least two persons have the same birthday. (b) What is the smallest n such that p(n) exceeds 0.5. Consider day and month only, and forget about leap years.
Probability Theory and Random Variables 7 A random variable X (Zufallsvariable, variable aléatoire) is a function from a sample space Ω of a given random experiment into the real numbers. I.e. each outcome of the experiment is related to a real number. Example 11 An urn contains 2 red balls and 3 white ones. We draw one ball after the other (without putting it back) until we get a red one. The number of moves is our random number X. The probability distribution function (PDF, Wahrscheinlichkeitsverteilung, distribution de probabilité) of a random variable is defined as F(x) = P[X x] In the case of a discrete random variable we have F(x) = In Example 11:
Probability Theory and Random Variables 8 The PDF has the following properties: (1) lim F(x) = 0 (2) lim F(x) = 1 (3) if x 1 < x 2 then F(x 1 ) F(x 2 ),.e. F(x) increases monotonically There are two important functions that are defined for random variables: mean or average or expectation value (Mittelwert oder Erwartungswert, moyenne ou espérance mathématique) and variance (Varianz, variance). E[X] = expectation value Var[X] = variance The square root of the variance is known as standard deviation (Streuung, écart type)
Probability Theory and Random Variables 9 Given a basic experiment with two possible outcomes (sucess and failure) with constant probabilities p (for success) and q = 1 - p (for failure). The experiment is repeated n times, and the number of successful outcomes is taken as the value of a random variable X. Let p(k) be the probability for the event that X takes the value of k: p(k) = P[X = k]. Then p Binomial (k,n,p) = with q = 1 - p This is the famous binomial distribution. Its expectation value and variance are: E[X] = Var[X] = For large values of n (and if p and q are not too close to zero) the evaluation of p(k,n,p) may be numerically difficult or even impossible.the binomial distribution may then be approximated by the normal distribution with the same mean and variance. p Normal (x,n,p) f(x) = with µ = and σ = Example 12 A telephone operator services 10 incoming telephone calls per hour in the average. Let us consider the minutes of an hour as "basic experiments" and declare an arriving call as a "successful outcome" of the basic experiment. What is the probability for exactly k = 12 "successful minutes" within a given hour? Binomial distribution: With the normal distribution we get: Evidently the number of "successful minutes" is not necessarily identical to the number of arriving calls, because it may happen that more than one call arrives within a single minute. We can improve the result of our probability calculation by considering "successful seconds".
Probability Theory and Random Variables 10 In order to do so, we replace: n = 60 n = p = p = Note that while applying this substitution, the product n p (equal to the mean value of the binomial distribution!) remains unchanged. We can even go further and take the limits for n and p 0 in a way that n p = µ remains constant. We then obtain the famous Poisson distribution: p Poisson (k,µ) = The mean value is E[X] = µ and the variance is also Var[X] = σ 2 = µ. In our Example 12 we obtain for k = 12 and µ = 10: p Poisson (12,10) = Example 13 In Example 12, what is the probability for the event that less than seven calls arrive in a given hour? The probability distribution function F(x) gives us the answer: F(6) = P[X 6] = Example 14 Consider again the situation of our telephone operator. We start our observation (and our clock) at an arbitrarily chosen moment in time and measure the time until the first telephone call arrives. We take this time as our random variable T. Note that the starting point on the time axis is not important. We can choose it wherever we like. In particular, we can choose it at the moment, when a telephone call arrives. If we do so, T is the time between two arriving calls: the interarrival time, or pause, for short. Clearly, T is a continuous random variable. However, we shall first work with a discrete time interval t (e.g a minute) and measure T in units of this interval. We may then calculate the probability of a pause T of a certain maximum length, say t = 12 min: F(t) = P[T t] =
Probability Theory and Random Variables 11 In our example λ = 10 h -1 = 1/6 min -1 represents the average arrival rate. We then have F(t) = In our numerical example: F(12 min) = However, the random number T should be measured in much shorter time units t in order to get a "more correct" result for F(t). The solution is to take the limit t 0 and n, such that t = n t remains constant. The result is the exponential distribution: F(t) = We have proved the following important fact: If the basic events of an experiment are independent from each other, then their number in a given interval of time is Poisson distributed, while the interarrivel time distribution is exponential. If X is a continuous random variable (like the interarrival time distribution in the preceeding example) we cannot use the definitions for the mean value and the average given earlier. In order to be able to define these functions for continuous random variables too, we define the probability density function (pdf) f(x) as the derivative of the probability distribution function F(x): F(x) = P[X x]. (probability distribution function); f(x) = f(x) dx = F(x + dx) - F(x) = P[x X x + dx]
Probability Theory and Random Variables 12 The mean value or average and the variance of a continuous variable is defined as: E[X] = Var[X] = Exercise 7 We throw a 5 cent coin and a fair die and take the absolute value of the difference of the numbers that appear. The result is our random number X. Draw the probability distribution function F(x) and calculate mean and variance. Exercise 8 Use a mathematical software package to draw a picture of the binomial distribution p Binomial (k,n,p) for n = 20, p = 0.3. Compare it with a normal distribution having the same mean and variance. Exercise 9 Use a mathematical software package to draw a picture of the Poisson distribution p Poisson (k,µ) for µ = 25, k = 0...100. Exercise 10 Calculate mean and variance of the exponential distribution F(t) = 1 - e -λ t Exercise 11 A particle detector counts 15 events per minute on the average. (a) What is probability of counting more than 20 events during a particular minute? (b) Find the probability of a pause shorter than 0.1 sec between two consecutive events.