Chapter 1. Sets and probability. 1.3 Probability space

Random processes - Chapter 1. Sets and probability 1 Random processes Chapter 1. Sets and probability 1.3 Probability space 1.3 Probability space

Random processes - Chapter 1. Sets and probability 2 Probability space Random experiment A random experiment is an experiment which can be repeated under perfect control, yet the result of which is not known in advance. 1.3 Probability space/ 1.3.1 Probability space

Random processes - Chapter 1. Sets and probability 3 Sample space A sample space is the collection of all possible outcomes of an experiment. Tossing a coin is a random experiment since the result is not determined in advance. The sample space in this case is S = {head, tail}. Probability space A probability space is the triplet of a sample space, a sigma algebra, and a probability measure (to be defined shortly). For example, when the sample space is S, asigma algebra of S is F, and the probability measure is P, the probability space can be denoted as (S, F,P). 1.3 Probability space/ 1.3.1 Probability space

Random processes - Chapter 1. Sets and probability 4 Sample space A sample space is often denoted by S or Ω. An element of a sample space is called a sample point, an elementary event, or an elementary outcome. In the tossing of a coin, the sample space is S = {head, tail} and head and tail are sample points. In the rolling of a die, the sample space is S = {1, 2, 3, 4, 5, 6} and 1, 2, 3, 4, 5, and 6 are sample points. 1.3 Probability space / 1.3.2 Sample space

Random processes - Chapter 1. Sets and probability 5 Event space An event space is a sigma algebra obtained from a sample space. An element of the event space (that is, a subset of the sample space) is called an event. An event in probability theory corresponds to a set in set theory. Consider the tossing of a coin. Then, S = {head, tail}. When the event space is F = {S,, {head}, {tail}}, the events are S, {head}, {tail}, and{}. 1.3 Probability space / 1.3.3 Event space

Random processes - Chapter 1. Sets and probability 6 Smallest sigma algebra, power set, biggest sigma algebra When the sample space is Ω, {Ω,φ} is the smallest sigma algebra, i.e., the smallest event space. The collection of all the subsets of the sample space Ω is called the power set, the biggest sigma algebra of Ω. When two sets M and N are equivalent - that is, if M N - the power sets of M and N are also equivalent. Sigma field generated by G Given a sample space Ω, consider an arbitrary class G of subsets of Ω. The smallest sigma field containing all of the sets in G is called the sigma field generated by G and is denoted by σ(g). Here, by smallest we mean that if a sigma field contains G, then it also contains σ(g). 1.3 Probability space / 1.3.3 Event space

Random processes - Chapter 1. Sets and probability 7 Borel field, Borel sigma field When the sample space is the real line R, the Borel field (or, the Borel sigma field) is defined as the sigma field generated by all the open intervals of the form (a, b). The members of the Borel field are called Borel sets. We shall denote the Borel field by B(R), and hence B(R) =σ (all open intervals). 1.3 Probability space / 1.3.3 Event space

Random processes - Chapter 1. Sets and probability 8 Probability measure Measurable space The pair (Ω, F) of a sample space Ω and an event space F is called a measurable space. 1.3 Probability space / 1.3.4 Probability measure

Random processes - Chapter 1. Sets and probability 9 Probability measure For the measurable space (Ω, F), a set function is called a probability measure if it assigns a real number P (F ) toasetf Funder the constraint of the following four axioms: If F F, P(F ) 0. P (Ω) = 1. If event sequences E i,i=1, 2,,n are mutually exclusive, ( n ) n P E i = P (E i ). i=1 If event sequences E i,i=1, 2, are mutually exclusive, ( ) P E i = P (E i ). i=1 i=1 i=1 1.3 Probability space / 1.3.4 Probability measure

Random processes - Chapter 1. Sets and probability 10 Consider the sample space Ω = {0, 1} and event space F = {{0}, {1}, Ω, }. Then, a probability measure P in this measurable space (Ω, F) is as follows: P (F )= 1 4 3 4, when F = {0},, when F = {1}, 0, when F =, 1, when F =Ω. Properties of probability measure Property 1 P (F c )=1 P(F). P (F ) 1. P ( ) =0. If E F, P(E) P (F ). P ( F i ) P (F i ) (Boole s inequality). i=1 i=1 1.3 Probability space / 1.3.4 Probability measure

Random processes - Chapter 1. Sets and probability 11 Property 2 Let {F i } be a countable partition of the sample space Ω. Then, P (G) = i P (G F i ). Property 3 Denoting by P (F i1 F i2 F ir ) the sum over ( ) n r = n! i 1 <i 2 < <i r choosing r numbers from {1, 2,,n}, wehave ( n ) n P F i = P (F i ) P (F i1 F i2 )+ i=1 i=1 i 1 <i 2 +( 1) r+1 P (F i1 F i2 F ir ) i 1 <i 2 < <i r +( 1) n+1 P (F 1 F 2 F n ). r!(n r)! ways of When n =2,wehaveP (E F )=P (E)+P (F ) P (E F ). 1.3 Probability space / 1.3.4 Probability measure

Random processes - Chapter 1. Sets and probability 12 Discrete space and probability mass function Probability mass function (pmf) A real function p(ω) assigning a real number to each sample point ω inasample space is called a probability mass function (pmf) or a mass function if it satisfies the following two conditions. Two-point pmf p(ω) 0, ω Ω, p(ω) = 1. ω Ω Consider the sample space Ω = {x 1,x 2 } and a number p 1 (0, 1). Then the following function p is called the two-point pmf. p(x 1 )=1 p 1, p(x 2 )=p 1. 1.3 Probability Space / 1.3.5 Discrete space and probability mass function

Random processes - Chapter 1. Sets and probability 13 Binary pmf, Bernoulli pmf When x 1 =0and x 2 =1in the two-point pmf, we have p(0) = 1 p 1, p(1) = p 1, which is called the binary or Bernoulli pmf: The binary distribution is usually denoted by b(1,p 1 ). Uniform pmf When the sample space is Ω=Z n = {0, 1,,n 1}, the following function p is called the uniform pmf p(k) = 1 n, k Z n. 1.3 Probability Space / 1.3.5 Discrete space and probability mass function

Random processes - Chapter 1. Sets and probability 14 Binomial pmf Consider the sample space Ω=Z n+1 = {0, 1,,n} and a number p 1 (0, 1). Then the following function p is called the binomial pmf and the distribution is denoted by b(n, p 1 ). ( ) n p(k) = p k k 1(1 p 1 ) n k, k Z n+1. 1.3 Probability Space / 1.3.5 Discrete space and probability mass function

Random processes - Chapter 1. Sets and probability 15 Geometric pmf Consider the sample space Ω={1, 2, 3, } and a number p 1 (0, 1). Then the following function p is called the geometric pmf. p(k) =(1 p 1 ) k 1 p 1, k Ω. 1.3 Probability Space / 1.3.5 Discrete space and probability mass function

Random processes - Chapter 1. Sets and probability 16 Poisson pmf Consider the sample space Ω=Z + = {0, 1, 2, }and a number λ (0, ). Then the following function p is called the Poisson pmf and the distribution is denoted by P (λ). p(k) = λk e λ, k Z +. k! 1.3 Probability Space / 1.3.5 Discrete space and probability mass function

Random processes - Chapter 1. Sets and probability 17 Negative binomial pmf, Pascal pmf Consider the sample space Ω={0, 1, } and two numbers p 1 (0, 1) and r {1, 2, }. Then the following function p is called the negative binomial pmf. p(x) = ( ) x + r 1 p r x 1(1 p 1 ) x, x Ω. 1.3 Probability Space / 1.3.5 Discrete space and probability mass function

Random processes - Chapter 1. Sets and probability 18 Continuous space and probability density function Probability density function (pdf) Given a measurable space (Ω, F) =(R, B(R)), a real-valued function f with the properties Ω f(r) 0, r Ω, f(r)dr = 1 is called a probability density function (pdf) or a density function. The set function P defined by f as P (F )= is a probability measure. F f(r)dr, F B(R), 1.3 Probability Space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 19 Uniform pdf, rectangular pdf The following pdf is called uniform pdf, and the distribution is denoted by U(a, b), b>a. f(r) = 1 b a, r (a, b). 1.3 Probability Space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 20 Exponential pdf The following pdf with λ>0 is called the exponential pdf. f(r) =λe λr, r 0. 1.3 Probability Space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 21 Double exponential pdf, Laplace pdf The following pdf with λ>0 is called the double exponential pdf. f(r) = λ 2 e λ r, r R. 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 22 Gaussian pdf, normal pdf The following pdf is called the Gaussian pdf and the distribution is denoted by N(m, σ 2 ). { } 1 f(r) = exp (r m)2, r R. 2πσ 2 2σ 2 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 23 Cauchy pdf The following pdf with α>0 is called a Cauchy pdf. f(r) = α π 1 r 2 + α2, r R. 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 24 Rayleigh pdf The following pdf is called a Rayleigh pdf. f(r) = r { } α exp r2, r 0. 2 2α 2 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 25 Logistic pdf The following pdf with k>0 is called a logistic pdf. f(r) = ke kr (1 + e kr ) 2, r R. 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 26 Gamma pdf The following pdf with α>0 and β>0is called a Gamma pdf and the distribution is denoted by G(α, β). { 1 f(r) = exp r }, r 0. Γ(α)β αrα 1 β Here, is the gamma function. Γ(α) = 0 x α 1 exp{ x}dx 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 27 Central chi-square pdf When α = n/2 and β =2for a natural number n, the gamma distribution is called the central chi-square distribution with n degree of freedom and is denoted by χ(n). 1 { f(r) = exp r }, r 0. Γ(n/2)2 n/2rn/2 1 2 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 28 Beta pdf The following pdf with α>0 and β>0 is called a beta pdf and the distribution is denoted by B(α, β). Here, f(r) = rα 1 (1 r) β 1, 0 r 1. B(α, β) B(α, β) = is called the beta function, for which 1 0 + x α 1 (1 x) β 1 dx B(α, β) = Γ(α)Γ(β) Γ(α + β). 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 29 Beta pdf 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 30 Central t-pdf The following pdf is called a central t pdf with n degrees of freedom. Γ((n +1)/2) f(r) = Γ(n/2) nπ Here, n is a positive integer. (1+ r2 n ) (n+1)/2, r R. 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 31 Central F -pdf The following pdf is called the central F -pdf with (m, n) degrees of freedom and the distribution is denoted as F (m, n). Γ((m + n)/2) m ( m ) m/2 1 ( f(r) = Γ(m/2)Γ(n/2) n n r 1+ m ) (m+n)/2 n r, r 0. Here, m and n are natural numbers. 1.3 Random space / 1.3.6 Continuous space and probability density function

Random processes - Chapter 1. Sets and probability 32 Mixed probability measure Mixed probability measure Let P i,i=1, 2, be probability measures on a common measurable space (Ω, F), and let a i, i =1, 2, be nonnegative numbers with a i =1. Then the set i function P defined by P (F )= i a i P i (F ) is also a probability measure on (Ω, F). Such a probability measure is called a mixed probability measure. 1.3 probability space / 1.3.7 Mixed probability measure

Random processes - Chapter 1. Sets and probability 33 Mixed probability measure impulse functions, pdf, and pmf Consider a distribution obtained by combining a point mass of 1/4 at 0, apoint mass of 1/2 at 1, and a unform pdf between 0 and 1. The distribution can then be described by the pdf f(r) = 1 4 δ(r)+1 2 δ(r 1) + 1, r [0, 1]. 4 This example says implicitly that a pdf can be defined for discrete and mixed spaces by using impulse functions. 1.3 probability space / 1.3.7 Mixed probability measure

Random processes - Chapter 1. Sets and probability 34 Product space and multi-dimensional probability functions Joint pmf A real function p is called a k-dimensional pmf or joint pmf on a measurable discrete space (Ω k, F k ) if it satisfies p(x) = 1 and p(x) 0 for all x = x Ω k (x 0,x 1,,x k 1 ) Ω k. Multinomial pmf: Consider the sample space Ω k = Z k n+1 = {0, 1,,n} k and k numbers p j (0, 1),j =1, 2,,k.Then is called the multinomial pmf p(x) = Pr{X 1 = x 1,X 2 = x 2,,X k = x k } n!p x 1 1 px 2 2 px k k k = x 1!x 2! x k!, x i = n, x i 0, i=1 0, otherwise 1.3 Probability space / 1.3.8 Product space and multi-dimensional probability functions

Random processes - Chapter 1. Sets and probability 35 Joint pdf A real function f satisfying the two conditions below on a measurable space (R k, B(R) k ) is called a k-dimensional pdf or joint pdf. Here, dx = dx 0 dx 1 dx k 1. f(x) 0, x R k, R k f(x)dx = 1. Product pdf If f i,i=0, 1,,k 1 are one dimensional pdfs, the following function f(x) is called a product pdf f(x) = k 1 i=0 f i (x i ). 1.3 Probability space / 1.3.8 Product space and multi-dimensional probability functions

Random processes - Chapter 1. Sets and probability 36 Other definitions of probability Classical definition When all the outcomes from a random experiment are equally likely, the probability of an event can be defined by the ratio of the desired outcomes to the total number of outcomes. The probability of A is thus given by the ratio P (A) = N A N, where N is the number of all possible outcomes and N A is the number of desired outcomes for A. It should be noted that all outcomes are assumed to be equally likely in the classical definition. 1.3 Probability space / 1.3.9 Other definitions of probability / 1.3.9.1 Classical definition

Random processes - Chapter 1. Sets and probability 37 Relative frequency definition The probability is determined by the relative frequency of favorable outcomes in a number of repetitions of a random experiment. The probability p(a) of an event A is the limit n A P (A) = lim n n, where n A is the number of occurrences of A and n is the number of trials. 1.3 Probability space / 1.3.9 Other definitions of probability / 1.3.9.2 Relative frequency definition

Random processes - Chapter 1. Sets and probability 38 Limit event and continuity of probability* Increasing sequence, decreasing sequence When E n E n+1 for all integer n, the event sequence {E n,n 1} is called an increasing sequence. When E n E n+1 for all integer n, the event sequence {E n,n 1} is called a decreasing sequence. Monotone sequence and limit event, limit An increasing or decreasing sequence is called a monotone sequence. When an event sequence {E n,n 1} is a monotone sequence, the limit event or limit of {E n,n 1} is defined by lim n E n = i=1 E i, ({E n,n 1} is an increasing sequence), lim E n = E i, ({E n,n 1} is a decreasing sequence). n i=1 1.3 Probability space / 1.3.10 Limit event and continuity of probability*

Random processes - Chapter 1. Sets and probability 39 Lower limit For an event sequence, the set of elements in almost all events is called the lower limit lim inf of the event sequence. n Upper limit For an event sequence, the set of elements in infinitely many events is called the upper limit lim sup of the event sequence. n 1.3 Probability space / 1.3.10 Limit event and continuity of probability*

Random processes - Chapter 1. Sets and probability 40 Convergence and limit for a general sequence Since lim inf n lim inf n lim inf n E n lim sup n E n when lim sup E n,wehave n E n for any sequence {E n,n 1}, wehavelim sup E n n lim inf n E n. In other words, when lim sup n lim sup n E n = liminf n = E. E n E n = In such a case, we say {E n,n 1} converges to E, and write E n E or lim E n = E. The event E is called the limit event or limit of {E n,n 1}. n E n 1.3 Probability space / 1.3.10 Limit event and continuity of probability*

Random processes - Chapter 1. Sets and probability 41 Probability of limit event 1 When an event sequence {E n,n 1} is a monotone sequence, the probability of the limit event is equal to the limit of the probability of the events in the sequence. In other words, lim P (E n)=p ( lim E n ). n n Probability of limit event 2 When the limit event lim n E n of an event sequence {E n,n 1} exists, the probability of the limit event is equal to the limit of the probability of the events in the sequence. In other words, lim P (E n)=p ( lim E n ). n n 1.3 Probability space / 1.3.10 Limit event and continuity of probability*

Random processes - Chapter 1. Sets and probability 42 Continuity of probability Let us consider an event sequence {E n,n = 1, 2, }. When the limit of this sequence is E, E n converges to E and P (E n ) converges to P (E). Here, the relation lim P (E n)=p (lim E n ) n n is called the continuity of probability. 1.3 Probability space / 1.3.10 Limit event and continuity of probability*