INF FALL NATURAL LANGUAGE PROCESSING. Jan Tore Lønning

Transcription

1 1 INF FALL NATURAL LANGUAGE PROCESSING Jan Tore Lønning

2 2 Probability distributions Lecture 5, 5 September

3 Today 3 Recap: Bayes theorem Discrete random variable Probability distribution Discrete distributions Bernoulli trial Binomial distribution Continuous random variables/distributions Normal distribution

4 Last week 4 Probability space (covered) Random experiment (or trial) (no: forsøk) Outcomes (utfallene) Sample space (utfallsrommet) An event (begivenhet) Bayes theorem Discrete random variable (left overs) The probability mass function, pmf The cumulative distribution function, cdf The mean (or expectation) (forventningsverdi) The variance of a discrete random variable X The standard deviation of the random variable

5 Conditional probability 5 Conditional probability (betinget sannsynlighet) P( A B) P( A B) P( B) The probability of A happens if B happens A AB B

6 Bayes theorem Jargon: P(A) prior probability P(A B) posterior probability Extended form ) ( ) ( ) ( ) ( B P A P A B P B A P ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( A P A B P A P A B P A P A B P B P A P A B P B A P 6

7 Example 7 Disease: 1 out of 1000 are infected Test: Detects 99% of the infected 2% of the non-infected get a positive test Given a positive test: what is the chance you are ill?

8 8 Random variables

9 Random variable 9 A variable X in statistics is a property (feature) of an outcome of an experiment. Formally it is a function from a sample space (utfallsrom) to a value space X. When the value space X is numerical (roughly a subset of R n ), it is called a random variable There are two kinds: Discrete random variables Continuous random variables A third type of variable: categorical variable, when X is nonnumerical

10 Examples Throwing two dice, ={(1,1), (1,2), (1,6),(2,1), (6,6)} 1. The number of 6s is a random variable X, X ={0, 1, 2} 2. The number of 5 or 6s is a random variable Y, Y = X 3. The sum of the two dice, Z, Z ={2, 3,, 12} 2. A random person: 1. X, the height of the person X =[0, 3] (meters) 2. Y, the gender Y ={0, 1} (1 for female) Ex 2.1 is continuous, the other are discrete

11 11 Discrete random variables

12 Discrete random variable 12 The value space is a finite or a countable infinite set of numbers {x1, x2,, xn, } The probability mass function, pmf, p, which to each value yields p(x i ) = P(X=x i ) = P ({ X()=x i }) The cumulative distribution function, cdf, F(x i ) = P(X < x i ) = P ({ X() < x i }) Diagrams: Wikipedia

13 Examples 13 Throwing two dice, ={(1,1), (1,2), (1,6),(2,1), (6,6)} (1.3) The sum of the two dice, Z, Z ={2, 3,, 12} p Z (2) = P({(1,1)} = 1/36 p Z (7) = 6/36 F Z (7) = =21/36 (1.1) The number of 6s X, X ={0, 1, 2} p X (2) = P({(6,6)} = 1/36 p X (1) = P({(6,x) x6}+ P({(x,6) x6}=10/36 px(0) = 25/36

14 Mean example 14 Throwing two dice, what is the mean value of their sum? ( )/36= (2+2*3+3*4+4*5+5*6+6*7+5*8+ 2*11+12)/36= (1/36)2+(2/36)*3+(3/36)*4+ +(1/36)*12 = p(2)*2 + p(3)*3 + p(4)*4 + p(12)*12 = p(x)*x

15 Mean of a discrete random variable 15 The mean (or expectation) (forventningsverdi) of a discrete random variable X: X E ( X ) p( x) x Useful to remember ( X Y ) ( a bx) X x Y a b x Examples: One dice: 3.5 Two dice: 7 Ten dice: 35

16 More than mean 16 Mean doesn t say everything Examples (1.3) The sum of the two dice, Z, i.e. p Z (2) = 1/36,, p Z (7) = 6/36 etc (3.2) p 2 given by: p 2 (7)=1 p 2 (x)= 0 for x 7 (3.3) p 3 given by: p 3 (x)= 1/11 for x = 2,3,,12 Have the same mean but are very different

17 Variance 17 The variance of a discrete random variable X 2 Var ( X ) p( x)( x ) Observe that Var ( X ) E(( X It may be shown that this equals x E( X )) The standard deviation of the random variable 2 ) 2 E 2 ( X ) ( E( X )) 2 Var(X )

18 Examples 18 Throwing one dice = ( )/6=7/2 2 = ((1-7/2) 2 +(2-7/2) 2 + (6-7/2) 2 )/6 = (25+9+1)/4*3=35/12 (Ex 1.3) Throwing two dice: 35/6 (Ex 3.2) p 2, where p 2 (7)=1 has variance 0 (Ex 3.3) p 3, the uniform distribution, has variance: ((2-7) 2 + (12-7) 2 )/11 = ( )*2/11 = 10

19 From Lecture 1: Looking at data 19 Median, mean mode Median, quartiles Variance, standard deviation

20 20 Probability distributions Sannsynlighetsfordelinger

21 Examples of distributions 21 (1.3) The sum of the two dice, Z, i.e. p Z (2) = 1/36,, p Z (7) = 6/36 etc (3.2) p 2 given by: p 2 (7)=1 p 2 (x)= 0 for x 7 (3.3) p 3 given by: p 3 (x)= 1/11 for x = 2,3,,12

22 Examples of variance 22 Throwing one dice = ( )/6=7/2 2 = ((1-7/2) 2 +(2-7/2) 2 + (6-7/2) 2 )/6 = (25+9+1)/4*3=35/12 (Ex 1.3) Throwing two dice: 2 = 35/6 (Ex 3.2) p 2, where p 2 (7)=1 has variance 0 (Ex 3.3) p 3, the uniform distribution, has variance: ((2-7) 2 + (12-7) 2 )/11 = ( )*2/11 = 10

23 23 Discrete distributions Bernoulli trial Binomial distribution

24 Bernoulli trial 24 One experiment, two outcomes X ={0, 1} Write p for p(1) Then p(0) = 1-p Examples: Flipping a fair coin, p=1/2 Rolling a dice, getting a 6, p=1/6 The mean/expectation: 0*p(0)+1*p(1)=0+p=p Variance 2 Var ( X ) p( x)( x ) (1 p)(0 p) 2 x p(1 p) 2 2 p(1 p) Standard deviation σ = p(1 p)

25 Bernoulli trial and binomial distribution 25 The Bernoulli trial seems trivial, but can be used as a lego block for more interesting models Binomial: n trials let X be a random variabel counting the number of successes Possible values {0, 1, 2,, n} Consider the distribution p(k) Geometric: how many trials before the first success?

26 Binomial distribution 26 Binomial distribution (binomisk fordeling) Conducting n Bernoulli trials with the same probability and counting the number of successes Example flipping a fair coin n times, p(k): n=2: p(0)=1/4, p(1)=1/2, p(2) =1/4 n=3: p(0)=1/8, p(1)=3/8, p(2)=3/8, p(3)=1/8 n=4: (1,4,6,4,1)/16 n=5: (1,5,10,5,1)/32 p( k) n k 1 2 n where n k n! k!( n k)!

27 Binomial distribution 27 Binomial distribution (binomisk fordeling) General form: 0<p<1 n a natural number B(n,p) is given by b( k; n, p) n k p k (1 p) ( nk ) for k = 0, 1, n, where n k n! k!( n k)!

28 Binomial distribution 28 n = 20 p = 0.1 (blue), p = 0.5 (green) and p = 0.8 (red)

29 Binomial distribution 29 Mean/expectation, μ, of B(n,p) is np n Bernoulli trials Each Bernoulli trial has mean p The variance is np(1-p) Because the Bernoulli trials are independent Each Bernoulli trial has variance p(1-p) The variance of the sum of two independent random variables is the sum of their variances

30 Binomial distributions p= The relative variation gets smaller with growing N The pmf graph approaches a bell shape N σ σ N=4: N=16: N=64:

31 SciPy 31 import scipy from scipy import stats bin10 = stats.binom(10, 0.5) # N=10, p=0.5 bin10.pmf(3) # probability mass of 3, b(3;10,0.5) bin10.cdf(3) # cumulative distribution function at 3 bin10.var() # variance bin10.std() # standard deviation x = np.arange(11); plt.bar(x, bin_10.pmf(x))

32 32 Continuous random variables The normal distribution

33 Continuous random variables 33 P(X=a) = 0 for nearly all values a The probability mass function does not make sense The cumulative distribution function, cdf, given by F(a) = P(X<a) makes sense P(a<x<b) = F(b) - F(a) To calculate expectation and variance we must use integration instead of (infinite) sums. We skip the details!

34 Probability density function 34 The derivative of the cdf, F, is called the probability density function, pdf (sannsynlighetstetthet) We draw curves for pdf-s The pdf has a similar relationship to the cdf in the continuous case as the pmf has to the cdf in the discrete case pdf cdf

35 The normal distribution 35 Standard norm.dist. (red curve) Formulas (don't have to remember) f ( x) 1 e 2 x 2 2 Remember these: 0 1 z x z-score relates the general case to the standard case Tells how far x is from in terms of standard deviations General norm.dist N(,) f ( x) ( x) e 2 2 2

36 36 68% - 95% %

37 37 Example x Tallness of Norwegian young men (rough numbers): = 180 cm = 6cm z = ( )/6=1 (standard deviation) (100-68)/2%= 16% are taller than 186cm z How many are taller than 190cm? z = ( )/6 = 1.67 Prob. = (from table or software)