02 Background Minimum background on probability. Random process

Transcription

1 0 Background 0.03 Minimum background on probability Random processes Probability Conditional probability Bayes theorem Random variables Sampling and estimation Variance, covariance and correlation Probability and information: Entropy alessandro bogliolo isti information science and technology institute /9 A random process: Random process Can be repeated infinite times May provide mutually-exclusive results Provide an unpredictable result at each trial Elementary event (e: each possible result of a single trial Event space (X: the set of all elementary events Event (E: any set of elementary events (any subset of the event space X e E alessandro bogliolo isti information science and technology institute /9

2 Probability Relative frequency of E over trials: ratio of the number of occurrence of E (n E over the trials: f (E = n E / Probability of E (empirical definition: E lim f ( E Probability of E (assiomatic definition: E=0 if E is empty E= if E=X p E E E if E E ( E alessandro bogliolo isti information science and technology institute 3/9 Probability (properties. If all elementary events have the same probability, the probability of an event is given by its relative size : X E = card(e/card(x. 3. If E is a subset of E E E E E E E E E E E E E E E E E E alessandro bogliolo isti information science and technology institute 4/9

3 Conditional probability Joint probability of two events, outcomes of two random experiments nee EE lim Marginal probability E EE E E Conditional probability E E Decomposition EE E E E E E E E E alessandro bogliolo isti information science and technology institute 5/9 Independent events E and E are independent events if and only if E E E The oint probability of two independent events is equal to the product of their marginal probabilities ( E E E E p alessandro bogliolo isti information science and technology institute 6/9

4 Bayes Theorem Bayes theorem: given two events E and E, the conditional probability of E given E can be expressed as: E E E E E E The theorem provides the statistical support for statistical diagnosis based on the evaluation of the probability of a possible cause (E of an observed effect (E alessandro bogliolo isti information science and technology institute 7/9 Random variable Random variable x: variable representing the outcome of a random experiment Probability distribution function of x: ( F x x x x Probability density function of x: df x( x f x( a dx alessandro bogliolo isti information science and technology institute 8/9

5 Sampling and estimation Parent population of a random variable x: ideal set of the outcomes of all possible trials of a random process Sample of x: set of the outcomes of trials of the random process Sample parameters can be used as estimators to infer the values of the corresponding parameters of the parent population Example: E x ei ei x x e X i Expected value of x Sample average alessandro bogliolo isti information science and technology institute 9/9 Confidence of an estimator The quality of the estimator P of a given parameter P can be expressed in terms of: Confidence interval: limiting distance d between the estimator and the actual parameter Confidence level: probability c of finding the actual parameter within the confidence interval c Prob P P' The smaller the confidence interval d and the higher the confidence level c, the better The quality of an estimator grows with the number of samples For a fixed confidence level c, the size of confidence interval d decreases with the inverse of the square root of d alessandro bogliolo isti information science and technology institute 0/9

6 Variance, covariance, correlation Variance: ( x x Standard deviation: Covariance: Correlation: Cov( x, y Corr( x, y ( x x ( y y Cov( x, y x y The confidence interval of an estimator is proportional to alessandro bogliolo isti information science and technology institute /9 Entropy The Entropy of a random variable x provides a measure of the average uncertainty of the value of x H (x x i log x i i If the base- logarithm is used in the above equation, entropy is expressed in bits. E.g.: Coin toss experiment x 0 =head, x =tail H(x 0.5log log 0.5 log bit alessandro bogliolo isti information science and technology institute /9

7 Entropy and information Given a random variable defined on a given event space, its entropy is maximum when it is uniformly distributed on the event space (i.e., when all possible outcomes have the same probability H(x xi log xi log log i The maximum entropy of a random variable defined over a finite event space corresponds to the number of bits required to encode the outcome of the experiment! Entropy may take non-integer values E.g.: Dice roll experiment (6 possible outcomes H(x 6* log log alessandro bogliolo isti information science and technology institute 3/9 Entropy and information If the outcomes of a random process are not equally distributed, the uncertainty of the outcome of each trial is reduced and the entropy is lower than log Entropy (Coin toss 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0, 0, 0 0 0, 0, 0,3 0,4 0,5 0,6 0,7 0,8 0,9 P(head alessandro bogliolo isti information science and technology institute 4/9

8 Entropy of DA DA sequences are defined over an alphabet of symbols that occur equiprobably. The entropy of a random variable representing the DA base at a given position is H(x 4* log log 4 bits 4 4 In fact, bits are needed to encode each base alessandro bogliolo isti information science and technology institute 5/9 Information content of DA ( Each character of a given DA strand contains all the information required to reduce the uncertainty on the symbol inserted at that position Assume that the n-th character of the strand is A The probability distribution evaluated a posteriori is: P(A =, C=G=T=0 and the corresponding entropy is H(x log 3*0 log 0 0 The information content of a character is the difference between it s a-priori and a-posteriori entropy H H 0 bits before after alessandro bogliolo isti information science and technology institute 6/9

9 Information content of DA ( ow assume that, after some experiments, we know that the n-th position of a given DA strand may take only values A or C, so that the a-posteriori probability distribution is A=C=0.5, G=T=0 The amount of information provided by the experiments is H H bits before after otice that the information content of an experiment can be negative if the a-posteriori distribution is more uniform than the a-priori distribution alessandro bogliolo isti information science and technology institute 7/9 Relative entropy Given two probability functions p and q, their relative entropy is defined as: xi H ( p q xi log q( x i In general, H ( p q H ( q p If q is a uniform distribution, the relative entropy of p and q is equal to the difference between the entropy of q and the entropy of p H ( p q H ( q H ( p that is the information content of the experiment leading from q to p i alessandro bogliolo isti information science and technology institute 8/9

10 Mutual information The mutual information of two random variables x and y is defined as the relative entropy of x,y and xy: xi, y M (x; y xi, y log i, xi y The mutual information is the amount of information we acquire about the outcome of x when we know the outcome of y Is x and y are independent variables, M(x;y=0 M(x;y is maximum when x and y always covariate M(x;y is always non negative (that s why it is often used as a distance metric to measure the difference between two distributions alessandro bogliolo isti information science and technology institute 9/9