A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2011

Transcription

1 A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2011 Reading Chapter 5 (continued) Lecture 8 Key points in probability CLT CLT examples

2 Prior vs Likelihood Box & Tiao

3 Learning in Bayesian Estimation Box & Tiao

4

5

6

7

8

9 3 I. Mutually exclusive events: If a occurs then b cannot have occurred. Let c = a + b + or (same as a b) P (c) =P {a or b occurred} = P (a)+p(b) Let d = a b and (same as a b) P (d) =P {a and b occurred} =0 ifmutuallyexclusive II. Non-mutually exclusive events: P (c) =P {a or b} = P (a)+p (b) P (ab) }{{} III. Independent events: P (ab) P (a)p (b) Examples I. Mutually exclusive events toss a coin once: 2possibleoutcomesH&T H&Taremutuallyexclusive H&Tarenotindependent because P (HT)=P{heads & tails} =0soP (HT) P (H)P (T ).

10 4 II. Independent events toss a coin twice = experiment The outcomes of the experiment are events might be defined as: H 1 H 2 =eventthathon1sttoss,hon2nd H 1 T 2 =eventthathon1sttoss,ton2nd T 1 H 2 =eventthatton1sttoss,hon2nd T 1 T 2 =eventthatton1sttoss,ton2nd 1st toss 2nd toss H 1 H 2 H 1 T 2 T 1 H 2 T 1 T 2 note P (H 1 H 2 )=P (H 1 )P (H 2 )[aslongascoinnotalteredbetweentosses]

11 5 Random Variables Of interest to us is the distribution of probability along the realnumberaxis: Random variables assign numbers to events or, more precisely, map the event space into a set of numbers: a X(a) event number The definition of probability translates directly over to thenumbersthatareassignedbyrandomvariables. The following properties are true for a real random variable. 1. Let {X x} =eventthatther.v.x is less than the number x; definedforallx [this defines all intervals on the real number line to be events] 2. the events {X =+ } and {X = } have zero probability. (Otherwise, moments would not be finite, generally.) Distribution function: (CDF = Cumulative Distribution Function) properties: F X (x) =P {X x} P {all events A : X(A) x} 1. F X (x) isamonotonicallyincreasingfunctionofx. 2. F ( ) =0,F (+ ) =1 3. P {x 1 X x 2 } = F (x 2 ) F (x 1 ) Probability Density Function (pdf) properties: f X (x) = df X(x) dx 1. f X (x) dx = P {x X x + dx} 2. dx f X(x) =F X ( ) F X ( ) =1 0=1

12 All three measures are localization measures Other quantities are needed to measure the width and asymmetry of the PDF, etc.

13 6 Continuous r.v. s: derivativeoff X (x) exists x Discrete random variables: use delta functions to write the pdf in pseudo continuous form. e.g. coin flipping 1 heads Let X = 1 tails then f X (x) = 1 [δ(x +1)+δ(x 1)] 2 F X (x) = 1 [U(x +1)+U(x 1)] 2 Functions of a random variable: The function Y = g(x) isarandomvariablethatisamappingfrom some event A to a number Y according to: Y (A) =g[x(a)] Theorem, ify = g(x), then the pdf of Y is f Y (y) = n j=1 f X (x j ) dg(x)/dx x=xj, where x j,j =1,n are the solutions of x = g 1 (y). Note the normalization property is conserved (unit area). This is one of the most important equations! Example * Y = g(x) =ax + b dg dx = a g 1 (y) = x 1 = y b a f X (x 1 ) f Y (y) = dg(x 1 )/dx = a 1 f X ( y b a ).

14 Comment about natural random number generators

15 7 To check: show that dy f Y (y) =1 Example Suppose we want to transform from a uniform distribution to an exponential distribution: We want ant f Y (y) =exp( y). A typical random number generator gives f X (x) with { 1, 0 x<1; f X (x) = 0, otherwise. Choose y = g(x) = ln(x). Then: Moments dg dx = 1 x x 1 = g 1 (y) =e y f Y (y) = f X[exp( y)] 1/x 1 = x 1 =e y. Factoid: Poission events in time have spacings that are exponentially distributed We will always use angular brackets < > to denote average over an ensemble (integrating over an ensemble); time averages and other sample averages will be denoted differently. Expected value of a random variable: E(X) X = dx xf X (x) Arbitrary power: denotes expectation w.r.t. the PDF of x X n = dx x n f X (x) Variance: σ 2 x = X2 X 2 Function of a random variable: If y = g(x) and Y dy y f Y (y) then it is easy to show that Y = dx g(x)f X (x). Proof: y dy f Y (y) = dy n j=1 f X [x j (y)] dg[x j (y)]/dx

16 8 A change of variable: dy = dg dx yields the result. dx Central Moments: µ n = (X X ) n Moment Tests: Moments are useful for testing hypotheses such as whether a given PDF is consistent with data: E.g. Consistency with Gaussian PDF: kurtosis k = µ 4 3=0 µ 3/2 2 skewness parameter γ = µ 3 =0 µ 3/2 2 k > 0 4th moment proportionately larger larger amplitude tail than Gaussian and less probable values near the mean.

17 9 Uses of Moments: Often one wants to infer the underlying PDF of an observable, e.g. perhaps because determination of the PDF is tantamount to understanding the underlying physics of some process. Two approaches are: 1. construct a histogram and compare the shape with a theoretical shape. 2. determine some of the moments (usually low-order) and compare. Suppose the data are {x j,j =1,N} 1. One could form bins of size x and count how many x j fall into each bin. If N is large enough so that n k = # points in the k-th bin is also large, then a reasonably good estimate of the PDF can be made. (But beware of dependence of results on choice of binning.) 2. However, often times N is too small or one would like to determine only basic information about the shape of the distribution (is it symmetric?), or determine the mean and variance of the PDF or test whether the data are consistent with a given PDF (hypothesis testing). Some of the typical situations are: i) assume the data were drawn from a Gaussian parent PDF; estimate themeanandσ of the Gaussian [parameter estimation] ii) test whether the data are consistent with a Gaussian PDF [moment test] note that if the r.v. is zero mean then the PDF is determined solely by one parameter: σ 1 /2σ f X (x) = 2 2πσ 2 e x2 The moments are (n 1)σ n (n 1)!! σ n n even x n = 0 n odd Therefore, the n = 2 moment = 1st non-zero moment all other moments. This statement remains for more multi-dimensional Gaussian processes: Any moment of order higher than 3 is redundant... gaussianity. or can be used as a test for

18 10 Characteristic Function: Of considerable use is the characteristic function Φ X (ω) e iωx dx f X (x) e iωx. If we know Φ X (ω) then we know all there is to know about the PDF because f X (x) = 1 dω Φ X (ω) e iωx 2π is the inversion formula. If we know all the moments of f X (x), then we also can completely characterize f X (x). Similarly, the characteristic function is a moment-generating function: Φ X (ω) = e iωx n=0 (iωx) n = n! because the expectation of the sum = sum of the expectations. By taking derivatives we can show that or Φ ω ω=0 = i X 2 Φ ω 2 ω=0 = i2 X 2 k Φ ω k ω=0 = in X n n=0 (iω) n n! X n X n = i n n Φ ω n ω=0 =( i) n n Φ ω n ω=0 Price stheorem Characteristic functions are useful for deriving PDFs of combinations of r.v. s as well as for deriving particular moments.

19 11 Joint Random Variables Let X and Y be two random variables with their associated sample spaces. The actual events associated with X and Y may or may not be independent (e.g. throwing a die may map into X; choosing colored marbles from a hat may map into Y ). The relationship of the events will be described by the joint distribution function of X and Y : and the joint probability density function is F XY (x, y) P {X x, Y y} f XY (x, y) 2 F xy (x, y) x y (a two dimensional PDF) Note that the one dimensional PDF of X, for example, is obtained by integrating the joint PDF over all y: f X (x) = dy f XY (x, y) which corresponds to asking what the PFf of X is given that the certain event for Y occurs. Example: flip two coins a and b. Let heads =1; tails =0. Define 2 r.v. s: X = a + b; Y = a. With these definitions X + Y are statistically dependent. Characteristic function of joint r.v. s: Φ XY (ω 1, ω 2 ) = e i(ω1x+ω2y ) = dx dy e i(ω1x+ω2y) f XY (x, y). For x, y independent [ ][ ] Φ XY (ω 1, ω 2 )= dx f X (x) e iω1x dy f Y (y) e iω2y Φ X (ω 1 ) Φ Y (ω 2 ). Example for independent r.v. s: flip two coins a and b. As before, heads = 1 and tails = 0, let x = a, y = b (x and y are independent). Independent random variables Two random variables are said to be independent if the events mapping into one r.v. are independent of those mapping into the other.

20 12 In this case, joint probabilities are factorable so that F XY (x, y) = F X (x) F Y (y) f XY (x, y) = f X (x) f Y (y). Such factorization is plausible if one considers moments of independent r.v. s: X n Y m = X n Y m which follows from X n Y m dx dy x n y m f XY (x, y) =[ [ dx x n f X (x)] ] dy y m f Y (y).

21 13 Convolution theorem for sums of independent RVs If Z = X +Y where X, Y are independent random variables, then the PDF of Z is the convolution of the PDFs of X and Y : f Z (z) =f X (x) f Y (y) = dx f X (x) f Y (z x) = dx f X (z x) f Y (x). proof: By definition, Consider Now, as before, this is f Z (z) = d dz F Z(z) F z (z) =P {Z z} F Z (z) = P {X + Y z} = P {Y z X}. To evaluate this, first evaluate the probability P {Y z x} where x is just a number. Now P {Y z x} F Y (z x) z x dy f Y (y) but P {Y z X} is the probability that Y z x for all values of x so we need to integrate over x and weight by the probability of x: P {Y z X} = dx f X (x) z x dy F Y (y) that is, P {Y z X} is the expected value of F Y (z x). By the Leibniz integration formula d db we obtain the convolution results. g(b) a dω h(ω) h(g(b)) dg(b) db

22 14 Characteristic function of Z = X + Y For X, Y independent we have f Z = f X f Y Φ Z (ω) = e iωz = Φ X (ω) Φ Y (ω) Variance of Z: if variance of X and Y are σ 2 X, σ2 Y, then variance of Z is σ2 Z = σ2 X + σ2 Y. Assume X and Y and hence Z are zero mean r.v. s, thenwehave σ 2 X = x2 = i 2 2 φ x ω 2 (ω =0) = 2 φ x ω 2 (ω =0) σ 2 Y = y2 = 2 φ y ω 2 (ω =0) Using Price s theorem: σz 2 = Z2 = 2 φ Z (ω =0) ω2 = 2 ω 2 [φ X(ω) φ Y (ω)] ω=0 = ω [ = φ X [ φ X 2 φ Y ω 2 φ Y ω + φ Y + φ Y 2 φ x ω 2 φ ] X ω ω=0 +2 φ X ω φ ] Y. ω ω=0 We have discovered that variances add (independent variables only): σ 2 Z = σ 2 X + σ 2 Y.

23 Multivariate random variables: N dimensional The results for the bivariate case are easily extrapolated. If where the X j are all independent r.v. s, then Z = X 1 + X X N = N j=1 X j f Z (z) =f X1 f X2... f XN and and N Φ Z = Φ Xj (ω) j=1 N σz 2 σx 2 j. j=1

24 16 Central Limit Theorem: Let Z N = 1 N N X j j=1 where the X j are independent r.v. s with means and variances µ j X j σ 2 j = X 2 j X j 2. and the PDFs of the X j s are almost arbitrary. Restrictions on the distributions of each X j are that i) σ 2 j >m>0 m =constant ii) X n <M=constantforn>2 In the limit N,Z N becomes a Gaussian random variable with mean Z N = 1 N N µ j j=1 and variance σ 2 Z = 1 N N σj 2. j=1 Example: supposethex j are all uniformly distributed between ± 1 2,so f X (x) =Π(x) sin πf πf = sin ω 2 ω/2

25 17 Thus the characteristic function is Φ j (ω) = e iωx j = sin ω/2 ω/2 Graphically: Gaussian N =2 N =3 N = e x2 ( sin ω 2 ω/2 )2 sin ω/2 ( ω/2 )3 e ω2 From the convolution results we have ( sin ω/2 φ NZ N (ω) = ω/2 From the transformation of random variables we have that ) N f ZN (x) = Nf NZ N ( Nx) and by the scaling theorem for Fourier transforms φ ZN (ω) =φ NZ N ( ω ) ( sin ω/2 N ) N. N = ω/2 N

26 Now or if the CLT holds: lim N φ Z N (ω) =e 1 2 ω2 σ 2 Z f ZN (x) = 1 2πσ 2 Z e x2 /2σ 2 Z. 18 Consistency with this limiting form can be seen by expanding φ ZN for small ω ( ω/2 N 1 3! φ ZN (ω) (ω/2 N) 3 ) N ω/2 ω 2 1 N 24 that is identical to the expansion of exp ( ω 2 σ 2 Z /2).

27 CLT Comments A sum of Gaussian RVs is automatically a Gaussian RV (can show using characteristic functions) Convergence to a Gaussian form depends on the actual PDFs of the terms in the sum and their relative variances Exceptions exist!

28 19 CLT: Example of a PDF that does not work The Cauchy distribution and its characteristic function are f X (x) = α π Φ(w) = e α ω 1 α 2 + x 2 Now has a characteristic function Z N = 1 N N x j j=1 Φ N (ω) =e Nα ω / N By inspection the exponential will not converge to a Gaussian. Instead, the sum of N Cauchy RVs is a Cauchy RV. Is the Cauchy distribution a legitimate PDF? No! The variance diverges: X 2 = dx x 2 α π 1 α 2 + x 2.

29 A CLT Problem Consider a set of N quantities that are i.i.d. (independently and identically distributed) with zero mean {a i, i =1,...,N} a i = 0 a i a j = σ 2 aδ ij We are interested in the cross correlation between all unique pairs C N = 1 N X i<j N X = N(N 1)/2 a i a j = 1 N X N 1 i=1 What do you expect <C N > to be? What do you expect the PDF of C N to be? N j=i+1 a i a j

30 A CLT Problem (2) Note: The number of independent quantities (random variables) is N The sum C N has terms that are products of i.i.d. variables Any given term in the sum is s.i. of some of the other terms The PDF of products is different from the PDF of individual factors In the limit N >> 1 there should be many independent terms in the sum N=2: Can show that PDF is symmetric (odd order moments = 0) N>2: Can show that the third moment 0 What gives?

31

32 20 Conditional Probabilities & Bayes Theroem We have considered P (ζ), the probability of an event ζ. Also obeying axioms of probability are conditional probabilities: P (ψ ζ), the probability of the event ψ given that the event ζ has occurred. Recast the axioms as P (ψ ζ) P (ψζ) P (ζ) I. P (ψ ζ) 0 II. P (ψ ζ)+p( ψ ζ) =1 III. P (ψζ η) = P (ψ η)p (ζ ψη) = P (ζ η)p (ψ ζη) How does this relate to experiments? Use the product rule: P (ζ ψη) = P (ζ η)p (ψ ζη) P (ψ η) or, letting M = model (or hypothesis), D = data and I = background information (assumptions), Terms: P (M DI) =P (M I) P (D MI) P (D I) prior: P (M I) sampling distribution for D: P (D MI) (also called likelihood for M) prior predictive for D: P (D I) (also called global likelihood for M or evidence for M)

33 21 Particular strengths of Bayesian method include: 1. One must often be explicit about what is assumed about I, the background information. 2. In assessing models, we get a PDF for parameters rather than just point estimates. 3. Occam s razor (simpler models win, all else being equal) is easily invoked when comparing models. We may have many different models, M i that we wish to compare. Form the odds ratio: fromtheposterior PDFs: P (M i DI): O i,j P (M i DI) P (M j DI) = P (M i I) P (D M i I) P (M j I) P (D M j I).

34 22 Example Data: {k i },i=1,...,n, drawn from Poisson process Poisson PDF: P k = λk e λ k! Want: mean of process Frequentist approach: We need an estimator for the mean; consider the likelihood f(λ) = n 1 n P (k i )= n i=1 k i! λ i=1 ki e nλ. i=1 Maximizing, [ df dλ =0=f(λ) n + λ 1 ] n k i i=1 we obtain an estimator for the mean is k = 1 n n k i. i=1

35 23 Bayesian approach: Likelihood (as before): P (D MI) = n 1 n P (k i )= n ı=1 k i! λ i=1 k i e nλ. i=1 Prior: Assume P (M I) =P (λ I) P (λ I)λ λ U(λ) Prior Predictive: P (D I) dλ U(λ)P (D MI) = n n x n ı=1 k i! Γ(n x). Combining all the above, we find P (λ {k i }I) = nn x Γ(n x) λn x e nλ U(λ) Note that rather than getting a point estimate for the mean, we get a PDF for its value. For hypothesis testing, this is much more useful than a point estimate.