Lecture 11: Probability, Order Statistics and Sampling

Transcription

1 5-75: Graduate Algorithms February, 7 Lecture : Probability, Order tatistics and ampling Lecturer: David Whitmer cribes: Ilai Deutel, C.J. Argue Exponential Distributions Definition.. Given sample space Ω equipped with probability measure p, a random variable is a function X : Ω R Definition.. Given a random variable X, a probability density function (PDF) f X for X satisfies: P (a X b) b Remark.3. + f X (x)dx a f X (x)dx Example.4. Exponential distribution with parameter : X Exp() { e x : x f X (x) : otherwise 3.5 X Exp() X Exp() X Exp(3) fx(x) x Definition.5. The cumulative distribution function (CDF) for X is: F X Pr(X x) x f X (x)dx Example.6. Exponential distribution with parameter : X Exp() For x, x F X (x) e x dx e t x e x

2 Remark.7. In general, f X (x) df X(x) dx Definition.8. The expected value of X is: E[X] + xf X (x)dx Example.9. For an exponential distribution with parameter : X Exp() E[X] + xe x dx Proposition. (Memoryless Property of Exp()). If X Exp() then for any reals s, t, Pr[X s + t X s] Pr[X t] Proof. Pr[X s + t X s] ( e (s+t)) ( e s ) e (s+t) e s e t Pr[X t] Order tatistics Consider n i.i.d random variables X,..., X n. Definition.. X (k) is the k th smallest of X,..., X n. That is, X (i) are a reordering of X i s.t. X () X ()... X (n) Note that any single X (i) is a random variable, so it has a PDF f X and CDF F X. Example.. Let X i Exp(). In order to have X () x we need to have X i x (which has probability F X (x)) for exactly one value of i and X i > x (probability F X (x)) for the remaining n values of i. There are n ways to choose which X i x, thus we have Thus X () Exp(n), so E[X () ] n. We now establish the expectation of X (n). Claim.3. E[X (n) ] Θ( log n ). f X() (x) f X (x)( F X (x)) n n e x(n ) n ne nx

3 Proof. Let i X (i+) X (i) for i. X () X () X (3) X (n ) X (n) n i is distributed as min(x,..., X n i ) X (i) conditioned on min(x,..., X n i ) X (i). By the memoryless property of exponential random variables, [X (j) X (j) X (i) ] X (j). Thus i is the minimum of n i exponential random variables with parameter, so by example., i Exp((n i)), and E[ i ] (n i). Then: n E[X (n) ] E[ i ] + E[X () ] i n i (n i) + n ( n ) log n We now bound the variation of X (n). Claim.4. Pr[X (n) log n ] n Proof. Thus we have [ Pr X i c log n ] ( ) c log n F X Letting c gives Pr[X (n) log n ] n. 3 ampling from distributions 3. Uniform distributions Let U be the uniform distribution on [, ]. { : x [, ] PDF: f U (x) : otherwise x : x [, ] CDF: F U (x) : x : x ( ) e c log n n c [ Pr X (n) c log n ] [ n Pr X i c log n ] n c 3

4 We will show how to use U to generate samples from other distributions. Example 3.. Unif(, ) X U X Unif(, ) 3. Inverse transform sampling ay that we want to sample from random variable X and we have Y U. We want to find some function g such that g(y ) is distributed like X. To this end, note that if g(y ) is an increasing function such that F g(y ) F X then o g (z) F X (z) g(z) F X (z) Example 3.. X Exp() F g(y ) (z) Pr(g(Y ) z) Pr(Y g (z)) g (z) To solve for F X F X (x) { e x : x : otherwise Thus Y e X e X Y X ln( Y ) X ln( y) ln( Y ) ln(y ) Exp() Exp() The second line follows since Y and Y are both distributed as U. 3.3 Drawing from a normal distribution A standard normal random variable N (, ), mean, variance, is given by PDF: f(x) π e x / CDF: F (x) x e t / dt π How do you sample from a normal distribution? The following theorem suggests that standard inverse transform sampling will not suffice. Theorem 3.3. F (x) is not an elementary function 4

5 3.3. Box-Muller sampling Consider -dimensional normal (X, Y ) (X N (, ), Y N (, )) with PDF: f(x, y) π e x / e y / π e x In polar coordinates, f(r, θ) π e r / (circular/spherical symmetry: doesn t depend on θ). +y Let R be the random variable X + Y. The CDF F R (r) is given by F R (r) Pr(R r) (integral of f(r, θ) over disc of radius r) π r r r e u/ π e r / r dr dθ e r / r dr e r / We can now apply inverse transform sampling du (set u r ) Y e R / R ln( Y ) Thus F R (y) ln( y). To sample X N (, ), first draw from D normal (polar). Let V, W U. Return (R, θ), where R ln(v ) and θ πw. The rectangular coordinates, (R cos θ, R sin θ), are independently distributed as N (, ) Polar form of Box-Muller We can improve on this by eliminating the trigonometric functions, whose computation is costly. We will show an alternate way to sample normal variables via efficiently sampling a point from the uniform distribution on the unit disk. Consider generating a random variable as follows:. Choose (X, Y ) uniformly at random on unit disk {X : X }. Let X + Y ( ) ln ln 3. Return X, Y We will show that this is equivalent to Box-Muller. Claim 3.4. If (X, Y ) is chosen uniformly at random from the unit disk, then X + Y U. 5

6 Proof. It suffices to show that Pr[X + Y t] t Pr[X + Y t] Pr[ X + Y t] Area of circle with radius t Area of circle with radius πt π t Also, symmetry implies that angle θ tan ( Y ) is uniform in [, π] X o (X, Y ) uniform on unit disk is distributed as ( V cos πw, V sin πw ) for V, W U. ( ) ln ln V and X, Y (R cos θ, R sin θ) above ampling from the unit disk To complete, we demonstrate how to generate uniform random point on unit disk without trigonometry. Intuitively, we throw darts at the square [, ] [, ] until we hit the unit disk. Let V, W U, X U, Y U so that X and Y are uniform on [, ]. Let X + Y. Algorithm:. Draw V, W, calculate X, Y,. If, return (X, Y ) 3. Else, try again This is an example of Rejection ampling. The probability of success in each iteration is Area of circle Area of square π 4, so in expectation we need to sample 4 π 4 3 times. 6