IEOR 165 Lecture 10 Distribution Estimation

Transcription

1 IEOR 165 Lecture 10 Distribution Estimation 1 Motivating Problem Consider a situation were we ave iid data x i from some unknown distribution. One problem of interest is estimating te distribution tat is generating te data. Tere are many useful examples of tis abstract problem, including: We are analyzing a telepone call center, and we measure te amount of time required to provide service to eac incoming pone call. Toug an exponential distribution is a commonly used model, it can be useful to estimate te distribution of te call lengts to elp validate te use of an exponential distribution model. In some cases, we may find by using te data tat an exponential distribution is not a good model. We are analyzing midterm scores witin a class, and would like to gain a better understanding of ow well te class scored on te exam. We are taking pictures wit a digital camera, and would like to determine if our picture is underexposed or overexposed. Witin te problem of estimating te distribution using iid x i, tere are two distinct problems: 1. Estimate te cdf F x (u). 2. Estimate te pdf f x (u). We will consider eac of tese problems. We will drop te x subscript to simply te notation. 2 Empirical Distribution Function One natural estimator for te cdf F (u) is known as te empirical distribution function ˆF (u), and it is defined as ˆF n (u) = 1 n 1(X i u). n In fact, te Glivenko-Cantelli teorem tell us tat ( P lim ˆF ) n (u) F (u) = 0 = 1. sup n u R Tis means te empirical distribution function ˆF n (u) as good statistical problems, because it means tat wit probability 1 te discrepancy between ˆF (u) and F (u) goes to 0. 1

2 2.1 Example: Telepone Call Data Suppose te lengts of calls at a call center are {0.66, 0.05, 0.27, 1.26, 1.51, 0.38, 1.79, 0.94, 0.48, 0.89}. Q: Plot te empirical distribution function. A: Te first step is to plot te location of eac data point (top plot). Te second step is to draw te empirical distribution function (middle plot) by increasing te function value by 1/n = 1/10 = 0.1 at te location of eac data point. Te bottom plot compares te empirical distribution to te true distribution function (an exponential distribution wit rate parameter λ = 1) tat was used to generate tis data. 3 Callenge wit Density Estimation Toug te empirical distribution function ˆF n (u) as good statistical problems, it is not wellsuited for estimating te density (i.e., te pdf) of te distribution. Recall tat wen a density 2

3 exists, it is equal to te derivative of te distribution function. And so te reason tat ˆF n (u) is not well-suited for estimating te pdf is tat it is not differentiable in te normal sense. To better understand tis, recall tat te Dirac delta function is defined as a measure suc tat { 1, if 0 A δ(a) = 0, oterwise Informally, te Dirac delta function is a function defined suc tat { 0, if u 0 δ(u) = +, if u = 0 and + g(u)δ(u)du = g(0). Ten, we can define an estimate of te density function by ˆf n edf (u) = 1 n since te following relationsip olds 1(t u) R n δ(u x i ) edf ˆf n (dt) = ˆF n (u). Observe tat for a differentiable distribution F (u) wit corresponding density f(u), we ave tat u F (u) = f(t)dt = 1(t u)f(t)dt. ˆf edf n So we can interpret te (u) as a derivative of te non-differentiable ˆF n (u). However, tere edf is a problem wit using tis ˆf n (u) as an estimate of te density. It essentially places all te density mass at points were te data x i as been measured; te density is zero at every oter point on te real line. Tis is not a good estimate of te density because if we know tat our unknown density is continuous, ten we would rater ave an estimate tat interpolates between measured data points. 3.1 Example: Telepone Call Data Suppose te lengts of calls at a call center are {0.66, 0.05, 0.27, 1.26, 1.51, 0.38, 1.79, 0.94, 0.48, 0.89}. R 3

4 Q: Plot te estimated pdf using te derivative of te empirical distribution function. A: We plot te location of eac data point and give te Dirac delta an amplitude of 1/n = 1/10 = Histograms Te simplest estimate of te density is instead a istogram. Te idea of a istogram is to 1. begin by specifying a support for te distribution, meaning specifying te range over wic te density is non-zero; suppose we specify tat te support is [u, v]; 2. next specify a set of bins partitioning te support, meaning specifying a set of strictly increasing values b 0 = u < b 1 <... < b N = v tat start at u and end at v; 3. we count te number of data points falling into eac bin; specifically, we define n C j = 1(b j 1 x i < b j ), for j = 1,..., N 1 C N = n 1(b N 1 x i b N ). 4. finally, we define our istogram estimate of te density by C j, if b n(b j b j 1 ) j 1 u < b j ˆf n is C (u) = N n(b, if u = b N b N 1 ) N 0, oterwise Note tat instead of specifying te edges of te bins b 0,..., b N, we could ave instead specified te desired number of bins N and set te bin edges to be b j = u + j (v u)/n. 4

5 4.1 Example: Telepone Call Data Suppose te lengts of calls at a call center are {0.66, 0.05, 0.27, 1.26, 1.51, 0.38, 1.79, 0.94, 0.48, 0.89}. Q: Plot te istogram using bin edges {0, 0.5, 1, 1.5, 2, 10}. A: We first count te number of data points falling into eac bin. We ave 4,3,1,2,0 points in bins 0 0.5, 0.5 1, 1 1.5, 1.5 2, 2 10, respectively. Te next step is to use te formulate C j n(b j b j 1 ) to normalize te data counts: 4 10 (0.5 0) = 0.8 Finally, we plot te istogram: 3 = (1 0.5) 1 = (1.5 1) 2 = (2 1.5) 0 = 0 10 (10 2) 5

6 Tis top plot is te istogram, and te bottom plot compares te istogram to te actual pdf tat was used to generate te data. 4.2 Example: Image Exposure Histograms are useful in taking and editing digital potograps. For instance, an image wit correct exposure and its istogram is: An overexposed image and its istogram is: 6

7 An underexposed image and its istogram is: 5 Kernel Density Estimation 5.1 Motivation Histogram estimates of te density f(u) are not continuous, and so it is interesting to consider oter approaces tat produce continuous estimates of te density. One idea is to combine te idea of a istogram wit te density estimate (written in terms of Dirac deltas) tat was generated by differentiating te empirical distribution function. To start, recall tat te problem of te density estimate generated by te empirical distribution function ˆf n edf (u) = 1 n n δ(u x i ) is tat it is zero at all points except te x i were data was seen. Te next ting to note is tat te istogram works by giving widt to counting te amount of data in some region; te Dirac deltas were problematic because tey give zero widt to counting data in some region. 5.2 Kernel Function Given tese two ideas, a proposal for anoter approac to estimate te density is to give te Dirac deltas some widt by replacing tem by copies of a kernel function K( ) : R R tat as finite support: K(u) = 0 for u 1; even symmetry: K(u) = K( u); 7

8 positive values: K(u) > 0 for u < 1; unit integral: K(u)du = 1. R Furtermore, we would like to be able to specify te widt of te copies of te kernel functions. We will use te variable to denote bandwidt, and note tat te function 1 K(u/) as support over [, ]; even symmetry 1 K(u/) = 1 K( u/); positive values 1 K(u/) > 0 for u < ; unit integral: R 1 K(u/)du = 1; maximum value of 1 max u K(u). An important conceptual point is tat 1 K(u/) weakly converges to a Dirac delta wen 0. It is interesting to consider some examples of kernel functions. Te table below lists some common examples: Kernel Function Equation Uniform K(u) = 1 1( u 1) 2 Triangular K(u) = (1 u )1( u 1) Epanecnikov K(u) = 3(1 4 u 2 )1( u 1) Quartic/biweigt K(u) = 15(1 16 u 2 ) 2 1( u 1) Te Epanecnikov kernel is plotted below, and te oter kernel functions essentially look te same. 8

9 5.3 Estimate Te corresponding estimate of te density is ten given by ˆf n kde (u) = edf ˆf n (u) 1 K(u/), were te symbol denotes a convolution. Recall tat a convolution of two functions f, g is defined as f(x) g(x) = f(τ) g(τ x)dτ. R In our context, we can simplify te convolution integral, wic results in te following equivalent equation for te density estimate ˆf n kde (u) = 1 n ( ) u xi K. n Te kernel density estimate naturally generalizes to iger dimensions: ˆf n kde (u) = 1 n ( ) u xi K, n d were d is te dimension of te random variables (i.e., x i R d ). 5.4 Example: Telepone Call Data Suppose te lengts of calls at a call center are {0.66, 0.05, 0.27, 1.26, 1.51, 0.38, 1.79, 0.94, 0.48, 0.89}. Q: Suppose we coose = 0.5, and tat we use te Epanecnikov kernel. Compute te estimated pdf using te kernel density approac at te point u = 0.7. A: We first compute te quantities K ( ) u x i for eac data point. We ave K(( )/0.5) = K(0.08) = 3/4 ( ) = K(( )/0.5) = K(1.3) = 0 K(( )/0.5) = K(0.86) = 3/4 ( ) = K(( )/0.5) = K( 1.12) = 0 K(( )/0.5) = K( 1.62) = 0 K(( )/0.5) = K(0.64) = 3/4 ( ) = K(( )/0.5) = K( 2.18) = 0 K(( )/0.5) = K( 0.48) = 3/4 ( ) = K(( )/0.5) = K(0.44) = 3/4 ( ) = K(( )/0.5) = K( 0.38) = 3/4 ( ) =

10 Finally, we compute ˆf kde n (0.7) = 1 n n ( ) 0.7 xi K = = Plotting te kernel density estimate (KDE) by and is difficult. A computer is typically used, and standard packages exist to do tis. One important point is tat te estimate is sensitive to te value of te bandwidt. Unfortunately, te bandwidt cannot be cosen using cross-validation. An example of te estimated density for tree different values of and compared to te true pdf is sown below: 10