Random fraction of a biased sample

Transcription

1 Random fraction of a biased sample old models and a new one Statistics Seminar Salatiga Geurt Jongbloed TU Delft & EURANDOM work in progress with Kimberly McGarrity and Jilt Sietsma September 2, Delft University of Technology Outline of the talk Waiting time paradox Cut cylinder model for precipitates in steel Equations of the inverse problem Functions of interest Estimation and (some) asymptotics Further problems and future directions September 2, 212 2

2 Waiting time paradox September 2, Waiting time paradox X 1,X 2,..., i.i.d. with cdf F, inter-arrival times September 2, 212 4

3 Waiting time paradox X 1,X 2,..., i.i.d. with cdf F, inter-arrival times Distribution length selected inter-arrival interval: x F (w) (x) = ydf(y) ydf(y) = 1 x ydf(y), m F length-biased distribution corresponding to F. September 2, Waiting time paradox X 1,X 2,..., i.i.d. with cdf F, inter-arrival times Distribution length selected inter-arrival interval: x F (w) (x) = ydf(y) ydf(y) = 1 x ydf(y), m F length-biased distribution corresponding to F. Observed remaining waiting time: Z = UY, with Y F (w), U Unif(, 1) and T U September 2, 212 4

4 Waiting time paradox X 1,X 2,..., i.i.d. with cdf F, inter-arrival times Distribution length selected inter-arrival interval: x F (w) (x) = ydf(y) ydf(y) = 1 x ydf(y), m F length-biased distribution corresponding to F. Observed remaining waiting time: Z = UY, with Y F (w), U Unif(, 1) and T U Z g F (z) =(1 F (z))/m F September 2, Observed remaining waiting time: Z = UY, with Y F (w), U Unif(, 1) and Y U Z g F (z) =(1 F (z))/m F Paradox : X Exp(λ) Z Exp(λ) Problem: Estimate F from Z 1,...,Z n iid g F. September 2, 212 5

5 Observed remaining waiting time: Z = UY, with Y F (w), U Unif(, 1) and Y U Z g F (z) =(1 F (z))/m F Paradox : X Exp(λ) Z Exp(λ) Problem: Estimate F from Z 1,...,Z n iid g F. Some observations: Random Fraction of Biased Sample September 2, Observed remaining waiting time: Z = UY, with Y F (w), U Unif(, 1) and Y U Z g F (z) =(1 F (z))/m F Paradox : X Exp(λ) Z Exp(λ) Problem: Estimate F from Z 1,...,Z n iid g F. Some observations: Random Fraction of Biased Sample Statistical inverse problem September 2, 212 5

6 Observed remaining waiting time: Z = UY, with Y F (w), U Unif(, 1) and Y U Z g F (z) =(1 F (z))/m F Paradox : X Exp(λ) Z Exp(λ) Problem: Estimate F from Z 1,...,Z n iid g F. Some observations: Random Fraction of Biased Sample Statistical inverse problem Shape constrained estimation problem September 2, Cut Cylinder model September 2, 212 6

7 Cut Cylinder model September 2, Cut Cylinder model Model: circular cylinders randomly distributed in 3D height H, squared radius X, (X, H) f The cut cylinders have height H and squared radius Y : (Y,H) f (w) (y, h) = 1 m f yf(y, h) with m f = h x xf(x, h) dx dh = Ef X September 2, 212 8

8 Observations ( squared half width, height): (Z, H) = ( (1 U 2 )Y,H ) where (Y,H) f (w), U Unif(, 1), (Y,H) U. September 2, Observations ( squared half width, height): (Z, H) = ( (1 U 2 )Y,H ) where (Y,H) f (w), U Unif(, 1), (Y,H) U. Therefore: g(z,h) = 1 2 y yfx (y) dy z f(x, h) x z dx September 2, 212 9

9 Observations ( squared half width, height): (Z, H) = ( (1 U 2 )Y,H ) where (Y,H) f (w), U Unif(, 1), (Y,H) U. Therefore: g(z,h) = 1 2 y yfx (y) dy z f(x, h) x z dx This leads to inverse problem: estimate (aspects of) f based on i.i.d. sample (Z 1,H 1 ),...,(Z n,h n ) from g Which aspects? September 2, Aspects of interest Covariance between X and H σ X,H = E f( XH) E f XEf H September 2, 212 1

10 Aspects of interest Covariance between X and H σ X,H = E f( XH) E f XEf H Marginal distribution function of X x F X (x) =P (X x) = x u= h= f(u, h) dh du September 2, Aspects of interest Covariance between X and H σ X,H = E f( XH) E f XEf H Marginal distribution function of X x F X (x) =P (X x) = x u= h= f(u, h) dh du Distribution of Aspect Ratio R = X/H r F AR (r) =P (R r) = September 2, x= h= x/r f(x, h) dh dx

11 Simulation settings f(x, h) =(x h)e x 1 [<h<x< ] September 2, Simulation settings f(x, h) =(x h)e x 1 [<h<x< ] g(z,h) = 8 ( ) z h e z 1 [<h<z] + 8e z (( ) ( ) π 2 + z h IG 2,h z + h ze (h z) ) 1 [<z<h] where IG(α, u) = September 2, u v α 1 e v dv.

12 Simulation settings September 2, Simulation settings f(x, h) =(x h)e x 1 [<h<x< ] F X (x) =1 1 2 e x (x 2 +2x +2) September 2,

13 Simulation settings f(x, h) =(x h)e x 1 [<h<x< ] F X (x) =1 1 2 e x (x 2 +2x +2) F AR (r) =e 1 r 2 3 2r 1/r e v2 dv September 2, Simulation settings F(X).5 F(R) X R September 2,

14 Cross-moment relation 2D-3D Cross moment relation: for α> 1and β such that moments exist πγ(α +1) E g Z α H β = 2m f Γ(α +3/2) E fx α+1/2 H β September 2, Cross-moment relation 2D-3D Cross moment relation: for α> 1and β such that moments exist πγ(α +1) E g Z α H β = 2m f Γ(α +3/2) E fx α+1/2 H β z 1/2 g Z (z) = π 2m f September 2,

15 Cross-moment relation 2D-3D Cross moment relation: for α> 1and β such that moments exist πγ(α +1) E g Z α H β = 2m f Γ(α +3/2) E fx α+1/2 H β z 1/2 g Z (z) = π 2m f E f XH = 2m f Γ(3/2)E g H π = πeg H E g Z 1/2 September 2, Plug-in estimator for covariance From the cross-moment relations: σ XH = πeg H π 2 E gz 1/2 H E g Z 1/2 ˆσ X,H = π n i=1 (H i π i H i ) 2 Z 1/2 n i=1 Z 1/2 i September 2,

16 Plug-in estimator for covariance From the cross-moment relations: σ XH = πeg H π 2 E gz 1/2 H E g Z 1/2 π n ˆσ X,H = i=1 (H i π 2 Z 1/2 i H i ) n i=1 Z 1/2 i Consistent and asymptotically normal: n ( ) ˆσ log n X,H σ X,H D N(,νf 2 ) September 2, Inverse relation for joint density Identifiability and estimation approaches follow immediately from the inverse relation: f(x, h) = d dx x (z x) 1/2 g(z,h) dz z 1/2 g Z (z) dz September 2,

17 Inverse relation for joint density Identifiability and estimation approaches follow immediately from the inverse relation: f(x, h) = d dx September 2, x Special case: marginal of X f X (x) = d dx x Wicksell s corpuscle problem! (z x) 1/2 g(z,h) dz z 1/2 g Z (z) dz (z x) 1/2 g Z (z) dz z 1/2 g Z (z) dz Wicksell s problem September 2,

18 Plug-in estimator for F X Based on inverse relation: F n (x) =1 V n(x) V n (), where V n (x) = x (z x) 1/2 dg Z,n (z) = 1 n n i=1 1 [Zi >x] Zi x September 2, Plug-in estimator for F X Based on inverse relation: V n (x) = x September 2, F n (x) =1 V n(x) V n (), where (z x) 1/2 dg Z,n (z) = 1 n n i=1 Consistent and asymptotically normal: n log n (V n(x) V (x)) D N (,g Z (x)) 1 [Zi >x] Zi x

19 Plug-in estimator for F X, n = F(X) X September 2, Plug-in estimator for F X, n = F(X) X September 2,

20 Projected plug-in estimator for F X Monotone projection of V n : U n (x) = x V n (y) dy ˆV n (x) =rdcm(u n ) x ˆF n (x) =1 ˆV n (x) ˆV n () September 2, Projected plug-in estimator for F X Monotone projection of V n : U n (x) = x V n (y) dy ˆV n (x) =rdcm(u n ) x ˆF n (x) =1 ˆV n (x) ˆV n () Consistent and asymptotically normal: n ( ˆVn (x) V (x)) D N (, 12 ) log n g(x) September 2,

21 Plug-in- and projected estimator for F X F(X) n = X September 2, Plug-in- and projected estimator for F X F(X) n = X September 2,

22 Two histograms, B = F emp (X=3) F(X=3) F iso (X=3) F(X=3) Plug-in: empirical variance.34 Projected: empirical variance.24. September 2, Graphical check Variance of Estimator for F(X=3) Var y =.55x Isotonic Variance Empirical Variance September 2,

23 Plug-in estimator of AR distribution Distribution of Aspect Ratio R = X/H in terms of g: r P (R r) =1 V (r) V () with V (r) = h= z=(hr) 2 g(z,h) z (hr) 2 dz dh plug in empirical distribution of (Z i,h i ) projected estimator based on plug-in estimator September 2, Plug-in- and projected estimator for F AR F(R) n = R September 2,

24 Plug-in- and projected estimator for F AR F(R) n = R September 2, Two histograms, B = F emp (R=5) F(R=5) F iso (R=5) F(R=5) Plug-in: empirical variance.2 Projected: empirical variance.14. September 2, 212 3

25 Recall special instances of RFBS-models in 1D cut-cylinder model for precipitates in steel direct- and inverse problem; aspects of interest nonparametric plug-in estimators projected estimators September 2, Things to do Study plug-in estimators of distributions of other quantities of interest asymptotically Study projected estimators asymptotically; prove variance reduction phenomenon Study estimators of the bivariate distribution function, explore regularization possibilities Relax model assumptions, apply to real data September 2,

26 Real picture 147 Slice distance (µm) September 2, distance (µm) References GROENEBOOM, P.AND J. (1995) Isotonic estimation and rates of convergence in Wicksell s problem. ANNALS OF STATISTICS SEN, B.AND WOODROOFE, M. (212) Bootstrap Confidence Intervals for Isotonic Estimators in a Stereological Problem. BERNOULLI WATSON, G.S. (1971). Estimating functionals of particle size distributions. BIOMETRIKA WICKSELL, S.D. (1925) The corpuscle problem. BIOMETRIKA September 2,

27 Thanks for your attention September 2, Thanks for your attention Questions? September 2,