Inverse problems in statistics

Transcription

1 Inverse problems in statistics Laurent Cavalier (Université Aix-Marseille 1, France) Yale, May p. 1/35

2 Introduction There exist many fields where inverse problems appear Astronomy (Hubble satellite). Econometrics (instrumental variables). Financial mathematics (model calibration). Medical image processing (X-rays). Yale, May p. 2/35

3 Introduction There exist many fields where inverse problems appear Astronomy (Hubble satellite). Econometrics (instrumental variables). Financial mathematics (model calibration). Medical image processing (X-rays). These are problems where we have indirect observations of an object (a function) that we want to reconstruct. Yale, May p. 2/35

4 Inverse problems Let H et G be Hilbert spaces. Let A be a continuous linear operator from H into G. Yale, May p. 3/35

5 Inverse problems Let H et G be Hilbert spaces. Let A be a continuous linear operator from H into G. Given g G find f H such that Af = g. Yale, May p. 3/35

6 Inverse problems Let H et G be Hilbert spaces. Let A be a continuous linear operator from H into G. Given g G find f H such that Af = g. Solving an inverse problem Inversion of the operator A. Yale, May p. 3/35

7 Inverse problems Let H et G be Hilbert spaces. Let A be a continuous linear operator from H into G. Given g G find f H such that Af = g. Solving an inverse problem Inversion of the operator A. If A 1 is not continuous the problem is called ill-posed. Yale, May p. 3/35

8 Inverse problems Let H et G be Hilbert spaces. Let A be a continuous linear operator from H into G. Given g G find f H such that Af = g. Solving an inverse problem Inversion of the operator A. If A 1 is not continuous the problem is called ill-posed. Observe g ε a noisy version of g, then f ε = A 1 g ε could be far from f. Yale, May p. 3/35

9 Inverse problems Let H et G be Hilbert spaces. Let A be a continuous linear operator from H into G. Given g G find f H such that Af = g. Solving an inverse problem Inversion of the operator A. If A 1 is not continuous the problem is called ill-posed. Observe g ε a noisy version of g, then f ε = A 1 g ε could be far from f. Importance of the notion of noise or error. Yale, May p. 3/35

10 Linear inverse problems Let H and G two separable Hilbert spaces. Let A be a known linear bounded operator from the space H to G. Yale, May p. 4/35

11 Linear inverse problems Let H and G two separable Hilbert spaces. Let A be a known linear bounded operator from the space H to G. Let the model : Y = Af + εξ, where Y is the observation, f H unknown, A a continuous linear operator fom H into G, ξ is a white noise, ε corresponds to the noise level. Yale, May p. 4/35

12 Linear inverse problems Let H and G two separable Hilbert spaces. Let A be a known linear bounded operator from the space H to G. Let the model : Y = Af + εξ, where Y is the observation, f H unknown, A a continuous linear operator fom H into G, ξ is a white noise, ε corresponds to the noise level. Reconstruct (estimate) f with the observation Y. Yale, May p. 4/35

13 Linear inverse problems Let H and G two separable Hilbert spaces. Let A be a known linear bounded operator from the space H to G. Let the model : Y = Af + εξ, where Y is the observation, f H unknown, A a continuous linear operator fom H into G, ξ is a white noise, ε corresponds to the noise level. Reconstruct (estimate) f with the observation Y. Projection of a white noise on any orthonormal basis {ψ k } gives a sequence of i.i.d. standard Gaussian random variables. Yale, May p. 4/35

14 Discrete model of inverse problems The standard discrete sample statistical model for linear inverse problems is Y i = Af(X i ) + ξ i, i = 1,...,n, where (X 1,Y 1 ),.., (X n,y n ) are observed (we may assume X i [0, 1]), f is an unknown function in L 2 (0, 1), A is an operator from L 2 (0, 1) into L 2 (0, 1), and ξ i are i.i.d. zero-mean Gaussian random variables of variance σ 2. Yale, May p. 5/35

15 Discrete model of inverse problems The standard discrete sample statistical model for linear inverse problems is Y i = Af(X i ) + ξ i, i = 1,...,n, where (X 1,Y 1 ),.., (X n,y n ) are observed (we may assume X i [0, 1]), f is an unknown function in L 2 (0, 1), A is an operator from L 2 (0, 1) into L 2 (0, 1), and ξ i are i.i.d. zero-mean Gaussian random variables of variance σ 2. Noise level is related to number of observations by ε 1/ n. Yale, May p. 5/35

16 Singular value decomposition A major property of compact operators is that they have a discrete spectrum. Yale, May p. 6/35

17 Singular value decomposition A major property of compact operators is that they have a discrete spectrum. Suppose A A compact operator with a known basis of eigenfunctions in H: A Aϕ k = b 2 k ϕ k. Yale, May p. 6/35

18 Singular value decomposition A major property of compact operators is that they have a discrete spectrum. Suppose A A compact operator with a known basis of eigenfunctions in H: A Aϕ k = b 2 k ϕ k. Singular Value Decomposition (SVD) of A : Yale, May p. 6/35

19 Singular value decomposition A major property of compact operators is that they have a discrete spectrum. Suppose A A compact operator with a known basis of eigenfunctions in H: A Aϕ k = b 2 k ϕ k. Singular Value Decomposition (SVD) of A : Aϕ k = b k ψ k, A ψ k = b k ϕ k, where b k > 0 are the singular values, {ϕ k } o.n.b. on H, {ψ k } o.n.b. on G. Yale, May p. 6/35

20 Singular value decomposition A major property of compact operators is that they have a discrete spectrum. Suppose A A compact operator with a known basis of eigenfunctions in H: A Aϕ k = b 2 k ϕ k. Singular Value Decomposition (SVD) of A : Aϕ k = b k ψ k, A ψ k = b k ϕ k, where b k > 0 are the singular values, {ϕ k } o.n.b. on H, {ψ k } o.n.b. on G. A linear bounded compact operator between two Hilbert spaces may really be seen as an infinite matrix. Yale, May p. 6/35

21 Projection on {ψ k } Projection of Y on {ψ k } : Yale, May p. 7/35

22 Projection on {ψ k } Projection of Y on {ψ k } : Y,ψ k = Af,ψ k + ε ξ,ψ k Yale, May p. 7/35

23 Projection on {ψ k } Projection of Y on {ψ k } : Y,ψ k = f,a ψ k + ε ξ,ψ k Yale, May p. 7/35

24 Projection on {ψ k } Projection of Y on {ψ k } : Y,ψ k = f,a ψ k + ε ξ,ψ k = b k f,ϕ k + ε ξ,ψ k Yale, May p. 7/35

25 Projection on {ψ k } Projection of Y on {ψ k } : Y,ψ k = f,a ψ k + ε ξ,ψ k = b k f,ϕ k + εξ k where {ξ k } standard Gaussian sequence i.i.d., by projection of a white noise ξ on the o.n.b. {ψ k }. Yale, May p. 7/35

26 Sequence space model Equivalent Sequence space model y k = b k θ k + εξ k, k = 1, 2,..., where {θ k } coefficients of f, ξ k N(0, 1) i.i.d., b k 0 singular values. Yale, May p. 8/35

27 Sequence space model Equivalent Sequence space model y k = b k θ k + εξ k, k = 1, 2,..., where {θ k } coefficients of f, ξ k N(0, 1) i.i.d., b k 0 singular values. Estimate θ = {θ k } with the observation Y = {Y k }. Use L 2 risk, it is equivalent to estimate f. Yale, May p. 8/35

28 Sequence space model Equivalent Sequence space model y k = b k θ k + εξ k, k = 1, 2,..., where {θ k } coefficients of f, ξ k N(0, 1) i.i.d., b k 0 singular values. Estimate θ = {θ k } with the observation Y = {Y k }. Use L 2 risk, it is equivalent to estimate f. Remark that b k 0 weaken the signal θ k. Yale, May p. 8/35

29 Sequence space model Equivalent Sequence space model y k = b k θ k + εξ k, k = 1, 2,..., where {θ k } coefficients of f, ξ k N(0, 1) i.i.d., b k 0 singular values. Estimate θ = {θ k } with the observation Y = {Y k }. Use L 2 risk, it is equivalent to estimate f. Remark that b k 0 weaken the signal θ k. Ill-posed problem. Yale, May p. 8/35

30 Inversion We have to invert in some sense the operator A. Yale, May p. 9/35

31 Inversion We have to invert in some sense the operator A. Thus, we obtain the model : where σ k = b 1 k. X k = b 1 k y k = θ k + εσ k ξ k, k = 1, 2,... Yale, May p. 9/35

32 Inversion We have to invert in some sense the operator A. Thus, we obtain the model : X k = b 1 k y k = θ k + εσ k ξ k, k = 1, 2,... where σ k = b 1 k. In the case where the problem is ill-posed the variance term grows to infinity. Yale, May p. 9/35

33 Inversion We have to invert in some sense the operator A. Thus, we obtain the model : X k = b 1 k y k = θ k + εσ k ξ k, k = 1, 2,... where σ k = b 1 k. In the case where the problem is ill-posed the variance term grows to infinity. In this model the aim is to estimate {θ k } by use of {X k }. When k is large the noise in X k may then be very large, making the estimation difficult. Yale, May p. 9/35

34 Inversion We have to invert in some sense the operator A. Thus, we obtain the model : X k = b 1 k y k = θ k + εσ k ξ k, k = 1, 2,... where σ k = b 1 k. In the case where the problem is ill-posed the variance term grows to infinity. In this model the aim is to estimate {θ k } by use of {X k }. When k is large the noise in X k may then be very large, making the estimation difficult. (see Donoho (1995), Mair and Ruymgaart (1996), Johnstone (1999) and C. and Tsybakov (2002)...). Yale, May p. 9/35

35 Difficulty of inverse problems Yale, May p. 10/35

36 Difficulty of inverse problems σ k 1 : Direct problem. Yale, May p. 10/35

37 Difficulty of inverse problems σ k 1 : Direct problem. σ k k β, β > 0 : Mildly ill-posed problem. Yale, May p. 10/35

38 Difficulty of inverse problems σ k 1 : Direct problem. σ k k β, β > 0 : Mildly ill-posed problem. σ k exp(βk), β > 0 : Severely ill-posed problem. Yale, May p. 10/35

39 Difficulty of inverse problems σ k 1 : Direct problem. σ k k β, β > 0 : Mildly ill-posed problem. σ k exp(βk), β > 0 : Severely ill-posed problem. Parameter β is called degree of ill-posedness. Yale, May p. 10/35

40 Examples There exist many examples of operators for which the SVD is known: Yale, May p. 11/35

41 Examples There exist many examples of operators for which the SVD is known: Convolution. Yale, May p. 11/35

42 Examples There exist many examples of operators for which the SVD is known: Convolution. Tomography. Yale, May p. 11/35

43 Examples There exist many examples of operators for which the SVD is known: Convolution. Tomography. Instrumental variables. Yale, May p. 11/35

44 Circular convolution The framework of deconvolution is perhaps one of the most well-known inverse problem. It is used in many applications as econometrics, physics, astronomy, medical image processing. For example, it corresponds to the problem of a blurred signal that one wants to recover from indirect data. Yale, May p. 12/35

45 Circular convolution The framework of deconvolution is perhaps one of the most well-known inverse problem. It is used in many applications as econometrics, physics, astronomy, medical image processing. For example, it corresponds to the problem of a blurred signal that one wants to recover from indirect data. Consider the following convolution operator Af(t) = r f(t) = 1 0 r(t x)f(x)dx, x [0, 1], where r is a known 1-periodic symetric real convolution kernel in L 2 [0, 1]. In this model, A is a linear bounded self-adjoint operator from L 2 [0, 1] to L 2 [0, 1]. Yale, May p. 12/35

46 Blurred cameraman (a) (b) Yale, May p. 13/35

47 Convolution model Define then the following model Y (t) = r f(t) + ε ξ(t), x [0, 1], where Y is observed, f is an unknown periodic function in L 2 [0, 1] and ξ(t) is a white noise on L 2 [0, 1]. Yale, May p. 14/35

48 Convolution model Define then the following model Y (t) = r f(t) + ε ξ(t), x [0, 1], where Y is observed, f is an unknown periodic function in L 2 [0, 1] and ξ(t) is a white noise on L 2 [0, 1]. The SVD basis is then clearly here the Fourier basis {ϕ k (t)}. Yale, May p. 14/35

49 Convolution model Define then the following model Y (t) = r f(t) + ε ξ(t), x [0, 1], where Y is observed, f is an unknown periodic function in L 2 [0, 1] and ξ(t) is a white noise on L 2 [0, 1]. The SVD basis is then clearly here the Fourier basis {ϕ k (t)}. We make the projection on {ϕ k (t)}, in the Fourier domain, and obtain y k = b k θ k + εξ k, where b k = r(x) cos(2πkx)dx for even k, θ k are the Fourier coefficients of f, and ξ k are i.i.d. N(0, 1). Yale, May p. 14/35

50 Tomography scan Yale, May p. 15/35

51 Instrumental variables An economic relationship between a response variable Y and a vector X of explanatory variables is represented by Y i = f(x i ) + U i, i = 1,...,n, where f has to be estimated and U i are the errors. Yale, May p. 16/35

52 Instrumental variables An economic relationship between a response variable Y and a vector X of explanatory variables is represented by Y i = f(x i ) + U i, i = 1,...,n, where f has to be estimated and U i are the errors. This model does not characterize the function f if U is not constrained. The problem is solved if E(U X) = 0. Yale, May p. 16/35

53 Instrumental variables An economic relationship between a response variable Y and a vector X of explanatory variables is represented by Y i = f(x i ) + U i, i = 1,...,n, where f has to be estimated and U i are the errors. This model does not characterize the function f if U is not constrained. The problem is solved if E(U X) = 0. In many structural econometrics models some components of X are endogeneous. Yale, May p. 16/35

54 Instrumental variables An economic relationship between a response variable Y and a vector X of explanatory variables is represented by Y i = f(x i ) + U i, i = 1,...,n, where f has to be estimated and U i are the errors. This model does not characterize the function f if U is not constrained. The problem is solved if E(U X) = 0. In many structural econometrics models some components of X are endogeneous. If Y denotes wages and X, level of education, among other variables. The error U includes, ability, not observed, but influences wages. Yale, May p. 16/35

55 Instrumental variables An economic relationship between a response variable Y and a vector X of explanatory variables is represented by Y i = f(x i ) + U i, i = 1,...,n, where f has to be estimated and U i are the errors. This model does not characterize the function f if U is not constrained. The problem is solved if E(U X) = 0. In many structural econometrics models some components of X are endogeneous. If Y denotes wages and X, level of education, among other variables. The error U includes, ability, not observed, but influences wages. High ability tends to have high level of education, then education and ability are correlated, and thus X and U also. Yale, May p. 16/35

56 Instrumental variables Nevertheless, suppose that we observe another set of data, W i where W is called an instrumental variable for which E(U W) = E(Y f(x) W) = 0. Yale, May p. 17/35

57 Instrumental variables Nevertheless, suppose that we observe another set of data, W i where W is called an instrumental variable for which E(U W) = E(Y f(x) W) = 0. This equation characterizes f by a Fredholm equation of the first kind. Estimation of the function f is in fact an ill-posed inverse problems. Yale, May p. 17/35

58 Instrumental variables Nevertheless, suppose that we observe another set of data, W i where W is called an instrumental variable for which E(U W) = E(Y f(x) W) = 0. This equation characterizes f by a Fredholm equation of the first kind. Estimation of the function f is in fact an ill-posed inverse problems. Not exactly our model of Gaussian white noise, but closely related. Yale, May p. 17/35

59 Instrumental variables Nevertheless, suppose that we observe another set of data, W i where W is called an instrumental variable for which E(U W) = E(Y f(x) W) = 0. This equation characterizes f by a Fredholm equation of the first kind. Estimation of the function f is in fact an ill-posed inverse problems. Not exactly our model of Gaussian white noise, but closely related. Inverse problems have been the topic of many articles in the econometrics literature, see Florens (2003), Hall and Horowitz (2005), Chen and Reiss (2009). Yale, May p. 17/35

60 Inverse problem and sequence space Let the model : Y = Af + εξ, where Y is the observation, f H unknown, A a continuous linear compact operator fom H into G, ξ is a white noise, ε is the noise level. Yale, May p. 18/35

61 Inverse problem and sequence space Let the model : Y = Af + εξ, where Y is the observation, f H unknown, A a continuous linear compact operator fom H into G, ξ is a white noise, ε is the noise level. By using the SVD, we obtain the equivalent sequence space model : X k = θ k + εσ k ξ k, k = 1, 2,.... where σ k. Yale, May p. 18/35

62 Inverse problem and sequence space Let the model : Y = Af + εξ, where Y is the observation, f H unknown, A a continuous linear compact operator fom H into G, ξ is a white noise, ε is the noise level. By using the SVD, we obtain the equivalent sequence space model : X k = θ k + εσ k ξ k, k = 1, 2,.... where σ k. The aim is to estimate (reconstruct) the function f (or the sequence {θ k }) by use of observations. Yale, May p. 18/35

63 Linear estimators Consider here a specific family of estimators. Yale, May p. 19/35

64 Linear estimators Consider here a specific family of estimators. Let λ = (λ 1,λ 2,...) be a sequence of nonrandom weights. Every sequence λ defines a linear estimator ˆθ(λ) = (ˆθ 1, ˆθ 2,...), ˆθk = λ k X k Yale, May p. 19/35

65 Linear estimators Consider here a specific family of estimators. Let λ = (λ 1,λ 2,...) be a sequence of nonrandom weights. Every sequence λ defines a linear estimator ˆθ(λ) = (ˆθ 1, ˆθ 2,...), ˆθk = λ k X k and ˆf(λ) = ˆθ k ϕ k. k=1 Yale, May p. 19/35

66 Linear estimators Consider here a specific family of estimators. Let λ = (λ 1,λ 2,...) be a sequence of nonrandom weights. Every sequence λ defines a linear estimator ˆθ(λ) = (ˆθ 1, ˆθ 2,...), ˆθk = λ k X k and ˆf(λ) = ˆθ k ϕ k. k=1 The L 2 risk of a linear estimator is E ˆf(λ) f 2 = R(θ,λ) = (1 λ k ) 2 θk 2 + ε2 k=1 k=1 σ 2 k λ2 k. Yale, May p. 19/35

67 Classes of linear estimators Projection estimators (spectral cut-off), λ k = I(k N), N > 0. Yale, May p. 20/35

68 Classes of linear estimators Projection estimators (spectral cut-off), λ k = I(k N), N > 0. Tikhonov regularization (penalized), λ k = γσk 2α, α 1, γ > 0. Yale, May p. 20/35

69 Classes of linear estimators Projection estimators (spectral cut-off), λ k = I(k N), N > 0. Tikhonov regularization (penalized), λ k = γσk 2α, α 1, γ > 0. Landweber iteration, λ k = 1 (1 σ 2 k )n,n N. Yale, May p. 20/35

70 Classes of linear estimators Projection estimators (spectral cut-off), λ k = I(k N), N > 0. Tikhonov regularization (penalized), λ k = γσk 2α, α 1, γ > 0. Landweber iteration, Choice of N,γ or n? λ k = 1 (1 σ 2 k )n,n N. Yale, May p. 20/35

71 Ellipsoid of coefficients Assume that f belongs to functional class corresponding to ellipsoids Θ in space of coefficients {θ k }: { } Θ = Θ(a,L) = θ : a 2 k θ2 k L, k=1 where a = {a k }, where a k > 0,a k and L > 0. Yale, May p. 21/35

72 Ellipsoid of coefficients Assume that f belongs to functional class corresponding to ellipsoids Θ in space of coefficients {θ k }: Θ = Θ(a,L) = { θ : } a 2 k θ2 k L, k=1 where a = {a k }, where a k > 0,a k and L > 0. For large values of k coefficients θ k will be decreasing with k and then be small. Yale, May p. 21/35

73 Ellipsoid of coefficients Assume that f belongs to functional class corresponding to ellipsoids Θ in space of coefficients {θ k }: Θ = Θ(a,L) = { θ : } a 2 k θ2 k L, k=1 where a = {a k }, where a k > 0,a k and L > 0. For large values of k coefficients θ k will be decreasing with k and then be small. Assumptions on coefficients θ k usually related to properties (smoothness) on f. Yale, May p. 21/35

74 Sobolev classes Introduce the Sobolev classes { W(α,L) = f = k=1 θ k ϕ k : θ Θ(α,L) } Yale, May p. 22/35

75 Sobolev classes Introduce the Sobolev classes { W(α,L) = f = k=1 θ k ϕ k : θ Θ(α,L) } where Θ(α,L) = Θ(a,L) with a = {a k } polynomial such that a 1 = 0 and a k = { (k 1) α for k odd, k α for k even, where α > 0, L > 0. We have also k = 2, 3,..., Yale, May p. 22/35

76 Sobolev classes Introduce the Sobolev classes { W(α,L) = f = k=1 θ k ϕ k : θ Θ(α,L) } where Θ(α,L) = Θ(a,L) with a = {a k } polynomial such that a 1 = 0 and a k = { (k 1) α for k odd, k α for k even, where α > 0, L > 0. We have also W(α,L) = { f periodic : 1 0 k = 2, 3,..., } (f (α) (t)) 2 dt π 2α L. Yale, May p. 22/35

77 Rates of convergence Function f has Fourier coefficients in some ellipsoid, and the problem is mildly, severely ill-posed or even direct. Rates appear in the following table : Yale, May p. 23/35

78 Rates of convergence Function f has Fourier coefficients in some ellipsoid, and the problem is mildly, severely ill-posed or even direct. Rates appear in the following table : Problem/Functions Direct problem Mildly ill-posed Severely ill-posed Sobolev ε 4α 2α+1 ε 4α 2α+2β+1 (log 1 ε ) 2α Yale, May p. 23/35

79 Comments Rates usually depend on smoothness α of function f and on degree of ill-posedness β. Yale, May p. 24/35

80 Comments Rates usually depend on smoothness α of function f and on degree of ill-posedness β. When β increases rates are slower. Yale, May p. 24/35

81 Comments Rates usually depend on smoothness α of function f and on degree of ill-posedness β. When β increases rates are slower. In direct model, standard rates for nonparametric estimation. For example, 2α/(2α + 1) with Sobolev classes. Yale, May p. 24/35

82 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. Yale, May p. 25/35

83 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. In minimax sense, optimal choice for N. However, choice depends on smoothness α and on degree of ill-posedness β. Yale, May p. 25/35

84 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. In minimax sense, optimal choice for N. However, choice depends on smoothness α and on degree of ill-posedness β. Even if operator A (and its degree β) is known, no real meaning to consider smoothness of f as known. Yale, May p. 25/35

85 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. In minimax sense, optimal choice for N. However, choice depends on smoothness α and on degree of ill-posedness β. Even if operator A (and its degree β) is known, no real meaning to consider smoothness of f as known. Notion of adaptation and oracle inequalities, i.e. how to choose bandwidth N without prior assumptions on f. Yale, May p. 25/35

86 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). Yale, May p. 26/35

87 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). Define the oracle λ 0 as R(θ,λ 0 ) = inf λ Λ R(θ,λ). The oracle corresponds to the best possible choice in Λ, i.e. the one which minimizes the risk. Yale, May p. 26/35

88 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). Define the oracle λ 0 as R(θ,λ 0 ) = inf λ Λ R(θ,λ). The oracle corresponds to the best possible choice in Λ, i.e. the one which minimizes the risk. However, this is not an estimator since the risk depends on the unknown θ, the oracle will depend also. Yale, May p. 26/35

89 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). Define the oracle λ 0 as R(θ,λ 0 ) = inf λ Λ R(θ,λ). The oracle corresponds to the best possible choice in Λ, i.e. the one which minimizes the risk. However, this is not an estimator since the risk depends on the unknown θ, the oracle will depend also. An oracle is the best in the family, but it knows the true θ. Yale, May p. 26/35

90 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. Yale, May p. 27/35

91 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). Yale, May p. 27/35

92 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). This method goes back to Akaike Information Criteria (AIC) in Akaike (1973) and Mallows C p (1973). Yale, May p. 27/35

93 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). This method goes back to Akaike Information Criteria (AIC) in Akaike (1973) and Mallows C p (1973). Originally, the URE was in the context of regression estimation. Nowadays, it is a basic adaptation tool for many statistical models. Yale, May p. 27/35

94 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). This method goes back to Akaike Information Criteria (AIC) in Akaike (1973) and Mallows C p (1973). Originally, the URE was in the context of regression estimation. Nowadays, it is a basic adaptation tool for many statistical models. This idea appears also in all the cross-validation techniques. Yale, May p. 27/35

95 URE in inverse problems For inverse problems, this method was studied in C., Golubev, Picard and Tsybakov (2002), where exact oracle inequalities were obtained. Yale, May p. 28/35

96 URE in inverse problems For inverse problems, this method was studied in C., Golubev, Picard and Tsybakov (2002), where exact oracle inequalities were obtained. In this setting, the functional U(X,λ) = (1 λ k ) 2 (X 2 k ε2 σ 2 k ) + ε2 σ 2 k λ2 k k=1 k=1 Yale, May p. 28/35

97 URE in inverse problems For inverse problems, this method was studied in C., Golubev, Picard and Tsybakov (2002), where exact oracle inequalities were obtained. In this setting, the functional U(X,λ) = (1 λ k ) 2 (X 2 k ε2 σ 2 k ) + ε2 σ 2 k λ2 k k=1 k=1 is an unbiased estimator of R(θ,λ): R(θ,λ) = E θ U(X,λ), λ. Yale, May p. 28/35

98 Data-driven choice Unbiased risk estimation suggests to minimize over λ Λ the functional U(X,λ) in place of R(θ,λ). Yale, May p. 29/35

99 Data-driven choice Unbiased risk estimation suggests to minimize over λ Λ the functional U(X,λ) in place of R(θ,λ). This leads to the following data-driven choice of λ: λ = arg min λ Λ U(X,λ). Yale, May p. 29/35

100 Data-driven choice Unbiased risk estimation suggests to minimize over λ Λ the functional U(X,λ) in place of R(θ,λ). This leads to the following data-driven choice of λ: λ = arg min λ Λ U(X,λ). Define then the estimator θ by θ k = λ k X k. Yale, May p. 29/35

101 Assumptions Denote S = ( maxλ Λ k=1 σ4 k λ2 k min λ Λ k=1 σ4 k λ2 k ) 1/2. Let the following assumptions hold. Yale, May p. 30/35

102 Assumptions Denote S = ( maxλ Λ k=1 σ4 k λ2 k min λ Λ k=1 σ4 k λ2 k ) 1/2. Let the following assumptions hold. For any λ Λ 0 < k=1 σ 2 k λ2 k <, max sup λ k 1. λ Λ k Yale, May p. 30/35

103 Assumptions Denote S = ( maxλ Λ k=1 σ4 k λ2 k min λ Λ k=1 σ4 k λ2 k ) 1/2. Let the following assumptions hold. For any λ Λ 0 < k=1 σ 2 k λ2 k <, max sup λ k 1. λ Λ k There exists a constant C 1 > 0 such that, uniformly in λ Λ, σ 4 k λ2 k C 1 σ 4 k λ4 k. k=1 k=1 Yale, May p. 30/35

104 Oracle inequality for URE Theorem. Suppose σ k k β, β 0. Assume that Λ is finite with cardinality D and belongs to the family of Projection, Tikhonov or Pinsker estimators. There exist constants γ,c > 0 such that θ l 2, we have for B large enough, E θ θ θ 2 (1 + γb 1 ) min λ Λ R(θ,λ) + BC ε 2 (log(ds)) 2β+1. Yale, May p. 31/35

105 Oracle inequality for URE Theorem. Suppose σ k k β, β 0. Assume that Λ is finite with cardinality D and belongs to the family of Projection, Tikhonov or Pinsker estimators. There exist constants γ,c > 0 such that θ l 2, we have for B large enough, E θ θ θ 2 (1 + γb 1 ) min λ Λ R(θ,λ) + BC ε 2 (log(ds)) 2β+1. The data-driven choice by URE mimics the oracle. Yale, May p. 31/35

106 Simulations Discrete model : inverse problem. Y (i) = g f ( i m ) + ε mξ(i), i = 1,...,m, Yale, May p. 32/35

107 Simulations Discrete model : inverse problem. where Y (i) = g f ( i m ) + ε mξ(i), i = 1,...,m, f(t) = 0.5n(t, 0.4, 0.12) + 0.5n(t, 0.7, 0.08), and g(t) = exp( 10 t 0.5 ), β 2. Yale, May p. 32/35

108 Simulations Discrete model : inverse problem. where and Y (i) = g f ( i m ) + ε mξ(i), i = 1,...,m, f(t) = 0.5n(t, 0.4, 0.12) + 0.5n(t, 0.7, 0.08), g(t) = exp( 10 t 0.5 ), β 2. Here m = 1000 et ε 2 = Signal/Noise = 100. Estimator by truncated Fourier series. Yale, May p. 32/35

109 Simulations Discrete model : inverse problem. where and Y (i) = g f ( i m ) + ε mξ(i), i = 1,...,m, f(t) = 0.5n(t, 0.4, 0.12) + 0.5n(t, 0.7, 0.08), g(t) = exp( 10 t 0.5 ), β 2. Here m = 1000 et ε 2 = Signal/Noise = 100. Estimator by truncated Fourier series. With ε 2 k N σ 2 k 1/ log(1/ε). Yale, May p. 32/35

110 Simulations Discrete model : inverse problem. where and Y (i) = g f ( i m ) + ε mξ(i), i = 1,...,m, f(t) = 0.5n(t, 0.4, 0.12) + 0.5n(t, 0.7, 0.08), g(t) = exp( 10 t 0.5 ), β 2. Here m = 1000 et ε 2 = Signal/Noise = 100. Estimator by truncated Fourier series. With ε 2 k N σ 2 k 1/ log(1/ε). Yale, May p. 32/35

111 True function f. Estimator f. 3 Estimation de f Yale, May p. 33/35

112 Oracle by projection. Estimator f Risque Quadratique Signal/Bruit Yale, May p. 34/35

113 Comments Simulations correspond more or less to theory. Yale, May p. 35/35

114 Comments Simulations correspond more or less to theory. Limitation on the size of the family. Yale, May p. 35/35

115 Comments Simulations correspond more or less to theory. Limitation on the size of the family. Method not always stable enough. Yale, May p. 35/35

116 Comments Simulations correspond more or less to theory. Limitation on the size of the family. Method not always stable enough. Need for stronger penalties than the URE penalty (or AIC). Yale, May p. 35/35

117 Comments Simulations correspond more or less to theory. Limitation on the size of the family. Method not always stable enough. Need for stronger penalties than the URE penalty (or AIC). Different method called Risk Hull Method, defined in C. and Golubev (2006). Yale, May p. 35/35