Inverse problems in statistics

Transcription

1 Inverse problems in statistics Laurent Cavalier (Université Aix-Marseille 1, France) YES, Eurandom, 10 October 2011 p. 1/32

2 Part II 2) Adaptation and oracle inequalities YES, Eurandom, 10 October 2011 p. 2/32

3 Part II 2) Adaptation and oracle inequalities Regularization methods Classes of functions Rates of convergence Adaptation Oracle inequalities Unbiased risk estimation (URE) YES, Eurandom, 10 October 2011 p. 2/32

4 Inverse problem and sequence space Let the model : Y = Af +εξ, where Y is the observation, f H unknown, A a continuous linear compact operator fom H into G, ξ is a white noise, ε is the noise level. YES, Eurandom, 10 October 2011 p. 3/32

5 Inverse problem and sequence space Let the model : Y = Af +εξ, where Y is the observation, f H unknown, A a continuous linear compact operator fom H into G, ξ is a white noise, ε is the noise level. By using the SVD, we obtain the equivalent sequence space model : X k = θ k +εσ k ξ k, k = 1,2,... where σ k. YES, Eurandom, 10 October 2011 p. 3/32

6 Inverse problem and sequence space Let the model : Y = Af +εξ, where Y is the observation, f H unknown, A a continuous linear compact operator fom H into G, ξ is a white noise, ε is the noise level. By using the SVD, we obtain the equivalent sequence space model : X k = θ k +εσ k ξ k, k = 1,2,... where σ k. The aim is to estimate (reconstruct) the function f (or the sequence {θ k }) by use of observations. YES, Eurandom, 10 October 2011 p. 3/32

7 Regularization methods In ill-posed inverse problems, find regularization methods to get fine reconstruction of f. YES, Eurandom, 10 October 2011 p. 4/32

8 Regularization methods In ill-posed inverse problems, find regularization methods to get fine reconstruction of f. The normal equation is A Y = A Af +ε A ξ. YES, Eurandom, 10 October 2011 p. 4/32

9 Regularization methods In ill-posed inverse problems, find regularization methods to get fine reconstruction of f. The normal equation is A Y = A Af +ε A ξ. One has to estimate the solution (A A) 1 A Y. Problem in ill-posed situation is that operator A A not invertible. YES, Eurandom, 10 October 2011 p. 4/32

10 Regularization methods In ill-posed inverse problems, find regularization methods to get fine reconstruction of f. The normal equation is A Y = A Af +ε A ξ. One has to estimate the solution (A A) 1 A Y. Problem in ill-posed situation is that operator A A not invertible. With regularization methods, get some inversion of a related operator. YES, Eurandom, 10 October 2011 p. 4/32

11 Regularization methods We call a regularization method an estimator defined by ˆf γ = Φ γ (A A)A Y, where Φ γ C(σ(A A)) depending on some regularization parameter γ > 0. YES, Eurandom, 10 October 2011 p. 5/32

12 Regularization methods We call a regularization method an estimator defined by ˆf γ = Φ γ (A A)A Y, where Φ γ C(σ(A A)) depending on some regularization parameter γ > 0. Examples of regularization methods or estimators which are commonly used. YES, Eurandom, 10 October 2011 p. 5/32

13 Regularization methods We call a regularization method an estimator defined by ˆf γ = Φ γ (A A)A Y, where Φ γ C(σ(A A)) depending on some regularization parameter γ > 0. Examples of regularization methods or estimators which are commonly used. Methods defined in the spectral domain even if some may be computed without using the whole spectrum. YES, Eurandom, 10 October 2011 p. 5/32

14 Estimation procedures Suppose now that operator A is compact. Using SVD, obtain sequence space model and statisticians prefer to work with. YES, Eurandom, 10 October 2011 p. 6/32

15 Estimation procedures Suppose now that operator A is compact. Using SVD, obtain sequence space model and statisticians prefer to work with. Many regularization methods may be expressed in a statistical framework, and usually correspond to some known estimation method in statistics. YES, Eurandom, 10 October 2011 p. 6/32

16 Estimation procedures Suppose now that operator A is compact. Using SVD, obtain sequence space model and statisticians prefer to work with. Many regularization methods may be expressed in a statistical framework, and usually correspond to some known estimation method in statistics. Notion of regularization not really used in statistics. However, more standard definition related. YES, Eurandom, 10 October 2011 p. 6/32

17 Linear estimators Consider here a specific family of estimators. YES, Eurandom, 10 October 2011 p. 7/32

18 Linear estimators Consider here a specific family of estimators. Let λ = (λ 1,λ 2,...) be a sequence of nonrandom weights. Every sequence λ defines a linear estimator ˆθ(λ) = (ˆθ 1,ˆθ 2,...), ˆθk = λ k X k YES, Eurandom, 10 October 2011 p. 7/32

19 Linear estimators Consider here a specific family of estimators. Let λ = (λ 1,λ 2,...) be a sequence of nonrandom weights. Every sequence λ defines a linear estimator ˆθ(λ) = (ˆθ 1,ˆθ 2,...), ˆθk = λ k X k and ˆf(λ) = ˆθ k ϕ k. k=1 YES, Eurandom, 10 October 2011 p. 7/32

20 Linear estimators Consider here a specific family of estimators. Let λ = (λ 1,λ 2,...) be a sequence of nonrandom weights. Every sequence λ defines a linear estimator ˆθ(λ) = (ˆθ 1,ˆθ 2,...), ˆθk = λ k X k and ˆf(λ) = ˆθ k ϕ k. k=1 The L 2 risk of a linear estimator is E ˆf(λ) f 2 = R(θ,λ) = (1 λ k ) 2 θ 2 k +ε2 k=1 k=1 σ 2 k λ2 k. YES, Eurandom, 10 October 2011 p. 7/32

21 Equivalence in sequence space By use of the SVD in the spectral theorem one obtains for a general regularization method ˆf(λ) = Φ γ (A A)A Y = Φ γ (b 2 k )b ky k ϕ k = Φ γ (b 2 k )b2 k X kϕ k, k=1 k=1 YES, Eurandom, 10 October 2011 p. 8/32

22 Equivalence in sequence space By use of the SVD in the spectral theorem one obtains for a general regularization method ˆf(λ) = Φ γ (A A)A Y = Φ γ (b 2 k )b ky k ϕ k = Φ γ (b 2 k )b2 k X kϕ k, k=1 k=1 corresponds to the special case of linear estimator with λ k = Φ γ (b 2 k )b2 k. YES, Eurandom, 10 October 2011 p. 8/32

23 Projection estimators Standard projection weights λ k = I(k N) The projection estimator (also called spectral cut-off). YES, Eurandom, 10 October 2011 p. 9/32

24 Projection estimators Standard projection weights λ k = I(k N) The projection estimator (also called spectral cut-off). ˆθ(N) = { X k, k N, 0, k > N. The value N is called the bandwidth. YES, Eurandom, 10 October 2011 p. 9/32

25 Projection estimators Standard projection weights λ k = I(k N) The projection estimator (also called spectral cut-off). ˆθ(N) = { X k, k N, 0, k > N. The value N is called the bandwidth. The projection estimator is then defined by ˆf N = N k=1 X k ϕ k. YES, Eurandom, 10 October 2011 p. 9/32

26 Projection estimators Standard projection weights λ k = I(k N) The projection estimator (also called spectral cut-off). ˆθ(N) = { X k, k N, 0, k > N. The value N is called the bandwidth. The projection estimator is then defined by ˆf N = N k=1 X k ϕ k. Estimate first N coefficients θ k by empirical counter-part X k and then estimate remainder terms by 0 for k > N. YES, Eurandom, 10 October 2011 p. 9/32

27 Tikhonov regularization Tikhonov method is one well-known regularization method. YES, Eurandom, 10 October 2011 p. 10/32

28 Tikhonov regularization Tikhonov method is one well-known regularization method. Tikhonov regularization method (Tikhonov (1963)). In this method, minimize functional L γ (ϕ): { Aϕ Y 2 +γ ϕ 2}, inf ϕ H where γ > 0 is some tuning parameter. YES, Eurandom, 10 October 2011 p. 10/32

29 Tikhonov regularization Tikhonov method is one well-known regularization method. Tikhonov regularization method (Tikhonov (1963)). In this method, minimize functional L γ (ϕ): { Aϕ Y 2 +γ ϕ 2}, inf ϕ H where γ > 0 is some tuning parameter. The minimum is then attained by ˆf γ = (A A+γI) 1 A Y. YES, Eurandom, 10 October 2011 p. 10/32

30 Tikhonov regularization Tikhonov method is one well-known regularization method. Tikhonov regularization method (Tikhonov (1963)). In this method, minimize functional L γ (ϕ): { Aϕ Y 2 +γ ϕ 2}, inf ϕ H where γ > 0 is some tuning parameter. The minimum is then attained by ˆf γ = (A A+γI) 1 A Y. Choice of γ very sensitive since characterizes balance between fitting and smoothing. YES, Eurandom, 10 October 2011 p. 10/32

31 Tikhonov estimator Define Tikhonov estimator by its form in SVD domain : λ k = 1 1+γσ 2 k. YES, Eurandom, 10 October 2011 p. 11/32

32 Tikhonov estimator Define Tikhonov estimator by its form in SVD domain : λ k = 1 1+γσ 2 k. In special case where A = I, estimator defined and computed as a modified version of Tikhonov regularization and called spline (Wahba (1990)). YES, Eurandom, 10 October 2011 p. 11/32

33 Tikhonov estimator Define Tikhonov estimator by its form in SVD domain : λ k = 1 1+γσ 2 k. In special case where A = I, estimator defined and computed as a modified version of Tikhonov regularization and called spline (Wahba (1990)). In parametric context of linear regression, method called ridge regression, see Hoerl (1962). YES, Eurandom, 10 October 2011 p. 11/32

34 Tikhonov estimator Define Tikhonov estimator by its form in SVD domain : λ k = 1 1+γσ 2 k. In special case where A = I, estimator defined and computed as a modified version of Tikhonov regularization and called spline (Wahba (1990)). In parametric context of linear regression, method called ridge regression, see Hoerl (1962). Improve on least-squares estimator when singular values of design matrix are close to 0. YES, Eurandom, 10 October 2011 p. 11/32

35 Landweber iteration Standard method, based on idea to minimize Aϕ Y by steepest descent (i.e. Gradient descent algorithm). YES, Eurandom, 10 October 2011 p. 12/32

36 Landweber iteration Standard method, based on idea to minimize Aϕ Y by steepest descent (i.e. Gradient descent algorithm). Choose direction h equals to minus gradient (here approximate gradient). Thus, obtain h = A (Aϕ Y). YES, Eurandom, 10 October 2011 p. 12/32

37 Landweber iteration Standard method, based on idea to minimize Aϕ Y by steepest descent (i.e. Gradient descent algorithm). Choose direction h equals to minus gradient (here approximate gradient). Thus, obtain h = A (Aϕ Y). Recursion formula ϕ 0 = ˆf 0 = 0 and for some µ > 0, ˆf n = ˆf n 1 µa (Aˆf n 1 Y), YES, Eurandom, 10 October 2011 p. 12/32

38 Landweber iteration Standard method, based on idea to minimize Aϕ Y by steepest descent (i.e. Gradient descent algorithm). Choose direction h equals to minus gradient (here approximate gradient). Thus, obtain h = A (Aϕ Y). Recursion formula ϕ 0 = ˆf 0 = 0 and for some µ > 0, ˆf n = ˆf n 1 µa (Aˆf n 1 Y), Method is called Landweber iteration. YES, Eurandom, 10 October 2011 p. 12/32

39 Landweber iteration Standard method, based on idea to minimize Aϕ Y by steepest descent (i.e. Gradient descent algorithm). Choose direction h equals to minus gradient (here approximate gradient). Thus, obtain h = A (Aϕ Y). Recursion formula ϕ 0 = ˆf 0 = 0 and for some µ > 0, ˆf n = ˆf n 1 µa (Aˆf n 1 Y), Method is called Landweber iteration. By induction, n 1 ˆf n = j=0 (I µa A) j µa Y. YES, Eurandom, 10 October 2011 p. 12/32

40 Properties of Landweber Parameter µ chosen such that µ A A 1 with strong influence on convergence. YES, Eurandom, 10 October 2011 p. 13/32

41 Properties of Landweber Parameter µ chosen such that µ A A 1 with strong influence on convergence. Regularization parameter is the number of iterations n. YES, Eurandom, 10 October 2011 p. 13/32

42 Properties of Landweber Parameter µ chosen such that µ A A 1 with strong influence on convergence. Regularization parameter is the number of iterations n. Computational point of view, method faster than Tikhonov, since no need to invert an operator. YES, Eurandom, 10 October 2011 p. 13/32

43 Properties of Landweber Parameter µ chosen such that µ A A 1 with strong influence on convergence. Regularization parameter is the number of iterations n. Computational point of view, method faster than Tikhonov, since no need to invert an operator. Landweber iteration has some important drawbacks. YES, Eurandom, 10 October 2011 p. 13/32

44 Properties of Landweber Parameter µ chosen such that µ A A 1 with strong influence on convergence. Regularization parameter is the number of iterations n. Computational point of view, method faster than Tikhonov, since no need to invert an operator. Landweber iteration has some important drawbacks. the number of iterations may be large. YES, Eurandom, 10 October 2011 p. 13/32

45 Pinsker estimator Pinsker estimator (Pinsker (1980)), a special class of linear estimators defined by weights (Belitser and Levit (1995)) YES, Eurandom, 10 October 2011 p. 14/32

46 Pinsker estimator Pinsker estimator (Pinsker (1980)), a special class of linear estimators defined by weights (Belitser and Levit (1995)) λ k = (1 c ε a k ) +, where c ε is the solution of the equation ε 2 k=1 σ 2 k a k(1 c ε a k ) + = c ε L, with x + = max(0,x) and a k > 0. YES, Eurandom, 10 October 2011 p. 14/32

47 Pinsker estimator Pinsker estimator (Pinsker (1980)), a special class of linear estimators defined by weights (Belitser and Levit (1995)) λ k = (1 c ε a k ) +, where c ε is the solution of the equation ε 2 k=1 σ 2 k a k(1 c ε a k ) + = c ε L, with x + = max(0,x) and a k > 0. In context of estimation in ellipsoids, attains optimal rate of convergence, but also exact minimax constant. YES, Eurandom, 10 October 2011 p. 14/32

48 Classes of linear estimators Projection estimators (spectral cut-off), λ k = I(k N), N > 0. YES, Eurandom, 10 October 2011 p. 15/32

49 Classes of linear estimators Projection estimators (spectral cut-off), Tikhonov regularization, λ k = I(k N), N > 0. λ k = (1+γσ 2 k ) 1, γ > 0. YES, Eurandom, 10 October 2011 p. 15/32

50 Classes of linear estimators Projection estimators (spectral cut-off), Tikhonov regularization, Landweber iteration, λ k = I(k N), N > 0. λ k = (1+γσ 2 k ) 1, γ > 0. λ k = 1 (1 σ 2 k )n,n N. YES, Eurandom, 10 October 2011 p. 15/32

51 Classes of linear estimators Projection estimators (spectral cut-off), Tikhonov regularization, Landweber iteration, Pinsker filter. λ k = I(k N), N > 0. λ k = (1+γσ 2 k ) 1, γ > 0. λ k = 1 (1 σ 2 k )n,n N. λ k = (1 c ε a k ) +, c ε > 0. YES, Eurandom, 10 October 2011 p. 15/32

52 Classes of linear estimators Projection estimators (spectral cut-off), Tikhonov regularization, Landweber iteration, Pinsker filter. λ k = I(k N), N > 0. λ k = (1+γσ 2 k ) 1, γ > 0. λ k = 1 (1 σ 2 k )n,n N. λ k = (1 c ε a k ) +, c ε > 0. Choice of N,γ or n? YES, Eurandom, 10 October 2011 p. 15/32

53 Ellipsoid of coefficients Assume that f belongs to functional class corresponding to ellipsoids Θ in space of coefficients {θ k }: { Θ = Θ(a,L) = θ : k=1 a 2 k θ2 k L } where a = {a k }, where a k > 0,a k and L > 0., YES, Eurandom, 10 October 2011 p. 16/32

54 Ellipsoid of coefficients Assume that f belongs to functional class corresponding to ellipsoids Θ in space of coefficients {θ k }: { Θ = Θ(a,L) = θ : k=1 a 2 k θ2 k L } where a = {a k }, where a k > 0,a k and L > 0. For large values of k coefficients θ k will be decreasing with k and then be small., YES, Eurandom, 10 October 2011 p. 16/32

55 Ellipsoid of coefficients Assume that f belongs to functional class corresponding to ellipsoids Θ in space of coefficients {θ k }: { Θ = Θ(a,L) = θ : k=1 a 2 k θ2 k L } where a = {a k }, where a k > 0,a k and L > 0. For large values of k coefficients θ k will be decreasing with k and then be small. Assumptions on coefficients θ k usually related to properties (smoothness) on f., YES, Eurandom, 10 October 2011 p. 16/32

56 Sobolev classes Introduce the Sobolev classes { W(α,L) = f = k=1 θ k ϕ k : θ Θ(α,L) } YES, Eurandom, 10 October 2011 p. 17/32

57 Sobolev classes Introduce the Sobolev classes { W(α,L) = f = k=1 θ k ϕ k : θ Θ(α,L) } where Θ(α,L) = Θ(a,L) with a = {a k } polynomial such that a 1 = 0 and { (k 1) α for k odd, a k = k α for k even, where α > 0, L > 0. We have also k = 2,3,..., YES, Eurandom, 10 October 2011 p. 17/32

58 Sobolev classes Introduce the Sobolev classes { W(α,L) = f = k=1 θ k ϕ k : θ Θ(α,L) } where Θ(α,L) = Θ(a,L) with a = {a k } polynomial such that a 1 = 0 and { (k 1) α for k odd, a k = k α for k even, k = 2,3,..., where α > 0, L > 0. We have also { W(α,L) = f periodic : 1 0 (f (α) (t)) 2 dt π 2α L }. YES, Eurandom, 10 October 2011 p. 17/32

59 Analytic classes Consider also more restrictive conditions and classes of functions { } A(α,L) = f = k=1 θ k ϕ k : θ Θ A (α,l) YES, Eurandom, 10 October 2011 p. 18/32

60 Analytic classes Consider also more restrictive conditions and classes of functions { } A(α,L) = f = k=1 θ k ϕ k : θ Θ A (α,l) where a = {a k } exponential, α > 0, and L > 0, a k = exp(αk). YES, Eurandom, 10 October 2011 p. 18/32

61 Analytic classes Consider also more restrictive conditions and classes of functions { } A(α,L) = f = k=1 θ k ϕ k : θ Θ A (α,l) where a = {a k } exponential, α > 0, and L > 0, a k = exp(αk). Classes of analytical functions (functions which admit an analytical continuation into a band of the complex plane, Ibragimov and Khasminskii (1984)). YES, Eurandom, 10 October 2011 p. 18/32

62 Analytic classes Consider also more restrictive conditions and classes of functions { } A(α,L) = f = k=1 θ k ϕ k : θ Θ A (α,l) where a = {a k } exponential, α > 0, and L > 0, a k = exp(αk). Classes of analytical functions (functions which admit an analytical continuation into a band of the complex plane, Ibragimov and Khasminskii (1984)). These functions are then very smooth. YES, Eurandom, 10 October 2011 p. 18/32

63 Optimal rates of convergence Function f has Fourier coefficients in some ellipsoid, and the problem is mildly, severely ill-posed or even direct. Rates appear in the following table : YES, Eurandom, 10 October 2011 p. 19/32

64 Optimal rates of convergence Function f has Fourier coefficients in some ellipsoid, and the problem is mildly, severely ill-posed or even direct. Rates appear in the following table : Problem/Functions Sobolev Analytic Direct problem Mildly ill-posed ε 4α 2α+1 ε 2 (log 1 ε ) ε 4α 2α+2β+1 ε 2 (log 1 ε )2β+1 Severely ill-posed (log 1 ε ) 2α ε 4α 2α+2β YES, Eurandom, 10 October 2011 p. 19/32

65 Comments Rates usually depend on smoothness α of function f and on degree of ill-posedness β. YES, Eurandom, 10 October 2011 p. 20/32

66 Comments Rates usually depend on smoothness α of function f and on degree of ill-posedness β. When β increases rates are slower. YES, Eurandom, 10 October 2011 p. 20/32

67 Comments Rates usually depend on smoothness α of function f and on degree of ill-posedness β. When β increases rates are slower. In direct model, standard rates for nonparametric estimation. For example, 2α/(2α + 1) with Sobolev classes. YES, Eurandom, 10 October 2011 p. 20/32

68 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. YES, Eurandom, 10 October 2011 p. 21/32

69 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. In minimax sense, optimal choice for N. However, choice depends on smoothness α and on degree of ill-posedness β. YES, Eurandom, 10 October 2011 p. 21/32

70 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. In minimax sense, optimal choice for N. However, choice depends on smoothness α and on degree of ill-posedness β. Even if operator A (and its degree β) is known, no real meaning to consider smoothness of f as known. YES, Eurandom, 10 October 2011 p. 21/32

71 Comments To attain optimal rate with projection estimator, choose N corresponding to optimal trade-off between bias and variance. In minimax sense, optimal choice for N. However, choice depends on smoothness α and on degree of ill-posedness β. Even if operator A (and its degree β) is known, no real meaning to consider smoothness of f as known. Notion of adaptation and oracle inequalities, i.e. how to choose the tuning parameter (N, γ or n) without prior assumptions on f. YES, Eurandom, 10 October 2011 p. 21/32

72 Minimax adaptive procedures The starting point of the approach of minimax adaptation is a collection A = {Θ α } of classes Θ α l 2. YES, Eurandom, 10 October 2011 p. 22/32

73 Minimax adaptive procedures The starting point of the approach of minimax adaptation is a collection A = {Θ α } of classes Θ α l 2. The statistician knows that θ belongs to some member Θ α of the collection A, but does not know exactly which one. YES, Eurandom, 10 October 2011 p. 22/32

74 Minimax adaptive procedures The starting point of the approach of minimax adaptation is a collection A = {Θ α } of classes Θ α l 2. The statistician knows that θ belongs to some member Θ α of the collection A, but does not know exactly which one. If Θ α is a smoothness class, this assumption can be interpreted as follows : the underlying function has some smoothness, but one does not know the degree of smoothness. YES, Eurandom, 10 October 2011 p. 22/32

75 Minimax adaptive procedures An estimator θ is called minimax adaptive on the scale of classes A if for every Θ α A the estimator θ attains the optimal rate of convergence (Lepskii (1990)). YES, Eurandom, 10 October 2011 p. 23/32

76 Minimax adaptive procedures An estimator θ is called minimax adaptive on the scale of classes A if for every Θ α A the estimator θ attains the optimal rate of convergence (Lepskii (1990)). Estimator which adapts to the unknown smoothness of the function. YES, Eurandom, 10 October 2011 p. 23/32

77 Minimax adaptive procedures An estimator θ is called minimax adaptive on the scale of classes A if for every Θ α A the estimator θ attains the optimal rate of convergence (Lepskii (1990)). Estimator which adapts to the unknown smoothness of the function. In some cases, no estimator attains (exactly) the optimal rate on the whole scale. One has often to pay a price for adaptation (Lepskii (1992)). YES, Eurandom, 10 October 2011 p. 23/32

78 Minimax adaptive procedures An estimator θ is called minimax adaptive on the scale of classes A if for every Θ α A the estimator θ attains the optimal rate of convergence (Lepskii (1990)). Estimator which adapts to the unknown smoothness of the function. In some cases, no estimator attains (exactly) the optimal rate on the whole scale. One has often to pay a price for adaptation (Lepskii (1992)). Lepski procedure in inverse problems (Goldenshluger and Pereverzev (2000), Bauer and Hohage (2005) or Mathé (2006)). YES, Eurandom, 10 October 2011 p. 23/32

79 Comments Minimax adaptive estimators are really important in statistics from a theoretical and from a practical point of view. YES, Eurandom, 10 October 2011 p. 24/32

80 Comments Minimax adaptive estimators are really important in statistics from a theoretical and from a practical point of view. Estimator which automatically adapts to the unknown smoothness of the underlying function. YES, Eurandom, 10 October 2011 p. 24/32

81 Comments Minimax adaptive estimators are really important in statistics from a theoretical and from a practical point of view. Estimator which automatically adapts to the unknown smoothness of the underlying function. Indeed, it implies that these estimators are optimal for any possible parameter in the collection A. YES, Eurandom, 10 October 2011 p. 24/32

82 Comments Minimax adaptive estimators are really important in statistics from a theoretical and from a practical point of view. Estimator which automatically adapts to the unknown smoothness of the underlying function. Indeed, it implies that these estimators are optimal for any possible parameter in the collection A. Practical point of view, it garantees a good accuracy of the estimator for a very large choice of functions. YES, Eurandom, 10 October 2011 p. 24/32

83 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). YES, Eurandom, 10 October 2011 p. 25/32

84 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). Define the oracle λ 0 as R(θ,λ 0 ) = inf λ Λ R(θ,λ). The oracle corresponds to the best possible choice in Λ, i.e. the one which minimizes the risk. YES, Eurandom, 10 October 2011 p. 25/32

85 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). Define the oracle λ 0 as R(θ,λ 0 ) = inf λ Λ R(θ,λ). The oracle corresponds to the best possible choice in Λ, i.e. the one which minimizes the risk. However, this is not an estimator since the risk depends on the unknown θ, the oracle will depend also. YES, Eurandom, 10 October 2011 p. 25/32

86 Oracle Consider now a linked, but different point of view. Assume that a class of estimators is fixed, i.e. that the class of possible weights λ Λ is given (projection, Tikhonov,...). Define the oracle λ 0 as R(θ,λ 0 ) = inf λ Λ R(θ,λ). The oracle corresponds to the best possible choice in Λ, i.e. the one which minimizes the risk. However, this is not an estimator since the risk depends on the unknown θ, the oracle will depend also. An oracle is the best in the family, but it knows the true θ. YES, Eurandom, 10 October 2011 p. 25/32

87 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. YES, Eurandom, 10 October 2011 p. 26/32

88 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). YES, Eurandom, 10 October 2011 p. 26/32

89 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). This method goes back to Akaike Information Criteria (AIC) in Akaike (1973) and Mallows C p (1973). YES, Eurandom, 10 October 2011 p. 26/32

90 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). This method goes back to Akaike Information Criteria (AIC) in Akaike (1973) and Mallows C p (1973). Originally, the URE was in the context of regression estimation. Nowadays, it is a basic adaptation tool for many statistical models. YES, Eurandom, 10 October 2011 p. 26/32

91 Unbiased risk estimation A very natural idea in statistics is to estimate this unknown risk using the available data, and then to minimize this estimator of the risk. A classical approach to this minimization problem is based on the principle of unbiased risk estimation (URE) (Stein (1981)). This method goes back to Akaike Information Criteria (AIC) in Akaike (1973) and Mallows C p (1973). Originally, the URE was in the context of regression estimation. Nowadays, it is a basic adaptation tool for many statistical models. This idea appears also in all the cross-validation techniques. YES, Eurandom, 10 October 2011 p. 26/32

92 URE in inverse problems For inverse problems, this method was studied in C., Golubev, Picard and Tsybakov (2002), where exact oracle inequalities were obtained. YES, Eurandom, 10 October 2011 p. 27/32

93 URE in inverse problems For inverse problems, this method was studied in C., Golubev, Picard and Tsybakov (2002), where exact oracle inequalities were obtained. In this setting, the functional U(X,λ) = (1 λ k ) 2 (X 2 k ε2 σ 2 k )+ε2 σ 2 k λ2 k k=1 k=1 YES, Eurandom, 10 October 2011 p. 27/32

94 URE in inverse problems For inverse problems, this method was studied in C., Golubev, Picard and Tsybakov (2002), where exact oracle inequalities were obtained. In this setting, the functional U(X,λ) = (1 λ k ) 2 (X 2 k ε2 σ 2 k )+ε2 σ 2 k λ2 k k=1 k=1 is an unbiased estimator of R(θ,λ): R(θ,λ) = E θ U(X,λ), λ. YES, Eurandom, 10 October 2011 p. 27/32

95 Data-driven choice Unbiased risk estimation suggests to minimize over λ Λ the functional U(X,λ) in place of R(θ,λ). YES, Eurandom, 10 October 2011 p. 28/32

96 Data-driven choice Unbiased risk estimation suggests to minimize over λ Λ the functional U(X,λ) in place of R(θ,λ). This leads to the following data-driven choice of λ: λ = argmin λ Λ U(X,λ). YES, Eurandom, 10 October 2011 p. 28/32

97 Data-driven choice Unbiased risk estimation suggests to minimize over λ Λ the functional U(X,λ) in place of R(θ,λ). This leads to the following data-driven choice of λ: λ = argmin λ Λ U(X,λ). Define then the estimator θ by θ k = λ k X k. YES, Eurandom, 10 October 2011 p. 28/32

98 Assumptions Denote S = ( maxλ Λ k=1 σ4 k λ2 k min λ Λ k=1 σ4 k λ2 k ) 1/2. Let the following assumptions hold. YES, Eurandom, 10 October 2011 p. 29/32

99 Assumptions Denote S = ( maxλ Λ k=1 σ4 k λ2 k min λ Λ k=1 σ4 k λ2 k ) 1/2. Let the following assumptions hold. For any λ Λ 0 < k=1 σ 2 k λ2 k <, max sup λ k 1. λ Λ k YES, Eurandom, 10 October 2011 p. 29/32

100 Assumptions Denote S = ( maxλ Λ k=1 σ4 k λ2 k min λ Λ k=1 σ4 k λ2 k ) 1/2. Let the following assumptions hold. For any λ Λ 0 < k=1 σ 2 k λ2 k <, max sup λ k 1. λ Λ k There exists a constant C 1 > 0 such that, uniformly in λ Λ, σ 4 k λ2 k C 1 σ 4 k λ4 k. k=1 k=1 YES, Eurandom, 10 October 2011 p. 29/32

101 Oracle inequality for URE Theorem. Suppose σ k k β, β 0. Assume that Λ is finite with cardinality D and belongs to the family of Projection, Tikhonov or Pinsker estimators. There exist constants γ,c > 0 such that ε > 0, θ l 2, we have for B large enough, E θ θ θ 2 (1+γB 1 )min λ Λ R(θ,λ)+BC ε 2 (log(ds)) 2β+1. YES, Eurandom, 10 October 2011 p. 30/32

102 Oracle inequality for URE Theorem. Suppose σ k k β, β 0. Assume that Λ is finite with cardinality D and belongs to the family of Projection, Tikhonov or Pinsker estimators. There exist constants γ,c > 0 such that ε > 0, θ l 2, we have for B large enough, E θ θ θ 2 (1+γB 1 )min λ Λ R(θ,λ)+BC ε 2 (log(ds)) 2β+1. The data-driven choice by URE mimics the oracle. YES, Eurandom, 10 October 2011 p. 30/32

103 Comments Oracle inequalities prove that the estimator has a risk of the order of the oracle. YES, Eurandom, 10 October 2011 p. 31/32

104 Comments Oracle inequalities prove that the estimator has a risk of the order of the oracle. We are interested in data-driven methods, and then automatic, which more or less mimic the oracle. YES, Eurandom, 10 October 2011 p. 31/32

105 Comments Oracle inequalities prove that the estimator has a risk of the order of the oracle. We are interested in data-driven methods, and then automatic, which more or less mimic the oracle. The oracle approach is in some sense the opposite of the minimax approach. YES, Eurandom, 10 October 2011 p. 31/32

106 Comments Oracle inequalities prove that the estimator has a risk of the order of the oracle. We are interested in data-driven methods, and then automatic, which more or less mimic the oracle. The oracle approach is in some sense the opposite of the minimax approach. Fix a family of estimators and choose the best one among them. YES, Eurandom, 10 October 2011 p. 31/32

107 Comments Oracle inequalities prove that the estimator has a risk of the order of the oracle. We are interested in data-driven methods, and then automatic, which more or less mimic the oracle. The oracle approach is in some sense the opposite of the minimax approach. Fix a family of estimators and choose the best one among them. In the minimax approach, best accuracy for functions which belong to some function class. YES, Eurandom, 10 October 2011 p. 31/32

108 Comments The oracle approach is often used as a tool in order to obtain adaptive estimators. YES, Eurandom, 10 October 2011 p. 32/32

109 Comments The oracle approach is often used as a tool in order to obtain adaptive estimators. best estimator in a given class often attains the optimal rate of convergence. YES, Eurandom, 10 October 2011 p. 32/32

110 Comments The oracle approach is often used as a tool in order to obtain adaptive estimators. best estimator in a given class often attains the optimal rate of convergence. On the other hand, the minimax theory may be viewed as a justification for oracle inequality. YES, Eurandom, 10 October 2011 p. 32/32

111 Comments The oracle approach is often used as a tool in order to obtain adaptive estimators. best estimator in a given class often attains the optimal rate of convergence. On the other hand, the minimax theory may be viewed as a justification for oracle inequality. Indeed, one may ask if the given family of estimators is satisfying. YES, Eurandom, 10 October 2011 p. 32/32

112 Comments The oracle approach is often used as a tool in order to obtain adaptive estimators. best estimator in a given class often attains the optimal rate of convergence. On the other hand, the minimax theory may be viewed as a justification for oracle inequality. Indeed, one may ask if the given family of estimators is satisfying. Possible answer comes from minimax results, which prove that a given family gives optimal estimators. YES, Eurandom, 10 October 2011 p. 32/32