Statistical Inverse Problems and Instrumental Variables

Size: px

Start display at page:

Download "Statistical Inverse Problems and Instrumental Variables"

Anastasia Tucker
5 years ago
Views:

1 Statistical Inverse Problems and Instrumental Variables Thorsten Hohage Institut für Numerische und Angewandte Mathematik University of Göttingen Workshop on Inverse and Partial Information Problems: Methodology and Applications RICAM, Linz,

2 Collaborators Frank Bauer (Linz) Laurent Cavalier (Marseille) Jean-Pierre Florens (Toulouse) Jan Johannnes (Heidelberg) Enno Mammen (Mannheim) Axel Munk (Göttingen)

3 outline 1 A Newton method for nonlinear statistical inverse problems 2 Oracle inequalities 3 Nonparametric instrumental variables and perturbed operators

4 statistical inverse problem problem: Let X, Y be separable Hilbert spaces and F : D(F ) X Y a Fréchet differentiable, one-to-one operator. Estimate a given indirect observations in the form of a random process F 1 is not continuous! Y = F (a ) + σξ + δζ. ξ normalized stochastic noise: a Hilbert space satisfying Eξ = 0 and Cov ξ 1 σ 0 stochastic noise level ζ Y normalized deterministic noise, ζ = 1 δ 0 deterministic noise level

5 the algorithm The Newton equation F [â k ](â k+1 â k ) = Y F (â k ), k = 1, 2,... is regularized in each step by Tikhonov regularization with initial guess a 0 and regularization parameter α k = α 0 q k, q (0, 1): â k+1 := argmin a X F [â k ](a â k )+F (â k ) Y 2 Y +α k+1 a a 0 2 X

6 What is this for linear problems? If F = T is linear, the iteration formula simplifies to â k+1 := argmin a X Ta Y 2 Y + α k+1 a a 0 2 X. The iteration steps decouple in the sense that none of the previous iterate appears in the formula for â k+1. Bias and variance must be balanced by proper choice of the stopping index.

7 What if â k / D(F) for some k? Since typically D(F ) X and the stochastic noise σξ can be arbitrarily large, there exists a positive probability that â k / D(F ) in each Newton step. Emergency stop : If this happens, we stop the Newton iteration and return a 0 as estimator of a. We will have to show that the probability the such an emergency stop is necessary rapidly tends to 0 with the stochastic noise level σ.

8 Can we improve on the qualification of Tikhonov regularization? Replace Tikhonov regularization by iterated Tikhonov regularization: â (0) k+1 := a 0 â (j) k+1 := argmin a X â k+1 := â (m) k+1 closed formula: { F [â k ](a â k ) + F (â k ) Y 2 Y } j = 1,..., m +α k+1 a â (j 1) k+1 2 X ( â k+1 := a 0 + g αk+1 F [â k ] F [â k ] ) F [â k ] ( ) Y F(â k ) + F [â k ](â k a 0 ) r α (λ) := ( α α+λ) m, gα (λ) := 1 λ (1 r α(λ))

9 deterministic convergence analysis: references: B. Kaltenbacher, A. Neubauer, O. Scherzer. Iterative Regularization Methods for Nonlinear Ill-Posed Problems. Radon Series on Computational and Applied Mathematics, de Gruyter, Berlin, 2008 A. B. Bakushinsky and M. Y. Kokurin. Iterative Methods for Approximate Solution of Inverse Problems. Springer, Dordrecht, A. B. Bakushinsky. The problem of the convergence of the iteratively regularized Gauss-Newton method. Comput. Maths. Math. Phys., 32: , The following results are from: F. Bauer, T. Hohage and A. Munk. Iteratively Regularized Gauss-Newton Method for Nonlinear Inverse Problems with Random Noise. preprint, under revision for SIAM J. Numer. Anal.

10 error decomposition Let T := F [a ] and T k := F [â k ]. The error E k = â k a in the kth Newton step can be decomposed into an approximation error E app k+1 := r α k+1 (T T )E 0, a propagated data noise error and a nonlinearity error Ek+1 noi := g α k+1 (Tk T k)tk (δζ + σξ), E nl k+1 := g αk+1 (T k T k)t k (F (a ) F (â k ) + T k E k ) + ( r αk+1 (T k T k) r αk (T T ) ) E 0, i.e. E k+1 = E app k+1 + E noi k+1 + E nl k+1.

11 crucial lemma Lemma Under certain assumptions discussed below there exists γ nl > 0 such that E nl k γ nl ( app E k + Ek noi ) k = 1,..., K max.

12 assumptions of the lemma source condition: There exists a sufficiently small source w Y such that a 0 a = T w α 0 sufficiently large such that E 0 q m E app 1 Lipschitz condition: For all a 1, a 2 D(F ) F [a 1 ] F [a 2 ] L a 1 a 2. choice of K max : E noi k K max := max {k N : } C stop αk

13 on the proof of the lemma The proof uses an straightforward induction argument in k. The following properties of iterated Tikhonov regularization are used: There exists γ app > 0 such that for all k E app k+1 app E γ app E app k k+1 This rules out methods with infinite qualification such as Landweber iteration! The propagated data noise is an ordered process in the sense that E noi noi E for all k. k k+1

14 optimal deterministic rates Corollary For deterministic errors (σ = 0) define the optimal stopping index by ( K := min {K max, K }, K := argmin k N E app k + δ ). αk Then there exist constants C, δ 0 > 0 such that ( â K a C inf E app k N k + δ ) for all δ (0, δ 0 ]. αk In particular, under the Hölder source condition a 0 a = Λ(T T ) w with µ [ 1 2, m] we obtain ) â K a = O ( w 1 2µ+1 δ 2µ 2µ+1,

15 propagated data noise error We make the following assumptions on the variance term V (a, α) := g α (F [a] F [a])f [a] ξ 2 : There exists a known function ϕ noi such that (EV (a, α)) 1/2 ϕ noi (α) α (0, α 0 ] and a D(F ). There are constants 1 < γ noi γ noi < such that γ noi ϕ noi (α k+1 )/ϕ noi (α k ) γ noi, k N 0. (exponential inequality) λ 1, λ 2 > 0 a D(F ) α (0, α 0 ] τ 1 P {V (a, α) τev (a, α)} λ 1 e λ 2τ.

16 optimal rates for known smoothness Theorem Assume that {a : a a 0 2R} D(F) and define the optimal stopping index ( K := argmin k N E app k + δ ) + σϕ noi (α k ). αk If â k a 0 2R for k = 1,..., K, set K := K, otherwise K := 0. Then there exist constants C > 1 and δ 0, σ 0 > 0 such that ( E â K a 2) 1/2 C min k N for all δ (0, δ 0 ] and σ (0, σ 0 ]. ( E app k + δ ) + σϕ noi (α k ) αk Short: The Newton method achieves the same rate as iterated Tikhonov applied to the linearized problem.

17 outline 1 A Newton method for nonlinear statistical inverse problems 2 Oracle inequalities 3 Nonparametric instrumental variables and perturbed operators

18 oracle parameter choice rules Consider an inverse problem Y = F (a ) + σξ + δζ and a family {R α : Y X } of regularized inverses of F. An oracle parameter choice rule α or for the method {R α } and the solution a is defined by sup ζ 1 E R αor (Y ) a 2 = inf α sup ζ 1 E R α (Y ) a 2 An oracle inequality for some given parameter choice rule α = α (Y, σ, δ) is an estimate of the form sup E R α (Y ) a 2 χ(σ, δ) sup E R αor (Y ) a 2. ζ 1 ζ 1 In the optimal case χ(σ, δ) 1 as σ, δ 0. E. Candès. Modern statistical estimation via oracle inequalities. Acta Numerica, 15: , 2006.

19 typical convergence results in deterministic regularization theory In deterministic theory convergence results for parameter choice rules typically contain a comparison with all other reconstruction methods R : Y X In this case one cannot consider only one a X, otherwise the optimal method would be R(Y ) a. Hence, estimates must be uniform over a smoothness class S X, which is typically defined by a source condition. E.g. sup a S sup ζ 1 C inf R R α (F(a) + δζ) a sup sup R(F(a ) + δζ) a. a S ζ 1

20 T. T. Cai and M. G. Low. nonparametric estimation over shrinking neighborhoods: superefficiency and adaptation. Ann. Stat., 33: , oracle inequalities are more precise Proposition Let R α := (αi + T T ) 1 T (Tikhonov regularization) and A = {(T T ) µ w : w ρ} with ρ > 0 and µ (0, 1]. Then for all a A inf R sup a S sup ζ 1 R(T (a) + δζ) a sup δ>0 inf α>0 sup ζ 1 R α (Ta + δζ) a = In other words: For every element a in the smoothness class A there exists an error level δ > 0 for which the classical deterministic error bounds are suboptimal by an arbitrarily large factor! Deterministic analog to superefficiency.

21 balancing principle for nonlinear inverse problems Let â 0, â 1,..., â Kmax be estimators of a such that â k a Φ noi (k) + Φ app (k) + Φ nl (k), k K max. Φ app is unknown and non-increasing. Φ noi is known and non-decreasing. Φ nl is unknown and satisfies for some γ nl > 0 Φ nl (k) γ nl ( Φnoi (k) + Φ app (k) ), k = 0,..., K max.

22 oracle inequality Lepskiĭ balancing principle: { â k bal := min k K max : k â m 4(1 + γ nl )Φ noi (m), m = k + 1,..., K max } Theorem (Bauer, Hohage, Munk) Assume that Φ noi (k + 1) γ noi Φ noi (k) for some constant γ noi <. Then â kbal a ( 6(1 + γ nl )γ noi min Φapp (k) + Φ noi (k) ). k=1,...,k max extension of a result for the linear case γ nl = 0 by P. Mathé and S. Pereverzev. Regularization of some linear ill-posed problems with discretized random noisy data. Math. Comp., 75: , See also O. V. Lepskiĭ. On a problem of adaptive estimation in Gaussian white noise. Theory Probab. Appl., 35: , P. Mathé. The Lepskiĭ principle revisited. Inverse Problems, 22:L11 L15, 2006.

23 deterministic errors, unknown smoothness We return now to the Newton method for nonlinear inverse problems. Corollary Let u = F (a ) + δζ. Then â kbal a 6(1 + γ nl )γ noi inf k N ( Φ app (k) + δ ) αk

24 stochastic noise, unknown smoothness Corollary Let u = F (a ) + σξ + δζ. Furthermore let k bal be chosen by the Lepskiĭ balancing principle if â k B 2R (a 0 ) for k = 1,..., K max and k bal := 0 else. Then there exists a constant C > 0 such that for σ, δ small enough ( E â kbal a 2) ( 1/2 C min E app k N k + δ ) + (ln σ 1 )σϕ noi (α k ) αk

25 Can the logaritmic factor be avoided? In general, no! Counter-example: A. Tsybakov. On the best rate of adaptive estimation in some inverse problems. C. R. Acad. Sci. Paris, 300: , However, for linear compact operators with polynomially decaying singular values, yes! L. Cavalier, G. K. Golubev, D. Picard, and A. B. Tsybakov. Oracle inequalities for inverse problems. Ann. Stat., 30: , L. Cavalier and A. Tsybakov, Sharp adaption for inverse problems with random noise. Prob. Theor. Rel. Fields, 123: , L. Cavalier and G. K. Golubev. Risk hull method for inverse problems. Ann. Stat., to appear.

26 Unbiased Risk Estimation Let Y = Ta + ɛ and a α = T R α Y a linear estimator of a depending on α > 0. To estimate the risk R(α, a ) := E a α a 2 assume an independent copy of the noise ɛ is available and consider U(Y, α, ɛ) := T R α Y 2 2 R α (Y + ɛ), Y ɛ. Then U(Y, α, ɛ) is an unbiased estimator of the risk up to an additive constant since EU(Y, α, ɛ) = E T R α Y 2 2E R α (y + ɛ + ɛ), y + ɛ ɛ = E a α 2 2 R α y, y 2E R α ɛ, ɛ + 2E R α ɛ, ɛ = E a α 2 2 K R α y, a = E a α 2 2E K R α Y, a = R(α, a ) a 2.

27 A condition for bounding the variance of U To bound the variance of U the following condition is used in the analysis: ( tr g α (T T ) 2) C tr ((T T ) 2 g α (T T ) 4) This condition is satisfied for the truncated singular value decomposition, but violated for Tikhonov regularization, Landweber iteration and ν-methods.

28 A modified iterated Tikhonov regularization For given m = 2, 3,... compute an estimator by and â (0) α := margmin a X ( Ta Y 2 + α a 2) â (l) α := argmin a X ( Ta Y 2 + α a â (l 1) α 2), l = 1,..., m. Then for exact data Y = Ta a â (m) = r α (T T )a with r α (λ) := ( ) α j α + (j + 1)λ. α + λ α + λ The method satisfies the usual assumptions and has qualification m 1. Moreover, it satisfies the condition on the previous slide if the singular values of T decay polynomially.

29 outline 1 A Newton method for nonlinear statistical inverse problems 2 Oracle inequalities 3 Nonparametric instrumental variables and perturbed operators

30 introduction regression problem: Estimate a function a given n independent observations (X i, Z i ), i = 1,..., n of random variables X, Z satisfying Z = a(x) + ɛ where ɛ is an unobservable nuisance variable satisfying E(ɛ X) = 0. Often the assumption E(ɛ X) = 0 is violated. We will show that by solving an ill-posed inverse problem one can still estimate a if there exists another observable quantity W, which is sufficiently correlated with X and satisfies E(ɛ W ) = 0.

31 Estimating hourly wages as a function of the education level Z i : hourly wage of indiviual i X i : level of education of individual i unknown: a(x) := E(Z X) Here it seems unlikely that the wage X and the nuisance variable ɛ = Z a(x) are uncorrelated since there are other variables such intellegence and stamina which influence both X and Z. However, we may choose W e.g. as distance of the individuals appartment from college and reasonably assume that E(ɛ W ) = 0. P. Hall and J.L. Horowitz. Nonparametric methods for inference in the presence of instrumental variables. Ann. Stat., 33: , 2005.

32 a linear first kind integral equation From the observed data (X i, Z i, W i ) we can estimate the joint density f (x, y, w). We have E(X = x W = w)g(x)dx = E(Z W = w) for all w. x Setting k(w, x) := E(X = x W = w) and u(w) := E(Z W = w) we obtain the linear integral equation k(w, x)g(x) dx = u(w). Note that both the right hand side and the kernel are noisy since they have to be estimated from the data.

33 a nonlinear integral equation Often the assumption E(X W ) = 0 can be replaced by the stronger independence assumption ɛ, W independent, Eɛ = 0. The first assumption is equivalent to f (ɛ + a(x), x, w) dx = f W (ɛ + a(x), x)f Y,X (w) dx for all x, w where f W and F Y,X denote the marginal densities w.r.t. W and Y, X, respectively. This is a nonlinear integral equation with a noisy kernel, which can be solved by regularized Newton methods. joint work with J.P.Florens, J. Johannes and E. Mammen

34 related work also leading to a nonlinear integral equation: J.L. Horowitz and S. Lee. Nonparametric instrumental variables estimation of a quantile regression model. Econometrica, 75: , Proof of convergence is modelled after the following paper: N. Bissantz, T. Hohage, A. Munk Consistency and rates of Convergence of Nonlinear Tikhonov regularization with random noise. Inverse Problems, 20: , It uses a Hölder source condition which seems unnatural in this context since the estimated kernels of the integral operators are smooth.

Empirical Risk Minimization as Parameter Choice Rule for General Linear Regularization Methods

Empirical Risk Minimization as Parameter Choice Rule for General Linear Regularization Methods Frank Werner 1 Statistical Inverse Problems in Biophysics Group Max Planck Institute for Biophysical Chemistry,