# Linear convergence of iterative soft-thresholding

Save this PDF as:
Size: px
Start display at page:

## Transcription

1 arxiv: v3 [math.fa] 11 Dec 007 Linear convergence of iterative soft-thresholding Kristian Bredies and Dirk A. Lorenz ABSTRACT. In this article, the convergence of the often used iterative softthresholding algorithm for the solution of linear operator equations in infinite dimensional Hilbert spaces is analyzed in detail. We formulate the algorithm in the framework of generalized gradient methods and present a new convergence analysis. The analysis bases on new techniques like the Bregman-Taylor distance. As main result we show that the algorithm converges with linear rate as soon as the underlying operator satisfies the so-called finite basis injectivity property or the minimizer possesses a so-called strict sparsity pattern. This result quantifies the experience that the iterative soft-thresholding converges slowly because it is argued that the constant of the linear rate is close to one. Moreover it is shown that the constants can be calculated explicitly if some knowledge on the operator is available i.e. for compact operators). Furthermore, the techniques also can be used to establish linear convergence for related methods such as the iterative thresholding algorithm for joint sparsity and the accelerated gradient projection method. 1. Introduction This paper is concerned with the convergence analysis of numerical algorithms for the solution of linear inverse problems in the infinite-dimensional setting with so-called sparsity constraints. The background for this type of problem is, for example, the attempt to solve the linear operator equation Ku = f in an infinite-dimensional Hilbert space which models the connection between some quantity of interest u and some measurements f. Often, the measurements f contain noise which makes the direct inversion ill-posed and practically impossible. Thus, instead of considering the linear equation, a regularized problem is posed for which the solution is stable with respect to noise. A common approach is to regularize by minimizing a Tikhonov Math Subject Classifications. 65J, 46N10, 49M05. Keywords and Phrases. Iterative soft-thresholding, inverse problems, sparsity constraints, convergence analysis, generalized gradient projection method.

2 Kristian Bredies and Dirk A. Lorenz functional [9, 18, 31]. A special class of these regularizations has been of recent interest, namely of the type Ku f min + u l α k u k. 1.1) These problems model the fact that the quantity of interest u is composed of a few elements, i.e. it is sparse in some given, countable basis. To make this precise, let A : H 1 H be a bounded operator between two Hilbert spaces and let {ψ k } be an orthonormal basis of H 1. Denote with B : l H 1 the synthesis operator Bu k ) = k u kψ k. Then the problem can be rephrased as Au f min + u H 1 α k u, ψ k ABu f min + u l α k u k. Indeed, solutions of this type of problem admit only finitely many non-zero coefficients and often coincide with the sparsest solution possible [1, 13, 1, 3]. In this paper we consider operators of the form K = AB only, i.e. bounded linear operators applied after a synthesis operator for an orthonormal basis. Unfortunately, the numerical solution of the above non-smooth) minimization problem is not straightforward. There is a vast amount of literature dealing with efficient computational algorithms for equivalent formulations of the problem [10,15,17,19,4,5,9,36], both in the infinite-dimensional setting as well as for finitely many dimensions, but mostly for the finitedimensional case. An often-used, simple but apparently slow algorithm is the iterative soft-thresholding or thresholded Landweber) procedure which is known to converge in the strong sense in infinite dimensions [9] regardless of the special structure of the problem. The algorithm is simple: it just needs an initial value u 0 and an operator with K 1. The iteration just reads as u n+1 = S α u n K Ku n f) ), Sα w) ) k = sgnw k) [ ] w k α k +. In practice it is important to know moreover convergence rates for the algorithms or at least an estimate for the distance to a minimizer to evaluate the fidelity of the outcome of the computations. The convergence proofs in the infinite-dimensional case presented in [9], and for generalizations in [7], however, come without any rate of convergence. To the best knowledge of the

3 Linear convergence of iterative soft-thresholding 3 authors, [4] contains the first results about the convergence rate for iterative algorithms for linear inverse problems with sparsity constraints in infinite dimensions. There, an iterative hard-thresholding procedure has been proposed for which, if K is injective, a convergence rate of On 1/ ) could be established. The purpose of this paper is to develop a general framework for the convergence analysis of algorithms for the problem 1.1), especially for the iterative soft-thresholding algorithm. We show that the iterative soft-thresholding algorithm converges linearly in almost every case. To this end, we formulate the iterative soft-thresholding as a generalized gradient projection method which leads to a new proof for the strong convergence which is independent of the proof given in [9]. The techniques used for our approach give some new insight in the properties of the iterative soft-thresholding related methods. We distinguish two key properties which lead to linear convergence. The first is called finite basis injectivity FBI) and is a property of the operator K only. Definition 1. An operator K : l H mapping into a Hilbert space has the finite basis injectivity property, if for all finite subsets I N the operator K I is injective, i.e. for all u,v l with Ku = Kv and u k = v k = 0 for all k / I it follows u = v. The second definition is a condition on a solution of the minimization problem 1.1). Definition. A solution u of 1.1) possesses a strict sparsity pattern if whenever u k = 0 for some k there follows K Ku f) k < α k. The main result can be summarized by the following: Theorem 1. Let K : l H, K 0 be a linear and continuous operator as well as f H. Consider the sequence {u n } given by the iterative softthresholding procedure u n+1 = S snα u n s n K Ku n f) ), with step size Ssnαw) ) k = sgnw k) [ w k s n α k ] + 1.) 0 < s s n s < / K 1.3) and a u 0 l such that α k u 0 k <. Then, there is a minimizer u such that u n u in l. Moreover, suppose that either 1. K possesses the FBI property, or. u possesses a strict sparsity pattern.

4 4 Kristian Bredies and Dirk A. Lorenz Then, u n u with a linear rate, i.e. there exists a C > 0 and a 0 λ < 1 such that u n u Cλ n. We state a few remarks on this theorem and on the FBI property. Remark 1 Strong convergence). Assuming that the strong convergence of the iterative soft-thresholding 1.) is proved, linear convergence can be proven easily when the FBI property is fulfilled. Indeed, it is clear that a minimizer u of 1.1) has finite support. Moreover, due to the strong convergence, there is a finite set I suppu and an iteration index n 0 such that suppu n I for all n n 0 see also the proof of Theorem 3). From that index on, the iteration is applied to the indices k I only and hence, the iteration mapping is u n+1 I = S sα u n I sk I ) K I u n I f) ). Due to the FBI property this mapping is a contraction and linear convergence is proven. We are grateful to one anonymous referee for pointing out this fact. However, the above argumentation does not provide an estimate for the convergence since there is no easy way to estimate the index n 0. In this paper we do not build on the previous convergence results from [9] but develop an independent analysis which provides deeper insight in the properties of iterative soft-thresholding. Moreover, our approach provides an estimate for the convergence rate is case the operator K is compact, see Theorem 4 at the end of Section 4. Finally, it can easily be adapted to establish linear convergence of related algorithms, see Section 5. Remark Relation to the RIP). Let us note that the FBI property is of the same flavor as the so-called RIP restricted isometry property) which is an important ingredient in compressive sampling [1, 6, 30]. An operator K R p n has the RIP of order S if for some natural number S n there exists a constant δ S [0,1[ such that for every subset T {1,...,n} with T S and every c R T it holds 1 δ S ) c K T c 1 + δ S ) c. The RIP is not suited for our purpose, because in the infinite dimensional case even injective operators does not need to fulfill the RIP, even of order one, and hence, many ill-posed problems could not be handled. One the other hand, the FBI property does not hold for a simple operator K = AB where A is the identity and B is an overcomplete dictionary in a finite dimensional space while the RIP of some order may still be fulfilled in this case. Remark 3 Examples for operators with the FBI property). In the context of inverse problems with sparsity constraints, the FBI property is natural, since the operators A are often injective. Prominent examples

5 Linear convergence of iterative soft-thresholding 5 are the Radon transform [7], solution operators for partial differential equations, e.g. in heat conduction problems [8] or inverse boundary value problems like electrical impedance tomography [8]. The combination with a synthesis operator B for an orthonormal basis does not influence the injectivity. Moreover, the restriction to orthonormal bases can be relaxed. The results presented in this paper also hold if the system {ψ k } is a frame or even a dictionary as long as the FBI property is fulfilled. This is for example the case for a frame which consists of two orthonormal basis where no element of one basis can be written as a finite linear combination of elements of the other. This is typically the case, e.g. for a trigonometric basis and the Haar wavelet basis on a compact interval. One could speak of FBI frames or FBI dictionaries. Remark 4 Strict sparsity pattern). This condition can be interpreted as follows. We know that the weighted l 1 -regularization Φ imposes sparsity on a solution u in the sense that u k = 0 for all but finitely many k, hence the name sparsity constraint. For the remaining indices, the equations K Ku ) k = K f α k sgnu k ) are satisfied which corresponds to an approximate solution of the generally ill-posed equation Ku = f in a certain way. Now the condition that the solutions of 1.1) possess a strict sparsity pattern says that u k = 0 for some index k can occur only because of the sparsity constraint but never for the solution of the linear equation. We emphasize that Theorem 1 states that whenever {u n } converges to a solution u with strict sparsity pattern, then the speed of convergence has to linear. Besides the continuity, no restrictions on the linear operator K have to be made. The proof of Theorem 1 will be divided into three sections. First, in Section, we introduce a framework in which iterative soft-thresholding according to 1.) can be interpreted as a generalized gradient projection method. We derive descent properties for generalized gradient methods and show under which conditions we can obtain linear convergence in Section 3. It will turn out that splitting the functional distance to the minimizer into a Bregman and Taylor part, respectively, and estimating this Bregman-Taylor distance appropriately will ensure that the conditions are satisfied. We show in Section 4 that a Bregman-distance estimate for problems of the type 1.1) leads to a new convergence proof for the iterative soft-thresholding. In case K fulfills the FBI property, an estimate on the Bregman-Taylor distance leads, together with the general considerations on the generalized gradient projection methods, directly to the desired linear convergence speed. In the case where the limit u possesses a strict sparsity pattern, additional considerations regarding the iteration procedure which are based on the convergence of {u n } establish the claimed linear rate.

7 Linear convergence of iterative soft-thresholding 7 generalize the conditional gradient method, see [5]. With some modifications, a generalized conditional gradient method for 1.1) can be developed, yielding an iterative hard-shrinkage procedure [4] which converges with rate On 1/ ) in case K is injective. Here our aim is to develop a suitable analogon for the gradient projection method. We will motivate the generalization by taking a certain view at the problem. Recall that the first-order necessary conditions for the minimization problem.1) are as follows: If u Ω is a solution, then or equivalently, u sf u ) u, v u 0 for all v Ω, u optimal u = P Ω u sf u ) ) for each s > 0. So u has to be a fixed point of a certain class of mappings. The numerical algorithm then arises from performing the corresponding fixed-point iteration where the s may possibly vary in each iteration step and is often chosen such that it ensures a certain descent of the objective functional F. For example, if F is Lipschitz continuous with constant L, one requires that 0 < s s n s < L.3) be satisfied for all n. Note that in order to deal with the trivial case L = 0, we agree that /L =. The generalization to functionals of the type.) can be derived as follows. Assume that F is still differentiable and not necessarily convex but Φ is a general proper, convex and lower semi-continuous functional mapping H R { }. Now, the first-order necessary conditions can analogously be formulated as follows. If u is optimal, then, for s > 0, u sf u ) u, v u s Φv) Φu ) ) for all v H. Denoting by J s the proximity operator of sφ, i.e. the operator J s : w argmin v H v w + sφv),.4) the above is equivalent to u = J s u sf u ) ). This is again a fixed-point equation. Likewise, one can consider the corresponding fixed-point iteration, which we call the generalized gradient projection method. The motivation here is that, for suitable s, the operation u J s u sf u) ) yields a suitable functional descent for.). Since the part u sf u) corresponds to a gradient-descent step for the minimization of F and J s also takes Φ into account, it is suggested that such a property indeed holds.

8 8 Kristian Bredies and Dirk A. Lorenz Assuming that F is Lipschitz continuous with constant L, the above can be put in formal terms as follows. Algorithm Choose a u 0 H with Φu 0 ) < and set n = 0.. Compute the next iterate u n+1 according to u n+1 = J sn u n s n F u n ) ). where s n satisfies.3) and J s is defined in.4). 3. Set n := n + 1 and continue with Step. Remark 5 Forward-backward splitting and the relaxed method). The generalization of the gradient projection method leads to a special case of the so-called proximal forward-backward splitting method which amounts to the iteration u n+1 = u n + t n J sn u sn F u n ) + b n ) ) + a n u n) where t n [0,1] and {a n }, {b n } are absolutely summable sequences in H. In [7], it is shown that this method converges strongly to a minimizer under appropriate conditions. There exist, however, no statements about convergence rates so far. As already mentioned, the aim here is to establish convergence rates under certain conditions. Here, we restrict ourselves to the special case of the generalized gradient projection method. It can, however, be noted that methods extending the generalized gradient projection method can easily be derived in analogy. One example is the relaxed generalized gradient projection method which is iterating with u n+1 = u n + t n J sn u n s n F u n ) ) u n).5) where t n [0,1]. As we can see later, if 0 < t t n 1, then the same convergence rates apply to this method as well. Now, it is easy to see that the iterative soft-thresholding algorithm 1.) is a special case of this generalized gradient projection method in case the functionals F : l R and Φ : l ], ] are chosen according to { Ku f Fu) =, Φu) = α k u k, if the sum converges, else.6) where K : l H is linear and continuous between the Hilbert spaces l and H, f H and {α k } is sequence satisfying α k α > 0 for all k.

9 Linear convergence of iterative soft-thresholding 9 Here, F u) = K Ku f), so in each iteration step of Algorithm 1 we have to solve u n s n K Ku n f) v min v l + s n α k v k for which the solution is given by soft-thresholding, i.e. v = S snα u n s n K Ku n f) ), with S according to 1.), see [9], for example. Since the Lipschitz constant associated with F does not exceed K, this result can be summarized as follows: Proposition 1. Let K : l H be a bounded linear operator, f H and 0 < α < α k. Let F and Φ be chosen according to.6). Then Algorithm 1 with step-size {s n } according to 1.3) coincides with the iterative softthresholding procedure 1.). Here and in the following, we also agree to set / K = in 1.3) for the trivial case K = Convergence of the generalized gradient projection method In the following, conditions which ensure convergence of the generalized gradient projection method are derived. The key is the descent of the functional F + Φ in each iteration step. The following lemma states some basic properties of one iteration. Lemma 1. Let F be differentiable with F Lipschitz continuous with constant L and Φ be proper, convex and lower semi-continuous. Set v = J s u sf u) ) as in.4) for some s > 0 and denote by Then it holds: D s u) = Φu) Φv) + F u), u v 3.1) w H : Φw) Φv) + F u), w v v u D s u) s F + Φ)v) F + Φ)u) 1 sl Proof. Since v solves the problem min v H v u + sf u) + sφv) u v, w v. 3.) s 3.3) ) D s u). 3.4)

10 10 Kristian Bredies and Dirk A. Lorenz it immediately follows that the subgradient relation u sf u) v s Φv) is satisfied, see [16,3] for an introduction to convex analysis and subdifferential calculus. This can be rewritten to u sf u) v, w v s Φw) Φv) ) for all w H, while rearranging and dividing by s proves the inequality 3.). The inequality 3.3) follows by setting w = u in 3.). To show inequality 3.4), we observe F + Φ)v) F + Φ)u) + D s u) = Fv) Fu) + F u), u v = 1 0 F u + tv u) ) F u), v u dt. Using the Cauchy-Schwarz inequality and the Lipschitz continuity we obtain F + Φ)v) F + Φ)u) + D s u) 1 0 tl v u dt = L v u. Finally, applying the estimate 3.3) for D s and rearranging leads to 3.4). Remark 6 A weaker step-size condition). If the step-size in the generalized gradient projection method is chosen such that s n s < /L, then we can conclude from 3.4) that F + Φ)u n+1 ) F + Φ)u n ) δd sn u n ) 3.5) where δ = 1 sl. Of course, the constraint on the step size is only sufficient to guarantee such a decrease. A weaker condition is the following: 1 0 F u n + tu n+1 u n ) ) F u n ), u n+1 u n dt 1 δ)d sn u n ) 3.6) for some δ > 0. Regarding the proof of Lemma 1, it is easy to see that this condition also leads to the estimate 3.5). Unfortunately, 3.6) can only be verified a-posteriori, i.e. with the knowledge of the next iterate u n+1. So one has to guess an s n and check if 3.6) is satisfied, otherwise a different s n has to be chosen. In practice, this means that one iteration step is lost and consequently more computation time is needed, reducing the advantages of a more flexible step size. While the descent property 3.5) can be proven without convexity assumptions on F, we need such a property to estimate the distance of the functional values to the global minimum of F +Φ in the following. With the

11 Linear convergence of iterative soft-thresholding 11 convexity of F given, we are able to confine a condition which ensures linear convergence to the minimizer. To this end, we introduce for any sequence {u n } H according to Algorithm 1 the values ) r n = F + Φ)u n ) min F + Φ)u). 3.7) u H The convergence rate is determined by the speed the values of r n vanish as n goes to infinity. Proposition. Let F be convex and continuously differentiable with Lipschitz continuous derivative. Let {u n } be a sequence generated by Algorithm 1 such that the step-sizes are bounded from below, i.e. s n s > 0, and that we have F + Φ)u n+1 ) F + Φ)u n ) δd sn u n ) for a δ > 0 with D sn u n ) according to 3.1). 1. If F + Φ is coercive, then the values r n according to 3.7) satisfy r n 0 with rate On 1 ), i.e. there exists a C > 0 such that r n Cn 1.. If for a minimizer u and some c > 0 the values r n from 3.7) satisfy u n u cr n, 3.8) then {r n } vanishes exponentially and {u n } converges linearly to u, i.e. there exists a C > 0 and a λ [0,1[ such that u n u Cλ n. Proof. We first prove an estimate for r n and then treat the cases separately. For this purpose, pick an optimal u H and observe that the decrease in each iteration step can be estimated by r n r n+1 = F + Φ)u n ) F + Φ)u n+1 ) δd sn u n ), according to the assumptions. Note that D sn u n ) 0 by 3.3), so {r n } is non-increasing. Use the convexity of F to deduce r n Φu n ) Φu ) + F u n ), u n u = D sn u n ) + F u n ), u n+1 u + Φu n+1 ) Φu ) D sn u n ) + un u n+1, u n+1 u s n D sn u n ) + un+1 u sn Dsn u n )

12 1 Kristian Bredies and Dirk A. Lorenz by applying the Cauchy-Schwarz inequality as well as 3.) and 3.3). With the above estimate on r n r n+1 and 0 < s < s n we get δ u n+1 u δr n r n r n+1 ) + rn r n ) s We now turn to prove the first statement of the proposition. Assume that F + Φ is coercive, so from the fact that {r n } is non-increasing follows that u n u has to be bounded by a C 1 > 0. Furthermore, 0 r n r n+1 r 0 <, implying and consequently δr n r0 + δs 1 C 1 ) rn r n+1 qrn δ ) r n r n+1, q = r0 + > 0. δs 1 C 1 Standard arguments then give the rate r n = On 1 ), we repeat them here for convenience. The above estimate on r n r n+1 as well the property that {r n } is non-increasing yields which, summed up, leads to 1 1 = r n r n+1 r n+1 r n r n r n+1 r n q q r n r n n 1 1 = 1 nq rn 1 nq + r0 1 r n r 0 r i=0 i+1 r i and consequently, since q > 0, to the desired rate r n 1 nq + r 1 0 Cn 1. Regarding the second statement, assume that there is a c > 0 such that u n u cr n for some optimal u and each n. Starting again at 3.9) and applying Young s inequality yields, for each ε > 0, δr n r n r n+1 ) + δε un+1 u s + r n r n+1 ε. Choosing ε = sc 1 and exploiting the assumption u n+1 u cr n+1 as well as the fact r n+1 r n then imply δr n r n r n+1 ) + δ r n + r n r n+1 sc 1.

13 Linear convergence of iterative soft-thresholding 13 Rearranging this yields the estimate r n r n+1 δsc 1 sc r n which in turn establishes the exponential decay rate r n+1 1 δsc 1 ) sc 1 r n λ r n, with λ = 1 δsc 1 ) 1/ [0,1[. + 1 sc ) Using u n u cr n again finishes the proof: u n u cr n ) 1/ cr 0 ) 1/ λ n. Remark 7 On the choice s = 1/L). By combining Lemma 1 and Proposition, we get linear convergence of sequences generated by Algorithm 1 for any sequence of step-sizes {s n } satisfying.3) under the assumption 3.8). Condition.3) is in particular true for a constant step-size 0 < s < /L. Alternatively, the weaker step-size condition 3.6) and s n s > 0 could also be used and yields the same convergence rate. Also note that the constant λ from 3.10) depends on δ, s and c only and gets worse as δ and s become small as well as c becomes large. An appropriate step-size rule can help to make s and δ become larger and consequently yielding a smaller λ. For example, as in Remark 6 one gets δ = 1 sl/, so with the choice s = s = s n = L 1 the term δs is maximized and leads to δ = 1/ and λ = 1 4+cL) 1 1/. Of course, this is only with respect to the estimates made in Lemma 1, so in practice one may not necessarily observe better convergence. Also, better step-size rules still may lead to an improvement of the functional descent, see [10]. Remark 8 Descent for the relaxed method). If F is convex, one can employ the relaxed version of the generalized gradient projection method.5) and still obtains F + Φ)u n ) F + Φ)u n+1 ) t n F + Φ)u n ) F + Φ) J s u n s n F u n )) )) tδd sn u n ) if s n is chosen such that 3.5) is satisfied and 0 < t t n 1. Moreover, one can easily derive a slightly modified version of 3.3): u n u n+1 s n t nd sn u n ).

14 14 Kristian Bredies and Dirk A. Lorenz Thus, the statements of Proposition remain true for the relaxed generalized gradient projection method but possibly with different constants. Proposition tells us that we only have to establish 3.8) to obtain strong convergence with linear convergence rate. This can be done with determining how fast the functionals F and Φ vanish at some minimizer. This can be made precise by introducing the following notions which also turn out to be the essential ingredients to show 3.8): First, define for a minimizer u H the functional Rv) = F u ), v u + Φv) Φu ). 3.11) Note that if the subgradient of Φ in u is unique, R is the Bregman distance of Φ in u, a notion which is extensively used in the analysis of descent algorithms [3, 33]. Moreover, we make use of the remainder of the Taylor expansion of F, Tv) = Fv) Fu ) F u ), v u. 3.1) Remark 9 On the Bregman distance). In many cases the Bregmanlike distance R is enough to estimate the descent properties, see [4,33]. For example, in case that Φ is the p-th power of a norm of a -convex Banach space X, i.e. Φu) = u p X with p ]1,], which is moreover continuously embedded in H, one can show that v u X C 1 Rv) holds on each bounded set of X, see [37]. Consequently, with j p = 1 p p X denoting the duality mapping with gauge t t p 1, v u C v u X C 1 C v p X u p X p j pu ), v u ) = crv) observing that R is in this case the Bregman distance. Often, Tikhonov functionals for inverse problems admit such a structure, e.g. Ku f min + u l α u k p, a regularization which is also topic in [9]. As one can see in complete analogy to Proposition 1, the generalized gradient projection method also amounts to the iteration proposed there, so as a by-product and after verifying that the prerequisites of Proposition indeed hold, one immediately gets a linearly convergent method. However, in the case that Φ is not sufficiently convex, the Bregman distance alone is not sufficient to obtain the required estimate on the r n.

15 Linear convergence of iterative soft-thresholding 15 Φv) F + Φ)v) R Φu ) F u ), v u T Fv) + F + Φ)u ) FIGURE 1: Illustration of the Bregman-like distance R and the Taylor distance T for a convex Φ and a smooth F. Note that for the optimal value u it holds F u ) Φu ). This is the case for F and Φ according.6). In this situation we also have to take the Taylor distance T into account. Figure 1 shows an illustration of the values R and T. One could say that the Bregman distance measures the error corresponding to the Φ part while the Taylor distance does the same for the F part. The functionals R and T possess the following properties: Lemma. Consider the problem.) where F is convex, differentiable and Φ is proper, convex and lower semi-continuous. If u H is a solution of.) and v H is arbitrary, then the functionals R and T according to 3.11) and 3.1), respectively, are non-negative and satisfy Rv) + Tv) = F + Φ)v) F + Φ)u ). Proof. The identity is obvious from the definition of R and T. For the non-negativity of R, note that since u is a solution, it holds that F u ) Φu ). Hence, the subgradient inequality reads as Φu ) F u ), v u Φv) Rv) 0. Likewise, the property Tv) 0 is a consequence of the convexity of F. Now it follows immediately that Rv) = Tv) = 0 whenever v is a minimizer. To conclude this section, the main statement about the convergence of the generalized gradient projection method reads as: Theorem. Let F be a convex, differentiable functional with Lipschitzcontinuous derivative with associated Lipschitz constant L), Φ be proper,

16 16 Kristian Bredies and Dirk A. Lorenz convex and lower semi-continuous and {u n } be a sequence generated by Algorithm 1 with step-size according to.3). Moreover, suppose that u H is a solution of the minimization problem.). If, for each M R there exists a constant cm) > 0 such that v u cm) Rv) + Tv) ) 3.13) for each v satisfying F + Φ)v) M and Rv) and Tv) defined by 3.11) and 3.1), respectively, then {u n } converges linearly to the unique minimizer u. Proof. A step-size chosen according to.3) implies, by Lemma 1, the descent property 3.5) with δ = 1 sl/. In particular, from 3.5) follows that {r n } is non-increasing also remember 3.3) means in particular that D sn u n ) 0). Now choose M = F +Φ)u 0 ) < for which, by assumption, a c > 0 exists such that u n u c Ru n ) + Tu n ) ) = cr n. Hence, the prerequisites for Proposition are fulfilled and consequently, u n u with a linear rate. Finally, the minimizer has to be unique: If u is also a minimizer, then u plugged into 3.13) gives u u = 0 and consequently u = u. 4. Convergence rates for the iterative soft-thresholding method We now turn to the proof of the main result, Theorem 1, which collects the results of Sections and 3. Within this section, we consider the regularized inverse problem Ku f min + α k u k 4.1) u l where K : l H is a linear and continuous operator and f H as well as α k α > 0. It is known that at least one minimizer exists. We have already seen in Proposition 1 that splitting the above Tikhonov functional into F and Φ according to.6) yields the equivalence of the associated iterative soft-thresholding procedure and a generalized gradient projection method. Our aim is, on the one hand, to apply Proposition in order to get strong convergence from the descent rate On 1 ). On the other hand, we will show the applicability of Theorem for K possessing the FBI property which implies the desired convergence speed. Note that F defines a convex and differentiable functional with derivative F u) = K Ku f) which is Lipschitz continuous with estimate

17 Linear convergence of iterative soft-thresholding 17 L K on the associated constant. Moreover, Φ is proper, convex and lower semi-continuous. Thus, we only have to ensure appropriate step-sizes according to.3) which is immediate when 1.3) is satisfied) and to verify 3.13). The latter will be done, among a Bregman-distance estimate, in the following lemma, which is also serving as the crucial prerequisite for showing convergence. Lemma 3. For each minimizer u of 4.1) and each M R, there exists a c 1 M,u ) and a subspace U l with finite-dimensional complement such that for the Bregman-like distance 3.11) it holds that Rv) c 1 M,u ) P U v u ) 4.) whenever F + Φ)v) M with F and Φ defined by.6). If K moreover satisfies the FBI property, there is a c M,u,K) > 0 such that, whenever F + Φ)v) M, the associated Bregman-Taylor distance according to 3.11) and 3.1) satisfies Rv) + Tv) c M,u,K) v u. 4.3) Proof. Let u be a minimizer of 4.1) and assume that v l satisfies Φv) M for a M 0. Then, Rv) = α k v k α k u k + wk v k u k ) 4.4) where w = F u ) = K Ku f). Now since w Φu ) we have wk α k sgnu k ) for each k note that = sgn ) with sgn0) = [ 1,1]), meaning that α k vk u k ) + wk v k u k ) 0 for each k. Denote by I = {k 1 : w k = α k } which has to be finite since w l implies > k I w k = k I α k k I α = I α. Moreover, wk 0 as k, so there has to be a ρ < 1 such that w k /α k ρ for each k N\I. Also, if k N\I, then wk ρα k which means in particular that u k = 0 since the opposite contradicts w k α k sgnu k ). So, one can estimate 4.4): Rv) = α k v k α k u k + wk v k u k ) k/ I α k v k + w k v k k/ I α k 1 ρ) v k 1 ρ)α k/ I v k u k 1 ρ)α v k u k ) 1/ k/ I

18 18 Kristian Bredies and Dirk A. Lorenz using the fact that one can estimate the l -sequence norm with the l 1 - sequence norm, see [4] for example. With U = {v l : v k = 0 for k I}, the above also reads as Rv) 1 ρ)α P U v u ) with P U being the orthogonal projection onto U in l. Next, observe that α P U v α v 1 Φv) M + 1, hence we have P U v 1 α/m + 1). Consequently, Rv) 1 ρ)α M + 1 P Uv u ) = c 1 M,u ) P U v u ) which corresponds to the estimate 4.). Finally, Φv) M whenever F + Φ)v) M and there is no v such that F + Φ)v) < 0. Hence, for each M R there is a constant for which 4.) holds whenever F + Φ)v) M which is the desired statement for R. To prove 4.3), suppose K possesses the FBI property. Recall that Tv) can be expressed by Tv) = Fv) Fu ) F u ), v u = Kv f = Kv u ) Ku f K Ku f), v u. 4.5) The claim now is that there is a CM,u,K) such that u CM,u,K) c 1 M,u ) P U u + 1 Ku ) for each u l. We will derive this constant directly. First split u = P U u+p U u, so we can estimate, with the help of the inequalities of Cauchy- Schwarz and Young ab a /4 + b for a,b 0), Ku = KP U u KP U u 4 KP U u 4 + KP U u, KP U u + KP Uu KP Uu K P U u. Since K fulfills the FBI property, the operator restricted to U is injective on the finite-dimensional space U, so there exists a cu,k) > 0 such that cu,k) P U u KP U u for all u l. Hence, P U u 4 cu,k) 1 1 K P U u + 1 Ku )

19 Linear convergence of iterative soft-thresholding 19 and consequently u = P U u + P U u 4 cu,k) 1 1 K cu,k)) P U u + 1 Ku ) K + cu,k) + 4c 1 M,u ) c1 cu,k)c 1 M,u M,u ) P U u + 1 ) Ku ) giving a constant cm,u,k) > 0 since U depends on u. This finally yields the statement v u cm,u,k) c 1 M,u ) P U v u ) + 1 Kv u ) ) cm,u,k) Rv) + Tv) ), consequently, 4.3) holds for c M,u,K) = cm,u,k) 1. In the following, we will see that the estimate 4.) considered in Ru n ) already leads to strong convergence of the iterative soft-thresholding procedure. Nevertheless, we utilize 4.3) later, when the linear convergence result will be proven. Lemma 4. Let K : l H be a linear and continuous operator as well as f H. Consider the sequence {u n } which is generated by the iterative softthresholding procedure 1.) with step-sizes {s n } according to 1.3). Then, {u n } converges to a minimizer in the strong sense. Proof. Since the Lipschitz constant for F satisfies L K, the step sizes are fulfilling.3) which implies, by Lemma 1, the descent property 3.5) with δ = 1 s K /. This means in particular that the associated functional distances {r n } are non-increasing since 3.3) in particular gives that D sn u n ) 0). Moreover, the descent result in Proposition yields that the iterates {u n } satisfy Ru n ) r n On 1 ). Since F + Φ)u n ) r n r 0 = M, we can apply Lemma 3 and the estimate 4.) leads to strong convergence of {P U u n }, i.e. P U u n P U u. Next, consider the complement parts {P U u n } in the finite-dimensional space U. Since P U u n u n α 1 Φu n ) r 0, the sequence {P U u n } is contained in a relative compact set in U, hence there is a strong) accumulation point u U. Together with P U u n P U u we can conclude that there is a subsequence satisfying u n l P U u + u = u. Moreover, {u n } is a minimizing sequence, so u has to be a minimizer. Finally, the whole sequence has to converge to u : The mappings T n u) = J sn u sn F u) ) satisfy T n u) T n v) I s n K K)u v) u v for all u,v l, since all proximal mappings J sn are non-expansive and s n K. So if, for an arbitrary ε > 0 there exists a n such that u n u ε,

20 0 Kristian Bredies and Dirk A. Lorenz then u n+1 u = T n u n ) T n u ) u n u ε since u is minimizer and hence a fixed point of each T n see Section ). By induction, u n u strongly in l. From the strong convergence of the iterative soft-thresholding procedure, which is already proven in [9] with the approach of surrogate functionals and non-expansive mappings, there follows linear convergence in many cases. As already mentioned in Remark 1, it will be the case if K possesses the FBI property, which will be rigorously proven in the following. It is worth mentioning that the linear rate follows directly without using the strong convergence result of Lemma 4. With the notions of FBI property and strict sparsity pattern from Definitions 1 resp., one is able to show linear convergence as soon as one of this two situations is given. Theorem 3. Let K : l H be a linear and continuous operator as well as f H. Consider the sequence {u n } given by the iterative softthresholding procedure 1.) with step sizes {s n } according to 1.3). Then, there is a minimizer u such that u n u in l. Moreover, suppose that either 1. K possesses the FBI property, or. u possesses a strict sparsity pattern. Then, u n u with a linear rate, i.e. there exists a C > 0 and a 0 λ < 1 such that u n u Cλ n. For the proof, we refer to Appendix A. Corollary 1. In particular, choosing the step-size constant, i.e. s n = s with s ]0, K [ also leads to linear convergence under the prerequisites of Theorem 3. Remark 10 The weak step-size condition as accelerated method). As already mentioned in Remark 6, the condition on the step-size can be relaxed. In the particular setting that Fu) = 1 Ku f, the estimate 3.6) reads as 1 0 K K u n + tu n+1 u n ) ) K Ku n, u n+1 u n dt Now, the choice s n according to = Kun+1 u n ) 1 δ)d sn u n ) s n Ku n+1 u n ) 1 δ) u n+1 u n, 4.6)

21 Linear convergence of iterative soft-thresholding 1 is sufficient for the above, since one has the estimate 3.3). Together with the boundedness 0 < s s n, this is exactly the step-size Condition B) in [10]. Hence, as seen in Remark 7, the choice gives sufficient descent. Consequently, linear convergence remains valid for such an accelerated iterative soft-thresholding procedure if K possesses the FBI property, see Remark A.1. The constants in the estimates of Lemma 3 and Theorem 3 are in general not computable unless the solution is determined. Nonetheless there are some situations in which prior knowledge about the operator K can be used to estimate the decay rate. Theorem 4. Let K : l H, K 0 be a compact, linear operator fulfilling the FBI property and define { Ku } σ k = inf u : u 0, u l = 0 for all l k as well as { Ku µ k = sup u } : u 0, u l = 0 for all l < k. Furthermore, choose k 0 such that µ k0 α /4 f ) with allowed on the right-hand side). Let {u n } be a sequence generated by the iterative softthresholding algorithm 1.) with initial value u 0 = 0 and constant step-size s = K for the minimization of 1.1) and let u denote a minimizer. Then it holds that u n u Cλ n with λ = max for some C 0. σ k0 1 4σ k0 + 8 K,1 σ k0 α ) 1/ 4σ k0 α + σ k0 + K ) K f The proof is given Appendix B. 5. Convergence of related methods In this section, we show how linear convergence can be obtained for some related methods. In particular, iterative thresholding methods for minimization problems with joint sparsity constraints as well as an accelerated gradient projection method are considered. Both algorithms can be written as a generalized gradient projection method, hence the analysis carried out in Sections and 3 can be applied, demonstrating the broad range of applications.

22 Kristian Bredies and Dirk A. Lorenz 5.1 Joint sparsity constraints First, we consider the situation of so-called joint sparsity for vector-valued problems, see [,0,35]. The problems considered are set in the Hilbert space l ) N for some N 1 which is interpreted such that for u l ) N the k-th component u k is a vector in R N. Given a linear and continuous operator K : l ) N H, some data f H, a norm of R N and a sequence α k α > 0, the typical inverse problem with joint sparsity constraints reads as Ku f min + α k u k. 5.1) u l ) N In many applications, = q for some 1 q. To apply the generalized gradient projection method for 5.1), we split the functional into Fu) = Ku f, Φu) = α k u k. Analogously to Proposition 1, one needs to know the associated proximal mappings J s which can be reduced to the computation of the proximal mappings for on R N. These are known to be I + s ) 1 x) = I P { s})x) where P { s} denotes the projection to the closed s-ball associated with the dual norm. Again, as can be seen in analogy to Proposition 1, the generalized gradient projection method for 5.1) is essentially given by the iteration u n+1 = S snα u n s n K Ku n f) ), ) Ssnαw) k = I P { s nα k })w k ) 5.) where {s n } satisfies an appropriate step-size rule, e.g. according to 1.3) or 4.6). Let us examine this method with respect to convergence. First, fix a minimizer u which satisfies the optimality condition w = K Ku f) Φu ). As one knows from convex analysis, this can also be formulated pointwise, and Asplund s characterization of see [34], Proposition II.8.6) leads to wk α k if u k = 0 w k u k = α k if u k 0 where w k u k denotes the usual inner product of w k and u k in RN. Now, one can proceed in complete analogy to the proof of Lemma 3 in order to get an estimate of the associated Bregman distance: One constructs I = {k N : w k u k = α k} as well as the closed subspace U = {v l ) N :

23 Linear convergence of iterative soft-thresholding 3 v k = 0 if k / I} for which U is finite-dimensional. Furthermore, we have ρ = sup k/ I w k /α k < 1 and, by equivalence of norms in R N, one gets C 0,c 0 > 0 such that c 0 x x C 0 x with x = x x) for all x RN. Then, for a given M R, whenever F + Φ)v) M, Rv) 1 ρ)α c 0 1/ v k u k M + 1)C ) = c1 M,u ) P U v u ), 0 k/ I establishing an analogon of 4.). If K moreover satisfies the FBI property, then one also gets an analogon to 4.3), i.e. Rv) + Tv) c M,u,K) v u whenever F + Φ)v) M, by arguing analogously to Lemma 3. Since these two inequalities are the essential ingredient for proving convergence as well as the linear rate, compare Lemma 4 and Theorem 3, there holds: Theorem 5. The iterative soft-thresholding procedure 5.) for the minimization problem 5.1) converges to a minimizer in the strong sense in l ) N if the step-size rule 1.3) is satisfied. Furthermore, the convergence will be at linear rate if K possesses the FBI property and the step-size rule 4.6) as well as 0 < s < s n is satisfied. In particular, this is the case when s n s < / K. 5. Accelerated gradient projection methods An alternative approach to implement sparsity constraints for linear inverse problems is based on minimizing the discrepancy within a weighted l 1 -ball [10]. With the notation used in Section 4, the problem can be generally formulated as min u Ω Ku f, Ω = { u l : } α k u k ) For this classical situation of constrained minimization, one finds that the generalized gradient projection method and the gradient projection method coincide for Fu) = 1 Ku f and Φ = I Ω ), see Section, and yield the iteration proposed in [10]. Consequently, classical convergence results hold for a variety of step-size rules [14], including the Condition B) introduced in [10], see also Remark 10. Let us note that linear convergence results can be obtained with the same techniques which have been used to prove Theorem 3: First, consider the Bregman distance R associated with Φ = I Ω in a minimizer u Ω. With w = K Ku f), the optimality condition reads as w, v u 0 for all v Ω α 1 w = w, u

24 4 Kristian Bredies and Dirk A. Lorenz where α 1 w ) k = α 1 k w k which is in l since α k α > 0. Introduce I = {k : α 1 k w k = α 1 w } which has to be finite since otherwise w / l, see the proof of Lemma 3. Suppose that w 0 which corresponds to Ku f), so sup k/ I α 1 k w k / α 1 w = ρ < 1. Moreover, α k u k = 1 and sgnu k ) = sgnw k ) for all k with u k 0, 5.4) k I since k I α k u k < 1 leads to the contradiction α 1 w = w, u = α 1 k w k α ku k α 1 k w k α ku k + α 1 w α k u k k/ I k I ρ α k u k + α k u k ) α 1 w < α 1 w k/ I k I while sgnu k ) sgnw k ) for some k with u k 0 implies the contradiction α 1 w = α 1 k w k α ku k < α 1 k w k α ku k α 1 w. Moreover, k I α k u k = 1 also yields u k = 0 for all k / I. Furthermore, observe that the equation for the signs in 5.4) gives k I w k u k = α 1 w. For v / Ω we have Rv) =, so estimate the Bregman distance for v Ω as follows: Rv) = w, v u = α 1 k w k α ku k v k) α 1 k w k α kv k k I k/ I α 1 w α 1 w α k v k ρ α 1 w α k v k k I 1 ρ) k/ I α k v k 1 ρ)α P U v 1 k/ I where U = {u l : u k = 0 for k I}. Using that v v 1 as well as α v α v 1 1 for all v Ω finally gives, together with P U u = 0, Rv) 1 ρ)α P U v u ) for all v l. If K possesses the FBI property, one can, analogously to the argumentation presented in the proof of Lemma 3, estimate the Bregman-Taylor distance such that, for some cu,k) > 0, Rv) + Tv) cu,k) v u for all v l.

25 Linear convergence of iterative soft-thresholding 5 By Theorem, the gradient projection method for 5.3) converges linearly. This remains true for each accelerated step-size choice according to Condition B) in [10], see Remark 10. This result can be summarized in the following theorem. Theorem 6. Assume that K : l H satisfies the FBI property, α k α > 0 and f H \ KΩ) where KΩ) = {Ku : αu 1 1}. Then, the gradient projection method for the minimization problem 5.3) converges linearly, whenever the step-sizes {s n } satisfy 4.6) for some δ > 0 as well as 0 < s < s n. This is in particular the case for s n s < / K. 6. Conclusions We conclude this article with a few remarks on the implications of our results. We showed that, in many cases, iterative soft-thresholding algorithms converge with linear rate and moreover that there are situations in which the constants can be calculated explicitly. Hence, this estimates may serve as a stopping criterion. The convergence proof base on descent arguments for a generalized gradient projection method as well as Bregman and Bregman-Taylor distance estimates. It is worth noting that the iterative thresholding converges linearly although the iteration itself is not a contraction its Lipschitz constant is one). Also note that we did not show a property like u n+1 u n C u n u n 1 for some C < 1. Moreover, the factor λ, which determines the speed within the class of linearly-convergent algorithms, always depends on the operator K but in the considered cases also on the initial value u 0 and a solution u. Unfortunately, the dependence on a solution can cause λ to be arbitrarily close to 1, meaning that the iterative soft-thresholding converges arbitrarily slow in some sense. This is often observed in practice, especially in the context of ill-posed inverse problems, where one has to compute a huge amount of iterations before reaching a reasonable accuracy. Explicit expressions for the constants are derived only for the special case of compact operators as in Theorem 4. A key ingredient for proving the convergence result is the FBI property. It is interesting to note that this property plays a central role for the convergence analysis since it is of similar flavor as the RIP. The RIP is central in the analysis of the properties of the minimizers and now it is seen that it is also of importance for the performance of algorithms. A similar observation is made in the analysis of Newton methods applied to minimization problems with sparsity constraints [4]. As we have moreover seen, in case the operator does not fulfill the FBI property, linear convergence can still be obtained whenever the iterative thresholding procedure converges to a solution with strict sparsity pattern. This result is closely connected with

26 6 Kristian Bredies and Dirk A. Lorenz the fact that 1.1), considered on a fixed sign pattern, is a quadratic problem, and hence the iteration becomes linear from some index on. The latter observation is also basis of a couple of different algorithms [15,19,9]. At last we want to remark that Theorem on linear convergence of the generalized gradient projection method holds in general and hence the conclusion of Theorem 3, which is an application of this result to sparsity constraints, may be generalized to other problems with similar structure as well as similar algorithms, see Section 5. Especially the techniques used for the proof of Theorem 3 are applicable to other penalty terms as the one considered here. In particular, convergence holds for a class of penalty functionals for which an appropriate estimate for the Bregman distance is available, also see Remark 9 which discusses this for norms of a -convex Banach space. If such an estimate does not hold, the Bregman-Taylor distance can be employed, as it has been done in the case of sparsity constraints. Finally, the framework of the generalized gradient projection method is very general and also allows for modifications such as taking other step-sizes, see the Remarks 6 and 10 on choosing accelerating step-sizes, or introducing the relaxed version of the generalized gradient projection method, see Remarks 5 and 8, with the statements remaining valid by only minor adaptation of the proofs. A. Proof of Theorem 3 Proof of Theorem 3. Observe that the prerequisites of Lemma 4 are fulfilled, so there exists a minimizer u such that u n u in l. Thus, we have to show that each of the two cases stated in the theorem leads to a linear convergence rate. Consider the first case, i.e. K possesses the FBI property. We utilize that, by Lemma 3, the Bregman-Taylor distance according to 3.11) and 3.1) can be estimated such that 3.13) is satisfied for some c > 0. This implies, by Theorem, the linear convergence rate. For the second case, let u possesses a strict sparsity pattern. Define, analogously to the above, the subspace U = {v l : v k = 0 if u k 0}. The desired result then is implied by the fact that there is an n 0 such that each u n+1 with n n 0 can be written as u n+1 = I s n P U K KP U )u n u ) + u. For this purpose, we introduce the notations w n = K Ku n f), w = K Ku f) and recall the optimality condition w Φu ) which can

### Linear convergence of iterative soft-thresholding

Linear convergence of iterative soft-thresholding arxiv:0709.1598v4 [math.fa] 9 Apr 008 Kristian Bredies and Dirk A. Lorenz ABSTRACT. In this article a unified approach to iterative soft-thresholding algorithms

### SPECTRAL PROPERTIES OF THE LAPLACIAN ON BOUNDED DOMAINS

SPECTRAL PROPERTIES OF THE LAPLACIAN ON BOUNDED DOMAINS TSOGTGEREL GANTUMUR Abstract. After establishing discrete spectra for a large class of elliptic operators, we present some fundamental spectral properties

### Iterated hard shrinkage for minimization problems with sparsity constraints

Iterated hard shrinkage for minimization problems with sparsity constraints Kristian Bredies, Dirk A. Lorenz June 6, 6 Abstract A new iterative algorithm for the solution of minimization problems which

### Optimization and Optimal Control in Banach Spaces

Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex non-smooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,

### Functional Analysis. Franck Sueur Metric spaces Definitions Completeness Compactness Separability...

Functional Analysis Franck Sueur 2018-2019 Contents 1 Metric spaces 1 1.1 Definitions........................................ 1 1.2 Completeness...................................... 3 1.3 Compactness......................................

### Contents: 1. Minimization. 2. The theorem of Lions-Stampacchia for variational inequalities. 3. Γ -Convergence. 4. Duality mapping.

Minimization Contents: 1. Minimization. 2. The theorem of Lions-Stampacchia for variational inequalities. 3. Γ -Convergence. 4. Duality mapping. 1 Minimization A Topological Result. Let S be a topological

### On Riesz-Fischer sequences and lower frame bounds

On Riesz-Fischer sequences and lower frame bounds P. Casazza, O. Christensen, S. Li, A. Lindner Abstract We investigate the consequences of the lower frame condition and the lower Riesz basis condition

### ADJOINTS, ABSOLUTE VALUES AND POLAR DECOMPOSITIONS

J. OPERATOR THEORY 44(2000), 243 254 c Copyright by Theta, 2000 ADJOINTS, ABSOLUTE VALUES AND POLAR DECOMPOSITIONS DOUGLAS BRIDGES, FRED RICHMAN and PETER SCHUSTER Communicated by William B. Arveson Abstract.

### Convex Optimization Notes

Convex Optimization Notes Jonathan Siegel January 2017 1 Convex Analysis This section is devoted to the study of convex functions f : B R {+ } and convex sets U B, for B a Banach space. The case of B =

### min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;

Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many

### A brief introduction to trace class operators

A brief introduction to trace class operators Christopher Hawthorne December 2015 Contents 1 Introduction 1 2 Preliminaries 1 3 Trace class operators 2 4 Duals 8 1 Introduction The trace is a useful number

### Finite-dimensional spaces. C n is the space of n-tuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product

Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a pre-hilbert space, or a unitary space) if there is a mapping (, )

### Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings

Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be

### Convergence rates in l 1 -regularization when the basis is not smooth enough

Convergence rates in l 1 -regularization when the basis is not smooth enough Jens Flemming, Markus Hegland November 29, 2013 Abstract Sparsity promoting regularization is an important technique for signal

### Banach Spaces V: A Closer Look at the w- and the w -Topologies

BS V c Gabriel Nagy Banach Spaces V: A Closer Look at the w- and the w -Topologies Notes from the Functional Analysis Course (Fall 07 - Spring 08) In this section we discuss two important, but highly non-trivial,

### Your first day at work MATH 806 (Fall 2015)

Your first day at work MATH 806 (Fall 2015) 1. Let X be a set (with no particular algebraic structure). A function d : X X R is called a metric on X (and then X is called a metric space) when d satisfies

### Regularization of linear inverse problems with total generalized variation

Regularization of linear inverse problems with total generalized variation Kristian Bredies Martin Holler January 27, 2014 Abstract The regularization properties of the total generalized variation (TGV)

### c 2011 International Press Vol. 18, No. 1, pp , March DENNIS TREDE

METHODS AND APPLICATIONS OF ANALYSIS. c 2011 International Press Vol. 18, No. 1, pp. 105 110, March 2011 007 EXACT SUPPORT RECOVERY FOR LINEAR INVERSE PROBLEMS WITH SPARSITY CONSTRAINTS DENNIS TREDE Abstract.

### Optimization methods

Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on first-order information,

### Introduction to Real Analysis Alternative Chapter 1

Christopher Heil Introduction to Real Analysis Alternative Chapter 1 A Primer on Norms and Banach Spaces Last Updated: March 10, 2018 c 2018 by Christopher Heil Chapter 1 A Primer on Norms and Banach Spaces

### Morozov s discrepancy principle for Tikhonov-type functionals with non-linear operators

Morozov s discrepancy principle for Tikhonov-type functionals with non-linear operators Stephan W Anzengruber 1 and Ronny Ramlau 1,2 1 Johann Radon Institute for Computational and Applied Mathematics,

### INDUSTRIAL MATHEMATICS INSTITUTE. B.S. Kashin and V.N. Temlyakov. IMI Preprint Series. Department of Mathematics University of South Carolina

INDUSTRIAL MATHEMATICS INSTITUTE 2007:08 A remark on compressed sensing B.S. Kashin and V.N. Temlyakov IMI Preprint Series Department of Mathematics University of South Carolina A remark on compressed

### 08a. Operators on Hilbert spaces. 1. Boundedness, continuity, operator norms

(February 24, 2017) 08a. Operators on Hilbert spaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ [This document is http://www.math.umn.edu/ garrett/m/real/notes 2016-17/08a-ops

### Some SDEs with distributional drift Part I : General calculus. Flandoli, Franco; Russo, Francesco; Wolf, Jochen

Title Author(s) Some SDEs with distributional drift Part I : General calculus Flandoli, Franco; Russo, Francesco; Wolf, Jochen Citation Osaka Journal of Mathematics. 4() P.493-P.54 Issue Date 3-6 Text

### Robust error estimates for regularization and discretization of bang-bang control problems

Robust error estimates for regularization and discretization of bang-bang control problems Daniel Wachsmuth September 2, 205 Abstract We investigate the simultaneous regularization and discretization of

### Proximal-like contraction methods for monotone variational inequalities in a unified framework

Proximal-like contraction methods for monotone variational inequalities in a unified framework Bingsheng He 1 Li-Zhi Liao 2 Xiang Wang Department of Mathematics, Nanjing University, Nanjing, 210093, China

### 4 Hilbert spaces. The proof of the Hilbert basis theorem is not mathematics, it is theology. Camille Jordan

The proof of the Hilbert basis theorem is not mathematics, it is theology. Camille Jordan Wir müssen wissen, wir werden wissen. David Hilbert We now continue to study a special class of Banach spaces,

### Proximal methods. S. Villa. October 7, 2014

Proximal methods S. Villa October 7, 2014 1 Review of the basics Often machine learning problems require the solution of minimization problems. For instance, the ERM algorithm requires to solve a problem

### A NEW ITERATIVE METHOD FOR THE SPLIT COMMON FIXED POINT PROBLEM IN HILBERT SPACES. Fenghui Wang

A NEW ITERATIVE METHOD FOR THE SPLIT COMMON FIXED POINT PROBLEM IN HILBERT SPACES Fenghui Wang Department of Mathematics, Luoyang Normal University, Luoyang 470, P.R. China E-mail: wfenghui@63.com ABSTRACT.

### Lecture 5 : Projections

Lecture 5 : Projections EE227C. Lecturer: Professor Martin Wainwright. Scribe: Alvin Wan Up until now, we have seen convergence rates of unconstrained gradient descent. Now, we consider a constrained minimization

### Statistics 612: L p spaces, metrics on spaces of probabilites, and connections to estimation

Statistics 62: L p spaces, metrics on spaces of probabilites, and connections to estimation Moulinath Banerjee December 6, 2006 L p spaces and Hilbert spaces We first formally define L p spaces. Consider

### LECTURE 3 Functional spaces on manifolds

LECTURE 3 Functional spaces on manifolds The aim of this section is to introduce Sobolev spaces on manifolds (or on vector bundles over manifolds). These will be the Banach spaces of sections we were after

### Chapter 3: Baire category and open mapping theorems

MA3421 2016 17 Chapter 3: Baire category and open mapping theorems A number of the major results rely on completeness via the Baire category theorem. 3.1 The Baire category theorem 3.1.1 Definition. A

### ECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference

ECE 18-898G: Special Topics in Signal Processing: Sparsity, Structure, and Inference Sparse Recovery using L1 minimization - algorithms Yuejie Chi Department of Electrical and Computer Engineering Spring

### Unconstrained optimization

Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout

### STAT 200C: High-dimensional Statistics

STAT 200C: High-dimensional Statistics Arash A. Amini May 30, 2018 1 / 57 Table of Contents 1 Sparse linear models Basis Pursuit and restricted null space property Sufficient conditions for RNS 2 / 57

### Your first day at work MATH 806 (Fall 2015)

Your first day at work MATH 806 (Fall 2015) 1. Let X be a set (with no particular algebraic structure). A function d : X X R is called a metric on X (and then X is called a metric space) when d satisfies

### Analysis Finite and Infinite Sets The Real Numbers The Cantor Set

Analysis Finite and Infinite Sets Definition. An initial segment is {n N n n 0 }. Definition. A finite set can be put into one-to-one correspondence with an initial segment. The empty set is also considered

### A SYNOPSIS OF HILBERT SPACE THEORY

A SYNOPSIS OF HILBERT SPACE THEORY Below is a summary of Hilbert space theory that you find in more detail in any book on functional analysis, like the one Akhiezer and Glazman, the one by Kreiszig or

### Dual methods for the minimization of the total variation

1 / 30 Dual methods for the minimization of the total variation Rémy Abergel supervisor Lionel Moisan MAP5 - CNRS UMR 8145 Different Learning Seminar, LTCI Thursday 21st April 2016 2 / 30 Plan 1 Introduction

### Necessary conditions for convergence rates of regularizations of optimal control problems

Necessary conditions for convergence rates of regularizations of optimal control problems Daniel Wachsmuth and Gerd Wachsmuth Johann Radon Institute for Computational and Applied Mathematics RICAM), Austrian

### Metric Spaces and Topology

Chapter 2 Metric Spaces and Topology From an engineering perspective, the most important way to construct a topology on a set is to define the topology in terms of a metric on the set. This approach underlies

### A Short Course on Frame Theory

A Short Course on Frame Theory Veniamin I. Morgenshtern and Helmut Bölcskei ETH Zurich, 8092 Zurich, Switzerland E-mail: {vmorgens, boelcskei}@nari.ee.ethz.ch April 2, 20 Hilbert spaces [, Def. 3.-] and

### An introduction to some aspects of functional analysis

An introduction to some aspects of functional analysis Stephen Semmes Rice University Abstract These informal notes deal with some very basic objects in functional analysis, including norms and seminorms

### Eigenvalues and Eigenfunctions of the Laplacian

The Waterloo Mathematics Review 23 Eigenvalues and Eigenfunctions of the Laplacian Mihai Nica University of Waterloo mcnica@uwaterloo.ca Abstract: The problem of determining the eigenvalues and eigenvectors

### Iterative regularization of nonlinear ill-posed problems in Banach space

Iterative regularization of nonlinear ill-posed problems in Banach space Barbara Kaltenbacher, University of Klagenfurt joint work with Bernd Hofmann, Technical University of Chemnitz, Frank Schöpfer and

### Best approximations in normed vector spaces

Best approximations in normed vector spaces Mike de Vries 5699703 a thesis submitted to the Department of Mathematics at Utrecht University in partial fulfillment of the requirements for the degree of

### Exercises to Applied Functional Analysis

Exercises to Applied Functional Analysis Exercises to Lecture 1 Here are some exercises about metric spaces. Some of the solutions can be found in my own additional lecture notes on Blackboard, as the

### A DECOMPOSITION THEOREM FOR FRAMES AND THE FEICHTINGER CONJECTURE

PROCEEDINGS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 00, Number 0, Pages 000 000 S 0002-9939(XX)0000-0 A DECOMPOSITION THEOREM FOR FRAMES AND THE FEICHTINGER CONJECTURE PETER G. CASAZZA, GITTA KUTYNIOK,

### Iowa State University. Instructor: Alex Roitershtein Summer Exam #1. Solutions. x u = 2 x v

Math 501 Iowa State University Introduction to Real Analysis Department of Mathematics Instructor: Alex Roitershtein Summer 015 Exam #1 Solutions This is a take-home examination. The exam includes 8 questions.

### Optimality Conditions for Constrained Optimization

72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

Chapter 8 Gradient Methods An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Introduction Recall that a level set of a function is the set of points satisfying for some constant. Thus, a point

### Master 2 MathBigData. 3 novembre CMAP - Ecole Polytechnique

Master 2 MathBigData S. Gaïffas 1 3 novembre 2014 1 CMAP - Ecole Polytechnique 1 Supervised learning recap Introduction Loss functions, linearity 2 Penalization Introduction Ridge Sparsity Lasso 3 Some

### Some Background Material

Chapter 1 Some Background Material In the first chapter, we present a quick review of elementary - but important - material as a way of dipping our toes in the water. This chapter also introduces important

### Elements of Convex Optimization Theory

Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a self-contained overview of elements of convex optimization

### Math Tune-Up Louisiana State University August, Lectures on Partial Differential Equations and Hilbert Space

Math Tune-Up Louisiana State University August, 2008 Lectures on Partial Differential Equations and Hilbert Space 1. A linear partial differential equation of physics We begin by considering the simplest

### Maths 212: Homework Solutions

Maths 212: Homework Solutions 1. The definition of A ensures that x π for all x A, so π is an upper bound of A. To show it is the least upper bound, suppose x < π and consider two cases. If x < 1, then

### Bayesian Paradigm. Maximum A Posteriori Estimation

Bayesian Paradigm Maximum A Posteriori Estimation Simple acquisition model noise + degradation Constraint minimization or Equivalent formulation Constraint minimization Lagrangian (unconstraint minimization)

### ADVANCED CALCULUS - MTH433 LECTURE 4 - FINITE AND INFINITE SETS

ADVANCED CALCULUS - MTH433 LECTURE 4 - FINITE AND INFINITE SETS 1. Cardinal number of a set The cardinal number (or simply cardinal) of a set is a generalization of the concept of the number of elements

### 1. Bounded linear maps. A linear map T : E F of real Banach

DIFFERENTIABLE MAPS 1. Bounded linear maps. A linear map T : E F of real Banach spaces E, F is bounded if M > 0 so that for all v E: T v M v. If v r T v C for some positive constants r, C, then T is bounded:

### THE INVERSE FUNCTION THEOREM FOR LIPSCHITZ MAPS

THE INVERSE FUNCTION THEOREM FOR LIPSCHITZ MAPS RALPH HOWARD DEPARTMENT OF MATHEMATICS UNIVERSITY OF SOUTH CAROLINA COLUMBIA, S.C. 29208, USA HOWARD@MATH.SC.EDU Abstract. This is an edited version of a

### LECTURE 15: COMPLETENESS AND CONVEXITY

LECTURE 15: COMPLETENESS AND CONVEXITY 1. The Hopf-Rinow Theorem Recall that a Riemannian manifold (M, g) is called geodesically complete if the maximal defining interval of any geodesic is R. On the other

### Tree sets. Reinhard Diestel

1 Tree sets Reinhard Diestel Abstract We study an abstract notion of tree structure which generalizes treedecompositions of graphs and matroids. Unlike tree-decompositions, which are too closely linked

### Least Sparsity of p-norm based Optimization Problems with p > 1

Least Sparsity of p-norm based Optimization Problems with p > Jinglai Shen and Seyedahmad Mousavi Original version: July, 07; Revision: February, 08 Abstract Motivated by l p -optimization arising from

### Math 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces.

Math 350 Fall 2011 Notes about inner product spaces In this notes we state and prove some important properties of inner product spaces. First, recall the dot product on R n : if x, y R n, say x = (x 1,...,

### Spanning and Independence Properties of Finite Frames

Chapter 1 Spanning and Independence Properties of Finite Frames Peter G. Casazza and Darrin Speegle Abstract The fundamental notion of frame theory is redundancy. It is this property which makes frames

### CONVERGENCE PROPERTIES OF COMBINED RELAXATION METHODS

CONVERGENCE PROPERTIES OF COMBINED RELAXATION METHODS Igor V. Konnov Department of Applied Mathematics, Kazan University Kazan 420008, Russia Preprint, March 2002 ISBN 951-42-6687-0 AMS classification:

### I teach myself... Hilbert spaces

I teach myself... Hilbert spaces by F.J.Sayas, for MATH 806 November 4, 2015 This document will be growing with the semester. Every in red is for you to justify. Even if we start with the basic definition

### arxiv: v1 [math.fa] 1 Nov 2017

NON-EXPANSIVE BIJECTIONS TO THE UNIT BALL OF l 1 -SUM OF STRICTLY CONVEX BANACH SPACES V. KADETS AND O. ZAVARZINA arxiv:1711.00262v1 [math.fa] 1 Nov 2017 Abstract. Extending recent results by Cascales,

### V (v i + W i ) (v i + W i ) is path-connected and hence is connected.

Math 396. Connectedness of hyperplane complements Note that the complement of a point in R is disconnected and the complement of a (translated) line in R 2 is disconnected. Quite generally, we claim that

### Hilbert spaces. 1. Cauchy-Schwarz-Bunyakowsky inequality

(October 29, 2016) Hilbert spaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ [This document is http://www.math.umn.edu/ garrett/m/fun/notes 2016-17/03 hsp.pdf] Hilbert spaces are

### A New Look at First Order Methods Lifting the Lipschitz Gradient Continuity Restriction

A New Look at First Order Methods Lifting the Lipschitz Gradient Continuity Restriction Marc Teboulle School of Mathematical Sciences Tel Aviv University Joint work with H. Bauschke and J. Bolte Optimization

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.

DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### Lecture 7: Semidefinite programming

CS 766/QIC 820 Theory of Quantum Information (Fall 2011) Lecture 7: Semidefinite programming This lecture is on semidefinite programming, which is a powerful technique from both an analytic and computational

### On Friedrichs inequality, Helmholtz decomposition, vector potentials, and the div-curl lemma. Ben Schweizer 1

On Friedrichs inequality, Helmholtz decomposition, vector potentials, and the div-curl lemma Ben Schweizer 1 January 16, 2017 Abstract: We study connections between four different types of results that

### INF-SUP CONDITION FOR OPERATOR EQUATIONS

INF-SUP CONDITION FOR OPERATOR EQUATIONS LONG CHEN We study the well-posedness of the operator equation (1) T u = f. where T is a linear and bounded operator between two linear vector spaces. We give equivalent

### PARTIAL REGULARITY OF BRENIER SOLUTIONS OF THE MONGE-AMPÈRE EQUATION

PARTIAL REGULARITY OF BRENIER SOLUTIONS OF THE MONGE-AMPÈRE EQUATION ALESSIO FIGALLI AND YOUNG-HEON KIM Abstract. Given Ω, Λ R n two bounded open sets, and f and g two probability densities concentrated

### Variational Formulations

Chapter 2 Variational Formulations In this chapter we will derive a variational (or weak) formulation of the elliptic boundary value problem (1.4). We will discuss all fundamental theoretical results that

### REAL RENORMINGS ON COMPLEX BANACH SPACES

REAL RENORMINGS ON COMPLEX BANACH SPACES F. J. GARCÍA PACHECO AND A. MIRALLES Abstract. In this paper we provide two ways of obtaining real Banach spaces that cannot come from complex spaces. In concrete

### Topology, Math 581, Fall 2017 last updated: November 24, Topology 1, Math 581, Fall 2017: Notes and homework Krzysztof Chris Ciesielski

Topology, Math 581, Fall 2017 last updated: November 24, 2017 1 Topology 1, Math 581, Fall 2017: Notes and homework Krzysztof Chris Ciesielski Class of August 17: Course and syllabus overview. Topology

### ON A CLASS OF NONSMOOTH COMPOSITE FUNCTIONS

MATHEMATICS OF OPERATIONS RESEARCH Vol. 28, No. 4, November 2003, pp. 677 692 Printed in U.S.A. ON A CLASS OF NONSMOOTH COMPOSITE FUNCTIONS ALEXANDER SHAPIRO We discuss in this paper a class of nonsmooth

### On the Weak Convergence of the Extragradient Method for Solving Pseudo-Monotone Variational Inequalities

J Optim Theory Appl 208) 76:399 409 https://doi.org/0.007/s0957-07-24-0 On the Weak Convergence of the Extragradient Method for Solving Pseudo-Monotone Variational Inequalities Phan Tu Vuong Received:

### David Hilbert was old and partly deaf in the nineteen thirties. Yet being a diligent

Chapter 5 ddddd dddddd dddddddd ddddddd dddddddd ddddddd Hilbert Space The Euclidean norm is special among all norms defined in R n for being induced by the Euclidean inner product (the dot product). A

### Statistical regularization theory for Inverse Problems with Poisson data

Statistical regularization theory for Inverse Problems with Poisson data Frank Werner 1,2, joint with Thorsten Hohage 1 Statistical Inverse Problems in Biophysics Group Max Planck Institute for Biophysical

### s P = f(ξ n )(x i x i 1 ). i=1

Compactness and total boundedness via nets The aim of this chapter is to define the notion of a net (generalized sequence) and to characterize compactness and total boundedness by this important topological

### FUNCTION BASES FOR TOPOLOGICAL VECTOR SPACES. Yılmaz Yılmaz

Topological Methods in Nonlinear Analysis Journal of the Juliusz Schauder Center Volume 33, 2009, 335 353 FUNCTION BASES FOR TOPOLOGICAL VECTOR SPACES Yılmaz Yılmaz Abstract. Our main interest in this

### COMPLEXITY OF SHORT RECTANGLES AND PERIODICITY

COMPLEXITY OF SHORT RECTANGLES AND PERIODICITY VAN CYR AND BRYNA KRA Abstract. The Morse-Hedlund Theorem states that a bi-infinite sequence η in a finite alphabet is periodic if and only if there exists

### Calculus of Variations. Final Examination

Université Paris-Saclay M AMS and Optimization January 18th, 018 Calculus of Variations Final Examination Duration : 3h ; all kind of paper documents (notes, books...) are authorized. The total score of

### Cambridge University Press The Mathematics of Signal Processing Steven B. Damelin and Willard Miller Excerpt More information

Introduction Consider a linear system y = Φx where Φ can be taken as an m n matrix acting on Euclidean space or more generally, a linear operator on a Hilbert space. We call the vector x a signal or input,

### Regularization in Banach Space

Regularization in Banach Space Barbara Kaltenbacher, Alpen-Adria-Universität Klagenfurt joint work with Uno Hämarik, University of Tartu Bernd Hofmann, Technical University of Chemnitz Urve Kangro, University

### Compressibility of Infinite Sequences and its Interplay with Compressed Sensing Recovery

Compressibility of Infinite Sequences and its Interplay with Compressed Sensing Recovery Jorge F. Silva and Eduardo Pavez Department of Electrical Engineering Information and Decision Systems Group Universidad

### Boundedly complete weak-cauchy basic sequences in Banach spaces with the PCP

Journal of Functional Analysis 253 (2007) 772 781 www.elsevier.com/locate/jfa Note Boundedly complete weak-cauchy basic sequences in Banach spaces with the PCP Haskell Rosenthal Department of Mathematics,

### On the complexity of the hybrid proximal extragradient method for the iterates and the ergodic mean

On the complexity of the hybrid proximal extragradient method for the iterates and the ergodic mean Renato D.C. Monteiro B. F. Svaiter March 17, 2009 Abstract In this paper we analyze the iteration-complexity

### On John type ellipsoids

On John type ellipsoids B. Klartag Tel Aviv University Abstract Given an arbitrary convex symmetric body K R n, we construct a natural and non-trivial continuous map u K which associates ellipsoids to

### INSTITUTE of MATHEMATICS. ACADEMY of SCIENCES of the CZECH REPUBLIC. A universal operator on the Gurariĭ space

INSTITUTE of MATHEMATICS Academy of Sciences Czech Republic INSTITUTE of MATHEMATICS ACADEMY of SCIENCES of the CZECH REPUBLIC A universal operator on the Gurariĭ space Joanna Garbulińska-Wȩgrzyn Wiesław

### arxiv: v1 [math.na] 9 Feb 2013

STRENGTHENED CAUCHY-SCHWARZ AND HÖLDER INEQUALITIES arxiv:1302.2254v1 [math.na] 9 Feb 2013 J. M. ALDAZ Abstract. We present some identities related to the Cauchy-Schwarz inequality in complex inner product

### EXISTENCE OF SOLUTIONS FOR CROSS CRITICAL EXPONENTIAL N-LAPLACIAN SYSTEMS

Electronic Journal of Differential Equations, Vol. 2014 (2014), o. 28, pp. 1 10. ISS: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu ftp ejde.math.txstate.edu EXISTECE OF SOLUTIOS

### Douglas-Rachford splitting for nonconvex feasibility problems

Douglas-Rachford splitting for nonconvex feasibility problems Guoyin Li Ting Kei Pong Jan 3, 015 Abstract We adapt the Douglas-Rachford DR) splitting method to solve nonconvex feasibility problems by studying