Optimality and complexity for constrained optimization problems with nonconvex regularization

Transcription

1 Opimaliy and complexiy for consrained opimizaion problems wih nonconvex regularizaion Wei Bian Deparmen of Mahemaics, Harbin Insiue of Technology, Harbin, China, Xiaojun Chen Deparmen of Applied Mahemaics, The Hong Kong Polyechnic Universiy, Hong Kong, China, In his paper, we consider a class of consrained opimizaion problems where he feasible se is a general closed convex se and he objecive funcion has a nonsmooh, nonconvex regularizer. Such regularizer includes widely used SCAD, MCP, logisic, fracion, hard hresholding and non-lipschiz L p penalies as special cases. Using he heory of he generalized direcional derivaive and he Clarke angen cone, we derive a firs order necessary opimaliy condiion for local minimizers of he problem, and define he generalized saionary poin of i. The generalized saionary poin is he Clarke saionary poin when he objecive funcion is Lipschiz coninuous a his poin, and he scaled saionary poin when he objecive funcion is no Lipschiz coninuous a his poin. We prove he consisency beween he generalized direcional derivaive and he limi of he classic direcional derivaives associaed wih he smoohing funcion. Moreover we show ha finding a global minimizer of such opimizaion problems is srongly NP-hard and esablish posiive lower bounds for he absolue value of nonzero enries in every local minimizer of he problem if he regularizer is concave in an open se. Key words : Consrained nonsmooh nonconvex opimizaion; direcional derivaive consisency; opimaliy condiion; lower bound heory; complexiy MSC2000 subjec classificaion : Primary: 49K35, 90C26; secondary: 65K05, 90C46 1. Inroducion In his paper, we consider he following consrained opimizaion problem min x X f(x) := Θ(x) + c(h(x)), (1) where Θ : R n R and c : R m R are coninuously differeniable, h : R n R m is coninuous, and X R n is a nonempy closed convex se. Of paricular ineres of his paper is when h is no convex, no differeniable, or even no Lipschiz coninuous. Problem (1) includes many problems in pracice. For insance, he following minimizaion problem min l x u,ax b Θ(x) + m φ( D T i x p p) (2) is a special case of (1), where l {R } n, u {R } n, A R n, b R, D i R n r, p (0, 1] and φ : R + R + is coninuous. Such problem arises from image resoraion (Chan and Liang [12], Chen e al. [17], Nikolova e al. [36]), signal processing (Brucksein e al. [9]), variable selecion (Fan and Li [22], Huang e al. [27], Huang e al. [29], Zhang [44]), ec. Anoher special case of (1) is he following problem m Θ(x) + max{α i m T i x, 0} p, (3) min x X where α i R and m i R n, which has araced much ineres in machine learning, wireless communicaion (Liu e al. [33]), informaion heory, daa analysis (Fan and Peng [23], Huber [28]), 1

2 2 W. Bian and X. Chen: Opimaliy and complexiy ec. Moreover, a number of consrained opimizaion problems can be reformulaed as problem (1) by using he exac penaly mehod wih nonsmooh or non-lipschiz coninuous penaly funcions (Auslender [3]). The generic naure of he firs and second order opimaliy condiions in nonlinear programming are reaed in Spingarn and Rockafellar [39]. When X = R n and c(h(x)) = x p p (0 < p < 1), he affine scaled firs and second order necessary condiions for local minimizers of (1) are esablished in Chen e al. [18]. By using subspace echniques, Chen e al. [16] exended he firs and second order necessary condiions o c(h(x)) = Dx p p wih D R m n. However, he opimaliy condiions in Chen e al. [16, 18] are weaker han he Clarke opimaliy condiions in Clarke [19] for p = 1. In his paper, we will derive a necessary opimaliy condiion for he non-lipschiz consrained opimizaion problem (1), which reduces o he Clarke opimaliy condiion when he objecive funcion in (1) is locally Lipschiz coninuous. A poin x is called a Clarke saionary poin of f if f is locally Lipschiz a x and here is V f(x ) such ha (V, x x ) 0, x X, (4) where f(x) = con{v f(y) v, f is differeniable a y, y x} is he Clarke subdifferenial of f and con denoes he convex hull. From Theorem 9.61 and (b) of Corollary 8.47 in Rockafellar and Wes [38], he subdifferenial associaed wih a smoohing funcion G f(x) = con{v x f(x k, µ k ) v, for x k x, µ k 0 }, is nonempy and bounded, and f(x) G f(x). In Burke and Hoheisel [10], Burke e al. [11], Chen [14], Rockafellar and Wes [38], i is shown ha many smoohing funcions saisfy he gradien consisency f(x) = G f(x). (5) The gradien consisency is an imporan propery of he smoohing mehods, which guaranees he convergence of smoohing mehods wih adapive updaing schemes of smoohing parameers o a saionary poin of he original problem. Due o he non-lipschiz coninuiy of he objecive funcion f, Clarke opimaliy condiion (4) canno be applied o (1). In Jahn [30], Jahn inroduced a direcional derivaive for Lipschiz consrained opimizaion problems f ( x; v) = lim sup y x, y X 0, y + v X f(y + v) f(y), which is equal o he Clarke generalized direcional derivaive a he inerior poins of X. In his paper, we exend he direcional derivaive in Jahn [30] o he non-lipschiz consrained opimizaion problem (1). Using he exended direcional derivaive and he Clarke angen cone, we derive necessary opimaliy condiions. The new opimaliy condiions are equivalen o he opimaliy condiions in Bian e al. [8], Chen e al. [16, 18], when he objecive funcion is no Lipschiz coninuous, and o he Clarke opimaliy condiion (4) when he objecive funcion is Lipschiz coninuous. Moreover, we esablish he consisency beween he generalized direcional derivaive and he limi of he classic direcional derivaives associaed wih he smoohing funcion. The direcional derivaive consisency guaranees he convergence of smoohing mehods o a generalized saionary poin of (1). Problem (1) includes he regularized minimizaion problem as a special case when Θ(x) is a daa fiing erm and c(h(x)) is a regularizaion erm (also called a penaly erm in some aricles).

3 W. Bian and X. Chen: Opimaliy and complexiy 3 In sparse opimizaion, nonconvex non-lipschiz regularizaion provides more efficien models o exrac he essenial feaures of soluions han he convex regularizaion (Bian and Chen [6], Charrand and Saneva [13], Chen [14], Chen e al. [17], Fan and Li [22], Huang e al. [27], Huang e al. [29], Loh and Wainwrigh [34], Lu [35], Nikolova e al. [36], Wang e al. [42], Zhang [44]). The SCAD penaly funcion in Fan and Li [22] and he MCP funcion in Zhang [44] have various desirable properies in variable selecion. Logisic and fracion penaly funcions yield edge preservaion in image resoraion (Nikolova e al. [36]). The l p norm penaly funcion wih 0 < p < 1 owns he oracle propery in saisics (Fan and Li [22], Knigh and Fu [31]). Nonconvex regularized M-esimaor is proved o have he saisical accuracy and predicion error esimaion in Loh and Wainwrigh [34]. Moreover, he lower bound heory of he l 2 -l p regularized minimizaion problem in Chen e al. [17, 18], a special case of (1), saes ha he absolue value of each componen of any local minimizer of he problem is eiher zero or greaer han a posiive consan. The lower bound heory no only helps us o disinguish zero and nonzero enries of coefficiens in sparse high-dimensional approximaion (Charrand and Saneva [13], Huang e al. [27]), bu also brings he resored image closed conours and nea edges (Chen e al. [17]). In his paper, we exend he lower bound heory of he l 2 -l p regularizaion minimizaion problem o problems (2) and (3) wih 0 < p 1 which include he mos widely used models in saisics and sparse reconsrucion. Moreover, we exend he complexiy resuls of he l 2 -l p regularizaion minimizaion problem in Chen e al. [15] o problem (2) wih a concave funcion φ and 0 < p 1. We show ha he concaviy of penaly funcions is a key propery for boh he lower bound heory and he srong NP hardness. Such exension of he lower bound heory and complexiy is no rivial because of he general consrains and weak condiions on φ. The res of he paper is organized as follows. In secion 2, we firs define a generalized direcional derivaive and presen is properies. Nex, we derive necessary opimaliy condiions for a local minimizer of problem (1), and prove he direcional derivaive consisency associaed wih smoohing funcions. In secion 3, we presen he compuaional complexiy and he lower bound heory of problem (2). In our noaion, R + = [0, ) and R ++ = (0, ). For x R n, 0 < p < and δ > 0, x p p = n x i p, B δ (x) means he open ball cenered a x wih radius δ. For a closed convex subse Ω R n, in(ω) means he inerior of Ω, cl(ω) means he closure of Ω and m(ω) denoes he elemen in Ω wih he smalles Euclidean norm. P X [x] = arg min{ z x 2 : z X } denoes he orhogonal projecion from R n o X. N ++ = {1, 2,...}. 2. Opimaliy condiions Inspired by he generalized direcional derivaive and he angen cone, we presen a firs order necessary opimaliy condiion for local minimizers of he consrained opimizaion problem (1), which is equivalen o he Clarke necessary condiion for locally Lipschiz opimizaion problems and sronger han he necessary opimaliy condiions for he non-lipschiz opimizaion problems in he exising lieraure. A he end of his secion, we prove he direcional direcive consisency associaed wih smoohing funcions We suppose he funcion h in (1) has he following version h(x) := (h 1 (D T 1 x), h 2 (D T 2 x),..., h m (D T mx)) T (6) where D i R n r, h i (i = 1,..., m) : R r R is coninuous, bu no necessarily Lipschiz coninuous Generalized direcional derivaive Definiion 1. A funcion ϕ : R n R is said o be Lipschiz coninuous a(near) x R n if here exis posiive numbers L x and δ such ha ϕ(y) ϕ(z) L x y z 2, y, z B δ (x). Oherwise, ϕ is said o be no Lipschiz coninuous a(near) x R n.

4 4 W. Bian and X. Chen: Opimaliy and complexiy For a fixed x R n, denoe and define I x = {i {1, 2,..., m} : h i is no Lipschiz coninuous a D T i x}, (7) V x = {v : D T i v = 0, i I x }, (8) { h x i (D T hi (D T i x) i I x i x) = h i (D T i x) i I x, which is Lipschiz coninuous a D T i x, i = 1, 2,..., m. Specially, we le V x = R n when I x =. And hen we le f x (x) = Θ(x) + c(h x (x)), (9) wih h x (x) := (h x 1(D T 1 x), h x 2(D T 2 x),..., h x m(d T mx)) T. The funcion f x (x) is Lipschiz coninuous a x and f x ( x) = f( x). The generalized direcional derivaive in Clarke [19] of f x a x in he direcion v R n is defined as Specially, when f is regular, f x( x; v) = lim sup y x, 0 f x (y + v) f x (y). (10) f x( x; v) = f x( x; f x ( x + v) f x ( x) v) = lim. 0 The generalized direcional derivaive in (10) is generalized in Jahn [30] and used in Aude and Dennis [2], Jahn [30] for locally Lipschiz consrained opimizaion. The generalizaion moives us o use he following generalized direcional derivaive of f x a x X in he direcion v R n f x( x; v) = lim sup y x, y X 0, y + v X The definiions in (10) and (11) coincide when x in(x ). Proposiion 1. For any x X and v V x, f x (y + v) f x (y). (11) f ( x; v) = lim sup y x, y X 0, y + v X f(y + v) f(y) exiss (12) and equals o f x( x; v) defined in (11). Proof. Fix x X and v V x. For y R n and > 0, here exiss z beween h(y) and h(y + v) such ha c(h(y + v)) c(h(y)) = c(z) T (h(y + v) h(y)) = c(z) T (h x (y + v) h x (y)). Then, f(y + v) f(y) = Θ(y + v) Θ(y) + c(z)t (h x (y + v) h x (y)). By he Lipschiz coninuiy of Θ and h x i a x, here exis δ > 0 and L > 0 such ha f(y+v) f(y) L, y B δ ( x), (0, δ). Thus, he generalized direcional derivaive of f a x X in he direcion v V x defined in (12) exiss.

5 W. Bian and X. Chen: Opimaliy and complexiy 5 Le {y n } and { n } be he sequences such ha y n X, n 0, y n x, y n + n v X and he upper limi in (12) holds. Using he Lipschiz coninuiy of h x i a x again, we can ge he subsequences {y nk } {y n } and { nk } { n } such ha By he above analysis, hen h x (y nk + nk v) h x (y nk ) lim exiss. (13) k nk f(y f nk + nk v) f(y nk ) ( x; v) = lim k nk h x (y nk + nk v) h x (y nk ) = Θ( x) + c(z) z=h( x) lim. k nk (14) By virue of (11), we have f x( x; f x (y nk + nk v) f x (y nk ) v) lim k nk h x (y nk + nk v) h x (y nk ) = Θ( x) + c(z) z=h x ( x) lim. k nk (15) Using h( x) = h x ( x), (14) and (15), we obain f x( x; v) f ( x; v). On he oher hand, by exracing he sequences {y nk } and { nk } such ha he upper limi in (11) holds and he limi in (13) exiss wih hem, similar o he above analysis, we find ha f ( x; v) f x( x; v). Therefore, f ( x; v) = f x( x; v). Noice ha he generalized direcional derivaive of f a x X in he direcion v V x defined in (12) involves only he behavior of f a x in he hyperplane V x Clarke angen cone Since X is a nonempy closed convex subse of R n, he disance funcion relaed o X is a nonsmooh, Lipschiz coninuous funcion, defined by d X (x) = min{ x y 2 : y X }. The Clarke angen cone o X a x X, denoed as T X (x), is defined by T X (x) = {v R n : d X (x; v) = 0}. Assumpion 1. Assume ha X = X 1 X 2 and in(x 1 ) X 2, where X 1 R n is a nonempy closed convex se and X 2 = {x Ax = b} wih A R n, b R. Under Assumpion 1, we can obain he following properies of he Clarke angen cones o X 1, X 2 and X. Lemma 1. The following saemens hold. (1) in(t X1 (x)), x X 1 ; (2) T X2 (x) = cl{λ(c x) : c X 2, λ 0}, x X 2 ; (3) T X1 X 2 (x) = T X1 (x) T X2 (x), x X. Proof. (1) Fix x X 1 and denoe ˆx in(x 1 ). Le ϵ > 0 be a consan such ha ˆx+B ϵ (0) in(x 1 ). We shall show ha ˆx x + B ϵ (0) T X1 (x), and hence ˆx x in(t X1 (x)). By he convexiy of X 1, d X1 (x) is a convex funcion and for v ˆx x + B ϵ (0), we noice ha x + v (1 )x + (ˆx + B ϵ (0)) X 1, x X 1, 0 1.

6 6 W. Bian and X. Chen: Opimaliy and complexiy Then, d X 1 (x; v) = d d X1 (x + λv) d X1 (x) X 1 (x; v) = lim = 0, λ 0 λ which confirms ha v T X1 (x). (2) Since X 2 is defined by a class of affine equaliies, we have T X2 (x) = cl{λ(c x) : c X 2, λ 0}. (3) By in(x 1 ) X 2, 0 in(x 1 X 2 ), hen T X1 X 2 (x) = T X1 (x) T X2 (x) (Aubin and Cellina [1, pp.141]). Since in(t X1 (x)), for a vecor v in(t X1 (x)), here exiss a scalar ϵ > 0 such ha y + w T X1 (x), for all y T X1 (x) B ϵ (x), w B ϵ (v) and 0 < ϵ. We ofen call in(t X1 (x)) he hyperangen cone o X 1 a x. And by Lemma 1 (2), we have x + v X 2, x X 2, 0, v T X2 (x) Necessary opimaliy condiion Denoe r-in(t X (x)) = in(t X1 (x)) T X2 (x). Since f is no assumed o be locally Lipschiz coninuous, he calculus heory developed in Aude and Dennis [2] canno be direcly applied o f. The nex lemma exends calculus resuls for he unconsrained case in Clarke [19] and he consrained case in Aude and Dennis [2]. For any x X, from 0 r-in(t X (x)) V x, we know r-in(t X (x)) V x. Lemma 2. For x X and v T X ( x) V x, f ( x; v) = lim f ( x; w). w v w r-in(t X ( x)) V x Proof. By he locally Lipschiz coninuiy of h x, here are ϵ > 0 and L x > 0 such ha h x (x) h x (y) 2 L x x y 2, x, y B ϵ ( x). (16) Le {w k } r-in(t X ( x)) V x be a sequence of direcions converging o a vecor v T X ( x) V x. By {w k } r-in(t X ( x)), le ϵ k > 0 be such ha x + w k X 1 whenever x X B ϵk ( x) and 0 < ϵ k. By Lemma 1 (2), we obain x + v X 2, x + w k X 2, 0, x X. Then, for all w k, i gives f f(x + v) f(x) ( x; v) = lim sup x x, x X 0, x + v X = lim sup x x, x X 0, x + v X x + w k X = lim sup x x, x X 0, x + v X x + w k X f(x + v) f(x) f(x + w k ) f(x) + f(x + v) f(x + w k). Le δ > 0 be such ha x + w k B ϵ ( x) for any x B δ ( x), 0 < δ and k N ++. By he Lipschiz condiion in (16), we have h x(x + v) h x (x + w k ) 2 L x v w k 2, x B δ ( x), 0 < < δ, k N ++. (17)

7 W. Bian and X. Chen: Opimaliy and complexiy 7 From he mean value heorem, here exiss z beween h(x + v) and h(x + w k ) such ha f(x + v) f(x + w k ) =Θ(x + v) Θ(x + w k ) + c(z) T (h(x + v) h(x + w k )) =Θ(x + v) Θ(x + w k ) + c(z) T (h x (x + v) h x (x + w k )). Then, for any x B δ ( x),0 < δ, we have f(x + v) f(x + w k) L Θ v w k 2 + L c L x v w k 2, where L Θ = sup{ Θ(y) 2 : y B ϵ ( x)} and L c = sup{ c(z) T z=h(y) 2 : y B ϵ ( x)}. Thus, (17) implies f ( x; w k ) L Θ v w k 2 L c L x v w k 2 f ( x; v) f ( x; w k ) + L Θ v w k 2 + L c L x v w k 2, k N ++. As k goes o infiniy, he above inequaliy follows f ( x; v) = lim k f ( x; w k ). Since {w k } is an arbirary sequence in r-in(t X ( x)) V x converging o v, we obain he resul in his lemma. Noe ha he above lemma is no necessarily rue when r-in(t X (x)) is empy. A similar example can be given following he idea in Aude and Dennis [2, Example 3.10]. Tha is why we pu he assumpion in(x 1 ) X 2 a he beginning of his secion. Based on Lemmas 1-2, he following heorem gives he main heoreical resul of his secion. Theorem 1. If x is a local minimizer of (1), hen f (x, v) 0 for every direcion v T X (x ) V x. Proof. Suppose x is a local minimizer of f over X and le w r-in(t X (x )) V x. There exis ϵ > 0 and L x > 0 such ha f(x ) f(x), and h x (x) h x (y) 2 L x x y 2, x, y X B ϵ (x ). (18) Since w in(t X1 (x )), here exiss ϵ (0, ϵ] such ha x + w X 1, x X 1 B ϵ (x ), 0 ϵ. By Lemma 1 (2), x + w X, x X B ϵ (x ), 0 < ϵ. And hen we can choose δ (0, ϵ] such ha x, x + w, x + w B ϵ (x ) X, x B 2(x ) X, 0 < δ. By (18), for all x B 2(x ) X, 0 < < δ, we obain Thus, h x (x + w) h x (x + w) lim x B 2 (x ) X x + w X, 0 h x (x) h x (x ) x x 2 2 2L x 2L x. h x (x + w) h x (x + w) h x (x) h x (x ) = 0. (19) From he mean value heorem, here exis z 1 beween h(x ) and h(x + w), and z 2 beween h(x ) and h(x + w) such ha c(h(x + w)) c(h(x)) = c(z 1) T (h(x + w) h(x)) = c(z 1) T (h x (x + w) h x (x)) c(h(x + w)) c(h(x )) c(z 2) T (h(x + w) h(x )) c(z 2) T (h x (x + w) h x (x )). (20)

8 8 W. Bian and X. Chen: Opimaliy and complexiy By (19), (20) and he coninuous differeniabiliy of Θ, we have Thus, f(x + w) f(x) lim [ x B 2 (x ) X x + w X, 0 = c(z) T z=h(x ) =0. f(x + w) f(x ) ] lim [ h x (x + w) h x (x) x B 2 (x ) X x + w X, 0 h x (x + w) h x (x ) ] f(x + w) f(x) lim sup [ x x, x X 0, x + w X By f(x + w) f(x ) 0 for 0 < ϵ, (21) implies f (x ; w) = lim sup x x, x X 0, x + w X f(x + w) f(x ) ] 0. (21) f(x + w) f(x) By Lemma 2, we can give ha f (x ; v) 0 for any v T X (x ) V x. Based on Theorem 1, we give a new definiion of a generalized saionary poin of problem (1). Definiion 2. x X is said o be a generalized saionary poin of (1), if f (x ; v) 0 for every v T X (x ) V x. I is worh noing ha a generalized saionary poin x is a Clarke saionary poin of problem (1) when f is Lipschiz coninuous a x. Remark 1. Suppose h i (Di T x) is regular in X \N i, where 0. N i = {x X : h i is no Lipschiz coninuous a D T i x}, i = 1, 2,..., m. For x X, he regulariy assumpion allows us o define V x by V x ={v : for any i I x, here exiss δ > 0 such ha h i ( x + v) = h i ( x) holds for all 0 δ}, which is a bigger se han V x given in (8). Hence a generalized saionary poin defined in Definiion 2 can be more robus wih his V x. For example, if f is defined as in (3), I x = {i {1, 2,..., m} : m T i x = α i } and we can le V x = {v : m T i v 0, i I x }, which includes {v : m T i v = 0, i I x } as a proper subse. We noice ha a generalized saionary poin defined in Definiion 2 is a scaled saionary poin defined in Bian and Chen [6], Bian e al. [8], Chen e al. [16], Ge e al. [25] for he special cases of (2) wih 0 < p < 1. Moreover i is sronger han a scaled saionary poin for he Lipschiz case, since i is a Clarke saionary poin bu a scaled saionary poin is no necessarily a Clarke saionary poin for he Lipschiz opimizaion problem Direcional derivaive consisency In his subsecion, we show ha he generalized direcional derivaive of f defined in (12) can be represened by he limi of a sequence of direcional derivaives of a smoohing funcion of f. This propery is imporan for developmen of numerical algorihms for nonconvex non-lipschiz consrained opimizaion problems. Definiion 3. (Chen [14]) Le g : R n R be a coninuous funcion. We call g : R n [0, ) R a smoohing funcion of g, if g(, µ) is coninuously differeniable for any fixed µ > 0 and lim z x,µ 0 g(z, µ) = g(x) holds for any x R n.

9 W. Bian and X. Chen: Opimaliy and complexiy 9 Le h(x, µ) = ( h 1 (D T 1 x, µ), h 2 (D T 2 x, µ),..., h m (D T mx, µ)) T, where h i is a smoohing funcion of h i in (6). Then f(x, µ) := Θ(x) + c( h(x, µ)) is a smoohing funcion of f. Since f(x, µ) is coninuously differeniable abou x for any fixed µ > 0, he generalized direcional derivaive of i wih respec o x can be given by f (x, µ; v) = lim sup y x, y X 0, y + v X f(y + v, µ) f(y) = x f(x, µ), v. (22) Theorem 2. Suppose h i is coninuously differeniable in X \N i, i {1, 2,..., m}, where N i = {x : h i is no Lipschiz coninuous a D T i x}, hen lim x f(xk, µ k ), v = f (x; v), v V x. (23) x k X, x k x, µ k 0 Proof. Le x k be a sequence in X converging o x and {µ k } be a posiive sequence converging o 0. For w V x, by he closed form of x f(xk, µ k ), we have x f(xk, µ k ), w = Θ(x k ), w + x h(xk, µ k ) c(z) z= h(xk,µ k ), w = Θ(x k ), w + c(z) z= h(xk,µ k ), x h(x k, µ k ) T w, (24) where x h(xk, µ k ) T w = ( x h1 (D T 1 w, µ k ) T w,..., x hm (D T mx k, µ k ) T w) T. For i I x, by w V x, we obain Di T w = 0, hen x hi (Di T x k, µ k ) T w = z hi (z, µ k ) T z=d i kd T T x i w = 0. Define { hi h x i (D T (D T i x, µ) i I x, i x, µ) = h i (D T i = 1, 2,..., m. i x, µ) i I x, Denoe h x (x, µ) = ( h x 1(D T 1 x, µ), h x 2(D T 2 x, µ),..., h x m(d T mx, µ)) T. Then, Thus, coming back o (24), we obain x h(xk, µ k ) T w = x h x (x k, µ k ) T w. x f(x k, µ k ), w = Θ(x k ), w + c(z) z= h(xk,µ k ), x h x (x k, µ k ) T w = Θ(x k ), w + x h x (x k, µ k ) c(z) z= h(xk,µ k ), w = Θ(x k ) + x h x (x k, µ k ) c(z) z= h(xk,µ k ), w. (25) Since h i is coninuously differeniable a D T i x for i I x and h x ( x) = h( x), we obain lim k Θ(x k) + x h x (x k, µ k ) c(z) z= h(xk,µ k ) = Θ( x) + h x ( x) c(z) z=h( x) = f x ( x), (26) where f x is defined in (9). Thus, f ( x, w) = f x( x, w) = f x ( x), w = lim k Θ(x k ) + x h x (x k, µ k ) c(z) z= h(xk,µ k ), w = lim k x f(x k, µ k ), w, (27) where he firs equaion uses Proposiion 1, he hird uses (26) and he fourh uses (25). Now we give anoher consisency resul on subspace V x.

10 10 W. Bian and X. Chen: Opimaliy and complexiy Lemma 3. Le x k be a sequence in X wih a limi poin x. For w V x, here exiss a sequence {x kl } {x k } such ha w V xkl, l N ++. Proof. If his lemma is no rue, hen here is K N ++ such ha w V xk, k K. By he definiion of V xk, here exiss i k I xk such ha D T i k w 0, k K. By I xk {1, 2,..., m}, here exis j {1, 2,..., m} and a subsequence of {x k }, denoed as {x kl }, such ha j I xkl and Dj T w 0. Noe ha j I xkl implies h j is no Lipschiz coninuous a Dj T x kl. Since he non-lipschiz poins of h j is a closed subse of R n, h j is also no Lipschiz coninuous a Dj T x, which means j I x. By w V x, we obain Dj T w = 0, which leads a conradicion. Therefore, he saemen in his lemma holds. Based on he consisency resuls given in Theorem 2 and Lemma 3, he nex corollary shows he generalized saionary poin consisency of he smoohing funcions. Corollary 1. Le {ϵ k } and {µ k } be posiive sequences converging o 0. Wih he condiions on h in Theorem 2, if x k saisfies x f(xk, µ k ), v ϵ k for every v T X (x k ) V x k B 1 (0), hen any accumulaion poin of {x k } X is a generalized saionary poin of (1). Proof. Le x be an accumulaion poin of {x k }. Wihou loss of generaliy, we suppose lim k x k = x. For w r-in(t X ( x)) V x B 1 (0), from Lemma 3, we can suppose w V xk, k N ++. By w r-in(t X ( x)), here exiss ϵ > 0 such ha x + sw X, x X B ϵ ( x), 0 s ϵ. (28) Since x k is converging o x, here exiss K N ++ such ha x k X B ϵ ( x), k K. By (28), we have x k + sw X, k K, 0 s ϵ. From he convexiy of X, we obain w T X (x k ). From Theorem 2, we have f ( x, w) 0. Then, for any ρ > 0, we have f ( x; ρv) = lim sup y x, y X 0, y + ρv X =ρ lim sup y x, y X s 0, y + sv X f(y + ρv) f(y) f(y + sv) f(y) s = ρf ( x; v) 0. (29) Thus, f ( x; v) 0 for every v r-in(t X ( x)) V x B 1 (0) implies f ( x; v) 0 for every v r-in(t X ( x)) V x. By Lemma 2, i is easy o verify ha f ( x, v) 0 holds for any v T X ( x) V x, which means ha x is a generalized saionary poin of (1). Remark 2. Suppose he gradien consisency associaed wih he smoohing funcion h i holds a is Lipschiz coninuous poins, ha is { lim z x,µ 0 x h i (D T i x, µ)} h i (D T i x), x X, i I x, (30)

11 W. Bian and X. Chen: Opimaliy and complexiy 11 hen { lim z x,µ 0 Θ(z) + h x (x, µ) c(z) z=hx (x)} f x (x), x X. (31) Since f x is Lipschiz coninuous a x, i gives f x( x, v) = max{ ξ, v : ξ f x ( x)}. (32) Similar o he calculaion in (27), by (31) and (32), we obain f ( x, w) = f x( x, w) = max{ ξ, w : ξ f x ( x)} lim sup Θ(x k ) + x h x (x k, µ k ) c(z) z= h(xk,µ k ), w k = lim sup x f(x k, µ k ), w. k Thus, he conclusion in Corollary 1 can be rue wih (30), which is weaker han he sric differeniabiliy of h i in X \N i, i {1, 2,..., m}. Some condiions can be found in Clarke [19] o ensure (30). Specially, when he funcion h in f is wih he form h(x) := (h 1 (d T 1 x), h 2 (d T 2 x),..., h m (d T mx)) T wih d i R n, by Clarke [19, Theorem (i)], he regulariy of h i (d T i x) in X \N i is a sufficien condiion for he saemen in Theorem 2. Corollary 1 shows ha one can find a generalized saionary poin of (1) by using he approximae firs order opimaliy condiion of min x X f(x, µ). Since f(x, µ) is coninuously differeniable for any fixed µ > 0, many numerical algorihms can find a saionary poin of min x X f(x, µ) (Beck and Teboulle [4], Curis and Overon [20], Leviin and Polyak [32], Nocedal and Wrigh [37], Ye [43]). We use one example o show he validiy of he firs order necessary opimaliy condiion and he consisency resul given in his secion. Example 1. Consider he following minimizaion problem min f(x) := (x 1 + 2x 2 1) 2 + λ 1 max{x1 + x 2 + 1, 0} + λ 2 x2, s.. x X = {x R 2 : 1 x 1, x 2 1}. (33) This problem is an example of (1) wih Θ(x) = (x 1 + 2x 2 1) 2, c(y) = λ 1 y 1 + λ 2 y 2, h 1 (D T 1 x) = max{x1 + x 2 + 1, 0} and h 2 (D T 2 x) = x 2, where D 1 = (1, 1) T, D 2 = (0, 1) T. Define he smoohing funcion of f as f(x, µ) = (x 1 + 2x 2 1) 2 + λ 1 ψ(x1 + x 2 + 1, µ) + λ 2 θ(x2, µ), wih ψ(s, µ) = 1(s + s s > µ, s µ 2 ), θ(s, µ) = s 2 2µ + µ s µ. 2 Here, we use he classical projeced algorihm wih Armijo line search o find an approximae generalized saionary poin of min x X f(x, µ). There exiss α > 0 such ha x PX [ x α x f( x, µ)] = 0 if and only if x is a generalized saionary poin of min x X f(x, µ), which is also a Clarke saionary poin of min x X f(x, µ) for any fixed µ > 0. We call xk an approximae saionary poin of min x X f(x, µk ), if here exiss α k > 0 such ha x k P X [x k α k x f(xk, µ k )] 2 α k µ k, which can be found in a finie number of ieraions by he analysis in Bersekas [5]. Choose he iniial ierae x 0 = (0, 0) T. For differen values of λ 1 and λ 2 in (33), he simulaion resuls are lised in Table 1, where f indicaes he opimal funcion value of (33), where he ieraion is erminaed when µ k 10 6.

12 12 W. Bian and X. Chen: Opimaliy and complexiy λ 1 λ 2 accumulaion poin x I x V x f(x ) f 8 2 ( 1.000, 0.000) T {1} {v = (a, a) T : a R} (0.982, 0.000) T {2} {v = (a, 0) T : a R} ( 1.000, 0.962) T R Table 1. Simulaion resuls in Example x 40 x 30 x 2 0 x 160 x x 1 Figure 1. Trajecory of x k in Example 1 wih λ 1 = 8 and λ 2 = 2 When λ 1 = 8, λ 2 = 2, since h 2 (D T 2 x) is coninuously differeniable a x, for v V x, by h 1 (D T 1 (x + v)) = h 1 (D T 1 x ), > 0, we obain f (x Θ(y + v) Θ(y) + λ 2 h 2 (D2 T (y + v)) λ 2 h 2 (D2 T y) ; v) = lim sup y x, y X 0, y + v X = Θ(x ) + λ 2 h 2(D T 2 x )D 2, v = 4v v 2, where v 1 = v 2 by v V x, and v 1 R + by x 1 = and he condiion x + v X in f (x ; v). Then, f (x ; v) 0, v V x, which means ha ( 1.000, 0.000) T is a generalized saionary poin of (33). Similarly, when λ 1 = 0.1, λ 2 = 0.2: where v 2 = 0 by v V x ; when λ 1 = 0.5, λ 2 = 0.1: f (x ; v) = Θ(x ) + λ 1 h 1(D T 1 x )D 1, v = 0.036v 2, f (x ; v) = Θ(x ) + λ 1 h 1(D T 1 x )D 1 + λ 2 h 2(D T 2 x )D 2, v = 0.102v 1, where v 1 R + by x 1 = This gives f (x ; v) 0, for all v V x. Thus, he accumulaion poins in Table 1 are generalized saionary poins of (33) wih differen values of λ 1 and λ 2. Furhermore, he rajecory of x k of he smoohing algorihm for (33) wih λ 1 = 8, λ 2 = 2 are picured in Figure 1 wih he isolines of f in X. 3. Nonconvex regularizaion In his secion, we focus on problem (2) wih he funcion φ saisfying he following assumpion. Assumpion 2. Assume ha φ : R + R + wih φ(0) = 0 is coninuously differeniable, nondecreasing and concave on (0, ), and φ is locally Lipschiz coninuous on R ++.

13 W. Bian and X. Chen: Opimaliy and complexiy 13 The funcion φ() = saisfies Assumpion 2. I is known ha problem (2) wih X = R n, φ() = and p (0, 1) is srongly NP hard bu enjoys lower bound heory. However, he complexiy and lower bound heory of problem (2) wih a general convex se X and he class of funcions φ saisfying Assumpion 2 have no been sudied. In his secion, we show ha he key condiion for he complexiy and lower bound heory is ha he funcion φ(z p ) is sricly concave in an open inerval Compuaional complexiy In his subsecion, we will show he srong NP-hardness of he following problem n min Hx c φ( x i p ), (34) where H R s n, c R s and 0 < p 1. Lemma 4. φ( s p ) + φ( p ) φ( s + p ), s, R. Proof. Define ψ(α) = φ(α + s p ) φ(α) on [0, + ). Then from he concaviy of φ, ψ (α) = φ (α + s p ) φ (α) 0, which implies ψ( p ) ψ(0). Thus, φ( p + s p ) φ( p ) + φ( s p ). Since + s p p + s p and φ is non-decreasing on [0, + ), we obain φ( + s p ) φ( p ) + φ( s p ). Firs, we give wo preliminary resuls for proving he srong NP-hardness of (34) wih 0 < p 1. The firs is for p = 1 and he second is for 0 < p < 1. Lemma 5. Suppose φ is sricly concave and wice coninuously differeniable on [τ 1, τ 2 ] wih τ 1 > 0 and τ 2 > τ 1. There exiss γ > 0 such ha when γ > γ and p = 1, he minimizaion problem min z R g(z) = γ z τ γ z τ φ( z p ), (35) has a unique soluion z (τ 1, τ 2 ). Proof. Since φ is wice coninuously differeniable in [τ 1, τ 2 ], here exiss α > 0 such ha 0 φ (s) α and α φ α (s) 0, s [τ 1, τ 2 ]. Le γ = max{, α 2(τ 2 τ 1 } and suppose γ > γ. ) 4 Noe ha g(z) > g(0) = γτ1 2 + γτ2 2 for all z < 0, and g(z) > g(τ 2 ) = γ(τ 2 τ 1 ) 2 + φ(τ 2 ) for all z > τ 2. Then, he minimum poin of g(z) mus lie wihin [0, τ 2 ]. To minimize g(z) on [0, τ 2 ], we check is firs derivaive g (z) = 2γ(z τ 1 ) + 2γ(z τ 2 ) + φ (z), 0 < z τ 2. When 0 < z τ 1, g (z) = 4γz 2γτ 1 2γτ 2 + φ (z) 2γτ 1 2γτ 2 + α < 0, which means ha g(z) is sricly decreasing on [0, τ 1 ]. Therefore, he minimum poin of g(z) mus lie wihin (τ 1, τ 2 ]. Consider solving g (z) = 2γ(z τ 1 ) + 2γ(z τ 2 ) + φ (z) = 0 on (τ 1, τ 2 ]. Calculae g (z) = 4γ + φ (z) > 0. And we have g (τ 2 ) = 2γτ 2 2γτ 1 +φ (τ 2 ) > 0, g (τ 1 ) < 0. Therefore, here exiss a unique z (τ 1, τ 2 ) such ha g ( z) = 0, which is he unique global minimum poin of g(z) in R. For he case ha 0 < p < 1, we need a weaker condiion on φ o obain a similar resul as in Lemma 5. Lemma 6. Suppose φ is wice coninuously differeniable on [τ p 1, τ p 2 ] wih τ 2 > τ 1 > 0. There exiss γ > 0 such ha when γ > γ and 0 < p < 1, he minimizaion problem (35) has a unique soluion z (τ 1, τ 2 ). Proof. Firs, here exiss α > 0 such ha 0 φ (s) α and α φ (s) 0, s [τ p 1, τ p 2 ]. Le γ > γ, where γ = max{ 2φ(( τ 1+τ 2 ) p ) 2 pατ p 1 2p 2 1 ατ1 + ατ p 2 1,, }. (τ 2 τ 1 ) 2 2(τ 2 τ 1 ) 4

14 14 W. Bian and X. Chen: Opimaliy and complexiy Similar o he analysis in Lemma 5, he minimum poin of g(z) mus lie wihin [0, τ 2 ]. When z [0, τ 1 ], g(z) γ(τ 2 τ 1 ) 2, hen by γ > 2φ(( τ 1 +τ 2 2 ) p ) (τ 2 τ 1, we have ) 2 g(z) > g( τ 1 + τ 2 ), z [0, τ 1 ]. 2 Thus, he minimum poin of g(z) mus lie wih in (τ 1, τ 2 ]. pατ p 1 1 To minimize g(z) on (τ 1, τ 2 ], we check is firs derivaive. By γ >, we have 2(τ 2 τ 1 ) g (τ 1 ) = 2γ(τ 1 τ 2 ) + pφ (τ p 1 )τ p 1 1 < 0, and by φ 0, we ge g (τ 2 ) = 2γ(τ 2 τ 1 ) + pφ (τ p 2 )τ p 1 2 > 0. Now we consider he soluion of he consrained equaion g (z) = 2γ(z τ 1 ) + 2γ(z τ 2 ) + pφ (z p )z p 1 = 0, z (τ 1, τ 2 ]. 2p 2 ατ +ατ p 2 We calculae ha g (z) = 4γ + p 2 φ (z p )z 2p 2 + p(p 1)φ (z p )z p > 0 since γ >. Combining i wih g (τ 1 ) < 0 and g (τ 2 ) > 0, here exiss a unique z (τ 1, τ 2 ) such ha g ( z) = 0, which 4 is he unique global minimizer of g(z) in R. Since φ is locally Lipschiz coninuous in R ++, φ is coninuously differeniable almos everywhere in R ++. If φ is sricly concave in (τ, τ) wih τ > τ > 0, here exis τ 1 > 0 and τ 2 > τ 1 wih [τ p 1, τ p 2 ] (τ, τ) such ha φ is sricly concave and wice coninuously differeniable on [τ p 1, τ p 2 ]. Thus, he sric concaviy of φ in an open inerval of R + is sufficien for he exisence of [τ p 1, τ p 2 ] wih τ 2 > τ 1 > 0 such ha φ is sricly concave and wice coninuously differeniable on i. And here is no oher condiion needed o guaranee he supposiion of φ in Lemma 6. Theorem Minimizaion problem (34) is srongly NP-hard for any given 0 < p < If φ is srongly concave in an open inerval of R +, hen minimizaion problem (34) is srongly NP-hard for p = 1. Proof. Now we presen a polynomial ime reducion from he well-known srongly NP-hard pariion problem (Garey and Johnson [24]) o problem (34). The 3-pariion problem can be described as follows: given a mulise S of n = 3m inegers {a 1, a 2,..., a n } wih sum mb, is here a way o pariion S ino m disjoin subses S 1, S 2,..., S m, such ha he sum of he numbers in each subse is equal? Given an insance of he pariion problem wih a = (a 1, a 2,..., a n ) T R n. We consider he following minimizaion problem in form (34): min x P (x) = m j=1 + γ n α i x ij β 2 + γ n m x ij τ j=1 n m x ij τ 1 2 j=1 n m ( φ( x ij p )), where he parameers τ 1, τ 2 and γ saisfy he supposiions in Lemma 5 for p = 1 and hem in Lemma 6 for 0 < p < 1. From Lemma 4, we have min P (x) x n m n m n m min γ x ij τ γ x ij τ ( φ( x ij p )) x ij ( j=1 j=1 j=1 ) n m m m = min γ x ij τ γ x ij τ φ( x ij p ) x ij j=1 j=1 j=1 n min γ z τ γ z τ φ( z p ). z j=1 (36) (37)

15 W. Bian and X. Chen: Opimaliy and complexiy 15 φ 1 φ 4 φ 5 φ 6 p = 1 τ 1 none (0, λ) (λ, aλ) (0, aλ) τ 2 none (τ 1, λ) (τ 1, aλ) (τ 1, aλ) 0 < p < 1 τ 1 (0, ) (0, λ) (λ, ) (0, λ) (λ, aλ) (aλ, ) (0, aλ) (aλ, ) τ 2 (τ 1, ) (τ 1, λ) (τ 1, ) (τ 1, λ) (τ 1, aλ) (τ 1, ) (τ 1, aλ) (τ 1, ) Table 2. Parameers for differen poenial funcions in Remark 3 By Lemmas 5-6 and he sric concaviy of φ(z p ) on [τ 1, τ 2 ], we can always choose one of x ij o be z ( 0) and he ohers are 0 for any i = 1, 2,..., n such ha he las inequaliy in (37) becomes o be an equaliy and P (x) ng(z ). Now we claim ha here exiss an equiable pariion o he pariion problem if and only if he opimal value of (36) equals o ng(z ). Firs, if S can be evenly pariioned ino m ses, hen we define x ik = z, x ij = 0 for j k if a i belongs o S k. These x ij provide an opimal soluion o P (x) wih opimal value ng(z ). On he oher hand, if he opimal value of P (x) is ng(z ), hen in he opimal soluion, for each i, here is only one elemen in {x ij : 1 j m} is nonzero. And we mus also have n α ix ij β = 0 holds for any 1 j m, which implies ha here exiss a pariion o se S ino m disjoin subses such ha he sum of he numbers in each subse is equal. Thus his heorem is proved. Remark 3. Many penaly funcions saisfy he condiions in Lemma 5 and Lemma 6, such as he logisic penaly funcion in Nikolova e al. [36], fracion penaly funcion in Nikolova e al. [36], hard hresholding penaly funcion in Fan [21], SCAD funcion in Fan and Li [22] and MCP funcion in Zhang [44]. The sof hresholding penaly funcion in Huang e al. [27], Tibshirani [40] only saisfies he condiions in Lemma 6. Here, we lis he formulaions of hese penaly funcions below. For φ 2 and φ 3, all choices of τ 1 and τ 2 in R ++ wih τ 1 < τ 2 saisfy he condiions in Lemma 5 and Lemma 6. For he oher four penaly funcions, he opional parameers of τ 1 and τ 2 are given in Table 2. sof hresholding penaly funcion: φ 1 (s) = λs, logisic penaly funcion : φ 2 (s) = λ log(1 + as), fracion penaly funcion: φ 3 (s) = λ as, 1+as hard hresholding penaly funcion: φ 4 (s) = λ 2 (λ s) 2 +, smoohly clipped absolue deviaion (SCAD) penaly funcion: φ 5 (s) = λ s minimax concave penaly (MCP) funcion: wih λ > 0 and a > 0. 0 φ 6 (s) = λ min{1, (a /λ) + }d, a 1 s 0 (1 aλ ) +d, 3.2. Lower bound heory In his subsecion, we will esablish he lower bound heory for local minimizers of (2) wih a special consrain, ha is m min f(x) := Θ(x) + φ( D T i x p p) s.. x X = {x : Ax b}, (38)

16 16 W. Bian and X. Chen: Opimaliy and complexiy where D i = (D i1,..., D ir ) wih D ij R n, j = 1, 2,..., r, A = (A 1,..., A q ) T R q n wih A i R n, i = 1, 2,..., q, and b = (b 1, b 2,..., b q ) T R q. Denoe M he se of all local minimizers of (38). In his subsecion, we suppose ha here exiss β > 0 such ha sup x M 2 Θ(x) 2 β. For x X, le I ac (x) = {i {1, 2,..., q} : A T i x b i = 0} be he se of acive inequaliy consrains a x. Theorem 4. Le p = 1 in (38). There exis consans θ > 0 and ν 1 > 0 such ha if φ (0+) > ν 1, hen any local minimizer x of (38) saisfies eiher D T i x 1 = 0 or D T i x 1 θ, i {1, 2,..., m}. Proof. We divide M ino he finie disjoin ses M 1, M 2,..., M s such ha all elemen x in each se have he same following values: (i) sign values sign(d T ix) for i = 1, 2,..., m, = 1, 2,..., r; (ii) index values I ac (x) and I x = {i {1, 2,..., m} : D T i x = 0}. Firs, we will prove ha here exis θ 1,1 > 0 and κ 1,1 such ha eiher D T 1 x 1 = 0 or D T 1 x 1 θ 1,1, x M 1, (39) when φ (0+) > βκ 1,1. Specially, if he values of D1 T x 1 are same for all x M 1, hen he saemen in (39) holds naurally. In wha follows, we suppose ha here are a leas wo elemens in M 1 wih differen values of D1 T x 1. Suppose x M 1 is a local minimizer of minimizaion problem (38) saisfying D1 T x = 0. Then, here exiss δ > 0 such ha m f( x) = min{θ(x) + φ( D T i x 1 ) : x x 2 δ, Ax b} = min{θ(x) + i I x φ( D T i x 1 ) : x x 2 δ, Ax b, D T i x = 0 for i I x }, which implies ha x is a local minimizer of he following consrained minimizaion problem min f x (x) := Θ(x) + i I x φ( D T i x 1 ) s.. Ax b, D T i x = 0, i I x. (40) Since φ is locally Lipschiz coninuous in R ++, by he second order opimaliy necessary condiion, here exiss ξ i (φ (s)) s= D T i x 1 such ha where v T 2 Θ( x)v + ξ i i I x {1,2,...,r} sign(d T i x)d iv T 2 0, v V x, (41) V x = {v : D T j v = 0 for j I x and A T k v = 0 for k I ac ( x)}. (42) By ξ i 0, i = 1, 2,..., m, (41) gives ξ 1 2 sign(d1 x)d T 1v T 2 Θ( x) 2 v 2 2, v V x. (43) {1,2,...,r}

17 W. Bian and X. Chen: Opimaliy and complexiy 17 Fix x M 1. For c R r wih c i { 1, 0, 1}, consider he following consrained convex minimizaion problem min v 2 2 s.. v V x,c = {v : c D1v T = 1 and v V x }. (44) {1,...,r} When V x,c, unique exisence of he opimal soluion of (44) is guaraneed, denoed by v x,c. Take all possible choices of nonzero vecors c R r wih c i { 1, 0, 1} such ha V x,c, which are finie, and we define κ 1,1 = max v x,c 2 2, which is a posiive number and same for all elemens in M 1 from he decomposiion mehod for M. Since here is anoher elemen in M 1, denoed as ˆx, such ha D T 1 x 1 D T 1 ˆx 1, hen ṽ = 1 D T 1 x 1 D T 1 ˆx 1 ( x ˆx) V x,c. Thus, he unique soluion of (44) exiss in his case and (43) holds wih i, which follows If φ (0+) > βκ 1,1, le ξ 1 βκ 1,1. θ 1,1 = inf{ > 0 : φ () exiss and φ () βκ 1,1 }, (45) by he upper semiconinuiy of (φ ())) on R ++, we obain ha D T 1 x 1 θ 1,1. By he randomiciy of x M 1 saisfying D1 T x 1 0 in he above analysis, (43) implies ξ 1 2 sign(d1x)d T 1v T 2 Θ(x) 2 v 2 2, v V x, {1,2,...,r} holds for any x M 1 saisfying D T 1 x 1 0. Since κ 1,1 is same for all elemens in M 1, he saemen in (39) holds. Similarly, for any i = 1,..., m, j = 1,..., s, here exis θ i,j > 0 and κ i,j > 0 such ha eiher D T i x 1 = 0 or D T i x 1 θ i,j, x M j, when φ (0+) > βκ i,j. Therefore, we can complee he proof for his heorem wih ν 1 = max{βκ i,j : i = 1,..., m, j = 1,..., s} and θ = min{θ i,j : i = 1,..., m, j = 1,..., s}. If here exiss consan ν 1 > 0 such ha φ (0+) ν 1, by he concaviy of φ and φ 0, here mus exis ν p > 0 such ha φ (0+) ν p. However, he converse does no hold. The following heorem presens he lower bound heory for he case ha 0 < p < 1 using he exisence of ν p > 0 such ha φ (0+) ν p. Theorem 5. Le 0 < p < 1 in (38). If here exiss ν p > 0 such ha φ (0+) ν p, hen here exiss a consan θ > 0 such ha any local minimizer x of (38) saisfies eiher D T i x p = 0 or D T i x p θ, i {1, 2,..., m}.

18 18 W. Bian and X. Chen: Opimaliy and complexiy Proof. We divide M by he mehod in Theorem 4 and we will also prove ha here exiss θ 1,1 > 0 such ha eiher D T 1 x p = 0 or D T 1 x p θ 1,1, x M 1. (46) Specially, if he values of D T 1 x p are same for all x M 1, hen he saemen in (46) holds naurally. In wha follows, we also suppose ha here are a leas wo elemens in M 1 wih differen values of D T 1 x p. Similar o he analysis in Theorem 4, x is a local minimizer of minimizaion problem (38) saisfying D T 1 x p 0 implies ha x is a local minimizer of he minimizaion problem min f x (x) := Θ(x) + i I x φ( D T i x p p) s.. Ax b, D T i x = 0, i I x. (47) By he second order opimaliy necessary condiion for he minimizers of (47), here exiss ξ i (φ (s)) s= D T i x p such ha p v T 2 Θ( x)v + i I x + i I x p(p 1)φ (s) s= D T i x p p ξ i ( Ti p D T i x p 1 sign(d T i x)d T iv ( T i D T i x p 2 (D T iv) 2 ) 2 ) 0, v V x, where V x is same as in (42) and T i = { {1, 2,..., r} : Di T x 0}, i = 1, 2,..., m. Then, by ξ i 0, i = 1, 2,..., m and D1 T x p 0, we obain ( ) p(1 p)φ (s) s= D T 1 x p D p 1 x T p 2 (D1v) T 2 v T 2 Θ( x)v, v V x. (48) T i Fix x M 1. For T 1, consider he following consrained convex opimizaion min v 2 2 s.. v V x, = {v : D T 1v = 1 and v V x }. (49) When V x,, unique exisence of he opimal soluion of (49) is guaraneed, denoed by v x,. Take all possible choices of T 1 such ha V x,c, which are finie, and we define κ 1,1 = max v x, 2 2, which is also a posiive number same for all elemens in M 1. Since here is anoher elemen in M 1, denoed as ˆx, such ha D T 1 x p D T 1 ˆx p. Then, here exiss 1 {1, 2,..., r} such ha D T 1 1 x D T 1 1 ˆx. Thus, ṽ = 1 D T 1 1 x D T 1 1 ˆx ( x ˆx) V x, 1, which follows he exisence of he unique soluion of (49) exiss wih = 1, denoed as v x, 1. By he decomposiion mehod for M, we have sign(d1 x) T = sign(d1ˆx), T which implies 1 T 1. Le v = v x, 1 in (48), by φ 0, we have p(1 p)φ (s) s= D T 1 x p p DT 1 1 x p 2 βκ 1,1. (50)

19 W. Bian and X. Chen: Opimaliy and complexiy 19 D1 T 1 x D1 T x p implies D1 T 1 x p 2 D1 T x p 2 p, hen (50) gives p(1 p)φ (s) s= D T 1 x p p DT 1 x p 2 p βκ 1,1. (51) By he concaviy of φ, lim φ (s) s= p p 2 lim φ (1) p 2 = 0. From φ (0+) ν 2, lim 0 φ ( p ) p 2 = +. Le θ 1,1 = inf{ > 0 : φ ( p ) p 2 = βκ 1,1 p(1 p) }, which is an exisen number larger han 0. Therefore, (51) implies D T 1 x p θ 1,1. Similar o he analysis in Theorem 4, he saemen in his heorem holds. Remark 4. For he oher cases, such as he regularizaion erm is given by m φ i(max{d T i x, 0} p ) wih d i R n, he lower bound heory in Theorems 4-5 can also be guaraneed under he same condiions. Moreover, he lower bound heories in Theorems 4-5 can also be exended o he more general case wih he objecive funcion f(x) := Θ(x) + m φ i( D T i x p p). All he poenial funcions in Remark 3 saisfy he condiions in Theorem 5, bu only φ 2, φ 3, φ 4 and φ 6 may mee he condiions in Theorem 4 under some condiions on he parameers, which shows he superioriy of he non-lipschiz regularizaion in sparse reconsrucion. While our paper [7](2014) was under review, we became aware of an independen line of relaed work on compuaional complexiy by Ge e al. [26](2015). Our conribuion is differen in ha we show ha he concaviy of penaly funcions is a key propery no only for he srong NP hardness bu also for he nice lower bound heory. 4. Conclusions In Theorem 1, we derive a firs order necessary opimaliy condiion for local minimizers of problem (1) based on he new generalized direcional derivaive (12) and he Clarke angen cone. The generalized saionary poin ha saisfies he firs order necessary opimaliy condiion is a Clarke saionary poin when he objecive funcion f is locally Lipschiz coninuous near his poin, and a scaled saionary poin if f is non-lipschiz a he poin. Moreover, in Theorem 2 we esablish he direcional derivaive consisency associaed wih smoohing funcions and in Corollary 1 we show ha he consisency guaranees he convergence of smoohing algorihms o a saionary poin of problem (1). Compuaional complexiy and lower bound heory of problem (1) are also sudied o illusrae he negaive and posiive news of he concave penaly funcion in applicaions. Acknowledgmens. The work in he presen paper was suppored by he NSF foundaion ( , ) of China, HIT.BRETIII , PIRS of HIT No.A201402, and parly by Hong Kong Research Gran Council gran (PolyU5001/12P) References [1] Aubin JP, Cellina A (1984) Differenial Inclusion: Se-Valued Maps and Viabiliy Theory (Springer- Verlag, Berlin) [2] Aude C, Dennis Jr. JE (2006) Mesh adapive direc search algorihms for consrained opimizaion. SIAM J. Opim. 17: [3] Auslender A (1997) How o deal wih he unbounded in opimizaion: heory and algorihms. Mah. Program. 79: 3-18 [4] Beck A, Teboulle M (2012) Smoohing and firs order mehods: a unified framework. SIAM J. Opim. 22:

20 20 W. Bian and X. Chen: Opimaliy and complexiy [5] Bersekas DP (1976) On he Goldsein-Leviin-Polyak gradien projecion mehod. IEEE Trans. Auom. Conrol AC-21: [6] Bian W, Chen X (2013) Wors-case complexiy of smoohing quadraic regularizaion mehods for non- Lipschizian opimizaion. SIAM J. Opim. 23: [7] Bian W, Chen X (2014) Opimaliy condiions and complexiy for non-lipschiz consrained opimizaion problems. Preprin, Deparmen of Applied Mahemaics, The Hong Kong Polyechnic Universiy [8] Bian W, Chen X, Ye Y (2015) Complexiy analysis of inerior poin algorihms for non-lipschiz and nonconvex minimizaion. Mah. Program. 149: [9] Brucksein AM, Donoho DL, Elad M (2009) From sparse soluions of sysems of equaions o sparse modeling of signals and images. SIAM Review 51: [10] Burke JV, Hoheisel T (2013) Epi-convergen smoohing wih applicaions o convex composie funcions. SIAM J. Opim. 23: [11] Burke JV, Hoheisel T, Kanzow C (2013) Gradien consisency for inegral-convoluion smoohing. Se-Valued Var. Anal. 21: [12] Chan RH and Liang HX (2014) Half-quadraic algorihm for l p -l q problems wih applicaions o TV-l 1 image resoraion and compressive sensing. Springer Lecure Noes in Compuer Science 8293: [13] Charrand R, Saneva V (2008) Resriced isomery properies and nonconvex compressive sensing. Inverse Probl. 24: 1-14 [14] Chen X (2012) Smoohing mehods for nonsmooh, nonconvex minimizaion. Mah. Program. 134: [15] Chen X, Ge D, Wang Z, Ye Y (2014) Complexiy of unconsrained L 2 -L p minimizaion. Mah. Program. 143: [16] Chen X, Niu L, Yuan Y (2013) Opimaliy condiions and smoohing rus region Newon mehod for non-lipschiz opimizaion. SIAM J. Opim. 23: [17] Chen X, Ng M, Zhang C (2012) Nonconvex l p regularizaion and box consrained model for image resoraion. IEEE Trans. Image Processing 21: [18] Chen X, Xu F, Ye Y (2010) Lower bound heory of nonzero enries in soluions of l 2 -l p minimizaion. SIAM J. Sci. Compu. 32: [19] Clarke FH (1983) Opimizaion and Nonsmooh Analysis (John Wiley, New York) [20] Curis FE, Overon ML (2012) A sequenial quadraic programming algorihm for nonconvex, nonsmooh consrained opimizaion. SIAM J. Opim. 22: [21] Fan J (1997) Commens on Waveles in saisics: a review by A. Anoniadis. Sa. Mehod. Appl. 6: [22] Fan J, Li R (2001) Variable selecion via nonconcave penalized likelihood and is oracle properies. J. Amer. Sais. Assoc. 96: [23] Fan J, Peng H (2004) Nonconcave penalized likelihood wih a diverging number of parameers. Ann. Sa. 32: [24] Garey MR, Johnson DS (1979) Compuers and Inracabiliy: A Guide o he Theory of NP- Compleeness (W. H. Freeman & Co Ld, New York) [25] Ge D, Jiang X, Ye Y (2011) A noe on he complexiy of L p minimizaion. Mah. Program. 21: [26] Ge D, Wang Z, Ye Y, Yin, H (2015) Srong NP-hardness resul for regularized L q -minimizaion problems wih concave penaly funcions. Preprin. [27] Huang J, Horowiz JL, Ma S (2008) Asympoic properies of bridge esimaors in sparse highdimensional regression models. Ann. Sais. 36: [28] Huber P (1981) Robus Esimaion (Wiley, New York) [29] Huang J, Ma S, Xue H, Zhang C (2009) A group bridge approach for variable selecion. Biomerika 96:

21 W. Bian and X. Chen: Opimaliy and complexiy 21 [30] Jahn J (1996) Inroducion o he Nonlinear Opimizaion (Springer-Verlag, Berlin) [31] Knigh K, Fu WJ (2000) Asympoics for lasso-ype esimaors. Ann. Sa. 28: [32] Leviin ES, Polyak BT (1966) Consrained minimizaion problems. USSR. Compu. Mah. Mah. Phys. 6: 1-50 [33] Liu YF, Dai YH, Ma S (2013) Join power and admission conrol via linear programming deflaion. IEEE Trans. Signal Processing 61: [34] Loh P and Wainwrigh MJ (2014) Regularized M-esimaors wih nonconvexiy: saisical and algorihmic heory for local opima. J. Mach. Learn. Res. 1: 1-56 [35] Lu Z (2014) Ieraive reweighed minimizaion mehods for l p regularized unconsrained nonlinear programming. Mah. Program. 147: [36] Nikolova M, Ng MK, Zhang S, Ching WK (2008) Efficien reconsrucion of piecewise consan images using nonsmooh nonconvex minimizaion. SIAM J. Imaging Sci. 1: 2-25 [37] Nocedal J, Wrigh SJ (2006) Numerical Opimizaion (Springer, New York) [38] Rockafellar RT, Wes R J-B (1998) Variaional Analysis (Springer, Berlin) [39] Spingarn JE, Rockafellar RT (1979) The genaric naure of opimaliy condiions in nonlinear programming. Mah. Oper. Res. 4: [40] Tibshirani R (1996) Shrinkage and selecion via he Lasso. J. Roy. Sais. Soc. Ser. B 58: [41] Vazirani V (2003) Approximaion Algorihms (Springer, Berlin) [42] Wang Z, Liu H, Zhang T (2014) Opimal compuaional and saisical raes of convergence for sparse nonconvex learning problems. Ann. Sa. 42: [43] Ye Y (1997) Inerior Poin Algorihms: Theory and Analysis (John Wiley & Sons, Inc., New York) [44] Zhang CH (2010) Nearly unbiased variable selecion under minimax concave penaly. Ann. Sais. 38: