Optimality and complexity for constrained optimization problems with nonconvex regularization

Transcription

1 Opimaliy and complexiy for consrained opimizaion problems wih nonconvex regularizaion Wei Bian Deparmen of Mahemaics, Harbin Insiue of Technology, Harbin, China, Xiaojun Chen Deparmen of Applied Mahemaics, The Hong Kong Polyechnic Universiy, Hong Kong, China, In his paper, we consider a class of consrained opimizaion problems where he feasible se is a general closed convex se and he objecive funcion has a nonsmooh, nonconvex regularizer. Such regularizer includes widely used SCAD, MCP, logisic, fracion, hard hresholding and non-lipschiz L p penalies as special cases. Using he heory of he generalized direcional derivaive and he angen cone, we derive a firs order necessary opimaliy condiion for local minimizers of he problem, and define he generalized saionary poin of i. We show ha he generalized saionary poin is he Clarke saionary poin when he objecive funcion is Lipschiz coninuous a his poin, and saisfies he exising necessary opimaliy condiions when he objecive funcion is no Lipschiz coninuous a his poin. Moreover, we prove he consisency beween he generalized direcional derivaive and he limi of he classic direcional derivaives associaed wih he smoohing funcion. Finally, we esablish a lower bound propery for every local minimizer and show ha finding a global minimizer is srongly NP-hard when he objecive funcion has a concave regularizer. Key words : Consrained nonsmooh nonconvex opimizaion; opimaliy condiion; generalized direcional derivaive; direcional derivaive consisency; numerical propery MSC2000 subjec classificaion : Primary: 49K35, 90C26; secondary: 65K05, 90C46 1. Inroducion In his paper, we consider he following consrained opimizaion problem min x X f(x) := Θ(x) + c(h(x)), (1) where Θ : R n R and c : R m R are coninuously differeniable, h : R n R m is coninuous, and X R n is a nonempy closed convex se. Of paricular ineres of his paper is when h is no convex, no differeniable, or even no Lipschiz coninuous a some poins, and f has a leas one local minimizer over X. Problem (1) includes many problems in pracice. For insance, he following minimizaion problem min l x u,ax b f(x) := Θ(x) + m ϕ( D T i x p p) (2) is a special case of (1), where l (R { }) n, u (R { }) n, A R n, b R, D i R n r, p (0, 1] and ϕ : R + R + is coninuous. Such problem arises from image resoraion (Chan and Liang [16], Chen e al. [22], Nikolova e al. [44]), signal processing (Brucksein e al. [12]), variable selecion (Fan and Li [27], Huang e al. [33], Huang e al. [35], Zhang [55]), ec. Anoher special case of (1) is he following problem min x X m f(x) := Θ(x) + ϕ(max{α i d T i x, 0} p ), (3) 1

2 2 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion wih α i R and d i R n, which has araced much ineres in machine learning, wireless communicaion (Liu e al. [39, 40]), informaion heory, daa analysis (Fan and Peng [28], Huber [34]), ec. Moreover, a number of consrained opimizaion problems can be reformulaed as problem (1) by using he exac penaly mehod wih nonsmooh or non-lipschiz coninuous penaly funcions (Auslender [3]). The generic naure of he firs and second order opimaliy condiions in nonlinear programming are reaed by Spingarn and Rockafellar [49]. When f is locally Lipschiz coninuous and X = R n, x is called a Clarke saionary poin of (1) if f (x ; v) 0, v R n, (4) where f (x ; v) is he Clarke generalized direcional derivaive of f a x in direcion v (Clarke [24]), defined by f (x f(y + v) f(y) ; v) = lim sup. (5) y x, 0 From he following relaion beween he Clarke generalized direcional derivaive and Clarke subdifferenial (Clarke [24]) f(x) := {s R n : f (x; v) v T s, v R n } and f (x; v) = max{v T s : s f(x)}, condiion (4) is equivalen o 0 f(x ). For he consrained opimizaion, a Clarke saionary poin of locally Lipschiz coninuous funcion f over X is defined by he exisence of ξ f(x ) saisfying ξ, x x 0, x X. (6) Then, he Clarke generalized direcional derivaive in (5) is generalized by Jahn [36] and used in Aude and Dennis [2], Jahn [36] for Lipschiz consrained opimizaion problem, defined by f (x ; v; X ) = lim sup y x, y X 0, y + v X f(y + v) f(y). (7) Based on he direcional derivaive in (7), Jahn [36, Theorem 3.46] presened a necessary opimaliy condiion for a local minimizer x of f over X, i.e. f (x ; x x ; X ) 0, x X. (8) When in(t X (x )), he necessary opimaliy condiions in (6) and (8) are equivalen o f (x ; v; X ) 0, v T X (x ), (9) where T X (x ) is he angen cone o X a x. Due o he non-lipschiz coninuiy of he objecive funcion f in (1), he Clarke opimaliy condiions in (6), (8) or (9) canno be direcly applied o problem (1). For a proper, lower semi-coninuous funcion f : R n R (possibly non-lipschiz), he limiing (Mordukhovich) subdifferenial and horizon subdifferenial (Rockafellar and Wes [47]) are defined respecively as f(x) = {v : x k f x, v k v wih lim inf z x k f(z) f(xk ) v k,z x k z x k 0, k}, f(x) = {v : x k f x, λ k v k v, λ k 0 wih lim inf z x k f(z) f(xk ) v k,z x k z x k 0, k},

3 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion 3 where λ k 0 means λ k > 0 and λ k 0, and x k following consrain qualificaion f x means x k x and f(x k ) f(x). Based on he for x o be a local minimizer of (1) i is necessary ha f( x) N X ( x) = {0}, (10) 0 f( x) + N X ( x) (11) (Rockafellar and Wes [47, Theorem 8.15]). The consrain qualificaion (10) holds naurally if f is locally Lipschiz coninuous a x or x is an inerior poin of X (can be non-lipschiz coninuous a x). When X = R n and c(h(x)) := x p p (0 < p < 1), he affine scaled firs and second order necessary opimaliy condiions for local minimizers of (1) are esablished in Chen e al. [23]. By using subspace echniques, Chen e al. [21] exended he firs and second order necessary opimaliy condiions o problem (1) wih X = R n and c(h(x)) := Dx p p. Some oher necessary opimaliy condiions for some special formas of (2) and (3) are sudied in Bian and Chen [6, 8], Bian e al. [9], Ge e al. [30], Liu e al. [40]. However, he opimaliy condiions in Bian and Chen [6, 8], Bian e al. [9], Chen e al. [21, 23], Ge e al. [30], Liu e al. [40] are weaker han he Clarke opimaliy condiions given in (6), (8) and (9) for p = 1. The minimizers of problem (1) in pracical problems are ofen he non-lipschiz poin and on he boundary of X, which may lead o he unsaisfacion of qualiy (10). In his paper, we will derive a necessary opimaliy condiion for he non-lipschiz consrained opimizaion problem (1), which holds wihou he consrain qualificaion (10) and reduces o he Clarke opimaliy condiion when he objecive funcion in (1) is locally Lipschiz coninuous a his poin. When f is locally Lipschiz coninuous, from Theorem 9.61 and Corollary 8.47 (b) in Rockafellar and Wes [47], he subdifferenial associaed wih a smoohing funcion G f(x) = con{v x f(x k, µ k ) v, for x k x, µ k 0 }, is nonempy and bounded, and f(x) G f(x), where con denoes he convex hull. In Burke and Hoheisel [13], Burke e al. [14], Chen [19], Rockafellar and Wes [47], i is shown ha many smoohing funcions saisfy he gradien consisency f(x) = G f(x). (12) The gradien consisency is an imporan propery of he smoohing mehods, which guaranees he convergence of smoohing mehods wih adapive updaing schemes of smoohing parameers o a saionary poin of he original problem. In his paper, we exend he direcional derivaive in Jahn [36] o he consrained opimizaion problem (1), whose objecive funcion may no be Lipschiz coninuous a some poins. Using he exended direcional derivaive and he angen cone, we derive a necessary opimaliy condiion for local minimizers of problem (1), and define he generalized saionary poin of (1). We show ha he generalized saionary poin is he Clarke saionary poin when he objecive funcion is Lipschiz coninuous a his poin defined in (6), (8) and (9), and saisfies he exising necessary opimaliy condiions when he objecive funcion is no Lipschiz coninuous a his poin defined in Bian and Chen [6, 8], Bian e al. [9], Chen e al. [21, 23], Ge e al. [30], Liu e al. [40]. Moreover, we esablish he consisency beween he generalized direcional derivaive and he limi of he classic direcional derivaives associaed wih he smoohing funcion. The direcional derivaive consisency guaranees he convergence of smoohing mehods o a generalized saionary poin of (1).

4 4 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion Problem (1) includes he regularized minimizaion problem as a special case when Θ(x) is a daa fiing erm and c(h(x)) is a regularizaion erm (also called a penaly erm in some aricles). In sparse opimizaion, nonconvex non-lipschiz regularizaion provides more efficien models o exrac he essenial feaures of soluions han he convex regularizaion (Bian and Chen [6], Charrand and Saneva [17], Chen [19], Chen e al. [22], Fan and Li [27], Huang e al. [33], Huang e al. [35], Loh and Wainwrigh [41], Lu [42], Nikolova e al. [44], Wang e al. [53], Zhang [55]). The SCAD penaly funcion in Fan and Li [27] and he MCP funcion in Zhang [55] have various desirable properies in variable selecion. Logisic and fracion penaly funcions yield edge preservaion in image resoraion (Nikolova e al. [44]). The l p norm penaly funcion wih 0 < p < 1 possesses he oracle propery in saisics (Fan and Li [27], Knigh and Fu [37]). Nonconvex regularized M-esimaor is proved o have he saisical accuracy and predicion error esimaion in Loh and Wainwrigh [41]. Moreover, he lower bound heory of he l 2 -l p regularized minimizaion problem in Chen e al. [22, 23], a special case of (1), saes ha he absolue value of each componen of any local minimizer of he problem is eiher zero or greaer han a posiive consan. The lower bound heory no only helps us o disinguish zero and nonzero enries of coefficiens in sparse high-dimensional approximaion (Charrand and Saneva [17], Huang e al. [33]), bu also brings he resored image closed conours and nea edges (Chen e al. [22]). In his paper, we exend he lower bound heory of he l 2 -l p regularizaion minimizaion problem o problem (1) wih consrains {x : Ax b}, which includes he mos widely used models in saisics and sparse reconsrucion as special cases. From he new bound heory, we can derive many ineresing lower and upper bound resuls for differen special problems. Moreover, we prove he srong NP-hardness of problem (1) via a special model of i, which generalizes he compuaional complexiy resuls in Chen e al. [20], Ge e al. [30], Liu e al. [40]. Such exensions are no rivial because of he general consrains in problem (1). We show ha he concaviy of he regularizaion erm is a key propery for boh he lower bound heory and he srong NP hardness of (1). The bound propery gives he posiive news of problem (1) in applicaions, while he srong NP-hardness indicaes is negaive aspec in numerical compuaion, which furher illusraes he imporance and necessiy for presening good necessary opimaliy condiions of (1). The res of his paper is organized as follows. In secion 2, we firs define a generalized direcional derivaive and presen is properies. Nex, we derive a necessary opimaliy condiion for he local minimizers of problem (1), and prove he direcional derivaive consisency associaed wih smoohing funcions. In secion 3, we presen he numerical properies of problem (1) wih a special consrain from he bound propery of is local minimizers and is compuaional complexiy. In our noaion, R + = [0, ), R ++ = (0, ) and N = {1, 2,...}. For x R n, 0 < p < and δ > 0, x p p = n x i p, B δ (x) means he open ball cenered a x wih radius δ. For a se Ω R n, in(ω) means he inerior of Ω, cl(ω) means he closure of Ω, Ω sands for is cardinaliy and and P Ω [x] = arg min{ z x 2 : z Ω} denoes he orhogonal projecion from R n o Ω. For locally Lipschiz coninuous funcion φ : R n R, φ (s+) and φ (s ) indicae he derivaive of φ a s on he righ hand side and lef hand side, respecively. For Π consised by a class of column vecors of R n, spanπ indicaes he subspace of R n spanned by he elemens in Π. 2. Opimaliy condiions Inspired by he generalized direcional derivaive and he angen cone, we presen a firs order necessary opimaliy condiion for local minimizers of he consrained opimizaion problem (1), which reduces o he Clarke necessary opimaliy condiion a Lipschiz coninuous poin and o he necessary opimaliy condiions in he exising lieraures (Bian and Chen [6, 8], Bian e al. [9], Chen e al. [21, 23], Ge e al. [30], Liu e al. [40]) a non-lipschiz poins. Thus, he generalized saionary poin based on he derived necessary opimaliy condiion provides a unified form for he saionary poins of problem (1) wih a coninuous objecive funcion. A he end of his secion, we prove he direcional direcive consisency associaed wih smoohing

5 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion 5 funcions, which gives some hins on how o find a generalized saionary poins of (1) in numerical compuaion. We suppose he funcion h in (1) has he following represenaion h(x) := (h 1 (D T 1 x), h 2 (D T 2 x),..., h m (D T mx)) T (13) where D i R n r, h i (i = 1,..., m) : R r R is coninuous, bu no necessarily Lipschiz coninuous Tangen cone Since X is a nonempy closed convex subse of R n, he angen cone o X a x X, denoed as T X (x), is he se consising of all angen vecors [47, Proposiion 6.2], where we call a vecor v R n a angen vecor o X a x, if here are a sequence {x k } of elemens in X converging o x and a sequence {λ k } of posiive numbers converging o 0 such ha v = lim k x k x λ k. Assumpion 1. Assume ha X can be expressed by X = X 1 X 2 wih a nonempy closed convex se X 1, X 2 := {x : Ax b} and in(x 1 ) X 2, where A R n and b R. Denoe A T i and b i he ih row of A and b, and le C x = {i : A T i x b i = 0} for x X 2. Under Assumpion 1, we can obain he following properies of he angen cones o X 1, X 2 and X. Lemma 1. Suppose Assumpion 1 holds. Then he following saemens hold. (1) T X2 (x) = {v : A T i v 0, i C x }, x X 2 ; (2) in(t X1 (x)) T X2 (x), x X ; (3) T X1 X 2 (x) = T X1 (x) T X2 (x), x X. Proof. (1) and (3) can be obained by Borwein and Lewis [10, Corollary 6.3.7] and Rockafellar and Wes [47, Theorem 6.42] respecively. Fix x X. By in(x 1 ) and from Rockafellar and Wes [47, Example 6.22,Example 6.24], in(t X1 ( x)) and ˆx x in(t X1 ( x)) wih ˆx in(x 1 ). Using Assumpion 1 and (1), we ge ˆx x in(t X1 ( x)) T X2 ( x) wih ˆx in(x 1 ) X Generalized direcional derivaive Definiion 1. A funcion φ : R n R is said o be Lipschiz coninuous a(near) x R n if here exis posiive numbers L x and δ such ha φ(y) φ(z) L x y z 2, y, z B δ (x). Oherwise, φ is said o be no Lipschiz coninuous a x. For a fixed x R n, denoe and define I x = {i {1, 2,..., m} : h i is no Lipschiz coninuous a D T i x}, (14) V x = {v : D T i v = 0, i I x }, (15) { h x i (D T hi (D T i x) i I x i x) := h i (D T i x) i I x, which is Lipschiz coninuous a D T i x, i = 1, 2,..., m. In paricular, we le V x = R n when I x =. And hen we define f x (x) := Θ(x) + c(h x (x)), (16)

6 6 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion wih h x (x) := (h x 1(D T 1 x), h x 2(D T 2 x),..., h x m(d T mx)) T. We noice ha he generalized direcional derivaive of f a x X in direcion v defined in (7) involves only he behavior of f around x in X. Moreover, f ( x; v; X ) exiss for any v T X ( x) if f is Lipschiz coninuous a x. In paricular, when x in(x ), f ( x; v) = f ( x; v; X ). However, he objecive funcion f in (1) may no be Lipschiz coninuous a some poins, which implies ha f ( x; v; X ) may no exis for some x X and v T X ( x). So, we consider he generalized direcional derivaive of f over X in direcions v T X ( x) V x when f is no Lipschiz coninuous a x. There are various generalized direcional derivaives, such as he lower subderivaive d f(x)(v), upper subderivaive d + f(x)(v) and regular subderivaive ˆdf(x)(v) for he exended real-valued funcion defined in Rockafellar and Wes [47] and he Clarke direcional derivaive in (7) for realvalued funcion. When f(x 1, x 2 ) = x 1 1 x 2 and X = R n, which is an example modeled by he objecive funcion in (1), f ( x; v; X ) = 1 2, bu d+ f( x)(v) = + for x = (1, 1) and v = (0, 1) V x ; f ( x; v; X ) = 1 2, bu d f( x)(v) = ˆdf( x)(v) = for x = (2, 0) and v = (1, 0) V x. This moivaes us o use he Clarke generalized direcional derivaive wih he consrains in (7) for problem (1) and we will prove he exisence of f ( x; v; X ) for any x X and v T X ( x) V x. Proposiion 1. For any x X and v T X ( x) V x, f ( x; v; X ) = lim sup y x, y X 0, y + v X f(y + v) f(y) exiss (17) and equals f x( x; v; X ) defined in (7). Proof. Fix x X and v V x. For y R n and > 0, here exiss z beween h(y) and h(y + v) such ha c(h(y + v)) c(h(y)) = c(z) T (h(y + v) h(y)) = c(z) T (h x (y + v) h x (y)). Then, f(y + v) f(y) = Θ(y + v) Θ(y) + c(z)t (h x (y + v) h x (y)). By he Lipschiz coninuiy of Θ and h x a x, here exis δ > 0 and L > 0 such ha f(y+v) f(y) L, y B δ ( x), (0, δ). Thus, he generalized direcional derivaive of f a x X in he direcion v V x defined in (17) exiss. Le {y j } and { j } be he sequences such ha y j X, j 0, y j x, y j + j v X and f(y j + j v) f(y j ) lim = f ( x; v; X ). j j Using he Lipschiz coninuiy of h x a x again, we can ge he subsequences {y jk } {y j } and { jk } { j } such ha h x (y jk + jk v) h x (y jk ) lim exiss. (18) k jk By he above analysis, hen f(y f jk + jk v) f(y jk ) ( x; v; X ) = lim k jk = Θ( x) + c(z) T h x (y jk + jk v) h x (y jk ) z=h( x) lim. k jk (19)

7 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion 7 By virue of (17), we have f x( x; f x (y jk + jk v) f x (y jk ) v; X ) lim k jk = Θ( x) + c(z) T h x (y jk + jk v) h x (y jk ) z=h x ( x) lim. k jk (20) Using h( x) = h x ( x), (19) and (20), we obain f x( x; v; X ) f ( x; v; X ). On he oher hand, by exracing he sequences {y jk } and { jk } such ha he upper limi in (7) holds for f x and he limi in (18) exiss wih hem, similar o he above analysis, we find ha f ( x; v; X ) f x( x; v; X ). Therefore, f ( x; v; X ) = f x( x; v; X ) Necessary opimaliy condiion Denoe r-in(t X (x)) = in(t X1 (x)) T X2 (x). By Lemma 1 (2), r-in(t X (x)) is no empy for any x X. For a vecor v in(t X1 (x)), here exiss a scalar δ 1 > 0 such ha y + w X 1, for all y X 1 B δ1 (x), w B δ1 (v) and 0 < δ 1. We ofen call in(t X1 (x)) he hyperangen cone o X 1 a x. Whenever in(t X1 (x)), cl(in(t X1 (x))) = T X1 (x) and cl(r-in(t X (x))) = T X (x) (Rockafellar [48]). By Lemma 1 (1), for v T X2 (x), here is δ 2 > 0 such ha y +v X 2, y X 2 B δ2 (x), 0 < δ 2. Therefore, for any vecor v r-in(t X (x)), here exiss a scalar ɛ > 0 such ha y + v X, for all y X B ɛ (x), 0 < ɛ. (21) Since f in (1) may no be locally Lipschiz coninuous a some poins, he calculus heory developed in Aude and Dennis [2] canno be direcly applied. The nex lemma exends calculus resuls for he unconsrained case in Clarke [24] and he consrained case in Aude and Dennis [2]. Lemma 2. For x X and v T X ( x) V x, if r-in(t X ( x)) V x, hen f ( x; v; X ) = lim f ( x; w; X ). w v w r-in(t X ( x)) V x Proof. By he locally Lipschiz coninuiy of h x, here are ɛ > 0 and L x > 0 such ha h x (x) h x (y) 2 L x x y 2, x, y B ɛ ( x). (22) Le {w k } r-in(t X ( x)) V x be a sequence of direcions converging o a vecor v T X ( x) V x. By {w k } r-in(t X ( x)) and (21), here exiss ɛ k > 0 such ha x + w k X whenever x X B ɛk ( x) and 0 < ɛ k. Then, for all w k, i gives f ( x; v; X ) = lim sup x x, x X 0, x + v X = lim sup x x, x X 0, x + v X x + w k X = lim sup x x, x X 0, x + v X x + w k X f(x + v) f(x) f(x + v) f(x) f(x + w k ) f(x) + f(x + v) f(x + w k). (23)

8 8 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion Le δ > 0 be such ha x + w k B ɛ ( x) for any x B δ ( x), 0 < δ and k N. By he Lipschiz propery in (22), we have h x (x + v) h x (x + w k ) L x v w k 2, x B δ ( x), 0 < < δ, k N. 2 From he mean value heorem, here exiss z beween h(x + v) and h(x + w k ) such ha f(x + v) f(x + w k ) =Θ(x + v) Θ(x + w k ) + c(z) T (h(x + v) h(x + w k )) =Θ(x + v) Θ(x + w k ) + c(z) T (h x (x + v) h x (x + w k )). Then, for any x B δ ( x) and (0, δ), we have f(x + v) f(x + w k ) L Θ v w k 2 + L c L x v w k 2, where L Θ = sup{ Θ(y) 2 : y B ɛ ( x)} and L c = sup{ c(z) z=h(y) 2 : y B ɛ ( x)}. Thus, (23) implies f ( x; w k ; X ) L Θ v w k 2 L c L x v w k 2 f ( x; v; X ) f ( x; w k ; X ) + L Θ v w k 2 + L c L x v w k 2, k N. As k goes o infiniy, he above inequaliy gives f ( x; v; X ) = lim k f ( x; w k ; X ). Since {w k } is an arbirary sequence in r-in(t X ( x)) V x converging o v, we obain he resul in his lemma. Noe ha he limi in Lemma 2 is no necessarily rue when r-in(t X ( x)) is empy, even for he case V x = R n. A similar example can be given following he idea in Aude and Dennis [2, Example 3.10]. Some oher condiions on X o ensure ha he se consising of he vecors v saisfying (21) is no empy can be used o obain he limi in Lemma 2. A sufficien condiion for (21) is in(x 1 ) X 2. Based on Lemmas 1-2, he following heorem gives he main heoreical resul of his secion. Theorem 1. Suppose he funcion h in (1) has he form in (13) and Assumpion 1 holds for X. If x is a local minimizer of (1) and r-in(t X (x )) V x, hen f (x ; v; X ) 0 for every direcion v T X (x ) V x. Proof. Suppose x is a local minimizer of f over X and le w r-in(t X (x )) V x. Since w r-in(t X (x )), by (21), here exiss ɛ > 0 such ha x + w X, x X B ɛ (x ), 0 ɛ. And here exis ɛ (0, ɛ] and L x > 0 such ha f(x ) f(x), x X B ɛ (x ), and h x (x) h x (y) 2 L x x y 2, x, y X B ɛ (x ). (24) Then, we can choose δ (0, ɛ] such ha x, x+w, x +w B ɛ (x ) X, x B 2(x ) X, 0 < δ. By (24), for all x B 2(x ) X, 0 < < δ, we obain h x (x + w) h x (x + w) h x (x) h x (x ) x x 2 2L x 2L x. Thus, lim x B 2 (x ) X x + w X, 0 h x (x + w) h x (x + w) 2 h x (x) h x (x ) = 0. (25)

9 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion 9 From he mean value heorem, here exis z 1 beween h(x) and h(x + w), and z 2 beween h(x ) and h(x + w) such ha (c(h(x + w)) c(h(x))) (c(h(x + w)) c(h(x ))) = c(z 1 ) T (h(x + w) h(x)) c(z 2 ) T (h(x + w) h(x )) = c(z 1 ) T (h x (x + w) h x (x)) c(z 2 ) T (h x (x + w) h x (x )). (26) Using (25), (26) and he coninuous differeniabiliy of Θ, we have [ ] f(x + w) f(x) lim f(x + w) f(x ) x B 2 (x ) X x + w X, 0 [ = c(z) T hx (x + w) h x (x) z=h(x ) Thus, =0. lim x B 2 (x ) X x + w X, 0 h ] x (x + w) h x (x ) [ ] f(x + w) f(x) lim sup f(x + w) f(x ) 0. (27) x x, x X 0, x + w X By f(x + w) f(x ) 0 for 0 < ɛ, (27) implies f (x ; w; X ) = lim sup x x, x X 0, x + w X f(x + w) f(x) By Lemma 2, we can give ha f (x ; v; X ) 0 for any v T X (x ) V x. Based on Theorem 1, we give a new definiion of he generalized saionary poin of problem (1). Definiion 2. x X is said o be a generalized saionary poin of (1), if r-in(t X (x )) V x = or f (x ; v; X ) 0 for every v T X (x ) V x. T X (x) V x is nonempy for any x X, bu f (x ; v; X ) 0, v T X (x ) V x is no necessarily rue a he local minimizer x of f over X when r-in(t X (x )) V x =. For example, if f(x 1, x 2 ) = x x 2 1 and X = {(x 1, x 2 ) : (x 1 1) 2 + (x 2 2) 2 1}, hen f (x ; v; X ) = 1, where 2 x = (1, 1) T is he global minimizer of f over X and v = (1, 0) T T X (x ) V x. So Definiion 2 is reasonable and robus. We call f Lipschiz coninuous a x X in direcion v, if here exis L > 0 and ε > 0 such ha 0. f( x + v) f( x) L v 2, (0, ε). When Assumpion 1 holds, if f is no Lipschiz coninuous a x X in direcion x x for any x X, r-in(t X (x)) V x = or T X (x) V x = {0}, which implies ha x is a rivial generalized saionary poin of (1). Corollary 1. Suppose he funcion h in (1) has he form in (13) and Assumpion 1 holds. Then he following saemens hold. (1) When f is Lipschiz coninuous a x X, x is a generalized saionary poin of (1) defined in Definiion 2 if and only if i is a Clarke saionary poin of (1). (2) When X := {x : Ax b} wih A R n and b R, x is a generalized saionary poin of (1) defined in Definiion 2 if and only if f (x ; v; X ) 0, for every v T X (x ) V x.

10 10 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion Proof. When f is Lipschiz coninuous a x X, by r-in(t X (x)) and V x = R n, x is a generalized saionary poin of (1) if and only if f (x ; v; X ) 0 for every v T X (x ), which means ha x is a Clarke saionary poin of (1). When X := {x : Ax b}, by leing X 1 = R n and X 2 = {x : Ax b}, we find ha r-in(t X (x )) = T X2 (x ) and hen 0 r-in(t X (x )) V x. So saemen (2) is rue. Remark 1. Suppose h i is regular in {Di T x : x X \N i }, where N i = {x X : h i is no Lipschiz coninuous a D T i x}, i = 1, 2,..., m. Then, he saemen in Theorem 1 holds wih Ṽ x insead of V x, where Ṽ x ={v : for any i I x, here exiss δ > 0 such ha h i (D T i (x + v)) = h i (D T i x ) holds for all 0 δ}. Since V x Ṽ x, he generalized saionary poin defined in Definiion 2 can be more robus wih Ṽ x insead of V x for some cases. For example, if f is modeled by (2), Ṽ x = V x ; however, if f is defined as in (3), Ṽ x = {v : d T i v 0, i {i : d T i x = α i }}, (29) which includes V x = {v : d T i v = 0, i {i : d T i x = α i }} as a proper subse. Oher necessary opimaliy condiions for he special cases of (1) are sudied by Bian and Chen [6, 8], Bian e al. [9], Chen e al. [21], Ge e al. [30], Liu e al. [40]. Bian and Chen [8] considered a special case of (1) modeled by (2) wih l = ( ) n, u = n and r = 1, ha is m min f(x) := Θ(x) + ϕ( d T i x p ) (30) s.. x X := {x : Ax b}. Recall C x = {i : A T i x b i = 0}. If x X is a local minimizer of (30), here exiss a nonnegaive vecor γ R C x such ha Z T x ( Θ( x) + i I x (28) pϕ (s) s= d T i x p dt i x p 1 sign(d T i x)d i + i C x γ i A i ) = 0, (31) where I x = {i : d T i x = 0} and Z x is a marix whose columns form an orhogonal basis of he null space of {d i : d T i x = 0} (Bian and Chen [8]). Mos recenly, Liu e al. [40] considered a special case of (1) modeled by (3) wih ϕ() := and X := {x : Ax b}. They called x X a KKT poin of (3), if here exiss a nonnegaive vecor λ R I x such ha x = P X ( x L( x, λ)), (32) where L(x, λ) = Θ(x) + i J x (α i d T i x) p + i I x λ i (α i d T i x) wih J x = {i : α i d T i x > 0} and I x = {i : α i d T i x = 0}. For problems (30) and (3), he nex proposiion shows he expressions of he opimaliy condiions in Bian and Chen [8] and Liu e al. [40] wih he generalized direcional derivaive and angen cone. Proposiion 2. (1) When problem (1) reduces o problem (30), for x X, here holds f ( x; v; X ) 0, v T X ( x) Ṽ x (31) holds wih a nonnegaive vecor γ R C x ; (2) When problem (1) reduces problem (3) wih X := {x : Ax b}, for x X, here holds f ( x; v; X ) 0, v T X ( x) Ṽ x (32) holds wih a nonnegaive vecor λ R I x.

11 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion 11 Proof. Firs, we esablish he direcion = in saemen (1). From he definiion of f x in (16) for (30), we can easily verify ha f x ( x) = Θ( x) + i I x pϕ (s) s= d T i x p dt i x p 1 sign(d T i x)d i. If x saisfies (31) wih a nonnegaive vecor γ R C x, by he definiion of Z x, we obain f x ( x) + i C x γ i A i span{d i : i I x }, which implies he exisence of a nonnegaive vecor κ R I x such ha f x ( x) + i C x γ i A i = i I x κ i d i. Since T X ( x) = {v : A T i v 0, i C x }, Ṽ x = {v : d T i v = 0, i I x } and γ 0, for any v T X ( x) Ṽ x, by Proposiion 1, i gives f ( x; v; X ) = f x ( x), v = i I x κ i d i i C x γ i A i, v 0. Then, we prove he direcion = in saemen (1). Denoe S = {x : d T i x = 0, i I x }. By Ṽ x = T S ( x) and Proposiion 1, f ( x; v; X ) 0, v T X ( x) Ṽ x, implies Since T X S ( x) T X ( x) T S ( x), (33) gives f x ( x), v 0, v T X ( x) T S ( x). (33) f x ( x) N X S ( x). From he definiion of X and S, N X S ( x) N X ( x) + N S ( x) = { i C x γ i A i : γ i 0} + { i I x λ i d i : λ i R}, which ensures he exisence of a nonnegaive vecor γ R C x such ha f x ( x) i C x γ i A i { i I x λ i d i : λ i R}. Thus, Z T x ( f x ( x) + i C x γ i A i ) = 0. Nex, we show he second saemen in his proposiion. For problem (3), Ṽ x = {v : d T i v 0, i I x }. Suppose x X saisfies (32) wih λ 0, by he projecion inequaliy, i holds Θ( x) i J x p(α i d T i x) p 1 d i i I x λ i d i, x x 0, x X, (34) which implies Θ( x) i J x p(α i d T i x) p 1 d i i I x λ i d i, v 0, v T X ( x). By he definiion of Ṽ x and I x, we obain Θ( x) i J x p(α i d T i x) p 1 d i, v 0, v T X ( x) Ṽ x. (35)

12 12 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion Proposiion 1 gives f ( x; v; X ) = Θ( x) i J x p(α i d T i x) p 1 d i, v. (36) Then, (35) and (36) guaranee f ( x; v; X ) 0, v T X ( x) Ṽ x. Conversely, by (36), similar o he analysis for he firs saemen, f ( x; v; X ) 0, v T X ( x) Ṽ x, implies Θ( x) + i J x p(α i d T i x) p 1 d i N X ( x) + N S ( x), where N S ( x) = { i I x λ i ( d i ) : λ i 0} wih S = {x : α i d T i x 0, i I x }. Then, here exiss a nonnegaive vecor λ R I x such ha Θ( x) + i J x p(α i d T i x) p 1 d i + i I x λ i d i N X ( x). Thus, Θ( x) + i J x p(α i d T i x) p 1 d i i I x λ i d i, x x 0, x X, which implies x saisfies (32) wih a nonnegaive vecor λ. When problem (1) reduces o (30) and (3), by Remark 1, he necessary opimaliy condiion given in Theorem 1 holds wih Ṽ x insead of V x. And from Corollary 1, he opimaliy condiions in Bian and Chen [8] and Liu e al. [40] are equivalen o he necessary opimaliy condiions given in his paper for problems (30) and (3). Similarly, he generalized saionary poin defined in Definiion 2 reduces o he firs order necessary opimaliy condiions given or used in Bian and Chen [6], Bian e al. [9], Chen e al. [21, 23], Ge e al. [30] for he special cases of (2) wih 0 < p < 1. Thus, Definiion 2 provides a uniform version of he exising necessary opimaliy condiions for he Lipschiz and non-lipschiz problems modeled by (1). In compuaion, by he properies of generalized direcional derivaive and angen cone, some closed forms can be derived from our opimaliy condiion for differen special cases. However, when p = 1 in (2) or (3), a generalized saionary poin defined in Definiion 2 is a Clarke saionary poin, while he scaled saionary poin, firs-order saionary poin, scaled KKT poin and KKT poin in Bian and Chen [6, 8], Bian e al. [9], Chen e al. [21, 23], Ge e al. [30], Liu e al. [40] are no necessarily Clarke saionary poins. All of hese no only show he robusness of he opimaliy condiions given in his paper, bu also illusrae he superioriy of he generalized direcion derivaive defined in (17) for sudying he opimaliy condiions of problem (1). For a locally Lipschiz funcion f, we have f(x) = con f(x), where is he Clarke subdifferenial, is he limiing (Mordukhovich) subdifferenial and con denoes he convex hull (Rockafellar and Wes [47, Theorem 8.49]). Thus a Clarke saionary poin defined in (6) is necessary bu no sufficien o be a limiing (Mordukhovich) saionary poin defined in (11) when f is locally Lipschiz coninuous. Some ineresing resuls on his opic can also be found in Burke e al. [15]. When he consrain qualificaion (10) holds, he necessary opimaliy condiion given in his paper is also weaker han condiion (11). However, he consrain qualificaion (10) is likely o be unsaisfied for many non-lipschiz opimizaion problems and f(x) may be empy a some non-lipschiz poins of f.

13 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion Direcional derivaive consisency In his subsecion, we show ha he generalized direcional derivaive of f defined in (17) can be represened by he limi of a sequence of direcional derivaives of a smoohing funcion of f. This propery is imporan for he developmen of numerical algorihms in nonsmooh opimizaion. Definiion 3. Le φ : R n R be a coninuous funcion. We call φ : R n [0, ) R a smoohing funcion of φ, if φ(, µ) is coninuously differeniable for any fixed µ > 0 and lim z x,µ 0 φ(z, µ) = φ(x) holds for any x R n. When φ is Lipschiz coninuous a x, we call he gradien consisency associaed wih he smoohing funcion φ holds a x, if { lim φ(z, x µ)} φ( x). (37) z x,µ 0 Some condiions o guaranee (37) can be found in Chen [19], Burke and Hoheisel [13], Burke e al. [14], Rockafellar and Wes [47]. Le h(x, µ) := ( h 1 (D T 1 x, µ), h 2 (D T 2 x, µ),..., h m (D T mx, µ)) T, where h i is a smoohing funcion of h i in (13) and he gradien consisency associaed wih h i holds a is Lipschiz coninuous poins. Then f(x, µ) := Θ(x) + c( h(x, µ)) is a smoohing funcion of f. Since f(, µ) is coninuously differeniable for any fixed µ > 0, he generalized direcional derivaive of i wih respec o x can be given by f (x, µ; v; X ) = lim sup y x, y X 0, y + v X f(y + v, µ) f(y, µ) = x f(x, µ), v. (38) Theorem 2. Suppose he funcion h in (1) has he form in (13), and h i is coninuously differeniable a Di T x for x X \N i wih N i = {x : h i is no Lipschiz coninuous a Di T x}, i {1, 2,..., m}, hen lim x k X, x k x, µ k 0 x f(xk, µ k ), v = f ( x; v; X ), v V x. (39) Proof. Le x k be a sequence in X converging o x and {µ k } be a posiive sequence converging o 0. For v V x, by he differeniabiliy of f(x k, µ k ), we have where x f(xk, µ k ), v = Θ(x k ), w + c(z) z= h(xk,µ k ), x h(x k, µ k ) T v, (40) x h(xk, µ k ) T v = ( x h1 (D T 1 x k, µ k ) T v,..., x hm (D T mx k, µ k ) T v) T. For i I x, by v V x, we obain Di T v = 0, hen x hi (Di T x k, µ k ) T v = z hi (z, µ k ) T z=d i kd T T x i v = 0. Define { hi h x i (D T (D T i x, µ) i I x, i x, µ) := h i (D T i = 1, 2,..., m. i x, µ) i I x, Denoe h x (x, µ) = ( h x 1(D T 1 x, µ), h x 2(D T 2 x, µ),..., h x m(d T mx, µ)) T. Then, Thus, coming back o (40), we obain x h(xk, µ k ) T v = x h x (x k, µ k ) T v. x f(x k, µ k ), v = Θ(x k ), v + c(z) z= h(xk,µ k ), x h x (x k, µ k ) T v = Θ(x k ) + x h x (x k, µ k ) c(z) z= h(xk,µ k ), v. (41) Since h i is coninuously differeniable a D T i x for i I x and h x ( x) = h( x), we obain lim k Θ(x k) + x h x (x k, µ k ) c(z) z= h(xk,µ k ) = Θ( x) + h x ( x) c(z) z=h( x) = f x ( x), (42)

14 14 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion where f x is defined in (16). Thus, f ( x; v; X ) = f x( x; v; X ) = f x ( x), v = lim k Θ(x k ) + x h x (x k, µ k ) c(z) z= h(xk,µ k ), v = lim k x f(x k, µ k ), v, (43) where he firs equaion uses Proposiion 1, he hird uses (42) and he fourh uses (41). Now we give anoher consisency resul on he subspace V x. Lemma 3. Le {x k } be a sequence in X wih a limi poin x. For w V x, here exiss a sequence {x kl } {x k } such ha w V xkl, l N. Proof. If his lemma is no rue, hen here is K N such ha w V xk, k K. By he definiion of V xk, here exiss i k I xk such ha D T i k w 0, k K. From I xk {1, 2,..., m}, here exis j {1, 2,..., m} and a subsequence of {x k }, denoed as {x kl }, such ha j I xkl and Dj T w 0. Noe ha j I xkl implies h j is no Lipschiz coninuous a Dj T x kl. Since he non-lipschiz poins of h j is a closed subse of R n, h j is also no Lipschiz coninuous a Dj T x, which means j I x. By w V x, we obain Dj T w = 0, which leads o a conradicion. Therefore, he saemen in his lemma holds. For x X, from he definiions of r-in(t X ( x)) and V x, r-in(t X ( x)) V x implies r-in(t X ( x)) V x B 1 (0). Based on he consisency resuls given in Theorem 2 and Lemma 3, we give a corollary o show he generalized saionary poin consisency of he smoohing funcions. Corollary 2. Suppose he funcion h in (1) has he form in (13). Le {ɛ k } and {µ k } be posiive sequences converging o 0. Wih he condiions on h in Theorem 2, if x k X saisfies r-in(t X (x k )) V x k = or x f(x k, µ k ), v ɛ k, v T X (x k ) V x k B 1 (0), (44) hen any accumulaion poin of {x k } is a generalized saionary poin of (1). Proof. Le x be an accumulaion poin of {x k }. Then, here exiss a subsequence of {x k } (also denoed as {x k }) converging o x, i.e. lim k x k = x. Wihou loss of generaliy, we suppose r-in(t X ( x)) V x. If no, by Definiion 2, he saemen in his corollary holds naurally. Firs, we will show ha here is a subsequence of {x k } (also denoed as {x k }) such ha r-in(t X (x k )) V x k. Denoe v r-in(t X ( x)) V x. By Lemma 3, we can suppose v V x k. v in(t X1 ( x)) guaranees he exisence of K N such ha v in(t X1 (x k )), k K. Since C x k and C x are he subses of {1, 2,..., }, by heir definiions, here exiss a subsequence of {x k } (also denoed as {x k }) such ha C x k C x. From Lemma 1 (1), we obain T X2 ( x) T X2 (x k ). Thus, here is a subsequence of {x k } (also denoed as {x k }) such ha v r-in(t X (x k )) V x k. For w r-in(t X ( x)) V x B 1 (0), from Lemma 3, we can suppose w V xk, k N.

15 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion 15 By w r-in(t X ( x)), here exiss ɛ > 0 such ha x + sw X, x X B ɛ ( x), 0 s ɛ. (45) Since x k converges o x, here exiss K N such ha x k X B ɛ ( x), k K. By (45), we have x k + sw X, k K, 0 s ɛ. From he convexiy of X, we obain w T X (x k ). From Theorem 2, we have f ( x; w; X ) 0. Then, for any ρ > 0, we have f ( x; ρv; X ) = lim sup y x, y X 0, y + ρv X =ρ lim sup y x, y X s 0, y + sv X f(y + ρv) f(y) f(y + sv) f(y) s = ρf ( x; v; X ) 0. (46) Thus, f ( x; v; X ) 0 for every v r-in(t X ( x)) V x B 1 (0) implies f ( x; v; X ) 0 for every v r-in(t X ( x)) V x. By Lemma 2, i is easy o verify ha f ( x; v; X ) 0 holds for any v T X ( x) V x, which means ha x is a generalized saionary poin of (1). From Corollary 2, i is easy o prove ha any accumulaion poin of he generalized saionary poins of min x X f(x, µk ) defined by he gradien of f(x, µ) is a generalized saionary poin of (1) defined in Definiion 2 as µ k ends o 0. Remark 2. Similar o he proof for Theorem 2 and Corollary 2, he consisence resul in hem holds wih Ṽ x insead of V x, which is defined in (28) and comes from he necessary opimaliy condiion for he minimizers of (1) in Remark 1. Remark 3. The gradien consisency of h i a is Lipschiz coninuous poins implies Since f x is Lipschiz coninuous a x, i gives { lim z x,µ 0 Θ(z) + x h x (z, µ) c(y) y= h x (z,µ) } f x( x). (47) f x( x; w; R n ) = max{ ξ, w : ξ f x ( x)}. (48) Similar o he calculaion in (43), by (47) and (48), we obain f ( x; w; R n ) = f x( x; w; R n ) = max{ ξ, w : ξ f x ( x)} lim sup Θ(x k ) + x h x (x k, µ k ) c(z) z= h(xk,µ k ), w k = lim sup x f(x k, µ k ), w. k Thus, when X = R n, if x k saisfies x f(xk, µ k ), v ɛ k for every v T X (x k ) V x k B 1 (0), he conclusions in Theorem 2 and Corollary 2 can be rue wihou he coninuous differeniabiliy of h i a D T i x for x X \N i, i {1, 2,..., m}. In paricular, when he funcion h in f has he form h(x) := (h 1 (d T 1 x), h 2 (d T 2 x),..., h m (d T mx)) T wih d i R n, by Clarke [24, Theorem (i)], he regulariy of h i (d T i x) in X \N i is a sufficien condiion for he saemen in Theorem 2 and Corollary 2. The saemen ha x k is an approximae saionary poin of f(x, µ k ) over X, i.e. x f(x k, µ k ), v ɛ k for every v T X (x k ) B 1 (0) is a sufficien condiion for (44). Corollary 2 shows ha one can find a generalized saionary poin of (1) by using he approximae saionary poins of min x X f(x, µ) defined by he gradien of f(x, µ).

16 16 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion Since f(, µ) is coninuously differeniable for any fixed µ > 0, many numerical algorihms can find a saionary poin of min x X f(x, µ) (Beck and Teboulle [4], Curis and Overon [25], Leviin and Polyak [38], Nocedal and Wrigh [46], Ye [54]). In wha follows, we use one example o show he validiy of he firs order necessary opimaliy condiion in Theorem 1 and he consisency resul given in Corollary 2. Example 1. Consider he following minimizaion problem min f(x) := (x 1 + 2x 2 1) 2 + λ 1 max{x1 + x 2 + 1, 0} + λ 2 x2, s.. x X := {x R 2 : 1 x 1, x 2 1}. (49) This problem is an example of (1) wih Θ(x) = (x 1 + 2x 2 1) 2, c(y) = λ 1 y 1 + λ 2 y 2, h 1 (D T 1 x) = max{x1 + x 2 + 1, 0} and h 2 (D T 2 x) = x 2, where D 1 = (1, 1) T, D 2 = (0, 1) T. Define he smoohing funcion of f as f(x, µ) = (x 1 + 2x 2 1) 2 + λ 1 ψ(x1 + x 2 + 1, µ) + λ 2 θ(x2, µ), wih ψ(s, µ) = 1(s + s s > µ, s µ 2 ) and θ(s, µ) = s 2 2µ + µ s µ. 2 Here, we use he classical projeced algorihm wih Armijo line search o find an approximae generalized saionary poin of min x X f(x, µ). There exiss α > 0 such ha x PX [ x α x f( x, µ)] = 0 if and only if x is a generalized saionary poin of min x X f(x, µ), which is also a Clarke saionary poin of min x X f(x, µ) for any fixed µ > 0. We call xk an approximae saionary poin of min x X f(x, µk ), if here exiss α k > 0 such ha x k P X [x k α k x f(xk, µ k )] 2 µ k, which can be found in a finie number of ieraions by he analysis in Bersekas [5]. Choose iniial ierae x 0 = (0, 0) T and le he ieraion be erminaed when µ k For differen values of λ 1 and λ 2 in (49), he simulaion resuls are lised in Table 1, where f indicaes he opimal funcion value of (49). In wha follows, we will show ha he accumulaion poins in Table 1 are he generalized saionary poins of (49) defined in Definiion 2. Moreover, i is ineresing ha hese poins are global minimizers of (49). λ 1 λ 2 accumulaion poin x I x V x f(x ) f 8 2 ( 1.000, 0.000) T {1} {v = (a, a) T : a R} (0.982, 0.000) T {2} {v = (a, 0) T : a R} ( 1.000, 0.962) T R Table 1. Simulaion resuls in Example 1 When λ 1 = 8 and λ 2 = 2, since h 2 (D T 2 x) is coninuously differeniable a x, for v V x, by h 1 (D T 1 (x + v)) = h 1 (D T 1 x ), > 0, we obain f (x ; v; X ) = Θ(x ) + λ 2 h 2(D T 2 x )D 2, v = 4v v 2, where v 1 = v 2 by v V x, and v 1 R + by v T X (x ). Then, f (x ; v; X ) 0, v T X (x ) V x, which means ha ( 1.000, 0.000) T is a generalized saionary poin of (49). Similarly, when λ 1 = 0.1 and λ 2 = 0.2: where v 2 = 0 by v T X (x ) V x ; f (x ; v; X ) = Θ(x ) + λ 1 h 1(D T 1 x )D 1, v = 0.036v 2,

17 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion x 40 x 30 x 2 0 x 160 x x 1 Figure 1. Trajecory of x k in Example 1 wih λ 1 = 8 and λ 2 = 2 when λ 1 = 0.5 and λ 2 = 0.1: f (x ; v; X ) = Θ(x ) + λ 1 h 1(D T 1 x )D 1 + λ 2 h 2(D T 2 x )D 2, v = 0.102v 1, where v 1 R + by v T X (x ) V x. This gives f (x ; v; X ) 0, for all v T X (x ) V x. Furhermore, he rajecory of x k of he smoohing algorihm for (49) wih λ 1 = 8 and λ 2 = 2 is picured in Figure 1 wih he isolines of f in X. From Example 1, we find ha he proposed algorihm wih he classical projeced algorihm for finding he approximaed generalized saionary poin can find a global minimizer of problem (49), which is of course a generalized saionary poin of i. Finding global minimizers via generalized saionary poins for problm (1) is an ineresing problem for furher sudy. 3. Numerical properies In his secion, we focus on he numerical properies of problem (1) wih he lower bound propery of is local minimizers and is compuaional complexiy. I is known ha problem (2) wih X = R n, ϕ() := and p (0, 1) enjoys lower bound propery (Chen e al. [23]) bu is srongly NP-hard (Chen e al. [20]). Owning o he beer numerical properies of differen penaly funcions or regularizaion erms in various applicaions and he imporance of he lower bound propery in pracice, we will explore a wider bound propery and he compuaional complexiy of (1) wih X := {x : Ax b}, where A = (A 1,..., A ) T R n wih A i R n, i = 1, 2,...,, and b = (b 1, b 2,..., b ) T R. In his secion, we suppose he funcion h in (1) can be represened as wih d i R n. h(x) := (h 1 (d T 1 x), h 2 (d T 2 x),..., h m (d T mx)) T, (50) 3.1. Bound propery For x X, le C x = {j {1, 2,..., } : A T j x b j = 0} be he se of acive inequaliy consrains a x, L x be he index se such ha h i is no LC 1 a d T i x for i L x, where we call a funcion φ : R n R is LC 1 (or C 1,1 ) a x, if φ is coninuously differeniable and is gradien is locally Lipschiz coninuous a x (Hiriar-Urruy e al. [32]). For a funcion φ : R n R of LC 1, 2 φ(x) is he generalized Hessian marix of φ a x, which is he generalized derivaive of φ(x) in Clarke s sense (Clarke [24]), i.e. 2 φ(x) = con{m : x k x wih φ wice differeniable a x k and 2 φ(x k ) M }.

18 18 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion Le M be he se of local minimizers of problem (1). To show he local minimizers of problem (1) own he lower bound propery, we need he following assumpions. Assumpion 2. There exiss β 0 such ha sup x M 2 Θ(x) 2 β. Assumpion 3. Funcion h in (1) has he form in (50), he poins a which h i is no LC 1 are finie and {η 2 h i (y) y=d T i x : x X } is non-posiive, i = 1, 2,..., m. Assumpion 4. Funcion c is of LC 1 on is domain, c(s) s=h(x) 0, x X, and all elemens in {C 2 c(s) s=h(x) : x X } is negaive semi-definie. Remark 4. When Θ(x) := Hx ω 2 2 or Θ(x) := log( Hx ω ) wih H R s n and ω R s, Assumpion 2 holds naurally wih β = 2 H T H 2. And he boundedness of M also guaranees Assumpion 2. Assumpions 3 and 4 for c(h(x)) are also saisfied by many models. For example, le c(y) = m y i, h i (z) = ϕ( z p ) or h i (z) = ϕ(max{0, z} p ), i = 1, 2,..., m, hen c(h(x)) wih 0 < p 1 saisfies Assumpions 3 and 4 when ϕ is wih one of he following expressions sof hresholding penaly funcion (Tibshirani [51]): ϕ 1 (s) = λs, logisic penaly funcion (Nikolova e al. [44]): ϕ 2 (s) = λ log(1 + as), fracion penaly funcion (Nikolova e al. [44]): ϕ 3 (s) = λ as, 1+as hard hresholding penaly funcion (Fan [26]): ϕ 4 (s) = λ 2 (λ s) 2 +, smoohly clipped absolue deviaion (SCAD) penaly funcion (Fan and Li [27]): ϕ 5 (s) = λ s 0 min{1, (a /λ) + }d, a 1 minimax concave penaly (MCP) funcion (Zhang [55]): wih λ > 0 and a > 0. ϕ 6 (s) = λ s 0 (1 aλ ) +d, Theorem 3. Suppose Assumpions 2-4 hold. Then, for any a R and i {1, 2,..., m}, if here exiss α i > 0 such ha i c(s) s=h(x) α i, x X, hen here exis θ i > 0 and κ i > 0 such ha any local minimizer x of (1) wih X := {x : Ax b} saisfies (1) h i (a+) > κ i = eiher d T i x a + θ i or d T i x a; (2) h i (a ) > κ i = eiher d T i x a θ i or d T i x a. Proof. By C x {1, 2,..., } and L x {1, 2,..., m}, we divide M ino he finie disjoin ses M 1, M 2,..., M s, such ha for each M i, C x and L x are he same for all elemens x M i. Then, W x = {v : d T i v = 0 for i L x and A T k v 0 for k C x } is he same for all elemens x in each se M 1, M 2,..., M s. So, we le C k, L k and W k denoe C x, L x and W x for x M k, k = 1, 2,..., s, respecively. For ī {1, 2,..., m}, we firs prove he saemen (1) holds for any x M 1. If ī L 1, hen saemen (1) holds naurally by Assumpion 3. Nex, we consider i for ī L 1. Le x M 1 and define h x (x) := (h x 1(d T 1 x), h x 2(d T 2 x),..., h x m(d T mx)) wih { h x i (d T hi (d T i x) i L 1 i x) = h i (d T i x) i L 1. Then, here exiss δ > 0 such ha f( x) = min{θ(x) + c(h(x)) : x x 2 δ, Ax b} = min{θ(x) + c(h x (x)) : x x 2 δ, Ax b, d T i x = d T i x for i L 1 },

19 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion 19 which implies ha x is a local minimizer of he following consrained minimizaion problem min f x (x) := Θ(x) + c(h x (x)) s.. Ax b, d T i x = d T i x, i L 1. (51) Since f x is LC 1 a x, by he second order necessary opimaliy condiion for he minimizers of (51) given by Hiriar-Urruy e al. [32, Corollary 3.1] and he finie sum rule in Clarke [24, Proposiion 2.3.3], for every v W 1, here exis C v ( c(s)) s=h( x) and η v i 2 h x i (z) z=d T i x for i L 1 such ha v T 2 Θ( x)v + v T h x ( x)c v ( h x ( x)) T v + i L 1 i c(s) s=h( x) η v i d T i v 2 0, (52) where he consrain qualificaion can be ignored for (51) (Sun and Yuan [50, Definiion 8.2.8, Corollary 8.2.9]). By Assumpions 2-4, we obain α ī η v ī dt ī v 2 v T 2 Θ( x)v β v 2 2, v W 1. (53) Wihou loss of generaliy, we suppose ha here are infinie elemens in M 1, which means ha here exiss wo elemens in M 1 wih differen values of d T ī x, and he same values of dt i x for i L 1, denoed as x and ˆx. If no, he resul in saemen (1) holds naurally. Consider he following consrained convex opimizaion min v 2 2 s.. v W 1 ī = {v : dt ī v = 1 and v W 1 }. (54) 1 By ˆx( x ˆx) W 1 d T ī x dṱ ī, unique exisence of he opimal soluion of (54) is guaraneed, denoed i by v 1. Le v = ī v1 ī in (53), hen we have η v ī κī, (55) wih κ ī = β v 1 ī 2 2/α ī. If h ī (a+) > κī, by Assumpion 3, which means h ī (a+) < κī, le θ ī = inf{ > 0 : h ī (a + ) exiss and h ī (a + ) κī }. (56) By he upper semiconinuiy of (h ī ()) around dt ī x and ηv ī (h ī (s)) s=d T ī x, (55) implies eiher d T ī x a + θī or d T ī x a. By he randomiciy of x M 1 and he invariance of κ 1 and ī θ1 for all elemens in M ī 1, he saemen in (1) holds for he elemens in M 1. Similar analysis can be given for M 2,...,M s. Therefore, we can complee he proof for saemen (1). Saemen (2) can be shown by using he same argumens. From he proof for Theorem 3, we find ha he concaviy of c(h(x)) is he key condiion o guaranee he resuls in Theorem 3. Based on he resuls in Theorem 3, we sudy he following special case of (1) m f(x) := Θ(x) + ϕ(max{α i d T i x, 0} p ), (57) min Ax b wih α i R and d i R n, i = 1, 2,..., m. Assumpion 5. ϕ : R + R + is coninuously differeniable, non-decreasing and concave on R ++, and ϕ is locally Lipschiz coninuous on R ++.

20 20 W. Bian and X. Chen: consrained opimizaion problems wih nonconvex regularizaion By Assumpion 5, we can obain he following lower bound resuls for problem (57). Corollary 3. Suppose Θ saisfies Assumpion 2 and ϕ saisfies Assumpion 5. (1) For 0 < p < 1, if ϕ (0+) > 0, hen here exiss a consan θ > 0 such ha any local minimizer x of (57) saisfies eiher α i d T i x θ or α i d T i x 0, i {1, 2,..., m}; (58) (2) For p = 1, here exis consans θ > 0 and κ > 0 such ha if ϕ (0+) > κ, hen any local minimizer x of (57) saisfies (58). Proof. Define c(y) = m y i and h i (d T i x) = ϕ(max{α i d T i x, 0} p ), i = 1, 2,..., m. I is easy o verify ha funcions c and h i, i = 1, 2,..., m, saisfy Assumpions 3 and 4. For i = 1, 2,..., m, we have h i (α i ) = when 0 < p < 1, and h i (α i ) = ϕ (0+) when p = 1, by Theorem 3 wih a = α i, which implies he saemens in his corollary. When problem (1) reduces o he following special case: min Ax b f(x) := Θ(x) + m ϕ( d T i x p ). (59) Denoe c(y) = m y i and h i (d T i x) = ϕ( d T i x p ), i = 1, 2,..., m. Suppose ϕ saisfies Assumpion 5, hen h i (0+) = h i (0 ) = for (59) wih 0 < p < 1, and h i (0+) = h i (0 ) = ϕ (0+) for (59) wih p = 1. Therefore, he resuls in Corollary 3 hold for problem (59) when (58) is replaced by eiher d T i x θ or d T i x = 0, i {1, 2,..., m}. (60) If here exiss a consan κ > 0 such ha ϕ (0+) κ, by he concaviy of ϕ and ϕ 0, ϕ (0+) > 0 holds obviously. However, he converse does no hold. So boh (57) and (59) can own he lower bound propery based on he weaker condiion for 0 < p < 1 han for p = 1, which shows he superioriy of he non-lipschiz regularizaion in sparse reconsrucion. For he poenial funcions in Remark 5, only ϕ 2, ϕ 3, ϕ 4 and ϕ 6 may mee he condiions in Corollary 3 for p = 1, bu all of hem saisfy he condiions in Corollary 3 for 0 < p < 1. Moreover, he bound resul in Theorem 3 can also be exended o he problem modeled by min Ax b f(x) := Θ(x) + m ϕ i ( D T i x p p), (61) wih 0 < p 1 and D i R n r, i = 1, 2,..., m. Also wih he condiions in Corollary 3, we can obain he following esimaion Similar exension can also be done for eiher D T i x p θ or D T i x p = 0, i {1, 2,..., m}. (62) min Ax b f(x) := Θ(x) + m ϕ i ( max{0, D T i x} p ). The lower bound resul in Chen e al. [23] is meaningful, since i no only indicaes he exisence of he lower bound propery for any local minimizers of he unconsrained l 2 -l p minimizaion wih 0 < p < 1, bu also presens a lower bound value. Due o he generaliy of he considered model in (1), Theorem 3 only proves he exisence of he bound propery for problem (1) wih general affine inequaliy consrains. However, following he proof of Theorem 3, we can obain explici values of κ i and θ i in Theorem 3 in he following wo seps.