OUTLINE l p -Norm Constrained Quadratic Programming: Conic Approximation Methods Wenxun Xing Department of Mathematical Sciences Tsinghua University, Beijing Email: wxing@math.tsinghua.edu.cn
OUTLINE OUTLINE 1 l p -Norm Constrained Quadratic Programming 2 3 4 5
Data fitting l 2 -norm: least-square data fitting min Ax b 2 s.t. x R n. When A is full rank in column, then x = A T A 1 A T b. A 2nd-order conic programming formulation min s.t. t Ax b 2 t x R n. Experts in numerical analysis prefer the direct calculation much more than the optimal solution method.
l 1 - norm problem l 1 -norm. min x 1 s.t. Ax = b x R n. A linear programming formulation min s.t. n i=1 t i t i x i t i, i = 1, 2,..., n Ax = b t, x R n.
Heuristic method for finding a sparse solution Regressor selection problem: A potential regressors, b to be fit by a linear combination of A min Ax b 2 s.t. cardx k x Z n +. It is NP-hard. Let m = 1, A = a 1, a 2,..., a n, b = 1 n 2 i=1 a i, k n 2. It is a partition problem. Heuristic method. min Ax b 2 + γ x 1 s.t. x R n. Ref. S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
Regularized approximation min Ax b 2 + γ x 1 s.t. x R n. l 1 -norm and l 2 -norm constrained programming min t 1 + γt 2 s.t. Ax b 2 t 1 x 1 t 2 x R n, t 1, t 2 R. The objective function is linear, the first constraint is a 2nd-order cone and the 2nd is a 1st-order cone. It is a convex optimization problem of polynomially solvable.
p-norm domain Black: 1-norm. Red: 2-norm. Green: 3-norm. Yellow: 8-norm.
Convex l p -norm problems p-norm domain is convex p 1. For set {x x p 1}, the smallest one is the domain with p = 1, which is the smallest convex set containing integer points { 1, 1} n. For p 1, the l p -norm problems with linear objective or linear constraints are polynomially solvable. Variants of l p -norm problems should be considered.
Variants of l p -norm problems l 2 -norm constrained quadratic problem min x T Qx + q T x s.t. Ax b 2 c T x c T x = d 0 x R n. l 1 -norm constrained quadratic problem min s.t. x T Qx + q T x x 1 k x R n, where Q is a general symmetric matrix.
l p -Norm Constrained Quadratic Programming min where p 1. 1 2 xt Qx + q T x s.t. 1 2 xt Q i x + q T i x + c i 0, i = 1, 2,..., m Ax b p c T x x R n,
QCQP reformulation where D R n. min 1 2 xt Q 0 x + q T 0 x + c 0 s.t. 1 2 xt Q i x + q T i x + c i 0, i = 1, 2,..., m x D,
p-norm form l 1 -norm problem min x T Qx + q T x s.t. x 1 k x R n. Denote D = {x R n x 1 k}. QCQP form min x T Qx + q T x s.t. x D.
2-norm form 2-norm problem min x T Qx + q T x s.t. Ax b 2 c T x c T x = d 0 x R n. Denote D = { x R n Ax b 2 c T x } QCQP form min s.t. x T Qx + q T x c T x = d x D.
Lifting reformulation min f x = 1 2 xt Q 0 x + q T 0 x + c 0 s.t. g i x = 1 2 xt Q i x + q T i x + c i 0, i = 1, 2,..., m QCQP x D. Denote: F = {x D g i x 0, i = 1, 2,..., m}. Lifting min 1 2c 0 q0 T 2 X q 0 Q 0 1 2c s.t. i qi T 2 X 0, i = 1, 2,..., m q i Q i T 1 1 X =, x F. x x
Convex reformulation min 1 2c 0 q0 T 2 X q 0 Q 0 1 2c s.t. i qi T 2 X 0, i = 1, 2,..., m q i Q i 1 0 X = 1 0 0 T 1 1 X clconv x F x x.
Linear conic programming reformulation min 1 2c 0 q0 T 2 X q 0 Q 0 1 2c s.t. i qi T 2 X 0, i = 1, 2,..., m q i Q i 1 0 X = 1 0 0 T 1 1 X clcone x F x x. It is a linear conic programming and has the same optimal value with QCQP.
Quadratic-Function Conic Programming PRIMAL min 1 2c 0 q0 T 2 V q 0 Q 0 1 2c s.t. i qi T 2 V 0, i = 1, 2,..., m QFCP q i Q i 1 0 V = 1 0 0 T V DF = cl 1 1 cone, x F. x x F R n, A B = traceab T,
Quadratic-Function Conic Programming DUAL s.t. max σ 2σ + 2c 0 + 2 m i=1 λ ic i q 0 + m i=1 λ iq i T q 0 + m i=1 λ iq i Q 0 + m i=1 λ iq i σ R, λ R m +, D F F R n, D F = U Sn+1 1 x T U 1 x 0, x F.
Properties of the Quadratic-Function Cone Cone of nonnegative quadratic functions Sturm and Zhang, MOR 28, 2003. T D F = U 1 1 Sn+1 U 0, x F x x. If F, then D F is the dual cone of D F and vice versa. If F is a bounded nonempty set, then DF 1 1 = cone x x T, x F. If intf, then D F and D F are proper.
Properties The complexity of checking whether V D F or U D F depends on F. When F = R n, D F = Sn+1 +. When F = R n +, DF is the copositive cone! Ref: recent survey papers I. M. Bomze, EJOR, 2012 2163; Mirjam Dür, Recent Advances in Optimization and its Applications in Engineering, 2010; J.-B. Hiriart-Urruty and A. Seeger, SIAM Review 524, 2010. Relaxation or restriction D F S n + D F. Approximation: Computable cover of F.
Checking U D F is an optimization problem! D F = U Sn+1 1 x T U 1 x 0, x F. Theorem U D F if and only if the optimal value of the following problem is not negative min 1 x s.t. x F. T U 1 x If F is a p-norm constraint, then it is a p-norm constrained quadratic programming.
Easy cases min T 1 U x s.t. x F. 1 x If F is a p-norm constraint, then it is a p-norm constrained quadratic programming. When F = {x R n 1 2 xt Px + p T x + d 0}, P 0, intf, it is computable. { When F = Socn = x R n } x T Px c T x, P 0, intf, it is computable.
A special case of p-norm constrained quadratic programming min s.t. 1 2 xt Qx + q T x x p k x R n, where p 1. Equivalent formulation min s.t. 1 2 xt Qx + 1 k tqt x x p t t = k x R n.
Homogenous quadratic constrained model min s.t. 1 2 xt Qx + 1 k tqt x x p t t = k x R n. Homogenous quadratic form T min 1 t 2 x s.t. 0 1 k qt 1 k q Q t = k t { t, x R R n x p t } x t x
of the problem Homogenous: It is polynomially computable when p = 2. min x T Qx s.t. x Socn = { x R n } x T Px c T x, where Q is a general symmetric matrix, P is positive definite and Socn + 1 has an interior Ref: Ye Tian et. al., JIMO 93, 2013. Variant? min s.t. x T Qx + q T x Ax b 2 c T x c T x = d 0 x R n.
of the problem Homogeneous QP over the 1st-order cone is NP-hard min s.t. x 0 x x 0 x T Q x 0 x Focn + 1, where Focn + 1 = {x 0, x R R n x 1 x 0 }, and Q is a general symmetric matrix. It is NP-hard.
of the problem A cross section problem T 1 min Q x s.t. x 1 1 x R n 1 x Guo et. al. conjectured NP-hard Ref: Xiaoling Guo et. al., JIMO 103, 2014. It is NP-hard Ref: Yong Hsia, Optimization Letters 8, 2014.
of the problem A general case p 1. min s.t. t x t x T Q t x { t, x R R n x p t }. Zhou et. al. conjectured NP-hard Ref: Jing Zhou et. al., PJO to appear, 2014. Provided with many solvable subcases.
Quadratic-Function Conic Programming PRIMAL min 1 2c 0 q0 T 2 V q 0 Q 0 1 2c s.t. i qi T 2 V 0, i = 1, 2,..., m q i Q i 1 0 V = 1 0 0 V D F. QFCP F R n, A B = traceab T, T DF 1 1 = cl cone, x F. x x
Quadratically Constrained Quadratic Programming QCQP Theorem If F, then the QFCP primal, its dual and the QCQP have the same optimal objective value. Theorem Suppose F, G 1 and G 2 be nonempty sets. Denote vf, vg 1 and vg 2 be the optimal objective value of the QFCP with F selecting different sets respectively. i If G 1 G 2, then D G1 D G2 and D G 1 D G 2. ii If F G 1 G 2, then vf vg 1 vg 2.
Relaxation Relaxation C DF and computable. min 1 2c 0 q0 T 2 q 0 Q 0 s.t. v 11 = 1 The worst one: C = S n+1 +. V 1 2 H i V 0, i = 1, 2,..., s V = v ij C,
Ellipsoid Cover of Bounded Feasible Set Easy case: Quadratic-function cone over one ellipsoid constraint. Theorem Let F = {x R n gx 0}, where gx = 1 2 xt Qx + q T x + c, intf and Q S++. n For an n + 1 n + 1 real symmetric matrix V, V DF if and only if 1 2c q T 2 V 0 q Q V S+ n+1.
Ellipsoid Cover of Bounded Feasible Set Ellipsoid cover Lu et al, 2011 Theorem Let G = G 1 G 2 G s, where G i = {x R n 1 2 xt B i x + b T i x + d i 0}, 1 i s, are ellipsoids with an interior, then D G = D G 1 + D G 2 + + D G s. And V DG if and only if the following system is feasible V = V 1 + V 2 + + V s 2d i bi T V i 0, i = 1, 2,..., s 1 2 b i B i V i S n+1 +, i = 1, 2,..., s.
Ellipsoid Cover of Bounded Feasible Set min H 0 V s.t. V 11 = 1 H i V 0, i = 1, 2..., m V = V 1 + + V s [ ] d i bi T V i 0, V i 0, i = 1, 2,..., s. b i B i EC It is a SDP, computable!
Ellipsoid Cover: Decomposition Theorem Under some assumptions, if V = V1 +... + V s is an optimal solution of EC, then for each j, j = 1,.., s, there exists a decomposition of V j = n j [ ] [ 1 1 µ ji i=1 x ji x ji ] T for some n j > 0, x ji G j, µ ji > 0 and n j i=1 µ ji = [Y j ] 11. Moreover, V can be decomposed in the form of V = n s j [ ] [ 1 1 µ ji j=1 i=1 with x ji G j, µ ji > 0 and s nj j=1 i=1 µ ji = V11 = 1. x ji x ji ] T
Ellipsoid Cover: Step 1 Cover the feasible set F with some ellipsoids. Step 2 Solve EC. Step 3 Decompose the optimal solution of EC and find a x ji with the smallest objective value sensitive point. Step 4 Check if the sensitive point x ji F. If it is, then it is a global optimum of QCQP. Otherwise, cover G j with two smaller ellipsoids. Repeat above procedure. Step 5 The approximation objective values converge to the optimal value of QCQP. Applications: QP Lu et al, to appear in OPT, 2014, 0-1 knapsack Zhou et al, JIMO 93, 2013, to detect copositve cone Deng et al, EJOR 229, 2013 etc.
Adaptive ellipsoid covering
Applications to p-norm problems: bounded feasible sets p-norm problem min x T Qx + q T x s.t. x p k x R n. F = D = { x R n x p k }. 2-norm problem min x T Qx + q T x s.t. Ax b 2 c T x c T x = d, x R n. D = { x R n Ax b 2 c T x }, F = { x D c T x = d }.
lp -Norm Constrained Quadratic Programming Second-order Cone Cover W. Xing Sept. 2-4, 2014, Peking University
For the least square problem, why the 2nd-order conic model is not used generally? Can we have more efficient algorithms than the interior point method for SDP?
Thank You!