Continuous Optimisation, Chpt 9: Semidefinite Optimisation Peter J.C. Dickinson DMMP, University of Twente p.j.c.dickinson@utwente.nl http://dickinson.website/teaching/2017co.html version: 28/11/17 Monday 27th November 2017 Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 1/1
Book Semidefinite Optimization M.J. Todd http://people.orie.cornell.edu/miketodd/soa5.ps Primal Problem: Chpt 2: Problems Dual problems: Chpt 4: Duality Examples: Chpt 3: Examples Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 2/1
Table of Contents 1 Introduction 2 Symmetric Matrices Inner Product Nonnegative symmetric matrices Primal and Dual Problem 3 Positive Semidefinite Cone Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 3/1
Inner Product For a vector space V, we say that, : V V R is an inner product if the following properties hold: 1 Symmetry: x, y = y, x for all x, y V, 2 Linearity 1: λx, y = λ x, y for all x, y V, λ R, 3 Linearity 2: x + y, z = x, z + y, z for all x, y, z V, 4 Positive definiteness: x, x > 0 for all x V \ {0}. Example For vector space R n and a matrix A S n, A O have an (induced) inner product x, y A = x T Ay. Usually consider the standard inner product x, y = x T y = x, y I. Example For B = {x R n : x 2 1}, we can consider the space of continuous functions from B to R, which has an inner product f, g = B f (x)g(x)dx. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 4/1
Primal and Dual Problem Consider convex cone K V and c, a 1,..., a m V and b R m : min x c, x s. t. a i, x = b i for all i = 1,..., m (P) x K, max y b T y s. t. c m y i a i K i=1 y R m. (D) K := {z V x, z 0 for all x K}. Ex. 9.1 Show that x feas(p), y feas(d) c, x b T y. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 5/1
Symmetric Matrices Definition 9.1 For space of symmetric matrices, S n, define inner product A, B := trace(ab) = n i,j=1 a ijb ij. The definitions/results from the previous two lectures can be naturally extended for this, noting that S n is a space of dimension n(n + 1). 1 2 Lemma 9.2 For A R n m and B R m n have trace(ab) = trace(ba). Lemma 9.3 Have ab T + ba T, X = 2a T Xb for all a, b R n, X S n. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 6/1
Nonnegative symmetric matrices Cone of Nonnegative symmetric matrices: N n := {X S n x ij 0 for all i, j} = conic{e i e T j + e j e T i i, j}. (N n ) = {e i e T j + e j e T i i, j} = {Y S n e i e T j + e j e T i, Y 0 for all i, j} = {Y S n y ji + y ij 0 for all i, j} = N n. Closed convex cone as intersection of closed convex cones. Pointed as ±X N n ±x ij 0 for all i, j X = O. Full-dimensional as dual to a pointed convex cone. Denote A B iff A B N n. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 7/1
Primal and Dual Problem Proper cone K S n and C, A 1,..., A m S n and b R m : min X C, X s. t. A i, X = b i for all i = 1,..., m (P) X K, max b T y y s. t. C m i=1 y ia i K (D) y R m. K := {Z S n X, Z 0 for all X K}. (P) and (D) are dual problems to each other. Slater s condition Strong duality. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 8/1
Table of Contents 1 Introduction 2 Symmetric Matrices 3 Positive Semidefinite Cone Definition Correlation matrices Eigenvalue problems Approximating Quadratic Problems Polynomial Optimisation Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 9/1
Positive semidefinite cone Positive Semidefinite cone, PSD n := {X S n v T Xv 0 v R n } = conic{bb T b R n }. Ex. 9.2 Prove that (PSD n ) = PSD n. Ex. 9.3 Prove that PSD n is a proper cone. Denote A B iff A B PSD n. Solvers Commercial: MOSEK. Free: SDPT3, SEDUMI Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 10/
Correlation matrices Definition 9.4 For two random variables [( X)( 1, X 2, define their correlation to be corr(x 1, X 2 ) := E X1 µ 1 X2 µ 2 σ 1 σ 2 )]. For vector of random variables X = ( ) T X 1 X n define correlation matrix corr(x ) S n s.t. [corr(x )] ij = corr(x i, X j ). Theorem 9.5 For A R n n, there exists a distribution such that A = corr(x ) iff A PSD n and a ii = 1 for all i. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 11/
Example What is the maximum possible corr(x 1, X 2 ) given corr(x 1, X 3 ) = 0.6 and corr(x 2, X 3 ) = 0? max y 1 1 y 1 0.6 s. t. y 1 1 0 0 0.6 0 1 Ex. 9.4 What is the dual problem to the example above? Ex. 9.5 Formulate as a semidefinite problem, the problem of finding the minimum possible corr(x 1, X 2 ) given 0.5 corr(x 1, X 3 ) 0.6 and 0.1 corr(x 2, X 3 ) 0. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 12/
Eigenvalue problems Lemma 9.6 For A S n and s, t R {± } with s t, all the eigenvalues of A are between s and t if and only if si A ti. (Where I S n is the identity matrix.) Example Find x R m such that the absolute values of the eigenvalues of C n i=1 A ix i are as small as possible: min x,t t s. t. ti C n A i x i ti. i=1 Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 13/
Approximating quadratic problems min x R n x T Qx + q T x s. t. x T A i x + a T i x = α i for all i = 1,..., m, min x R n s. t. Q, xx T + q T x A i, xx T + a T i x = α i for all i = 1,..., m ( ) ( ) ( ) 1 x T T 1 1 x xx T = PSD n+1, x x (1) (2) min x R n, X S n Q, X + q T x s. t. A i, X + a T i x = α i for all i = 1,..., m ( ) (3) 1 x T PSD n+1 x X val(1) = val(2) val(3). Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 14/
Examples Example min x 2 1 + 2x 2 2 4x 1 x 2 + 2x 2 s. t. x 2 1 + x 2 2 x 1 x 2 4x 1 = 1, x R 2, min s. t. ( 1 2 2 2 ( 1 1/2 1/2 1 ), X + 2x 2 ), X 4x 1 = 1, ( ) 1 x T PSD 3. x X Ex. 9.6 For the following problem, give a finite upper bound and formulate a PSD problem which would give a lower bound. min 2x1 2 + 2x2 2 6x 1 x 2 + 2x 2 4x 3 s. t. 2x1 2 + x2 2 2x 1 x 2 4x 1 = 1 2x 2 x3 2 = 0, x R 3. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 15/
Introduction Symmetric Matrices Positive Semidefinite Cone Certifying nonegativity Lemma 9.7 For w : Rn Rp and A PSDp, letting f (x) = [w(x)]t A [w(x)], we have that f (x) 0 for all x Rn. Example For f : R R given below we have f (x) 0 for all x R: T x 2 1 1 x f (x) sin x 1 3 2 sin x cos x 1 2 3 cos x 2x 2 2x sin x 2x cos x + 3 sin2 x 4 sin x cos x + 3 cos2 x. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 16/
Sum-of-squares polynomials Definition 9.8 Consider a polynomial f : R n R with degree 2d and let v : R n R p be the vector of monomials of x of degree up to d. We say that f (x) is a sum-of-squares polynomial if there exists A PSD p such that f (x) = [v(x)] T A [v(x)]. This is a sufficient condition for having f (x) 0 for all x R n. Example For n = 2, d = 2 we have v = ( 1 x 1 x 2 x 2 1 x 1 x 2 x 2 2 ) T. Remark 9.9 There are (n+d)! n!d! monomials of x R n of degree up to d. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 17/
Example We have that f (x) = f 0 + f 1 x + f 2 x 2 + f 3 x 3 + f 4 x 4 is a sum-of-squares polynomial if and only if there exists A PSD 3 such that 1 f (x) x x 2 T a 11 a 12 a 13 1 a 12 a 22 a 23 x a 13 a 23 a 33 x 2 a 11 + 2a 12 x + (2a 13 + a 22 )x 2 + 2a 23 x 3 + a 33 x 4. Equivalently, f is a sum-of-squares polynomial if and only if there exists a 13 R such that 1 f 0 2 f 1 a 13 1 2 f 1 1 f 2 2a 13 2 f 3 PSD 3. 1 a 13 2 f 3 f 4 Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 18/
Polynomial Optimisation Lemma 9.10 For a polynomial f (x) of degree 2d we have min{f (x)} = max {λ : f (x) λ 0 for all x x λ Rn } max{λ : f (x) λ is a sum-of-squares polynomial}. Example λ For f (x) = f 0 + f 1 x + f 2 x 2 + f 3 x 3 + f 4 x 4 we have min{f (x)} max{λ : f (x) λ is a sum-of-squares polynomial} x λ 1 f 0 λ = max λ,a 13 λ : 2 f 1 a 13 1 2 f 1 1 f 2 2a 13 2 f 3 PSD 3 1 a 13 2 f. 3 f 4 Extendable for constraints, see e.g. [Sums of Squares, Moment Matrices and Optimization Over Polynomials, M. Laurent, 2009]. Peter J.C. Dickinson http://dickinson.website CO17, Chpt 9: Semidefinite Optimisation 19/