Convex Optimization & Parsimony of L p-balls representation

Convex Optimization & Parsimony of L p -balls representation LAAS-CNRS and Institute of Mathematics, Toulouse, France IMA, January 2016

Motivation Unit balls associated with nonnegative homogeneous polynomials An important convexity property Optimality in unit ball representation

It is well-known that the shape of the Euclidean unit ball B 2 = { x : n i=1 x 2 i 1 } has spectacular geometric properties w.r.t. other shapes. For instance, the sphere has the smallest surface area among all surfaces enclosing a given volume and it encloses the largest volume among all closed surfaces with a given surface area. Hilbert and Cohn-Vossen even describe eleven geometric properties of the sphere...

But B 2 has also another spectacular (non-geometric) property related to its algebraic representation which is obvious even to people with a little background in Mathematics: Its defining polynomial x g (2) (x) := n i=1 x i 2 cannot be simpler! Indeed, among all quadratic homogeneous polynomials x g(x) = i j g ij x i x j that define a bounded unit ball { x : g(x) 1 }, g (2) is the one that minimizes the cardinality pseudo-norm" g 0 := #{ (i, j) : g ij 0 }. Only n coefficients of g (2) (out of potentially O(n 2 )) do not vanish and there cannot be less than n non zero coefficients to define a bounded ball { x : g(x) 1 }.

The same is true for the L d -unit ball B d = { x : n i=1 x i d 1 } and its defining polynomial x g (d) (x) := i x i d for any even integer d > 2, when compared to any other homogeneous polynomial g of degree d whose sublevel set { x : g(x) 1 } has finite Lebesgue volume (and so g is necessarily nonnegative). Indeed, again g (d) 0 = n, i.e., out of potentially ( ) n+d 1 d coefficients of g (d), only n do not vanish!

Let Hom[x] d R[x] d be the space of homogeneous polynomials of degree d. In other words, g (d) is an optimal solution of the optimization problem P 0 : inf g { g 0 : vol (G) ρ d ; g Hom[x] d }, where the minimum is taken over all homogeneous polynomials of degree d, and ρ d denotes the Lebesgue volume of the L d -unit ball B d.

Question I: Can we recover this parsimony property by minimizing the sparsity-inducing l 1 -norm g 1 := α g α, instead of the nasty" cardinality pseudo-norm 0? In other words, Is g (d) also an optimal solution of the optimization problem P 1 : inf g { g 1 : vol (G) ρ d ; g Hom[x] d }, where the minimum is taken over all homogeneous polynomials of degree d, and ρ d denotes the Lebesgue volume of the L d -unit ball B d.

Question II: What is an optimal solution if one now minimizes the weighted l 2 -norm g 2 := α c α g α 2, c α = ( i α i)! α 1! α n! instead of the l 1 -norm 1? In other words, What is an optimal solution of the optimization problem P 2 : inf g { g 2 : vol (G) ρ d ; g Hom[x] d }, where the minimum is taken over all homogeneous polynomials of degree d, and ρ d denotes the Lebesgue volume of the L d -unit ball B d.

Given d N, let v d (x) := (x α ) be the vector of all monomials x α of degree α = d. Restrict to SOS homogeneous polynomials of degree 2d x g Q (x) := v d (x) T Qv d (x), for some real symmetric Q 0. Question III: What is an optimal solution if one minimizes the small rank-inducing norm trace(q) (also known as the nuclear norm of Q) instead of the l 1 -norm g Q 1? In other words, What is an optimal solution of the optimization problem P 3 : inf { trace(q) : vol (G Q) ρ d ; g Hom[x] d }, Q 0 One expect a solution g Q which consists of a small number of squares.

Finally we will consider generalized" homogeneous polynomials x g(x) := α g α x α, α Q, and investigate problem P 1 : inf g { g 1 : vol (G) ρ d }, in this new context.

A little detour g : R n R is a positively homogeneous (PH) function of degree" d R if for every λ > 0, g(λ x) = λ d g(x), x. g : R n R is a homogeneous function of degree d R if for every λ, g(λ x) = λ d g(x), x. For instance x x is positively homogeneous of degree d = 1 (but not homogeneous).

Theorem Let g be a nonnegative PH of degree 0 < d R such that G := {x : g(x) 1} has finite Lebesgue volume: vol ({x : g(x) y}) = y n/d exp( g) dx. Γ(1 + n/d) R n Was already proved in Morosov & Shakirov 1 in the context of homogeneous polynomials and Integral discriminants. Also extends to quasi-homogeneous polynomials 1 Morosov & Shakirov. Introduction to integral discriminants, J. High Energy Phys. 12 (2009)

A simple proof via Laplace transform Let y v(y) := vol({x : g(x) y}). As g is PH of degree d then so is v with degree n/d. Moreover v(y) = 0 for all y < 0. Therefore v(y) = y n/d v(1) = y n/d vol({x : g(x) 1}). In particular, its Laplace transform λ L v (λ) satisfies: L v (λ) = 0 = v(1) exp( λy)v(y) dy 0 exp( λy)y n/d dy = v(1) Γ(1 + n/d) λ 1+n/d.

On the other hand, for 0 < λ R, ( L v (λ) = exp( λy) and so 0 {x:g(x) y} ( ) = exp( λy)dy dx [by Fubini] R n g(x) = 1 exp( λg(x)) dx λ R n 1 = λ 1+n/d exp( g(z)) dz [by homogenity] R n vol({x : g(x) 1}) = v(1) = 1 Γ(1 + n/d) dx ) dy R n exp( g(z)) dz.

In particular, computing the non Gaussian integral exp( g) dx reduces to computing the volume of the level set G := {x : g(x) 1},... which is the same as solving the optimization problem: where : max µ µ(g) s.t. µ + ν = λ µ(b \ G) = 0 B is a box [ a, a] n containing G and λ is the Lebesgue measure.

... we know how to approximate as closely as desired µ(g) and any FIXED number of moments of µ, by solving an appropriate hierarchy of semidefinite programs (SDP). via... the moment-lp or the moment-sos appoach

Convex Optimization & Parsimony of Lp -balls representation

... But the resulting SDPs can be numerically difficult to solve Solving the dual reduces to approximating the indicator function I(G) by polynomials of increasing degrees Gibbs effect, etc. However some tricks can help a lot! (see: Approximate volume and integration for basic semi algebraic sets, Henrion, Lasserre and Savorgnan, SIAM Review 51, 2009.)

III. Convexity An interesting issue is to analyze how the Lebesgue volume vol {x R n : g(x) 1}, (i.e. vol (G)) changes with g. Corollary Let C d R[x] d be the convex cone of homogeneous polynomials of degree at most d such that vol(g) is finite. Then the function f : C d R, g f (g) := vol(g) = dx, g C d, G is a PHF of degree n/d, strictly convex on C d.

Convexity (continued) Corollary (continued) Moreover, if g int(c d ) then f is differentiable and: f (g) 1 = x α exp( g) dx g α Γ(1 + n/d) R n = d + n x α dx d G 2 f (g) 1 = x α+β exp( g) dx g α g β Γ(1 + n/d) R n 2n + 3d = x α+β dx d G

A consequence of convexity Recall that if K R n is compact then computing the ellipsoid ξ of minimum volume containing K is a classical problem whose optimal solution is called the Löwner-John ellipsoid. So consider the following natural extension of Löwner-John s problem: Find an homogeneous polynomial g R[x] 2d, g C 2d, such that its sub level set G := {x : g(x) 1} contains K and has minimum volume among all such levels sets with this inclusion property. A convex optimization problem with a unique optimal solution, with characterization as in the Löwner-John problem! Lass: Math. Program. 152 (2015), 559 591.

But this strict convexity of the function g f (g) := vol(g) has other consequences... Indeed... Combining this strict convexity property with the standard KKT-optimality conditions in convex optimization,... provides us with the tool to solve our initial problems P 1, P 2 and P 3.

Back to problem P 1 With g 1 the l 1 -norm of coefficients of g R[x] d, solve P 1 : inf g { g 1 : vol (G) ρ d ; g Hom[x] d }. Theorem (Lass (2015)) The L d -unit ball polynomial x g (d) (x) = n i=1 x 2d i, x R n, is the unique optimal solution of P 1 and of ˆP 1 : inf g { vol(g) : g 1 n; g Hom[x] d }.

Sketch of the proof Rewrite P 1 as: λ α : inf λ,g α s.t. λ α ±g α, α : α = d g α = 0, α : α < d R n exp( g(x))dx Γ(1 + n/d) ρ d Write the KKT-optimality conditions at a candidate g int(c d ). Convex problem and Slater s condition holds true implies that the KKT-optimality conditions at g int(c d ) are both necessary and sufficient for optimality. Show that g (d) satisfies the KKT-optimality conditions.

For this we use the explicit expression for the gradient f (g (d) ) of g f (g) = vol(g). Recall that where f (g (d) ) g α = n + d d B d x α dx, α = d, B d = {x : n i=1 x d i 1 } = {x : g (d) (x) 1 }. We also need the inequality x α dx x1 B d B d d dx, α = d.

The result extends L d -balls B d = {x : n i=1 x i d 1}, where 0 < d is rational (and not necessarily d > 1, e.g. d = 1/2). For this we need introduce... generalized homogeneous polynomials" x g(x) := α g α n x i α i, x R n, i=1 where: There are finitely many non-zero coefficient (g α ). The α s are positive rationals with n i=1 α i = d

A little detour With q N, define the lattice Z n q := {α Z n : q α Z n } and with 0 < d Z q, the finite set N n d,q := { 0 α Zn q : α i d }, and the finite-dimensional vector space of Homogeneous functions i Hom[x] q d := { α N n d,q g α n x i α i : g α R}. i=1 I. We then consider P 1q : inf g { g 1 : vol (G) ρ d ; g Hom[x] q d }.

II. We next prove that the L d -generalized polynomial x g (d) (x) := n x i d, x R n, i=1 is the unique optimal solution of P 1q. Indeed with g Hom[x] q d and G = {x : g(x) 1}, the crucial feature which we exploit is that the function f : Hom[x] q d R: g f (g) := G dx = 1 Γ(1 + n/d) R n exp( g(x)) dx, is strictly convex on the interior of its domain... with gradient f (g) = n + d x α dx g α n G

Back to Problem P 2 Recall that with P 2 : inf g { g 2 : vol (G) ρ d ; g Hom[x] d }, g 2 := α c α g α 2, c α = ( i α i)! α 1! α n! Theorem (Lass (2015)) Problem P 2 has a unique optimal solution g 2 Hom[x] d which a sum of d-powers of linear forms, i.e.: g 2 (x) = s θ k z k, x d, x R n, k=1 for some positive weights (θ k ) and points (z k ) R n, k = 1,..., s with s ( ) n+d 1 d

Sketch of proof As P 1, problem P 2 is a convex optimization problem and Slater s condition holds. Uniqueness follows from the strict convexity of the volume function. Write g 2 (x) = α =d α! α 1! α n! g 2,α xα. As for problem P 1 one writes the KKT-optimality conditions at a candidate optimal solution g2 Hom[x] d where f is differentiable. This yields x α dx g2,α = M G, α = d. dx G

Which implies that (g2,α ) is the moment vector of a probability measure, hence an element of the DUAL of the space of nonnegative homogeneous polynomials of degree d. But this also implies that g2 is a positive sum of d-powers of linear forms! Theorem (Lass (2015)) For d = 2, 4, 6, 8 g 2 (x) = θ ( n i=1 ) d/2 ( d/2 xi 2 = θ g (x)) (2), for some scaling factor θ > 0 and so G is homothetic to the Euclidean ball!! With n = 2, d = 4, g 2 (x) = (x 2 1 + x 2 2 )2 = 1 6 (x 1 + x 2 ) 4 + 1 6 (x 1 x 2 ) 4 + 2 3 (x 4 1 + x 4 2 ).

Problem III Given d N, let v d (x) := (x α ) be the vector of all monomials x α of degree α = d. Restrict to SOS homogeneous polynomials of degree 2d x g Q (x) := v d (x) T Qv d (x), for some real symmetric Q 0. Question III: What is an optimal solution if one minimizes the small rank-inducing norm trace(q) (also known as the nuclear norm of Q) instead of the l 1 -norm g Q 1?

In other words, What is an optimal solution of the optimization problem P 3 : inf { trace(q) : vol (G Q) ρ d ; g Hom[x] d }, Q 0 One expect a solution g Q which consists of a small number of squares.

Recall that C d Hom[x] d is the convex cone of homogeneous polynomials g of degree d such that vol (G) <. For symmetric matrices Q 0 such that g Q int(c d ) Q v(q) := vol (G Q ); g Q int(c d ). Then v(q) = n n + d G Q v d/2 (x)v d/2 (x) T dx.

Theorem Problem P 3 has an optimal solution and there exists a unique homgeneous polynomial g3 Hom[x] d associated with any solution Q of P 3. In addition: Ψ := I (n + d) trace(q ) n ρ d and trace(q Ψ) = 0. G Q v d/2 (x)v d/2 (x) T dx 0 Conversely if Q satisfies the above with g Q int(c d ) and v(q ) = ρ d then Q is an optimal solution of P 3 and g 3 = g Q.

Theorem (continued) In particular for d = 2, 4 and for all d = 4N (provided that n is sufficiently large) the unique optimal solution g 3 Hom[x] d is the polynomial x g 3 (x) = (ρ 2/ρ d ) d/n ( n i=1 x 2 i ) d/2. That is, the Euclidean ball is the unique optimal solution!

Some references J.B. Lasserre. A Generalization of Löwner-John s ellipsoid Theorem, Math. Program. 152 (2015) J.B. Lasserre. Convex Optimization and Parsimony of L p -balls representation, SIAM J. Optim. (2016)

THANK YOU!!