Lecture 4: Linear and quadratic problems linear programming examples and applications linear fractional programming quadratic optimization problems (quadratically constrained) quadratic programming second-order cone programming examples and applications 1
Linear program (LP) linear program: minimize c T x subject to Gx h, Ax = b P c xopt standard form LP minimize c T x subject to Ax = b, x 0 c T x constant (widely used in LP literature & software) variations: e.g.,, maximize c T x subject to Ax b 2
force/moment generation with thrusters rigid body with center of mass at origin p = 0 R 2 n forces with magnitude u i, acting at p i = (p ix,p iy ), in direction θ i θ i (p ix,p iy ) u i resulting horizontal force: F x = n i=1 u icosθ i resulting vertical force: F y = n i=1 u isinθ i resulting torque: T = n i=1 (p iyu i cosθ i p ix u i sinθ i ) force limits: 0 u i 1 (thrusters) fuel usage: u 1 + + u n problem: find u i that yield given desired forces and torques and minimize fuel usage 3
can be expressed as LP: minimize 1 T u subject to Fu = f des 0 u i 1, i = 1,...,n where F = cosθ 1 cosθ n sinθ 1 sinθ n, p 1y cosθ 1 p 1x sinθ 1 p ny cosθ n p nx sinθ n f des = (F des x, Fdes y, Tdes ), 1 = ( 1, 1, 1 ) 4
Converting LP to standard form inequality constraints: write a T i x b i as a T i x + s i = b i, s i 0 s i is called slack variable associated with a T i x b i unconstrained variables: write x i R as x i = x + i x i, x+ i,x i 0 Example. thruster problem in standard form [ minimize 1 T 0 ][ ] u s [ ] [ u F 0 subject to 0, s I I ][ u s ] = [ f des 1 ] 5
Piecewise-linear minimization minimize max i (c T i x + d i) subject to Ax b max i (c T i x + d i ) c T i x + d i x express as minimize t subject to c T i x + d i t, Ax b an LP in variables x R n, t R 6
Blind Channel Equalization via Linear Programming H(!) G(!) LTI channel; impulse response h: complex, unknown; QAM modulation Goal: design equalizer G(ω) such that H(ω)G(ω) = e jdω, where D is delay. Let x i be the channel output, and y k be the equalizer output. It is known that minimize max k Re(y k ) subject to Re(g 1 ) + Im(g 1 ) = 1 channel equalization. where Re(g 1 ) + Im(g 1 ) = 1 is a normalizing constraint. Note that Re(y k )= N i= N Re(x k i )Re(g i ) Im(x k i )Im(g i ) is linear in g. 7
Blind Channel Equalization via Linear Programming Note that max k { Re(y k ) } is nonsmooth in g. Smoothing approach: approximating the -norm by p-norm, with p large, max{ Re(y k ) } k ( ) 1/p Re(y k ) p and then remove the exponent 1 p to obtain a smooth objective. Gradient descent. Linear programming formulation: minimize t subject to Re(g 1 ) + Im(g 1 ) = 1 N t Re(x k i )Re(g i ) Im(x k i )Im(g i ) t, k i= N The above formulation requires fewer samples, and enjoys faster convergence. k 8
Blind Channel Equalization via Linear Programming 10 1 0.5 5 0 0.5 0 5 0 10 10 5 0 5 10 (a) Channel output 0.5 0 5 10 15 20 (b) Real part 0.5 0 5 10 15 20 (c) Imaginary part 4 2 1 0.5 1 0.5 0 0 0 2 0.5 0.5 4 4 2 0 2 4 (d) Equalized output 1 0 10 20 30 (b) (e) Real part 1 0 10 20 30 (f) Imaginary part 9
l - and l 1 -norm approximation constrained l - (Chebychev) approximation minimize Ax b subject to Fx g write as minimize t subject to Ax b t1, Ax b t1 Fx g constrained l 1 -approximation minimize Ax b 1 subject to Fx g write as minimize 1 T y subject to Ax b y, Ax b y Fx g 10
Extensions of thruster problem opposing thruster pairs minimize i u i subject to Fu = f des u i 1, i = 1,...,n can express as LP given f des, can express as LP minimize Fu f des subject to 0 u i 1, i = 1,...,n given f des, minimize # thrusters on subject to Fu = f des 0 u i 1, i = 1,...,n can t express as LP (# thrusters on is quasiconcave!) 11
Design centering find largest ball inside a polyhedron P = {x a T i x b i, i = 1,...,m} P x c center is called Chebychev center ball {x c + u u r} lies in P if and only if i.e.,, sup{a T i x c + a T i u u r} b i, i = 1,...,m, hence, finding Chebychev center is an LP: a T i x c + r a i b i, i = 1,...,m maximize r subject to a T i x c + r a i b i, i = 1,...,m 12
Linear fractional program objective function is quasiconvex sublevel sets are polyhedra like LP, can be solved very efficiently c T x + d minimize f T x + g subject to Ax b, f T x + g > 0 extension: c T i minimize max x + d i i=1,...,k fi Tx + g i subject to Ax b, f T i x + g i > 0, i = 1,...,K objective function is quasiconvex sublevel sets are polyhedra 13
state space model: Minimum-time optimal control x(t + 1) = Ax(t) + Bu(t), t = 0,1,...,K x(0) = x 0, u(t) 1, t = 0,1,...,K variables: u(0),...,u(k) settling time f(u(0),...,u(k)) is inf {T x(t) = 0 for T t K + 1} f is quasiconvex function of (u(0),...,u(k)): f(u(0),u(1),...,u(k)) T if and only if for all t = T,...,K + 1 x(t) = A t x 0 + A t 1 Bu(0) + + Bu(t 1) = 0 min-time optimal control problem: minimize f(u(0),u(1),...,u(k)) subject to u(t) 1, t = 0,...,K 14
Min-time control example three unit masses, connected by two unit springs with equilibrium length one, u(t) R 2 is force on left & right mass over time interval (0.15t,0.15(t + 1)] u 1 (t) u 2 (t) problem: pick u(0),...,u(k) to bring masses to positions ( 1,0,1) (at rest), as quickly as possible, from initial condition ( 3, 0, 2) (at rest) optimal solution: u 1 u 2 positions 1 0 1 1 0 1 2 1 0 1 2 3 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 4 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 t t t right mass center mass left mass 15
Optimal transmitter power allocation m transmitters, mn receivers all at same frequency transmitter i wants to transmit to n receivers labeled (i,j), j = 1,...,n transmitter k receiver (i, j) transmitter i A ijk is path gain from transmitter k to receiver (i,j) N ij is (self) noise power of receiver (i,j) variables: transmitter powers p k, k = 1,...,m 16
at receiver (i, j): signal power: noise plus interference power: S ij = A iji p i I ij = k i A ijk p k + N ij signal to interference/noise ratio (SINR): S ij /I ij problem: choose p i to maximize smallest SINR: A iji p i maximize min i,j k i A ijkp k + N ij subject to 0 p i p max... a (generalized) linear fractional program 17
Nonconvex extensions of LP Boolean LP or zero-one LP: minimize c T x subject to Ax b Fx = g x i {0,1} integer LP: these are in general minimize c T x subject to Ax b Fx = g x i not convex problems extremely difficult to solve 18
Quadratic functions and forms quadratic function f(x) = x T Px + 2q T x + r [ ] T [ ][ x P q x = 1 q T r 1 ] convex if and only if P 0 quadratic form f(x) = x T Px convex if and only if P 0 Euclidean norm f(x) = Ax + b (f 2 is a convex quadratic function... ) 19
Minimizing a quadratic function minimize f(x) = x T Px + 2q T x + r nonconvex case (P 0): unbounded below proof: take x = tv, t, where Pv = λv, λ < 0 convex case (P 0): x is optimal if and only if two cases: q range(p): f > q range(p): unbounded below f(x) = 2Px + 2q = 0 important special case, P 0: unique optimal point x opt = P 1 q; f = r q T P 1 q 20
Least-squares minimize Euclidean norm (squared) (A = [a 1 a n ] full rank, skinny) minimize Ax b 2 = x T (A T A)x 2b T x + b T b geometrically: project b on span({a 1,...,a n }) b 0 x ls span({a 1,...,a n }) solution: set gradient equal to zero x ls = (A T A) 1 A T b general solution, without rank assumption: x ls = A b + v A is Moore-Penrose inverse of A, v N(A) 21
Least-norm solution of linear equations (A full rank, fat) minimize x 2 subject to Ax = b feasible if b R(A) geometrically: project 0 on {x Ax = b} 0 x ln {x Ax = b} solution: x ln = A T (AA T ) 1 b general solution, without rank assumption: x ln = A b 22
Extension: linearly constrained least-squares minimize Ax b subject to Cx = d (C R r s full rank, skinny) can be solved by elimination: write {x Cx = d} = {Fw + g w R s r } span(f) = nullspace(c) Cg = d (any solution) and solve (may or may not be a good idea in practice) minimize AFw + Ag b 23
Minimizing a linear function with quadratic constraint minimize c T x subject to x T Ax 1 x opt (A = A T 0) c x opt = A 1 c/ c T A 1 c proof. Change of variables y = A 1/2 x, c = A 1/2 c minimize c T y subject to y T y 1 optimal solution: y opt = c/ c Note: the problem is easy even without the condition A 0 24
Quadratic program (QP) quadratic objective, linear inequalities & equalities minimize x T Px + 2q T x + r subject to Ax b, Fx = g convex optimization problem if P 0 very hard problem if P 0 25
QCQP and SOCP quadratically constrained quadratic programming (QCQP): minimize x T P 0 x + 2q T 0 x + r 0 subject to x T P i x + 2q T i x + r i 0, i = 1,...,L convex if P i 0, i = 0,...,L nonconvex QCQP very difficult second-order cone programming (SOCP): minimize c T x subject to A i x + b i e T i x + d i, i = 1,...,L includes QCQP (QP, LP) 26
Robust linear program linear program minimize c T x subject to a T i x b i, i = 1,...,m suppose a i are uncertain robust linear program Note a i E i = {a i + F i u u 1} minimize c T x subject to a T i x b i a i E i, i = 1,...,m a T i x b i a i E i a T i x + ut F T i x b i u 1 a T i x + F ix b i hence, robust LP is SOCP minimize c T x subject to a T i x + F ix b i, i = 1,...,m 27
Phased-array antenna beamforming (x i,y i ) θ omnidirectional antenna elements at positions (x 1,y 1 ), (x 2,y 2 ),..., (x n,y n ) unit plane wave incident from angle θ induces in ith element a signal e j(x i cosθ+y i sinθ ωt), (j = 1, frequency ω, wavelength 2π) demodulate to get output e j(x i cosθ+y i sinθ) (assuming noise/interference free) linearly combine with complex weights w i : y(θ) = n i=1 w ie j(x i cosθ+y i sinθ) y(θ) is (complex) antenna array gain pattern or beam pattern y(θ) gives sensitivity of array as function of incident angle θ depends on design variables Re w, Im w (called antenna array weights) design problem: choose w to achieve desired gain pattern 28
Sidelobe level minimization make y(θ) small for θ θ tar > α (θ tar : target direction; 2α: beamwidth) sidelobe level y(θ) 50 θtar = 30 10 via least-squares (discretize angles) minimize i y(θ i) 2 subject to y(θ tar ) = 1 (sum over angles outside beam) least-squares problem with two (real) linear equality constraints 29
minimize sidelobe level (discretize angles) (max over angles outside beam) minimize max i y(θ i ) subject to y(θ tar ) = 1 can be cast as SOCP minimize t subject to y(θ i ) t y(θ tar ) = 1 y(θ) 50 θtar = 30 sidelobe level 10 30
Other variants convex (& quasiconvex) extensions: y(θ 0 ) = 0 (null in direction θ 0 ) w is real (amplitude only shading) w i 1 (attenuation only shading) minimize σ 2 n i=1 w i 2 (thermal noise power in y) minimize beamwidth given a maximum sidelobe level nonconvex extension: maximize number of zero weights 31
Robust adaptive beamforming In the presence of noise and interference, the received signal is given by x(θ,t) = s(t)a(θ) + i(θ,t) + n(t) s(t) is the desired signal, a(θ) is the steering vector of the desired user; in the ideal case a(θ) = e j(x i cosθ+y i sinθ), estimated in practice i(θ,t) is the interference signal; n(t) is noise beamformer output: y(θ,t) = w H x(θ,t) minimum variance distortionless response (MVDR) beamformer: minimize w H R x w, subject to w H ā = 1 where R x is the data covariance matrix, ā is the estimated steering vector. robust MVDR as a SOCP: minimize w H R x w, subject to Re(w H (ā + a)) 1, a ɛ minimize w H R x w, subject to Re(w H ā) 1 + ɛ w 32
Optimal receiver location N transmitter frequencies 1,...,N transmitters @ a i, b i use frequencyi(a i, b i R 2 ) transmitters @ a 1, a 2,..., a N are wanted transmitters @ b 1, b 2,..., b N are interfering x a 3 a 2 a 1 b 1 b 2 b 3 (signal) receiver power from a i : x a i α (interfering) receiver power from b i : x b i α (α 2.1) worst signal to interference ratio as a function of receiver position x: S/I = min i x a i α x b i α what is the optimal receiver location? 33
S/I is quasiconcave on {x S/I 1}, i.e.,, on {x x a i x b i, i = 1,...,N} b 3 a 3 a 2 b 2 a 1 b 1 can use bisection; every iteration is a convex quadratic feasibility problem 34