Near-Optimal Linear Recovery from Indirect Observations

Transcription

1 Near-Optimal Linear Recovery from Indirect Observations joint work with A. Nemirovski, Georgia Tech nemirovs/statopt LN.pdf Les Houches, April, 2017

2 Situation: In the nature there exists a signal x known to belong to a given convex compact set X R n. We observe corrupted by noise affine image of the signal: ω = Ax + σξ Ω = R m A: given m n sensing matrix ξ: random observation noise Our goal is to recover the image Bx of x under a given affine mapping B: R n R ν. Risk of a candidate estimate x( ) : Ω R ν is defined as { } Risk[ x X ] = sup E ξ Bx x(ax + σξ) 2 2 x X Risk 2 is the worst-case, over x X, expected 2 2 recovery error

3 Agenda: Under appropriate assumptions on X, we are to show that One can build, in a computationally efficient fashion, the (nearly) best, in terms of risk, estimate from the family of linear estimates x(ω) = x H (ω) = H T ω [H R m ν ] The resulting linear estimate is nearly optimal among all estimates, linear and nonlinear alike

4 Linear estimation of signal in Gaussian noise... Kuks & Olman, 1971, 1972 Rao 1972, 1973, Pilz, 1981, 1986,..., Drygas, 1996, Christopeit & Helmes, 1996, Arnold & Stahlecker, 2000,... Pinsker 1980, Efromovich & Pinsker, 1981, 1982, Efromovich & Pinsker 1996, Golubev, Levit & Tsybakov, 1996,..., Efromovich, 2008,... Donoho, Liu, McGibbon,

5 Risk of linear estimation Assuming that ξ is zero mean with unit covariance matrix, we can easily compute the risk of a linear estimate x H (ω) = H T ω Risk 2 [ x H X ] = max E { ξ [B H T A]x σh T ξ 2} 2 x X { = max [B H T A]x x X σ2 E ξ {Tr(H T ξξ T H} } = σ 2 Tr(H T H) + max x X Tr(xxT [B T A T H][B H T A]). Note: building the minimum risk linear estimate reduces to solving convex minimization problem [ min H φ(h) := max x X Tr(xxT [B T A T H][B H T A]) + σ 2 Tr(H T H) ]. ( ) Convex function φ is given implicitly and can be difficult to compute, making ( ) difficult as well

6 Fact: essentially, the only cases when ( ) is known to be easy are those when X is given as a convex hull of finite set of moderate cardinality X is an ellipsoid: for W S n and S 0 ( ) max x T Sx 1 Tr(xxT W ) = λ max S 1/2 W S 1/2. where λ max ( ) is the maximal eigenvalue. When X is a box, computing φ is NP-hard... When φ is difficult to compute, we can to replace φ in the design problem ( ) with an efficiently computable convex upper bound ϕ(h). We are about to consider a family of sets X ellitopes for which reasonably tight bounds ϕ of desired type are available

7 An ellitope is a set X R n given as where X = {x R n : y R N, t T : x = P y, y T S k y t k, 1 k K} P is a given n N matrix (we can assume that P = I n ), S k 0 are positive semidefinite matrices with k S k 0 T is a convex compact subset of K-dimensional nonnegative orthant R K + such that T contains some positive vectors T is monotone: if 0 t t and t T, then t T as well. Note: every ellitope is a symmetric w.r.t. the origin convex compact set

8 Examples [A.] A centered at the origin ellipsoid (K = 1, T = [0; 1]) [B.] (Bounded) intersection of K ellispoids/elliptic cylinders centered at the origin (T = {t R K : 0 t k 1, k N}) [C.] Box {x R n : 1 x i 1} (T = {t R n : 0 t k 1, k K = n}, x T S k x = x 2 k ) [D.] X = {x R n : x p 1} with p 2 (T = {t R n + : t p/2 1}, x T S k x = x 2 k, k K = n) Ellitopes admit fully algorithmic calculus: if X i, 1 i I, are ellitopes, so are linear images of X i inverse linear images of X i under linear embeddings i X i X 1... X I X X I

9 Observation Let X = {x R n : t T : x T S k x t k, 1 k K} be an ellitope. Given a quadratic form x T W x, W S n, one has where max x X xt W x = max x X Tr(xxT W ) max Q Q Tr(QW ), We conclude that Q := {Q S n : Q 0, t T : Tr(QS k ) t k, k K}. and φ(h) ϕ(h) := σ 2 Tr(H T H) + max Q Q Tr( Q(A T H B T )(H T A B) ), Risk 2 [ x H X ] min H ϕ(h)

10 This attracts our attention to the optimization problem Opt P = min H { ϕ(h) = max Q Q Note that (P) is the primal problem [ [ σ 2 Tr(H T H) + Tr ( Q(A T H B T )(H T A B) ) ]}. (P ) }{{} Φ(H,Q) min H max Q Q Φ(H, Q) associated with the convex-concave saddle point function Φ(H, Q). problem associated with Φ(H, Q) is that is, the problem Opt D = max Q Q { ψ(q) := min H max Q Q ] [ ] min Φ(H, Q), H The dual [ σ 2 Tr(H T H) + Tr ( Q(A T H B T )(H T A B) )]}. (D) By the Sion-Kakutani theorem, (P) and (D) are solvable with equal optimal values: Opt D = Opt P = Opt

11 Note that the minimizer of Φ(, Q) can be easily computed: so that H(Q) = (σ 2 I m + AQA T ) 1 AQB T, ψ(q) = Tr ( B[Q QA T (σ 2 I m + AQA T ) 1 AQ]B T ), and the dual problem reads { Opt D = max Tr ( B[Q QA T (σ 2 I m + AQA T ) 1 AQ]B ) T, Q,t } (D) Q 0, t T, Tr(QS k ) t k, k K In fact, both (P) and (D) can be cast as Semidefinite Optimization problems. In particular, (P) can be rewritten as { [ Opt = min σ 2 Tr(H T H) + φ T (λ) : k λ ] } ks k B T A T H H,λ B H T 0, λ 0 A I ν where φ T : R K R is the support function of T : φ T (λ) = max t T λt t. Note that (P) is efficiently solvable whenever T is computationally tractable. (P )

12 Bottom line: Given matrices A R m n, B R k n and an ellitope X = {x R n : t T : x T S k x t k, 1 k K} ( ) consider the convex optimization problems Opt P = min H ϕ(h) and OptD = max Q Q ψ(q), where Q := {Q S n : Q 0, t : Tr(QS k ) t k, k K}. The optimal values of two problems coincide, Opt P = Opt D = Opt. When noise ξ satisfies E{ξ} = 0, and E{ξξ T } = I m, the risk of the linear estimate x H ( ) induced by the optimal solution H to the problem (this solution clearly exists provided that σ > 0) satisfies the risk bound Risk[ x H X ] Opt. We are to compare the bound Opt for the risk of x H to the minimax risk Risk Opt [X ] = inf Risk[ x X ]. x( )

13 Bayesian risks Minimax risk Risk Opt [X ] is defined as the worst, over the signals of interest, performance of x( ) Bayesian risk is the average performance, with the average taken over some prior probability distribution on the signals. For the problem of 2 -recovering Bx via noisy observation ω = Ax + σξ, ξ P this alternative reads as follows: (!) Given a probability distribution π of signals x R n, find an estimate x( ) which minimizes Risk 2 ( x π) := π { } Bx x(ax + σξ) 2 2 P (dξ) π(dx) R m the average, over the distribution π of signals x, of expected 2 2 estimation error of Bx via observation Ax + σξ

14 Let P x,ω be the induced by π and P ξ joint distribution of (x, ω = Ax + σξ) on R n x Rm ω. P x,ω gives rise to marginal distribution P ω of ω, conditional distribution P x ω of x given ω. We have Risk 2 ( x π) = Bx x(ω) 2 2 P x,ω(dx, dω) R n x Rm ω = R m { R n Bx x(ω) 2 2 P x ω(dx) } P ω (dω) Assuming that the probability distribution π possesses finite second moments, one has min x( ) R n R m Bx x(ω) 2 2 P x,ω(dx) = Bx x (ω) 2 2 P x,ω(dx), R n R m where x (ω) = BxP x ω (dx). R n

15 Corollary [Gauss-Markov theorem]: Let x R n and ξ R m be independent zeromean Gaussian random vectors. Assuming σ > 0 and the covariance matrix of ξ to be positive definite, conditional, given ω, distribution of x is normal, and the conditional expectation x (ω) is a linear function of ω, as a result, an optimal solution x ( ) to the risk minimization problem { } E x π,ξ Bx x(ax + σξ) 2 2 min x( ) exists and is a linear function of ω = Ax + σξ. In particular, when ξ N (0, I m ) and x N (0, Q), one has x (ω) = [ [σ 2 I m + AQA T )] 1 AQB ] T ω Risk 2 ( x N (0, Q)) = Tr ( B[Q QA T [σ 2 I m + AQA T ] 1 AQ]B ) T

16 Course of actions (Pinsker s program) Let N (0, Q) be a Gaussian prior for the signal x which sits on X with high probability. Then by the Gauss-Markov theorem the ( slightly reduced ) quantity ψ(q) = Tr(B[Q QA T [σ 2 I m + AQA T ] 1 AQ]B T ) would be a lower bound on Risk 2 Opt. Note that E η N (0,Q) {η T Sη} = Tr(SQ). Thus, selecting Q 0 according to t T : Tr(QS k ) t k, k K we ensure that η N (0, Q) sits in X on average. Imposing on Q 0 restriction t T : Tr(QS k ) ρt k, k K, [ρ > 0] we enforce η N (0, Q) to take values in X with probability controlled by ρ and approaching 1 as ρ

17 The above considerations give rise to parametric optimization problem Opt (ρ) = max Q 0 {ψ(q) : t T : Tr(QS k) ρt k, 1 k K} (P ρ ) We may expect that for small ρ a slightly corrected Opt (ρ) is a lower bound on Risk 2 Opt. As we have just seen, Opt (1) = Opt (!). Since the optimal value of the (concave) optimization problem (P ρ ) is a a concave function of ρ, we have Opt (ρ) ρopt, 0 < ρ < 1. Now, all we need is a simple result as follows: Lemma Let S and Q be positive semidefinite n n matrices with ρ := Tr(SQ) 1, and let η N (0, Q). Then Prob { η T Sη > 1 } 1 ρ+ρ ln(ρ) e 2ρ

18 We arrive at the following Theorem. Let us associate with ellitope X = {x R n : t T : x T S k x t k, k K} the convex compact set Q = {Q S n : Q 0, t T : Tr(QS k ) t k, k K}, and the quantity M = max Q Q Tr(BQB T ). Then the linear estimate x H (ω) = H T ω of Bx, x X, via observation ω = Ax+σξ, ξ N (0, I m ), given by the optimal solution H to the convex optimization problem { } λ Opt = min φ T (λ) + σ 2 Tr(HH T [ 0 ) : k λ ks k B T A T H H,λ 0 satisfies the risk bound Risk[ x H X ] Opt ( 6 ln B H T A I k ] 8M 2 K Risk 2 Opt [X ] ) Risk Opt [X ]

19 Numerical illustration In these experiments B is n n identity matrix, n n sensing matrix A is a randomly rotated matrix with singular values λ j, 1 j n, forming a geometric progression, with λ 1 = 1 and λ n = In the first experiment the signal set X 1 is an ellipsoid: n X 1 = {x R n : j 2 x 2 j 1}, that is, K = 1, S 1 = n j=1 j2 e j e T j (e j are basic orths), and T = [0, 1]. Theoretical suboptimality factor in the interval [31.6, 73.7] in this experiment. j=1 In the second experiment, the signal set X is the box: X = {x R n : j x j 1, 1 j n} [K = n, S k = k 2 e k e T k, k = 1,..., K, T = [0, 1]K ]. Theoretical suboptimality factor in the interval [73.2, 115.4]

20 Recovery on ellipsoids: risk bounds as functions of the noise level σ, dimension n = 32. Left plot: upper and lower bounds of the risk; right plot: suboptimality ratios Recovery on ellipsoids: risk bounds as functions of problem dimension n, noise level σ = Left plot: upper and lower risk bounds; right plot: suboptimality ratios.

21 Recovery on a box: risk bounds as functions of the noise level σ, dimension n = 32. Left plot: upper and lower bounds of the risk; right plot: suboptimality ratios Recovery on a box: risk bounds as functions of problem dimension n, noise level σ = Left plot: upper and lower risk bounds; right plot: suboptimality ratios.

22 Extensions 1. Relative risks When very large signals are allowed, one may switch from the usual risk to its relative version S-risk defined as follows: Given a positive semidefinite risk calibrating matrix S we set RiskS[ x X ] = min { τ : Eξ { Bx x(ax + σξ) 2 2} τ[1 + x T Sx] x X } Note: setting S = 0 recovers the usual plain risk. Results on design of near-optimal, in terms of plain risk, linear estimates extend directly to the case of S-risk

23 Design of near optimal linear estimate x H (ω) = H T ω is given by an optimal solution (H, τ, λ ) to the convex optimization problem { [ Opt = min τ : k λ ] } ks k + τs B T A T H 0, σ 2 Tr(HH T ) + φ H,τ,λ B H T T (λ) τ, λ 0 A I k For the resulting estimate, it holds RiskS[ x H X ] Opt, provided ξ is zero mean with unit covariance matrix. Near-optimality properties of the estimate x H remain the same as in the case of plain risk: when ξ N (0, I m ), one has ( ) RiskS[ x H X ] 8KM 2 6 ln RiskS 2 Opt [X ] RiskS Opt [X ], where { } M = max Q Tr(BQB T ) : Q 0, t T : Tr(QS k ) t k, 1 k K, and RiskS Opt [X ] = inf RiskS[ x X ]. x( )

24 In the case X = R n, the best linear estimate is yielded by the optimal solution to the convex problem Opt = min H,τ { τ : [ ] τs B T A T H B H T A I k } 0, σ 2 Tr(HH T ) τ ( ) A feasible solution τ, H to ( ) gives rise to linear estimate x H (ω) = H T ω such that RiskS[ x H R n ] τ, provided ξ is zero mean with unit covariance matrix. Proposition Assume that B 0 and ( ) is feasible. Then the problem is solvable, and its optimal solution Opt, H gives rise to the linear estimate x H (ω) = H T ω with S-risk Opt. When ξ N (0, I m ), this estimate is minimax optimal: RiskS[ x H R n ] = Opt = RiskS Opt [R n ]

25 2. Spectratopes We say that a set X R n is a basic spectratope, if it can be represented in the form X = { x R n : t T : Rk 2 [x] t ki dk, 1 k K } where [S 1 ] R k [x] = n i=1 x ir ki are symmetric d k d k matrices linearly depending on x R n (i.e., matrix coefficients R ki belong to S n ) [S 2 ] T R K + is a convex compact subset of RK + and is monotone: which contains a positive vector 0 t t T t T. [S 3 ] Whenever x 0, it holds R k [x] 0 for at least one k K. A spectratope is a linear image Y = P X of a basic spectratope. We refer to D = k d k as size of the spectratope Y

26 Examples [A.] Any ellitope is a spectratope. [B.] Let L be a positive definite d d matrix. Then the matrix box X = {X S d : L X L} = {X S d : R 2 [X] := [L 1/2 XL 1/2 ] 2 I d } is a basic spectratope. As a result, a bounded set X R n given by a system of two-sided LMI s, specifically, X = {x R n : t T : t k L k S k [x] t l L k, 1 k K} where S k [x] are symmetric d k d k matrices linearly depending on x, L k 0 and T satisfies S 2, is a basic spectratope: X = {x R n : t T : R 2 k [x] t ki dk, k K} [R k [x] = L 1/2 k S k [x]l 1/2 k ] Same as ellitopes, spectratopes admit fully algorithmic calculus

27 Bounding quadratic forms over ellitopes Proposition Let G be a symmetric n n matrix, X R n be given by spectratopic representation, and let and Opt = where R k (Λ) : Sd k Opt = max x X xt Gx min Λ={Λ k } k K { φt (λ[λ]) : Λ k 0, P T GP k R k [Λ k] } (QP R) Sn is the conjugate linear mapping, [R k (Λ)] ij = 1 2 Tr ( Λ[R ki R kj + R kj R ki ] ), 1 i, j n, φ T is the support function of T, and for Λ = {Λ k S d k } k K, λ[λ] = [Tr[Λ 1 ];...; Tr[Λ K ]]. Then (QPR) is solvable, and Opt Opt 2 max[ln(2d), 1]Opt

28 Remark The result of the proposition has some history. Nemirovski, Roos and Terlaky, 1999 X is an intersection of centered at the origin ellipsoids/elliptic cylinders J. and Nemirovski 2016 X is ellitope, with tighter bound Opt Opt 4 ln(5k)opt. Note that in the case of an ellitope, (QPR) results in a somewhat worse suboptimality factor O(1) ln( K k=1 Rank(S k))

29 Building linear estimate Proposition Consider convex optimization problem { (B H T A) T (B H T A) Opt = min τ : k H,Λ,τ σ 2 Tr(H T H) + φ T (λ[λ]) τ R k (Λ k) Problem ( ) is solvable, and its feasible solution (H, λ, τ) induces a linear estimate x H = H T ω of Bx, x X, via observation ω = Ax + σξ, ξ N (0, I) with the maximal over X risk not exceeding τ. } ( )

30 Proposition Let X be a spectratope, and let Q = {Q S n + : t T : R k[q] t k I dk, k K}. The set Q is a nonempty convex compact set containing a neighbourhood of the origin, so that the quantity is well defined and positive. M = max Q Q Tr(BQBT ), The efficiently computable linear estimate x H (ω) = H T solution of ( ) is nearly optimal in terms of the risk: ( ) Risk[ x H X ] 2 8DM 2 2 ln RiskS 2 Opt [X ] Risk Opt [X ], where Risk Opt [X ] = inf Risk[ x X ] x( ) is the minimax risk associated with X, and D = k d k. ω yielded by the optimal

31 3. Norms We say that the norm is spectratopic-representable if the unit ball B of the conjugate norm is a spectratope: B = MY, Y = { x R n : r R : Sl 2 [x] r li fl, 1 l L }, where S l R f l f l, r l, l = 1,..., L and R is a valid spectratopic data. We denote F = l f l the size of B. Examples p -norm with 1 p 2 B is the unit ball of q -norm with 1 p + 1 q = B is an affine image of the the direct product of unit balls of norms and 2 combined norm min x=u+v M 1 u 1 + M 2 v 2 B is the intersection B B2 of unit balls of norms M T 1 and M T 2 2 nuclear norm Sh,1 B is the unit ball of the spectral norm Sh,... spectral norm Sh, is difficult

32 For an estimate x of Bx, let Risk [ x X ] = sup x X E ξ N (0,Im ){ Bx x(ax + σξ) }. Proposition Consider the convex optimization problem { Opt = min H,Λ,Υ,Υ,Θ φ T (λ[λ]) + φ R (λ[υ]) + φ R (λ[υ ]) + σtr(θ) : Λ = {Λ k 0, k K}, Υ [ = {Υ l 0, l L}, Υ = {Υ l 0, l L}, k R k [Λ ] 1 k] 2 [BT A T H]M 1 2 M T [B H T A] l S l [Υ 0, [ l] 1 Θ 2 HM ] 1 2 M T H T l S l [Υ l ] 0. Here for Λ = {Λ i S m i } i I λ[λ] = [Tr[Λ 1 ];...; Tr[Λ I ]], and φ T [R k [Λ k]] ij = 1 2 Tr(Λ k[r ki k Rkj k [S l [Υ l]] ij = 1 2 Tr(Υ l[s li l Slj l + Rkj k Rki + S lj l Sli k ]), where R k[x] = i x ir ki, l ]), where S l[y] = i y is li, and φ R are the support function of T and R. The problem is solvable, and the H-component H of its optimal solution yields linear estimate x H (ω) = H T ω such that Risk [ x( ) X ] Opt

33 Near-optimality of linear estimation on spectratopes Proposition Let M 2 = max W { Eη N (0,In ) BW 1/2 η 2 : W Q := {W S n + : t T : R k[w ] t k I dk, 1 k K} }. Then there is an efficiently computable linear estimate x H = H ω which satisfies ( ) 2DM 2 Risk [ x H X ] Opt C ln(2f ) ln Risk 2 Risk [X ],Opt[X ], where C is a positive absolute constant, [ ] Risk Opt [X ] = inf sup E ξ N (0,Im ){ Bx x(ax + σξ) } x( ) x X the infimum being taken over all estimates, and D = k d k, F = l f l

34 The key component Lemma Let Y be an N ν matrix, let be a norm on R ν such that the unit ball B of the conjugate norm is the spectratope, and let ζ N (0, Q) for some positive semidefinte N N matrix Q. Then the upper bound on φ Q (Y ) := E{ Y T ζ } yielded by the SDP relaxation, that is, the optimal value Opt[Q] of the convex optimization problem { Opt[Q] = min Θ,Υ is tight, namely, φ R (λ[υ]) + Tr(Θ) : Υ = {Υ l 0, 1 l L}, Θ S m, [ ] } 1 Θ 2 Q1/2 Y M 1 2 M T Y T Q 1/2 l S l [Υ 0 l] ψ Q (Y ) Opt[Q] ( 8F 4 ln 2 e 1/4 2 e 1/4 where F = l f l is the size of the spectratope B ) ψ Q (Y ),