Sparse analysis Lecture II: Hardness results for sparse approximation problems

Transcription

1 Sparse analysis Lecture II: Hardness results for sparse approximation problems Anna C. Gilbert Department of Mathematics University of Michigan

2 Sparse Problems Exact. Given a vector x R d and a complete dictionary Φ, solve arg min c 0 s.t. x = Φc c i.e., find a sparsest representation of x over Φ. Error. Given ɛ 0, solve arg min c 0 s.t. x Φc 2 ɛ c i.e., find a sparsest approximation of x that achieves error ɛ. Sparse. Given k 1, solve arg min x Φc 2 s.t. c 0 k c i.e., find the best approximation of x using k atoms.

3 Computational complexity How hard is it to solve these problems? How difficult are these problems as compared to well-studied problems; e.g., Primes, SAT? What is the asymptotic complexity of the problem? Is there a feasible algorithm that is guaranteed to solve the problem? Sparse approximation problems are at least as hard as SAT

4 Complexity theory: Decision problems Fundamental problem type: output is Yes or No RelPrime. Are a and b relatively prime? SAT. Boolean expression. Given a Boolean expression with and, or, not, variables, and parentheses only, is there some assignment of True and False to variables that makes expression True? D-Exact. Given a vector x R d, a complete d N dictionary Φ, and a sparsity parameter k, does there exist a vector c R N with c 0 k such that Φc = x? Can be used as a subroutine (partially) to solve Exact

5 Complexity theory: Languages Encode input as a finite string of 0s and 1s Definition A language L is a subset of the set of all (finite length) strings. Decision problem = language L Given input x, decide if x L or if x / L RelPrime = {binary encodings of pairs (a,b) s.t. gcd(a, b) = 1} Algorithm A L for deciding L { Yes A L (x) = No iff x L otherwise

6 Complexity theory: P P = polynomial time Definition P is the class of languages L that are decidable in polynomial time; i.e., there is an algorithm A L such that x L iff A L (x) = Yes (otherwise A L (x) = No) there is some n so that for all inputs x, the running time of A L on x is less than x n

7 Complexity theory: NP NP Not Polynomial time

8 Complexity theory: Verify vs. determine Verifying existence of k-sparse vector c with Φc = x easier than determining existence Given witness c, check (i) c 0 = k and (ii) Φc = x Checks can be done in time polynomial in size of Φ, x, k Definition A language L has a (polynomial time) verifier if there is an algorithm V such that L = {x w s.t. V accepts (x, w)} (and algorithm V runs in time polynomial in the length of x).

9 NP NP = nondeterministic polynomial time Definition NP is the class of languages L that have polynomial time verifiers.

10 NP NP = nondeterministic polynomial time Definition NP is the class of languages L that have polynomial time verifiers. P = membership decided efficiently NP = membership verified efficiently P = NP?

11 NP NP = nondeterministic polynomial time Definition NP is the class of languages L that have polynomial time verifiers. P = NP P = membership decided efficiently NP = membership verified efficiently P = NP? NP P

12 Complexity theory: Reductions Problem A (efficiently) reduces to B means a(n efficient) solution to B can be used to solve A (efficiently) If we have an algorithm to solve B, then we can use that algorithm to solve A; i.e., A is easier to solve than B reduces does not confer simplification here Definition A P B if there s polynomial time computable function f s.t. w A f (w) B. B at least as hard as A

13 Complexity theory: NP-hard Definition A NP-complete if (i) A NP and (ii) for all X NP, X P A. NP-hard B A NP-complete Definition B NP-hard if there is A NP-complete s.t. A P B. NP P X

14 Examples RelPrime Are a and b relatively prime? in P Euclidean algorithm, simple Primes Is x a prime number? in P highly non-trivial algorithm, does not determine factors Factor Factor x as a product of powers of primes. in NP not known to be NP-hard X3C Given a finite universe U, a collection X of subsets X 1, X 2,..., X N s.t. X i = 3 for each i, does X contain a disjoint collection of subsets whose union = U? NP-complete

15 NP-hardness Theorem Given an arbitrary redundant dictionary Φ, a signal x, and a sparsity parameter k, it is NP-hard to solve the sparse representation problem D-Exact. [Natarajan 95,Davis 97]

16 NP-hardness Theorem Given an arbitrary redundant dictionary Φ, a signal x, and a sparsity parameter k, it is NP-hard to solve the sparse representation problem D-Exact. [Natarajan 95,Davis 97] Corollary Sparse, Error, Exact are all NP-hard.

17 NP-hardness Theorem Given an arbitrary redundant dictionary Φ, a signal x, and a sparsity parameter k, it is NP-hard to solve the sparse representation problem D-Exact. [Natarajan 95,Davis 97] Corollary Sparse, Error, Exact are all NP-hard. Corollary Given an arbitrary redundant dictionary Φ and a signal x, it is NP-hard to approximate (in error) the solution of Sparse to within any factor. [Davis 97]

18 Exact Cover by 3-sets: X3C Definition Given a finite universe U, a collection X of subsets X 1, X 2,..., X N s.t. X i = 3 for each i, does X contain a disjoint collection of subsets whose union = U?

19 Exact Cover by 3-sets: X3C Definition Given a finite universe U, a collection X of subsets X 1, X 2,..., X N s.t. X i = 3 for each i, does X contain a disjoint collection of subsets whose union = U? NP-complete problem.

20 Exact Cover by 3-sets: X3C Definition Given a finite universe U, a collection X of subsets X 1, X 2,..., X N s.t. X i = 3 for each i, does X contain a disjoint collection of subsets whose union = U? NP-complete problem. X1 X2 X3 XN u

21 Proposition Any instance of X3C is reducible in polynomial time to D-Exact. X3C P D-Exact Proof. Let Ω = {1, 2,..., N} index Φ. Set ϕ i = 1 Xi. X1 X2 X3 XN Select x = (1, 1,..., 1), k = 1 3 U. Suppose have solution to X3C. Sufficient to check if Sparse solution has zero error. Assume solutions of X3C indexed by Λ. Set c opt = 1 Λ. Φc opt = x. u = Sparse solution has zero error and D-Exact returns Yes.

22 Proposition Any instance of X3C is reducible in polynomial time to D-Exact. X3C P D-Exact Proof. Let Ω = {1, 2,..., N} index Φ. Set ϕ i = 1 Xi. X1 X2 X3 XN Select x = (1, 1,..., 1), k = 1 3 U. Suppose have solution to X3C. Sufficient to check if Sparse solution has zero error. Assume solutions of X3C indexed by Λ. Set c opt = 1 Λ. Φc opt = x. u = Sparse solution has zero error and D-Exact returns Yes. Suppose c opt is optimal solution of Sparse Φc opt = x then c opt contains k 3 1 U nonzero entries and D-Exact returns Yes. Each column of Φ has 3 nonzero entries = {X i i supp(c opt )} is disjoint collection covering U.

23 What does this mean? Bad news Given any polynomial time algorithm for Sparse, there is a dictionary Φ and a signal x such that algorithm returns incorrect answer Pessimistic: worst case Cannot hope to approximate solution, either

24 What does this mean? Bad news Given any polynomial time algorithm for Sparse, there is a dictionary Φ and a signal x such that algorithm returns incorrect answer Pessimistic: worst case Cannot hope to approximate solution, either Good news Natural dictionaries are far from arbitrary Perhaps natural dictionaries admit polynomial time algorithms Optimistic: rarely see worst case Hardness depends on instance type

25 Hardness depends on instance Redundant dictionary Φ input signal x NP-hard arbitrary arbitrary depends on choice of Φ fixed fixed compressive sensing random (distribution?) random (distribution?) random signal model

26 Leverage intuition from orthonormal basis Suppose Φ is orthogonal, Φ 1 = Φ T Solution to Exact problem is unique c = Φ 1 x = Φ T x i.e., c l = x, ϕ l hence, x = l x, ϕ l ϕ l.

27 Leverage intuition from orthonormal basis Solution to Sparse problem similar Let l 1 arg max l x, ϕ l Set c l1 x, ϕ l1 Residual r x c l1 ϕ l1

28 Leverage intuition from orthonormal basis Solution to Sparse problem similar Let l 1 arg max l x, ϕ l Set c l1 x, ϕ l1 Residual r x c l1 ϕ l1 Let l 2 arg max l r, ϕ l = arg max l x c l1 ϕ l1, ϕ l = arg max l l1 x, ϕ l Set c l2 r, ϕ l2. Update residual r x (c l1 ϕ l1 + c l2 ϕ l2 )

29 Leverage intuition from orthonormal basis Solution to Sparse problem similar Let l 1 arg max l x, ϕ l Set c l1 x, ϕ l1 Residual r x c l1 ϕ l1 Let l 2 arg max l r, ϕ l = arg max l x c l1 ϕ l1, ϕ l = arg max l l1 x, ϕ l Set c l2 r, ϕ l2. Update residual r x (c l1 ϕ l1 + c l2 ϕ l2 ) Repeat k 1 times.

30 Leverage intuition from orthonormal basis Solution to Sparse problem similar Let l 1 arg max l x, ϕ l Set c l1 x, ϕ l1 Residual r x c l1 ϕ l1 Let l 2 arg max l r, ϕ l = arg max l x c l1 ϕ l1, ϕ l = arg max l l1 x, ϕ l Set c l2 r, ϕ l2. Update residual r x (c l1 ϕ l1 + c l2 ϕ l2 ) Repeat k 1 times. Set c l 0 for l l 1, l 2,..., l k.

31 Leverage intuition from orthonormal basis Solution to Sparse problem similar Let l 1 arg max l x, ϕ l Set c l1 x, ϕ l1 Residual r x c l1 ϕ l1 Let l 2 arg max l r, ϕ l = arg max l x c l1 ϕ l1, ϕ l = arg max l l1 x, ϕ l Set c l2 r, ϕ l2. Update residual r x (c l1 ϕ l1 + c l2 ϕ l2 ) Repeat k 1 times. Set c l 0 for l l 1, l 2,..., l k. Approximate x Φc = k t=1 x, ϕ l t ϕ lt. Check: algorithm generates list of coeffs of x over basis in descending order (by absolute value).

32 Geometry Why is orthogonal case easy? inner products between atoms are small it s easy to tell which one is the best choice r, ϕ j = x c i ϕ i, ϕ j = x, ϕ j c i ϕ i, ϕ j When atoms are (nearly) parallel, can t tell which one is best

33 Summary Sparse approximation problems are NP-hard At least as hard as other well-studied problems Hardness result of arbitrary input: dictionary and signal Intuition from orthonormal basis suggests some feasible solutions under certain conditions on redundant dictionary Next lecture: geometric contraints/properties of dictionaries that permit feasible solutions