The Johnson-Lindenstrauss Lemma in Linear Programming

Transcription

1 The Johnson-Lindenstrauss Lemma in Linear Programming Leo Liberti, Vu Khac Ky, Pierre-Louis Poirion CNRS LIX Ecole Polytechnique, France Aussois COW 2016

2 The gist Goal: solving very large LPs min{c x Ax = b x 0} Trade-off: approximate / wrong with low probability: OK Means: project cols of Ax = b to random subspace T, get with high probability Ax = b x 0 TAx = Tb x 0 Bisection: solve LP using [TAx = Tb x 0] as oracle 2

3 Plan Restricted Linear Membership Johnson-Lindenstrauss Lemma Applying JLL to RLM Towards solving LPs 3

4 Restricted Linear Membership 4

5 Linear feasibility with constrained multipliers Restricted Linear Membership (RLM) Given vectors A 1,...,A n,b R m and X R n, is there x X s.t. b = i nx i A i? RLM X is a fundamental problem class, which subsumes: Linear Feasibility Problem (LFP) with X = R n + Integer Feasibility Problem (IFP) with X = Z n + Efficient solution of LFP/IFP yields sol. of LP/IP via bisection 5

6 The shape of a set of points Lose dimensions but not too much accuracy Given A 1,...,A n R m find k m and points A 1,...,A n R k s.t. A and A have almost the same shape What is the shape of a set of points? A A congruent sets have the same shape Approximate congruence: A,A have almost the same shape if i < j n (1 ε) A i A j A i A j (1+ε) A i A j for some small ε > 0 Assume norms are all Euclidean 6

7 Losing dimensions in the RLM Given X R n and b,a 1,...,A n R m, find k m, b,a 1,...,A n Rk such that: x X b = i nx i A i } {{ } high dimensional with high probability iff x X b = i nx i A i } {{ } low dimensional If this is possible, then solve RLM X (b,a ) Since k m, solving RLM X (b,a ) should be faster RLM X (b,a ) = RLM X (b,a) with high probability 7

8 Losing dimensions = projection In the plane, hopeless line 2 line 1 In 3D: no better 8

9 The Johnson-Lindenstrauss Lemma 9

10 Johnson-Lindenstrauss Lemma Thm. Given A R m with A = n and ε > 0 there is k O( 1 ε2 lnn) and a k m matrix T s.t. x,y A (1 ε) x y Tx Ty (1+ε) x y If k m matrix T is sampled componentwise from N(0, 1 k ), then A and TA have almost the same shape Discrete approximations of N(0, 1 k ) can also be used, e.g. P(T ij = 1 k ) = P(T ij = 1 k ) = 1 6, P(T ij = 0) = 2 3 (This makes T sparser) 10

11 Sampling to desired accuracy Distortion has low probability: x,y A P( Tx Ty (1 ε) x y ) 1 n 2 x,y A P( Tx Ty (1+ε) x y ) 1 n 2 Probability pair x, y A distorting Euclidean distance: union bound over ( ) n 2 pairs P( (A and TA have almost the same shape)) ( n 2 ) 2 n 2 = 1 1 n P(A and TA have almost the same shape) 1 n re-sampling T gives JLL with arbitrarily high probability 11

12 Sketch of a possible JLL proof 90% 90% 90% n=3 n=11 n=101 Thm. Let T be a k m rectangular matrix with each component sampled from N(0, 1 k ), and u R m s.t. u = 1. Then E( Tu 2 ) = 1 dt d S m Tu O S m 1 t 1 1 t 2 12

13 In practice Empirical estimation of C in k = Cε 2 lnn: C 1.8 [Venkatasubramanian & Wang 2011] Empirically, sample T very few times (e.g. once will do!) on average Tx Ty x y, and distortion decreases exponentially with n We only need a logarithmic number of dimensions in function of the number of points Surprising fact: k is independent of the original number of dimensions m 13

14 Typical applications of JLL Problems involving Euclidean distances only Euclidean clustering k-means, k-nearest neighbors Linear regression min x Ax b 2 where A is m n with m n 14

15 Applying the JLL to the RLM 15

16 Projecting infeasibility Thm. T : R m R k a JLL random projection, b,a 1,...,A n R m a RLM X instance. For any given vector x X, we have: (i) If b = n i=1 (ii) If b n i=1 (iii) If b n i=1 x i A i then Tb = n x i A i then P ( i=1 Tb n x i TA i i=1 x i TA i ) 1 2e Ck y i A i for all y X R n, where X is finite, then ( P y X Tb n i=1 y i TA i ) for some constant C > 0 (independent of n,k). 1 2 X e Ck [VPL, arxiv: v1/math.oc] 16

17 Proof (ii) Cor. ε (0,1) and z R m, there is a constant C such that Proof By the JLL P((1 ε) z Tz (1+ε) z ) 1 2e Cε2 k Lemma If z 0, there is a constant C such that P(Tz 0) 1 2e Ck Proof Consider events A : Tz 0 and B : (1 ε) z Tz (1+ε) z A c B =, othw Tz = 0 (1 ε) z Tz = 0 z = 0, contradiction B A P(A) P(B) 1 e Cε2k by Corollary Holds ε (0,1) hence result Now it suffices to apply the Lemma to Ax b 17

18 Consequences of the main theorem (i) and (ii): checking certificates given x, with high probability b = ix i A i Tb = ix i TA i (iii) RLM X whenever X is polynomially bounded e.g. knapsack set {x {0,1} n α i x i d} for a fixed d with α > 0 i n (iii) hints that LFP case is more complicated as X = R n + is not polynomially bounded 18

19 Separating hyperplanes When X is large, project separating hyperplanes instead Convex C R m, x C: then hyperplane c separating x, C In particular, true if C = cone(a 1,...,A n ) for A R m We aim to show x C Tx TC with high probability As above, if x C then Tx TC by linearity of T real issue is proving the converse 19

20 Projecting the separation Thm. Given c,b,a 1,...,A n R m of unit norm s.t. b / cone{a 1,...,A n } pointed, ε > 0, c R m s.t. c b < ε, c A i ε (i n), and T a random projector: P [ Tb / cone{ta 1,...,TA n } ] 1 4(n+1)e C(ε2 ε 3 )k for some constant C. Proof Let A be the event that T approximately preserves c χ 2 and c + χ 2 for all χ {b,a 1,...,A n }. Since A consists of 2(n + 1) events, by the JLL Corollary (squared version) and the union bound, we get Now consider χ = b P(A) 1 4(n+1)e C(ε2 ε 3 )k Tc,Tb = 1 4 ( T(c+b) 2 T(c b) 2 ) by JLL 1 4 ( c+b 2 c b 2 )+ ε 4 ( c+b 2 + c b 2 ) = c b+ε < 0 and similarly Tc,TA i 0 [VPL, arxiv: v1/math.oc] 20

21 Is this useful? Previous results look like: orig. LFP infeasible P(proj. LFP infeasible) 1 p(n)e Cr(ε)k where p, r two polynomials Pick a suitable δ > 0 Choose k O( 1 Cr(ε) (lnp(n)+ln 1 δ )) so that RHS 1 δ Preserve infeasibility with probability 1 δ Useful for m n large enough that k m 21

22 Consequences of projecting separations Applicable to LFP Probability depends on ε (the larger the better) Largest ε given by LP max{ε 0 c b ε i n (c A i ε)} If cone(a 1,...,A n ) is almost non-pointed, ε can be very small 22

23 Projecting minimum distances to a cone Thm.: minimum distance to a cone is approximately preserved This result also works with non-pointed cones Trade-off: need larger k,m,n We appear to be all set for LFPs Using bisection and LFP, also for LPs 23

24 Main theorem for LFP projections Established so far: Thm. Given δ > 0, sufficiently large m n such that: for any LFP input A,b where A is m n we can sample a random k m matrix T with k m and P(orig. LFP feasible proj. LFP feasible) 1 δ 24

25 Towards solving LPs 25

26 Some results on uniform dense LFP Matrix product TA takes too long (call this an implementation detail and don t count it) Infeasible instances (sizes from to ): Uniform ǫ k CPU saving accuracy ( 1, 1) m 30% 50% ( 1, 1) m 92% 0% ( 1, 1) m 99.2% 0% (0, 1) m 10% 100% (0, 1) m 90% 100% (0, 1) m 97% 100% Feasible instances: similar CPU savings obviously 100% accuracy 26

27 Certificates Ax = b TAx = Tb by linearity, however Thm.: For x 0 s.t. TAx = Tb, Ax = b with probability 0 Can t get certificate for original LFP using projected LFP! 27

28 Can we solve LPs by bisection? Projected certificate is infeasible in original problem Only get approximate optimal objective function value No bound on error, no idea about how large m,n should be Validated on large enough NetLib instances (with k 0.95m) 28

29 Certificate retrieval from dual LFP b r e a k i n g n e w s! Primal min{c x Ax = b x 0} dual max{b y A y c} Run bisection on projected LFP, threshold v l at itn. l Proj. LFP infeasible unbounded dual ray λ l s.t. (Tb) λ l > v l (TA) λ l c b (T λ l ) > v l A (T λ l ) c T λ l is a certificate for the original dual L = set of itn. indices s.t. projected LFP infeasible lim(t λ l L l ) = λ (opt. dual sol.) l In practice: last iteration l, get λ T λ l Complementarity cond. m basic cols of A, solve for x 29

30 Current work Implementation of certificate retrieval from dual Random projections directly on dual LP allows explicit feasibility & optimality guarantees Results on projecting Integer Programs we have many of these already! Certificate retrieval using Plotkin-Shmoys-Tardos algorithm 30

31 Some references K. Vu, P.-L. Poirion, L. Liberti, Using the Johnson-Lindenstrauss lemma in linear and integer programming, arxiv report v1/math.OC, 2015 K. Vu, P.-L. Poirion, L. Liberti, Gaussian random projections for Euclidean membership problems, arxiv report v1/math.OC, 2015 W. Johnson and J. Lindenstrauss, Extensions of Lipschitz mappings into a Hilbert space, in Contemporary Mathematics, Vol. 26, , 1984 S. Dasgupta and A. Gupta, An elementary proof of a theorem of Johnson and Lindenstrauss, Random Structures and Algorithms, 22(1):60-65, 2003 S. Venkatasubramanian, Q. Wang, The Johnson-Lindenstrauss transform: an empirical study, in Proceedings of ALENEX, 2011 J. Nash, C 1 Isometric embeddings, in Annals of Mathematics, 60(3): ,