Project Discussions: SNL/ADMM, MDP/Randomization, Quadratic Regularization, and Online Linear Programming

Transcription

1 Project Discussions: SNL/ADMM, MDP/Randomization, Quadratic Regularization, and Online Linear Programming Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. yyye 1

2 Project-I: SNL x i x j 2 = d 2 ij, (i, j) N x, i < j, a k x j 2 = ˆd 2 kj, (k, j) N a, (1) SDP relaxation for solving solve (1): Find a symmetric matrix Z S d+n such that min 0 Z s.t. Z 1:d,1:d = I, (0; e i e j )(0; e i e j ) T Z = d 2 ij, i, j N x, i < j, (a k ; e j )(a k ; e j ) T Z = ˆd 2 kj, k, j N a, Z 0. (2) Also a simple nonlinear least squares (NLS) approach to solve (1): min (ij) N x ( xi x j 2 d 2 ij ) 2 + (kj) N a ( a k x j 2 d 2 kj) 2 (3) 2

3 P-I: Questions Run some randomly generated problems (in 1D, 2D and 3D) with few (2, 3 and 4) anchors and tens sensors to compare the SOCP, SDP, and NLS approaches. Use the SDP solution X = [ x 1, x 2,..., x n ] of Z as the initial solution for model (3) and apply the Steepest Descent Method for a number steps. How is the final solution come out after steepest descent? Apply ADMM to the split nonlinear least squares: min (ij) N x [ (xi x j ) T (y i y j ) d 2 ij ] [ ] 2 + (kj) N a (a k x j ) T (a k y j ) d 2 kj s.t. x j y j = 0, j. (4) Apply Steepest Descent and Feasible Projection Method to SDP Relaxation (Slide 11 of Lecture 11). 3

4 P-I: Objective Regularization for SDP Rank-Reduction for SNL Consider Anchor-Free SNL: minimize (i,j) E α ij subject to (e i e j )(e i e j ) T Y = d 2 ij + α ij, (i, j) E, minimize Y 0. (i,j) E α ij + λ r(y ) subject to (e i e j )(e i e j ) T Y = d 2 ij + α ij, (i, j) E, Y 0, where r(.) is a nonnehative regularization function. For example, the matrix p-norm function: 1/p r(y ) = Y p = j λ(y ) j p, 0 < p 1. 4

5 P-I: Tensegrity (Tensional-Integrity) Objective for 1-D SNL An Anchor-Free SNL example: a Unit-Distance Chain (e i e i+1 )(e i e i+1 ) T Y = 1, i = 1,..., n 1 Y 0. For certain graphs, to select a subset edges to maximize and/or a subset of edges to minimize is guaranteed to finding the lowest rank SDP solution Tensegrity Method. To Maximize 5

6 P-I: The Chain Graph Example Consider: max e 3 e 3 Y s.t. e 1 e T 1 Y = 1, (e 1 e 2 )(e 1 e 2 ) T Y = 1, (e 2 e 3 )(e 2 e 3 ) T Y = 1, Y 0 S 3, The dual is min y 1 + y 2 + y 3 s.t. y 1 e 1 e T 1 + y 2 (e 1 e 2 )(e 1 e 2 ) T + y 3 (e 2 e 3 )(e 2 e 3 ) T S = e 3 e 3, S 0 S 3, 6

7 P-I: Interpretation of the Dual What s the interpretation of the dual? What s the interpretation of complementarity? Dual variables are stresses (internal-forces) on edges, and the objective is the total potential of the graph. At the optimal solution, stresses reach an equilibrium or balanced state with an external force added at node 3. (1; 2; 3)(1; 2; 3) T S = 0 implies S(1; 2; 3) = 0; or ( 3e1 e T 1 + 3(e 1 e 2 )(e 1 e 2 ) T + 3(e 2 e 3 )(e 2 e 3 ) T e 3 e 3 ) (1; 2; 3) = 0, that is, 3e 1 3(e 1 e 2 ) 3(e 2 e 3 ) = 3e 3. 7

8 P-I: The Kissing Number Problem Given a unit sphere centered at the origin, the maximum number of other unit spheres, in dimension d, can touch or kiss the centered unit-sphere? The Kissing Problem as a Graph Realization: (e i e j ) T Y (e i e j ) 4, i j, e T i Y e i = 4, i Y 0; (Rank(Y ) = d) A Tensegrity regularization objective function is constructed to find the lowest rank solution up to dimension 3. 8

9 Figure 1: Kissing Localization in 2D 9

10 Project-II: First-Order Methods and Value-Iteration for MDP MDP problem with m states and total n actions: min x j A 1 c j x j j A m c j x j s.t. j A 1 (e 1 γp j )x j j A m (e m γp j )x j = e,... x j... 0, j, (5) maximize y m i=1 y i where y i represents the cost-to-go value in state i. subject to y i γp T j y c j, j A i, i. Question 1: Prove that in (5) every basic feasible solution represent a policy, i.e., the basic variables have exactly one variable from each state i. Furthermore, prove each basic variable value is no less than 1, and the sum of all basic variable values is m 1 γ. 10

11 P-II: First-Order Methods and Value-Iteration for MDP The Value-Iteration (VI) Method is, starting from any y 0, y k+1 i = min j A i {c j + γp T j y k }, i. Question 2: Prove the contraction result: y k+1 y γ y k y, k. where y is the fixed-point or optimal value vector, that is, yi = min j A i {c j + γp T j y }, i. Question 3: In the VI method, if starting with any vector y 0 y and assuming y 1 y 0, then prove the following entry-wise monotone property: y y k+1 y k, k. On the other hand, if we start from a vector such that yi 0 < min j A i {c j + γp T j y0 }, i (y 0 in the interior of the feasible region), then prove the entry-wise monotone property: y y k+1 y k, k. 11

12 P-II: Randomized VI Rather than go through all state values in each iteration, we modify the VI method, call it RamdomVI: In the kth iteration, randomly select a subset of states B k and do y k+1 i = min j A i {c j + γp T j y k }, i B k. (6) In RandomVI, we only update a subset of state values at random in each iteration. Question 4: What can you tell the convergence of the RandomVI method? Does it make a difference with the classical VI method? How is the sample size affect the performance? Use simulated computational experiments to verify your claims. Rather than randomly select a subset of all states in each iteration, suppose we build an influence tree from a given subset of states, say B, for all sates, denoted by I(B), that are connected by any state in B. Then when states in B are updated in the current iteration, then selected a subset of states in I(B) for updating in the next iteration. Redo the computational experiments using this strategy for a sparsely connected (p j is a very sparse distribution vector for each action j) MDP network. In doing so, many unimportant or irrelevant states may be avoided which results a state-reduction. 12

13 P-II: Cyclic VI Question 5: Here is another modification, called CyclicVI: In the kth iteration do Initialize ỹ k = y k. For i = 1 to m ỹ k i = min j A i {c j + γp T j ỹk } (7) y k+1 = ỹ k. In the CyclicVI method, as soon as a state value is updated, we use it to update the rest of state values. What can you tell the convergence of the CyclicVI method? Does it make a difference with other VI methods? Use simulated computational experiments to verify your claims. How is this cyclic method related to the method at the bottom of Question 4? 13

14 P-II: Randomly Permuted Cyclic VI In the CyclicVI method, rather than with the fixed cycle order from 1 to m, we follow a random permutation order, or sample without replacement to update the state values. More precisely, in the kth iteration do 0. Initialize ỹ k = y k and B k = {1, 2,..., m} 1. Randomly select i B k ỹ k i = min j A i {c j + γp T j ỹk } (8) remove i from B k and return to Step y k+1 = ỹ k. We call it the randomly permuted CyclicVI or RPCyclicVI in short What can you tell the convergence of the RPCyclicVI method? Does it compare with other VI methods? Use simulated computational experiments to verify your claims. 14

15 Project-III: Quadratic Regularization and Path-Following Algorithms Question 1: Quadratic Regularization Method described in Lecture #12, and try it on your favorite nonlinear and nonconvex minimization problem. d k (λ) = arg min d (c k ) T d dt Q k d λ d 2 (9) where parameterλ max{0, λ min (Q k )}. Then consider the one-variable function ϕ(λ) := f(x k + d k (λ)) and do the one-variable minimization of ϕ(λ). Let λ k = arg min ϕ(λ) and do update x k+1 = x k + d k (λ k ). First use the direct method to solve the convex QP problem (9) for any given λ. Then use first-order QP algorithms to solve (9) to accuracy error ϵ k. What is a good strategy to select ϵ k? Decrease ϵ k as k increases? At what rate? How to do the one-variable minimization problem fast? Try different approached, Golden Search, Bisection Search, Newton s Method, or simply the back-tracking strategy presented page 28 of Lecture #10? Also, is the final λ k a useful initial point for searching λ k+1? 15

16 P-III: Path-Following Algorithms for Unconstrained Optimization Question 2: Now consider the Path-Following Method described in Lecture #12. Start from a solution x k that approximately satisfies f(x) + λx = 0, with λ = λ k > 0. (10) When f is convex, such a solution x(λ) exists for any λ > 0 and we let x(λ) = arg min f(x) + λ 2 x 2 and they form a path down to x(0). Sub-Question 2.1: What would happen if f is not convex but it is bounded from below? 16

17 P-III: Path-Following Algorithms for Unconstrained Optimization Let the approximation path error at x k with λ = λ k be (x k ) + λ k x k 1 2β λk. Then, we like to compute a new iterate x k+1 such that f(x k+1 ) + λ k+1 x k+1 1 2β λk+1, where 0 λ k+1 < λ k. Sub-Question 2.2: What is a good strategy to decrease λ? 17

18 P-III: Path-Following Algorithms for Unconstrained Optimization After λ k is replaced by a smaller λ k+1, we aim to find a solution x such that f(x) + λ k+1 x = 0. One way is to start from x k and apply the Newton Method: f(x k ) + 2 f(x k )d + λ k+1 (x k + d) = 0, 2 f(x k )d + λ k+1 d = f(x k ) λ k+1 x k. or (11) which is also a convex quadratic minimization problem if f is convex. Sub-Question 2.3: Again, use first-order QP algorithms to solve the convex QP problem to accuracy error ϵ k. What is a good strategy to select ϵ k? Decreaseϵ k as k increases? At what rate? Sub-Question 2.4: Finally, how to modify the path-following algorithm for nonconvex f minimization? What should we do if the matrix of (11) is singular? Could some random perturbation be helpful here? 18

19 Project-IV: An Online Linear Programming/Resource Allocation Example order 1(t = 1) order 2(t = 2)... Inventory(b) Price(π t ) $100 $30... Decision x 1 x 2... Pants Shoes T-shirts Jacket Socks

20 P-IV: Offline Formulation LP max π T x s.t. A T x b, x e, x 0, CPCAM max π T x + U(s) s.t. A T x + s = b, s 0 x e, x 0. U( ), in the form U(s) = i u(s i), is a strcitly concave (risk aversion) and increasing value function of the possible slack variables to value the uncertain revenue of remaining resources. Question 1: The optimal s so is the multipliers/prices of first set of constraints. Exponential u(s i ) = b (1 exp( s i /b)) for some positive constant b. Logarithmic u(s i ) = b log(s i ) or u(s i ) = b log(1 + s i ) for some positive constant b. b (1 (1 s i /b) 2 ) 0 s i b Quadratic: u(s i ) = b s i b for some positive constant b. 20

21 P-IV: Online Formulation and SCPM Given the already-made (t 1) order-fill decisions x 1,..., x t 1, the t th order-fill decision: max π t x t + i u(s i) + t 1 j=1 π j x j s.t. a t x t + s = b t 1 j=1 a j x j, x t 1 x t 0. t 1 j=1 π j x j is collected revenue, and t 1 j=1 a j x j is outstanding allocations before the new arrival. max π t x t + i u(s i) s.t. a t x t + s = b t 1, x t 1, x t 0, max π t x t + i u(z a itx t b t 1 i ) s.t. x t 1, x t 0; where b t 1 = b t 1 j=1 a j x j. Question 2 and 3: KKT conditions of the simplified problem and experiments. 21

22 P-IV: Price Mechanism and SLPM The problem would be easy if there is an ideal price vector: Bid 1(t = 1) Bid 2(t = 2)... Inventory(b) p Bid(π t ) $100 $30... Decision x 1 x 2... Pants $45 Shoes $45 T-shirts $10 Jackets $55 Hats $15 Could such a ideal price vector be learned? 22

23 P-IV: Model Assumptions Main Assumptions The columns (π t, a t ) arrive in a random order. We know the total number of columns n a priori. Other technical assumptions 0 a it 1, for all (i, t); π t 0 for all t The algorithm/mechanism quality is evaluated on the expected performance over all the permutations comparing to the offline optimal solution, i.e., an algorithm A is c-competitive if and only if [ n ] E σ π t x t (σ, A) c OP T (A, π), t=1 where OP T (A, π) is the maximal objective value of the offline model. 23

24 P-IV: Comments and Theorems on the Online Model The online approach is distribution-free so that it allows for great robustness in practical problems. The second assumption is necessary for one to obtain a near optimal solution. However, it can be relaxed to an approximate knowledge of n or the length of decision horizon. Both assumptions are reasonable and standard in many engineering and science applications. Theorem 1 For any fixed 0 < ϵ < 1, there is no online algorithm for solving the linear program with competitive ratio 1 ϵ if B < log(m) ϵ 2. Theorem 2 For any fixed 0 < ϵ < 1, there is a 1 ϵ competitive online algorithm for solving the linear program if ( ) m log (n/ϵ) B Ω. Agrawal, Wang and Y [Operations Research 2014] 24 ϵ 2

25 P-IV: Key Ideas to Prove Negative and Positive Result Consider m = 1 and inventory level B, one can construct an instance where OP T = B, and there will be a loss of B with a high probability, which give an approximation ratio 1 1. B Consider general m and inventory level B for each good. We are able to construct an instance to decompose the problem into log(m) separable problems, each of which has an inventory level B/ log(m) on a composite single good and OP T = B/ log(m). Then, with hight probability each single good case has a loss of B/ log(m) and the total loss of B log(m). Thus, approximation ratio is at best 1 log(m) B. The proof of the positive result is constructive and based on a learning policy. There is no distribution known so that any type of stochastic optimization models is not applicable. Unlike dynamic programming, the decision maker does not have full information/data so that a backward recursion can not be carried out to find an optimal sequential decision policy. Thus, the online algorithm needs to be learning-based, in particular, learning-while-doing. 25

26 P-IV: One-Time Learning Algorithm We start with a simple Set x t = 0 for all 1 t ϵn; Solve the ϵ portion of the problem maximize x subject to ϵn t=1 π tx t ϵn t=1 a itx t (1 ϵ)ϵb i i = 1,..., m 0 x t 1 t = 1,..., ϵn and get the optimal Lagrange/dual solution ˆp; Determine the future allocation x t as: x t = 0 if π t ˆp T a t 1 if π t > ˆp T a t as long as a it x t b i t 1 j=1 a ijx j for all i; otherwise, set x t = 0. 26

27 P-IV: One-Time Learning Algorithm Result Theorem 3 For any fixed ϵ > 0, the one-time learning algorithm is (1 ϵ) competitive for solving the linear program when B Ω ( ) m log (n/ϵ) ϵ 3 Outline of the Proof: With high probability, we clear the market; With high probability, the revenue is near-optimal if we include the initial ϵ portion revenue; With high probability, the first ϵ portion revenue, a learning cost, doesn t contribute too much. Then, we prove that the one-time learning algorithm is (1 ϵ) competitive under condition B 6m log(n/ϵ) ϵ 3. But this is one ϵ factor higher than the lower bound... Question 4: numerical experiments. 27

28 P-IV: Dynamic Price Updating Algorithm In the dynamic price learning algorithm, we update the price at time ϵn, 2ϵn, 4ϵn,..., till 2 k ϵ 1. At time l {ϵn, 2ϵn,...}, the price vector is the optimal Lagrange/dual solution to the following linear program: maximize x subject to l t=1 π tx t l t=1 a itx t (1 h l ) l n b i i = 1,..., m 0 x t 1 t = 1,..., l where h l = ϵ n l ; and this price vector is used to determine the allocation for the next immediate period, which is doubled each update. Question 5: numerical experiments. In the dynamic algorithm, we update the prices log 2 (1/ϵ) times during the entire time horizon. The numbers h l basically balances the probability that the inventory ever gets violated and the lost of revenue due to the factor 1 h l. Choosing large h l (more conservative) at the beginning periods and smaller h l (more aggressive) at the later periods, one can now control the loss of revenue. 28

29 P-IV: Related Work on Random-Permutation Sufficient Condition Kleinberg [2005] B 1 ϵ 2, for m = 1 Devanur et al [2009] Feldman et al [2010] Agrawal et al [2010] Molinaro/Ravi [2013] Kesselheim et al [2014] Gupta/Molinaro [2014] Agrawal/Devanur [2014] B m log n ϵ 3 B m log n ϵ 2 OP T m2 log(n) ϵ 3 and OP T m log n ϵ or OP T m2 log n ϵ 2 B m2 log m ϵ 2 B log m ϵ 2 B log m ϵ 2 B log m ϵ 2 Learning Dynamic One-time One-time Dynamic Dynamic Dynamic* Dynamic* Dynamic* Table 1: Comparison of several existing results 29

30 P-IV: Summary and Future Questions on OLP B = log m ϵ 2 is now a necessary and sufficient condition (differing by a constant factor). Thus, they are near-optimal online algorithms for a very general class of online linear programs. The algorithms are distribution-free and/or non-parametric, thereby robust to distribution/data uncertainty. The dynamic learning has the feature of learning-while-doing, and is provably better than one-time learning by a factor. Buy-and-sell or double market? Price-Posting multi-good model? Online Utility Formulation for Resource Allocation? Question 6: multi-layer resource allocation/supply-chain management 30