Convex Optimization of Graph Laplacian Eigenvalues
|
|
- Alberta Kelly
- 6 years ago
- Views:
Transcription
1 Convex Optimization of Graph Laplacian Eigenvalues Stephen Boyd Stanford University (Joint work with Persi Diaconis, Arpita Ghosh, Seung-Jean Kim, Sanjay Lall, Pablo Parrilo, Amin Saberi, Jun Sun, Lin Xiao. Thanks to Almir Mutapcic.) ICM 2006 Madrid, August 29, 2006
2 Outline some basic stuff we ll need graph Laplacian eigenvalues convex optimization and semidefinite programming the basic idea some example problems distributed linear averaging fastest mixing Markov chain on a graph fastest mixing Markov process on a graph its dual: maximum variance unfolding conclusions ICM 2006 Madrid, August 29,
3 Outline some basic stuff we ll need graph Laplacian eigenvalues convex optimization and semidefinite programming the basic idea some example problems distributed linear averaging fastest mixing Markov chain on a graph fastest mixing Markov process on a graph its dual: maximum variance unfolding conclusions
4 (Weighted) graph Laplacian graph G = (V,E) with n = V nodes, m = E edges edge weights w 1,...,w m R l (i,j) means edge l connects nodes i, j incidence matrix: A il = 1 edge l enters node i 1 edge l leaves node i 0 otherwise (weighted) Laplacian: L = Adiag(w)A T L ij = w l l (i, j) l (i,k) w l i = j 0 otherwise ICM 2006 Madrid, August 29,
5 Laplacian eigenvalues L is symmetric; L1 = 0 we ll be interested in case when L 0 (i.e., L is PSD) (always the case when weights nonnegative) Laplacian eigenvalues (eigenvalues of L): 0 = λ 1 λ 2 λ n spectral graph theory connects properties of graph, and λ i (with w = 1) e.g.: G connected iff λ 2 > 0 (with w = 1) ICM 2006 Madrid, August 29,
6 Convex spectral functions suppose φ is a symmetric convex function in n 1 variables then ψ(w) = φ(λ 2,...,λ n ) is a convex function of weight vector w examples: φ(u) = 1 T u (i.e., the sum): ψ(w) = n λ i = n λ i = TrL = 21 T w (twice the total weight) i=2 i=1 φ(u) = max i u i : ψ(w) = max{λ 2,...,λ n } = λ n (spectral radius) ICM 2006 Madrid, August 29,
7 More examples φ(u) = min i u i (concave) gives ψ(w) = λ 2, algebraic connectivity (concave function of w) φ(u) = i 1/u i (with u i > 0): ψ(w) = n i=2 1 λ i proportional to total effective resistance of graph, TrL ICM 2006 Madrid, August 29,
8 Outline some basic stuff we ll need graph Laplacian eigenvalues convex optimization and semidefinite programming the basic idea some example problems distributed linear averaging fastest mixing Markov chain on a graph fastest mixing Markov process on a graph its dual: maximum variance unfolding conclusions
9 Convex optimization x R n is optimization variable minimize f(x) subject to x X f is convex function (can maximize concave f by minimizing f) X R n is closed convex set roughly speaking, convex optimization problems are tractable, easy to solve numerically (tractability depends on how f and X are described) ICM 2006 Madrid, August 29,
10 Symmetry in convex optimization permutation (matrix) π is a symmetry of problem if f(πz) = f(z) for all z, πz X for all z X if π is a symmetry and the convex optimization problem has a solution, it has a solution invariant under π (if x is a solution, so is average over {x, πx,π 2 x,...}) ICM 2006 Madrid, August 29,
11 Duality in convex optimization primal: minimize f(x) subject to x X dual: maximize g(y) subject to y Y y is dual variable; dual objective g is concave; Y is closed, convex (various methods can be used to generate g, Y) p (d ) is optimal value of primal (dual) problem weak duality: for any x X, y Y, f(x) g(y); hence, p g(y) strong duality: for convex problems, provided a constraint qualification holds, there exist x X, y Y with f(x ) = g(y ) hence, x is primal optimal, y is dual optimal, and p = d ICM 2006 Madrid, August 29,
12 Semidefinite program (SDP) a particular type of convex optimization problem minimize c T x subject to n i=1 x ia i B, Fx = g variable is x R n ; data are c, F, g, symmetric matrices A i, B means with respect to positive semidefinite cone generalization of linear program (LP) minimize c T x subject to n i=1 x ia i b, Fx = g (here a i, b are vectors; means componentwise) ICM 2006 Madrid, August 29,
13 SDP dual primal SDP: minimize c T x subject to n i=1 x ia i B, Fx = g dual SDP: maximize TrZB ( ν T g subject to Z 0, F T ν + c ) + TrZA i i = 0, i = 1,...,n with (matrix) variable Z, (vector) ν ICM 2006 Madrid, August 29,
14 SDP algorithms and applications since 1990s, recently developed interior-point algorithms solve SDPs very effectively (polynomial time, work well in practice) many results for LP extended to SDP SDP widely used in many fields (control, combinatorial optimization, machine learning, finance, signal processing, communications, networking, circuit design, mechanical engineering,... ) ICM 2006 Madrid, August 29,
15 Outline some basic stuff we ll need graph Laplacian eigenvalues convex optimization and semidefinite programming the basic idea some example problems distributed linear averaging fastest mixing Markov chain on a graph fastest mixing Markov process on a graph its dual: maximum variance unfolding conclusions
16 The basic idea some interesting weight optimization problems have the common form minimize φ(w) = φ(λ 2,...,λ n ) subject to w W where φ is symmetric convex, and W is closed convex these are convex optimization problems we can solve them numerically (up to our ability to store data, compute eigenvalues... ) for some simple graphs, we can get analytic solutions associated dual problems can be quite interesting ICM 2006 Madrid, August 29,
17 Outline some basic stuff we ll need graph Laplacian eigenvalues convex optimization and semidefinite programming the basic idea some example problems distributed linear averaging fastest mixing Markov chain on a graph fastest mixing Markov process on a graph its dual: maximum variance unfolding conclusions
18 Distributed averaging x 1 1 W 12 2 x2 W 13 W 14 W 24 3 x 4 x 4 3 W 45 x 5 5 W 36 W 46 6 x 6 each node of connected graph has initial value x i (0) R; goal is to compute average 1 T x(0)/n using distributed iterative method applications in load balancing, distributed optimization, sensor networks ICM 2006 Madrid, August 29,
19 Distributed linear averaging simple linear iteration: replace each node value with weighted average of its own and its neighbors values; repeat x i (t + 1) = W ii x i (t) + where W ii + j N i W ij = 1 j N i W ij x j (t) = x i (t) j N i W ij (x i (t) x j (t)) we ll assume W ij = W ji, i.e., weights symmetric weights W ij determine whether convergence to average occurs, and if so, how fast classical result: convergence if weights W ij (i j) are small, positive ICM 2006 Madrid, August 29,
20 Convergence rate vector form: x(t + 1) = Wx(t) (we take W ij = 0 for i j, (i,j) E) W satisfies W = W T, W1 = 1 convergence lim t W t = (1/n)11 T ρ(w (1/n)11 T ) = W (1/n)11 T < 1 ρ is spectral radius; is spectral norm asymptotic convergence rate given by W (1/n)11 T convergence time is τ = 1/log W (1/n)11 T ICM 2006 Madrid, August 29,
21 Connection to Laplacian eigenvalues identifying W ij = w l for l (i,j), we have W = I L convergence rate given by W (1/n)11 T = I L (1/n)11 T = max{ 1 λ 2,..., 1 λ n } = max{1 λ 2,λ n 1}... a convex spectral function, with φ(u) = max i 1 u i ICM 2006 Madrid, August 29,
22 Fastest distributed linear averaging minimize W (1/n)11 T subject to W S, W = W T, W1 = 1 optimization variable is W; problem data is graph (sparsity pattern S) in terms of Laplacian eigenvalues with variable w R m minimize max{1 λ 2, λ n 1} these are convex optimization problems so, we can efficiently find the weights that give the fastest possible averaging on a graph ICM 2006 Madrid, August 29,
23 Semidefinite programming formulation introduce scalar variable s to bound spectral norm minimize s subject to si I L (1/n)11 T si (for Z = Z T, Z s si Z si) an SDP (hence, can be solved efficiently) ICM 2006 Madrid, August 29,
24 Constant weight averaging a simple, traditional method: constant weight on all edges, w = α1 yields update x i (t + 1) = x i (t) + j N i α(x j (t) x i (t)) a simple choice: max-degree weight, α = 1/max i d i d i is degree (number of neighbors) of node i best constant weight: α 2 = λ 2 + λ n (λ 2, λ n are eigenvalues of unweighted Laplacian, i.e., with w = 1) for edge transitive graph, w l = α is optimal ICM 2006 Madrid, August 29,
25 A small example max-degree best constant optimal ρ = W (1/n)11 T τ = 1/ log(1/ρ) ICM 2006 Madrid, August 29,
26 Optimal weights (note: some are negative!) λ i (W):.643,.643,.106, 0.000,.368,.643,.643, ICM 2006 Madrid, August 29,
27 Larger example 50 nodes, 200 edges max-degree best constant optimal ρ = W (1/n)11 T τ = 1/ log(1/ρ) ICM 2006 Madrid, August 29,
28 Eigenvalue distributions 6 4 max-degree best constant optimal ICM 2006 Madrid, August 29,
29 Optimal weights out of 250 are negative ICM 2006 Madrid, August 29,
30 Another example a cut grid with n = 64 nodes, m = 95 edges edge width shows weight value (red for negative) τ = 85; max-degree τ = 137 ICM 2006 Madrid, August 29,
31 Some questions & comments how much better are the optimal weights than the simple choices? for barbell graphs K n K n, optimal weights are unboundedly better than max-degree, optimal constant, and several other simple weight choices what size problems can be handled (on a PC)? interior-point algorithms easily handle problems with 10 4 edges subgradient-based methods handle problems with 10 6 edges any symmetry can be exploited for efficiency gain what happens if we require the weights to be nonnegative? (we ll soon see) ICM 2006 Madrid, August 29,
32 Least-mean-square average consensus include random noise in averaging process: x(t + 1) = W x(t) + v(t) v(t) i.i.d., Ev(t) = 0, Ev(t)v(t) T = I steady-state mean-square deviation: δ ss = lim t E 1 n (x i (t) x j (t)) 2 = i<j n i=2 1 λ i (2 λ i ) for ρ = max{1 λ 2, λ n 1} < 1 another convex spectral function ICM 2006 Madrid, August 29,
33 Outline some basic stuff we ll need graph Laplacian eigenvalues convex optimization and semidefinite programming the basic idea some example problems distributed linear averaging fastest mixing Markov chain on a graph fastest mixing Markov process on a graph its dual: maximum variance unfolding conclusions
34 Random walk on a graph Markov chain on nodes of G, with transition probabilities on edges P ij = Prob(X(t + 1) = j X(t) = i) we ll focus on symmetric transition probability matrices P (everything extends to reversible case, with fixed equilibrium distr.) identifying P ij with w l for l (i,j), we have P = I L same as linear averaging matrix W, but here W ij 0 (i.e., w 0, diag(l) 1) ICM 2006 Madrid, August 29,
35 Mixing rate probability distribution π i (t) = Prob(X(t) = i) satisfies π(t + 1) T = π(t) T P since P = P T and P1 = 1, uniform distribution π = (1/n)1 is stationary, i.e., ((1/n)1) T P = ((1/n)1) T π(t) (1/n)1 for any π(0) iff µ = P (1/n)11 T = I L (1/n)11 T < 1 µ is called second largest eigenvalue modulus (SLEM) of MC SLEM determines convergence (mixing) rate, e.g., sup π(t) (1/n)1 tv ( n/2 ) µ t π(0) associated mixing time is τ = 1/log(1/µ) ICM 2006 Madrid, August 29,
36 Fastest mixing Markov chain problem minimize µ = I L (1/n)11 T = max{1 λ 2,λ n 1} subject to w 0, diag(l) 1 optimization variable is w; problem data is graph G same as fast linear averaging problem, with additional nonnegativity constraint W ij 0 on weights convex optimization problem (indeed, SDP), hence efficiently solved ICM 2006 Madrid, August 29,
37 Two common suboptimal schemes max-degree chain: w = (1/max i d i )1 Metropolis-Hastings chain: w l = 1, where l (i,j) max{d i, d j } (comes from Metropolis method of generating reversible MC with uniform stationary distribution) ICM 2006 Madrid, August 29,
38 Small example optimal transition probabilities (some are zero) max-degree M.-H. optimal (fastest avg) SLEM µ (.643) mixing time τ (2.27) ICM 2006 Madrid, August 29,
39 Larger example 50 nodes, 200 edges max-degree M.-H. optimal (fastest avg) SLEM µ (.902) mixing time τ (9.7) ICM 2006 Madrid, August 29,
40 Optimal transition probabilities 82 edges (out of 200) edges have zero transition probability distribution of positive transition probabilities: ICM 2006 Madrid, August 29,
41 Subgraph with positive transition probabilities ICM 2006 Madrid, August 29,
42 Another example a cut grid with n = 64 nodes, m = 95 edges edge width shows weight value (dotted for zero) mixing time τ = 89; Metropolis-Hastings mixing time τ = 120 ICM 2006 Madrid, August 29,
43 Some analytical results for path, fastest mixing MC is obvious one (P i,i+1 = 1/2) for any edge-transitive graph (hypercube, ring,... ), all edge weights are equal, with value 2/(λ 2 + λ n ) (unweighted Laplacian eigenvalues) ICM 2006 Madrid, August 29,
44 Commute time for random walk on graph P ij proportional to w l, for l (i,j); P ii = 0 P not symmetric, but MC is reversible can normalize w as 1 T w = 1 commute time C ij : time for random walk to return to i after visiting j expected commute time averaged over all pairs of nodes is C = 1 n 2 n i,j=1 EC ij = 2 (n 1) n i=2 1 λ i (Chandra et al, 1989) called total effective resistance... another convex spectral function ICM 2006 Madrid, August 29,
45 Minimizing average commute time find weights that minimize average commute time on graph: minimize C = 2/(n 1) n i=2 1/λ i subject to w 0, 1 T w = 1 another convex problem of our general form ICM 2006 Madrid, August 29,
46 Outline some basic stuff we ll need graph Laplacian eigenvalues convex optimization and semidefinite programming the basic idea some example problems distributed linear averaging fastest mixing Markov chain on a graph fastest mixing Markov process on a graph its dual: maximum variance unfolding conclusions
47 Markov process on a graph (continuous-time) Markov process on nodes of G, with transition rate w l 0 between nodes i and j, for l (i,j) probability distribution π(t) R n satisfies heat equation π(t) = Lπ(t) π(t) = e tl π(0) π(t) converges to uniform distribution (1/n)1, for any π(0), iff λ 2 > 0 (asymptotic) convergence as e λ 2t ; λ 2 gives mixing rate of process λ 2 is concave, homogeneous function of w (come from symmetric concave function φ(u) = min i u i ) ICM 2006 Madrid, August 29,
48 Fastest mixing Markov process on a graph maximize λ 2 subject to l d2 l w l 1, w 0 variable is w R m ; data is graph, normalization constants d l > 0 a convex optimization problem, hence easily solved allocate rate across edges so as maximize mixing rate constraint is always tight at solution, i.e., l d2 l w l = 1 when d 2 l = 1/m, optimal value is called absolute algebraic connectivity ICM 2006 Madrid, August 29,
49 Interpretation: Grounded unit capacitor RC circuit charge vector q(t) satisfies q(t) = Lq(t), with edge weights given by conductances, w l = g l charge equilibrates (i.e., converges to uniform) at rate determined by λ 2 with conductor resistivity ρ, length d l, and cross-sectional area a l, we have g l = a l /(ρd l ) ICM 2006 Madrid, August 29,
50 total conductor volume is l d la l = ρ l d2 l w l problem is to choose conductor cross-sectional areas, subject to a total volume constraint, so as to make the circuit equilibrate charge as fast as possible optimal λ 2 is.105; uniform allocation of conductance gives λ 2 =.073 ICM 2006 Madrid, August 29,
51 SDP formulation and dual alternate formulation: minimize d 2 l w l subject to λ 2 1, w 0 SDP formulation: minimize d 2 l w l subject to L I (1/n)11 T, w 0 dual problem: maximize Tr X subject to X ii + X jj X ij X ji d 2 l, 1 T X1 = 0, X 0 l (i,j) with variable X R n n ICM 2006 Madrid, August 29,
52 A maximum variance unfolding problem use variables x 1,...,x n R n, with X = xt 1. x T n [x 1 x n ] dual problem becomes maximum variance unfolding (MVU) problem maximize i x i 2 subject to x i x j d l, i x i = 0 l (i,j) position n points in R n to maximize variance, while respecting local distance constraints ICM 2006 Madrid, August 29,
53 similar to semidefinite embedding for unsupervised learning of manifolds (Weinberger & Saul 2004) surprise: duality between fastest mixing Markov process and maximum variance unfolding ICM 2006 Madrid, August 29,
54 Conclusions some interesting weight optimization problems have the common form minimize φ(w) = φ(λ 2,...,λ n ) subject to w W where φ is symmetric convex, and W is closed convex these are convex optimization problems we can solve them numerically (up to our ability to store data, compute eigenvalues... ) for some simple graphs, we can get analytic solutions associated dual problems can be quite interesting ICM 2006 Madrid, August 29,
55 Convex Optimization Cambridge University Press, 2004 Some references Fast linear iterations for distributed averaging Systems and Control Letters 53:65-78, 2004 Fastest mixing Markov chain on a graph SIAM Review 46(4): , 2004 Fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem to appear, SIAM Review 48(4), 2006 Minimizing effective resistance of a graph to appear, SIAM Review, 2007 Distributed average consensus with least-mean-square deviation MTNS p , 2006 these and other papers available at ICM 2006 Madrid, August 29,
Convex Optimization of Graph Laplacian Eigenvalues
Convex Optimization of Graph Laplacian Eigenvalues Stephen Boyd Abstract. We consider the problem of choosing the edge weights of an undirected graph so as to maximize or minimize some function of the
More informationFast Linear Iterations for Distributed Averaging 1
Fast Linear Iterations for Distributed Averaging 1 Lin Xiao Stephen Boyd Information Systems Laboratory, Stanford University Stanford, CA 943-91 lxiao@stanford.edu, boyd@stanford.edu Abstract We consider
More informationFASTEST MIXING MARKOV CHAIN ON A GRAPH. 1. Introduction Fastest mixing Markov chain problem.
FASTEST MIXING MARKOV CHAIN ON A GRAPH STEPHEN BOYD, PERSI DIACONIS, AND LIN XIAO Abstract. We consider a symmetric random walk on a connected graph, where each edge is labeled with the probability of
More informationThe Fastest Mixing Markov Process on a Graph and a Connection to a Maximum Variance Unfolding Problem
SIAM REVIEW Vol. 48, No. 4, pp. 681 699 c 006 Society for Industrial and Applied Mathematics The Fastest Mixing Markov Process on a Graph and a Connection to a Maximum Variance Unfolding Problem Jun Sun
More informationMINIMIZING EFFECTIVE RESISTANCE OF A GRAPH
MINIMIZING EFFECTIVE RESISTANCE OF A GRAPH ARPITA GHOSH, STEPHEN BOYD, AND AMIN SABERI Abstract. The effective resistance between two nodes of a weighted graph is the electrical resistance seen between
More informationELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications
ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March
More informationA Duality View of Spectral Methods for Dimensionality Reduction
A Duality View of Spectral Methods for Dimensionality Reduction Lin Xiao 1 Jun Sun 2 Stephen Boyd 3 May 3, 2006 1 Center for the Mathematics of Information, California Institute of Technology, Pasadena,
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationGraph and Controller Design for Disturbance Attenuation in Consensus Networks
203 3th International Conference on Control, Automation and Systems (ICCAS 203) Oct. 20-23, 203 in Kimdaejung Convention Center, Gwangju, Korea Graph and Controller Design for Disturbance Attenuation in
More informationDimension reduction for semidefinite programming
1 / 22 Dimension reduction for semidefinite programming Pablo A. Parrilo Laboratory for Information and Decision Systems Electrical Engineering and Computer Science Massachusetts Institute of Technology
More informationUpper bounds on algebraic connectivity via convex optimization
Linear Algebra and its Applications 418 (2006) 693 707 www.elsevier.com/locate/laa Upper bounds on algebraic connectivity via convex optimization Arpita Ghosh, Stephen Boyd Department of Electrical Engineering,
More informationA Duality View of Spectral Methods for Dimensionality Reduction
Lin Xiao lxiao@caltech.edu Center for the Mathematics of Information, California Institute of Technology, Pasadena, CA 91125, USA Jun Sun sunjun@stanford.edu Stephen Boyd boyd@stanford.edu Department of
More informationSEMIDEFINITE PROGRAM BASICS. Contents
SEMIDEFINITE PROGRAM BASICS BRIAN AXELROD Abstract. A introduction to the basics of Semidefinite programs. Contents 1. Definitions and Preliminaries 1 1.1. Linear Algebra 1 1.2. Convex Analysis (on R n
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More informationCSC Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming
CSC2411 - Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming Notes taken by Mike Jamieson March 28, 2005 Summary: In this lecture, we introduce semidefinite programming
More informationFast linear iterations for distributed averaging
Available online at www.sciencedirect.com Systems & Control Letters 53 (2004) 65 78 www.elsevier.com/locate/sysconle Fast linear iterations for distributed averaging Lin Xiao, Stephen Boyd Information
More informationFastest Distributed Consensus Averaging Problem on. Chain of Rhombus Networks
Fastest Distributed Consensus Averaging Problem on Chain of Rhombus Networks Saber Jafarizadeh Department of Electrical Engineering Sharif University of Technology, Azadi Ave, Tehran, Iran Email: jafarizadeh@ee.sharif.edu
More informationINTERIOR-POINT METHODS ROBERT J. VANDERBEI JOINT WORK WITH H. YURTTAN BENSON REAL-WORLD EXAMPLES BY J.O. COLEMAN, NAVAL RESEARCH LAB
1 INTERIOR-POINT METHODS FOR SECOND-ORDER-CONE AND SEMIDEFINITE PROGRAMMING ROBERT J. VANDERBEI JOINT WORK WITH H. YURTTAN BENSON REAL-WORLD EXAMPLES BY J.O. COLEMAN, NAVAL RESEARCH LAB Outline 2 Introduction
More informationIntroduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 Markov Chain Monte Carlo Methods Barnabás Póczos & Aarti Singh Contents Markov Chain Monte Carlo Methods Goal & Motivation Sampling Rejection Importance Markov
More informationInequalities for the spectra of symmetric doubly stochastic matrices
Linear Algebra and its Applications 49 (2006) 643 647 wwwelseviercom/locate/laa Inequalities for the spectra of symmetric doubly stochastic matrices Rajesh Pereira a,, Mohammad Ali Vali b a Department
More informationLecture: Introduction to LP, SDP and SOCP
Lecture: Introduction to LP, SDP and SOCP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html wenzw@pku.edu.cn Acknowledgement:
More informationLecture 15 Perron-Frobenius Theory
EE363 Winter 2005-06 Lecture 15 Perron-Frobenius Theory Positive and nonnegative matrices and vectors Perron-Frobenius theorems Markov chains Economic growth Population dynamics Max-min and min-max characterization
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Simulated Annealing Barnabás Póczos & Ryan Tibshirani Andrey Markov Markov Chains 2 Markov Chains Markov chain: Homogen Markov chain: 3 Markov Chains Assume that the state
More informationDistributed Optimization over Networks Gossip-Based Algorithms
Distributed Optimization over Networks Gossip-Based Algorithms Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Random
More informationGraph Partitioning Using Random Walks
Graph Partitioning Using Random Walks A Convex Optimization Perspective Lorenzo Orecchia Computer Science Why Spectral Algorithms for Graph Problems in practice? Simple to implement Can exploit very efficient
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko
More information6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC
6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC 2003 2003.09.02.10 6. The Positivstellensatz Basic semialgebraic sets Semialgebraic sets Tarski-Seidenberg and quantifier elimination Feasibility
More informationContinuous Optimisation, Chpt 9: Semidefinite Optimisation
Continuous Optimisation, Chpt 9: Semidefinite Optimisation Peter J.C. Dickinson DMMP, University of Twente p.j.c.dickinson@utwente.nl http://dickinson.website/teaching/2017co.html version: 28/11/17 Monday
More informationThe maximal stable set problem : Copositive programming and Semidefinite Relaxations
The maximal stable set problem : Copositive programming and Semidefinite Relaxations Kartik Krishnan Department of Mathematical Sciences Rensselaer Polytechnic Institute Troy, NY 12180 USA kartis@rpi.edu
More informationAgenda. Applications of semidefinite programming. 1 Control and system theory. 2 Combinatorial and nonconvex optimization
Agenda Applications of semidefinite programming 1 Control and system theory 2 Combinatorial and nonconvex optimization 3 Spectral estimation & super-resolution Control and system theory SDP in wide use
More informationc 2005 Society for Industrial and Applied Mathematics
SIAM J. MATRIX ANAL. APPL. Vol. 27, No. 2, pp. 532 546 c 2005 Society for Industrial and Applied Mathematics LEAST-SQUARES COVARIANCE MATRIX ADJUSTMENT STEPHEN BOYD AND LIN XIAO Abstract. We consider the
More information4. Algebra and Duality
4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone
More information6 Markov Chain Monte Carlo (MCMC)
6 Markov Chain Monte Carlo (MCMC) The underlying idea in MCMC is to replace the iid samples of basic MC methods, with dependent samples from an ergodic Markov chain, whose limiting (stationary) distribution
More informationIII. Applications in convex optimization
III. Applications in convex optimization nonsymmetric interior-point methods partial separability and decomposition partial separability first order methods interior-point methods Conic linear optimization
More informationapproximation algorithms I
SUM-OF-SQUARES method and approximation algorithms I David Steurer Cornell Cargese Workshop, 201 meta-task encoded as low-degree polynomial in R x example: f(x) = i,j n w ij x i x j 2 given: functions
More informationConsensus Problems on Small World Graphs: A Structural Study
Consensus Problems on Small World Graphs: A Structural Study Pedram Hovareshti and John S. Baras 1 Department of Electrical and Computer Engineering and the Institute for Systems Research, University of
More informationDeterminant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University
Determinant maximization with linear matrix inequality constraints S. Boyd, L. Vandenberghe, S.-P. Wu Information Systems Laboratory Stanford University SCCM Seminar 5 February 1996 1 MAXDET problem denition
More informationRandomized Gossip Algorithms
2508 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 52, NO 6, JUNE 2006 Randomized Gossip Algorithms Stephen Boyd, Fellow, IEEE, Arpita Ghosh, Student Member, IEEE, Balaji Prabhakar, Member, IEEE, and Devavrat
More informationConvex Optimization and l 1 -minimization
Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More informationMIT Algebraic techniques and semidefinite optimization May 9, Lecture 21. Lecturer: Pablo A. Parrilo Scribe:???
MIT 6.972 Algebraic techniques and semidefinite optimization May 9, 2006 Lecture 2 Lecturer: Pablo A. Parrilo Scribe:??? In this lecture we study techniques to exploit the symmetry that can be present
More informationSemidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5
Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5 Instructor: Farid Alizadeh Scribe: Anton Riabov 10/08/2001 1 Overview We continue studying the maximum eigenvalue SDP, and generalize
More informationContinuous Optimisation, Chpt 9: Semidefinite Problems
Continuous Optimisation, Chpt 9: Semidefinite Problems Peter J.C. Dickinson DMMP, University of Twente p.j.c.dickinson@utwente.nl http://dickinson.website/teaching/2016co.html version: 21/11/16 Monday
More informationLECTURE 13 LECTURE OUTLINE
LECTURE 13 LECTURE OUTLINE Problem Structures Separable problems Integer/discrete problems Branch-and-bound Large sum problems Problems with many constraints Conic Programming Second Order Cone Programming
More informationConvex Optimization M2
Convex Optimization M2 Lecture 8 A. d Aspremont. Convex Optimization M2. 1/57 Applications A. d Aspremont. Convex Optimization M2. 2/57 Outline Geometrical problems Approximation problems Combinatorial
More informationLECTURE NOTE #11 PROF. ALAN YUILLE
LECTURE NOTE #11 PROF. ALAN YUILLE 1. NonLinear Dimension Reduction Spectral Methods. The basic idea is to assume that the data lies on a manifold/surface in D-dimensional space, see figure (1) Perform
More informationSemidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 4
Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 4 Instructor: Farid Alizadeh Scribe: Haengju Lee 10/1/2001 1 Overview We examine the dual of the Fermat-Weber Problem. Next we will
More informationConvex sets, conic matrix factorizations and conic rank lower bounds
Convex sets, conic matrix factorizations and conic rank lower bounds Pablo A. Parrilo Laboratory for Information and Decision Systems Electrical Engineering and Computer Science Massachusetts Institute
More informationExample: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma
4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid
More informationGraph Laplacian Regularization for Large-Scale Semidefinite Programming
Graph Laplacian Regularization for Large-Scale Semidefinite Programming Kilian Q. Weinberger Dept of Computer and Information Science U of Pennsylvania, Philadelphia, PA 19104 kilianw@seas.upenn.edu Qihui
More informationOptimal Scaling of a Gradient Method for Distributed Resource Allocation 1
JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 129, No. 3, pp. 469 488, June 2006 ( C 2006) DOI: 10.1007/s10957-006-9080-1 Optimal Scaling of a Gradient Method for Distributed Resource Allocation
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationThe Miracles of Tropical Spectral Theory
The Miracles of Tropical Spectral Theory Emmanuel Tsukerman University of California, Berkeley e.tsukerman@berkeley.edu July 31, 2015 1 / 110 Motivation One of the most important results in max-algebra
More informationCommunities, Spectral Clustering, and Random Walks
Communities, Spectral Clustering, and Random Walks David Bindel Department of Computer Science Cornell University 26 Sep 2011 20 21 19 16 22 28 17 18 29 26 27 30 23 1 25 5 8 24 2 4 14 3 9 13 15 11 10 12
More informationInderjit Dhillon The University of Texas at Austin
Inderjit Dhillon The University of Texas at Austin ( Universidad Carlos III de Madrid; 15 th June, 2012) (Based on joint work with J. Brickell, S. Sra, J. Tropp) Introduction 2 / 29 Notion of distance
More informationSparse Covariance Selection using Semidefinite Programming
Sparse Covariance Selection using Semidefinite Programming A. d Aspremont ORFE, Princeton University Joint work with O. Banerjee, L. El Ghaoui & G. Natsoulis, U.C. Berkeley & Iconix Pharmaceuticals Support
More informationAnalysis and synthesis: a complexity perspective
Analysis and synthesis: a complexity perspective Pablo A. Parrilo ETH ZürichZ control.ee.ethz.ch/~parrilo Outline System analysis/design Formal and informal methods SOS/SDP techniques and applications
More informationA priori bounds on the condition numbers in interior-point methods
A priori bounds on the condition numbers in interior-point methods Florian Jarre, Mathematisches Institut, Heinrich-Heine Universität Düsseldorf, Germany. Abstract Interior-point methods are known to be
More informationRelaxations and Randomized Methods for Nonconvex QCQPs
Relaxations and Randomized Methods for Nonconvex QCQPs Alexandre d Aspremont, Stephen Boyd EE392o, Stanford University Autumn, 2003 Introduction While some special classes of nonconvex problems can be
More informationAdvances in Convex Optimization: Theory, Algorithms, and Applications
Advances in Convex Optimization: Theory, Algorithms, and Applications Stephen Boyd Electrical Engineering Department Stanford University (joint work with Lieven Vandenberghe, UCLA) ISIT 02 ISIT 02 Lausanne
More information- Well-characterized problems, min-max relations, approximate certificates. - LP problems in the standard form, primal and dual linear programs
LP-Duality ( Approximation Algorithms by V. Vazirani, Chapter 12) - Well-characterized problems, min-max relations, approximate certificates - LP problems in the standard form, primal and dual linear programs
More informationConvex Optimization in Classification Problems
New Trends in Optimization and Computational Algorithms December 9 13, 2001 Convex Optimization in Classification Problems Laurent El Ghaoui Department of EECS, UC Berkeley elghaoui@eecs.berkeley.edu 1
More informationAdvanced SDPs Lecture 6: March 16, 2017
Advanced SDPs Lecture 6: March 16, 2017 Lecturers: Nikhil Bansal and Daniel Dadush Scribe: Daniel Dadush 6.1 Notation Let N = {0, 1,... } denote the set of non-negative integers. For α N n, define the
More informationAgreement algorithms for synchronization of clocks in nodes of stochastic networks
UDC 519.248: 62 192 Agreement algorithms for synchronization of clocks in nodes of stochastic networks L. Manita, A. Manita National Research University Higher School of Economics, Moscow Institute of
More informationEE 227A: Convex Optimization and Applications October 14, 2008
EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider
More informationRandom Walks on Graphs. One Concrete Example of a random walk Motivation applications
Random Walks on Graphs Outline One Concrete Example of a random walk Motivation applications shuffling cards universal traverse sequence self stabilizing token management scheme random sampling enumeration
More informationFast ADMM for Sum of Squares Programs Using Partial Orthogonality
Fast ADMM for Sum of Squares Programs Using Partial Orthogonality Antonis Papachristodoulou Department of Engineering Science University of Oxford www.eng.ox.ac.uk/control/sysos antonis@eng.ox.ac.uk with
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationMarkov Chains, Random Walks on Graphs, and the Laplacian
Markov Chains, Random Walks on Graphs, and the Laplacian CMPSCI 791BB: Advanced ML Sridhar Mahadevan Random Walks! There is significant interest in the problem of random walks! Markov chain analysis! Computer
More informationData Analysis and Manifold Learning Lecture 9: Diffusion on Manifolds and on Graphs
Data Analysis and Manifold Learning Lecture 9: Diffusion on Manifolds and on Graphs Radu Horaud INRIA Grenoble Rhone-Alpes, France Radu.Horaud@inrialpes.fr http://perception.inrialpes.fr/ Outline of Lecture
More information15. Conic optimization
L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationPOLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS
POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS Sercan Yıldız syildiz@samsi.info in collaboration with Dávid Papp (NCSU) OPT Transition Workshop May 02, 2017 OUTLINE Polynomial optimization and
More informationSpectra of Large Random Stochastic Matrices & Relaxation in Complex Systems
Spectra of Large Random Stochastic Matrices & Relaxation in Complex Systems Reimer Kühn Disordered Systems Group Department of Mathematics, King s College London Random Graphs and Random Processes, KCL
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationReal Symmetric Matrices and Semidefinite Programming
Real Symmetric Matrices and Semidefinite Programming Tatsiana Maskalevich Abstract Symmetric real matrices attain an important property stating that all their eigenvalues are real. This gives rise to many
More information8.1 Concentration inequality for Gaussian random matrix (cont d)
MGMT 69: Topics in High-dimensional Data Analysis Falll 26 Lecture 8: Spectral clustering and Laplacian matrices Lecturer: Jiaming Xu Scribe: Hyun-Ju Oh and Taotao He, October 4, 26 Outline Concentration
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationLecture 20: Reversible Processes and Queues
Lecture 20: Reversible Processes and Queues 1 Examples of reversible processes 11 Birth-death processes We define two non-negative sequences birth and death rates denoted by {λ n : n N 0 } and {µ n : n
More informationSome Properties of the Augmented Lagrangian in Cone Constrained Optimization
MATHEMATICS OF OPERATIONS RESEARCH Vol. 29, No. 3, August 2004, pp. 479 491 issn 0364-765X eissn 1526-5471 04 2903 0479 informs doi 10.1287/moor.1040.0103 2004 INFORMS Some Properties of the Augmented
More informationMULTISCALE DISTRIBUTED ESTIMATION WITH APPLICATIONS TO GPS AUGMENTATION AND NETWORK SPECTRA
MULTISCALE DISTRIBUTED ESTIMATION WITH APPLICATIONS TO GPS AUGMENTATION AND NETWORK SPECTRA A DISSERTATION SUBMITTED TO THE DEPARTMENT OF AERONAUTICS AND ASTRONAUTICS AND THE COMMITTEE ON GRADUATE STUDIES
More informationLinearly-solvable Markov decision problems
Advances in Neural Information Processing Systems 2 Linearly-solvable Markov decision problems Emanuel Todorov Department of Cognitive Science University of California San Diego todorov@cogsci.ucsd.edu
More informationAgenda. 1 Cone programming. 2 Convex cones. 3 Generalized inequalities. 4 Linear programming (LP) 5 Second-order cone programming (SOCP)
Agenda 1 Cone programming 2 Convex cones 3 Generalized inequalities 4 Linear programming (LP) 5 Second-order cone programming (SOCP) 6 Semidefinite programming (SDP) 7 Examples Optimization problem in
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationLECTURE 15 Markov chain Monte Carlo
LECTURE 15 Markov chain Monte Carlo There are many settings when posterior computation is a challenge in that one does not have a closed form expression for the posterior distribution. Markov chain Monte
More informationminimize x x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x 2 u 2, 5x 1 +76x 2 1,
4 Duality 4.1 Numerical perturbation analysis example. Consider the quadratic program with variables x 1, x 2, and parameters u 1, u 2. minimize x 2 1 +2x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x
More information15-780: LinearProgramming
15-780: LinearProgramming J. Zico Kolter February 1-3, 2016 1 Outline Introduction Some linear algebra review Linear programming Simplex algorithm Duality and dual simplex 2 Outline Introduction Some linear
More informationStochastic Subgradient Methods
Stochastic Subgradient Methods Stephen Boyd and Almir Mutapcic Notes for EE364b, Stanford University, Winter 26-7 April 13, 28 1 Noisy unbiased subgradient Suppose f : R n R is a convex function. We say
More information1 The independent set problem
ORF 523 Lecture 11 Spring 2016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Tuesday, March 29, 2016 When in doubt on the accuracy of these notes, please cross chec with the instructor
More informationSemidefinite Programming
Semidefinite Programming Basics and SOS Fernando Mário de Oliveira Filho Campos do Jordão, 2 November 23 Available at: www.ime.usp.br/~fmario under talks Conic programming V is a real vector space h, i
More informationPrimal-Dual Geometry of Level Sets and their Explanatory Value of the Practical Performance of Interior-Point Methods for Conic Optimization
Primal-Dual Geometry of Level Sets and their Explanatory Value of the Practical Performance of Interior-Point Methods for Conic Optimization Robert M. Freund M.I.T. June, 2010 from papers in SIOPT, Mathematics
More informationOptimization over Polynomials with Sums of Squares and Moment Matrices
Optimization over Polynomials with Sums of Squares and Moment Matrices Monique Laurent Centrum Wiskunde & Informatica (CWI), Amsterdam and University of Tilburg Positivity, Valuations and Quadratic Forms
More informationConvergence Rate of Markov Chains
Convergence Rate of Markov Chains Will Perkins April 16, 2013 Convergence Last class we saw that if X n is an irreducible, aperiodic, positive recurrent Markov chain, then there exists a stationary distribution
More informationGeometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as
Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.
More informationAlgebraic Representation of Networks
Algebraic Representation of Networks 0 1 2 1 1 0 0 1 2 0 0 1 1 1 1 1 Hiroki Sayama sayama@binghamton.edu Describing networks with matrices (1) Adjacency matrix A matrix with rows and columns labeled by
More informationLecture: Expanders, in theory and in practice (2 of 2)
Stat260/CS294: Spectral Graph Methods Lecture 9-02/19/2015 Lecture: Expanders, in theory and in practice (2 of 2) Lecturer: Michael Mahoney Scribe: Michael Mahoney Warning: these notes are still very rough.
More informationCourse Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979)
Course Outline FRTN Multivariable Control, Lecture Automatic Control LTH, 6 L-L Specifications, models and loop-shaping by hand L6-L8 Limitations on achievable performance L9-L Controller optimization:
More informationMarkov Chains and MCMC
Markov Chains and MCMC Markov chains Let S = {1, 2,..., N} be a finite set consisting of N states. A Markov chain Y 0, Y 1, Y 2,... is a sequence of random variables, with Y t S for all points in time
More informationCopositive Programming and Combinatorial Optimization
Copositive Programming and Combinatorial Optimization Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria joint work with I.M. Bomze (Wien) and F. Jarre (Düsseldorf) IMA
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More information