CS Lecture 8 & 9. Lagrange Multipliers & Varitional Bounds
|
|
|
- Lenard Davidson
- 5 years ago
- Views:
Transcription
1 CS 6347 Lecture 8 & 9 Lagrange Multipliers & Varitional Bounds
2 General Optimization subject to: min ff 0() R nn ff ii 0, h ii = 0, ii = 1,, mm ii = 1,, pp 2
3 General Optimization subject to: min ff 0() R nn ff 0 is not necessarily convex ff ii 0, h ii = 0, ii = 1,, mm ii = 1,, pp 3
4 General Optimization subject to: min ff 0() R nn ff ii 0, h ii = 0, ii = 1,, mm ii = 1,, pp Constraints can be arbitrary functions 4
5 Lagrangian mm LL, λλ, νν = ff 0 + ii=1 pp λλ ii ff ii + ii=1 νν ii h ii () Incorporate constraints into a new objective function λλ 0 and νν are vectors of Lagrange multipliers The Lagrange multipliers can be thought of as soft constraints 5
6 Duality Construct a dual function by minimizing the Lagrangian over the primal variables gg λλ, νν = inf LL(, λλ, νν) gg λλ, νν = whenever the Lagrangian is not bounded from below for a fixed λλ and νν 6
7 The Primal Problem subject to: min ff 0() R nn ff ii 0, h ii = 0, ii = 1,, mm ii = 1,, pp Equivalently, inf sup λλ 0,νν LL(, λλ, νν) 7
8 The Dual Problem Equivalently, sup λλ 0,νν sup λλ 0,νν inf gg(λλ, νν) LL(, λλ, νν) The dual problem is always concave, even if the primal problem is not convex 8
9 Primal vs. Dual Why? sup λλ 0,νν inf LL(, λλ, νν) inf gg λλ, νν LL(, λλ, νν) for all sup λλ 0,νν LL(, λλ, νν) LL, λλ, νν ff 0 ( ) for any feasible, λλ 0 is feasible if it satisfies all of the constraints Let be the optimal solution to the primal problem and λλ 0 gg λλ, νν LL, λλ, νν ff 0 9
10 Duality Under certain conditions, the two optimization problems are equivalent sup λλ 0,νν inf LL(, λλ, νν) = inf This is called strong duality sup λλ 0,νν LL(, λλ, νν) If the inequality is strict, then we say that there is a duality gap Size of gap measured by the difference between the two sides of the inequality 10
11 Slater s Condition For any optimization problem of the form subject to: min ff 0() R nn ff ii 0, ii = 1,, mm AAAA = bb where ff 0,, ff mm are convex functions, strong duality holds if there exists an such that ff ii < 0, ii = 1,, mm AAAA = bb 11
12 Some Examples Minimize 2 + yy 2 subject to + yy 2 Maximize log yy log yy zz log zz subject to, yy, zz 0 and + yy + zz = 1 Minimize subject to + yy 1
13 Approximate Marginal Inference Last week: approximate MAP inference Reparamaterizations Linear programming over the local marginal polytope Approximate marginal inference (e.g., pp(yy ii )) Sampling methods (MCMC, etc.) Variational methods (loopy belief propagation, TRW, etc.) 13
14 KL Divergence In order to perform approximate marginal inference, we will try to find distributions that approximate the true distribution Ideally, the marginals of the approximating distribution should be easy to compute For this, we need a notion of closeness of distributions 14
15 KL Divergence pp DD(pp qq = pp log qq Called the Kullback-Leibler divergence DD(pp qq 0 with equality if and only if pp = qq Not symmetric, DD(pp qq DD(qq pp) 15
16 Jensen's Inequality Let ff() be a convex function and aa ii 0 such that ii aa ii = 1 aa ii ff( ii ) ff ii aa ii ii ii Useful inequality when dealing with convex/concave functions When does equality hold? 16
17 KL Divergence pp DD(pp qq = pp log qq Suppose that we want to approximate the distribution pp with some other distribution qq in some family of distributions QQ Could minimize KL divergence in one of two ways arg min qq QQ DD(pp qq) arg min qq QQ DD(qq pp) 17
18 KL Divergence pp DD(pp qq = pp log qq Suppose that we want to approximate the distribution pp with some other distribution qq in some family of distributions QQ Could minimize KL divergence in one of two ways arg min qq QQ DD(pp qq) arg min qq QQ DD(qq pp) Called the M-projection Called the I-projection 18
19 KL Divergence pp DD(pp qq = pp log qq Suppose that we want to approximate the distribution pp with some other distribution qq in some family of distributions QQ Could minimize KL divergence in one of two ways arg min qq QQ DD(pp qq) arg min qq QQ DD(qq pp) As hard as the original inference problem Potentially easier 19
20 Variational Inference Let's let pp = 1 ZZ cc ψψ cc cc be the distribution that we want to approximate with distribution qq qq DD(qq pp = qq log pp = qq log qq() qq log pp = HH(qq) qq log pp = HH(qq) + log ZZ qq log ψψ cc cc CC = HH(qq) + log ZZ CC CC qq CC CC log ψψ cc cc 20
21 Variational Inference Let's let pp = 1 ZZ cc ψψ cc cc be the distribution that we want to approximate with distribution qq qq DD(qq pp = qq log pp = qq log qq() qq log pp = HH qq qq log pp = HH qq + log ZZ qq log ψψ cc cc CC Where have we seen this before? = HH qq + log ZZ CC CC qq CC CC log ψψ cc cc 21
22 MAP Integer Program max ττ such that ii VV ii ii,jj EE ττ ii ii log φφ ii ii + ττ iiii ii, jj log ψψ iiii ( ii, jj ) ii, jj ττ ii ii = 1 ii jj ττ iiii ( ii, jj ) = ττ ii ( ii ) ττ ii ii {0,1} For all ii VV For all ii, jj EE, ii For all ii VV, ii ττ iiii ii, jj {0,1} For all ii, jj EE, ii, jj 22
23 Variational Inference Let's let pp = 1 ZZ cc ψψ cc cc be the distribution that we want to approximate with distribution qq DD(qq pp = HH qq + log ZZ CC CC qq CC CC log ψψ cc cc Using the observation that the KL divergence is non-negative log ZZ HH qq + CC CC qq CC CC log ψψ cc cc 23
24 Variational Inference Let's let pp = 1 ZZ cc ψψ cc cc be the distribution that we want to approximate with distribution qq DD(qq pp = HH qq + log ZZ CC CC qq CC CC log ψψ cc cc Using the observation that the KL divergence is non-negative log ZZ HH(qq) + CC CC qq CC CC log ψψ cc cc This lower bound holds for any qq 24
25 Variational Inference Let's let pp = 1 ZZ cc ψψ cc cc be the distribution that we want to approximate with distribution qq DD(qq pp = HH qq + log ZZ CC CC qq CC CC log ψψ cc cc Using the observation that the KL divergence is non-negative log ZZ HH qq + CC CC qq CC CC log ψψ cc cc Maximizing this over qq gives equality 25
26 Variational Inference log ZZ HH(qq) + CC CC qq CC CC log ψψ cc cc The right hand side is a concave function of qq Despite that, this optimization problem is hard! (surprised?) Exponentially many distributions, qq We need a more compact way to express them Computing the entropy is non-trivial 26
27 Variational Inference log ZZ HH(qq) + CC CC qq CC CC log ψψ cc cc Two kinds of methods that are used to deal with these difficulties Mean-field methods: assume that the approximating distribution factorizes as qq ii VV qq ii ii Similar idea to naïve Bayes Relaxation based methods: replace hard pieces of the optimization with easier optimization problems Similar to the MAP IP -> MAP LP relaxation 27
28 Relaxation Approach log ZZ HH(qq) + CC CC qq CC CC log ψψ cc cc To handle the representation problem, we can use the same LP relaxation trick that we did before For each ττ in the marginal polytope, we can rewrite the RHS as log ZZ HH ττ + CC CC ττ CC CC log ψψ cc cc 28
29 Relaxation Approach log ZZ HH(qq) + CC CC qq CC CC log ψψ cc cc To handle the representation problem, we can use the same LP relaxation trick that we did before For each ττ in the marginal polytope, we can rewrite the RHS as log ZZ HH(ττ) + CC CC ττ CC CC log ψψ cc cc Maximum entropy over all ττ with these marginals 29
30 Relaxation Approach max ττ M HH ττ + CC CC ττ CC CC log ψψ cc cc Marginal polytope, MM, is intractable to optimize over Use the local polytope, TT! ττ CC CC = ττ ii ii ffffff aaaaaa CC, ii VV CC ii ii ττ ii ii = 1 ffffff aaaaaa ii VV 30
31 Relaxation Approach max ττ TT HH(ττ) + CC CC ττ CC CC log ψψ cc cc Even with the polytope relaxation, the optimization problem still remains challenging as computing the entropy remains nontrivial We will need to approximate the entropy as well For which distributions is it easy to compute the entropy? 31
32 Tree Reparameterization On a tree, the joint distribution factorizes in a special way pp 1,, nn = 1 ZZ ii VV pp ii ( ii ) ii,jj EE pp iiii ( ii, jj ) pp ii ii pp jj ( jj ) pp ii is the marginal distribution of the ii ttt variable and pp iiii is the maxmarginal distribution for the edge ii, jj EE This applies to clique trees as well (i.e., when the factor graph is a tree) 32
33 Tree Reparameterization On a tree, the joint distribution factorizes in a special way pp 1,, nn = 1 ZZ ii VV pp CC ( CC ) pp ii ( ii ) ii CC pp ii ii CC pp ii is the marginal distribution of the ii ttt variable and pp iiii is the maxmarginal distribution for the edge ii, jj EE This applies to clique trees as well (i.e., when the factor graph is a tree) 33
34 Entropy of a Tree Given this factorization, we can easily compute the entropy of a tree structured distribution HH TTTTTTTT = pp ii ii log pp ii ( ii ) ii V ii CC CC pp CC CC pp CC ( CC ) log ii CC pp ii ii This only depends on the marginals Use this as an approximation for general distributions! 34
35 Bethe Free Energy Combining these two approximations gives us the so-called Bethe free energy approximation max ττ TT HH BB ττ + CC CC ττ CC CC log ψψ cc cc where HH BB ττ = ττ ii ii log ττ ii ( ii ) ii V ii CC CC ττ CC CC ττ CC ( CC ) log ii CC ττ ii ii 35
36 Bethe Free Energy max ττ TT HH BB ττ + CC CC ττ CC CC log ψψ cc cc This is not a concave optimization problem for general graphs It is still difficult to maximize However, fixed points of loopy belief propagation correspond to saddle points of this objective over the local marginal polytope 36
Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers
Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex
14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
14 : Theory of Variational Inference: Inner and Outer Approximation
10-708: Probabilistic Graphical Models 10-708, Spring 2014 14 : Theory of Variational Inference: Inner and Outer Approximation Lecturer: Eric P. Xing Scribes: Yu-Hsin Kuo, Amos Ng 1 Introduction Last lecture
Probabilistic Graphical Models. Theory of Variational Inference: Inner and Outer Approximation. Lecture 15, March 4, 2013
School of Computer Science Probabilistic Graphical Models Theory of Variational Inference: Inner and Outer Approximation Junming Yin Lecture 15, March 4, 2013 Reading: W & J Book Chapters 1 Roadmap Two
Probabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Variational Inference IV: Variational Principle II Junming Yin Lecture 17, March 21, 2012 X 1 X 1 X 1 X 1 X 2 X 3 X 2 X 2 X 3 X 3 Reading: X 4
Duality (Continued) min f ( x), X R R. Recall, the general primal problem is. The Lagrangian is a function. defined by
Duality (Continued) Recall, the general primal problem is min f ( x), xx g( x) 0 n m where X R, f : X R, g : XR ( X). he Lagrangian is a function L: XR R m defined by L( xλ, ) f ( x) λ g( x) Duality (Continued)
14 : Theory of Variational Inference: Inner and Outer Approximation
10-708: Probabilistic Graphical Models 10-708, Spring 2017 14 : Theory of Variational Inference: Inner and Outer Approximation Lecturer: Eric P. Xing Scribes: Maria Ryskina, Yen-Chia Hsu 1 Introduction
13: Variational inference II
10-708: Probabilistic Graphical Models, Spring 2015 13: Variational inference II Lecturer: Eric P. Xing Scribes: Ronghuo Zheng, Zhiting Hu, Yuntian Deng 1 Introduction We started to talk about variational
Constrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
Variational algorithms for marginal MAP
Variational algorithms for marginal MAP Alexander Ihler UC Irvine CIOG Workshop November 2011 Variational algorithms for marginal MAP Alexander Ihler UC Irvine CIOG Workshop November 2011 Work with Qiang
Support Vector Machines. CSE 4309 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Support Vector Machines CSE 4309 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 A Linearly Separable Problem Consider the binary classification
13 : Variational Inference: Loopy Belief Propagation and Mean Field
10-708: Probabilistic Graphical Models 10-708, Spring 2012 13 : Variational Inference: Loopy Belief Propagation and Mean Field Lecturer: Eric P. Xing Scribes: Peter Schulam and William Wang 1 Introduction
Inferring the origin of an epidemic with a dynamic message-passing algorithm
Inferring the origin of an epidemic with a dynamic message-passing algorithm HARSH GUPTA (Based on the original work done by Andrey Y. Lokhov, Marc Mézard, Hiroki Ohta, and Lenka Zdeborová) Paper Andrey
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
Lecture 11. Kernel Methods
Lecture 11. Kernel Methods COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Andrey Kan Copyright: University of Melbourne This lecture The kernel trick Efficient computation of a dot product
Lagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
Lagrangian Duality and Convex Optimization
Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?
Probabilistic and Bayesian Machine Learning
Probabilistic and Bayesian Machine Learning Day 4: Expectation and Belief Propagation Yee Whye Teh ywteh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit University College London http://www.gatsby.ucl.ac.uk/
Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016
Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,
Charge carrier density in metals and semiconductors
Charge carrier density in metals and semiconductors 1. Introduction The Hall Effect Particles must overlap for the permutation symmetry to be relevant. We saw examples of this in the exchange energy in
Uncertain Compression & Graph Coloring. Madhu Sudan Harvard
Uncertain Compression & Graph Coloring Madhu Sudan Harvard Based on joint works with: (1) Adam Kalai (MSR), Sanjeev Khanna (U.Penn), Brendan Juba (WUStL) (2) Elad Haramaty (Harvard) (3) Badih Ghazi (MIT),
Advanced data analysis
Advanced data analysis Akisato Kimura ( 木村昭悟 ) NTT Communication Science Laboratories E-mail: akisato@ieee.org Advanced data analysis 1. Introduction (Aug 20) 2. Dimensionality reduction (Aug 20,21) PCA,
Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
Lecture #21. c T x Ax b. maximize subject to
COMPSCI 330: Design and Analysis of Algorithms 11/11/2014 Lecture #21 Lecturer: Debmalya Panigrahi Scribe: Samuel Haney 1 Overview In this lecture, we discuss linear programming. We first show that the
Posterior Regularization
Posterior Regularization 1 Introduction One of the key challenges in probabilistic structured learning, is the intractability of the posterior distribution, for fast inference. There are numerous methods
Bayesian Learning in Undirected Graphical Models
Bayesian Learning in Undirected Graphical Models Zoubin Ghahramani Gatsby Computational Neuroscience Unit University College London, UK http://www.gatsby.ucl.ac.uk/ Work with: Iain Murray and Hyun-Chul
Variational Inference (11/04/13)
STA561: Probabilistic machine learning Variational Inference (11/04/13) Lecturer: Barbara Engelhardt Scribes: Matt Dickenson, Alireza Samany, Tracy Schifeling 1 Introduction In this lecture we will further
Variations. ECE 6540, Lecture 02 Multivariate Random Variables & Linear Algebra
Variations ECE 6540, Lecture 02 Multivariate Random Variables & Linear Algebra Last Time Probability Density Functions Normal Distribution Expectation / Expectation of a function Independence Uncorrelated
Probabilistic Graphical Models
Probabilistic Graphical Models Lecture 9: Variational Inference Relaxations Volkan Cevher, Matthias Seeger Ecole Polytechnique Fédérale de Lausanne 24/10/2011 (EPFL) Graphical Models 24/10/2011 1 / 15
Convex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
Support Vector Machines: Maximum Margin Classifiers
Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind
Worksheets for GCSE Mathematics. Quadratics. mr-mathematics.com Maths Resources for Teachers. Algebra
Worksheets for GCSE Mathematics Quadratics mr-mathematics.com Maths Resources for Teachers Algebra Quadratics Worksheets Contents Differentiated Independent Learning Worksheets Solving x + bx + c by factorisation
Discrete Markov Random Fields the Inference story. Pradeep Ravikumar
Discrete Markov Random Fields the Inference story Pradeep Ravikumar Graphical Models, The History How to model stochastic processes of the world? I want to model the world, and I like graphs... 2 Mid to
CS249: ADVANCED DATA MINING
CS249: ADVANCED DATA MINING Support Vector Machine and Neural Network Instructor: Yizhou Sun yzsun@cs.ucla.edu April 24, 2017 Homework 1 Announcements Due end of the day of this Friday (11:59pm) Reminder
CS229T/STATS231: Statistical Learning Theory. Lecturer: Tengyu Ma Lecture 11 Scribe: Jongho Kim, Jamie Kang October 29th, 2018
CS229T/STATS231: Statistical Learning Theory Lecturer: Tengyu Ma Lecture 11 Scribe: Jongho Kim, Jamie Kang October 29th, 2018 1 Overview This lecture mainly covers Recall the statistical theory of GANs
Junction Tree, BP and Variational Methods
Junction Tree, BP and Variational Methods Adrian Weller MLSALT4 Lecture Feb 21, 2018 With thanks to David Sontag (MIT) and Tony Jebara (Columbia) for use of many slides and illustrations For more information,
Classical RSA algorithm
Classical RSA algorithm We need to discuss some mathematics (number theory) first Modulo-NN arithmetic (modular arithmetic, clock arithmetic) 9 (mod 7) 4 3 5 (mod 7) congruent (I will also use = instead
Lecture 3. STAT161/261 Introduction to Pattern Recognition and Machine Learning Spring 2018 Prof. Allie Fletcher
Lecture 3 STAT161/261 Introduction to Pattern Recognition and Machine Learning Spring 2018 Prof. Allie Fletcher Previous lectures What is machine learning? Objectives of machine learning Supervised and
EE364a Review Session 5
EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary
Expectation Propagation performs smooth gradient descent GUILLAUME DEHAENE
Expectation Propagation performs smooth gradient descent 1 GUILLAUME DEHAENE In a nutshell Problem: posteriors are uncomputable Solution: parametric approximations 2 But which one should we choose? Laplace?
12 : Variational Inference I
10-708: Probabilistic Graphical Models, Spring 2015 12 : Variational Inference I Lecturer: Eric P. Xing Scribes: Fattaneh Jabbari, Eric Lei, Evan Shapiro 1 Introduction Probabilistic inference is one of
Convex Optimization and Support Vector Machine
Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We
4. Algebra and Duality
4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone
13 : Variational Inference: Loopy Belief Propagation
10-708: Probabilistic Graphical Models 10-708, Spring 2014 13 : Variational Inference: Loopy Belief Propagation Lecturer: Eric P. Xing Scribes: Rajarshi Das, Zhengzhong Liu, Dishan Gupta 1 Introduction
Probabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Variational Inference II: Mean Field Method and Variational Principle Junming Yin Lecture 15, March 7, 2012 X 1 X 1 X 1 X 1 X 2 X 3 X 2 X 2 X 3
CS249: ADVANCED DATA MINING
CS249: ADVANCED DATA MINING Vector Data: Clustering: Part II Instructor: Yizhou Sun yzsun@cs.ucla.edu May 3, 2017 Methods to Learn: Last Lecture Classification Clustering Vector Data Text Data Recommender
Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark
Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian
A notion of Total Dual Integrality for Convex, Semidefinite and Extended Formulations
A notion of for Convex, Semidefinite and Extended Formulations Marcel de Carli Silva Levent Tunçel April 26, 2018 A vector in R n is integral if each of its components is an integer, A vector in R n is
5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
Convex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
EE 227A: Convex Optimization and Applications October 14, 2008
EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider
Linear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
Tutorial on Convex Optimization: Part II
Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation
Algorithms for Reasoning with Probabilistic Graphical Models. Class 3: Approximate Inference. International Summer School on Deep Learning July 2017
Algorithms for Reasoning with Probabilistic Graphical Models Class 3: Approximate Inference International Summer School on Deep Learning July 2017 Prof. Rina Dechter Prof. Alexander Ihler Approximate Inference
Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
A linear operator of a new class of multivalent harmonic functions
International Journal of Scientific Research Publications, Volume 7, Issue 8, August 2017 278 A linear operator of a new class of multivalent harmonic functions * WaggasGalibAtshan, ** Ali Hussein Battor,
Duality. Geoff Gordon & Ryan Tibshirani Optimization /
Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider
Convex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
Lecture 9: PGM Learning
13 Oct 2014 Intro. to Stats. Machine Learning COMP SCI 4401/7401 Table of Contents I Learning parameters in MRFs 1 Learning parameters in MRFs Inference and Learning Given parameters (of potentials) and
Bayesian Machine Learning - Lecture 7
Bayesian Machine Learning - Lecture 7 Guido Sanguinetti Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh gsanguin@inf.ed.ac.uk March 4, 2015 Today s lecture 1
I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
Lagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
Duality in Linear Programs. Lecturer: Ryan Tibshirani Convex Optimization /36-725
Duality in Linear Programs Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: proximal gradient descent Consider the problem x g(x) + h(x) with g, h convex, g differentiable, and
Section Notes 9. IP: Cutting Planes. Applied Math 121. Week of April 12, 2010
Section Notes 9 IP: Cutting Planes Applied Math 121 Week of April 12, 2010 Goals for the week understand what a strong formulations is. be familiar with the cutting planes algorithm and the types of cuts
Worksheets for GCSE Mathematics. Algebraic Expressions. Mr Black 's Maths Resources for Teachers GCSE 1-9. Algebra
Worksheets for GCSE Mathematics Algebraic Expressions Mr Black 's Maths Resources for Teachers GCSE 1-9 Algebra Algebraic Expressions Worksheets Contents Differentiated Independent Learning Worksheets
General Strong Polarization
General Strong Polarization Madhu Sudan Harvard University Joint work with Jaroslaw Blasiok (Harvard), Venkatesan Gurswami (CMU), Preetum Nakkiran (Harvard) and Atri Rudra (Buffalo) December 4, 2017 IAS:
Example: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma
4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid
Lecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module - 5 Lecture - 22 SVM: The Dual Formulation Good morning.
Topic: Primal-Dual Algorithms Date: We finished our discussion of randomized rounding and began talking about LP Duality.
CS787: Advanced Algorithms Scribe: Amanda Burton, Leah Kluegel Lecturer: Shuchi Chawla Topic: Primal-Dual Algorithms Date: 10-17-07 14.1 Last Time We finished our discussion of randomized rounding and
Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
The Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:
HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course
Lecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
17 Variational Inference
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms for Inference Fall 2014 17 Variational Inference Prompted by loopy graphs for which exact
Estimate by the L 2 Norm of a Parameter Poisson Intensity Discontinuous
Research Journal of Mathematics and Statistics 6: -5, 24 ISSN: 242-224, e-issn: 24-755 Maxwell Scientific Organization, 24 Submied: September 8, 23 Accepted: November 23, 23 Published: February 25, 24
Grover s algorithm. We want to find aa. Search in an unordered database. QC oracle (as usual) Usual trick
Grover s algorithm Search in an unordered database Example: phonebook, need to find a person from a phone number Actually, something else, like hard (e.g., NP-complete) problem 0, xx aa Black box ff xx
Solving Dual Problems
Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem
Bayesian Learning in Undirected Graphical Models
Bayesian Learning in Undirected Graphical Models Zoubin Ghahramani Gatsby Computational Neuroscience Unit University College London, UK http://www.gatsby.ucl.ac.uk/ and Center for Automated Learning and
ICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
Systems of Linear Equations
Systems of Linear Equations As stated in Section G, Definition., a linear equation in two variables is an equation of the form AAAA + BBBB = CC, where AA and BB are not both zero. Such an equation has
Lecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
16.1 L.P. Duality Applied to the Minimax Theorem
CS787: Advanced Algorithms Scribe: David Malec and Xiaoyong Chai Lecturer: Shuchi Chawla Topic: Minimax Theorem and Semi-Definite Programming Date: October 22 2007 In this lecture, we first conclude our
Lecture Support Vector Machine (SVM) Classifiers
Introduction to Machine Learning Lecturer: Amir Globerson Lecture 6 Fall Semester Scribe: Yishay Mansour 6.1 Support Vector Machine (SVM) Classifiers Classification is one of the most important tasks in
Dan Roth 461C, 3401 Walnut
CIS 519/419 Applied Machine Learning www.seas.upenn.edu/~cis519 Dan Roth danroth@seas.upenn.edu http://www.cis.upenn.edu/~danroth/ 461C, 3401 Walnut Slides were created by Dan Roth (for CIS519/419 at Penn
Integer Programming ISE 418. Lecture 8. Dr. Ted Ralphs
Integer Programming ISE 418 Lecture 8 Dr. Ted Ralphs ISE 418 Lecture 8 1 Reading for This Lecture Wolsey Chapter 2 Nemhauser and Wolsey Sections II.3.1, II.3.6, II.4.1, II.4.2, II.5.4 Duality for Mixed-Integer
EE/AA 578, Univ of Washington, Fall Duality
7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
An introductory example
CS1 Lecture 9 An introductory example Suppose that a company that produces three products wishes to decide the level of production of each so as to maximize profits. Let x 1 be the amount of Product 1
subject to (x 2)(x 4) u,
Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the
10701 Recitation 5 Duality and SVM. Ahmed Hefny
10701 Recitation 5 Duality and SVM Ahmed Hefny Outline Langrangian and Duality The Lagrangian Duality Eamples Support Vector Machines Primal Formulation Dual Formulation Soft Margin and Hinge Loss Lagrangian
Inference as Optimization
Inference as Optimization Sargur Srihari srihari@cedar.buffalo.edu 1 Topics in Inference as Optimization Overview Exact Inference revisited The Energy Functional Optimizing the Energy Functional 2 Exact
MAP estimation via agreement on (hyper)trees: Message-passing and linear programming approaches
MAP estimation via agreement on (hyper)trees: Message-passing and linear programming approaches Martin Wainwright Tommi Jaakkola Alan Willsky martinw@eecs.berkeley.edu tommi@ai.mit.edu willsky@mit.edu
CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 17: Combinatorial Problems as Linear Programs III. Instructor: Shaddin Dughmi
CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 17: Combinatorial Problems as Linear Programs III Instructor: Shaddin Dughmi Announcements Today: Spanning Trees and Flows Flexibility awarded
9. Interpretations, Lifting, SOS and Moments
9-1 Interpretations, Lifting, SOS and Moments P. Parrilo and S. Lall, CDC 2003 2003.12.07.04 9. Interpretations, Lifting, SOS and Moments Polynomial nonnegativity Sum of squares (SOS) decomposition Eample
The Derivative. Leibniz notation: Prime notation: Limit Definition of the Derivative: (Used to directly compute derivative)
Topic 2: The Derivative 1 The Derivative The derivative of a function represents its instantaneous rate of change at any point along its domain. There are several ways which we can represent a derivative,
Random Vectors Part A
Random Vectors Part A Page 1 Outline Random Vectors Measurement of Dependence: Covariance Other Methods of Representing Dependence Set of Joint Distributions Copulas Common Random Vectors Functions of
Lecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
Angular Momentum, Electromagnetic Waves
Angular Momentum, Electromagnetic Waves Lecture33: Electromagnetic Theory Professor D. K. Ghosh, Physics Department, I.I.T., Bombay As before, we keep in view the four Maxwell s equations for all our discussions.
Lecture Note 18: Duality
MATH 5330: Computational Methods of Linear Algebra 1 The Dual Problems Lecture Note 18: Duality Xianyi Zeng Department of Mathematical Sciences, UTEP The concept duality, just like accuracy and stability,
Lecture 7: Semidefinite programming
CS 766/QIC 820 Theory of Quantum Information (Fall 2011) Lecture 7: Semidefinite programming This lecture is on semidefinite programming, which is a powerful technique from both an analytic and computational