Submodular Functions, Optimization, and Applications to Machine Learning

Transcription

1 Submodular Functions, Optimization, and Applications to Machine Learning Spring Quarter, Lecture 4 Prof. Jeff Bilmes University of Washington, Seattle Department of Electrical Engineering April 4th, 2018 Clockwise from top left:v Lásló Lovász Jack Edmonds Satoru Fujishige George Nemhauser Laurence Wolsey András Frank Lloyd Shapley H. Narayanan Robert Bixby William Cunningham William Tutte Richard Rado Alexander Schrijver Garrett Birkhoff Hassler Whitney Richard Dedekind Prof. Jeff Bilmes + f (A) + f (B) f (A [ B) = f (Ar ) + 2f (C ) + f (Br ) = f (Ar ) +f (C ) + f (Br ) f (A \ B) = f (A \ B) EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F1/55 (pg.1/174)

2 Logistics Cumulative Outstanding Reading Review Read chapter 1 from Fujishige s book. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F2/55 (pg.2/174)

3 Logistics Review Announcements, Assignments, and Reminders Homework 1 out, due Monday, 4/9/ :59pm electronically via our assignment dropbox ( If you have any questions about anything, please ask then via our discussion board ( Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F3/55 (pg.3/174)

4 Logistics Class Road Map - EE563 L1(3/26): Motivation, Applications, & Basic Definitions, L2(3/28): Machine Learning Apps (diversity, complexity, parameter, learning target, surrogate). L3(4/2): Info theory exs, more apps, definitions, graph/combinatorial examples L4(4/4): Graph and Combinatorial Examples, Matrix Rank, Examples and Properties, visualizations L5(4/9): L6(4/11): L7(4/16): L8(4/18): L9(4/23): L10(4/25): L11(4/30): L12(5/2): L13(5/7): L14(5/9): L15(5/14): L16(5/16): L17(5/21): L18(5/23): L (5/28): Memorial Day (holiday) L19(5/30): L21(6/4): Final Presentations maximization. Review Last day of instruction, June 1st. Finals Week: June 2-8, Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F4/55 (pg.4/174)

5 Logistics Submodular on Hypercube Vertices Review Test submodularity via values on verticies of hypercube. Example: with V = n =2,thisis easy: With V = n =3,abitharder w How many inequalities? Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F5/55 (pg.5/174)

6 Logistics Subadditive Definitions Review Definition (subadditive) Afunctionf :2 V! R is subadditive if for any A, B V,wehavethat: f(a)+f(b) f(a [ B) (4.21) This means that the whole is less than the sum of the parts. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F6/55 (pg.6/174)

7 Logistics Superadditive Definitions Review Definition (superadditive) Afunctionf :2 V! R is superadditive if for any A, B V,wehavethat: f(a)+f(b) apple f(a [ B) (4.21) This means that the whole is greater than the sum of the parts. In general, submodular and subadditive (and supermodular and superadditive) are different properties. Ex: Let 0 <k< V, andconsiderf :2 V! R + where: ( 1 if A applek f(a) = 0 else This function is subadditive but not submodular. (4.22) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F7/55 (pg.7/174)

8 Logistics Modular Definitions Review Definition (modular) Afunctionthatisbothsubmodularandsupermodulariscalledmodular If f is a modular function, than for any A, B V,wehave f(a)+f(b) =f(a \ B)+f(A [ B) (4.21) In modular functions, elements do not interact (or cooperate, or compete, or influence each other), and have value based only on singleton values. Proposition If f is modular, it may be written as f(a) =f(;)+ X f({a}) a2a f(;) = c + X f 0 (a) (4.22) a2a which has only V +1parameters. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F8/55 (pg.8/174)

9 Logistics Complement function Review Given a function f :2 V! R, wecanfindacomplementfunction f :2 V! R as f(a) =f(v \ A) for any A. Proposition f is submodular iff f is submodular. Proof. follows from f(a)+ f(b) f(a [ B)+ f(a \ B) (4.26) f(v \ A)+f(V \ B) f(v \ (A [ B)) + f(v \ (A \ B)) (4.27) which is true because V \ (A [ B) =(V \ A) \ (V \ B) and V \ (A \ B) =(V \ A) [ (V \ B) (De Morgan s laws for sets). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F9/55 (pg.9/174)

10 Logistics Other graph functions that are submodular/supermodular These come from Narayanan s book Let G be an undirected graph. Let V (X) be the vertices adjacent to some edge in X E(G), then V (X) (the vertex function) is submodular. Let E(S) be the edges with both vertices in S V (G). Then E(S) (the interior edge function) is supermodular. Let I(S) be the edges with at least one vertex in S V (G). Then I(S) (the incidence function) is submodular. Recall (S), isthesetsizeofedgeswithexactlyonevertexin S V (G) is submodular (cut size function). Thus, we have I(S) =E(S) [ (S) and E(S) \ (S) =;, andthusthat I(S) = E(S) + (S). So we can get a submodular function by summing a submodular and a supermodular function. If you had to guess, is this always the case? Consider f(a) = + (A) + (V \ A). Guess,submodular, supermodular, modular, or neither? Exercise: determine which one and prove it. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F10/55 (pg.10/174) Review

11 Logistics Number of connected components in a graph via edges Recall, f :2 V! R is submodular, then so is f :2 V! R defined as f(s) =f(v \ S). Hence, if g :2 V! R is supermodular, thensoisḡ :2 V! R defined as ḡ(s) =g(v \ S). Given a graph G =(V,E), foreacha E(G), letc(a) denote the number of connected components of the (spanning) subgraph (V (G),A), withc :2 E! R +. c(a) is monotone non-increasing, c(a + a) c(a) apple 0. ECO ).to Then c(a) is supermodular, i.e., c(a + a) c(a) apple c(b + a) c(b) (4.40) with A B E \{a}. Intuition: an edge is more (no less) able to bridge separate components (and reduce the number of conected components) when edge is added in a smaller context than when added in a larger context. c(a) =c(e \ A) is number of connected components in G when we remove A; supermodularmonotonenon-decreasingbutnotnormalized. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F11/55 (pg.11/174) Review

12 Graph Strength So c(a) =c(e \ A), thenumberofconnectedcomponentsing when we remove A, issupermodular. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F12/55 (pg.12/174)

13 Graph Strength So c(a) =c(e \ A), thenumberofconnectedcomponentsing when we remove A, issupermodular. Maximizing c(a) would be a goal for a network attacker many connected components means that many points in the network have lost connectivity to many other points (unprotected network). oe ± ii Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F12/55 (pg.13/174)

14 Graph Strength So c(a) =c(e \ A), thenumberofconnectedcomponentsing when we remove A, issupermodular. Maximizing c(a) would be a goal for a network attacker many connected components means that many points in the network have lost connectivity to many other points (unprotected network). If we can remove a small set A and shatter the graph into many connected components, then the graph is weak. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F12/55 (pg.14/174)

15 Graph Strength So c(a) =c(e \ A), thenumberofconnectedcomponentsing when we remove A, issupermodular. Maximizing c(a) would be a goal for a network attacker many connected components means that many points in the network have lost connectivity to many other points (unprotected network). If we can remove a small set A and shatter the graph into many connected components, then the graph is weak. An attacker wishes to choose a small number of edges (since it is cheap) to shatter the graph into as many components as possible. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F12/55 (pg.15/174)

16 Graph Strength So c(a) =c(e \ A), thenumberofconnectedcomponentsing when we remove A, issupermodular. Maximizing c(a) would be a goal for a network attacker many connected components means that many points in the network have lost connectivity to many other points (unprotected network). If we can remove a small set A and shatter the graph into many connected components, then the graph is weak. An attacker wishes to choose a small number of edges (since it is cheap) to shatter the graph into as many components as possible. Let G =(V,E,w) with w : E! R+ be a weighted graph with non-negative weights. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F12/55 (pg.16/174)

17 Graph Strength So c(a) =c(e \ A), thenumberofconnectedcomponentsing when we remove A, issupermodular. Maximizing c(a) would be a goal for a network attacker many connected components means that many points in the network have lost connectivity to many other points (unprotected network). If we can remove a small set A and shatter the graph into many connected components, then the graph is weak. An attacker wishes to choose a small number of edges (since it is cheap) to shatter the graph into as many components as possible. Let G =(V,E,w) with w : E! R+ be a weighted graph with non-negative weights. For (u, v) =e 2 E, letw(e) be a measure of the strength of the connection between vertices u and v (strength meaning the difficulty of cutting the edge e). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F12/55 (pg.17/174)

18 Graph Strength Then w(a) for A E is a modular function w(a) = X e2a w e (4.1) so that w(e(g[s])) is the internal strength of the vertex set S. Notation: S is a set of nodes, G[S] is the vertex-induced subgraph of G induced by vertices S, E(G[S]) are the edges contained within this induced subgraph, and w(e(g[s])) is the weight of these edges. Eatin ' Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F13/55 (pg.18/174)

19 Graph Strength Then w(a) for A E is a modular function w(a) = X e2a w e (4.1) so that w(e(g[s])) is the internal strength of the vertex set S. Suppose removing A shatters G into a graph with c(a) > 1 components Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F13/55 (pg.19/174)

20 Graph Strength Then w(a) for A E is a modular function w(a) = X w e (4.1) e2a so that w(e(g[s])) is the internal strength of the vertex set S. Suppose removing A shatters G into a graph with c(a) > 1 components then w(a)/( c(a) 1) is like the effort per achieved/additional component for a network attacker. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F13/55 (pg.20/174)

21 Graph Strength Then w(a) for A E is a modular function w(a) = X e2a w e (4.1) so that w(e(g[s])) is the internal strength of the vertex set S. Suppose removing A shatters G into a graph with c(a) > 1 components then w(a)/( c(a) 1) is like the effort per achieved/additional component for a network attacker. Aformofgraphstrengthcanthenbedefinedasthefollowing: strength(g, w) = min A E(G): c(a)>1 w(a) c(a) 1 (4.2) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F13/55 (pg.21/174)

22 Graph Strength Then w(a) for A E is a modular function w(a) = X e2a w e (4.1) so that w(e(g[s])) is the internal strength of the vertex set S. Suppose removing A shatters G into a graph with c(a) > 1 components then w(a)/( c(a) 1) is like the effort per achieved/additional component for a network attacker. Aformofgraphstrengthcanthenbedefinedasthefollowing: strength(g, w) = min A E(G): c(a)>1 w(a) c(a) 1 (4.2) Graph strength is like the minimum effort per component. An attacker would use the argument of the min to choose which edges to attack. A network designer would maximize, over G and/or w, the graph strength, strength(g, w). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F13/55 (pg.22/174)

23 Graph Strength Then w(a) for A E is a modular function w(a) = X e2a w e (4.1) so that w(e(g[s])) is the internal strength of the vertex set S. Suppose removing A shatters G into a graph with c(a) > 1 components then w(a)/( c(a) 1) is like the effort per achieved/additional component for a network attacker. Aformofgraphstrengthcanthenbedefinedasthefollowing: strength(g, w) = min A E(G): c(a)>1 w(a) c(a) 1 (4.2) Graph strength is like the minimum effort per component. An attacker would use the argument of the min to choose which edges to attack. A network designer would maximize, over G and/or w, the graph strength, strength(g, w). Since submodularity, problems have strongly-poly-time solutions. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F13/55 (pg.23/174)

24 Submodularity, Quadratic Structures, and Cuts Lemma Let M 2 R n n be a symmetric matrix and m 2 R n be a vector. Then f :2 V "! R defined as c- 112 a Ix E { 0,13 " f(x) =m 1 X X M1 X (4.3) od is submodular iff the off-diagonal elements In of M are non-positive. Proof.... Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F14/55 (pg.24/174)

25 Submodularity, Quadratic Structures, and Cuts Lemma Let M 2 R n n be a symmetric matrix and m 2 R n be a vector. Then f :2 V! R defined as f(x) =m 1 X X M1 X (4.3) is submodular iff the off-diagonal elements of M are non-positive. Proof. Given a complete graph G =(V,E), recallthate(x) is the edge set with both vertices in X V (G), and that E(X) is supermodular. w... Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F14/55 (pg.25/174)

26 Submodularity, Quadratic Structures, and Cuts Lemma Let M 2 R n n be a symmetric matrix and m 2 R n be a vector. Then f :2 V! R defined as f(x) =m 1 X X M1 X (4.3) is submodular iff the off-diagonal elements of M are non-positive. Proof. Given a complete graph G =(V,E), recallthate(x) is the edge set with both vertices in X V (G), and that E(X) is supermodular. Non-negative modular weights w + : E! R +, w(e(x)) is also supermodular, so w(e(x)) is submodular. 9 - tzie m@... Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F14/55 (pg.26/174)

27 Submodularity, Quadratic Structures, and Cuts Lemma Let M 2 R n n be a symmetric matrix and m 2 R n be a vector. Then f :2 V! R defined as f(x) =m 1 X X M1 X (4.3) is submodular iff the off-diagonal elements of M are non-positive. Proof. Given a complete graph G =(V,E), recallthate(x) is the edge set with both vertices in X V (G), and that E(X) is supermodular. Non-negative modular weights w + : E! R +, w(e(x)) is also supermodular, so w(e(x)) is submodular. f is a modular function m 1 A = m(a) added to a weighted submodular function, hence f is submodular.... Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F14/55 (pg.27/174)

28 Submodularity, Quadratic Structures, and Cuts Proof of Lemma cont. Conversely, suppose f is submodular. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F15/55 (pg.28/174)

29 Submodularity, Quadratic Structures, and Cuts Proof of Lemma cont. Conversely, suppose f is submodular. Then 8u, v 2 V, f({u})+f({v}) f({u, v})+f(;) while f(;) =0. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F15/55 (pg.29/174)

30 Submodularity, Quadratic Structures, and Cuts Proof of Lemma cont. Conversely, suppose f is submodular. Then 8u, v 2 V, f({u})+f({v}) f({u, v})+f(;) while f(;) =0. This requires: - utv 0 apple f({u})+f({v}) f({u, v}) (4.4) = m(u)+ 1 2 M u,u + m(v)+ 1 2 M v,v (4.5) m(u)+m(v)+ 1 2 M u,u + M u,v M v,v (4.6) = M u,v (4.7) So that 8u, v 2 V, M u,v apple 0. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F15/55 (pg.30/174)

31 Set Cover and Maximum Coverage just Special cases of Submodular Optimization We are given a finite set U of m elements and a set of subsets U = {U 1,U 2,...,U n } of n subsets of U, sothatu i U and S i U i = U. ( cover. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F16/55 (pg.31/174)

32 Set Cover and Maximum Coverage just Special cases of Submodular Optimization We are given a finite set U of m elements and a set of subsets U = {U 1,U 2,...,U n } of n subsets of U, sothatu i U and S i U i = U. The goal of minimum set cover is to choose the smallest subset A [n], {1,...,n} such that S a2a U a = U. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F16/55 (pg.32/174)

33 Set Cover and Maximum Coverage just Special cases of Submodular Optimization We are given a finite set U of m elements and a set of subsets U = {U 1,U 2,...,U n } of n subsets of U, sothatu i U and S i U i = U. The goal of minimum set cover is to choose the smallest subset A [n], {1,...,n} such that S a2a U a = U. Maximum k cover: The goal in maximum coverage is, given an integer k apple n, selectk subsets, say {a 1,a 2,...,a k } with a i 2 [n] such that S k i=1 U a i is maximized. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F16/55 (pg.33/174)

34 Set Cover and Maximum Coverage just Special cases of Submodular Optimization We are given a finite set U of m elements and a set of subsets U = {U 1,U 2,...,U n } of n subsets of U, sothatu i U and S i U i = U. The goal of minimum set cover is to choose the smallest subset A [n], {1,...,n} such that S a2a U a = U. Maximum k cover: The goal in maximum coverage is, given an integer k apple n, selectk subsets, say {a 1,a 2,...,a k } with a i 2 [n] such that S k i=1 U a i is maximized. f :2 [n]! Z + where for A [n], f(a) = S a2a U a is the set cover function and is submodular. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F16/55 (pg.34/174)

35 Set Cover and Maximum Coverage just Special cases of Submodular Optimization We are given a finite set U of m elements and a set of subsets U = {U 1,U 2,...,U n } of n subsets of U, sothatu i U and S i U i = U. The goal of minimum set cover is to choose the smallest subset A [n], {1,...,n} such that S a2a U a = U. Maximum k cover: The goal in maximum coverage is, given an integer k apple n, selectk subsets, say {a 1,a 2,...,a k } with a i 2 [n] such that S k i=1 U a i is maximized. f :2 [n]! Z + where for A [n], f(a) = S a2a U a is the set cover function and is submodular. Weighted set cover: f(a) =w( S a2a U myoddn a) where w : U! R +. a (4*4) g. zu " W gsubmohla Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F16/55 (pg.35/174)

36 w :rv 1R ttvfv, wcnek w w to indented, : zv 1R )=aizyw von AEV WIA

37 Set Cover and Maximum Coverage just Special cases of Submodular Optimization We are given a finite set U of m elements and a set of subsets U = {U 1,U 2,...,U n } of n subsets of U, sothatu i U and S i U i = U. The goal of minimum set cover is to choose the smallest subset A [n], {1,...,n} such that S a2a U a = U. Maximum k cover: The goal in maximum coverage is, given an integer k apple n, selectk subsets, say {a 1,a 2,...,a k } with a i 2 [n] such that S k i=1 U a i is maximized. f :2 [n]! Z + where for A [n], f(a) = S a2a U a is the set cover function and is submodular. Weighted set cover: f(a) =w( S a2a U a) where w : U! R +. Both Set cover and maximum coverage are well known to be NP-hard, but have a fast greedy approximation algorithm, and hence are instances of submodular optimization. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F16/55 (pg.36/174)

38 Vertex and Edge Covers Also instances of submodular optimization Definition (vertex cover) A vertex cover (a vertex-based cover of edges ) in graph G =(V,E) is a set S V (G) of vertices such that every edge in G is incident to at least one vertex in S. Let I(S) be the number of edges incident to vertex set S. Thenwe wish to find the smallest set S V subject to I(S) = E. Definition (edge cover) - plastic A edge cover (an edge-based cover of vertices ) in graph G =(V,E) is a set F E(G) of edges such that every vertex in G is incident to at least one edge in F. Let V (F ) be the number of vertices incident to edge set F.Thenwe wish to find the smallest set F E subject to V (F )= V. can, Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F17/55 (pg.37/174)

39 Graph Cut Problems Also submodular optimization Minimum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F18/55 (pg.38/174)

40 Graph Cut Problems Also submodular optimization Minimum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Maximum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F18/55 (pg.39/174)

41 Graph Cut Problems Also submodular optimization Minimum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Maximum cut: Given a graph G =(V,E), findasetofverticess V that minimize WEE the cut (set of edges) between S and V \ S. Let :2 V! R + be the cut function, namely for any given set of nodes X V, (X) measures the number of edges between nodes X and V \ X i.e., (x) =E(X, V \ X). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F18/55 (pg.40/174)

42 Graph Cut Problems Also submodular optimization Minimum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Maximum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Let :2 V! R + be the cut function, namely for any given set of nodes X V, (X) measures the number of edges between nodes X and V \ X i.e., (x) =E(X, V \ X). Weighted versions, where rather than count, we sum the (non-negative) weights of the edges of a cut, f(x) =w( (X)). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F18/55 (pg.41/174)

43 Graph Cut Problems Also submodular optimization Minimum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Maximum cut: Given a graph G =(V,E), findasetofverticess V that minimize the cut (set of edges) between S and V \ S. Let :2 V! R + be the cut function, namely for any given set of nodes X V, (X) measures the number of edges between nodes X and V \ X i.e., (x) =E(X, V \ X). Weighted versions, where rather than count, we sum the (non-negative) weights of the edges of a cut, f(x) =w( (X)). Hence, Minimum cut and Maximum cut are also special cases of submodular optimization. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F18/55 (pg.42/174)

44 Matrix Rank functions Let V,with V = m be an index set of a set of vectors in R n for some n (unrelated to m). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F19/55 (pg.43/174)

45 mg Matrix Rank functions Let V,with V = m be an index set of a set of vectors in R n for some " n (unrelated to m). FVEV, XVEIR For a given set {v, v 1,v 2,...,v k },itmightormightnotbepossibleto find ( i ) i such that: tvtlr " x v = kx i x vi (4.8) i=1 If not, then x v is linearly independent of x v1,...,x vk. P( x. g) =p ( x ). ) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F19/55 (pg.44/174)

46 Matrix Rank functions Let V,with V = m be an index set of a set of vectors in R n for some n (unrelated to m). For a given set {v, v 1,v 2,...,v k },itmightormightnotbepossibleto find ( i ) i such that: x v = kx i x vi (4.8) i=1 If not, then x v is linearly independent of x v1,...,x vk. Let r(s) for S V be the rank of the set of vectors S. Thenr( ) is a submodular function, and in fact is called a matric matroid rank function. nyytt Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F19/55 (pg.45/174)

47 Example: Rank function of a matrix Given n m matrix X =(x 1,x 2,...,x m ) with x i 2 R n for all i. There are m length-n column vectors {x i } i Skip matrix rank example Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F20/55 (pg.46/174)

48 Example: Rank function of a matrix Given n m matrix X =(x 1,x 2,...,x m ) with x i 2 R n for all i. There are m length-n column vectors {x i } i Let V = {1, 2,...,m} be the set of column vector indices. Skip matrix rank example Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F20/55 (pg.47/174)

49 Example: Rank function of a matrix Given n m matrix X =(x 1,x 2,...,x m ) with x i 2 R n for all i. There are m length-n column vectors {x i } i Let V = {1, 2,...,m} be the set of column vector indices. For any A V,letr(A) be the rank of the column vectors indexed by A. Skip matrix rank example Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F20/55 (pg.48/174)

50 Example: Rank function of a matrix Given n m matrix X =(x 1,x 2,...,x m ) with x i 2 R n for all i. There are m length-n column vectors {x i } i Let V = {1, 2,...,m} be the set of column vector indices. For any A V,letr(A) be the rank of the column vectors indexed by A. r(a) is the dimensionality of the vector space spanned by the set of vectors {x a } a2a. Skip matrix rank example Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F20/55 (pg.49/174)

51 Example: Rank function of a matrix Given n m matrix X =(x 1,x 2,...,x m ) with x i 2 R n for all i. There are m length-n column vectors {x i } i Let V = {1, 2,...,m} be the set of column vector indices. For any A V,letr(A) be the rank of the column vectors indexed by A. r(a) is the dimensionality of the vector space spanned by the set of vectors {x a } a2a. Thus, r(v ) is the rank of the matrix X. Skip matrix rank example Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F20/55 (pg.50/174)

52 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 nsy m=8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.51/174)

53 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.52/174)

54 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.53/174)

55 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.54/174)

56 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.55/174)

57 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.56/174)

58 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.57/174)

59 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.58/174)

60 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.59/174)

61 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.60/174)

62 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.61/174)

63 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.62/174)

64 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.63/174)

65 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.64/174)

66 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.65/174)

67 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1<r(C) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.66/174)

68 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1 <r(c) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.67/174)

69 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1 <r(c) =2. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.68/174)

70 Example: Rank function of a matrix Consider the following 4 8 matrix, so V = {1, 2, 3, 4, 5, 6, 7, 8} = x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Let A = {1, 2, 3}, B = {3, 4, 5}, C = {6, 7}, A r = {1}, B r = {5}. Then r(a) =3, r(b) =3, r(c) =2. r(a [ C) =3, r(b [ C) =3. r(a [ A r )=3, r(b [ B r )=3, r(a [ B r )=4, r(b [ A r )=4. r(a [ B) =4, r(a \ B) =1 <r(c) =2. = 6= r(a)+r(b) =r(a [ B)+r(C) >r(a [ B)+r(A \ B) =5 Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F21/55 (pg.69/174)

71 Rank function of a matrix Let A, B V be two subsets of column indices. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F22/55 (pg.70/174)

72 Rank function of a matrix Let A, B V be two subsets of column indices. The rank of the two sets unioned together A [ B is no more than the sum of the two individual ranks. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F22/55 (pg.71/174)

73 Rank function of a matrix Let A, B V be two subsets of column indices. The rank of the two sets unioned together A [ B is no more than the sum of the two individual ranks. In a Venn diagram, let area correspond to dimensions spanned by vectors indexed by a set. Hence, r(a) can be viewed as an area. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F22/55 (pg.72/174)

74 Rank function of a matrix Let A, B V be two subsets of column indices. The rank of the two sets unioned together A [ B is no more than the sum of the two individual ranks. In a Venn diagram, let area correspond to dimensions spanned by vectors indexed by a set. Hence, r(a) can be viewed as an area. r(a) + r(b) r(a [ B) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F22/55 (pg.73/174)

75 . Rank function of a matrix Let A, B V be two subsets of column indices. The rank of the two sets unioned together A [ B is no more than the sum of the two individual ranks. In a Venn diagram, let area correspond to dimensions spanned by vectors indexed by a set. Hence, r(a) can be viewed as an area. r(a) + r(b) r(a [ B) If some of the dimensions spanned by A overlap some of the dimensions spanned by B (i.e., if 9 common span), then that area is counted twice in r(a)+r(b), sotheinequalitywillbestrict. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F22/55 (pg.74/174)

76 Rank function of a matrix Let A, B V be two subsets of column indices. The rank of the two sets unioned together A [ B is no more than the sum of the two individual ranks. In a Venn diagram, let area correspond to dimensions spanned by vectors indexed by a set. Hence, r(a) can be viewed as an area. r(a) + r(b) r(a [ B) 0 If some of the dimensions spanned by A overlap some of the dimensions spanned by B (i.e., if 9 common span), then that area is counted twice in r(a)+r(b), so the inequality will be strict. Any function where the above inequality is true for all A, B V is called subadditive. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F22/55 (pg.75/174)

77 Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.76/174)

78 Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Let C index vectors spanning all dimensions common to A and B. We call C the common span and call A \ B the common index. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.77/174)

79 . Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Let C index vectors spanning all dimensions common to A and B. We call C the common span and call A \ B the common index. f indices Let A r index vectors spanning dimensions spanned by A but not B. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.78/174)

80 Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Let C index vectors spanning all dimensions common to A and B. We call C the common span and call A \ B the common index. Let A r index vectors spanning dimensions spanned by A but not B. Let B r index vectors spanning dimensions spanned by B but not A. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.79/174)

81 Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Let C index vectors spanning all dimensions common to A and B. We call C the common span and call A \ B the common index. Let A r index vectors spanning dimensions spanned by A but not B. Let B r index vectors spanning dimensions spanned by B but not A. Then, r(a) =r(c)+r(a r ) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.80/174)

82 Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Let C index vectors spanning all dimensions common to A and B. We call C the common span and call A \ B the common index. Let A r index vectors spanning dimensions spanned by A but not B. Let B r index vectors spanning dimensions spanned by B but not A. Then, r(a) =r(c)+r(a r ) Similarly, r(b) =r(c)+r(b r ). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.81/174)

83 Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Let C index vectors spanning all dimensions common to A and B. We call C the common span and call A \ B the common index. Let A r index vectors spanning dimensions spanned by A but not B. Let B r index vectors spanning dimensions spanned by B but not A. Then, r(a) =r(c)+r(a r ) Similarly, r(b) =r(c)+r(b r ). Then r(a) +r(b) counts the dimensions spanned by C twice, i.e., r(a)+r(b) =r(a r )+2r(C)+r(B r ). (4.9) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.82/174)

84 Rank functions of a matrix Vectors A and B have a (possibly empty) common span and two (possibly empty) non-common residual spans. Let C index vectors spanning all dimensions common to A and B. We call C the common span and call A \ B the common index. Let A r index vectors spanning dimensions spanned by A but not B. Let B r index vectors spanning dimensions spanned by B but not A. Then, r(a) =r(c)+r(a r ) Similarly, r(b) =r(c)+r(b r ). Then r(a) +r(b) counts the dimensions spanned by C twice, i.e., r(a)+r(b) =r(a r )+2r(C)+r(B r ). (4.9) But r(a [ B) counts the dimensions spanned by C only once. r(a [ B) =r(a r )+r(c)+r(b r ) (4.10) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F23/55 (pg.83/174)

85 Rank functions of a matrix Then r(a) +r(b) counts the dimensions spanned by C twice, i.e., r(a) + r(b) = r(a r ) + 2r(C) + r(b r ) 1 ' ' ' Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F24/55 (pg.84/174)

86 Rank functions of a matrix Then r(a) +r(b) counts the dimensions spanned by C twice, i.e., r(a) + r(b) = r(a r ) + 2r(C) + r(b r ) But r(a [ B) counts the dimensions spanned by C only once. r(a [ B) = r(a r ) +r(c) + r(b r ) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F24/55 (pg.85/174)

87 Rank functions of a matrix Then r(a) +r(b) counts the dimensions spanned by C twice, i.e., r(a) + r(b) = r(a r ) + 2r(C) + r(b r ) But r(a [ B) counts the dimensions spanned by C only once. r(a [ B) = r(a r ) +r(c) + r(b r ) Thus, we have subadditivity: r(a)+r(b) r(a [ B). Canweadd more to the r.h.s. and still have an inequality? Yes. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F24/55 (pg.86/174)

88 Rank function of a matrix Note, r(a \ B) apple r(c). Why?VectorsindexedbyA \ B (i.e., the common index set) span no more than the dimensions commonly spanned by A and B (namely, those spanned by the professed C). r(a \ B) r(c) In short: Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F25/55 (pg.87/174)

89 Rank function of a matrix Note, r(a \ B) apple r(c). Why?VectorsindexedbyA \ B (i.e., the common index set) span no more than the dimensions commonly spanned by A and B (namely, those spanned by the professed C). r(a \ B) r(c) In short: Common span (blue) is more (no less) than span of common index (magenta). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F25/55 (pg.88/174)

90 Rank function of a matrix Note, r(a \ B) apple r(c). Why?VectorsindexedbyA \ B (i.e., the common index set) span no more than the dimensions commonly spanned by A and B (namely, those spanned by the professed C). r(a \ B) r(c) In short: Common span (blue) is more (no less) than span of common index (magenta). More generally, common information (blue) is more (no less) than information within common index (magenta). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F25/55 (pg.89/174)

91 . :. :. The Venn and Art of Submodularity A - + r(a) + r(b) r(a [ B) r(a \ B) {z } {z } {z } = r(a r ) + 2r(C) + r(b r ) = r(a r ) +r(c) + r(b r ) = r(a \ B) :.. r(altr(b)=r(aub7+r(= Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F26/55 (pg.90/174)

92 Polymatroid rank function Let S be a set of subspaces of a linear space (i.e., each s 2 S is a subspace of dimension 1). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F27/55 (pg.91/174)

93 Polymatroid rank function Let S be a set of subspaces of a linear space (i.e., each s 2 S is a subspace of dimension 1). For each X S, letf(x) denote the dimensionality of the linear subspace spanned by the subspaces in X. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F27/55 (pg.92/174)

94 Polymatroid rank function Let S be a set of subspaces of a linear space (i.e., each s 2 S is a subspace of dimension 1). For each X S, letf(x) denote the dimensionality of the linear subspace spanned by the subspaces in X. We can think of S as a set of sets of vectors from the matrix rank example, and for each s 2 S, letx s being a set of vector indices. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F27/55 (pg.93/174)

95 Polymatroid rank function Let S be a set of subspaces of a linear space (i.e., each s 2 S is a subspace of dimension 1). For each X S, letf(x) denote the dimensionality of the linear subspace spanned by the subspaces in X. We can think of S as a set of sets of vectors from the matrix rank example, and for each s 2 S, letx s being a set of vector indices. Then, defining f :2 S! R + as follows, f(x) =r([ s2s X s ) (4.11) we have that f is submodular, and is known to be a polymatroid rank function. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F27/55 (pg.94/174)

96 Polymatroid rank function Let S be a set of subspaces of a linear space (i.e., each s 2 S is a subspace of dimension 1). For each X S, letf(x) denote the dimensionality of the linear subspace spanned by the subspaces in X. We can think of S as a set of sets of vectors from the matrix rank example, and for each s 2 S, letx s being a set of vector indices. Then, defining f :2 S! R + as follows, f(x) =r([ s2s X s ) (4.11) we have that f is submodular, and is known to be a polymatroid rank function. In general (as we will see) polymatroid rank functions are submodular, normalized f(;) =0,andmonotonenon-decreasing(f(A) apple f(b) whenever A B). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F27/55 (pg.95/174)

97 Polymatroid rank function Let S be a set of subspaces of a linear space (i.e., each s 2 S is a subspace of dimension 1). For each X S, letf(x) denote the dimensionality of the linear subspace spanned by the subspaces in X. We can think of S as a set of sets of vectors from the matrix rank example, and for each s 2 S, letx s being a set of vector indices. Then, defining f :2 S! R + as follows, f(x) =r([ s2s X s ) (4.11) we have that f is submodular, and is known to be a polymatroid rank function. In general (as we will see) polymatroid rank functions are submodular, normalized f(;) =0,andmonotonenon-decreasing(f(A) apple f(b) whenever A B). We use the term non-decreasing rather than increasing, the latter of which is strict (also so that a constant function isn t increasing ). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F27/55 (pg.96/174)

98 Spanning trees Let E be a set of edges of some graph G =(V,E), andletr(s) for S E be the maximum size (in terms of number of edges) spanning forest in the vertex-induced graph, induced by vertices incident to edges S. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F28/55 (pg.97/174)

99 Spanning trees Let E be a set of edges of some graph G =(V,E), andletr(s) for S E be the maximum size (in terms of number of edges) spanning forest in the vertex-induced graph, induced by vertices incident to edges S. Example: Given G =(V,E), V = {1, 2, 3, 4, 5, 6, 7, 8}, E = {1, 2,...,12}. S = {1, 2, 3, 4, 5, 8, 9} E. Twospanningtrees have the same edge count (the rank of S) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F28/55 (pg.98/174)

100 Spanning trees to ( Xl from win slide. plate Let E be a set of edges of some graph G =(V,E), andletr(s) for S E be the maximum size (in terms of number of edges) spanning forest in the vertex-induced graph, induced by vertices incident to edges S. Example: Given G =(V,E), V = {1, 2, 3, 4, 5, 6, 7, 8}, E = {1, 2,...,12}. S = {1, 2, 3, 4, 5, 8, 9} E. Twospanningtrees have the same edge count (the rank of S) Then r(s) is submodular, and is another matrix rank function corresponding to the incidence matrix of the graph. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F28/55 (pg.99/174)

101 Summing Submodular Functions Given E, letf 1,f 2 :2 E! R be two submodular functions. Then is submodular. f :2 E! R with f(a) =f 1 (A)+f 2 (A) (4.16) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F30/55 (pg.100/174)

102 Summing Submodular Functions Given E, letf 1,f 2 :2 E! R be two submodular functions. Then f :2 E! R with f(a) =f 1 (A)+f 2 (A) (4.16) is submodular.this follows easily since f(a)+f(b) =f 1 (A)+f 2 (A)+f 1 (B)+f 2 (B) (4.17) f 1 (A [ B)+f 2 (A [ B)+f 1 (A \ B)+f 2 (A \ B) (4.18) = f(a [ B)+f(A \ B). (4.19) I.e., it holds for each component of f in each term in the inequality. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F30/55 (pg.101/174)

103 Summing Submodular Functions Given E, letf 1,f 2 :2 E! R be two submodular functions. Then f :2 E! R with f(a) =f 1 (A)+f 2 (A) (4.16) is submodular.this follows easily since f(a)+f(b) =f 1 (A)+f 2 (A)+f 1 (B)+f 2 (B) (4.17) f 1 (A [ B)+f 2 (A [ B)+f 1 (A \ B)+f 2 (A \ B) (4.18) = f(a [ B)+f(A \ B). (4.19) I.e., it holds for each component of f in each term in the inequality. In fact, any conic combination (i.e., non-negative linear combination) of submodular functions is submodular, as in f(a) = 1 f 1 (A)+ 2 f 2 (A) for 1, 2 0. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F30/55 (pg.102/174)

104 Summing Submodular and Modular Functions Given E, letf 1,m:2 E! R be a submodular and a modular function. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F31/55 (pg.103/174)

105 Summing Submodular and Modular Functions Given E, letf 1,m:2 E! R be a submodular and a modular function. Then f :2 E! R with f(a) =f 1 (A) m(a) (4.20) is submodular (as is f(a) =f 1 (A)+m(A)). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F31/55 (pg.104/174)

106 Summing Submodular and Modular Functions Given E, letf 1,m:2 E! R be a submodular and a modular function. Then f :2 E! R with f(a) =f 1 (A) m(a) (4.20) is submodular (as is f(a) =f 1 (A)+m(A)). This follows easily since f(a)+f(b) =f 1 (A) m(a)+f 1 (B) m(b) (4.21) f 1 (A [ B) m(a [ B)+f 1 (A \ B) m(a \ B) (4.22) = f(a [ B)+f(A \ B). (4.23) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F31/55 (pg.105/174)

107 Summing Submodular and Modular Functions Given E, letf 1,m:2 E! R be a submodular and a modular function. Then f :2 E! R with f(a) =f 1 (A) m(a) (4.20) is submodular (as is f(a) =f 1 (A)+m(A)). This follows easily since f(a)+f(b) =f 1 (A) m(a)+f 1 (B) m(b) (4.21) f 1 (A [ B) m(a [ B)+f 1 (A \ B) m(a \ B) (4.22) = f(a [ B)+f(A \ B). (4.23) That is, the modular component with m(a) +m(b) =m(a [ B) +m(a \ B) never destroys the inequality. Note of course that if m is modular than so is m. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F31/55 (pg.106/174)

108 Restricting Submodular functions Given E, letf :2 E! R be a submodular functions. And let S E be an arbitrary fixed set. Then is submodular. f 0 :2 E! R with f 0 (A), f(a \ S) (4.24) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F32/55 (pg.107/174)

109 Restricting Submodular functions Given E, letf :2 E! R be a submodular functions. And let S E be an arbitrary fixed set. Then is submodular. Proof. f 0 :2 E! R with f 0 (A), f(a \ S) (4.24) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F32/55 (pg.108/174)

110 Restricting Submodular functions Given E, letf :2 E! R be a submodular functions. And let S E be an arbitrary fixed set. Then is submodular. Proof. Given A B E \ v, consider f 0 :2 E! R with f 0 (A), f(a \ S) (4.24) f((a + v) \ S) f(a \ S) f((b + v) \ S) f(b \ S) (4.25) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F32/55 (pg.109/174)

111 Restricting Submodular functions Given E, letf :2 E! R be a submodular functions. And let S E be an arbitrary fixed set. Then is submodular. Proof. Given A B E \ v, consider f 0 :2 E! R with f 0 (A), f(a \ S) (4.24) f((a + v) \ S) f(a \ S) f((b + v) \ S) f(b \ S) (4.25) If v/2 S, thenbothdifferencesoneachsizearezero. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F32/55 (pg.110/174)

112 Restricting Submodular functions Given E, letf :2 E! R be a submodular functions. And let S E be an arbitrary fixed set. Then is submodular. Proof. Given A B E \ v, consider f 0 :2 E! R with f 0 (A), f(a \ S) (4.24) f((a + v) \ S) f(a \ S) f((b + v) \ S) f(b \ S) (4.25) If v/2 S, thenbothdifferencesoneachsizearezero.if v 2 S, thenwecan consider this f(a 0 + v) f(a 0 ) f(b 0 + v) f(b 0 ) (4.26) with A 0 = A \ S and B 0 = B \ S. Since A 0 B 0,thisholdsdueto submodularity of f. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F32/55 (pg.111/174)

113 ' Summing Restricted Submodular Functions Given V,letf 1,f 2 :2 V! R be two submodular functions and let S 1,S 2 be two arbitrary fixed sets. Then Si Sr IV. \, f :2 V! R with f(a) =f 1 (A \ S 1 )+f 2 (A \ S 2 ) (4.27) ± is submodular. This follows easily from the preceding two results. 25 ' 112 fz : 25 R Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F33/55 (pg.112/174)

114 Summing Restricted Submodular Functions Given V,letf 1,f 2 :2 V! R be two submodular functions and let S 1,S 2 be two arbitrary fixed sets. Then f :2 V! R with f(a) =f 1 (A \ S 1 )+f 2 (A \ S 2 ) (4.27) is submodular. This follows easily from the preceding two results. Given V,letC = {C 1,C 2,...,C k } be a set of subsets of V,andforeach C 2C,letf C :2 V! R be a submodular function. Then f :2 V! R with f(a) = X C2C f C (A \ C) (4.28) is submodular. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F33/55 (pg.113/174)

115 Summing Restricted Submodular Functions Given V,letf 1,f 2 :2 V! R be two submodular functions and let S 1,S 2 be two arbitrary fixed sets. Then f :2 V! R with f(a) =f 1 (A \ S 1 )+f 2 (A \ S 2 ) (4.27) is submodular. This follows easily from the preceding two results. Given V,letC = {C 1,C 2,...,C k } be a set of subsets of V,andforeach C 2C,letf C :2 V! R be a submodular function. Then f :2 V! R with f(a) = X C2C f C (A \ C) (4.28) is submodular. This property is critical for image processing and graphical models. For example, let C be all pairs of the form {{u, v} : u, v 2 V }, or let it be all pairs corresponding to the edges of some undirected graphical model. We plan to revisit this topic later in the term. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F33/55 (pg.114/174)

116 Max - normalized Given V,letc 2 R V + be a given fixed vector. Then f :2 V! R +,where is submodular and normalized (we take f(;) =0). Proof. Consider max j2a c j + max j2b c j which follows since we have that and f(a) = max j2a c j (4.29) =. max c j + max c j (4.30) j2a[b j2a\b max(max j2a c j, max j2b c j) = max j2a[b c j (4.31) min(max j2a c j, max j2b c j) max c j (4.32) j2a\b Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F34/55 (pg.115/174)

117 Max Given V,letc2R V be a given fixed vector (not necessarily non-negative). Then f :2 V! R, where f(a) = max c j (4.33) j2a is submodular, where we take f(;) apple min j c j (so the function is not normalized). Proof. The proof is identical to the normalized case. doent need to by na Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F35/55 (pg.116/174)

118 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.117/174)

119 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. S = {1,...,s} is a set of sites (e.g., cities, clients) needing service. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.118/174)

120 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. S = {1,...,s} is a set of sites (e.g., cities, clients) needing service. Let c ij be the benefit (e.g., 1/c ij is the cost) of servicing site i with facility location j. vote - Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.119/174)

121 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. S = {1,...,s} is a set of sites (e.g., cities, clients) needing service. Let c ij be the benefit (e.g., 1/c ij is the cost) of servicing site i with facility location j. Let m j be the benefit (e.g., either 1/m j is the cost or m j is the cost) to build a plant at location j. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.120/174)

122 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. S = {1,...,s} is a set of sites (e.g., cities, clients) needing service. Let c ij be the benefit (e.g., 1/c ij is the cost) of servicing site i with facility location j. Let m j be the benefit (e.g., either 1/m j is the cost or m j is the cost) to build a plant at location j. Each site should be serviced by only one plant but no less than one. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.121/174)

123 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. S = {1,...,s} is a set of sites (e.g., cities, clients) needing service. Let c ij be the benefit (e.g., 1/c ij is the cost) of servicing site i with facility location j. Let m j be the benefit (e.g., either 1/m j is the cost or m j is the cost) to build a plant at location j. Each site should be serviced by only one plant but no less than one. Define f(a) as the delivery benefit plus construction benefit when the locations A F are to be constructed. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.122/174)

124 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. S = {1,...,s} is a set of sites (e.g., cities, clients) needing service. Let c ij be the benefit (e.g., 1/c ij is the cost) of servicing site i with facility location j. Let m j be the benefit (e.g., either 1/m j is the cost or m j is the cost) to build a plant at location j. Each site should be serviced by only one plant but no less than one. Define f(a) as the delivery benefit plus construction benefit when the locations A F are to be constructed. We can define the (uncapacitated) facility location function f(a) = X j2a m j + X i2s max j2a c ij. (4.34) Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.123/174)

125 Facility/Plant Location (uncapacitated) w. plant benefits Let F = {1,...,f} be a set of possible factory/plant locations for facilities to be built. S = {1,...,s} is a set of sites (e.g., cities, clients) needing service. Let c ij be the benefit (e.g., 1/c ij is the cost) of servicing site i with facility location j. Let m j be the benefit (e.g., either 1/m j is the cost or m j is the cost) to build a plant at location j. Each site should be serviced by only one plant but no less than one. Define f(a) as the delivery benefit plus construction benefit when the locations A F are to be constructed. We can define the (uncapacitated) facility location function f(a) = X j2a m j + X i2s max j2a c ij. (4.34) Goal is to find a set A that maximizes f(a) (the benefit) placing a bound on the number of plants A (e.g., A applek). Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F36/55 (pg.124/174)

126 Facility/Plant Location (uncapacitated) Core problem in operations research, early motivation for submodularity. Goal: as efficiently as possible, place facilities (factories) at certain locations to satisfy sites (at all locations) having various demands. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F37/55 (pg.125/174)

127 Facility/Plant Location (uncapacitated) Core problem in operations research, early motivation for submodularity. Goal: as efficiently as possible, place facilities (factories) at certain locations to satisfy sites (at all locations) having various demands. Prof. Jeff Bilmes EE563/Spring 2018/Submodularity - Lecture 4 - April 4th, 2018 F37/55 (pg.126/174)