Optimal Transport And WGAN

Transcription

1 Optimal Transport And WGAN Yiping Lu 1 1 School of Mathmetical Science Peking University Data Science Seminar, PKU, 2017 Yiping Lu (pku) Geometry Of WGAN 1/34

2 Outline 1 Introduction To Optimal Transport. 2 Minkowski Type Problems Picewise Linear Function And Power Diagram 3 WGAN Yiping Lu (pku) Geometry Of WGAN 2/34

3 Introduction To Optimal Transport. Outline 1 Introduction To Optimal Transport. 2 Minkowski Type Problems Picewise Linear Function And Power Diagram 3 WGAN Yiping Lu (pku) Geometry Of WGAN 3/34

4 Introduction To Optimal Transport. Optimal Transport Monge Objective: Calculate a transport map T # µ = υ which minimize the transport cost c(t ) = c(x, T (x))dµ(x) Kantorovich Objective: Calculate a transport plane minimize the transport cost c(π) = c(x, y)π(x, y) Yiping Lu (pku) Geometry Of WGAN 4/34

5 Introduction To Optimal Transport. Optimal Transport: Linear Programming View The optimal transport is a convex problem, which can be formulated as min C, F s.t. F i,j = q j i F i,j = q i is a special case of the linear programming: minc T x s.t.ax = b, x 0 j Yiping Lu (pku) Geometry Of WGAN 5/34

6 Introduction To Optimal Transport. Linear Programming Consider the dual problem of the linear programming problem. Dual: Primal: minb T y minc T x s.t.a T y c s.t.ax = b, x 0 and we have the relation: Yiping Lu (pku) Geometry Of WGAN 6/34

7 Introduction To Optimal Transport. Kantorovich Dual First, let us express the constraint γ Π(µ, υ) in the following way. φdµ + ψdυ (φ(x) + ψ(y))dγ sup φ,ψ X Y X Y X Y so that the primal problem can be expressed by min + sup φdµ + ψdυ (φ(x) + ψ(y))dγ γ 0 φ,ψ sup φ,ψ X Y X Y X Y then consider interchanging sup and inf: φdµ + ψdυ + inf (c(x, y) (φ(x) + ψ(y)))dγ γ 0 X Y Yiping Lu (pku) Geometry Of WGAN 7/34

8 Introduction To Optimal Transport. Kantorovich Dual If the central notion inthe original Monge-Kantorovich problem is cost, in the dual problem it is price. Imagine that a company offers to take care of all your transportation problem, buying bread at the bakeries and selling them to the cafes. Let ψ(x) be the price at which a basker of bread at the bakery x and selling them to the cafe y at the price φ(y) Let us maximize the profit: { } sup φ(y)dυ(y) ψ(x)dµ(y) φ(y) ψ(x) c(x, y) Y X Yiping Lu (pku) Geometry Of WGAN 8/34

9 Introduction To Optimal Transport. Kantorovich Dual It is easy to proof that { sup φ ψ c Y φ(y)dυ(y) inf π Π(µ,υ) { X Y If we describe a pair of prices (φ, ψ) as tight if X } ψ(x)dµ(y) } c(x, y)dπ(x, y) φ(y) = inf(ψ(x) + c(x, y)) x ψ(x) = sup(φ(y) c(x, y)) y The following formula can be seen as the definition of c transform. Yiping Lu (pku) Geometry Of WGAN 9/34

10 Introduction To Optimal Transport. c-cyclical Monotonicity Definition Once a function c : X R {+ } is given, we say that a set Γ X Y is c-cyclically monotone if for every k N, every permutation σ and every finite family of points (x 1, y 1 ),, (x k, y k ) Γ we have k c(x i, y i ) i=1 k c(x i, y σ(i) ) i=1 Yiping Lu (pku) Geometry Of WGAN 10/34

11 Introduction To Optimal Transport. c-cyclical Monotonicity Definition Once a function c : X R {+ } is given, we say that a set Γ X Y is c-cyclically monotone if for every k N, every permutation σ and every finite family of points (x 1, y 1 ),, (x k, y k ) Γ we have k c(x i, y i ) i=1 k c(x i, y σ(i) ) i=1 Theorem If γ is an optimal transport plane for the cost c and c is continuous, then spt(γ) is a CM-set. Yiping Lu (pku) Geometry Of WGAN 10/34

12 Introduction To Optimal Transport. c-cyclical Monotonicity Theorem Rockafellar s Theorem If Γ is a c CM set in X Y and c : X Y R, then there exists a c concave function φ : X R { } such that Γ {(x, y) X Y : φ(x) + φ c (y) = c(x, y)} Yiping Lu (pku) Geometry Of WGAN 11/34

13 Introduction To Optimal Transport. c-cyclical Monotonicity Theorem Rockafellar s Theorem If Γ is a c CM set in X Y and c : X Y R, then there exists a c concave function φ : X R { } such that Proof. Γ {(x, y) X Y : φ(x) + φ c (y) = c(x, y)} The function φ can be defined as φ(x) = inf{c(x, y n ) c(x n, y n ) + c(x n, y n 1 ) c(x n 1, y n 1 ) + +c(x 1, y 0 ) c(x 0, y 0 ) : n N, (x i, y i ) Γ} Yiping Lu (pku) Geometry Of WGAN 11/34

14 Introduction To Optimal Transport. Kantorovich Dual As a result, we can get a theorem as below. Theorem If c is C 1, φ is a Kantorovich potential for the cost c in the transport from µ to υ, and (x 0, y 0 ) belongs to the support of an optimal transport plane γ, then φ (x 0 ) = x c(x 0, y 0 ), provided φ is differentiable at x 0. Yiping Lu (pku) Geometry Of WGAN 12/34

15 Introduction To Optimal Transport. Kantorovich Dual As a result, we can get a theorem as below. Theorem If c is C 1, φ is a Kantorovich potential for the cost c in the transport from µ to υ, and (x 0, y 0 ) belongs to the support of an optimal transport plane γ, then φ (x 0 ) = x c(x 0, y 0 ), provided φ is differentiable at x 0. As an example, if the cost function has the following form c(x, y) = h(x y), h is strictly convex. Then there is exists an optimal transport plan γ for the cost c(x, y) and is unique of the form (id, T ) # µ. Moreover, ther exists a Kantorovich potential φ and T and the potentials φ are linked by T (x) = x ( h) 1 ( φ(x)) Yiping Lu (pku) Geometry Of WGAN 12/34

16 Introduction To Optimal Transport. Quadratic Case For the quadratic case c(x, y) = 1 2 x y 2 Theorem T (x) = x φ(x) = ( x 2 φ(x)) = u(x) 2 For function X : R n R, let us define u X = 1 2 x 2 X(x), then we have Proof. u X c(x) = sup y u X c = (u X ) 1 2 x x y 2 + X(y) = sup y ( ) 1 x y 2 y 2 X(y) Yiping Lu (pku) Geometry Of WGAN 13/34

17 Introduction To Optimal Transport. Quadratic Case We go futhur more on the quadratic case, we only need to minimize the x ydγ gives the same result. We can give the same result easier, actually we have φ(x 0 ) + φ (y 0 ) = x 0 y 0 for y 0 φ(x 0 ), which means Theorem For the quadratic case, there exists unique an optimal transport map T from µ to υ and it is of the form T = u for a convex function u Yiping Lu (pku) Geometry Of WGAN 14/34

18 Introduction To Optimal Transport. Quadratic Case We go futhur more on the quadratic case, we only need to minimize the x ydγ gives the same result. We can give the same result easier, actually we have φ(x 0 ) + φ (y 0 ) = x 0 y 0 for y 0 φ(x 0 ), which means Theorem For the quadratic case, there exists unique an optimal transport map T from µ to υ and it is of the form T = u for a convex function u Remark The φ above is called Kantorovich Potential. The u here is called Brenier Potential. Yiping Lu (pku) Geometry Of WGAN 14/34

19 Minkowski Type Problems Outline 1 Introduction To Optimal Transport. 2 Minkowski Type Problems Picewise Linear Function And Power Diagram 3 WGAN Yiping Lu (pku) Geometry Of WGAN 15/34

20 Minkowski Type Problems Minkowski Problem Theorem Suppose Ω is a compact convex polytope with non-empty interior in R n are distinct k points and A 1, A 2,, A k > 0 s.t. k i=1 A i = vol(ω). Then there exists a vector h = (h 1,, h k ) R k, unique up to adding the constant (c, c,, c), so that the piecewise linear convex function u(x) = max x Ω {x p i + h i } satisfies vol({x Ω u(x) = p i }) = A i Yiping Lu (pku) Geometry Of WGAN 16/34

21 Minkowski Type Problems Minkowski Problem Yiping Lu (pku) Geometry Of WGAN 17/34

22 Minkowski Type Problems Picewise Linear Function And Power Diagram Outline 1 Introduction To Optimal Transport. 2 Minkowski Type Problems Picewise Linear Function And Power Diagram 3 WGAN Yiping Lu (pku) Geometry Of WGAN 18/34

23 Minkowski Type Problems Picewise Linear Function And Power Diagram Piecewise Linear(PL) Function Definition PL Function: For P = {p 1,, p k } and h = (h 1,, h k ) R k, we define the PL convex function u h (x) to be u(x) = max{p i x + h i i = 1, 2,, k} The domain D(u ) of the dual function u is the convex hull of P and u (y) = min{ k t i h i t i 0, i=1 k t i = 1, i=1 k t i p i = y} i=1 Yiping Lu (pku) Geometry Of WGAN 19/34

24 Minkowski Type Problems Picewise Linear Function And Power Diagram Piecewise Linear(PL) Function PL-convex function f defined on a closed convex polyhedron produces a convex subdivision. Yiping Lu (pku) Geometry Of WGAN 20/34

25 Minkowski Type Problems Picewise Linear Function And Power Diagram Power Diagram Definition (Power Distance) Given a point y i R n with a power weight φ i the power distance is given by pow(x, y i ) = x y i 2 φ i Definition (Power Diagram) Given weighted points {(y i, φ i )}, the power diagram is the cell decompostion of R n, denote as V (φ) R n = k i=1 W i(φ), W i (φ) = {x R n pow(x, y i ) pow(x, y j ), j} Each cell is a convex polytope Yiping Lu (pku) Geometry Of WGAN 21/34

26 Minkowski Type Problems Picewise Linear Function And Power Diagram Power Diagram Now consider a eqaul construction as let h i = 1 2 (φ i y i 2 ), we construct the convex function u h (x) = max{ x, y i + h}, W i (h) = max{x p i + h i } i i Yiping Lu (pku) Geometry Of WGAN 22/34

27 Minkowski Type Problems Picewise Linear Function And Power Diagram Variation Proposition Suppose σ R is continuous defined on a compact convex domain Ω R n. If p 1,, p k R n are distinct and h R k so that vol(w i (h) Ω) > 0 for all i, then ω i (h) = W i (h) Ω σ(x) is a differentialable function in h so that for j i and W i (h) Ω and W i (h) Ω share a codimension-1 face F, ω i (h) = 1 σ F (x)da h j p i p j F where da is the area form on F and parital derivative is zero otherwise. Yiping Lu (pku) Geometry Of WGAN 23/34

28 Minkowski Type Problems Picewise Linear Function And Power Diagram Variation It is easy to observe that ω i h j Theorem = ω j h i, thus we can give our main theorem. Yiping Lu (pku) Geometry Of WGAN 24/34

29 Minkowski Type Problems Picewise Linear Function And Power Diagram Semi-discrete Optimal Mass Transport For our empirical distribution is defined as the sum of several Dirac measure v = k j=1 v jδ(y y j ) Define the discrete Kantorovich potential φ : Y R, φ(y j ) = φ j, then Y φdv = k φ j v j j=1 Define the c transformation of φ is given by and each cell is defined as φ c (x) = min 1 j k {c(x, y j) φ j } W i (φ) = {x X c(x, y i ) φ i c(x, y j ) φ j, 1 j k} Yiping Lu (pku) Geometry Of WGAN 25/34

30 Minkowski Type Problems Picewise Linear Function And Power Diagram Brenier s Approach We only consider the situation that the cost function is the L 2 distance. Here Then and at the same time u h (x) = k max i=1 { x, y i + h} W i (h) = {x X u h (x) = y i } Ω u h : W i (h) y i, i = 1, 2,, k Yiping Lu (pku) Geometry Of WGAN 26/34

31 WGAN Outline 1 Introduction To Optimal Transport. 2 Minkowski Type Problems Picewise Linear Function And Power Diagram 3 WGAN Yiping Lu (pku) Geometry Of WGAN 27/34

32 WGAN GAN/WGAN Objective Function: min max u U v GAN:φ = log WGAN:φ = id E x Dreal [φ(d v (x))] + E x DG [φ(1 D v (x))] Yiping Lu (pku) Geometry Of WGAN 28/34

33 WGAN GAN/WGAN Objective Function: min max u U v GAN:φ = log WGAN:φ = id Rewrite the WGAN Objective: min max u U v E x Dreal [φ(d v (x))] + E x DG [φ(1 D v (x))] E x Dreal [D v (x)] + E x DG [1 D v (x)] Yiping Lu (pku) Geometry Of WGAN 28/34

34 WGAN GAN/WGAN Objective Function: min max u U v GAN:φ = log WGAN:φ = id Rewrite the WGAN Objective: equal to min max u U v E x Dreal [φ(d v (x))] + E x DG [φ(1 D v (x))] min max u U v E x Dreal [D v (x)] + E x DG [1 D v (x)] E x Dreal [D v (x)] E x DG [D v (x)] Yiping Lu (pku) Geometry Of WGAN 28/34

35 WGAN GAN/WGAN Objective Function: min max u U v GAN:φ = log WGAN:φ = id Rewrite the WGAN Objective: equal to min max u U v E x Dreal [φ(d v (x))] + E x DG [φ(1 D v (x))] min max u U v E x Dreal [D v (x)] + E x DG [1 D v (x)] E x Dreal [D v (x)] E x DG [D v (x)] For if c(x, y) = x y, then φ c = φ(1-lip),approximate to W 1 Distance Yiping Lu (pku) Geometry Of WGAN 28/34

36 WGAN Geometric Generative Model Ecoding/Decoding process: This setp maps the smaples between the image space X and the latent space Z by deep neural networks, the encoding map is denoted as f θ : X Z and decoding map is g ξ : Z X Probability measure transformation process: This step transform a fixed distibution ξ P(Z ) to any given distribution µ P(Z ), the mapping is denoted as T : Z Z, T # ξ = µ. This step can either use conventional deep neural network or use explicit geometric/numerical methods. Yiping Lu (pku) Geometry Of WGAN 29/34

37 WGAN WGAN Yiping Lu (pku) Geometry Of WGAN 30/34

38 WGAN Geometric OMT Yiping Lu (pku) Geometry Of WGAN 31/34

39 WGAN Geometric Method Yiping Lu (pku) Geometry Of WGAN 32/34

40 WGAN Geometric Method Yiping Lu (pku) Geometry Of WGAN 33/34

41 WGAN Conclusion Yiping Lu (pku) Geometry Of WGAN 34/34