Shape Matching: A Metric Geometry Approach. Facundo Mémoli. CS 468, Stanford University, Fall 2008.

Transcription

1 Shape Matching: A Metric Geometry Approach Facundo Mémoli. CS 468, Stanford University, Fall

2 The Problem of Shape/Object Matching databases of objects objects can be many things: proteins molecules 2D objects (imaging) 3D shapes: as obtained via a 3D scanner 3D shapes: modeled with CAD software 3D shapes: coming from design of bone protheses text documents more complicated structures present in datasets (things you can t visualize)

3 3D objects: examples cultural heritage (Michelangelo project: search of parts in a factory of, say, cars face recognition: the face of an individual is a 3D shape... proteins: the shape of a protein reflects its function.. protein data bank:

4 Typical situation: classification assume you have database D of objects. assume D is composed by several objects, and that each of these objects belongs to one of n classes C 1,..., C n. imagine you are given a new object o, not in your database, and you are asked to determine whether o belongs to one of the classes. If yes, you also need to point to the class. One simple procedure is to say that you will assign object o the class of the closest object in D: class(o) = class(z) where z D minimizes dist(o, z) in order to do this, one first needs to define a notion dist of distance or dis-similarity between objects.

5 Another important point: invariances Are these two objects the same? X Y

6 Another important point: invariances Are these two objects the same? XY this is called invariance to rigid transformations

7 Another important points: invariances what about these two? roughly speaking, this corresponds to invariance to bending transformations..

8 invariances... The measure of dis-similarity dist must capture the type of invariance you want to encode in your classification system. =? dist(, )=0?

9 What we proposed in the course: 1. Decide what invariances you wish to incorporate. It is OK if you don t have invariances (Hausdorff + Wasserstein) 2. Represent shapes as metric spaces (or mm-spaces): Identify what metric is preserved by the notion of invariance you chose to consider. Endow shapes with that metric Choose weights that are meaningful for your application (if you don t have any reason to choose: then set them to be equal) 3. Define a metric on your class of objects. We studied the case of no invariances first.

10 No invariances.. You start out with a compact metric space Z, d (called ambient space, typically Z R k ). We saw two constructions: C Z,d Z H and C w Z,d Z W,p where C Z stands for all compact subsets of Z and C w Z for all weighted subsets of Z: that is, pairs A, µ A where µ A is a probability measure on Z s.t. supp µ A A. X Y X Y

11 Ambient space isometries..(extrinsic approach) Fix a compact metric space Z, d. Let I Z denote the isometry group of Z (when Z R k, I Z E k, that is, all Euclidean isometries.) In this case, we considered either objects in C Z denote your choice. or in C w Z. Let O Z Let dist be the corresponding metric (Hausdorff or Wasserstein). Then, we constructed distances between objects in O Z that were blind to isometries: dist iso X, Y : inf T I Z dist X, T Y for all X, Y O Z. X Y

12 Ambient space isometries..(extrinsic approach) Fix a compact metric space Z, d. Let I Z denote the isometry group of Z (when Z R k, I Z E k, that is, all Euclidean isometries.) In this case, we considered either objects in C Z denote your choice. or in C w Z. Let O Z Let dist be the corresponding metric (Hausdorff or Wasserstein). Then, we constructed distances between objects in O Z that were blind to isometries: dist iso X, Y : inf T I Z dist X, T Y for all X, Y O Z. XY

13 Ambient space isometries..(extrinsic approach) Fix a compact metric space Z, d. Let I Z denote the isometry group of Z (when Z R k, I Z E k, that is, all Euclidean isometries.) In this case, we considered either objects in C Z denote your choice. or in C w Z. Let O Z Let dist be the corresponding metric (Hausdorff or Wasserstein). Then, we constructed distances between objects in O Z that were blind to isometries: for all X, Y O Z. but dist this iso X, was Y : not infgeneral dist X, T enough... Y T I Z remember BENDS XY

14

15 The intrinsic approach... Regard shapes as metric spaces in themselves: no reference to any ambient space. That is, objects/shapes now are metric spaces or mm-spaces X, d X or X, d X,µ X. Let M denote collection of all metric spaces and M w the collections of all mm-spaces. Endow M and/or M w with a metric. We saw the following constructions: On M we put the Gromov-Hausdorff distance d GH,. On M w we put two metrics, S p and D p. These metrics could be called Gromov-Wasserstein metrics.

16 C Z C w Z M M w d H d W,p d GH D p S p

17 C Z C w Z M M w Extrinsic approach d H d W,p d GH D p S p

18 C Z C w Z M M w Extrinsic approach d H d W,p Intrinsic approach d GH D p S p

19 What is the relationship between the Intrinsic and Extrinsic Approaches? Answer known for Z R k, [M08-euclidean]. In class we saw the case of d GH vs. d iso H : for all X, Y Rk, d GH X,, Y, inf T E k d H X, T Y C k d GH X,, Y, 1 2 M 1 2 where M max diam X, diam Y. There is a similar claim valid for S p vs. d iso W,p.

20 Main points Define notion of distance on shapes. Get sampling consistency + stability for free. GH distance leads to hard combinatorial optimization problems. Relaxations of these do not appear to be correct. Gromov-Wasserstein distances are better. Both D p and S p yield quadratic optimization probs. with linear constraints. Sturm s S p requires large nbr. of constraints, then we argue for D p. Many lower bounds for D p are possible. These employ invariants previously used in the literature: Shape Distributions Eccentricities (Hamza-Krim) Shape contexts

21 Main Technical concepts Metric spaces, mm-spaces, isometries, approximate isometries, probability measures. Correspondences, measure couplings, metric couplings. Hausdorff distance. Wasserstein distance. Mass transportation. Gromov-Hausdorff distance. Gromov-Wasserstein distances. Invariants of mm-spaces

22 correspondences and the Hausdorff distance Definition [Correspondences] For sets A and B, a subset R A B is a correspondence (between A and B) if and and only if a A, there exists b B s.t. (a, b) R b B, there exists a A s.t. (a, b) R Let R(A, B) denote the set of all possible correspondences between sets A and B. Note that in the case n A = n B, correspondences are larger than bijections. 17

23 correspondences Note that when A and B are finite, R R(A, B) can be represented by a matrix ((r a,b )) {0, 1} n A n B s.t. r ab 1 b B a A r ab 1 a A b B 18

24 correspondences Note that when A and B are finite, R R(A, B) can be represented by a matrix ((r a,b )) {0, 1} n A n B s.t. r ab 1 b B a A r ab 1 a A b B Proposition Let (X, d) be a compact metric space and A, B X be compact. Then d H (A, B) = inf R R(A,B) d L (R) 19

25 correspondences and measure couplings Let (A, µ A ) and (B, µ B ) be compact subsets of the compact metric space (X, d) and µ A and µ B be probability measures supported in A and B respectively. Definition [Measure coupling] Is a probability measure µ on A B s.t. (in the finite case this means ((µ a,b )) [0, 1] n A n B ) a A µ ab = µ B (b) b B b B µ ab = µ A (a) a A Let M(µ A,µ B ) be the set of all couplings of µ A and µ B. Notice that in the finite case, ((µ a,b )) must satisfy n A + n B linear constraints. 20

26 correspondences and measure couplings Proposition [(µ R)] Given (A, µ A ) and (B, µ B ), and µ M(µ A,µ B ), then R(µ) := supp(µ) R(A, B). König s Lemma. [gives conditions for R µ] 21

27 Wasserstein distance d H (A, B) = inf R R(A,B) d L (R) (R µ) d W, (A, B) = inf µ M(µ A,µ B ) d L (R(µ)) (L L p ) d W,p (A, B) = inf µ M(µ A,µ B ) d L p (A B,µ) 22

28 Wasserstein distance d H (A, B) = inf R R(A,B) d L (R) (R µ) d W, (A, B) = inf µ M(µ A,µ B ) d L (R(µ)) (L L p ) d W,p (A, B) = inf µ M(µ A,µ B ) d L p (A B,µ) 22

29 Wasserstein distance d H (A, B) = inf R R(A,B) d L (R) (R µ) d W, (A, B) = inf µ M(µ A,µ B ) d L (R(µ)) (L L p ) d W,p (A, B) = inf µ M(µ A,µ B ) d L p (A B,µ) 22

30 Wasserstein distance d H (A, B) = inf R R(A,B) d L (R) (R µ) d W, (A, B) = inf µ M(µ A,µ B ) d L (R(µ)) (L L p ) d W,p (A, B) = inf µ M(µ A,µ B ) d L p (A B,µ) 22

31 GH distance 23

32 GH: definition d GH (X, Y ) = inf Z,f,g dz H(f(X),g(Y )) 24

33 correspondences and GH distance The GH distance between (X, d X ) and (Y, d Y ) admits the following expression: d (1) GH (X, Y ) = inf d D(d X,d Y ) inf R R(X,Y ) d L (R) where D(d X,d Y ) is a metric on X Y that reduces to d X and d Y and Y Y, respectively. on X X ( X Y ) X d X D Y D T d Y In other words: you need to glue X and Y in an optimal way. Note that D consists of n X n Y positive reals that must satisfy n X C n Y 2 + n Y C n X 2 linear constraints. 25 = d

34 Another expression for the GH distance For compact spaces (X, d X ) and (Y, d Y ) let d (2) GH (X, Y )=1 2 inf R max d X(x, x ) d Y (y, y ) (x,y),(x,y ) R We write, compactly, d (2) GH (X, Y )=1 2 inf R d X d Y L (R R) 26

35 Equivalence thm: Theorem [Kalton-Ostrovskii] For all X, Y compact, d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R d X d Y L (R R) 27

36 Relaxing the notion of correspondence from GH to GW d (1) GH d H d (2) GH d (1) GW,p?? d W,p d (2) GW,p 28

37 Shapes as mm-spaces, [M07] Remember: (X, d X,µ X ) 1. Specify representation of shapes. 2. Identify invariances that you want to mod out. 3. Describe notion of isomorphism between shapes (this is going to be the zero of your metric) 4. Come up with a metric between shapes (in the representation of 1.) Now we are talking of triples (X, d X,µ X ) where X is a set, d X a metric on X and µ X a probability measure on X. These objects are called measure metric spaces, or mm-spaces for short. two mm-spaces X and Y are deemed equal or isomorphic whenever there exists an isometry Φ : X Y s.t. µ Y (B) =µ X (Φ 1 (B) for all (measurable) sets B Y.

38 Remember Now, one works with mm-spaces: triples (X, d, ν) where (X, d) is a compact metric space and ν is a Borel probability measure. Two mm-spaces are isomorphic iff there exists isometry Φ : X Y s.t. µ X (Φ 1 (B)) = µ Y (B) for all measurable B Y. 1 3/4 1/4 1/2 1 1/2 30

39 The plan d (1) GH d (2) GH d H d (1) GW,p?? d (2) GW,p d W,p 31

40 The plan d (1) GH d (2) GH d H d (1) GW,p?? d (2) GW,p d W,p 31

41 The plan d (1) GH d (2) GH d H d (1) GW,p?? d (2) GW,p d W,p 31

42 The plan d (1) GH d (2) GH d H d (1) GW,p?? d (2) GW,p d W,p 31

43 The plan d (1) GH d (2) GH d H d (1) GW,p?? d (2) GW,p d W,p 31

44 d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R Γ X,Y L (R R) 32

45 d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R Γ X,Y L (R R) inf d,µ d Lp (µ) 1 2 inf µ Γ X,Y L p (µ µ) d (1) GW,p d (2) GW,p

46 d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R Γ X,Y L (R R) inf d,µ d Lp (µ) 1 2 inf µ Γ X,Y L p (µ µ) d (1) GW,p d (2) GW,p

47 d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R Γ X,Y L (R R) inf d,µ d Lp (µ) 1 2 inf µ Γ X,Y L p (µ µ) d (1) GW,p d (2) GW,p

48 d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R Γ X,Y L (R R) inf d,µ d Lp (µ) 1 2 inf µ Γ X,Y L p (µ µ) S p d (1) d (2) GW,p GW,p

49 d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R Γ X,Y L (R R) inf d,µ d Lp (µ) 1 2 inf µ Γ X,Y L p (µ µ) S p d (1) d (2) GW,p GW,p

50 d (1) GH d (2) GH inf d,r d L (R) 1 2 inf R Γ X,Y L (R R) inf d,µ d Lp (µ) 1 2 inf µ Γ X,Y L p (µ µ) S p D p d (1) d (2) GW,p GW,p

51 Can S p be equal to D p? Using the same proof as in the Kalton-Ostrovskii Thm., one can prove that S = D. Also, it is easy to see that for all p 1 S p D p. But the equality does not hold in general. One counterexample is as follows: take X =( n 1, ((d ij = 1)), (ν i =1/n)) and Y =({q}, ((0)), (1)). Then, for p [1, ) ( ) 1/p n 1 = D 1 (X, Y ) S 1 (X, Y )= 1 2 > 1 2 n Furthermore, these two (tentative) distances are not Lipschitz equivalent!! This forces us to analyze them separately. The delicate step is proving that dist(x, Y ) = 0 implies X Y. K. T. Sturm has analyzed S p. Analysis of D p is in [M07]. 1/3 1/4 { n } n= /2 1/2 1/3 1 1/3 1/ /4 1/4

52 Properties of D p Theorem 1 ([M07]). 1. Let X, Y and Z mm-spaces then 2. If D p X, Y 0 then X and Y are isomorphic. D p X, Y D p X, Z D p Y, Z. 3. Let X n x 1,..., x n X be a subset of the mm-space X, d, ν. Endow X n with the metric d and a prob. measure ν n, then 4. p q 1, then D p D q. 5. D d GH. D p X, X n d W,p ν, ν n. 35

53 The parameter p is not superfluous For p 1, let diam p X d X L p µ X µ X. The simplest lower bound for D p one has is based on the triangle inequality plus the observation that D p X, d X,µ X, q, 0, 1 diam p X Then, D p X, Y 1 2 diam p X diam p Y For example, when X and usual intrinsic metric): S n (spheres with uniform measure p gives diam S n π for all n N p 1 gives diam 1 S n π 2 for all n N p 2 gives diam 2 S 1 π 3 and diam 2 S 2 π

54 Lower bounds for D p in terms of invariants. Recall the invariants we defined before. Fix X G w and p 1. distribution of distances: F X : 0, 0, 1, t µ X µ X x, x d X x, x t. local distribution of distances: C X : X 0, 0, 1, t µ X x d X x, x t. eccentricities: s X,p : X R, x d X x, L p µ X. There are explicit lower bounds for D p in terms of these invariants, [M07]. These lower bounds are important in practice: yield LOPs, easy optimization problems. Solution can be used as initial condition for solving D p.

55 Shape Distributions [Osada-et-al] 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d

56 Shape Distributions [Osada-et-al] 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d

57 Shape Distributions [Osada-et-al] 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d

58 Shape Distributions [Osada-et-al] 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d

59 Shape Contexts 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d

60 Shape Contexts 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d

61 Shape Contexts 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d

62 Hamza-Krim j d 1,j N 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d j d 2,j N j d N,j N

63 Hamza-Krim j d 1,j N 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d j d 2,j N j d N,j N

64 Hamza-Krim j d 1,j N 0 d 12 d 13 d d 12 0 d 23 d d 13 d 23 0 d d 14 d 24 d j d 2,j N j d N,j N

65 The discrete case. The option proposed and analyzed by K.L Sturm [St06], reads S p (X, Y ) = inf d D(d X,d Y ) inf µ M(µ X,µ Y ) ( x,y d p (x, y)µ x,y ) 1/p The second option reads [M07] D p (X, Y ) = inf µ M(µ X,µ Y ) x,y d X (x, x ) d Y (y, y ) p µ x,y µ x,y x,y 1/p 41

66 The first option, S p = inf d D(d X,d Y ) inf µ M(µ X,µ Y ) ( x,y d p (x, y)µ x,y ) 1/p requires 2(n X n Y ) variables and n X + n Y plus n Y C n X 2 + n X C n Y 2 linear constraints. When p = 1 it yields a bilinear optimization problem. Our second option, D p (X, Y ) = inf µ M(µ X,µ Y ) x,y d X (x, x ) d Y (y, y ) p µ x,y µ x,y x,y 1/p requires n X n Y variables and n X + n Y linear constraints. It is a quadratic (generally non-convex :-( ) optimization problem (with linear and bound constraints) for all p. 42

67 Future Study families of shapes. Statistic of families of shapes. Partial shape matching. Connections with Persistent topology invariants (Frosini+others... Yi will describe Frosini s work) Comparison/matching of animated geometries (Peter will talk about this)

68 Future Study families of shapes. Statistic of families of shapes. Partial shape matching. Connections with Persistent topology invariants (Frosini+others... Yi will describe Frosini s work) Comparison/matching of animated geometries (Peter will talk about this)

69 Future Study families of shapes. Statistic of families of shapes. Partial shape matching. Connections with Persistent topology invariants (Frosini+others... Yi will describe Frosini s work) Comparison/matching of animated geometries (Peter will talk about this)

70 and more Explain/relate more methods using these ideas: what about eigenvalue based methods? Shape signatures [Reuter-et-al]: associate to each shape the sorted list of eigenvalues of the Laplacian on the shape. Leordaneau... from matching of pairs of points to matching of points. The GH distance and related Metric Geometry ideas are very powerful and can probably help uniformizing the treatment of many algorithmic procedure out there.

71 Bibliography [BBI] Burago, Burago and Ivanov. A course on Metric Geometry. AMS, 2001 [EK] A. Elad (Elbaz) and R. Kimmel. On bending invariant signatures for surfaces, IEEE Trans. on PAMI, 25(10): , [BBK] A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proc. Natl. Acad. Sci. USA, 103(5): (electronic), [HK] A. Ben Hamza, Hamid Krim: Probabilistic shape descriptor for triangulated surfaces. ICIP (1) 2005: [Osada-et-al] Robert Osada, Thomas A. Funkhouser, Bernard Chazelle, David P. Dobkin: Matching 3D Models with Shape Distributions. Shape Modeling International 2001: [SC] S. Belongie and J. Malik (2000). Matching with Shape Contexts. IEEE Workshop on Contentbased Access of Image and Video Libraries (CBAIVL- 2000). [Goodrich] M. Goodrich, J. Mitchell, and M. Orletsky. Approximate geometric pattern matching under rigid motions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(4): , [M08-partial] F.Memoli. Lp Gromov-Hausdorff distances for partial shape matching, preprint. [M08-euclidean]F. Mémoli. Gromov-Hausdorff distances in Euclidean spaces. NORDIA-CVPR [M07] F.Memoli. On the use of gromov-hausdorff distances for shape comparison. In Proceedings of PBG 2007, Prague, Czech Republic, [MS04] F. Memoli and G. Sapiro. Comparing point clouds. In SGP 04: Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, pages 32 40, New York, NY, USA, ACM. [MS05] F. Memoli and G. Sapiro. A theoretical and computational framework for isometry invariant recognition of point cloud data. Found. Comput. Math., 5(3): , 2005.

72