Object Recognition, Symmetry Detection, Jigsaw Puzzles, and Cancer

Transcription

1 Object Recognition, Symmetry Detection, Jigsaw Puzzles, and Cancer Peter J. Olver University of Minnesota olver Technion, May, 2015

2 Symmetry Definition. A symmetry of a set S is a transformation that preserves it: g S = S The set of symmetries forms a group, called the symmetry group of the set S.

3 Discrete Symmetry Group Rotations by 90 : G S = Z 4 Rotations + reflections: G S = Z 4 Z 4

4 Continuous Symmetry Group Rotations: G S =SO(2) Rotations + reflections: G S =O(2) Conformal Inversions: x y x = x 2 + y 2 y = x 2 + y 2 A continuous group is known as a Lie group in honor of Sophus Lie.

5 Continuous Symmetries of a Square R

6 Symmetry To define the set of symmetries requires a priori specification of the allowable transformations or, equivalently, the underlying geometry. G transformation group or pseudo-group of allowable transformations of the ambient space M Definition. A symmetry of a subset S M is an allowable transformation g G that preserves it: g S = S

7 What is the Symmetry Group? Allowable transformations: Rigid motions G =SE(2)=SO(2) R 2 G S = Z 4 Z 2

8 What is the Symmetry Group? Allowable transformations: Rigid motions G =SE(2)=SO(2) R 2 G S = {e}

9 Local Symmetries Definition. g G is a local symmetry of S M based at a point z S if there is an open neighborhood z U M such that g (S U) =S (g U) G z G thesetoflocal symmetries based at z. Global symmetries are local symmetries at all z S: G S G z G S = G z z S The set of all local symmetries forms a groupoid!

10 Groupoids Definition. A groupoid is a small category such that every morphism has an inverse. = Brandt (quadratic forms), Ehresmann (Lie pseudo-groups) Mackenzie, R. Brown, A. Weinstein Groupoids form the appropriate framework for studying objects with variable symmetry.

11 Groupoids Double fibration: G M σ σ source map τ target map τ M You are only allowed to multiply α β G if σ(α) =τ (β)

12 Groupoids Source and target of products: σ(α β) =σ(β) τ (α β) =τ (α) when σ(α) =τ (β) Associativity: α (β γ) =(α β) γ when defined Identity section: e : M G σ(e(x)) = x = τ (e(x)) α e(σ(α)) = α = e(τ (α)) α Inverses: σ(α) =x = τ (α 1 ), τ (α) =y = σ(α 1 ), α 1 α = e(x), α α 1 = e(y)

13 Jet Groupoids = Ehresmann The set of Taylor polynomials of degree n, or Taylor series (n = ) of local diffeomorphisms Ψ: M M forms a groupoid. Algebraic composition of Taylor polynomials/series is well-defined only when the source of the second matches the target of the first.

14 The Symmetry Groupoid Definition. The symmetry groupoid of S M is G S = { (g, z) z S, g G z } G S Source and target maps: σ(g, z) = z, τ (g, z) = g z. Groupoid multiplication and inversion: (h, g z) (g, z) =(g h, z) (g, z) 1 =(g 1,g z) Identity map: e(z) =(z, e) G S

15 What is the Symmetry Groupoid? G =SE(2) Corners: G z = G S = Z 4 Sides: G z generated by G S = Z 4 some translations 180 rotation around z

16 What is the Symmetry Groupoid? Cogwheels = Musso Nicoldi G S = Z 6 G S = Z 2

17 What is the Symmetry Groupoid? Cogwheels = Musso Nicoldi b a b a a b b b b a a a a b a a b a b b a b a b G S = Z 6 G S = Z 2

18 Geometry = Group Theory Felix Klein s Erlanger Programm (1872): Each type of geometry is founded on an underlying transformation group.

19 Plane Geometries/Groups Euclidean geometry: SE(2) rigid motions (rotations and translations) ( ) ( )( ) ( ) x cos θ sin θ x a = + y sin θ cos θ y b E(2) plus reflections? Equi-affine geometry: SA(2) area-preserving affine transformations: ( ) ( )( ) ( ) x α β x a = + αδ βγ =1 y γ δ y b Projective geometry: PSL(3) projective transformations: x = αx+ βy+ γ ρx+ σy+ τ y = λx+ µy+ ν ρx+ σy+ τ

20 The Equivalence Problem = ÉCartan G transformationgroupactingonm Equivalence: Determine when two subsets are congruent: Symmetry: S and S M S = g S for g G Find all symmetries or self-congruences: S = g S

21 Tennis, Anyone?

22 Invariants The solution to an equivalence problem rests on understanding its invariants. Definition. If G is a group acting on M, thenaninvariant is a real-valued function I : M R that does not change under the action of G: I(g z) =I(z) for all g G, z M If G acts transtively, there are no (non-constant) invariants.

23 Differential Invariants Given a submanifold (curve, surface,...) S M a differential invariant is an invariant of the prolonged action of G on its Taylor coefficients (jets): I(g z (k) ) = I(z (k) )

24 Euclidean Plane Curves G =SE(2) actsoncurves C M = R 2 The simplest differential invariant is the curvature, defined as the reciprocal of the radius of the osculating circle: κ = 1 r

25 Curvature

26 Curvature

27 Curvature r =1/κ

28 Euclidean Plane Curves: G =SE(2) Differentiation with respect to the Euclidean-invariant arc length element ds is an invariant differential operator, meaning that it maps differential invariants to differential invariants. Thus, starting with curvature κ, wecangenerateaninfinite collection of higher order Euclidean differential invariants: κ, dκ ds, d 2 κ ds 2, d 3 κ ds 3, Theorem. All Euclidean differential invariants are functions of the derivatives of curvature with respect to arc length: κ, κ s, κ ss,

29 Euclidean Plane Curves: G =SE(2) Assume the curve C M is a graph: y = u(x) Differential invariants: κ = u xx (1 + u 2, dκ x )3/2 ds = (1 + u2 x )u xxx 3u x u2 xx (1 + u 2, x )3 d 2 κ ds 2 = Arc length (invariant one-form): ds = 1+u 2 x dx, d ds = 1 1+u 2 x d dx

30 Equi-affine Plane Curves: G =SA(2)=SL(2) R 2 Equi-affine curvature: κ = 5 u xx u xxxx 3 u2 xxx 9 u 8/3 xx dκ ds = Equi-affine arc length: ds = 3 uxx dx d ds = 3 1 uxx d dx Theorem. All equi-affine differential invariants are functions of the derivatives of equi-affine curvature with respect to equi-affine arc length: κ, κ s, κ ss,

31 Plane Curves Theorem. Let G be an ordinary Lie group acting on M = R 2. Then for curves C M, thereexistsaunique(upto functions thereof) lowest order differential invariant κ and a unique (up to constant multiple) invariant differential form ds. Everyotherdifferentialinvariantcanbewrittenasa function of the curvature invariant and its derivatives with respect to arc length : κ, κ s, κ ss,. ordinary = transitive + no pseudo-stabilization.

32 Moving Frames The equivariant method of moving frames provides a systematic and algorithmic calculus for determining complete systems of differential invariants, invariant differential forms, invariant differential operators, etc., and the structure of the non-commutative differential algebra they generate.

33 Equivalence & Invariants Equivalent submanifolds S S must have the same invariants: I = I. Constant invariants provide immediate information: e.g. κ =2 κ =2 Non-constant invariants are not useful in isolation, because an equivalence map can drastically alter the dependence on the submanifold parameters: e.g. κ = x 3 versus κ = sinh x

34 Syzygies However, a functional dependency or syzygy among the invariants is intrinsic: e.g. κ s = κ 3 1 κ s = κ 3 1 Universal syzygies Gauss Codazzi Distinguishing syzygies. Theorem. (Cartan) Two regular submanifolds are locally equivalent if and only if they have identical syzygies among all their differential invariants.

35 Finiteness of Generators and Syzygies There are, in general, an infinite number of differential invariants and hence an infinite number of syzygies must be compared to establish equivalence. But the higher order differential invariants are always generated by invariant differentiation from a finite collection of basic differential invariants, and the higher order syzygies are all consequences of a finite number of low order syzygies!

36 Example Plane Curves If non-constant, both κ and κ s depend on a single parameter, and so, locally, are subject to a syzygy: κ s = H(κ) ( ) But then κ ss = d ds H(κ) =H (κ) κ s = H (κ) H(κ) and similarly for κ sss,etc. Consequently, all the higher order syzygies are generated by the fundamental first order syzygy ( ). Thus, for Euclidean (or equi-affine or projective or...) plane curves we need only know a single syzygy between κ and κ s in order to establish equivalence!

37 Signature Curves Definition. The signature curve Σ R 2 of a plane curve C R 2 is parametrized by the two lowest order differential invariants χ : C Σ = {( κ, dκ ds )} R 2 = Calabi, PJO, Shakiban, Tannenbaum, Haker Theorem. Two regular curves C and C are locally equivalent: C = g C if and only if their signature curves are identical: Σ=Σ = regular: (κ s,κ ss ) 0.

38 Theorem. equivalent: Continuous Symmetries of Curves For a connected curve, the following are All the differential invariants are constant on C: κ = c, κ s =0,... The signature Σ degenerates to a point: dim Σ =0 C is a piece of an orbit of a 1-dimensional subgroup H G C admits a one-dimensional local symmetry group

39 Discrete Symmetries of Curves Definition. The index of a completely regular point ζ Σ equals the number of points in C which map to it: i ζ =#χ 1 {ζ} Regular means that, in a neighborhood of ζ, thesignatureisan embedded curve no self-intersections. Theorem. If χ(z) = ζ is completely regular, then its index counts the number of discrete local symmetries of C.

40 The Index χ C Σ

41 The Curve x =cost cos2 t, y =sint sin2 t The Original Curve Euclidean Signature Equi-affine Signature

42 The Curve x =cost cos2 t, y = 1 2 x +sint sin2 t The Original Curve Euclidean Signature Equi-affine Signature

43 %#" &'()* $"" &'()5 %"" ##" $#" #""!#" 36782/2889)"+*:%$%:!"" #""!"" #"",-./0('12)3'142)&'()*,-./0('12)3'142)&'()5 "+"* "+""# "!"+""#!"+"* "+"* "+""# "!"+""#!"+"* "+"* "+""# "!"+""#!"+"*!"+"*#!"+"*#!"+"*#!"+"# " "+"# "+*!"+"# " "+"# "+*!"+"# " "+"# "+*

44 '(()*& :32*& &""" #," %"" #"" $"" 9," 6;(<505<<=*"+">&!&# #""!"" 8"","" -./012345*63475*'(()*& -./012345*63475*:32*& "+"& "+"", "!"+"",!"+"& "+"& "+"", "!"+"",!"+"& "+"& "+"", "!"+"",!"+"&!"+"&,!"+"&,!"+"&,!"+", " "+", "+&!"+", " "+", "+&!"+", " "+", "+&

45 Signatures κ Classical Signature κ s s Original curve κ Differential invariant signature

46 Signatures κ Classical Signature κ s s Original curve κ Differential invariant signature

47 Occlusions κ Classical Signature κ s s Original curve κ Differential invariant signature

48 3D Differential Invariant Signatures Euclidean space curves: C R 3 Σ = { ( κ, κ s,τ) } R 3 κ curvature, τ torsion Euclidean surfaces: S R 3 (generic) Σ = {( H,K,H,1,H,2,K,1,K,2 )} R 6 or Σ = {( H,H,1,H,2,H,11 )} R 4 H meancurvature, K Gausscurvature Equi affine surfaces: S R 3 (generic) Σ = {( P,P,1,P,2,P,11 )} R 4 P Pickinvariant

49 Vertices of Euclidean Curves Ordinary vertex: local extremum of curvature Generalized vertex: κ s 0 critical point circular arc straight line segment Mukhopadhya s Four Vertex Theorem: A simple closed, non-circular plane curve has n 4generalized vertices.

50 Counterexamples = Musso Nicoldi Generalized vertices map to a single point of the signature. Hence, the (degenerate) curves obtained by replace ordinary vertices with circular arcs of the same radius all have identical signature:

51 Bivertex Arcs Bivertex arc: κ s 0everywhereonthearcB C except κ s =0atthetwoendpoints The signature Σ = χ(b) ofabivertexarcisasinglearcthat starts and ends on the κ axis. κ s κ

52 Bivertex Decomposition v-regular curve finitely many generalized vertices C = m j=1 B j n V k k=1 B 1,...,B m bivertex arcs V 1,...,V n generalized vertices: n 4 Main Idea: Compare individual bivertex arcs, and then decide whether the rigid equivalences are (approximately) the same. D. Hoff & PJO, Extensions of invariant signatures for object recognition, J. Math. Imaging Vision 45 (2013),

53 Signature Metrics Used to compare signatures: Hausdorff Monge Kantorovich transport Electrostatic/gravitational attraction Latent semantic analysis Histograms Geodesic distance Diffusion metric Gromov Hausdorff & Gromov Wasserstein

54 Gravitational/Electrostatic Attraction Treat the two signature curves as masses or as oppositely charged wires. The higher their mutual attraction, the closer they are together. κ s κ

55 Gravitational/Electrostatic Attraction Treat the two signature curves as masses or as oppositely charged wires. The higher their mutual attraction, the closer they are together. In practice, we are dealing with discrete data (pixels) and so treat the curves and signatures as point masses/charges. κ s κ s κ κ

56 The Baffler Jigsaw Puzzle

57 Piece Locking Minimize force and torque based on gravitational attraction of the two matching edges.

58 The Baffler Solved

59 The Rain Forest Giant Floor Puzzle

60 The Rain Forest Puzzle Solved = D. Hoff & PJO, Automatic solution of jigsaw puzzles, J. Math. Imaging Vision 49 (2014)

61 3D Jigsaw Puzzles = Anna Grim, Tim O Connor, Ryan Schlecta

62 Broken Ostrich Egg Shell = Marshall Bern

63 Reassembling Humpty Dumpty

64 Benign vs. Malignant Tumors = A. Grim, C. Shakiban

65 Benign vs. Malignant Tumors

66 Benign vs. Malignant Tumors

67 Joint Invariant Signatures If the invariants depend on k points on a p-dimensional submanifold, then you need at least l>kp distinct invariants I 1,...,I l in order to construct a syzygy. Typically, the number of joint invariants is l = km r =(#points)(dimm) dim G Therefore, a purely joint invariant signature requires at least k r m p +1 points on our p-dimensional submanifold N M.

68 Joint Euclidean Signature z 0 a z 1 b e c d f z 3 z 2

69 Joint signature map: Σ: C 4 Σ R 6 a = z 0 z 1 b = z 0 z 2 c = z 0 z 3 d = z 1 z 2 e = z 1 z 3 f = z 2 z 3 Syzygies: = six functions of four variables Φ 1 (a, b, c, d, e, f) =0 Φ 2 (a, b, c, d, e, f) =0 Universal Cayley Menger syzygy C R 2 det 2a 2 a 2 + b 2 d 2 a 2 + c 2 e 2 a 2 + b 2 d 2 2b 2 b 2 + c 2 f 2 a 2 + c 2 e 2 b 2 + c 2 f 2 2c 2 =0

70 Joint Equi Affine Signature Requires 7 triangular areas: [ 012], [ 013], [ 014], [ 015], [ 023], [ 024], [ 025] z 1 z 2 z 0 z 5 z 3 z 4

71 Joint Invariant Signatures The joint invariant signature subsumes other signatures, but resides in a higher dimensional space and contains a lot of redundant information. Identification of landmarks can significantly reduce the redundancies (Boutin) It includes the differential invariant signature and semidifferential invariant signatures as its coalescent boundaries. Invariant numerical approximations to differential invariants and semi-differential invariants are constructed (using moving frames) near these coalescent boundaries.

72 Statistical Sampling Idea: Replace high dimensional joint invariant signatures by increasingly dense point clouds obtained by multiply sampling the original submanifold. The equivalence problem requires direct comparison of signature point clouds. Continuous symmetry detection relies on determining the underlying dimension of the signature point clouds. Discrete symmetry detection relies on determining densities of the signature point clouds.

73 Invariant Histograms To eliminate noise, use histograms based on joint invariants. Definition. The distance histogram of a finite set of points P = {z 1,...,z n } V is the function η P (r) =# { (i, j) 1 i<j n, d(zi,z j )=r }. Brinkman, D., & PJO, Invariant histograms, Amer. Math. Monthly 118 (2011) 2 24.

74 The Distance Set The support of the histogram function, supp η P = P R + is the distance set of P. Erdös distinct distances conjecture (1946): If P R m,then# P c m,ε (# P ) 2/m ε

75 Characterization of Point Sets Note: If P = g P is obtained from P R m by a rigid motion g E(n), then they have the same distance histogram: η P = η P. Question: Can one uniquely characterize, up to rigid motion, a set of points P {z 1,...,z n } R m by its distance histogram? = Tinkertoy problem.

76 Yes: η = 1, 1, 1, 1, 2, 2.

77 No: Kite Trapezoid η = 2, 2, 2, 10, 10, 4.

78 No: P = {0, 1, 4, 10, 12, 17} Q = {0, 1, 8, 11, 13, 17} R η = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17 = G. Bloom, J. Comb. Theory, Ser. A 22 (1977)

79 Theorem. (Boutin Kemper) Supposen 3orn m +2. Then there is a Zariski dense open subset in the space of n point configurations in R m that are uniquely characterized, up to rigid motion, by their distance histograms. = M. Boutin, G. Kemper, Adv. Appl. Math. 32 (2004)

80 Distinguishing Melanomas from Moles Melanoma Mole = A. Rodriguez, J. Stangl, C. Shakiban

81 Cumulative Global Histograms Red: melanoma Green: mole

82 Logistic Function Fitting Melanoma Mole

83 Logistic Function Fitting Residuals Melanoma = ± Mole = ± % Confidence

84 Limiting Curve Histogram

85 Limiting Curve Histogram

86 Limiting Curve Histogram

87 Sample Point Histograms Cumulative distance histogram: n =#P : Λ P (r) = 1 n + 2 n 2 s r η P (s) = 1 n 2 # { (i, j) d(zi,z j ) r }, Note η(r) = 1 2 n2 [Λ P (r) Λ P (r δ)] δ 1. Local distance histogram: λ P (r, z) = 1 n # { j d(z,zj ) r } = 1 n #(P B r (z)) Ball of radius r centered at z: B r (z) ={ v V d(v, z) r } Note: Λ P (r) = 1 n z P λ P (r, z) = 1 n 2 z P #(P B r (z)).

88 Limiting Curve Histogram Functions Length of a curve l(c) = C ds < Local curve distance histogram function h C (r, z) = l(c B r (z)) l(c) z V = The fraction of the curve contained in the ball of radius r centered at z. Global curve distance histogram function: H C (r) = 1 l(c) C h C (r, z(s)) ds.

89 Convergence Theorem. Let C be a regular plane curve. Then, for both uniformly spaced and randomly chosen sample points P C, the cumulative local and global histograms converge to their continuous counterparts: λ P (r, z) h C (r, z), Λ P (r) H C (r), as the number of sample points goes to infinity.

90 Square Curve Histogram with Bounds

91 Kite and Trapezoid Curve Histograms

92 Histogram Based Shape Recognition 500 sample points Shape (a) (b) (c) (d) (e) (f) (a) triangle (b) square (c) circle (d) 2 3rectangle (e) 1 3rectangle (f) star

93 Curve Histogram Conjecture Two sufficiently regular plane curves C and C have identical global distance histogram functions, so H C (r) =H C(r) for all r 0, if and only if they are rigidly equivalent: C C.

94 Proof Strategies Show that any polygon obtained from (densely) discretizing a curve does not lie in the Boutin Kemper exceptional set. Polygons with obtuse angles: taking r small, one can recover (i) the set of angles and (ii) the shortest side length from H C (r). Further increasing r leads to further geometric information about the polygon... Expand H C (r) inataylorseriesatr =0andshowthatthe corresponding integral invariants characterize the curve.

95 Taylor Expansions Local distance histogram function: Lh C (r, z) =2r κ2 r 3 + ( 1 40 κκ ss κ2 s κ4 ) r 5 +. Global distance histogram function: H C (r) = 2r L + r3 12L 2 C κ2 ds + r5 40L 2 C ( 3 8 κ4 1 9 κ2 s ) ds +.

96 Space Curves Saddle curve: z(t) =(cost, sin t, cos 2t), 0 t 2π. Convergence of global curve distance histogram function:

97 Surfaces Local and global surface distance histogram functions: h S (r, z) = area (S B r (z)) area (S) Convergence for sphere:, H S (r) = 1 area (S) h S (r, z) ds. S

98 Area Histograms Rewrite global curve distance histogram function: H C (r) = 1 L h C (r, z(s)) ds = 1 C L 2 χ r C C (d(z(s),z(s )) ds ds { 1, t r, where χ r (t) = 0, t > r, Global curve area histogram function A C (r) = 1 L χ 3 r (area C C C (z(ŝ),z(ŝ ),z(ŝ )) dŝdŝ dŝ, dŝ equi-affine arc length element L = Discrete cumulative area histogram 1 A P (r) = n(n 1)(n 2) z z z P χ r (area (z,z,z )), Boutin & Kemper: the area histogram uniquely determines generic point sets P R 2 up to equi-affine motion C dŝ

99 Area Histogram for Circle Joint invariant histograms convergence???

100 Triangle Distance Histograms Z =(...z i...) M sample points on a subset M R n (curve, surface, etc.) T i,j,k triangle with vertices z i,z j,z k. Side lengths: σ(t i,j,k )=(d(z i,z j ),d(z i,z k ),d(z j,z k )) Discrete triangle histogram: Triangle inequality cone S = σ(t ) K K = { (x, y, z) x, y, z 0, x+ y z, x + z y, y + z x } R 3.

101 Triangle Histogram Distributions Circle Triangle Square = Madeleine Kotzagiannidis

102 Practical Object Recognition Scale-invariant feature transform (SIFT) (Lowe) Shape contexts (Belongie Malik Puzicha) Integral invariants (Krim, Kogan, Yezzi, Pottman,...) Shape distributions (Osada Funkhouser Chazelle Dobkin) Surfaces: distances, angles, areas, volumes, etc. Gromov Hausdorff and Gromov-Wasserstein distances (Mémoli) = lower bounds