RECONSTRUCTION AND PATTERN RECOGNITION VIA THE CITTI- PETITOT-SARTI MODEL DARIO PRANDI, J.P. GAUTHIER, U. BOSCAIN LSIS, UNIVERSITÉ DE TOULON & ÉCOLE POLYTECHNIQUE, PARIS SÉMINAIRE STATISTIQUE ET IMAGERIE 19 JANUARY 2014 0
OUTLINE OF THE TALK: 1. 2. 3. The Citti-Petitot-Sarti model of the primary visual cortex Image Inpainting Image recognition
THE CITTI-PETITOT-SARTI MODEL OF THE PRIMARY VISUAL CORTEX
THE MATHEMATICAL MODEL ORIGIN OF THE MODEL: Hoffman (1989): structure of a contact manifold Petitot (1999): structure of a sub-riemannian manifold (Heisenberg group) THEN REFINED BY: Citti, Sarti (2006): structure of the rototranslations of the plane Agrachev, Boscain, Charlot, Gauthier, Rossi, D.P. (2010 ) ALSO STUDIED BY: Sachkov (2010-) Duits (2009 )
NEUROPHYSIOLOGICAL FACT STRUCTURE OF THE PRIMARY VISUAL CORTEX Hubel and Wiesel (Nobel prize 1981) observed that in the primary visual cortex V1, groups of neurons are sensitive to both positions and directions. Patterns of orientation columns and long-range horizontal connections in V1 of a tree shrew. Taken from Kaschube et al. (2008).
CITTI-PETITOT-SARTI MODEL V1 is modeled as the projective tangent bundle:
We need to discuss RECEPTIVE FIELDS: How a greyscale visual stimulus. is lifted to an state on SPONTANEOUS EVOLUTION: How a state on evolves via neuronal connections.
RECEPTIVE FIELDS Receptive fields are aimed to describe a family of neurons with a family of functions on the image plane. in It is widely accepted that a greyscale visual stimulus feeds a V1 neuron with an extracellular voltage given by A good fit for is the Gabor filter (sinusoidal wave multiplied by Gaussian function) centered at of orientation
GROUP THEORETICAL APPROACH Consider with group operation where is the rotation of. is the double covering of, so with appropriate care we can work here The left regular (unitary) representation of acting on is The quasi-regular (unitary) representation of acting on is
Definition: A lift operator is left-invariant if Theorem: (J.-P. Gauthier, D.P.) Let be a left-invariant lift such that is linear, is densely defined and bounded. Then, there exists such that REMARKS V1 receptive fields through Gabor filters define a left-invariant lift. The function is known as the wavelet transform of w.r.t..
SPONTANEOUS EVOLUTION IN V1 Two types of connections between neurons: Lateral connections: Between iso-oriented neurons, in the direction of their orientation. Represented by the integral lines of the vector field Local connections: Between neurons in the same hypercolumn. Represented by the integral lines of the vector field Connections made by V1 cells of a tree shew. Picture from Bosking et al. (1997)
EVOLUTION IN THE CITTI-PETITOT-SARTI MODEL Given two independent Wiener process and on, neuron excitation evolve according to the SDE Its generator equation for the evolution of a stimulus : yields to the hypoelliptic Highly anisotropic evolution Natural sub-riemannian interpretation Invariant under the action of
SEMI-DISCRETE MODEL Conjecture: The visual cortex can detect a finite (small) number of directions only ( 30). This suggests to replace with and with. LOCAL CONNECTIONS We replace by the jump Markov process on defined as follows. The time of the first jump is exponentially distributed, with probability on either side. We obtain a Poisson process with The infinitesimal generator is
THE EVOLUTION OPERATOR The semi-discrete evolution operator, acting on is then where, for, ADVANTAGES Already naturally discretized the variable. Invariant w.r.t. semi-discretized rototranslations.
IMAGE INPAINTING
ILLUSORY CONTOURS Two illusory contours Neurophysiological assumption: We reconstruct images through the natural diffusion in V1, which minimizes the energy required to activate unexcitated neurons.
ANTHROPOMORPHIC IMAGE RECONSTRUCTION Let with on be an image corrupted on. 0. Smooth by a Gaussian filter to get generically a Morse function 1. Lift to on 2. Evolve through the CPS semi-discrete evolution 3. Project the evolved back to REMARKS Reasonable results for small corruptions Adding some heuristic procedures yield very good reconstructions
SMOOTHING Even if images are not described by Morse functions, the retina smooths images through a Gaussian filter (Peichl & Wässle (1979), Marr & Hildreth (1980)) Theorem (Boscain, Duplex, Gauthier, Rossi 2012) The convolution of with a two dimensional Gaussian centered at the origin is generically a Morse function. Level lines of Morse functions
SIMPLE LIFT For simplicity, instead than the lift through Gabor filters we chose to lift to
EXAMPLE: LIFTING A CURVE In the continuous model a curve in is lifted to curve in, where The lift of
A REMARKABLE FEATURE When is a Morse function, the lift Lf is supported on a 2D manifold. (This is false if the angles are not projectivized!)
SEMI-DISCRETE EVOLUTION The hypoelliptic heat kernel of the semi-discretized operator has been explicitly computed, but is impractical from the numerical point of view. We use the following scheme: 1. For any compute, the Fourier transform of. 2. For any we let be the solution of the decoupled ODE on 3. For any the solution is the inverse Fourier transform of This scheme can be implemented by discretizing.
A VIEW THROUGH ALMOST-PERIODIC FUNCTIONS To avoid discretizing we can proceed as follows: Let be the grid of pixels of the image and be the pixel values. 1. Compute, the discrete Fourier transform (FFT) of on 2. Represent as the almost-periodic function 3. Evolve splitting the evolution in the ODE s, for each, The evolution of is exact!
PROJECTION Given the result of the evolution image by, we define the reconstructed
RESULTS Inpainting results from Boscain et al. (2014)
MUMFORD ELASTICA MODEL With the same techniques it is possible to treat the evolution equation underlying the Mumford Elastica model, which is associated with the operator Comparison of a reconstruction with the Petitot diffusion and the Mumfor diffusion from Boscain, Gauthier, P., Remizov (2014)
HIGHLY CORRUPTED IMAGES RECONSTRUCTION Based upon this diffusion and certain heuristic complements, we get nice results on images with more than 85% of pixels missing. Image from Boscain, Gauthier, P., Remizov (preprint)
IMAGE RECOGNITION
SPECTRAL INVARIANTS Given a group and a representation of on some topological space, a complete set of invariants is a map, from to some functional space such that If the above holds only for and in some residual subset of, the s are said to be a weakly complete set of invariants. We speak of spectral invariants whenever the transform of. s depend on the Fourier IMPORTANT CASES 1. Since any group acts on itself, we look for invariants for the action of the left regular representation of on 2. Given a semidirect product, we look for invariants for the quasi-regular representation of on.
PLAN 1. Case of an abelian group (e.g. invariants on w.r.t. translations) Fourier transform and Pontryagin duality Bispectral invariants 2. Case of (non-compact, non-abelian semi-direct product) Generalized Fourier transform and Chu duality Bispectral invariants for Invariants for lifts of functions in w.r.t. semidiscretized rototranslations.
FOURIER TRANSFORM ON ABELIAN GROUPS Let be a locally compact abelian group with Haar measure. The dual of is the set of of characters of, i.e., of continuous group homomorphism,. The Fourier transform of is the map on defined by Since is abelian and locally compact, the Fourier transform extends to an isometry w.r.t. the Haar measure on. Fundamental property: For any,
EXAMPLE: It holds, where for the corresponding element of is. The above defined Fourier transform then reduces to Then, REMARK In this case, it is clear that is indeed a group and that. This is an instance of Pontryagin duality, which works on all abelian groups: Theorem: (Pontryagin duality) The dual of is canonically isomorphic to. More precisely, defined as is a group isomorphism.
INVARIANTS FOR ABELIAN GROUPS Let be a locally compact abelian group and consider the action on of the left-regular representation (e.g. for this corresponds to translations). The power spectrum invariants for w.r.t. the action of the left regular representation are the functions These are widely used (e.g. in astronomy), but are not complete: Fix any s.t. which is not a character of, Let, so that, However, for any, since Indeed, if it was the case equivalent to being a character of. which, by Pontryagin duality, is
What is missing in the power spectrum invariants is the phase information. The bispectral invariants for regular representation are the functions w.r.t. the action of the left Theorem: The bispectral invariants are weakly complete on, where is compact. In particular, they discriminate on the residual set of those square-integrable s such that on an open-dense subset of. REMARKS Note that allows to recover the power spectral invariants The bispectral invariants are used in several areas of signal processing (e.g. to identify music timbre and texture, Dubnov et al. (1997))
PROOF OF WEAK COMPLETENESS Let be compactly supported and such that. Define, which is a continuous function on an open and dense set of satisfying. Since the bispectral invariants coincide it holds Since are compactly supported, and are continuous and hence can be extended to a measurable function on, still satisfying the above. Since every measurable character is continuous, this shows that hence. That is, there exists such that, and
GENERALIZED FOURIER TRANSFORM Let is a locally compact unimodular group with Haar measure not necessarily abelian. The dual of is the set of equivalence classes of unitary irreducible representations of. The (generalized) Fourier transform of is the map that to acting on the Hilbert space associates the Hilbert- Schmidt operator on defined by There exists a measure (the Plancherel measure) on w.r.t. the Fourier transform can be extended to an isometry. Fundamental property: For any,
CHU DUALITY Chu duality is an extension of the dualities of Pontryagin (for abelian groups) and Tannaka (for compact groups) to certain (non-compact) MAP groups. Here the difficulty is to find a suitable notion of bidual, carrying a group structure. See Heyer (1973). Let be a topological group is the set of all -dimensional continuous unitary representations of in. It is endowed with the compact-open topology. The Chu dual of is the topological sum is second countable if is so.
QUASI-REPRESENTATIONS A quasi-representation of is a continuous map from to such that for any and 1. 2. 3. 4. The Chu quasi-dual of is the union of all quasirepresentations of endowed with the compact-open topology. Setting and, is a Hausdorff topological group with identity. The mapping defined by is a continuous homomorphis, injective if the group is MAP. Definition: The group has the Chu duality property if is a topological isomorphism.
CHU DUAL A Moore group is a group whose irreducible representations are all finitedimensional. Theorem (Chu) The following inclusions hold The group is Moore (i.e. all its irreducible representations are finite-dimensional) and then it has Chu duality. The group is not MAP (i.e. almost-periodic functions do not form a dense subspace of continuous functions) and hence it does not have Cuu duality. This is why it is more convenient to work in the semi-discretized model
THE CASE OF Let us consider the (non-compact, non-abelian) Moore group. The unitary irreducible representations fall into two classes Characters: Any. -dimensional representations: For any representation that acts on as induces the one-dimensional representation we have the where is the shift operator. Since the Plancherel measure is supported on the -dimensional representations, bispectral invariants are generalized to as the following functions of :
THE LEFT REGULAR REPRESENTATION Theorem: The bispectral invariants are weakly complete w.r.t. the action of on, where is compact. In particular, they discriminate on the residual set of those square-integrable s such that is invertible for in an open-dense subset of. The proof is similar to the abelian one: Given with : Let for s.t. is invertible. Prove that can be extended to a quasi-representation Since is Moore, it has Chu duality and hence there exists s.t. for any unitary representation Finally, this implies
THE QUASI-REGULAR REPRESENTATION Consider now the quasi-regular representation acting on, which corresponds to rotation and translations. Fixed a lift invariants as. we define the bispectral Corollary: Let be an injective leftinvariant lift. Then, for any compact the bispectral invariants are complete for the action of on the subset of such that is invertible for in an open-dense subset of. Proof: Let be such that. Then
Unfortunately, the set is empty for regular left-invariant lifts: Let be the vector. Since we have Thus has at most rank and hence. Conjecture: The bispectral invariants are weakly complete w.r.t. the action of on compactly supported functions of. REMARK There exists non left-invariant lifts for which lift is residual, as the cyclic The price to pay is that we have to quotient away the translations before the lift. This suggest the following.
ROTATIONAL BISPECTRAL INVARIANTS The rotational bispectral invariants for and any the quantities are, for any Observe that is invariant under rotations but not under translations. A function is weakly cyclic if is a basis of for a.e.. Modifying the arguments used in the case of the left-regular representation, we can then prove the following. Theorem: Consider a regular left-invariant lift with weakly cyclic and such that a.e.. Then, the rotational bispectral invariants are weakly complete w.r.t. the action of rotations on for any. More precisely, they discriminate on weakly cyclic functions.
EXPERIMENTAL RESULTS In Smach et al. (2008), although the theory was not complete, some tests on standard academic databases have been carried out. They yielded results superior to standard strategies. ZM denotes the standard Zernike moments MD are the (non-complete) power spectrum invariants MD are the bispectral invariants Sample objects from and results obtained on the COIL-100 noisy database (from Smach et al.).
OTHER EXPERIMENTAL RESULTS Sample objects from and results obtained on the faces ORL database (from Smach et al.).
TEXTURE RECOGNITION Let be countable and invariant under the action of. A natural model for texture discrimination are almost periodic functions on in the Besicovitch class, i.e., functions are the pull-back of functions on the Bohr compactification. The theory above can be adapted to these spaces of functions, and an analog of the weakly completeness of rotational bispectral invariants holds. In this space the bispectral invariants are not complete, and thus the (analog of) the above conjecture is false. As already mentioned, when is finite this space can be used to exacty solve the hypoelliptic diffusion.
REMARK: SETTING The considerations of this part of the talk work in the general context of a semi-direct product where is an abelian locally compact group is a finite group the Haar measure on is invariant under the action, the natural action of on the dual has non-trivial stabilizer only w.r.t. the identity. The group law on is non-commutative: In our case:, and is the rotation of.
THANK YOU FOR YOUR ATTENTION