Toward a comprehensive framework for the spatiotemporal statistical analysis of longitudinal shape data

Transcription

1 IJCV manscript No. (will be inserted by the editor) Toward a comprehensive framework for the spatiotemporal statistical analysis of longitdinal shape data S. Drrleman X. Pennec A. Trové J. Braga G. Gerig N. Ayache Received: date / Accepted: date Abstract This paper proposes an original approach for the statistical analysis of longitdinal shape data. The proposed method allows the characterization of typical growth patterns and sbject-specific shape changes in repeated time-series observations of several sbjects. This can be seen as the extension of sal longitdinal statistics of scalar measrements to high-dimensional shape or image data. The method is based on the estimation of continos sbject-specific growth trajectories and the comparison of sch temporal shape changes across sbjects. Differences between growth trajectories are decomposed into morphological deformations, which accont for shape changes independent of the time, and time warps, which accont for different rates of shape changes over time. Given a longitdinal shape data set, we estimate a mean growth scenario representative of the poplation, and the variations of this scenario both in terms of shape changes and in terms of change in growth speed. Then, intrinsic statistics are derived in the space of spatiotemporal deformations, which characterize the typical variations in shape and in growth Stanley Drrleman Gido Gerig Scientific Compting and Imaging (SCI) Institte, 72 S. Central Drive, Salt Lake City, UT 84112, USA stanley@sci.tah.ed (Stanley Drrleman) Phone: , Fax: Stanley Drrleman Xavier Pennec Nicholas Ayache Asclepios team-project, INRIA Sophia Antipolis, 24 rote des Lcioles, 692 Sophia Antipolis, France Stanley Drrleman Alain Trové Centre de Mathématiqes et Lers Applications (CMLA), CNRS-ENS Cachan, 61 avene d Président Wilson, Cachan, France José Braga Laboratoire de paléoanthropologie assistée par ordinater, CNRS- Université de Tolose (Pal Sabatier), 37 allées Jles Gesde, 3773 Tolose, France speed within the stdied poplation. They can be sed to detect systematic developmental delays across sbjects. In the context of neroscience, we apply this method to analyze the differences in the growth of the hippocamps in children diagnosed with atism, developmental delays and in controls. Reslt sggest that grop differences may be better characterized by a different speed of matration rather than shape differences at a given age. In the context of anthropology, we assess the differences in the typical growth of the endocranim between chimpanzees and bonobos. We take advantage of this stdy to show the robstness of the method with respect to change of parameters and pertrbation of the age estimates. Keywords longitdinal data statistics growth shape regression spatiotemporal registration time warp 1 Spatiotemporal variability of longitdinal data Many scientific qestions can be expressed in terms of changes or alterations of a dynamical process. In camera srveillance, one aims at distingishing normal from abnormal behaviors in video seqences. In clinical stdies, one wants to characterize anatomical or fnctional changes de to disease progression, clinical intervention or therapy. In neroscience, one stdies the nerodevelopment or the nerodegeneration of the brain and its related strctres. In cardiac imaging, one looks for abnormal patterns in the heart motion. What make these qestions so challenging is that the evolving object of interest changes in appearance in different sitations. In video seqences for instance, we want to distingish a normal from an abnormal behavior behind the large variety of the shapes and the motions of the silhoettes. Similarly, every brain has a different shape, whereas its matration may follow some common patterns that we wold like precisely to describe and qantify.

2 2 From the point of view of data analysis and pattern theory, these problems can be addressed by the statistical analysis of longitdinal data sets. A longitdinal data set consists of the observation of a set of homologos objects (sch as silhoettes of people or anatomical strctres), each object being observed repeatedly at several time points. An abstract example of sch a data set is given in Fig. 1, which illstrates the sampling of individal growth trajectories of different sbjects. The analysis of sch longitdinal data sets shold lead to the qalitative and qantitative assessment of change trajectories, to the detection of common growth patterns shared in a poplation, and to the characterization of their appearances in different sbjects. Longitdinal analysis differs from the sal cross-sectional variability analysis in that it takes into accont the inherent correlation of repeated measrements of the same individals. It mst also provide a model of how an individal sbject s trajectory changes relative to another sbject. At the poplation level, we typically analyze how the sbjects are distribted within a grop by estimating a mean configration and its variance. For longitdinal data, the mean configration may be a mean growth scenario, which averages the growth patterns in the poplation. The analysis of its variance explains how each sbject s trajectory differs from the mean growth scenario. Sch a statistical approach based on mean and variance is well-known for scalar measrements and for analysis of cross-sectional shape data, for which the mean is sally called template or atlas. The extension of these concepts for longitdinal shape data is challenging, as no consenss has emerged abot how to combine shape changes over time and shape changes across sbjects. In this paper, we propose a consistent conceptal and comptational framework to address these qestions: (i) the estimation of sbject-specific trajectories via the introdction of a growth model as a smooth deformation of the baseline shape, (ii) the comparison of different trajectories via spatiotemporal mappings which align both the shape of different sbjects and the tempo of their respective evoltion, (iii) the estimation of a mean growth scenario representative of a given poplation, and (iv) the statistical analysis of the typical variations of this mean scenario in the stdied poplation. The proposed methodology does not reqire that the sbjects are observed with the same nmber of samples or at the same time-points. One of the main contribtions of this methodology is that it models the changes in individal trajectories both as morphological changes, which accont for the different appearances of the object, and as dynamical changes, which accont for different paces of evoltion. At the poplation level, this assmes that the development of different sbjects shares the same growth patterns, p to changes in shape and changes in the tempo of the development. This enables in particlar the characterization of the effect of a pathology as a systematic developmental delays in the growth of a given organ. The detailed explanation of the method and its related algorithms is given in Sec. 3. Sec. 2 explains how the proposed framework consistently embeds different concepts introdced in the literatre and highlights different possible modeling choices. In Sec. 4, we show how the method can be sed to characterize the effect of atism and developmental delay in the growth speed of the hippocamps. In Sec. 5, the method will be sed to qantitatively assess the relative developmental delay of the endocranial growth between bonobos and chimpanzees. We will show that this estimation is robst to parameter changes and changes in the age estimates of the samples. Fig. 1 Synthetic example of a longitdinal data set with 3 sbjects. Each sbject has been observed a few times and at different time-points. The aim of the spatiotemporal variability analysis is to describe the variability of this poplation in two ways: the geometrical variability (there is a circle, a sqare and a triangle), and the variability in terms of change of dynamics of evoltion (for instance, the sqare grows first at a faster rate than the circle and then slows down.) 2 An emerging framework for the analysis of longitdinal shape data This section presents a srvey of the literatre on the topic of longitdinal analysis of shape data. This will highlight which tools and concepts need to be linked into a common statistical framework. We will also make clear that different modeling choices are possible. We will propose to follow the approach that seems the more adapted to the targeted applications.

3 3 2.1 Previos research to design 4D statistical analysis D analysis meant as regression or tracking The first kind of so-called 4D-analysis proposes to estimate a continos seqence from a set of time-indexed shapes or images of the same sbject. In Mansi et al (29), one estimates a cross-sectional atlas from time series data and then analyzes the correlations between the modes of variability and the age of the sbject, considered as an explanatory variable. These correlations may be sed to estimate a synthetic growth scenario for a given individal. Other approaches, which do not rely on a cross-sectional atlas, inclde work by de Craene et al (29), in which the athors se Large Diffeomorphic Free Form Deformations to estimate time-varying deformations between the first and the last sample of a seqence of images. In the same spirit, Davis et al (27) propose to perform the regression of a seqence of images via a generalization of the kernel regression method to Riemannian manifolds. Growth scenarios cold also be estimated based on stochastic growth models as in Grenander et al (27); Trové and Vialard (21) or on twice differentiable flows of deformations as in Fishbagh et al (211). These methods are pre regression methods. If they are sed with several sbjects scanned several times, these regression methods retrn a single evoltion, the most probable evoltion in some sense. They do not take into accont that data at different time points may come from the same sbject or from different sbjects. It averages shape evoltions withot discarding the inter-sbject variability, which leads to fzzy estimation like the average of a set of non-registered images. By contrast, in Thompson et al (2); Gogtay et al (28), registrations between baseline and follow-p scans of the same sbject are performed and the evoltions of scalar measrements extracted from the registration are compared across sbjects. A main contribtion of or paper will be to extend this framework for scalar measrements to the high-dimensional space of shapes. In Khan and Beg (28), the athors propose to perform a regression of the image seqence of every sbject separately and then to average the time dependent velocity field of each regression to estimate a typical scenario of evoltion. This approach is limited to sitations where each seqence is registered in the same reference frame, bt no details of how to perform registration of time-indexed seqences of images is given D-registration of 4D-seqences The problem of registering individal trajectories has been investigated in different commnities. In Chandrashekara et al (23), the motion of the heart of each sbject is tracked throgh time. Then the registration between the baseline image of two sbjects is sed to transport the velocity field of the tracking from one sbject s space to the other. This approach cold also inclde the estimation of a template image at the baseline time-point sing sal cross-sectional atlas constrction methods, like in Ehrhardt et al (28); Qi et al (28, 29). All these methods assme that the inter-sbject variability can be captred considering only the baseline images. Using these deformations for registering the whole time-indexed seqence of images is argable, since they do not take into accont anatomical featres which may appear later in the seqence. This isse has been addressed in Peyrat et al (28) who proposed to register a time-indexed seqence by compting deformations between any pair of sccessive scans of the same sbject and between any pair of scans of two different sbjects at the same time-point. Sch an approach takes all temporal information into accont and therefore leads to a mch more robst registration scheme. However, this method assmes that every time-indexed seqence has exactly the same nmber of images, which are acqired at time points which correspond across sbjects. By contrast, in longitdinal stdies, only a few scans per sbject are available, and the nmber of scans may vary for different sbjects. This isse has been addressed in Hart et al (21), who proposed an interpolation scheme to average individal trajectories at every time-point independently. However, none of these approaches take into accont the inherent temporal correlations between sccessive inter-sbject registration. From a statistical point of view, this means that the inter-sbject variability at two different time points are considered as independent variables. As the sampling of the image seqence becomes finer and finer, the nmber of variables to estimate becomes larger and larger. A main contribtion of this paper will be precisely to define a generative statistical model, which takes into accont the temporal correlations between inter-sbject registrations at different time points, and to provide a way to estimate these correlations from a finite set of observations. In Gerig et al (26), a template image is bilt at every time-point independently. Then, the deformations between the baseline scan and the follow-p of any sbjects are compared to the deformation between the baseline atlas and the follow-p atlas. This approach focses on the analysis of the cross-sectional variability over time. However, since the template image is bilt at each time point independently, it is not clear whether the difference between the baseline atlas and the follow-p atlas is the average of every sbject s evoltion. Moreover, sch a method reqires that the distribtion of age in the longitdinal data set is clstered at two distinct ages, which is a special case. A similar approach has also been proposed in Aljabar et al (28).

4 Taking into accont temporal re-alignment The methods cited previosly propose a way to combine the sbject-specific growth with the inter-sbject variability: time-series image seqence are processed by a combination of 3D deformations. In particlar, the age at which the sbjects are scanned is considered an absolte time which corresponds across sbjects. This assmes that at a given age, every sbject is at the same development stage and that their anatomy can be compared. Sch procedres neglect possible developmental delays between sbjects, or some pathology affecting the cardiac pace, for instance, a key featre that we precisely want to detect. A spatiotemporal registration scheme shold register individal growth scenarios both in space (sal geometrical variations of the anatomy) and in time (change of the speed of evoltion). Time changes shold pt the ages of the sbjects into correspondence, which represents the same developmental stage. In Declerck et al (1998), a deformation of the 4D domain is provided via 4D planispheric transformations for the registration of the heart motion. In Perperidis et al (25), spatiotemporal deformations are compted. The temporal part is a 1D fnction showing the change of cardiac dynamics between the sorce and the target sbject. This temporal alignment is performed jointly with the registration of the anatomy. These methods focs on the registration between a pair of individal trajectories, and reqires a fine temporal sampling of the trajectories. A main contribtion of or paper will be to se sch spatiotemporal deformations for the inference of statistical properties at the poplation level, via the estimation of spatiotemporal atlases Ingredients for a spatiotemporal statistical model This review of the literatre shows that several aspects of the design of a 4D statistical analysis have been addressed separately by different athors, in different contexts and with different tools. There is a lack of a consistent framework to embed these concepts together, covering the estimation of individal trajectories and the inference of poplation statistics. In light of this review, a statistical framework for longitdinal data analysis might inclde: The estimation of a continos shape evoltion from a set of observations sparsely distribted in time. These individal trajectories cold be sed to compare the anatomy of two sbjects, who have not been scanned at the same age. They cold also be sed to analyze the speed of evoltion of a given sbject at any time-point. The comparison between individal trajectories, which shold measre not only morphological differences (commonly described by 3D deformations) bt also the temporal re-alignment which pt the developmental stages of different sbject into correspondence. This temporal re-alignment will detect different speeds of evoltion and therefore possible developmental delays between sbjects. A generative statistical model, which combines the two previos concepts to estimate evoltion patterns that are shared among a given poplation. The estimated statistics shold inclde a mean (a growth scenario representative of the poplation) and variance (the typical variations of this mean growth scenario evident in the poplation) Terminology The srvey of the crrent literatre also raises the problem of terminology: there is no consenss among athors abot which words refer to which concepts. In this paper, we will se the following definitions: Data: cross-sectional data is a set of samples, which are spposed to be comparable, or homologos (like samples drawn from a healthy adlt poplation, for instance). No notion of time is involved, or eqivalently, the effect of time or age on the data can be neglected. time-series data is a set of data that are indexed by any temporal marker like age, indicator of developmental stage, disease progression or index of a frame in a movie, for instance. No assmption is made that a sb-set of the samples correspond to the same object seen at different time-points. longitdinal data is a time-series data set, which contains repeated observations of individal sbjects over a period of time. As a conseqence, each sbject in the data set shold have been observed more than once at different time-points. Methods: shape/image regression, also called tracking, refers to the estimation of a continos evoltion model from a time-series data set. This tool estimates shape changes between discrete temporal observations or averages timeindexed observations into a single evoltion. spatiotemporal registration pts two individal trajectories into correspondence. This involves the notion of correspondence between shapes and between time-points. spatiotemporal or longitdinal data analysis measres the similarities and the differences between individal trajectories. It takes into accont the fact that individal sbjects were observed several times, which makes it more constrained than the analysis of the effect of time on the observations. According to these definitions, shape or image regression may be performed on time-series data, whereas spatiotemporal analysis can only be performed on longitdinal data.

5 5 2.2 Two possible generative models for longitdinal data Spatiotemporal variations of a typical growth model A generative statistical model is a set of hypotheses, which explain how individal trajectories cold be derived one from the others. In other words, it shold provide an answer to the two fndamental qestions: given the anatomy of one sbject at time t, how can we predict the anatomy of this sbject at a later time t > t? how can we derive the typical anatomy of another sbject at the same time-point? Once these answers are provided, we can easily define a generative statistical model at the poplation level. This model will assme the existence of a mean growth scenario representative of the poplation, sch that the individal trajectories can be seen as a derivation of this mean scenario. The mean scenario captres the invariants in the poplation and detects the growth patterns, which are shared among the sbjects. The derivation of the mean scenario captres the variance of this mean configration within the poplation. In light of the literatre srvey, there are at least two different ways to answer these qestions. We refer to these two paradigms as a sbject-specific approach and a timespecific approach Sbject-specific approach In the sbject-specific approach, a specific reference frame is attached to each sbject. The whole evoltion of each sbject is described within the same reference frame: the reference frames are atemporal. We assme that there is a template reference frame in which the evoltion is written by a time-varying shape M(t): the prototype scenario of evoltion in the poplation, which can be seen as the 4D analog to the template shape in 3D. We sally assme that the timevarying shape derives continosly from a template shape M at a reference time point t. This is written as: M(t) = χ t (M ), where χ t is a smoothly varying 3D deformation called the growth fnction (χ t = id so that M(t ) = M ). Change of coordinates from the template reference frame to each sbject s reference frame is modeled by 3D deformations φ s. Since these reference frames are atemporal, the deformations φ s do not depend on time. As a conseqence, the evoltion fnction M(t) has a different expression in each coordinate system: the evoltion of a sbject S is given as S(t) = φ(m(t)), also written as S(t) = φ(χ t (M )). It is as if a single object (M(t)) is seen by different observers in different coordinate systems. As illstrated in Fig. 2a, the change of coordinates φ s transports the evoltion fnction χ(t) from the template frame to the sbject s frame: χ S t = φ s χ t, so that S(t) = χ s t (M ). The evoltion mapping is therefore specific to each sbject. In this modeling, we can inclde time as an additional variable, so that the reference frame of each sbject is described by 3 spatial coordinates and 1 temporal coordinate. This means that both the anatomy and the age is relative to the sbject. This specific time variable can be called the physiological age of the sbject, as if each sbject has their own biological clock. In the reference frame of the prototype, the time wold be the absolte age, compted from the date of birth. Then, the 3D warp φ(x,y,z) needs to be generalized to a deformation of the nderlying 4D space: Φ(x,y,z,t). The most general form of a 4D-deformation is Φ(x,y,z,t) = (φ(x,y,z,t),ψ(x,y,z,t)), where φ(x,y,z,t) denotes the 3 spatial coordinates of Φ(x,y,z,t) (the morphological deformation) and ψ(x,y,z,t) its temporal coordinate (the time warp). Assming that ψ(x,y,z,t) depends on the spatial variables (x,y,z) means that different parts of the anatomy of a given sbject wold evolve at different speeds. This is definitely possible in applications involving mlti-shape comparisons. However, in this paper, we will assme that all points of the anatomy of a given sbject have always the same physiological age over time. In this case, ψ depends only on the time variable t: ψ(t). This assmption is likely to be valid in most longitdinal stdies, focsing on one specific strctre. The time warp ψ(t) maps the absolte age in the reference frame of the prototype to the physiological age of a given sbject. Note that this fnction shold be monotonic, assming that the seqence of events in every individal trajectory occr in the same order (from birth onwards) bt at a different pace. Since the change of coordinate maps φ are independent of time, they are of the form: φ(x,y,z). Therefore, in the sbject-specific setting, the 4D deformations are written as: Φ(x,y,z,t) = (φ(x,y,z),ψ(t)). The morphological deformation φ is sed to measre the geometrical variability. The time warp ψ is sed to detect possible developmental delays between sbjects. Note that the most general form of 4D deformations, withot any assmptions on the temporal dependency of the spatial part φ(x,y,z,t), cannot be sed in a statistical model. Indeed, sch models will be not identifiable, as there wold be an infinite nmber of different spatial/temporal combinations to explain the same data set Time-specific approach In the time-specific approach, every sbject is embedded into the same reference frame, which transports everyone over time. It is as if different objects are seen by a single observer. More precisely, there is a common reference frame at reference time t = (the origin of the world ) in which the

6 6 a- sbject-specific approach b- time-specific approach Fig. 2 Illstration of the hypotheses nderlying the sbject- and time-specific approaches. In the sbject-specific approach (left), one considers that one sbject is circle and the other is sqare : the difference is described by a single fnction φ, which maps circles to sqares. The evoltion of the first sbject is described by a fnction χ which maps a small circle to big circle. As a conseqence, the evoltion of the second sbject is described by another fnction χ φ which maps a small sqare to a big sqare. In the time-specific approach (right), one describes the evoltion by a niversal fnction χ, which tends to scale the shapes. At the first-time point, the difference between sbjects is described by a fnction φ which maps the small circle to the small sqare. At a later time, the inter-sbject variability has changed according to χ: now the difference between sbjects is described by φ t which maps a big circle to a big sqare. sbject-specific approach time-specific approach Fig. 3 Sbject- verss time-specific approach. In the sbject-specific approach (left) the mean scenario averages the individal trajectories. The inter-sbject variability is spposed to be constant over time. In the time-specific approach (right), every sbject is spposed to follow the same mean scenario of evoltion, p to a change of the initial conditions. The mean scenario describes how the inter-sbject variability evolves over time. anatomy of every sbject is described. The evoltion fnction χ t changes the geometry of this reference frame over time. At each time t, there is one single reference frame which embeds the anatomy of every sbject: this frame is niversal. The same fnction χ t applies for each sbject, so that the evoltion of any sbject is given by S(t) = χ t (S ), where S represents the anatomy of the sbject at the reference time t. In the common reference frame at t =, we assme that each sbject s anatomy S reslts from a deformation of the prototype anatomy M : S = φ s (M ). The deformations φ s describe the inter-sbject variability at time t =. In this framework, the mapping between the template and the sbject shape changes over time according to the evoltion fnction χ t. At a later time t, the template has evolved as M(t) = χ t (M ) and the sbject shape has evolved as S(t) = χ t (S ). This shows that the template-to-sbject registration has become: S(t) = φ t (M(t)) where φ t = χ t φ χt 1, as illstrated in Fig. 2b. Whereas the evoltion fnction is independent of the sbject, the inter-sbject variability is specific to time. We can also inclde possible developmental delays in this framework. If χ t is a niversal fnction which carries the anatomies over time, we can imagine that every sbject follow this niversal scenario at its own pace. There is a sbject specific time warp ψ, so that the evoltion of this sbject is given by χ ψ(t). However, we mst admit that this time-realignment fits less natrally into this time-specific framework than for the sbject-specific framework. In particlar, it is not clear how to distingish a developmental delay from a variation of the inter-sbject variability in this setting. This time-specific approach also defines a deformation of the nderlying 4D-space Φ(x,y,z,t). The morphological deformations φ now depend on time according to the evoltion fnction χ t. This leads to the particlar form of the 4D mapping: Φ(x,y,z,t) = (φ(x,y,z,ψ(t)),ψ(t)),

7 7 where the geometrical part has the form: φ(x,y,z,t) = χ t φ χt 1 (x,y,z). This last eqation is the constraint, which eventally makes the statistical model identifiable Which method for which problem? The sbject-specific approach focses on the variations of a growth scenario from sbjects to sbjects. One is interested in analyzing how individal trajectories vary across sbjects. The time-specific approach focses on the evoltion of the inter-sbject variability over time. One is more interested in the evoltion of the statistical properties (mean and variance) of the poplation over time, as illstrated in Fig. 3. Theses two approaches are based on different assmptions and lead to different statistical estimations. The sbject-specific approach is the only one to take into accont change of coordinates between sbjects, and therefore the only one to accommodate for scaling effects across sbjects. The time-specific paradigm ses a single diffeomorphic deformation to describe the evoltion of every sbject. This assmes that the strctre of two different sbjects, which are sperimposed at one time, will remain sperimposed in the ftre. Sch topological constraints are often nrealistic. Moreover, the statistical estimations in the sbject-specific paradigm are more robst when the nmber of sbjects is greater than the nmber of observations per sbjects, which is the case with the longitdinal data set on which we aim at applying this methodology in Sec. 4 and 5. For these reasons, the presented work will focs on the sbject-specific paradigm. 3 A sbject-specific approach sing 3D diffeomorphisms and 1D time warps In this section, we propose an instance of the sbjectspecific paradigm for the analysis of longitdinal shape data, given as point sets, crves or srfaces. Among several other possible choices, we will bild or methodology on the large diffeomorphic deformations setting for defining the registration between shapes. This setting is particlarly adapted to define statistical models sing deformations de to the metric properties of the considered space of diffeomorphisms (Vaillant et al, 24; Drrleman et al, 29a). In particlar, we will propose an extension of this framework to constrct monotonic 1D fnctions for or time warp in a very generic way. We will also consider the geometrical shape like crves and srfaces as crrents (Glanès, 25). This allows s to inherit from the statistical and comptational tools introdced in Drrleman et al (29a); Drrleman (21) for the estimation of representative shapes, called templates. We follow the approach in three steps otlined in the Introdction: (i) the estimation of individal growth trajectories via the inference of a growth model, (ii) the comparison of individal trajectories based on a morphological map and a time warp, (iii) the estimation of statistics from a set of individal trajectories: mean scenario of evoltion and analysis of the spatiotemporal variability. 3.1 Sketch of the method Growth model for individal shape evoltion Or prpose is to fit a continos shape evoltion to a discrete set of shapes (S i ) of the same sbject acqired at different time points (t i ). To infer sch a continos shape evoltion, we need a prior on the growth of the shape, called a growth model. Here, we hypothesize that the baseline shape S observed at time t = continosly and smoothly deforms over time. To be more precise, or growth model assmes that the evoltion of the shape S can be described by a continos flow of diffeomorphisms χ t. This means that for each t varying in the interval of interest [,T ], χ t is a diffeomorphism of the nderlying 2D or 3D space, which models the smooth and invertible deformation which maps the baseline at t = to its actal shape at time t. The diffeomorphisms vary continosly over time (the deformation χ t+δt is close to χ t ). Mapping the baseline shape S with the time-varying fnctions χ t leads to a continosly deforming shape S(t) = χ t (S ): the individal trajectory of the considered sbject. Note that this imposes that χ = Id, the identity map, so that S() = S. Given the set of discrete observations (S i ) at time-points t i, one needs to estimate the flow of diffeomorphisms χ t, which may have led to these observations. A Maximm A Posteriori (MAP) estimation, in the same framework as in Drrleman (21)[Chap. 5] leads the minimization of the discrepancy between the growth model at time t i (S(t i ) = χ ti (S )) at the actal observation S i, p to a reglarity constraint on the smoothness of the flow of diffeomorphisms (χ t ) t [,T ] : E(χ) = t i d(χ ti (M ),S i ) 2 + γ χ Reg(χ) (1) where d is a similarity measre between shapes, which will be the distance on crrents in the following, Reg(χ) a reglarity term, which will be the total kinetic energy of the deformation, and γ χ a scalar parameter qantifying the tradeoff between reglarity and fidelity to data. The optimization of this criterion will be explained in Section As an illstrative example, we sed five 2D profiles of hominid sklls which consist of six lines each 1, as shown in 1 sorce:

8 8 Fig. 4 Shape regression of a set of five 2D profiles of hominid sklls (in red). The Astralopithecs profile is chosen as the baseline S. The temporal regression comptes a continos flow of shapes S(t) (here in ble) sch that the deforming shape matches the observations at the corresponding time-points. It is estimated by fitting a growth model, which assmes a diffeomorphic correspondence between the baseline and every stage of evoltion (S(t) = χ t (S )), with the diffeomorphism χ t varying continosly in time. Fig. 4. Each profile correspond to a hominid (Astralopithecs, Homo Habilis, Homo Erects, Homo Neandertalensis and Homo sapiens sapiens) and is associated to an age (in millions of years). The regression infers a continos evoltion from the Astralopithecs to the Homo sapiens sapiens which matches the intermediate stages of evoltion. If there is only one data S 1 at time t 1 = T, the criterion (1) defines the registration of S to S 1. In the LDDMM framework, the reslt of sch a registration is a geodesic flow of diffeomorphism between t = and t = T that maps S close to S 1 (Miller et al, 22). With several data at sccessive time points, we will show in Sec that the reslt is a flow of diffeomorphism which is geodesic only between sccessive time points (i.e. piecewise geodesic). We will also show that the comptation of the regression fnctions χ t takes into accont all the observations S i in the past and ftre simltaneosly. Therefore, it differs from pairwise registration between consective shapes. For instance, if the trade-off γ χ tends to infinity (no fidelity-to-data term) the regression is a constant map χ t = Id for all t. As γ χ decreases, the piecewise geodesic regression matches the data with increasing goodness of fit. This framework allows s also to perform the regression even if several data are associated to the same time-point. This will be sed in Sec. 5 to estimate a mean growth scenario of a time-series cross-sectional data set. Note that if T is greater than the latest time-point of the data t max, then the regression fnction χ is constant over the interval: [t max,t ]. Therefore, the method extrapolates with constant shape otside the time interval [,t max ]. Sch an extrapolation will be needed to compare the evoltion of two sbjects, whose latest observation correspond to different time-points. Similarly, we can also extrapolate the evoltion fnction at time earlier than with a constant map, so that the evoltion fnction can be defined on any arbitrary time interval Spatiotemporal registration between pairs of growth scenarios We sppose now that we have two sbjects S and U which have been scanned several times each (bt not necessarily the same nmber of times and possibly at different ages). Let S ti (resp. U t j ) be the shapes of sbject S (resp. U) at ages t i (resp. t j ). We define a time-interval of interest which contains every t i s and t j s. Withot loss of generality, we can assme that this time interval of interest is of the form [,T ]. We infer an individal growth model S(t) from the data of the sorce sbject {S ti }, sing the procedre of the previos section. As a reslt, the continos shape evoltion S(t) is of the form: S(t) = χ t (S ) for t [,1]. Or goal is to define a spatiotemporal deformation of the continos evoltion S(t) into S (t) so that the deformed shapes S (t j ) at the time-points of the target t j match the shape U t j (thanks to the continos regression, we can define S(t j ) for the target time point t j even if the sorce has not been observed at this age.) For this prpose, we introdce two fnctions (sing the sbject-specific paradigm in Sec. 2): the 3D morphological deformation φ and the 1D time warp ψ. The morphological deformation is a 3D-fnction, which maps the geometry of the sorce to the geometry of the target (change of reference frame). Every frame of the sorce seqence S(t) is deformed sing the same fnction. The time warp ψ maps the time-points t within the time interval [,T ] to ψ(t). This fnction does not change the frames of the seqence S(t) bt change the speed at which the frames are displayed. It models the change of the dynamics of the evoltion of the sorce with respect to the evoltion of the target. We impose this 1D fnction to be monotonic, assming that the shape changes occr in the same order, even if at a different pace between sorce and target. The combination of these two fnctions gives the spatiotemporal deformation of the continos evoltion S(t), defined as:

9 9 Fig. 5 Illstrative pairwise registration: data preparation. The database is ct in two to compare the evoltion {Homo habilis-erects-neandertalensis} (red shapes) to the evoltion {Homo erects-sapiens sapiens} (green shapes). The later evoltion is translated in time, so that both evoltions start at the same time. Then, one performs a shape regression of the sorce shapes (ble shapes). The spatiotemporal registration of this continos sorce evoltion to the target shapes is shown in Fig. 6 and 7. Fig. 6 Illstrative pairwise registration: morphological deformation and time warp. Top row: The inpt data as prepared in Fig. 5 with the continos sorce evoltion (ble) sperimposed with the target shapes (green). Middle row: The morphological deformation φ is applied to each frame of the sorce evoltion. It shows that, independently of time, the skll is larger, ronder and the jaw less prominent dring the later evoltion relative to the earlier evoltion. Bottom row: The time warp ψ is applied to the evoltion of the second row. The ble shapes are moved along the time axis (as shown by dashed black lines), bt they are not deformed. This change of the speed of evoltion shows an acceleration of the later evoltion relative to the earlier evoltion. Taking this time warp into accont enables a better alignment of the sorce to the target shapes than only the morphological deformation. Note that the morphological deformation and the time warp are estimated simltaneosly, as the minimizers of a combined cost fnction. S (t) = φ (S(ψ(t))). (2) Using the fact that S(t) = χ t (S ), this becomes 2 : S (t) = φ(χ ψ(t) (S )). (3) 2 We notice that in the time-specific paradigm, this wold be S (t) = χ ψ(t) (φ(s )) (see Sec. 2) In a MAP setting, the estimation of the best possible spatiotemporal deformation (φ, ψ) of the sorce evoltion which fits the the target observations, leads to the minimization of the discrepancy between the deformed sorce at target s timepoints S (t j ) = φ(s(ψ(t j )) and the target s shape U t j : E(φ,ψ) = t j d ( φ(s(ψ(t j ))),U t j ) 2 +γ φ Reg(φ)+γ ψ Reg(ψ), (4)

10 1 where d is a distance between shapes, Reg(φ) and Reg(ψ) the measre of reglarity of the deformation φ and ψ and γ φ,γ ψ the sal scalar trade-offs between reglarity and fidelity to data. An illstration of spatiotemporal registration is shown in Fig. 5, 6 and 7. We se the same set of profiles of 2D hominids sklls as in Fig. 4. Here we want to compare the evoltion {Homo habilis-homo erects-homo neandertalensis} (called earlier evoltion) with the evoltion {Homo erects-homo sapiens sapiens} (called later evoltion). The differences may be de to a change of the shape of the skll, as well as a change of the dynamics of evoltion between the earlier and the later evoltion. Therefore, we divide the database into two grops, considered as two different sbjects, and translate the target back for million years, so that both evoltions start at the same time (this can be seen as a rigid temporal alignment as a pre-processing). See Fig. 5. The regression of the sorce data leads to a continos sorce evoltion S(t) shown in ble in the first row of Fig. 5. The estimation of the spatiotemporal deformation between the sorce and the target reslts in a morphological deformation φ and a time warp ψ, see Fig. 6. The morphological deformation shows that the jaw is less prominent and the skll larger and ronder dring the later evoltion than dring the earlier evoltion (second row in Fig. 6). The effect of the time warp is to accelerate the sorce evoltion to adjst to the rate of shape change between the target shapes (third row in Fig. 6). The graph of the time warp is plotted in Fig. 7a. It shows an almost linear increase in speed. The slope of the crve is of 1.66, ths meaning that the later evoltion evolves 1.66 times faster than the earlier evoltion. This vale is compatible with the growth speed of the skll dring this period according to the vales reported in the literatre and in Fig. 7b: between Homo erects and Homo sapiens sapiens the skll volme had grown at a rate of (15 9)/.7 = 86cm 3 per millions of years, whereas between Homo habilis and Homo neandertalensis, it had grown at (15 6)/1.7 = 53cm 3 per millions of years, namely 1.62 times faster Atlas estimation from longitdinal data sets In this section, we want to combine the previosly introdced growth model and spatiotemporal deformations to estimate statistics from a longitdinal database. Given the repeated observations of a grop of sbjects, we assme that each sbject s evoltion derives from the same prototype evoltion, called a mean scenario of evoltion. Each sbject-specific evoltion is derived from the mean scenario via its own spatiotemporal deformation. The analysis of the set of all the spatiotemporal deformations in the poplation will lead to the estimation of the typical variations of the mean scenario in the poplation (the variance of the pop- time in the sorce space (in millions of years) slope = time in the target space (in millions of years) a- time warp b- skll volme evoltion Fig. 7 Illstrative pairwise registration: analysis of the time warp. Top: plot of the 1D time warp ψ(t) ptting into correspondence the timepoints of the target shapes with that of the sorce. The x = y line (dashed in black) wold correspond to no dynamical change between sorce and target (ψ(t) = t). The slope indicates that the shape changes between target data occr 1.66 times faster than the changes in the sorce evoltion, once morphological differences has been discarded. Right: the graph of the skll volme over the hman evoltion as fond in the literatre (sorce: This crve shows that the increase in skll volme between Homo erects and Homo sapiens sapiens was 1.62 times faster than between Homo habilis and Homo neandertalensis (ratio between the slope of the two straight lines). This vale is compatible with the acceleration measred by the time warp: lation in a sense to be defined). We assme that the mean scenario of evoltion is given by the growth model of an nknown prototype shape M, called template in the seqel.

11 11 Formally, this means that there is a growth fnction χ t for t [,T ] and a template shape M, so that the mean scenario of evoltion is written as: M(t) = χ t (M ) with M() = M. For each sbject s (s = 1,...,N sbj ), the sbject-specific spatiotemporal deformation of the mean scenario is written as: S s (t) = φ s (M (ψ s (t))) for all t [,T ]. φ s is the morphological deformation for sbject s and ψ s its time warp. These two fnctions model how the anatomy of the sbject and the dynamics of evoltion can be derived from the prototype scenario of evoltion. Eventally, we sppose that the observation of the sbject s at time-point t s j, denoted Ss j, is the temporal sample from S s (t) at time point t s j, p to a random Gassian noise: S s j = Ss (t s j ) + εs j : ( ) S s j = φ s χ ψ s (t s j ) (M ) + ε s j, (5) where the Gassian variables ε s j are independent and identically distribted over the sbject-index s and the time-index j. This eqation is or generative statistical model, which explains how the observations can be seen as instances of a random process. The fixed parameters are the prototype shape M and the growth fnction χ t. The random parameters are the spatiotemporal deformations (φ,ψ) (each estimated (φ s,ψ s ) is an instance of these random deformations). Both the fixed and the random parameters are nknown and shold be estimated given the actal observations. In the same MAP setting as in the previos section, the estimation of the nknown parameters can be done by minimizing the following combined cost fnction: ) E ((ψ s ) s=1,...,nsbj,(φ s ) s=1,...,nsbj, χ,m { N sbj s=1 d(φ s (χ ψ s (t s t s j ) M ),S s (t s j)) 2 j + γ φ Reg(φ s ) + γ ψ Reg(ψ s ) + γ χ Reg(χ) The otpt is the prototype shape M, the growth fnction χ t and the set of spatiotemporal deformations φ s,ψ s for every sbject s. These variables are called a spatiotemporal atlas. In Sec. 3.3, we will show how we can perform statistics on the estimated deformations (φ s,ψ s ), like Principal Component Analysis for instance. Sch statistics will describe the changes in shape and the variations of the speed of evoltion across sbjects. To illstrate the method, we rn the atlas estimation given the two sbjects in Fig. 5: the first sbject (in red) consists of three shapes, the second sbject (in green) consists of two shapes. From these five shapes, the method retrns the estimated template, the mean scenario and the two spatiotemporal registrations of this mean scenario to each sbject. The estimated template M is given as a crrent, which does not = } (6) form a set of crves anymore Drrleman et al (29a). To give an illstration of the atlas, we map the yongest shape of each sbject to this crrent and pick the deformed shape that is the closest to the estimated template. Then, one rns one more iteration of the atlas algorithm, to show the mean scenario and the spatiotemporal registrations as deformations of this template shape. This is shown in Fig. 8. In particlar, the two time warps which pt into correspondence the evoltion stages of each sbject to the ones of the estimated mean scenario are shown in Fig. 9-b. Let s denote S 1 (t) the spatiotemporal deformation of the mean scenario, which is spposed to match the shape of the first sbject: S 1 (t) = φ 1 (M(ψ 1 (t))). Similarly, S 2 (t) = φ 2 (M(ψ 2 (t))) matches the shapes of ( the second sbject. Then, by definition, we have: S 2 (t) = φ 2 φ 1 1 (S 1(ψ1 1 (ψ 2(t)))) ). At the first glance, this sggests that (φ 2 φ 1 1,ψ 1 1 ψ 2) corresponds to the spatiotemporal registration between the first sbject (considered then as the sorce) to the second sbject (considered as the target). We sperimposed in Fig. 9-c the graph of ψ1 1 ψ 2 with the graph of the time warp estimated in the previos section and shown in Fig. 7-a. As expected, the two crves show a similar pattern, namely the overall acceleration of the sorce relative to the target. However, noticeable differences appear, in particlar in the slope of the crves. This can be explained by at least two reasons. First, what we called here S 1 (t) is not the same shape evoltion as the one compted in the pairwise registration case (first row in Fig. 5): the regression of the sorce sbject in the registration case did not take into accont any information abot the target shapes, whereas the mean scenario M(t) (and conseqently its deformation S 1 (t)) averages the growth patterns of both sbjects. Second, the reasoning above does not take into accont the residal errors into accont: assming that S 1 (t) and S 2 (t) are the tre evoltion of each sbject, then we have: S 1 (t) = φ 1 (M(ψ 1 (t)))+ε 1 (t) and S 2 (t) = φ 2 (M(ψ 2 (t))) + ε 2 (t), where ε 1 (t) and ε 2 (t) models the residals shape which contains noise, small-scale variations and everything else, which cannot be explained by the model. The sqared norm in the criterion to be optimized shows that we assme these residals to be Gassian random variables (see Drrleman (21) for more details). This shows therefore that S 2 (t) = φ 2 φ1 1 ( S 1 (ψ1 1 ψ 2 )(t)+φ 2 φ 1 (ε(ψ1 1 ψ 2 (t))), meaning that the residal error between the two scenarios S 1 (t) and S 2 (t) is no more Gassian. Therefore, to retrieve the same deformations and time warps, one wold need to change the sqared norm in the registration criterion to take into accont the distortion in the distribtion of the residals indced by the deformations. The discssion above highlights the main featres of the atlas constrction method. The main assmption is that the different sbjects derive from the same prototype scenario, and therefore share common growth patterns even if altered in their shape and timing. The atlas aims precisely at detecting

12 12 Fig. 8 Spatiotemporal atlas estimation given the two sbjects in Fig. 5. On the middle row is shown the estimated mean scenario of evoltion. The first frame of this scenario (far left) is the estimated template shape. Two pper rows represents the morphological deformation and then the time warp, which jointly maps the mean scenario to the shapes of the first sbject (red shapes). Tow lower rows represents the spatiotemporal deformation of the mean scenario to the shapes of the second sbject (green shapes). Black arrows indicate areas where the most important shape deformations occr. these common featres and the variations of their shape and pace in the poplation. Every pattern, which is specific to a given individal, is discarded from the atlas and remains in the residals. In this sense, the atlas is a statistical tool, which detects the reprodcible patterns in the poplation. Compared to pairwise registration, the advantages of the atlas constrction is that it can be applied to more than two sbjects and that it does not favor any particlar sbject in the poplation. Remark 1 (On the assmption of diffeomorphic maps) In this modeling, we sppose that the evoltion fnction χ and the morphological deformations φ are 3D diffeomorphisms and that the time warps ψ are 1D diffeomorphisms. The motivation and conseqences of choosing diffeomorphic maps are different in each case. The evoltion fnction χ maps the anatomy of a sbject over time. Setting χ as a diffeomorphism assmes a smooth one-to-one correspondence between any observed shapes of the same sbject. This incldes modes of growth like atrophy, dilatation, torqe, etc. However, this cannot model a tearing of the shape, its division or the creation of another disconnected component over time. This assmption is realistic in many practical case, like for the heart over a cycle or the macroscopic observation of a brain strctre dring infancy. The morphological deformations φ model the geometrical inter-sbject variability. Assming a smooth one-to-one correspondence between the anatomies of two different sbjects is more qestionable. As highlighted in Drrleman et al (211) and Drrleman (21)[Chap. 5], the diffeomorphism is sed to decompose the inter-sbject variability into two terms: the diffeomorphic geometrical variability captred in the deformations and the non-diffeomorphic variability in terms of textre captred in the residals (modeled by the random Gassian variables). Both terms can be sed for the statistical analysis, whereas in this work we will focs only on the geometric variability captred by the deformations. Extending the work of Drrleman et al (211) to analyze the

13 13 estimated template M deformed template 1 (M ) deformed template 2 (M ) a- Template and its deformation time (sbject 1) (in millions of years) time warp 1 1 o 2 time warp of pairwise registration time (sbject 2) (in millions of years) b-time warps virtal time in template space (in millions of years) time warp for sbj #1: 1 time warp for sbj #2: real time (in millions of years) c-correspondence between sbjects Fig. 9 Spatiotemporal atlas estimation: template and time warp. (a) the template and its morphological deformation to the first sbject (red) and the second sbject (green). This corresponds to the far left frames in the second, third and forth row in Fig. 8. (b) The graphs of the two time warps, mapping the sbjects growth speed to that of the mean scenario. When the crve is above x = y axis, the sbject s evoltion is in advance relative to the rate of shape changes given by the mean scenario. (c) Graph of the fnction: ψ1 1 ψ 2 (in ble), which maps the dynamics of the two sbjects. The dashed red crve is the time warp given by the pairwise registration as shown in Fig. 7-a. non-diffeomorphic variations wold be possible bt ot of the scope of this paper. The time warp ψ model the change of speed of evoltion between sbjects. The diffeomorphic assmption in 1D implies that the fnction is smooth and monotonic. The monotonic property assmes that the seqence of the events dring evoltion occr in the same order for every sbject (from birth to death). This is a very realistic (if not desirable) hypothesis, at least from a biological point of view. Moreover, assming the evoltion of a strctre is smooth (at least differentiable) like its inverse is also very realistic, so that one can speak abot the speed of an evoltion. Therefore, the time warps ψ are intrinsically diffeomorphic. Remark 2 On the noise model In (5), we assmed the noise of the data to be Gassian. This choice leads to the sqared distance between the deformed template and each observation in the criterion (6). The same assmption is made by the registration schemes that are driven by sm of sqared differences -like metrics. Thogh convenient, this noise model is argable. In the framework of crrents, the simlation of a Gassian noise is eqivalent to adding random Dirac delta crrents at the nodes of a reglar lattice, whose covariance matrix is given by the kernel (momenta close to each others tend to be correlated), which is not nlike a sensor noise (see Drrleman (21)[Chap. 3] for more details). Other noise models that are more closely related to the mesh strctre of the srfaces cold be sed, at the cost of a more complex MAP derivation. Note that, in or model, we spposed the sbjects data to be corrpted by noise, and not the template which is spposed to be a noise free ideal representation of the shape. Therefore, the reslting cost fnction (6) is not symmetric, as the observed sbjects shapes do not play the same role as the template.

14 Comptational framework and algorithms A generic way to bild diffeomorphisms In this section, we explain a way to bild generic 3D and 1D flows of diffeomorphisms which will be sed as a model for the deformations χ t, φ and ψ in the following. We se here the LDDMM framework (Trové, 1998; Dpis et al, 1998; Miller et al, 22) for constrcting 3D diffeomorphisms. We propose to adapt this framework to the constrction of 1D diffeomorphisms. 3D diffeomorphisms In the LDDMM framework, 3D diffeomorphisms are generated by integrating time-varying vector fields. Let v t (x) be a time-varying speed vector field which gives the velocity of a particle which is at position x at time t. A particle which is at position x at time t = moves to the position φ t (x) at time t. The fnction φ t (x) follows the differential eqation for t : { dφt (x) dt = v t (φ t (x)) φ (x) = x Under some conditions on the reglarity of the speed vector field explained in Trové (1998), the set of deformation φ t is a flow of diffeomorphisms of the 3D domain. Following this theory, we assme that the speed vector field belongs to a reprodcible kernel Hilbert space (RKHS), meaning the speed vector fields reslt from the convoltion between a sqare integrable vector field and a smoothing kernel K, which plays the role of a low-pass filter. In or applications, we will se a Gassian ) kernel, which writes K(x,y) = σ 2 exp ( x y 2 /λ 2 I for any points (x,y) in space and I the identity matrix. The spatial scale λ determines the typical scale at which points in space have a correlated speed, and therefore move in a consistent way. It determines the degree of smoothness of the deformations. Large scale means almost rigid deformations. Small scales favor deformations with many small-scale local variations. The parameter σ is a scaling factor, which in some cases cancels ot with the trade-offs in the criterion, as we will discss later. In this setting, we define the measre of reglarity of the flow of diffeomorphisms as the total kinetic energy of the flow between t = and t = T : Reg(φ) = T (7) v t 2 V dt (8) where. V denotes the RKHS norm associated to the kernel K. An important property, (proven in Joshi and Miller (2); Glanès (25) and extended in Drrleman (21)[Chap.4] in case of the matching term involves several time-points), states that the vector field in the RKHS V which achieves the best trade-off between this reglarity term and a fidelityto-data term has a finite-dimensional parameterization, if the fidelity-to-data term depends only on a finite nmber of points: Proposition 1 (Finite dimensional parameterization of minimizing vector field) Let E be a criterion of the form: E(v) = A i (φt v i (S)) + γ i T v t 2 V dt (9) where v t denotes a time-varying speed vector field, φt v the flow generated by this vector field in the sense of (7), S a discrete set of N points x i in the 3D domain and A i a set of positive and continos fnctions from R 3N to R. Then, the criterion E admits at least one minimm and the vector field which minimizes E over all possible vector field in the RKHS V is parameterized by a set of N timevarying vectors (α i (t)), sch that: v t (x) = N i=1 K(x,x i (t))α i (t) (1) for any points x, where x i (t) = φ v t (x i ) satisfies the flow eqations: dx i (t) dt = v t (x i (t)) = N j=1 K(x i (t),x j (t))α j (t) with x i () = x i The coples (x i (t),α i (t)) are called momenta. (11) The norm of the minimizing vector field in the RKHS V is given as: v t 2 V = N i=1 N j=1 α i (t) t K(x i (t),x j (t))α i (t), (12) The criterion depends therefore only on the set of L 2 fnctions α i (t). Given these fnctions and the initial positions x i, one can integrate (11) to generate the trajectories x i (t). Then the criterion E involves only the coples (x i (t),α i (t). 1D diffeomorphisms We can adapt this framework to the constrction of 1D diffeomorphisms, which will be sed as time warps in or method. Let the variable t R play the role of the spatial variable x in the constrction of 3D diffeomorphisms. We can bild a flow of 1D diffeomorphisms ψ (t) for the parameter in [,1] (here plays the previos role of t, since now t denotes a real time and not the integration variable) by integrating the flow eqation: { dψ (t) d = v (ψ (t)) ψ (t) = t (13) where v is now a scalar fnction, which gives the speed at which the time ψ (t) evolves. If it is positive, time tends to

15 15 accelerate. If it is negative, time tends to slow down. We impose that v is in 1D RKHS, ) determined by the kernel K(t,t ) = σ 2 exp ( t t 2 /λ 2. The scalar parameter λ determine the typical time-length at which two time-points t and t are changed in a correlated manner. An illstration of the constrction of sch 1D diffeomorphism is given in Fig. 1. Then, the same property as Prop. 1 applies. If E is a criterion of the form: E(v) = A(ψ 1 (t)) + γ v 2 V d (14) where t denotes a vector of time-points t 1,...,t N and A a positive and continos scalar fnction, then the minimm of E over the RKHS exists and is achieved for a speed fnction v of the form: v (t) = N i=1 K(t,ψ (t i ))β i () (15) where the time-varying scalars β i () are L 2 fnctions from [,1] to R. The norm of the speed fnction v in the RKHS is given by: v 2 V = N i=1 N j=1 K(ψ (t i ),ψ (t j ))β i ()β j () (16) Optimization of the regression criterion We optimize the regression criterion (1) assming that the regression fnction χ t is generated by a time-varying velocity field v χ, which belongs to the RKHS V χ determined by the 3D Gassian kernel K χ with standard deviation λ χ. Defining the reglarity criterion as Reg(χ) = T v t 2 V χ dt, the criterion to be minimized becomes: E(χ) = d(χ ti (S ),S i ) 2 + γ χ v t 2 V χ dt (17) t i where d is a similarity measre between shapes. Here we assme that the baseline S and the shape S i are sets of points, polygonal lines or meshes. We denote (x 1,...,x N ) the vertices of the baseline S. Wold d be either the sm of sqared differences between point positions or the distance on crrents in absence of point correspondence, the conditions of Proposition 1 are satisfied Glanès (25): the minimizing vector field v t is parameterized by the momenta (x p (t),α p (t)): v χ t (x) = N p=1 Kχ (x,x p (t))α p (t). As noticed in Sec , the regression criterion E is a fnction of the N L 2 fnctions α p (t). We provide this set of N fnctions with the metric indced by the kernel K χ, meaning that the inner-prodct between two sets of L 2 fnctions α p (t) and α p(t) is given by: N N p=1 q=1 (α p(t)) t K χ (x p (t),x q (t))α q (t)dt. The gradient of E with respect to the pth fnction α p (t) is an L 2 fnction denoted αp E(t), which is sch that for all: d dτ E(α 1(t),...,α p (t) + τε p (t),...,α N (t)) = N q=1 ε p (t) t K χ (x p (t),x q (t)) αq E(t)dt. In Appendix A, we show that this gradient is eqal to: αp E(t) = 2γ χ α p (t) + η χ p (t) (18) where η χ p (t) is the soltion of the linear set of backward integral eqations for all p: η p (t) = ( xp (t i )A i )1 {t ti }+ t i T t N q=1 ( α p () t η q () + α q () t η p () ) + 2γ χ α p () t α q () 1 k χ (x p (),x q ())d (19) where 1 {t ti } = 1 if t t i and otherwise, A i = d(χ ti (S ),S i ) 2 seen ( as a fnction of ) the points positions x p (t i ) and k χ (x,y) = exp x y 2 /λχ 2. The gradient descent scheme for the comptation of the regression S(t) = χ t (S ) is smmarized in Algorithm 1 in Appendix C. We start the gradient descent by setting α p (t) = for all t and p (χ t = Id and S(t) = S, for all t). Compting the gradient reqires first to integrate of the flow eqation (Eq. (11)) forward in time and then to compte the axiliary variable η χ (Eq. (19)) backward in time. In this last case, the initial conditions at t = T is given by xp (T )A T. Then the ODE is integrated for decreasing time t. As soon as a new time point t i is reached, a new contribtion xp (t i )A i is added to η χ (t). As a conseqence, αp E(t) (and therefore the momenta α p (t) and the vector field vt χ ) at time t depend on all the data which appear later than t. Once the momenta are pdated, the new positions x p (t) are compted by the integration of the flow eqation (11) forward in time (the initial condition is given at time t = by x p () = x p ). These positions at time t depend on the vector field vt χ for all time earlier than t. As a reslt, the positions x p (t) depend on all the data in past and ftre. This regression fits the best trajectory (χ t (S )) to all the data globally. This differs, for instance, from pairwise registrations between consective time-points, althogh both techniqes reslt in a piecewise geodesic flow.

16 16 Fig. 1 Constrction of 1D diffeomorphisms by integration of speed fnctions. In this illstration, we sppose the speed fnction to be constant (v independent of ): ψ(t) dt = v(ψ (t)). Left: The speed profile v is set as the convoltion of 3 constant momenta (β i ) with a Gassian kernel with standard deviation λ ψ = 4 (in red). The integration of the flow eqation with the initial condition ψ (t) = t is shown in ble: the bold ble crve corresponds to the final diffeomorphism at = 1, light ble crves correspond to ψ 1/6 (t), ψ 1/3 (t), ψ 1/2 (t), ψ 2/3 (t) and ψ 5/6 (t). Right: Illstration of the nmerical integration of the flow: ψ n+1 (t) = ψ n (t) + τv(ψ n (t)). The speed profile in red is shown along the y-axis. One can show easily that this scheme prodces only increasing fnction (invertible 1D fnction), when τ is chosen small enogh. For better nmerical accracy, we replace the Eler scheme in Algorithm 1 to integrate ODEs by a Eler scheme with prediction/correction, which has the same accracy as a Rnge- Ktta method of order 2. The comptational bottleneck of this algorithm is the comptation of every sm of the form N p=1 K(x q,x p )α p that need to be compted for all q. These comptations of complexity N 2 (where N is nmber of points in the baseline shape) can be efficiently approximated sing a linearly spaced grid and FFT (Drrleman, 21), or Fast Mltipole Approximations (Glanès, 25), with a nearly linear complexity. The comptation of the gradient reqires to compte the differentiation of the fidelity-to-data term: xp (t i )d(φ ti (S ) S i ) 2. If S i have the same nmber of points as S (i.e. N points), then d can be defined as the sm of sqared differences: A i = N p=1 x p (t i ) s i 2 p, where s i p denotes the points of S i. In this case, xp (t i )A i = 2(x p (t i ) s i p). In absence of point correspondence, the distance on crrents is sed, which can be differentiated as explained in Glanès (25); Drrleman (21) Optimization of the spatiotemporal registration criterion As explained in Sec , the morphological deformation φ and the time warp ψ are generated by the integration of flows of 3D and 1D velocity fields respectively. This means that they are the end-points φ = φ 1 and ψ = ψ 1 of the differential eqations: φ (x) = v φ (φ (x)) ψ (t) = v ψ (ψ (t)) (2) with the initial conditions: φ (x) = x and ψ (t) = t. Now, we assme that for every parameter, the 3D velocity fields v φ and 1D velocity profile v ψ belong to a RKHS with Gassian kernel K φ and K ψ, with standard deviation λ φ and λ ψ respectively. We denote {x p } p=1,...,n the set of points of the discrete shape S. The sorce trajectory S(t) = χ t (S ) is described by the moving points x p (t). Let {t j } j=1,...,ntarget be the time-points associated to target shapes.

17 17 The fidelity-to-data term in (4) depends on the variables φ 1 (x p (ψ 1 (t j ))) = φ 1 (x p, j ), where we denote (see Fig. 11 for an illstrative scheme): x p, j = x p (ψ 1 (t j )) (21) Therefore, the application of Proposition 1 leads to the following parameterization of the minimizing velocity fields: v φ (x) = and N p=1 N target v ψ (t) = j=1 N target K φ (x,φ (x p, j ))α p, j () (22) j=1 K ψ (t,ψ (t j ))β j () (23) The criterion (4) is a fnction of the N N target L 2 fnctions α p, j and the N target L 2 fnctions β j. Like for the regression case, we provide this set of fnctions with the metric indced by the kernel K φ and K ψ. The reglarity parameters in (4) are given by: Reg(φ) = and = Reg(ψ) = = v φ 2 V φ d p, j,p, j α p, j () t K φ (x p, j (),x p, j ())α p, j ()d v ψ 2 V ψ d j, j β j () t K ψ (t j (),t j ())β j ()d (25) (26) As shown in Appendix B, the gradient of the criterion with respect to the fnctions α p, j () (denoted αp, j E()) and to the fnctions β j () (denoted β j E()) is given by: αp,i E() = 2γ φ α p,i () + η p,i () β j E() = 2γ ψ β j () + ξ j () (27) where η i,p () satisfies the backward integral eqation: η p,i () = xp,i (1)A ( + N q=1 N target j=1 α p,i (s) t η q, j (s) + η p,i (s) t α q, j (s) + 2γ φ α p,i (s) t α q, j (s) ) 1 k φ (x p,i (s),x q, j (s))ds (28) Fig. 11 Illstrative scheme for the notations: x p denotes a generic point of the sorce shape, x p (t) = χ t (x) the continos evoltion of the sorce point, (ψ ) [,1] is a flow of 1D-diffeomorphism which moves the time-labels along the time-axis (in red), (φ ) [,1] is a flow of 3Ddiffeomorphism which moves the points of the sorce evoltion (in magenta), independently at each time-point. For the sake of simplicity, we introdce the notations x p,t () and t j () sch that: φ (x p, j ) = x p, j () x p, j () = x p, j = x p (ψ 1 (t j )) x p, j (1) = φ(x p, j ) ψ (t j ) = t j () t j () = t j t j (1) = ψ(t j ) (24) where A denotes the matching term N target j=1 d(φ 1 (S(ψ 1 (t j ))) U j ) 2 which is a fnction of the variables φ 1 (x p (ψ 1 (t j ))) = x p, j (1). and where ξ j () satisfies the backward integral eqation: ξ j () = + N p=1 N target k=1 ( ( dx p (t) dt t η p, j () t=t j (1)) β j (s) t ξ k (s) + ξ j (s) t β k (s) + 2γ ψ β j (s) t β k (s) ) 1 k ψ (t j (s),t k (s))ds (29) The axiliary space variable η() plls the gradient of the matching term from = 1 back to = along the space axis. Then, the vale η() is sed in the final conditions of the axiliary time variable ξ () in combination with the local speed of the sorce growth scenario, ths showing the spatiotemporal copling. The variable ξ () plls back this condition at = 1 back to = along the time axis. The gradient transports the driving force in the target space, namely

18 18 the gradient of the data term, back to sorce space along the spatiotemporal deformation. This transport is sed to pdate the momenta α p, j () and β j (), which parameterize the spatiotemporal deformation. In (28), the gradient of the matching term is compted as for the regression fnction. For instance, if the distance between sorce and target is the sm of sqared differences: A = N target j=1 N p=1 xp, j (1) U p, j 2, then the gradient is simply xp,i (1)A = 2(x p, j (1) U p, j ). In (29), one needs to compte the speed of the sorce growth scenario: dx p(t) dt. If one has stored the parameterization of the regression fnction (i.e. the momenta (x p (t),αp χ (t))), then one can compte explicitly: dx p (t) dt = dχ t(x p ) = vt χ (x p (t)) dt = N q=1 K χ (x p (t),x q (t))α χ q (t) (3) In or implementation, we only stored samples of the trajectories x p (t) and not the vectors α χ (t). So, we estimate this speed by a finite difference scheme: dx p(t) dt (t j ) = (x p (t j+1 ) x p (t j 1 ))/2. This allows s to still se this spatiotemporal registration even if the sorce evoltion has been compted with another regression method than the one presented in Sec The sketch of the gradient descent for this spatiotemporal registration scheme is given in Algorithm 2 in Appendix C. Note that we minimize the criterion with respect to the geometrical and the temporal parameters jointly, ths avoiding alternated minimization. The differentiation of the criterion proposed here is here, whereas an approximation was involved in Drrleman et al (29b); Drrleman (21). In practice, both differentiations leads to similar reslts. Remark 3 Note that since the growth model χ t is piecewise geodesic, the evoltion S(t) generated by χ t is not differentiable at the time-points t j : the continos S(t) may have different left and right derivatives. This point is discssed in depth in Drrleman (21)[Chap. 9], where an alternative optimization procedre is proposed, which ensres that an extremm of the registration criterion is achieved at convergence, even in presence of discontinos velocities. Another way to address the problem is to se the twice-differentiable growth model proposed in Fishbagh et al (211) Optimization of the criterion for atlas constrction The estimation of the 4D-atlas relies on one regression fnction χ t and N sbj spatiotemporal deformations (φ s,ψ s ) for s = 1,...,N sbj, where N sbj is the nmber of sbjects. We se the framework of Sec to constrct the 3D diffeomorphisms χ and φ s and the 1D diffeomorphisms ψ s. As a conseqence, every deformation satisfies a flow eqation as follows: χ t (x) = vt χ (χ t (x)), t t [,T ] φ(x) s = v φ s (φ s (x)), [,1],s = 1,...,N sbj ψ(t) s = v ψs (ψ s (t)), [,1],s = 1,...,N sbj (31) where we sppose that the velocity fields v χ (resp. v φ s and v ψs ) belong to a RKHS V χ (resp. V φ and V ψ ) determined by the Gassian kernel K χ (resp. K φ and K ψ ) with standard deviation λ χ (resp. λ φ and λ ψ ). We sppose that the prototype shape M (to be estimated) is given by a finite set of points {x p }. In this case, the application of Proposition 1 leads to the parameterization of the time-varying velocity fields by momenta as follows: vt χ (x) = K χ (x,x p (t))αp χ (t) p v φ s (x) = K φ ( x,x s p, j() ) αp, s j(), for s = 1,...,N sbj v ψs p,t s j (t) = K ψ ( t,t s j() ) β j s (), for s = 1,...,N sbj t s j where we denote: x p (t) = χ t (x p ) x s p, j() = φ s (x s p(t s j(1))) t s j() = ψ s (t s j) (32) (33) for all t [,T ] and [,1]. t s j denotes the N s target timepoints at which the sth sbject has been observed, which might be different for every sbject. The criterion for atlas estimation depends therefore on the N points of the template M = {x p } p=1,...,n, the N time t-varying vectors α p (t) for the regression fnction, the N N sbj s=1 Ns target -varying vectors αp, s j () for the morphological deformations and the N sbj s=1 Ns target -varying vectors β j s() for the time warps. This criterion can be written now as: J ( {αp, s j()},{β j s ()},{αp χ ) (t)},m = { N sbj Nsbj Ntarget s A ( {x s p, j(1)} ) + γ φ s=1 s=1 j=1 + γ ψ v ψs 2 V ψ d + γ χ T v φ s 2 d V φ } χ vt 2 V χ dt (34) ) where the matching term A ({x s p, j (1)} = d(φ s (χ ψ s (t s j ) M ),S s j )2 depends on the positions φ s 1 (M(ψ 1(t s j ))) = {(xs p, j (1)} p=1,...,n. To minimize this criterion, we adopt a 3-step alternating minimization procedre:

19 19 If the template M and the growth fnction χ t are fixed, the criterion is divided into N sbj independent fnctions. Their minimm is achieved for the spatiotemporal deformations (φ s,ψ s ), which maps the mean scenario χ t (M ) to the set of data S s j for each sbject s. These N sbj spatiotemporal registrations are compted sing Algorithm 2. If the N sbj spatiotemporal deformations (φ s,ψ s ) and the growth fnction χ t are fixed, the criterion to be minimized with respect to M is redced to: J(M ) = d(φ s, j (M ),S s j) 2, s, j where we denote Φ s, j = φ s χ ψ s (t s j ). These deformations are 3D-diffeomorphisms. This criterion has exactly the form of the criterion for sal 3D template estimation. If d is the distance on crrents, a soltion for the minimization of this convex criterion has been proposed in Drrleman et al (29a) and Drrleman (21)[Chap. 5, Algorithm 4]. As a conseqence, the template M is always given as a finite set of points {x p } p=1,...,n. If the template M and the N sbj spatiotemporal deformations (φ s,ψ s ) are fixed, the criterion to be minimized becomes: d(φ s (χ ψ s (t s j ) M ),S s (t s j)) 2 + γ χ Reg(χ). s, j This is not exactly the regression problem stated in Section becase of the deformation φ s in the matching term. To trn it into a regression problem, we approximate the matching term d(φ s (χ ψ s (t s j ) M ),S s (t s j )) by d(χ ψ s (t s j ) (M ),(φ s ) 1 (S s (t s j ))), meaning that the shapes of each sbject are matched back to the mean anatomy. This approximation is valid only for diffeomorphisms φ s whose Jacobian is close to the identity, since the sal metrics d are not left-invariant. As a reslt, the evoltion fnction χ t performs the temporal regression of the set of shapes (φ s ) 1 (S s j ) located at time-points ψs (t s j ). This regression problem can now be solved sing Algorithm 1. Frther investigations are needed in order to perform this regression withot this approximation, so that we can be consistent throghot the minimization procedre. To initialize the minimization, we set M as the mean crrent of the earliest data ((S1 s ) for every sbject s) and set all the momenta to zeros (χ,φ s,ψ s eqal identity map). The whole minimization procedre is smmarized in Algorithm 3 in Appendix C Parameters The overall framework depends on several parameters. There are 3 kernels of 3 distinct RKHS: K χ, K φ and K ψ. We se Gassian kernels determined by their standard deviations: λ χ, λ φ and λ ψ respectively. They determine the degree of smoothness (i.e. the scale at which points have a correlated speed) of the mean scenario of evoltion, the morphological deformations and the time warp. The first one compares with the scale of the geometrical variations of the strctre over time for a typical sbject (scale of the intrasbject variability). The second one compares with the scale of geometrical variations between different sbjects (geometrical inter-sbject variability). The third one compares with the typical time-scale at which the dynamics of evoltion changes from sbject to sbject. The ser mst also set the 3 trade-offs between reglarity and fidelity to data: γ χ,γ φ,γ ψ. In addition, one needs to set the metric d between shapes. In the framework of crrents, this metric depends on a kernel K W. We choose a Gassian kernel with standard deviation λ W. This parameter sets the typical scale at which shape variations are smoothed (see Drrleman (21)). The dimension of the trade-off γ χ, γ φ and γ ψ depends on the kinds of data that we deal with. The dimension of the data term in the criterions is L 2 (i.e. sqared length) for crves and L 4 (i.e. sqared area) for srfaces, where L denotes the dimension of a length. The parameter t has the dimension of time (denoted T) and the parameter is an integration parameter, which is normalized to fall in the nit interval [, 1] and therefore has no physical dimension. Therefore, the velocities vt χ, v φ, v ψ are of dimension LT 1, L and T respectively and the reglarity terms (integral of the sqared norm of the velocities) Reg(χ), Reg(φ) and Reg(ψ) are of dimension: L 2 T 1, L 2 and T 2 respectively. Eventally, the dimension of the trade-off γ is that of the ratio between the data term and the reglarity terms: crves srfaces γ χ T L 2 T γ φ none L 2 γ ψ L 2 T 2 L 4 T 2 In the ftre, we plan to normalize these constants so that their vales can be compared for different applications. More generally, one needs to better nderstand the balance between the spatial and temporal constraints and to find an atomatic way to estimate this parameters (which cold be considered as fixed effects in a Bayesian framework along the lines of Allassonnière and Khn (29)). 3.3 Statistical measres of spatiotemporal variability The constrction of the spatiotemporal atlas leads to the mean scenario M(t), which gives a representative mean of the stdied poplation, and the spatiotemporal deformations of this mean scenario to each sbject, which estimates the variance within the poplation. The criterion for the atlas constrction is not nlike the estimation of a Fréchet mean on the

20 2 manifold of the individal trajectories, the distance between two individal trajectories being given by the cost of the spatiotemporal deformation which connects them (see Miller et al (22) for this interpretation in the 3D case). In this section, we explain how one can compte intrinsic statistics on the spatiotemporal deformations: the mean and the principal modes of the morphological deformations and the time warps are defined as 3D and 1D diffeomorphisms themselves. De to the definition of the mean scenario, the mean of all deformations vanishes. Nevertheless, one can compte the means of poplation sb-grops to detect significant differences between them. The modes show the typical variations of the mean scenario within the poplation or within one sb-grop. They can be sed to drive the search for anatomical characterization of sb-grops. Besides the qantification of grop differences and the sal hypothesis testing, one important aspect of intrinsic statistics is that means and modes can be displayed as movies of shape evoltions, which is crcial for qalitative interpretation prposes. This can be sed to better nderstand the effect of a pathology and drive the search for bio-markers Statistics on initial momenta As shown in Miller et al (22), the flow of diffeomorphisms which minimize the registration criterion (4) or the atlas constrction criterion (6) are geodesic: they are the ones which minimize the length of the path (φ,ψ ) [,1] between the identity map Id and the actal diffeomorphisms (φ 1,ψ 1 ). These initial velocity plays the role of a tangentspace representation as in finite-dimensional Riemannian geometry (Pennec et al, 26): they are the eqivalent of the logarithm of the deformations. Since we perform templateto-sbjects registration (and not sbjects-to-template), every flows of diffeomorphisms φ s ( resp. ψ s ) starts from the same space, the one of the mean scenario, and therefore share the same tangent-space V φ (resp. V ψ ). As a conseqence, one can perform intrinsic statistics on these common vector spaces. Since the initial velocities are parameterized by a finite nmber of momenta, the statistics on deformations redces to statistics in an Eclidean space. For each sbject s, the 3D diffeomorphisms φ s are parameterized by momenta αp, s j () located at the points x p, j() = x p (t s j (1)), which is a sbset of the whole point set x p,k (the trajectories of every template point). Using zero-padding, every φ s is parameterized by a vector of the same dimension α s = {α s p,k } p,k. Similarly, each 1D diffeomorphism ψ s is characterized by momenta β j s () located at time-points t s j () = ts j, which is a sbset of the set of all time-points {t k}. Using zero-padding, every 1D diffeomorphism is characterized by a vector of the same dimension: β s = {β s k } k. One can compte a Principal Component Analysis (PCA) on the vectors α s and β s according to the metric on the RKHS V φ and V ψ as follows (see Drrleman (21)[Chap. 5] for more details). One bilds the mean vectors α = s α s /N sbj and β = s β s /N sbj and the centered vectors α s = α s α and β s = β s β. Then, one bilds the empirical matrices Σ φ and Σ ψ of size N sbj N sbj whose term s,s is given by: Σ φ s,s = α s, α s = V φ N k,k p,p =1 ( α p,k s )t K φ (x p,k,x p,k ) αs p,k Σ ψ s s s,s = β, β = K ψ (t k,t k ) β s β V ψ k k s k,k (35) Let E φ m and E ψ m the eigenvectors of the matrices Σ φ and Σ ψ associated to the mth largest eigenvales λ φ m and λ ψ m (these are vectors of dimension N sbj ). Then, as shown in Drrleman (21)[Chap. 5], the mth eigenmode is given by: N sbj α m = α + s=1 N sbj β m = β + s=1 E φ m,s(α s α) E ψ m,s(β s β) sch that α m α 2 V φ = λm φ and β m β 2 = λ ψ V ψ m. (36) Geodesic shooting for compting intrinsic means and modes Once one has compted the statistics on the tangentspaces V φ and V ψ, one needs to se the geodesic shooting eqations (the eqivalent of the exponential map in Riemannian manifold) to generate the 3D and 1D diffeomorphisms, whose initial velocities are parameterized by the compted mean or modes. The compted diffeomorphisms are called the mean or the modes of the deformations. Given vectors α and β (located at the points {x p,k } p,k and {t k }), the diffeomorphisms, whose initial velocities are parameterized by these vectors, are re-constrcted by integrating over the interval [,1]: dα p,k () d dx p,k () d = ( d xp,k ()v φ ) t αp,k () = v φ (x p,k ()) (37) where v φ (x) = p,k K φ (x,x p,k ())α p,k () (see Miller et al (26) for the proof). The positions of the points x p,k (1) = φ 1 (x p,k ) bilds a movie, which shows the mean (if α is sed as initial conditions) or the mth mode (if ±α m is sed as initial conditions) of the morphological deformations within the poplation or one of its sb-grop. The eqivalent eqations for 1D diffeomorphisms are given as: { dβk () = ( d tk ()v ψ ) t βk () d dt k () d = v ψ (t k ()) (38)

21 21 where v ψ (t) = k K ψ (t,t k ())β k (). The integration of this set of ODEs leads to the 1D diffeomorphism t k (1) = ψ 1 (t k ), called the mean time warp (if the mean vector β is sed as initial conditions) or the mth mode of the time warps (if the mth mode ±β m is sed as initial conditions). The geodesic shooting of the mean and the principal modes of the momenta leads to diffeomorphisms. In this sense, they are intrinsic statistics on an infinite-dimensional manifold of diffeomorphisms. In particlar, the mean or the modes of the time warps are all smooth monotonic fnctions. This differs from compting the point-by-point mean of the real-vales fnctions ψ s (t). Examples will be shown in the next sections. 4 Measres of developmental delays of deep brain strctres in atism In this section, we apply the tools introdced in Sec. 3 to analyze a longitdinal database of deep brain strctres segmented from images of atistic, developmental delayed and control children (Hazlett et al, 25, 211). Each child has been scanned twice: a baseline at abot age 3 years and a follow-p at abot age 5 years. The segmentation provides a set of 24 meshes for each strctre: 12 sbjects divided on 3 grops of 4 sbjects (atistics, developmental delays and controls), each sbject having two meshes (a baseline and a follow-p). As a pre-processing, all the meshes were co-registered via rigid transformations sing gmmreg (Jian and Vemri, 25). Or prpose is to show how or methodology can be sed to give a description of the effect of the pathology on the matration of the hippocamps and the amygdala of the right hemisphere. De to the limited nmber of sbjects involved, this stdy is mostly a proof of concept, aiming at showing the strengths and the limitations of or approach. 4.1 Spatiotemporal atlas estimation We estimate a spatiotemporal atlas by minimizing the criterion in (6) sing the algorithms described in Sec We set the time-interval of interest to [.5, 7.1] years with a time-step of.2 years. The parameters of the Gassian kernels were set to λ χ = 1 mm for the regression fnction, λ φ = 1 mm and σ φ = 1 for the morphological deformation and λ ψ = 1.5 years and σ ψ = 1 for the time warp. The typical scale on crrents λ W is set to 5 mm. The tradeoffs were set to γ χ = 1 4 mm 2 year, γ φ = 1 4 mm 2 and γ ψ = 1 6 mm 4 year 2. The diameter of the hippocamps is abot 25 mm. We refer the reader to Sec for a discssion abot the parameters sed (an empirical stdy of the impact of these parameters will be presented in the next section abot endocranial data). The otpt of the algorithm is a prototype shape, a mean scenario of evoltion of this prototype shape, and 12 spatiotemporal deformations of this mean scenario to the pair of meshes of each sbject. The analysis of the vale of the criterion at the minimm shows that one atistic patient has a residal significantly larger than the other sbject. This sbject can be considered as an otlier, as will discss later on. As shown in Drrleman et al (29a), the prototype shape is given as a crrent, which does not correspond to a mesh. For visalization prposes, and for the following volme comptations, we mapped one instance of the data to the prototype shape and sed it as a template. As a conseqence, the mean scenario and its spatiotemporal deformations can be seen as the continos evoltion of a mesh. Significant samples of the estimated mean growth scenario are shown in Fig. 12. The complete scenario can be seen in the companion movie (See Online Resorce 1). This mean scenario of evoltion shows that the prototype growth of the strctre is mch more complex than a pre volme scaling over time, and involves several non-linear growth patterns. The visal inspection of the companion movie shows mainly 3 phases of growth: from 1.5 to 2.5 years, the hippocamps tends to nfold giving it less crved aspect; from 2.5 to 4.5 years, the hippocamps strongly elongates in the antero/posterior direction; from 4.5 to 6 years, the extremities of the hippocamps tends to bend: the head toward the bottom and the tail toward the top, ths stretching the body of the hippocamps (red blob in the third frame in Fig. 12). This mean scenario of evoltion has been estimated along with its spatiotemporal deformation to each sbject. The spatiotemporal deformation takes into accont all the shape information and not only the size. Nevertheless, to illstrate the method, we compte the volme of the original meshes, of the template mesh and the deformed meshes. The evoltion of the volme of the mean hippocampal growth is shown in Fig. 13-a. Althogh the growth involves different non-linear patterns in shape as highlighted in Fig. 12, the volme extracted from this mean scenario evolves qite linearly between age 2 and 6 years. Otside this interval, the volme remains constant de to the bondary conditions (the growth fnction χ t eqals identity). In Fig. 13-b,c,d, we plot the volme evoltion of the mean scenario, once it has been registered to each sbject, taking into accont both the morphological deformation and the time warp: the morphological deformation changes the vales of the crve in Fig. 13a, whereas the time warp stretches or shortens the crve along the time axis. We sperimpose the volme of the original pairs of data for each sbject, as well as the volme evoltion compted from the pairwise registration between these pairs of srfaces. Note that there is no reason that the volme of the registered mean scenario corresponds to the volme evoltion compted from the pairwise registration. Indeed, the pairwise

22 22 Fig. 12 Mean growth scenario of the hippocamps. For significant frames are shown (lateral view). Color indicates the instantaneos speed of the srface deformation (best seen as a movie: see Online Resorce 1) registration take into accont only a pair of data, whereas the mean scenario integrates the information of the whole database: the mean scenario may contain growth patterns which are not present in a given sbject s evoltion. Moreover, the pairwise registration aims at minimizing the discrepancy between the two srfaces, whereas the deformed mean scenario is more constrained by the fact that we assme each sbject s pair of srfaces to reslt from a smooth deformation of a mean scenario. Having said that, we notice that volme evoltion of the deformed scenario does not deviate too mch from the volme evoltion compted from pairwise registration. This means that the morphological deformation acconts well for the different sizes of the strctres and that the time warp enables to adjst the slope of the crves to the different growth speed of each sbject. We notice that the crves for one atistic patient are not properly aligned (the patient for which the the volme decreases between the two observations in Fig. 13-b). This is the patient detected as an otlier. With the crrent set of parameters, it was too costly to deform the mean scenario to this sbject, which present a niqe pattern of size redction over time (this volme redction might be real or might be de to a segmentation inaccracy as well). We rn the atlas estimation with a more important weight for the time warp than for the morphological deformation (σ ψ /σ φ p to 1 instead of 1) and for larger scales for the time warps (λ ψ = 1.5 to 2.5), which redces the cost of time warps of large amplitde. We observe that the estimated mean scenario tends to show a volme redction near age 6 years (after a phase of volme increase from 2 to nearly 6 years) and the otlier is registered to the later part of the mean growth scenario: its time warp shows a strong advance in development of this sbject relative to the mean scenario. Nevertheless, this was done at the cost of less accrate registration of all other sbjects and the atlas with sch parameters was not optimal. Or statistical model prefers to treat this particlar sbject as an otlier. However, wold more sbjects be available showing a decreasing volme over time, the atlas wold be likely to take this into accont by estimating a growth scenario decomposed into a first phase of increasing volme and a second phase of decreasing volme. Then, the sbject with decreasing volme wold be systematically considered as delayed with respect to the sbject with increasing volme at the same age. Sch a poplation, however, wold probably violate or main assmption that the growth of the sbjects are homologos, in the sense that they derive from a common prototype scenario. It wold be better to consider the two sb-grops as two different poplations. 4.2 Analysis of the spatiotemporal variability Now, we analyze the spatiotemporal variability of the mean scenario in the poplation, and not only its effect on the volme distribtion. This variability is decomposed into a geometrical part captred by the morphological deformations φ s and a temporal part captred by the time warps ψ s. Preliminary tests performed on the initial momenta of the 3D diffeomorphisms φ s (see Sec. 3.3) do not show any correlations between the morphological deformations and the class of the sbject (atistics, developmental delays and controls). The mean initial momenta of the morphological deformations of each grop do not differ significantly from zero. The direction of the first mode of deformation is similar for each grop, bt the variance is larger for the atistics and developmental delays than for the controls. This mode essentially shows important variations in the elongation of

23 Volme (in cm 3 ) Volme (in cm 3 ) age (in years) age (in years) a- Mean scenario b- after registration to atistics Volme (in cm 3 ) Volme (in cm 3 ) age (in years) age (in years) c- to developmental delays d- to controls Fig. 13 Evoltion of the volme of the hippocamps. a- Volme evoltion of the mean scenario. b-to d-: Volme of the mean scenario, after it has been registered to each sbject (magenta crves). Black asterisks indicate the volme of the original data. Red, green and ble crves indicate the volme evoltion given by the pairwise registration between each sbject s data pair. The atistic otlier corresponds to the decreasing red crve in b- (decrease of volme between the two observations of this sbject) the hippocamps along with an enlargement of the body. By contrast, the depth of the hippocamps almost does not vary. The time warps are plotted in Fig. 14-a for every sbject. When the crve is above the y = x axis (resp. below the y = x axis), the evoltion of the sbject is in advance (resp. is delayed) relatively to the mean scenario. A slope greater than 1 (resp. smaller than 1) denotes an acceleration (resp. a speed redction) of the evoltion of the sbject compared to the evoltion of the mean scenario. The mean of the crves for each grop is plotted in Fig. 14-b. Althogh the mean of all crves seem to be biased (overall over the x = y axis), this bias is not proven to be statistically significant (ratio of mean over standard deviation being eqal to 1.21). We compte the intrinsic first mode of variations in the space of smooth monotonic fnctions (see Sec. 3.3) in Fig. 14-c, and in Fig. 14-d by exclding the atistic otlier. From these reslts, we cannot conclde that an atistic or a developmental delayed patient is systematically delayed or in advance compared to controls, even at a given age. However, both the atistics and the controls has a mch narrower variability interval than the developmental delays. It seems also that the atistic on average are more advanced than the control at the earlier stages of growth (between 2 and 3 years of age). This

24 Virtal Physiological Age Devlpt Delay 1 Atistics Controls Real Age (in years) Virtal Physiological Age Devlpt Delay 1 Atistics Controls Real Age (in years) a- raw crves b- mean monotonic fnctions 8 7 Virtal Physiological Age Real Age (in years) Devlpt Delay Atistics Controls Virtal Physiological Age Devlpt Delay 1 Atistics Controls Real Age (in years) c- 1st mode of variability d- 1st mode w/o the otlier Fig. 14 Estimated time warps from the hippocamps database. a- The monotonic crves indicates how the real age of each sbject maps to the virtal physiological stage estimated in the mean growth scenario. When crves are above the x = y axis, the sbject is in advance with respect to the mean scenario. The dashed red crve corresponds to the otlier. b- Intrinsic means of each grop (also monotonic fnctions). c- Limits of the first mode of variation at ± one standard deviation. d- Same as c, bt exclding the otlier. It shows that atistics tend to be in advance with respect to the control and that the developmental delays have a mch greater variance than the other two grops. sggests that the hippocamps develops faster among atistic children. To investigate this more in depth, we also compte the spatiotemporal atlas for the amygdala of the right hemisphere (sing the the parameters λ χ = 15 mm, λ φ = 15 mm, σ φ = 1, λ ψ = 1 year, σ ψ = 1, λ W = 3 mm, γ χ = 1 3 mm 2 year, γ φ = 1 3 mm 2 and γ ψ = 1 6 mm 4 year 2 ). Again, the analysis of the morphological deformations does not highlight informative patterns. By contrast, the analysis of the time warps shown in Fig. 15 reveals that the atistics and the controls share a similar pattern, namely a strong acceleration of the growth with respect to the mean scenario, bt at a different age. The acceleration occrs between age 2.5 and 3.5 years for the atistics and between age 4 and age 5 years for the controls. The developmental delays also display a similar pattern, bt it occrs at a more variable age. This confirms the hypothesis of an over-growth of the atistics compared to the controls at the earlier stages of development. This also

25 25 Virtal Physiological Age Devlpt Delay 1 Atistics Controls Real Age (in years) a - time warps for the 12 sbjects Virtal Physiological Age Devlpt Delay Atistics Controls Real Age (in years) b- first mode of variations Fig. 15 Estimated time warps from the amygdala database a- time warps for the 12 sbjects, b- limits of the first mode of variation at ± 1 standard deviation for each grop. Atistics and controls show the same evoltion pattern, namely a redction of speed with respect to the mean scenario (slope smaller than 1) and then a qick acceleration (slope greater than 1). This pattern for the atistics grop seems to occr later than for the control grop. The developmental delays presents also sch pattern bt at an arbitrary age. Mean and modes are compted as monotonic fnctions within the space of 1D diffeomorphisms. confirms that fact that the developmental delays do not bild a very homogeneos grop becase of mch more variable patterns. These preliminary reslts on both the geometrical and temporal parts of the variability sggest that the discriminative information between classes might not be inferred from the anatomical variability at a given age, bt rather from variations of the growth process. It sggests that atism may more strongly affect the growth speed of the deep brain strctres rather than its shape, a finding related to brain overgrowth discssed in Hazlett et al (211). Note that the hypothesis of an over-growth of the brain of atistic patient has been reported in the literatre, for instance in Corchesne et al (211). We believe that this new methodology is well adapted to test this hypothesis thanks to the introdction of the time warps, which models explicitly the possible developmental delays between sbjects both in shape and in size. Of corse, one wold need to test it on mch larger database: the more time-points per sbjects, the more constrained the mean scenario estimation; the more sbjects, the more robst the statistics. 5 Comparison of the endocast growth between chimpanzees and bonobos 5.1 Framework of the stdy In this section, we aim at sing or methodology to characterize the differences of growth patterns between the two closest hman relatives: the bonobo (Pan paniscs) and the chimpanzee (Pan troglodytes). We will also assess the robstness of the method with respect to parameter changes and changes in age labels. Since bonobos were discovered to science in 1929, the analysis of what distingishes them from the common chimpanzee has been controversial. After several morphological and behavioral stdies (Kroda, 1989; Shea, 1989; Kano, 1992; de Waal, 1995), the hypothesis has emerged that the bonobos may be a jvenilized version of the chimpanzee, in the sense that the growth of the bonobos may share common patterns with the one the chimpanzees bt with a different tempo. The tools that we have developed, and in particlar the introdction of the time warps, seem to be well adapted to test this hypothesis. For this prpose, we will se one the largest collections of endocasts available for the two species, which comes from the collection of the Msée de l Afriqe centrale in Tervren, Belgim. The endocast is a mold of the endocranim, which provides a replica of the inner srface of the skll and therefore has often played an important role for the analysis of the evoltion of the brain in fossil mammals. This data set consists of samples from wild-shot animals: 59 chimpanzees and 6 bonobos, with approximately eqal nmbers of male and female. They have been scanned with slice thickness between.33 and.5 mm. The segmentation of the endocasts sing itksnap (Yshkevich et al, 26) leads to srface meshes. These srfaces have been rigidly co-registered sing gmmreg (Jian and Vemri, 25).

26 26 It has been observed in Kinzey (1984) that the seqences of teeth emergence in bonobos and chimpanzees are essentially identical. This gives a way to estimate the dental age of each skll. We will se this dental age as a common proxy of growth, and not the tre age of the specimen, which is not available. As a conseqence, each skll has been associated to one the 6 dental ages defined in Shea (1989): infant, child, yong jvenile, old jvenile, sb-adlt and adlt. To refine the classification, we associated some sklls with the intermediate class child/yong jvenile. Age distribtion is shown in Fig 1. Withot loss of generality, we assme that each dental development stage lasts the same amont of time, namely 1 nit of time. Each nit of time has been discretized with 5 time-points, so that the samples are associated to the time-points t i = 5,1,15,2,25,3 according to the dental development of the specimen. The age child/yong jvenile has been associated to the time-point 13. # samples infant child child/yong jvenile Bonobos Chimpanzees yong old sbadlt jvenile jvenile adlt TOTAL Table 1 Distribtion of the dental ages across samples for both species: chimpanzee and bonobo. Obviosly, this database is cross-sectional by natre. It is nthinkable to have several observations of the same wild animals over time. To make the best of this sitation, we choose to estimate first a typical growth scenario for each species independently, applying the regression tool to the cross-sectional data. Second, we analyze the differences between the two growth scenarios sing the spatiotemporal registration to measre both morphological differences and possible developmental delays, a key featre we aim at detecting in regards to the bonobos hypothesis. An alternative approach wold consist in sing the spatiotemporal atlas constrction to estimate an hybrid growth scenario and its deformations to each species, considered as 2 sbjects. One the one hand, this wold prevent biasing the analysis by choosing a reference species and the interspecies comparison wold take into accont the fact that the different age grops have a different nmber of samples. On the other hand, the methodology we choose enables a more direct comparison between species. In particlar, the estimation of the species specific growth scenario, which is done independently for each species, is not constrained by the assmption that the two growth scenarios shold derived from the same hybrid scenario. 5.2 Typical growth scenario estimation for each species We choose the smallest endocast within the child class as the baseline S and associate it to the time point t = 2. Then, we perform a temporal shape regression of the endocasts (independently for each species), as explained in Sec and Algorithm 1. We set the typical spatial interaction between crrents λ W = 1 mm, the spatial scale of deformation consistency λ χ = 2mm and the trade-off between fidelity-to-data and reglarity γ χ = 1 3 mm 2 (nit of time). The diameter of the endocasts are typically between 6 and 7 mm. Significant samples of the species specific growth scenarios are shown in Fig. 16. The complete scenarios can be seen in the companion movies (see Online Resorce 2 (chimpanzees) and 3 (bonobos)). Besides the increase of volme, the most salient effect in both scenarios is an elongation along the posterior/anterior axis and a slight contraction along the sperior/inferior axis. As a conseqence, the endocast which has an almost spherical geometry at birth has an increasingly ellipsoidal geometry. However, it seems that the chimpanzee endocasts have a stronger anisotropy and that this anisotropy increases faster in time. The sbseqent spatiotemporal registration will measre the differences in both scenarios more precisely. The two growth scenarios differ considerably dring infancy and childhood, mainly becase of the small amont of data in infancy. The two infant chimpanzees have a larger endocasts compared to both the infant bonobos and the children chimpanzees. To have a more relevant estimation of the growth in infancy, we expect to scan more infant chimpanzees sklls in the ftre. Note that in the next section we will not take the infancy data into accont and will consider the growth scenarios starting at childhood. We can dedce from the growth scenario an estimation of the evoltion of the endocranial volme across ages, as shown in Fig. 17 (Note that we have not performed a regression of the volme bt of the shapes instead). Besides the evoltion in infancy, one intriging featre is the apparent decrease in endocranial volme of bonobos at sb-adlthood. This featre is also present in the endocranial volme distribtion in the original endocasts (mean and standard deviation are shown in Fig. 17): the mean of the volme at sb-adlthood is smaller than the one of old jveniles. However, the Mann-Whitney U test gives a p-vale of.47 when comparing the volme distribtion of old jveniles and sb-adlts: the median of the two distribtions are not proved to be statistically different. The test rn for every pair of consective distribtions shows a significant increase of volme in only three occasions: (i) between infancy and childhood for the bonobos (p-vale: ); (ii) between childhood and yong jvenility for the chimpanzees (p-vale:.7); (iii) and between old-jvenility and sb-adlthood for the chimpanzees (p-vale:.2).

27 27 Temporal regression of chimpanzees endocasts Temporal regression of bonobos endocasts Fig. 16 Temporal shape regression of endocast of the chimpanzees (top) and the bonobos (bottom) estimated from the original endocasts. In each species, the endocast seems to evolve from a spherical geometry at infancy to an ellipsoidal one at adlthood. However, the dynamics of sch changes seem to differ for both species. The qite nrealistic evoltion of the chimpanzee endocast at infancy is de to the small amont of data at this age (2). Here, only 6 stages of the growth are shown, althogh the estimated scenario is continos. Best seen as movies: see Online Resorce 2 (chimpanzees) and 3 (bonobos) 5.3 Spatiotemporal registration between the two growth scenarios We perform a spatiotemporal registration between the two estimated growth scenarios as explained in Sec and Algorithm 2. For the reasons explained in the previos section, we consider the part of these scenarios - between childhood and adlthood - discarding the portion between infancy and childhood. We consider the chimpanzee growth scenario as the reference scenario (i.e. the sorce). The bonobo scenario is sampled every 2 time-steps. These samples play the role of the target shapes. We set the scale of crrents to λw = 1mm as for the regression estimation. We rn the registration for different sets of parameters and pick the ones which enable to achieve the smallest discrepancy term in (4). This gives the scale of the morphological deformation: λ φ = 1mm, the scale of the time warp λ ψ = 1 nit of time (i.e. dration of one time-point), the spatial power σφ = 4, the temporal power σψ = 5, the morpho- logical trade-off γ φ = 1 5 mm2 and the temporal trade-off γ ψ = 1 5 mm4 /(nit of time). The morphological deformation changes the shape of each frame of the chimpanzee growth as shown in Fig. 18 (this is the eqivalent figre to the first and second row in Fig. 6, althogh we plot here an intermediate step of the deformation). It shows that, independently of the age, the bonobos endocasts are ronder than the chimpanzees. The movie of this deformation clearly shows a twist at the anterior and posterior part of the endocast (see companion movie: Online Resorce 4). The graph of the estimated time warp is shown in Fig. 19. This plot shows the correspondence between the ages of the bonobos and the chimpanzees. It mostly shows an important speed redction of the growth of the bonobos with respect to the chimpanzees between old-jvenility and sbadlthood. The almost constant slope of the crve dring this period of time indicates that the bonobos growth speed is.25 times that of the chimpanzees. The graph shows also that the bonobos seem to be slightly in advance with respect

28 28 Endocast Volme (in mm 3 ) x 1 5 Bonobos Chimpanzees Dental Age infant child yong jvenile old sb adlt jvenile adlt Fig. 17 From the continos shape regression shown in Fig. 16, we dedce an estimation of the evoltion of the endocast volme dring growth. Mean and standard deviation of the volme of the original endocasts are sperimposed. The intriging decrease of volme of bonobos at sb-adlthood is not shown to be statistically significant. The nrealistic regression at infancy of chimpanzees is de to the very small nmber of samples at this age (2). dental age of the chimpanzees old jvenile sb adlt adlt yong jvenile child Time Warp Limits of the 9% confidence interval (bootstrap) Limits of the 9% variation interval de to random age shifts Bonobos in advance w.r.t. Chimpanzees child slope =.24±.1 yong jvenile old jvenile sb adlt dental age of the bonobos Bonobos delayed w.r.t. Chimpanzees adlt Fig. 19 Time warp between chimpanzee and bonobo growth. It shows that the growth of the bonobos is in advance with respect to the chimpanzees at childhood and then that it drastically slows down dring jvenility (almost linearly by a factor.24 between old-jvenility and sb-adlthood). This delay seems to decrease at adlthood. Dashed magenta lines indicate the limits of the 9% confidence interval estimated by bootstrap. Dashed cyan lines indicate the limits of the 9% variation intervals de to random age shifts. Fig. 18 Morphological part of the spatiotemporal registration between chimpanzees and bonobos growth scenarios. The morphological deformation maps the morphological space of the chimpanzees to that of the bonobos, independently of the age. It is applied here to the chimpanzees endocasts at old jvenility: endocasts from the chimpanzees growth scenario (top row), their deformation to the bonobos space (bottom row) with an intermediate stage of deformation (middle row). This shows that, on average, the endocast of a chimpanzee is more elongated and less rond than the one of a bonobo. Note that the deformed endocasts do not match the ones of the bonobo growth at the same age, bt at the age given by the correspondence graph shown in Fig. 19. Best seen as a movie: see Online Resorce 4 to the chimpanzees at childhood and that the delay of the bonobos growth at sb-adlthood seems to be redced at adlthood. We show the effect of this spatiotemporal registration on the evoltion of endocast volme of the chimpanzees in Fig. 2(left). It shows that the spatiotemporal warping enables to match the volme evoltion of the chimpanzees endocast closer to that of the bonobos endocast. Besides volme, we also analyze the differences of a measre of the shape, namely the ratio between the height (in the sperior-inferior direction) and the width (in the anteroposterior direction) of the endocast. Ratio close to 1 indicates a ronded endocast in the sagittal plane. The evoltion of the ratio compted from the species specific scenario is shown in Fig. 2(right): the decrease of the crves indicates that the endocast become more and more asymmetric (ellipsoidal) dring growth. This ratio is always smaller for the chimpanzees than for the bonobos, ths showing a stronger asymmetry for the chimpanzees. As expected, the morphological deformation moves the crves of the chimpanzees closer to the one of the bonobos, except between sb-adlthood and adlthood. Indeed, we have shown in Fig. 18 that the morphological deformation tends give the endocast a more ellipsoidal aspect. The time warp tends to align the slope of the chimpanzee crve to the slope of the bonobos crve. However, we notice that the spatiotemporal warping aligns the evoltion of this ratio with less accracy than that of the volme (see Fig. 2(left)).