Pairwise active appearance model and its application to echocardiography tracking

Transcription

1 Pairwise active aearance model and its alication to echocardiograhy tracking S. Kevin Zhou 1, J. Shao 2, B. Georgescu 1, and D. Comaniciu 1 1 Integrated Data Systems, Siemens Cororate Research, Inc., Princeton, NJ, USA {shaohua.zhou, bogdan.georgescu, dorin.comaniciu}@siemens.com 2 Center for Automation Research, University of Maryland, College Park, MD, USA {shaojie}@cfar.umd.edu Abstract. We roose a airwise active aearance model (PAAM) to characterize statistical regularities in shae, aearance, and motion resented by a target that undergoes a series of motion hases, such as the left ventricle in echocardiograhy. The PAAM deicts the transition in motion hase through a Markov chain and the transition in both shae and aearance through a conditional Gaussian distribution. We learn from a database the joint Gaussian distribution of the shaes and aearances belonging to two consecutive motion hases (i.e., a air of motion hases), from which we analytically comute the conditional Gaussian distribution. We utilize the PAAM in tracking the left ventricle contour in echocardiograhy and obtain imroved tracking results in terms of localization accuracy when comared with exert-secified contours. 1 Introduction Characterizing shae, aearance, and motion is an imortant research toic in medical imaging alications. There is a wide literature on this toic; we here only focus on one articular tye of aroach active models. Active shae model (ASM) [1] deicts shae statistics using rincial comonent analysis (PCA). Active aearance model (AAM) [2] extends the ASM to model the aearance too with both shae and aearance are jointly modeled by PCA. The ASM and AAM are alicable to images only. To deal with a video, active aearance motion model (AAMM) [3] extends the AAM to characterize the motion in the video and is used for segmenting a satiotemoral object. One restriction of the AAMM is that no global motion is allowed before neighboring frames; hence the AAMM is not alicable to online tracking. Attemting to solving the tracking task, we resent a novel model called airwise active aearance model (PAAM) that characterizes shae, aearance and motion in one treatment. We aly the PAAM for tracking the left ventricle in 2D cardiac ultrasonograhy (or echocardiograhy). Echocardiograhy tracking [4 1] is challenging The work was done when Shao was with SCR in summer 25. We thank Dr. S. Krishnan of Siemens Medical Solutions for roviding the data.

2 t ED t ES (a) t ED +T (b) Fig. 1. (a)a cardiac cycle is divided into P = 9 motion hases. The blue curve shows the LV volume. (b) A ictorial illustration of the PAAM. due to severe imaging artifacts. Artifacts arise from ultrasound seckle noise (due to signal refraction and reverberation), signal droout, and are also characterized by missing, fake, and imroerly located anatomic structures. The left ventricle (LV) aearance changes are caused by fast movement of heart muscle, resiratory inferences, unnecessary transducer movement, etc. With the aid of the PAAM, we successfully regularize the otical flow measurement and obtain imroved shae tracking results. 2 Pairwise active aearance model Assuming that the target of interest undergoes a series of P motion hases indexed by = {1, 2,..., P }. Fig. 1(a) shows an examle of dividing a cardiac cycle into four equally sace motion hases in systole and five motion hases in diastole. Fig. 1(b) illustrates the underlying rincile of the PAAM. (i) The PAAM deicts the transition in motion hase through a Markov chain: it either stays at the current motion hase or roceeds to the next one. For examle, in the cardiac examle, given the end of diastole (ED) and the end of systole (ES) frames, one can easily determine which motion hase the current frame belongs to. (ii) The PAAM deicts the transition in both shae and aearance through a conditional Gaussian distribution. We learn from a database the joint Gaussian distribution of the shaes and aearances belonging to two consecutive motion hases (i.e., a air of motion hases), from which we analytically comute the conditional Gaussian distribution. 2.1 Learning the PAAM The shae is reresented by M s landmark oints, or equivalently a 2M s -dimensional vector S. The aearance A is reresented by an M g -dimensional vector. We concatenate the shae and aearance vectors at two consecutive motion hases to form aired data: s = [S T S T 1] T and a = [A T A T 1] T, where {1, 2,..., P } is the hase index. We assume that S. = SP and A. = AP. We follow the rocedure of learning the AAM for each air of motion hases: (i) Construct the shae subsace based on s using the rincial comonent analysis (PCA). The subsace can be reresented by: s s + P <s> b <s>, (1)

3 where P <s> is a subsace matrix (eigenvectors) describing a sufficient fraction of the total shae variation, b <s> is a vector containing the combination coefficients for each of the eigenvectors. (ii) Similarly, construct the aearance subsace based on a using the PCA. a ā + P <a> b <a>. (2) (iii) Aly a third PCA to the combination of shae and aearance: [ ] [ ] b b = <s> Q <s> W <a> b <a> Q c = Q <a> c, (3) where W <a> is a diagonal matrix that balances the energy discreancy between the shae and aearance models, Q is the eigenvector matrix, and c is a latent vector that controls both the shae and aearance models. We recaitulate the PAAM in a statistical jargon. Denote both shae and aearance by z = [S T, A T ] T. For the th air of motion hases, its distribution (z, z 1 ) = (S, A, S 1, A 1 ) is Gaussian, whose mean and covariance matrix are exressed as: µ = [ µ <z> µ <z> 1 ] [ Σ <z>,, Σ = Σ <z>, 1 Σ <z> 1, Σ<z> 1, 1 It is easy to see that the conditional robability (z z 1 ), which is actually used in tracking, is also Gaussian with mean and covariance matrix given as: µ <z> 1 = µ<z> + Σ <z>, 1 [Σ<z> 1, 1 ] 1 (z 1 µ <z> 1 ), (4) Σ <z> 1 = Σ<z>, Σ <z>, 1 [Σ<z> 1, 1 ] 1 Σ <z> 1,. (5) In ractice, when the Gaussian assumtion is not satisfactory, we grou the data into several clusters and learn the PAAM for each cluster to handle ossible data nonlinearity. ]. 2.2 Using the PAAM in tracking Tracking algorithms can be broadly divided into two categories, deending on the way in which online observations and offline learned models are integrated. (i) The models are embedded into the so-called observation likelihood. The motion arameters are used to deform the observation to best fit the likelihood. An examle is the famous active aearance model (AAM) [2]. (ii) Generic otical flow comutation is first conducted for each landmark; learned models are then alied to regularize the overall shae. An examle is the work of Zhou et al. [1], which is referred to as fusion aroach. We follow [1] due to its flexibility. The fusion aroach consists of two rocesses: observation and fusion. The observation rocess comutes otical flow for individual landmarks and the fusion rocess regularizes the whole contour. In this aer, we mainly focus on the fusion rocess. In the observation rocess, we utilize our earlier aroach

4 [9] where a nonarametric local aearance model (NLAM) is constructed on the fly to model the shae and aearance at a oint level. The outut of the observation rocess is the location and covariance matrix of the landmarks as well as the aearance and its uncertainty. At time instant t, the fusion rocess derives an otimal solution z t that minimizes the fusion cost d 2 t t 1 = d2 t t 1,1 + d2 t t 1,2, where d 2 t t 1,i = (z t z t t 1,i ) T C 1 t t 1,i (z t z t t 1,i ); i = 1, 2, (6) and z t t 1,i and C t t 1,i are the mean vector and covariance matrix, resectively. The first distance d 2 t t 1,1 in (6) arises from the observation rocess that rovides the mean vector z t t 1,1 and the covariance matrix C t t 1,1. The second distance d 2 t t 1,2 in (6) is from the PAAM (refer to (4) and (5)). There are two ossible situations from time t 1 to t: (a) there is no transition in the motion hase, i.e., staying at the same motion hase ; or (b) there is a transition in the motion hase from 1 to. z t t 1,2 = µ <z>, C t t 1,2 = Σ <z>, ; if (a). (7) z t t 1,2 = µ <z> 1, C t t 1,2 = Σ <z> 1 ; if (b). (8) When evaluating the above µ <z> 1 exactly defined in (4), we use z 1 = z t 1. It is easy to determine (a) or (b) in echocardiograhy by using the cardiac eriod T, the ED frame t ED, and the ES frame t ES. All these information is directly available from the video sequence file. We observe that when there is a motion transition, using the conditional robability (z z 1 ) is beneficial because z t t 1,2 is always udated during the iterations and hence adative to the revious observation z t 1. On the other hand, the covariance matrix Σ <z> 1 is fixed and hence re-comutable during training, which imroves comutational efficiency. Usually, C t t 1,2 is singular due to the high dimensionality of the shae and aearance vectors, thereby leading to a non-orthogonal subsace rojection roblem. Suose the rank of C t t 1,2 is q and its rank-q SVD is C t t 1,2 = U q Λ q U T q, the best fusion estimator that minimizes the fusion cost d 2 t t 1 is the so-called best linear unbiased estimate [1]: z t = U q (U T q C 1 t t 1,1 U q + Λ 1 q ) 1 (U T q C 1 t t 1,1 z t t 1,1 + Λ 1 q U T q z t t 1,2 ). (9) In ractice, because we cluster the data and learn several sub-models for each air of motion hases, the sub-model with the smallest fusion cost is selected. 3 Exerimental results We have 4 A4C (aical four-chamber) sequences and 32 A2C (aical twochamber) sequences. In total, there are about 1 A4C frames and about 92 A2C frames. We used 5-fold cross validation for erformance evaluation. The

5 = 1 = 6 c = 1 c = 2 c = 3 = 1 = 6 c = 1 c = 2 c = 3 Fig. 2. Examle of shae and aearance subsaces of the trained PAAM. In our exeriments we trained three sub-models for each air of motion hases. c: cluster index, : hase index. Rows in the figure corresond to clusters; columns corresond to hases. In the shae model, the red dot lines reresent the means of the subsaces, while the three blue solid lines in each lot reresent three eigenvectors associated with the to three eigenvalues in the corresonding subsaces. ground truth contours were generated by exerts. During testing, we assumed manual initialization at the middle frame between the ED and ES frames. [Prerocessing] Before training, we erformed the following rerocessing stes: (a) Video frames are samled and classified to different hases. Global aearance atches are croed out from each samled frame and then rigidly aligned to a mean shae in a 4 temlate using the thin-late slines waring algorithm. (b) Since echocardiograms have highly non-gaussian intensity histograms, we alied a nonlinear ultrasound-secific normalization method [3] to transform the non-gaussian intensity histogram to have a normal distribution. However, since this is only for the aearance, the joint sace of shae and aearance is hardly Gaussian even after this transformation. (c) The shae consists of 17 control oints, which means that the dimensionality of the shae vector is 34. The aearance atch contains 4 = 2 ixels. Since such a high dimension requires exensive comutation, we alied a rerocessing PCA to reduce the dimensionality of the aearance from 2 to around, before feeding them to train the PAAM. Using the rerocessed data, we trained the PAAM with P = 9 comonents, each comonent containing three sub-models. Fig. 2 illustrates the learned shae and aearance subsaces. [Two contour distances] To evaluate the tracking erformance, we need to measure the roximity between two contours. Rather than using the rigid Euclidean distance to measure the distance between two landmark oints, we roose a segmental Hausdorff distance (seghd) that allows certain degree of non-rigidity. As illustrated in Fig. 3(a), the seghd between two corresonding

6 (a) (b) Fig. 3. (a) Segmental Hausdorff distance. (b) Surrisal vector distance. (a) Fig. 4. The snashots of the tracking results of (a) an A4C sequence and (b) an A2C sequence. (b) landmark oints x and x on the two curves C and C, resectively, is defined as the Hausdorff distance (HD) between two segments ω(x) and ω(x ), where ω(x) defines a segment around x on the curve C. We further take the mean of the seghd of all landmarks as the distance between C and C, denoted by d seghd (C, C ). shd(x, x ) = HD(ω(x), ω(x )); d seghd (C, C ) = { shd(x, x )dc}/{ dc}. (1) x x The seghd measures only the hysical distance between two contours, ignoring their curvedness. Even when the two contours C and C have the same distance to the ground truth contour C in terms of d seghd, C and C can be differently erceived because they resent different curvedness. Feldman and Singh roosed [11] a surrisal vector sv to quantify how the curve is erceived. Fig. 3(b) illustrates the surrisal vector. The direction of sv is the same as the outward normal direction and the magnitude sv is a function of curvature. When at the highly-curved art of the contour, the sv is large; when at the flat art, it is small. Using the surrisal vector, we comute a surrisal vector distance d sur (C, C ) to characterize the roximity of two contours in their curvedness. sur(x, x ) = sv(x) sv(x ) 2 ; d sur (C, C ) = { sur(x, x )dc}/{ dc}. (11) [Tracking erformance] We first comared four methods whose results are reorted in Table 1 using the median and standard deviation of the contour distances for all testing video sequences in five folds. The SSD means the x x

7 (a) Sequences Segmental Hausdorff distance d seghd (ixels) SSD CD 2 NLAM PAAM A2C ± ± ± ±.6623 A4C ± ± ± ±.55 (b) Sequences Segmental Hausdorff distance d seghd (ixels) ASM AAM PASM PAAM A2C ± ± ± ±.6623 A4C ± ± ± ±.55 (c) Sequences Surrisal vector distance d sur SSD CD 2 NLAM P-AAM A2C.324 ± ± ± ±.11 A4C.324 ± ± ± ±.97 Table 1. Tracking erformance based on (a,b) the segmental Hausdorff distance and (c) the surrisal vector distance using (a,c) the SSD, CD 2, NLAM, and PAAM methods and (b)the ASM, AAM, PASM, and PAAM methods. general otical flow method using the sum of squared distance similarity function; the CD 2 using the similarity function in [8], which considers a simlified ultrasound image formation; and the NLAM using the method in [9]. The PAAM means regularizing the NLAM results using the PAAM. From Table 1(a), we observe that the NLAM imroves the tracking results significantly in terms of the seghd, comared with SSD and CD 2. Using the PAAM further decreases the seghd by some margin. The advantage of using the PAAM is highlighted when the surrisal vector distance is used. Using the NLAM only often results in a wiggly contour as every landmark is tracked indeendently. However, the PAAM successfully regularizes the wiggly contour into a smooth one. This regularization is quantized by the surrisal vector distance: the PAAM yields significant lower error as reflected in Table 1(c). Fig. 4 shows the tracking contours overlaid on samle frames of an A4C sequence and an A2C sequence. Next, we show that the effectiveness of shae, aearance and motion information when used as rior knowledge. Table 1(b) shows the erformance after regularizing the NLAM results using four different rior models in the fusion rocess. The ASM means using the hase-searate active shae model (ASM) only, without the airwise model. In other words, we trained ASMs for each of the nine hases. No motion and aearance information is interreted in the model. The AAM model means using the hase-searate AAM only, which takes into account shae and aearance. The third model uses the PASM (airwise ASM) model, with shae and motion but no aearance information involved. The last model is the PAAM model that jointly considers shae, aearance and motion. Table 1(b) tells that using more rior information results in decreased tracking error. It also indicates the order of the imortance of the three elements: shae > aearance > motion. For examle, the fact that the AAM rovides better erformance than the PASM suggests that the aearance information contributes more to the entire system than the motion information. [Comarison with the AAMM] We summarize the main differences between the AAMM and PAAM since both are able to cature shae, aearance, and motion. First, the AAMM is suitable to segment a satiotemoral target

8 in a sequence, but hardly fits to an online tracking task. Second, the AAMM assumes that the motion only comes from the heart beating. Little or no motion is introduced by external factors such as ultrasound transducer movement that is always resent in ractice. Third, the AAMM lacks adatability to different cases since it falls in the observation exlains model category. Finally, the AAMM is very high-dimensional, causing ineffective modeling caability due to difficulty in collecting enough data to cover desired variations, and exensive comutations in both training and testing. The roosed PAAM contains the romising roerties of the AAMM, with flexibility and adativity integrated. It also enhances the modeling caability and comutational efficiency. 4 Conclusion We have roosed the PAMM to reresent shae, aearance, and motion information. The shae and aearance knowledge is described by the model subsaces, while the inter-hase motion is described by aired data. We integrated the model into a fusion algorithm for tracking. In the exeriments, we alied the tracker to a large study of LV tracking and demonstrated robustness and accuracy, using the segmental Hausdorff distance and surrisal vector distance, in tracking both A4C and A2C echocardiograhic sequences. References 1. Cootes, T., Taylor, C.: Active shae models - smart snakes. In: BMVC. (1992) 2. Cootes, T., Edwards, G., Taylor, C.: Active aearance models. PAMI 23(6) (21) Bosch, J., et al.: Automatic segmentation of echocardiograhic sequences by active aearance motion models. IEEE Trans. Medical Imaging 21 (22) Mikic, I., Krucinski, S., Thomas, J.: Segmentation and tracking in echocardiograhic sequences: Active contours guided by otical flow estimates. IEEE Trans. Medical Imaging 17 (1998) Jacob, G., Noble, A., Blake, A.: Robust contour tracking in echocardiograhic sequence. In: Proc. Intl. Conf. on Comuter Vision. (1998) Ledesma-Carbayo, M., et al.: Cardiac motion analysis from ultrasound sequences using non-rigid registration. In: MICCAI. (21) Jacob, G., et al.: A shae-sace-based araoch to tracking myocardial borders and quantifying regional left-ventricular function alied in echocardiograhy. IEEE Trans. Medical Imaging 21 (22) Boukerroui, D., Alison, J., Brady, M.: Velocity estimation in ultrasound images: A block matching aroach. In: IPMI. (23) Georgescu, B., Zhou, X., Comaniciu, D., Rao, B.: Real-time multi-model tracking of myocardium in echocardiograhy using robust information fusion. In: MICCAI. (24) Zhou, X.S., Comaniciu, D., Guta, A.: An information fusion framework for robust shae tracking. PAMI 27(1) (25) Feldman, J., Singh, M.: Information along contours and object boundaries. Psychological Review 112 (25)