A Hierarchical Convolutional Neural Network for Mitosis Detection in Phase-Contrast Microscopy Images

Transcription

1 A Hierarchical Convolutional Neural Network for Mitosis Detection in Phase-Contrast Microscopy Images Yunxiang Mao and Zhaozheng Yin (B) Department of Computer Science, Missouri University of Science and Technology, Rolla, USA Abstract. We propose a Hierarchical Convolution Neural Network (HCNN) for mitosis event detection in time-lapse phase contrast microscopy. Our method contains two stages: first, we extract candidate spatial-temporal patch sequences in the input image sequences which potentially contain mitosis events. Then, we identify if each patch sequence contains mitosis event or not using a hieratical convolutional neural network. In the experiments, we validate the design of our proposed architecture and evaluate the mitosis event detection performance. Our method achieves 99.1 % precision and 97.2 % recall in very challenging image sequences of multipolar-shaped C3H10T1/2 mesenchymal stem cells and outperforms other state-of-the-art methods. Furthermore, the proposed method does not depend on hand-crafted feature design or cell tracking. It can be straightforwardly adapted to event detection of other different cell types. 1 Introduction Analyzing the proliferative behavior of stem cells in vitro plays an important role in many biomedical applications. Most of the analysis methods use fluorescent, luminescent or colorimetric microscopy images which are acquired by invasive methods, such as staining cells with fluorescent dyes and radiating them with the specific wavelength light. The invasive method damages cells viability or kills cells, which is not suitable for continuously monitoring the cell proliferation process. Phase-contrast microscopy, as a non-invasive imaging modality, offers the possibility to persistently monitor cells behavior in the culturing dish without altering them. Quantitatively analyzing the cell proliferation process relies on the accurate detection of mitosis events, in which the genetic material of an eukaryotic cell is equally divided, resulting in daughter cells. In fact, the process of a mitosis event consists of four stages: interphase, start of mitosis, formation of daughter cells, separation of daughter cells, as shown in Fig. 1. According to the four stages, This research was supported by NSF CAREER award IIS , NSF EPSCoR grant IIA , ISC and CBSE centers at Missouri S&T. c Springer International Publishing AG 2016 S. Ourselin et al. (Eds.): MICCAI 2016, Part II, LNCS 9901, pp , DOI: /

2 686 Y. Mao and Z. Yin Fig. 1. The process of a mitosis event. a mitotic cell have the following sequential actions: reduce its migration speed, shrink its size and increase its brightness, appear like a number 8, split into two daughter cells. In this paper, the mitosis detection is defined as detecting the time and location at which the daughter cells first appear (birth moment). 1.1 Related Work Several mitosis detection methods based on phase-contrast microscopy images have been proposed in the past decade. Liu et al. [2] proposed an approach based on Hidden Conditional Random Fields (HCRF) [1] in which mitosis candidate patch sequences are extracted through a 3D seeded region growing method, then HCRF is trained to classify each candidate patch sequence. This method achieves good performance on C3H10T1/2 stem cell datasets. Since only one label is assigned to patch sequence, this HCRF-based approach can classify each patch sequence into mitosis or nor, but it can not accurately localize the birth moment of the mitosis event in the patch sequence. A few extensions have been made on the HCRF-based approach. Huh et al. [3] proposed an Event-Detection CRF (EDCRF) in which each patch in a candidate sequence is assigned with one label. The birth moment of the mitosis event is determined based on the observation that if there exists a change from before mitosis to after mitosis label. Liu et al. [4] utilized a maximum-margin learning framework for training the HCRF and proposed a semi-markovian model to localize mitosis events. Cireşan et al. [5] utilized the DCNN as a pixel classifier for mitosis detection in individual breast cancer histology images. During the histology, the histologic specimens are stained and sandwiched, which makes it not suitable for detecting mitosis events in the time-lapse image sequences. 1.2 Motivation and Contributions The previous mitosis detection approaches either use handcrafted features or consider a single image for the input of DCNN architectures. If we attempt to detect mitosis events by a single image, we may lose the visual appearance change information during the whole process of a mitosis event. Furthermore, motion information hidden in the continuous image sequence can also aid the detection of mitosis event. Thus, we propose a Hierarchical Convolutional Neural Network (HCNN) for the task of mitosis detection, which utilizes the temporal appearance change information and motion information in continuous microcopy images.

3 A HCNN for Mitosis Detection in Phase-Contrast Microscopy Image Methodology Our proposed method takes a video sequence as the input, and detects when and where mitosis events occur in the sequence. It consists of two steps: first, candidate patch sequences that possibly contain mitosis events are extracted from the image sequences; then, each candidate patch sequence is classified by our Hierarchical Convolutional Neural Network (HCNN). 2.1 Mitosis Candidate Extraction The mitosis candidate extraction aims to eliminate the regions in the input image where mitosis events are highly unlikely to occur, and retrieve patch sequences in temporally continuous frames as the input to our classifier. We follow the similar process as in [3], except for that we we run flatfield correction (illumination normalization, [10]) on the observed images and use a Gaussian filter with standard derivation of 3 to smooth the backgroundsubtracted image. The time length of each mitosis event may be quite different. However, the most salient images during the mitosis are just a few images around the birth moment, so we choose a short fixed time length to extract candidate patch sequences. Each candidate patch sequences contains five image patches. The precision of mitosis events is low (1.2 %) after candidate extraction, thus we propose the HCNN in the next section to further improve the performance. 2.2 Hierarchical CNN Architecture The overall architecture of our proposed Hierarchical Convolutional Neural Network (HCNN) is illustrated in Fig. 2. The first set of input contains five consecutive patches in the candidate patch sequence, and the second set of input contains the five corresponding motion images computed by the central finite difference. Each of the ten convolutional neural networks in the first layer (CNN k 1,k [1, 10]) takes a single image as the input. In the second layer of Fig. 2. The overview of our proposed Hieratical CNN architecture

4 688 Y. Mao and Z. Yin Fig. 3. The architecture of CNNs in the first layer of our HCNN. Fig. 4. The architecture of CNNs in the second and last layer of our HCNN. our HCNN, we design two CNNs (CNN2 11 and CNN2 12 ) to learn joint features at the patch-sequence level from original patch sequences and their motion patch sequences separately. In the last layer of our HCNN, combined appearance and motion features are fed into the last CNN (CNN3 13 ) to make the final prediction. In the notation of CNNi k, i denotes the layer in our HCNN and k indexes the CNN out of the total 13 CNNs in our HCNN. The design of such an architecture has two motivations. First, mitosis is a continuous event. Instead of detecting the mitosis events by single frame, leveraging several nearby frames will be more reliable to detect the birth moments of mitosis events. Second, the movement pattern of mitotic cells are different from that of migration cells, thus utilizing the motion information should boost the classification performance. The first layer of our HCNN contains ten CNNs (CNN1 k,k [1, 10]), each of which classifies a single appearance or motion image at different time instants of a mitosis event. The ten CNNs shares the same architecture as shown in Fig. 3. There are three convolutional layers with each followed by a 2 2 max pooling layer. We add one more drop-out layer in case of over-fitting. The prediction layer outputs the label of the input image, indicating if the input image is the image at the specific time instant of a mitosis event. The architecture of CNNs in the second and last layer our HCNN (CNN2 11 CNN2 12 and CNN3 13 ) is shown in Fig. 4. The input to CNN2 11 is the combined features from the Fully-connection Layer 2 of CNN k 1,k [1, 5], leading to a 5120 vector. The input to CNN 12 2 is the combined features from the Fully-connection

5 A HCNN for Mitosis Detection in Phase-Contrast Microscopy Image 689 Layer 2 of CNN1 k,k [6, 10], and the input to CNN3 13 is the combined features from the Fully-connection Layer 3 of CNN2 11 and CNN Hierarchical CNN Training Since the overall HCNN has 13 CNNs, the number of parameters is quite large. If we train the whole HCNN at once, this will increase the training complexity. Given the limited amount of training data, this will also increase the risk of over-fitting. Therefore, we divide the training process into two steps as below Pretraining Each CNN Independently First, we train each CNN in three layers independently. For the first-layer CNNs (CNN k 1,k [1, 10]), we use the trained weights of the first CNN (e.g., CNN 1 1 ) as the initialization for the rest four CNNs (e.g., CNN k 1,k [2, 5]) to achieve faster convergence. When training the 13 CNNs, we set the batch size as 100 and the number of epochs as 20 with the learning rate gradually decreasing from 10 3 to The drop-out rate is set to be 0.5 for all drop-out layers Fine-Tuning Hieratical CNN After each CNN is properly pretrained, we fine-tune the complete HCNN. The prediction layers of CNNs in the first and second layers are bypassed and the error from the third-layer CNN (CNN3 13 ) is back-propagated to all the CNNs to updates the weights. 3 Experiments 3.1 Dataset We evaluate our proposed method in five phase contrast video sequences obtained from [4], with each containing 79, 94, 85, 120 and 41 mitosis cells, respectively. Each sequence consists of 1436 images (resolution: pixels). The location and time of mitosis events in the video sequences are provided as the ground truth. In order to train our HCNN, data expansion is performed to generate more positive training data. For each positive mitosis sequence, we rotate the images every 45 (8 variations), slightly translate the images (e.g., by 5 pixels) horizontally and/or vertically (9 variations), which generates 72 times of the original positive training data. We retrieve negative training sequences by our proposed candidate patch sequence extraction method. At last, the training data are balanced by randomly duplicating some positive data so that the numbers of positive samples and negative samples are even.

6 690 Y. Mao and Z. Yin 3.2 Evaluation Metric We adopt leave-one-out policy in the experiment, i.e., using four sequences for training and the rest one for testing. For testing, we use maximum-suppression to converge all the detection results based on their spatial and temporal locations and confidence scores. We use two evaluation metrics in our experiments. First, we evaluate the performance of mitosis occurrence detection in terms of the mean and standard deviation of precision, recall and F score on the five leaveone-out tests, without examining the timing of birth events. In this case, we define True Positive (TP) as a patch sequence contains a mitosis event, False Positive (FP) as it does not contain a mitosis event, and False Negative as a true positive is classified as negative. Second, the performance of mitosis detection is strictly evaluated in terms of the timing error of birth moments, i.e., those aforementioned true positive patch sequences will be considered as true positive only if the timing error of the mitosis event is equal or less than a certain threshold. The timing error is measured as the frame difference between the detection result and the ground truth. 3.3 Evaluation on the Hierarchical Architecture In this section, we show the effectiveness of each module in the proposed architecture design. We compare the performance of a single-appearance CNN (CNN1 3 ) targeted at the detection of the birth moment, a multi-appearance HCNN with the 5 original image patches as input (CNN1 1 to CNN1 5 + CNN2 11 ), a simple CNN which takes 10-channel images as input and our complete HCNN. As shown in Table 1, because single-appearance CNN cannot capture the temporal appearance change, the F-Score of single-appearance CNN is 5 % points lower than that of the multi-appearance HCNN which classify the whole patch sequence. With only the appearance information as input, the F-Score of multi-appearance HCNN is 10 % points lower than that of our HCNN that further incorporates the motion information. As proven in [9], fusing the temporal information in feature level is better than in input pixel level, thus our HCNN performs better than a simple CNN with 10-channel images as the input. Table 1. Mitosis occurrence detection accuracy of different designs. Model Precision (%) Recall (%) Fscore(%) Our HCNN 99.1 ± ± ± 1.3 CNN with multi-channel input 97.6 ± ± ± 1.2 Multi-appearance HCNN 90.9 ± ± ± 1.4 Single appearance CNN 85.9 ± ± ± 4.7

7 A HCNN for Mitosis Detection in Phase-Contrast Microscopy Image 691 Table 2. Comparison of mitosis detection accuracy. Model Precision (%) Recall (%) Fscore(%) Our HCNN 99.1 ± ± ± 1.3 MM-HCRF+MM-SMM 95.8 ± ± ± 2.0 MM-HCRF 82.8 ± ± ± 1.6 EDCRF 91.3 ± ± ± 0.7 CRF 90.5 ± ± ± 4.4 HMM 83.4 ± ± ± 3.4 SVM 68.0 ± ± ± 1.7 Table 3. Comparison of mitosis event timing accuracy. th Precision Recall Fscore Our HCNN [4] Our HCNN [4] Our HCNN [4] ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Comparisons We compare our method with six state-of-the-arts: Max-Margin Hidden Conditional Random Fields+Max-Margin Semi-Markov Model (MM-HCRF + MM- SMM) [4], EDCRF [3], HCRF [2], Hidden Markov Models (HMMs) [6], and Support Vector Machine (SVM) [7]. As shown in Table 2, our HCNN achieves an average precision of %, recall of and F score of %, which outperforms the state-of-the-arts by a large margin. When evaluating the mitosis detection in term of the timing error of birth event, we use four different thresholds th (1, 3, 5 and 10) to report the precision, recall. As shown in Table 3, our HCNN achieves better performance than (MM-HCRF + MM-SMM) [4]. The reason for that is two-fold. First, in [4], they extract hand-crafted SIFT features [8] from patch images, which is not the most suitable features descriptor compared with CNN; Second, their method labels each patch in the whole progress of mitosis, but the early frames and last frames may introduce noise in the model since the appearance representation of them are not clear. While we only focus on consecutive frames near the birth event, the appearance representations of these frames are clear and easy to be captured. 4 Conclusion In this paper, we propose a Hierarchical Convolutional Neural Network (HCNN) for mitosis event detection in phase-contrast microcopy images. We extract candidate patch sequences from the image sequence as the input to HCNN. In our

8 692 Y. Mao and Z. Yin HCNN architecture, we utilize both the appearance information and temporal cues hidden in patch sequences to identify the birth event of mitotic cells. Given the complex HCNN structure, we propose an efficient training methodology to learn the parameters inside HCNN and prevent the risk of over-fitting. In the experiments, we prove that the design of our HCNN is sound and our method outperforms other state-of-the-art by a large margin. References 1. Quattoni, A., et al.: Hidden conditional random fields. IEEE Trans. Pattern Anal. Mach. Intell. 29(10), (2007) 2. Liu, A., et al.: Mitosis sequence detection using hidden conditional random fields. In: Proceedings of IEEE International Symposium on Biomedical Imaging (2010) 3. Huh, S., et al.: Automated mitosis detection of stem cell populations in phasecontrast microscopy images. IEEE Trans. Med. Imaging 30(3), (2011) 4. Liu, A., et al.: A semi-markov model for mitosis segmentation in time-lapse phase contrast microscopy image sequences of stem cell populations. IEEE Trans. Med. Imaging 31(2), (2012) 5. Cireşan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Mitosis detection in breast cancer histology images with deep neural networks. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI LNCS, vol. 8150, pp Springer, Heidelberg (2013). doi: / Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), (1989) 7. Suykens, J., et al.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), (1999) 8. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), (2004) 9. Karpathy, A., et al.: Large-scale video classification with convolutional neural networks. In: Proceedings of CVPR (2014) 10. Murphy, D.: Fundamentals of Light Microscopy and Electronic Imaging. Wiley, New York (2001)