Fine-grained Classification

Transcription

1 Fine-grained Classification Marcel Simon Department of Mathematics and Computer Science, Germany Seminar Talk

2 Outline 1 Motivation Marcel Simon Fine-grained Classification 1

3 Outline 1 Motivation Marcel Simon Fine-grained Classification 2

4 Motivation Three birds, but only two species. Which two images show the same species? High intra-class, low inter-class variance! Marcel Simon Fine-grained Classification 3

5 Object Parts Required for every kind of localized features Problem: identification and robust detection Additional challenge: ambiguous location Marcel Simon Fine-grained Classification 4

6 Part Proposals from CNNs Pretrained CNNs contain inherent part detectors Part detectors are generic, shared among all classes of ImageNet Task: unsupervised selection of relevant part detectors for each object category Input images CNN Neural activation maps 256 part proposals Input Output ACCV Marcel Simon Fine-grained Classification 5

7 Random Part Selection Assumption: part detection are generic interest point detectors specialized on a specific pattern In classification compute features at these interesting areas 256 Part proposals Random selection Result: 8 selected parts

8 Model-based Part Selection Example: which part detector is relevant for birds? Idea: select parts which fit a part constellation model 256 Part proposals Part constellation model View 1: View 2 Result: 8 selected parts

9 Part Constellation Model Assuming constant normalized distance between parts Part locations are Gaussian distributed with mean relative to anchor Anchor Point Mean Part Locations Relative Offset Marcel Simon Fine-grained Classification 8

10 Learning Constellation Model Given the part proposal locations µ, estimate part model parameters Γ: ˆΓ = argmax Γ p(γ µ) =... = argmin Γ M N P i=1 p=1 v=1 V s i,v b v,p h i,p µ i,p (a i +d v,p ) 2 Marcel Simon Fine-grained Classification 9

11 Solving the Problem = argmin Γ M N P i=1 p=1 v=1 V s i,v b v,p h i,p µ i,p (a i +d v,p ) 2 }{{} t i,v,p {0,1} Solved by iteratively optimizing each variable independently Intuitive solutions for each variable, for example: N ˆd v,p = t i,v,p (µ i,p a i )/ ( n t i,v,p). i=1 i =1 Marcel Simon Fine-grained Classification 10

12 Classification Pipeline Part Selection Part proposals Part selection Feature Extraction Detect parts Feature extraction SVM Marcel Simon Fine-grained Classification 11

13 Results Train. Test Method Anno. Anno. CUB Birds 200 classes, images Accuracy Parts Bbox Göring et al. (2014) 57.8% Parts Bbox Simon et al. (2014) 62.5% Parts Bbox Donahue et al. (2014) 64.9% Bbox None Simon et al. (2014) 53.8% None None Xiao et al. (2015) (VGG19) 77.9% None None Ours, constellation (AlexNet) 68.5% None None No parts (VGG19) 71.9% None None Ours, random (VGG19) 79.4% None None Ours, constellation (VGG19) 81.0% After publication with citation: None None Google % None None Baidu %

14 Results Oxford flowers 102 classes, 8189 images Method Accuracy Angelova and Zhu (2013) 80.7% Murray and Perronnin (2014) 84.6% Azizpour et al. (2014) 91.3% No parts (AlexNet) 90.4% Ours, random (AlexNet) 90.3 ± 0.2% Ours, constellation (AlexNet) 91.7% No parts (VGG19) 93.1% Ours, random (VGG19) 94.2 ± 0.2% Ours, constellation (VGG19) 95.3% After publication with citation: Baidu % NA Birds 555 classes, images Train. Test Method Anno. Anno. Accuracy Parts Parts Van Horn et al. (2015) 75.0% None None No parts (GoogLeNet) 63.9% None None Ours, const. (GoogLeNet) 76.3% Marcel Simon Fine-grained Classification 13

15 Generic Classification Datasets Approach applicable to all classification datasets This is a large step compared to specialized fine-grained approaches Method Caltech classes, images Accuracy Zeiler and Fergus (2014) 74.20% Chatfield et al. (2014) 78.82% Simonyan and Zisserman (2014) (VGG19) 85.1% No parts (AlexNet) 71.44% Ours, random (AlexNet) 72.39% Ours, constellation (AlexNet) 72.57% No parts (VGG19) 82.44% Ours, constellation (VGG19) 84.10% Marcel Simon Fine-grained Classification 14

16 Influence of Number of Parts CUB Birds, VGG19, 256 available parts Accuracy in % Ours, constellation Ours, random parts Number of parts used Marcel Simon Fine-grained Classification 15

17 CNN part proposals Constellation model Random selection - Part constellation models for part proposal selection 81.0% on CUB , 76.3% on NA birds, no annotation More information: Marcel Simon Fine-grained Classification 16

18 References References References I Angelova, A. and Zhu, S. (2013). Efficient object detection and segmentation for fine-grained recognition. In CVPR. Azizpour, H., Razavian, A. S., Sullivan, J., Maki, A., and Carlsson, S. (2014). From generic to specific deep representations for visual recognition. CoRR, abs/ Chatfield, K., Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Return of the devil in the details: Delving deep into convolutional nets. In BMVC. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014). Decaf: A deep convolutional activation feature for generic visual recognition. In ICML. Göring, C., Rodner, E., Freytag, A., and Denzler, J. (2014). Nonparametric part transfer for fine-grained recognition. In CVPR. Murray, N. and Perronnin, F. (2014). Generalized max pooling. In CVPR. Simon, M., Rodner, E., and Denzler, J. (2014). Part detector discovery in deep convolutional neural networks. In ACCV. Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. CoRR, abs/ Marcel Simon Fine-grained Classification 17

19 References References References II Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., and Belongie, S. (2015). Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, pages Xiao, T., Xu, Y., Yang, K., Zhang, J., Peng, Y., and Zhang, Z. (2015). The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In CVPR. Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In ECCV. Marcel Simon Fine-grained Classification 18

20 References References Image References Bird images are taken from the CUB Dataset Marcel Simon Fine-grained Classification 19