Improving Image Similarity With Vectors of Locally Aggregated Tensors (VLAT)

Transcription

1 Improving Image Similarity With Vectors of Locally Aggregated Tensors (VLAT) David Picard and Philippe-Henri Gosselin ETIS / ENSEA - University of Cergy-Pontoise - CNRS UMR 8051 F Cergy-Pontoise Cedex, France 12 September 2011

2 Visual Object Classification Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

3 Plan 1 Vector representations using codebooks 2 Kernels on Bags 3 Vectors of Locally Aggregated Tensors 4 Experiments 5 Conclusion Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

4 Plan 1 Vector representations using codebooks 2 Kernels on Bags 3 Vectors of Locally Aggregated Tensors 4 Experiments 5 Conclusion Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

5 Codebooks Kernels on Bags VLAT Experiments Conclusion Descriptors Bags of Descriptors 1 Extraction of Zones of Interest Regions, Keypoints,... 2 Description of Zones of Interest Colors, Textures HoG, SIFT,... 3 Bag of Descriptors Bi = {bri }r bri D : descriptor r of image i Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

6 Vector representations using codebook Vector representations using codebook 1 Build a code book K-Means, Gaussian Mixture Models,... 2 Project Bags of Descriptors on the codebook Hard assignment Soft assignment... 3 Vector representation w i R m, with m words w ri R : weight of word r of image i Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

7 Vector representations using codebook Vector representations using codebook Advantages Small representation Low memory consumption Fast similarities Compatible with usual classifiers (knn, SVM,...) Drawbacks Performance depends on the codebook A lot of words are required (from 1k to 100k) Codebook computation in high dimensional spaces is difficult and unstable Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

8 Plan 1 Vector representations using codebooks 2 Kernels on Bags 3 Vectors of Locally Aggregated Tensors 4 Experiments 5 Conclusion Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

9 Kernels on Bags Change initial space Work directly on Bags of Descriptors B i = {b ri } r, b ri D Perform matching computations in a new space H Use a mapping function φ : D H D : visual space (where descriptors are) H : induced space (where matching is performed) Vector representation φ(b ri ) in the induced space H High dimension vectors (sometimes infinite) Matching is easier in large spaces Implicit use of induced space H using a kernel function : k(b ri, b sj ) = φ(b ri ), φ(b sj ) Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

10 Kernels on Bags Soft maximum (Shawe-Taylor) K softmax (B i, B j ) = b ri B i b sj B j k(b ri, b sj ) k kernel function K softmax kernel function Compare barycenters of bags : Φ(B i ) = φ(b ri ) b ri B i K softmax (B i, B j ) = Φ(B i ), Φ(B j ) = b ri B i b sj B j φ(b ri ), φ(b sj )) Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

11 Kernels on Bags Kernels on Bags Advantages As efficient as vote-based systems No large codebook Stable Easy parameter tuning Compatible with usual classifiers (knn, SVM,...) Drawbacks High computational complexity Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

12 Plan 1 Vector representations using codebooks 2 Kernels on Bags 3 Vectors of Locally Aggregated Tensors 4 Experiments 5 Conclusion Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

13 First idea : Approximation of Gaussian Kernel on Bags Taylor expansion : K (B i, B j ) = r,s = r,s e 2 2<b ri,b sj > σ 2 e σ 2 α p (< b ri, b sj >) p p Vectorization with Tensors : K (B i, B j ) = α p < p b ri, p b sj > r,s p = α p < p b ri, p b sj > p r s Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

14 Vectors of Locally Aggregated Tensors Second idea : Divide and center Inspired from Vectors of Locally Aggregated Descriptors (VLAD) method : Divide descriptor space in a few partition (from 16 to 64) Kernels on Bags are more efficient with smaller bags Clustering in high dimension spaces with few words is easy and stable Center descriptors in each cluster Relative comparisons are more accurate (cf Fisher kernels) Perform a matching in each cluster Sum up the matching results Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

15 Vectors of Locally Aggregated Tensors Proposed method Descriptor computation for order p : 1 Compute a small codebook and get centers c n 2 For each image i Compute tensor for each center c n Tn p (B i ) = p (b ri c n ) b ri such that NN(b ri )=c n 3 Concatenate : T p (B i ) = (T p 1 (B i)... T p m(b i )) 4 Normalize : ˆT p (B i ) = T p (B i )/ T p (B i ) L2 Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

16 Plan 1 Vector representations using codebooks 2 Kernels on Bags 3 Vectors of Locally Aggregated Tensors 4 Experiments 5 Conclusion Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

17 Image classification experiments Visual Object Classes Challenge 2007 Pascal Network Network of Excellence 9,963 images 20 categories SIFT descriptors Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

18 Image classification experiments Evaluation Protocol SVM Classifier Triangle kernel with L 2 distance Training on official train+val sets (5011 images) Testing on official test set (4952 images) Performance evaluation with official tool (Average Precision) Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

19 BOF VLAD VLAT Category 4k aeroplane ,2 61,0 62,0 64,6 65,9 66,1 bicycle ,3 38,6 40,3 45,3 47,2 49,2 bird ,7 34,6 35,1 38,1 39,6 39,5 boat ,7 55,8 55,3 57,4 59,5 58,7 bottle ,9 16,0 17,2 17,2 18,5 19,1 bus ,0 44,8 47,6 46,8 48,3 48,9 car ,8 68,3 69,2 70,5 71,1 71,0 cat ,2 37,9 40,9 40,8 41,1 43,1 chair ,2 40,2 40,4 41,3 43,1 42,8 cow ,7 22,7 25,3 25,2 26,9 27,3 diningtable ,3 25,9 28,3 25,7 30,4 30,4 dog ,1 32,4 34,0 36,6 37,0 37,3 horse ,8 65,2 66,7 70,7 70,6 70,8 motorbike ,2 47,2 49,2 49,6 51,1 51,3 person ,6 77,4 78,0 79,4 80,1 79,9 pottedplant ,9 14,4 17,0 14,9 15,8 16,7 sheep ,2 26,1 28,6 23,1 28,2 28,2 sofa ,3 36,2 35,0 39,4 39,7 38,9 train ,7 64,6 64,3 66,1 66,3 65,3 tvmonitor ,3 34,9 38,3 38,1 42,3 42,4 all ,4 42,2 43,6 44,5 46,1 46,4 Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

20 Image search experiments Holidays database BoW HE VLAD VLAT linear metric no no yes yes BoW = Bags of Words HE = Hamming Embedding VLAD = Vectors of Locally Aggregated Vectors VLAT = Vectors of Locally Aggregated Tensors (proposed) Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

21 Plan 1 Vector representations using codebooks 2 Kernels on Bags 3 Vectors of Locally Aggregated Tensors 4 Experiments 5 Conclusion Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22

22 Conclusion Vectors of Locally Aggregated Tensors (VLAT) Vectorization of Kernels on Bags Faster than Kernels on Bags Easier tuning than Bag of Words State of the art performance Object classification Image search New framework, many more to come! Vectors of Locally Aggregated Tensors (VLAT) D. Picard & P.H. Gosselin 12 September / 22