Information fusion for scene understanding

Transcription

1 Information fusion for scene understanding Philippe XU CNRS, HEUDIASYC University of Technology of Compiègne France Franck DAVOINE Thierry DENOEUX

2 Context of the PhD thesis Sino-French research collaboration Heudiasyc, UTC and CNRS Key Lab of Machine Perception, Peking University LIAMA, Sino-European Laboratory PSA Peugeot Citroën, Shanghai, Paris Research projects Labex MS2T ANR-NSFC PRETIV MPR LIAMA ICT-ASIA PREDiMap 2

3 Outline 1. Information fusion for scene understanding 2. Reasoning on sets with belief functions 3. Calibration of classifiers 4. Combination of pedestrian detectors 5. Local fusion in over-segmented images 3

4 Outline 1. Information fusion for scene understanding a) Scene understanding b) Combining pattern classifiers c) Bayesian fusion 2. Reasoning on sets with belief functions 3. Calibration of classifiers 4. Combination of pedestrian detectors 5. Local fusion in over-segmented images 4

5 Multi-sensors-based perception 5

6 Trainable multi-sensor fusion Sensors Perception Camera LIDAR Stereo Sensor Classes Sky Features Multi-class classification Grass Tree Road Obst 6

7 Non-trainable multi-sensor fusion Reasoning with images: What the driver sees Adapted for driver assistance Camera LIDAR Flexible and robust frame: Sensors can be added or removed New classes can be easily defined Sensors failure Stereo Classes Sky Sensor Grass Road Partial classification Tree Car Obst* 7

8 Probabilistic model Uncertainty is represented by an a posteriori probability distribution: P( x) Ignorance is represented by a uniform probability distribution: U Ω ω i = 1 Ω Source reliability: P Ω x,r ω i = P Ω x ω i P R r = 1 + U Ω ω i P R (r = 0) 8

9 Bayesian combination P ω i x 1, x 2 P ω i x 1 P(ω i x 2 ) P(ω i ) The estimation of the prior class distribution is a difficult task. Uniform prior distribution: P ω i = 1 Ω Validity: conditional independence P x 1, x 2 ω i = P x 1 ω i P(x 2 ω i ) 9

10 Combination rules Product Combination rules Support of the class ω i μ i = P ω i x j Average μ i + = P ω i x j Minimum Maximum μ i = min P ω i x j μ i = max P ω i x j Properties Prod. Avg. Min. Max. Uniform distribution as neutral element Idempotence Representation of categorical information Combination of contradictory information 10

11 How many classes do you see? 48 11

12 Sky Class organization Other stuff Ground Vegetation Infrastructure Movable Obstacle Four wheeled vehicle Road Building Pedestrian Lane marking Tree Vertical structures Two wheeled vehicle Grass Other kind Other kind Animal Other kind 12

13 Class refinement W S Sky Sky W G Ground 0.5 Ground 0.5 W V Vegetation Vegetation Vegetation Vegetation Q Grass0.25 Road0.25 Tree/Bush 0.17 Obstacles 0.17 Sky

14 Outlines 1. Information fusion for scene understanding 2. Reasoning on sets with belief functions a) Information representation b) Combination rules c) Operations over the frame of discernment 3. Calibration of classifiers 4. Combination of pedestrian detectors 5. Local fusion in over-segmented images 14

15 Information representation Mass function: m Ω 2 Ω [0,1] m Ω = 0, m Ω A = 1 Vacuous: m Ω Ω = 1 A Ω Simple A w : m A = 1 w, m Ω = w Discounting α m: α m A = 1 α m A α m Ω = 1 α m Ω + α 15

16 Information representation Belief and plausibility: Bel A = m(b), Pl A = m(b) B A B A Contour function: pl ω = Pl( ω ) Pignistic probability: BetP ω = A Ω,ω A m(a) A 16

17 Class refinement W S Sky Sky W G Ground 0.5 Ground 0.5 W V Vegetation Vegetation Vegetation Vegetation Q Grass 0.5 Road Tree/Bush Obstacles Sky 17

18 Dempster s rule of combination m 1 m 2 = 0, m 1 m 2 A = 1 1 κ B C=A m 1 (B)m 2 (C) where κ = B C= m B m(c) Hypothesis: the sources of information are supposed to be independent Properties: commutative, associative and has the vacuous mass as unique neutral element 18

19 Cautious rule m 1 m 2 = A w 1 A w 2 A w A = B A q(b) ( 1) B A +1, q A = B A m(b) Properties: idempotent, commutative, associative and has the vacuous mass as unique neutral element 19

20 Triangular norm-based rule m 1 m 2 = A w 1 A w 2 A, m 1 m 2 = A w 1 A w 2 A w 1 T s w 2 = w 1 w 2 if s = 0, w 1 w 2 if s = 1, log s 1 + sw 1 1 s w 2 1 s 1 otherwise. m 1 T s m 2 = A w 1 A T s w 2 A 20

21 Outlines 1. Information fusion for scene understanding 2. Reasoning on sets with belief functions 3. Calibration of classifiers a) Probabilistic calibration b) Evidential extension c) Experimental results 4. Combination of pedestrian detectors 5. Local fusion in over-segmented images 21

22 Posterior probability P(y = 1 s) Calibration of SVM scores SVM score s 22

23 Posterior probability P(y = 1 s) Uncertainty of the calibration SVM score s 23

24 Belief and plausibility Belief and plausibility Isotonic regression SVM score s 24

25 Logistic regression Sigmoid function: P y = 1 s h s θ = Likelihood function: exp(θ 0 + θ 1 s) L X θ = p i y i 1 pi 1 y i with p i = h xi (θ) Plausibility contour: pl x Ω ω, s = sup pl X Θ ln ω 1 1 θ 1 s, θ 1 25

26 Evidential logistic regression 26

27 Belief and plausibility Belief and plausibility Evidential logistic regression SVM score s SVM score s 27

28 Classification results Adult #train=600, #test= Australian #train=300, #test=390 Diabetes #train=300, #test=468 Scenario (a) (b) (c) (a) (b) (c) (a) (b) (c) Probabilistic 83,24 82,70 80,90 85,13 85,90 85,90 78,42 77,14 53,42 Inv. Pignistic 83,32 82,79 81,02 85,13 85,90 86,41 78,63 77,14 54,70 Likelihood 83,29 83,03 81,65 85,13 86,67 88,46 79,06 77,35 68,16 28

29 Belief and plausibility Decision boundary SVM score s 29

30 Outlines 1. Information fusion for scene understanding 2. Theory of belief functions 3. Calibration of classifiers 4. Combination of pedestrian detectors a) Calibration and clustering of bounding boxes b) Combination of detectors c) Experimental results 5. Local fusion in over-segmented images 30

31 # Algorithm Features Classifier Training 1 VJ Haar AdaBoost INRIA 2 HOG HOG Linear SVM INRIA 3 HikSVM HOG HIK SVM INRIA 4 LatSVM HOG Latent SVM PASCAL/INRIA 5 MultiResC HOG Latent SVM Caltech 6 PoseInv HOG AdaBoost INRIA 7 DBN HOG DeepNet INRIA/Caltech 8 MOCO HOG+LBP Latent SVM Caltech 9 paucboost HOG+COV paucboost INRIA 10 ACF Channels AdaBoost INRIA/Caltech 11 MultiFtr+Motion Multiple Linear SVM TUD-Motion 12 PLS Multiple PLS+QDA INRIA 13 Shapelet Gradients AdaBoost INRIA 14 ConvNet Pixels DeepNet INRIA 31

32 VJ [Viola02] HOG [Dalal05] ACF+SDt [Park13]

33 Clustering of bounding boxes Non-maximal suppression (NMS) Greedy: NMS with decreasing score =>Hierarchical clustering Overlap area: a = area BB i BB j area BB i BB j Threshold 1/2. 33

34 Calibrated probability Number of occurrences Calibration of the HOG detector Logistic 0.0 Isotonic SVM scores SVM scores There are almost infinitely more negative samples than positive samples. 34

35 Calibration of detectors 35

36 Combination rules The combination of multiple low scored BBs will lead to an even lower confident BB. Positive scores associated to BBs should be considered as supporting only the presence of a pedestrian. {1} α 1 1 α 2 = 1 α 1α 2 {1} α 1 1 α 2 = 1 α 1 α 2 {1} α 1 T s 1 α 2 = 1 α 1T s α 2 36

37 Distance between detectors Distance between two mass functions d(m 1, m 2 ): 1 A B 2 A B m 1 A m 2 A A,B Ω\{ } Distance between two simple mass functions: 0 d 1 α 1, 1 α 2 = α 1 α d 1 α 1, 0 0 = α 1 2 m 1 B m 2 B 1 2 Detectors: D C k, C l = 1 n i=1 n d(m k,i, m l,i )

38 Distance between detectors Detector clustering #20: CrossTalk #22: ACF #27: MultiFtr+Motion #28: MultiFtr+Motion+2Ped #8: MT-DPM #9: MT-DPM+Context Detector number #6: MultiResC #7: MultiResC+2Ped 38

39 Comparison of combination rules 39

40 Reasonable case scenario 40

41 Overall case scenario 41

42 Results 42

43 Results 43

44 Outlines 1. Information fusion for scene understanding 2. Reasoning on sets with belief functions 3. Calibration of classifiers 4. Combination of pedestrian detectors 5. Local fusion in over-segmented images a) Image over-segmentation b) Construction of detection modules c) Experimental results 44

45 Multi-sensor fusion Sensors Camera LIDAR Stereo Sensor Classes Sky Perception Detection modules Fusion in unified space Grass Tree Road Obst 45

46 46

47 Pixel-based module m ground = 1 m Ω = 1 m sky = 1 47

48 Stereo-based module 48

49 LiDAR-based module 49

50 Surface layout and vegetation modules 50

51 Temporal propagation module 51

52 Classification modules Module Frame of discernment #1 Pixel Ω S = {sky, sky} #2 Pixel Ω G = {ground, ground} #3 Stereo Ω G = {ground, ground} #4 LiDAR Ω G = {ground, ground} #5 Surface Λ = {ground, vertical, sky} #6 Texture Ω V = {vegetation, vegetation} #7 Optical flow multiple 52

53 Ground/Non-ground classification 53

54 Raw image Stereo LiDAR Pixel Optical flow Combination Ground truth 54

55 Three-class classification 55

56 Raw image Surface layout Probabilist Evidential Ground truth 56

57 Three-class classification 57

58 Probabilist Evidential Ground truth 58

59 Our contributions 1. Information fusion for scene understanding 2. Reasoning on sets with belief functions 3. Calibration of classifiers 4. Combination of pedestrian detectors 5. Local fusion in over-segmented images 59

60 Conclusions Flexible and robust fusion framework New detection modules can be added/removed New classes can be defined Calibration can be used to transform the output of many types of classifiers The combination framework can be used at both the object and pixel levels 60

61 Perspectives Combination with additional souces of information (GPS, maps, etc.) Combination at both object and segment level. Selection of subsets of classifiers Reinforcing existing algorithms 61

62 Publications [1] Ph. Xu, F. Davoine, J.-B. Bordes, H. Zhao and T. Denoeux. Multimodal Information Fusion for Urban Scene Understanding. Machine Vision and Applications (MVA), [2] Ph. Xu, F. Davoine, J.-B. Bordes and T. Denoeux. Fusion d informations pour la comprehension de scènes. Traitement du signal (TS), [3] Ph. Xu, F. Davoine and T. Denoeux. Evidential Logistic Regression for Binary SVM Classifier Calibration. International Conference on Belief Function (BELIEF), [4] Ph. Xu, F. Davoine and T. Denoeux. Evidential Combination of Pedestrian Detectors. British Machine Vision Conference (BMVC), [5] Ph. Xu, F. Davoine, J.-B. Bordes, H. Zhao and T. Denoeux. Information Fusion on Oversegmented Images: An Application for Urban Scene Understanding. International Conference on Machine Vision Applications (MVA), [6] J.-B. Bordes, Ph. Xu, F. Davoine, H. Zhao and T. Denoeux. Information Fusion and Evidential Grammars for Object Class Segmentation. IROS Workshop of Planning, Perception and Navigation for Intelligent Vehicles, [7] J.-B. Bordes, F. Davoine, Ph. Xu, T. Denoeux. Evidential Grammars for Image Interpretation. Application to multimodal traffic scene understanding. Integrated Uncertainty inknowledge Modelling and Decision Making (IUKM), [8] Ph. Xu, F. Davoine and T. Denoeux. Transformation de scores SVM en fonctions de croyance. Congrès national sur la Reconnaissance de Formes et l Intelligence Artificielle (RFIA), [9] Ph. Xu, F. Davoine, J.-B. Bordes and T. Denoeux. Fusion d informations sur des images sursegmentées : Une application à la comprehension de scènes routières. Congrès des jeunes chercheurs envision par ordinateur (ORASIS),