AI*IA 2003 Fusion of Multiple Pattern Classifiers PART III

Transcription

1 AI*IA 23 Fusion of Multile Pattern Classifiers PART III AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 49

2 Methods for fusing multile classifiers Methods for fusing multile classifiers can be classified according to the tye of information roduced by the individual classifiers (Xu et al., 992): Abstract-level oututs: each classifier oututs a unique class label for each inut attern Rank-level oututs: each classifier oututs a list of ossible classes, with ranking, for each inut attern Measurement-level oututs: each classifier oututs class confidence levels for each inut attern For each of the above categories, methods can be further subdivided into: Integration vs. Selection rules and Fixed rules vs. Trained Rules AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 5

3 Methods for fusing multile classifiers Integration Abstract-level Selection Fixed rules Measurements-level Trained rules Rank-level AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 5

4 Fixed Rules at the abstract-level The Majority Voting Rule Let us consider the N abstract ( cris ) classifiers oututs S (),, S (N) associated to the attern x. Majority Rule: Class label c i is assigned to the attern x if c i is the most frequent label in the cris classifiers oututs. Classifier S () =a i S = a, b, N = 3 { } x n Classifier 2 S (2) =b Majority Voting Rule a Classifier 3 S (3) =a AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 52

5 Usually N is odd. Majority Vote Rule The frequency of the winner class must be at least N/2 If the N classifiers make indeendent errors and they have the same error robability e<.5, then it can be shown that the error E of the majority voting rule is monotonically decreasing in N (Hansen and Salamon, 99): N N l i m e k N k ( e ) = > / 2 k N k N Clearly, erformances of majority vote quickly decreases for deendent classifiers AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 53

6 Majority Vote Rule vs. Classifiers Deendency: An Examle 3 Classifiers Two class task Classifiers C and C3 exhibit error correlations > P(,,) P(,,) < Classifier C Abstract C2 Oututs C3 Majority Class True Class AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 54

7 Trainable Rules at the abstract-level Rules based on the Bayes Aroach This kind of trained fusion rules are based on the Bayes aroach. Pattern x is assigned to the class c i if its osterior robability: () (2) ( N) () (2) ( N) j P( c S, S,..., S ) P( c S, S,..., S ) i i { } S c c, 2,..., c k > j i Fusion rules based on the Bayes Formula try to estimate, by an indeendent validation set, these osterior robabilities This fusion rule coincides with the multinomial statistical classifier, that is, the otimal statistical decision rule for discretevalued feature vectors (Raudys and Roli, 23) AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 55

8 Behaviour Knowledge Sace (BKS) In the BKS method, every ossible combination of abstract-level classifiers oututs is regarded as a cell in a look-u table. Each cell contains the number of samles of the validation set characterized by a articular value of class labels. Reject otion by a threshold is used to limit error due to ambiguous cells Classifiers oututs S (), S (2), S (3) Class,,,,,,,,,,,,,,,, P ( c = S =, S =, S = ) = = th () (2) (3) 76 AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 56

9 BKS Small-Samle Size Drawback If k is the number of classes and N is the number of the combined classifiers, BKS requires to estimate k N osterior robabilities. K and N are two critical arameter for the BKS rule, because the number of osterior robabilities to estimate increases very quickly. If this number is too high, we can have serious roblems, because the number of training samle is often small. BKS rule suffers a lot from the small samle size roblem AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 57

10 BKS Imrovements In order to avoid the small samle size roblem: -we can try to reduce the number of the arameters to estimate (Xu et al., 992 ; Kang, and Lee, 999). For examle, under the assumtion of classifier indeendence given the class (Xu et al., 992): P S S c P S c P c S P S () ( N ) ( j ) ( j ) ( j ) (,..., i ) = ( i ) ( i ) ( ) j j bel i P c S S P c S () ( N ) ( j ) ( ) = ( i,..., ) = ( i ) j -We can use noise injection to increase samle size of training set (Roli, Raudys, Marcialis, 22). AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 58

11 BKS Imrovements If the number of classifiers is small, BKS cells can become large and contain vectors of different classes, that is, ambiguous cells can exist. Classifiers oututs S (), S (2), S (3) Class,,,,,,,,,,,,,,,, Raudys and Roli (23) roosed a method to address this issue AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 59

12 Remarks on Abstract Level Fusers Abstract level methods are the most general fusion rules They can be alied to any ensemble of classifiers, even to classifiers of different tyes The majority voting rule is the simlest combining method This allows theoretical analyses (Lam and Suen, 997) When rior erformance is not considered, the requirements of time and memory are negligible As we roceed from simle rules to adative (weighted voting) and trained (BKS, Bayesian rule) the demands on time and memory quickly increase Trained rules imose heavy demands on the quality and size of data set AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 6

13 AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 6 Rank-level Fusion Methods Some classifiers rovide class scores, or some sort of class robabilities In general, if ={c,,c k } is the set of classes, these classifiers can rovide an ordered (ranked) list of class labels. Classifier c c c = = = c c c r r r = = = This information can be used to rank each class.

14 The Borda Count Method: an examle Let N=3 and k=4. = { a, b, c, d }. For a given attern, the ranked oututs of the three classifiers are as follows: Rank value Classifier Classifier 2 Classifier 3 4 c a b 3 b b a 2 d d c a c d AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 62

15 The Borda Count Method: an examle So, we have: r = r + r + r = = 8 ( ) ( 2 ) ( 3 ) a a a a ( ) ( 2 ) ( 3 ) b b b b ( ) ( 2 ) ( 3 ) a c c c ( ) ( 2 ) ( 3 ) a d d d r = r + r + r = = r = r + r + r = = 7 r = r + r + r = = 5 The winner-class is b because it has the maximum overall rank. AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 63

16 Remarks on Rank level Methods Advantages over abstract level (majority vote): ranking is suitable in roblems with many classes, where the correct class may aear often near the to of the list, although not at the to Examle: word recognition with sizeable lexicon Advantages over measurement level: rankings can be referred to soft oututs to avoid lack of consistency when using different classifiers rankings can be referred to soft oututs to simlify the combiner design Drawbacks: Rank-level methods are not suorted by clear theoretical underinnings Results deend on the scale of numbers assigned to the choices AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 64

17 AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 65 Measurement-level Fusion Methods Classifier Classifier j Classifier N k 2... j j j k 2... N N N k Fusion rule x i k a rg m a x i i Normalization of classifiers oututs is not a trivial task when combining classifiers with different outut ranges and different outut tyes (e.g., distances vs. membershi values).

18 Linear Combiners ave = N k i k = k i ( x) w ( x) Simle and Weighted averaging of classifiers oututs Simle average is the otimal fuser for classifiers with the same accuracy and the same air-wise correlations (Roli and Fumera) Weighted average is required for imbalanced classifiers, that is, classifiers with different accuracy and/or different air-wise correlations (Roli and Fumera) Imrovement of weighted average over simle average has been investigated theoretically and by exeriments (Roli and Fumera) AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 66

19 Bias/Variance Analysis in Linear Combiners The added error over the Bayes one can be reduced by linear combinations of classifiers oututs (Tumer and Ghosh; Roli and Fumera) i (x) Otimum boundary Obtained boundary j (x) f i (x) f j (x) Bayes Error Added Error x D i x* b x b D j AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 67

20 Bias/Variance Analysis in Linear Combiners Correlated and Unbiased Classifiers Added error of the simle average of N classifiers (Tumer and Ghosh): E a v e a d d ( N ) + " = E a d d N The reduction of variance comonent of the added error achieved by simle averaging deends on the correlation factor " between the estimation errors Negatively correlated estimation errors allow to achieve a greater imrovement than indeendent errors AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 68

21 Bias/Variance Analysis in Linear Combiners Correlated and Biased Classifiers Added error of the simle average of N classifiers (Tumer and Ghosh): E ( ) 2 + N " = # + $ $ s N 2s ( ) 2 ave add i j Simle averaging is effective for reducing the variance comonent, but not for the bias comonent So, individual classifiers with low biases should be referred Analysis of bias/variance for weighted average can be found in (Fumera and Roli) AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 69

22 Product and Order Statistics Fusers Fusers based on order statistics oerators are max, min, and med: max ( ) N : N x = ( x), i =,, k i i min ( ) : N x = ( x), i =,, k i i ( ) N N j i (* i ) i j= = P, i =,, k Product is obviously sensible to classifiers oututting robability estimates close to zero ( overconfident classifiers) & N N + : N : N 2 ( ) 2 ( ) i x + i x med if is even ( ) ' N i x = ( 2, i =,, k ' N + : N 2 ) ' ( ) i x if N is odd AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 7

23 A Theoretical Framework A theoretical framework for the roduct, min, med and max fusers was established by J. Kittler et al. (998) These rules can be formally derived assuming that the N individual classifiers use distinct feature vectors j (* / X, X,..., X ) i 2 the roduct and min rules are derived under the hyothesis that classifiers are conditionally statistically indeendent sum, max, median rules are derived under the further hyothesis that the a osteriori robabilities estimated by the classifiers do not deviate significantly from the class rior robabilities N AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 7

24 Fusers with Weights A natural extension of many fixed rules is the introduction of weights to the oututs of classifiers (weighted average, weighted majority vote) A simle criterion is to introduce weights roortional to the accuracy of each individual classifier Weights related to classes can also be comuted by extracting them from the confusion matrix of each classifier Methods for robust weights estimation are a matter of ongoing research AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 72

25 Stacked Fusion The k soft oututs of the N individual classifiers, ij (x), i=,,k, j=,,n, can be considered as features of a new classification roblem (classifiers outut feature sace) Classifiers can be regarded as feature extractors! Another classifier can be used as fuser: this is the so-called stacked aroach (D.H. Wolert, 992), or metaclassification (Giacinto and Roli, 997), brute-force aroach (L.I. Kuncheva, 2) To train the metaclassifier, the oututs of the N individual classifiers on an indeendent validation set must be used Exerts Boasting Issue! (Raudys, 23) An indeendent validation set is required AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 73

26 Pros and Cons of the Stacked aroach Pros: the meta-classifier can work in a enriched feature sace No classifiers deendency model is assumed Cons: The dimensionality of the outut sace increases very fast with the number of classes and classifiers. The meta-classifier should be trained with a data set different from the one used for the individual classifiers (Exerts Boasting Issue) The sace of classifiers oututs might not be well-behaved AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 74

27 Classifier Selection Goal: for each inut attern, select the classifier, if any, able to correctly classify it Problem formulation Two dimensional feature sace Partitioned in 4 regions R R4 R3 R2 Easy to see that selection outerforms individual classifiers if I am able to select the best classifier for each region Two critical issues: i)definition of regions; ii)selection algorithm AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 75

28 Classifier Selection Two main aroaches to design a classifier selection system: Static vs. Dynamic Selection Static Selection Regions are defined before classification. For each region, a resonsible classifier is identified Regions can be defined in different ways: Histogram method: Sace artition in bins Clustering algorithms AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 76

29 Dynamic Selection Classifier Selection SELECTION CONDITIONS based on estimates of classifiers accuracies in local regions of feature sace surrounding an unknown test attern X (Neighbourhood(X)) X 2 xxxxxxxx xxxxxx xxx X ^^^^^^^^^^ ^^^^^^^^ ^^^ X Test Pattern X Neighbourhood(X) Validation Patterns See works of Woods et al.; Giacinto and Roli AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 77

30 Selection Condition: An Examle X 2 xxxxxxxx xxxxxx xxx X ^^^^^^^^^^ ^^^^^^^^ ^^^ X Test Pattern X Neighbourhood(X) Validation Patterns Woods et al. (997) simly comuted classifier local accuracy as the ercentage of correctly classified atterns in the neighbourhood Giacinto and Roli (997, etc) roosed some robabilistic measures: j ˆ( P correct X *) = j N Pˆ ( ) (* X * ) W n= i n i n N W n= n AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 78

31 Some remarks on classifier selection Theoretically, local fusion rules can outerform global ones Selection can effectively handle correlated classifiers In ractice, large and reresentative data sets are necessary to design a good selection system Further work is necessary to develo robust methods for estimating classifier local accuracy Works on Adative Mixtures of Local Exerts (M. Jordan, et al.), not discussed for the sake of brevity, can be regarded as methods for dynamic classifier selection AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 79

32 Final Remarks on Fixed vs. Trained Fusers Fixed rules Simlicity Low memory and time requirements Well-suited for ensembles of classifiers with indeendent/low correlated errors and similar erformances Trained rules -Flexibility: otentially better erformances than fixed rules -Trained rules are claimed to be more suitable than fixed ones for classifiers correlated or exhibiting different erformances High memory and time requirements Heavy demands on the quality and size of the training set AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 8

33 MCS DESIGN Parts 2 and 3 showed that designer of MCS has a toolbox containing a large number of instruments for generating and fusing classifiers. Two main design aroaches have been defined so far Coverage otimisation methods Decision otimisation methods However, combinations of the two above methods and hybrid, ad hoc, methods are often used AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 8

34 Feature Design MCS DESIGN Ensemble Design Classifier Design Fuser Design Performance Evaluation Performance Evaluation Coverage otimisation methods: a simle fuser is given without any design. The goal is to create a set of comlementary classifiers that can be fused otimally -Bagging, Random Subsace, etc. Decision otimisation methods: a set of carefully designed and otimised classifiers is given and unchangeable, the goal is to otimise the fuser AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 82

35 MCS DESIGN Decision otimisation method to MCS design is often used when reviously carefully designed classifiers are available, or valid roblem and designer knowledge is available Coverage otimisation method makes sense when creating carefully designed, strong, classifiers is difficult, or time consuming Integration of the two basic aroaches is often used However, no design method guarantees to obtain the otimal ensemble for a given fuser or a given alication (Roli and Giacinto, 22) The best MCS can only be determined by erformance evaluation. AI*IA 23 Tutorial on Fusion of Multile Pattern Classifiers by F. Roli 83