Joint Entropy based Sampling in PBIL for MLFS

Transcription

1 Joint Entropy based Sampling in PBIL for MLFS In Jun Yu( 유인준 ) Artificial Intelligence Lab 1

2 1. Introduction Evolutionary Algorithms(EA) have recently received much attention from the Feature Selection community because of their global search ability [1]. One of EAs, PBIL is effective for feature selection because of simplicity and more accurate and faster than Genetic Algorithm(GA) [2] Artificial Intelligence Lab 2

3 1. Introduction The characteristic of PBIL is replacing the population to probability vector (PV), but PV does not consider relevance of features because of assumption that each feature is independent [2]. Therefore, we propose a feature selection method that considers the relevance between features using joint entropy in the PBIL sampling stage Artificial Intelligence Lab 3

4 2. Related Work 1. In [3] multiple probability vectors and an adaptive updating strategy are proposed and the resulting algorithm is tested on the geometrical design of the end region of power transformers. 2. In [4] a new probability update rule and sampling procedure are proposed using opposition based computing for maintaining diversity Artificial Intelligence Lab 4

5 3. Proposed Method PV ini alize probability vector. (Each posi on = 0.5) For (generations++) 1 Generate Samples 2 Find Best Sample 3 Update Probability Vector 4 Mutate Probability Vector End (Generations) Artificial Intelligence Lab 5

6 3. Proposed Method What is the JENT Average joint entropy of th feature and the currently selected features. : Joint entropy between features normalized between 1 and 1. : Index of the currently selected feaures. : Total number of the currently selected features Artificial Intelligence Lab 6

7 4. Experimental Settings Experiment : repeated 10 times The maximum number of Selected features : 30 Classifier : MLNB Population Size : 50 FFC(Fitness function calls) : 100 Datasets Dataset Domain Patterns Features Size Type Labels Scene Image Continuous 6 Yeast Biology Continuous 14 Genbase Text Binary 22 Medical Biology Binary 27 Slashdot Text Binary 45 Enron Text Binary Artificial Intelligence Lab 7

8 5. Experimental Results Hloss Dataset GA PSO EDA PBIL DPBIL2 PPBIL2 Proposed Scene Yeast Genbase Medical Slashdot Enron Avg. Rank Artificial Intelligence Lab 8

9 5. Experimental Results Mlacc Dataset GA PSO EDA PBIL DPBIL2 PPBIL2 Proposed Scene Yeast Genbase Medical Slashdot Enron Avg. Rank Artificial Intelligence Lab 9

10 5. Experimental Results Rloss Dataset GA PSO EDA PBIL DPBIL2 PPBIL2 Proposed Scene Yeast Genbase Medical Slashdot Enron Avg. Rank Artificial Intelligence Lab 10

11 6. References [1] B Xue, M Zhang, W Browne, X Yao, A Survey on Evolutionary Computation Approaches to Feature Selection, Evolutionary Computation, IEEE Transactions on 20 (4), [2] Shumeet Baluja, Population Based Incremental Learning: a method for integrating genetic search based function optimization and competitive learning, Technical report CMU CS [3] Yang, S. Y., et al. "A new implementation of population based incremental learning method for optimizations in electromagnetics." IEEE Transactions on Magnetics 43.4 (2007): [4] Ventresca, Mario, and Hamid R. Tizhoosh. "A diversity maintaining population based incremental learning algorithm." Information Sciences (2008): Artificial Intelligence Lab 11

12 Appendix A Competitive Learning A competitive learning network The inhibitory connections, between output units, ensure that only one output is turned on at a time. The output unit that is turned on is the one which has the largest net input. The excitatory connections contribute to the net input of the outputs Artificial Intelligence Lab 12

13 Appendix A Competitive Learning The activation of the output units is calculated by thefollowing formula: During training, the weights of the winning output unit are moved closer to the presented point by adjusting the weights according to the following rule (LR is the learning rate parameter): Artificial Intelligence Lab 13

14 Appendix A Competitive Learning After the network training is complete, the weight vectors for each of the output units can be considered prototype vectors for one of the discovered classes. The attributes with the large weights are the defining characteristics of the class represented by the output. It is the notion of creating a prototype vector which will be central to the discussions of PBIL Artificial Intelligence Lab 14

15 Appendix B Genetic Algorithm GA is search technique inspired by the evolutionary process of the natural world. Populations f1 f2 f3 f4 f loop Mutation Evaluation Function Crossover offspting Genetic Algorithm Procedure Artificial Intelligence Lab 15

16 Appendix B Genetic Algorithm Limitation of GA Once the population has converged, the ability for crossover operators to aid in exploring new portions of the function space is greatly hindered. The entire population may come to be dominated by very similar solution vectors when several consecutive generations do not develop novel high evaluation solution vectors. Can Not Maintain Dissimilarity!!! Artificial Intelligence Lab 16

17 Appendix C The Role of Mutation The role of mutation is to prevent the prototype vector from too quickly converging to an extreme value (either 0.0 or 1.0) in each of the bit positions Artificial Intelligence Lab 17

18 Appendix D Standard PBIL [ Baluja, 1994 ] P ini alize probability vector. (Each posi on = 0.5) For (generations++) 1 Generate Samples 2 Find Best Sample 3 Update Probability Vector 4 Mutate Probability Vector End (Generations) Artificial Intelligence Lab 18

19 Appendix D Standard PBIL [ Baluja, 1994 ] 1 Generate Samples for ( i++ ) # iis sample size generate sample vector according to probabili es in P end evaluate ( ) 2 Find Best Sample find vector corresponding to maximum evalua on Artificial Intelligence Lab 19

20 Appendix D Standard PBIL [ Baluja, 1994 ] 3 Update Probability Vector for ( j++ ) # j is Probability Vector Length End User Defined Constants LR : Learning Rate. (= 0.1) Artificial Intelligence Lab 20

21 Appendix D Standard PBIL [ Baluja, 1994 ] 4 Mutate Probability Vector for ( j++ ) # j is Probability Vector Length if ( ramdom (0,1] < )... end End User Defined Constants MP : Probability of mutation occurring in each position. (= 0.02) MS : Mutation Shift amout for mutation to affect the probability. (= 0.05) Artificial Intelligence Lab 21

22 Appendix E PBIL Issues Limitation of PBIL It is unlikely that the population members would be regenerated by sampling the probability vector [2]. It can also be a disadvantage as the collective knowledge accumulated from other searched individuals are not used properly [3] Artificial Intelligence Lab 22

23 Appendix E PBIL Issues Changing the Learning Rate The learning rate affects which portions of the function space will be explored. The setting of the learning rate has a direct impact on the trade off between exploration of the function space and exploitation of the exploration already conducted. exploration is the ability of the algorithm to search the function space thoroughly. exploitation refers to the algorithm s ability to use the information it has gained about the function space to narrow its future search Artificial Intelligence Lab 23

24 Appendix E PBIL Issues Extension to the prototype vector the prototype vector is only adjusted based upon the single best solution vector generated in the current generation. Alternatives 1 The first is to move the probability vector in the direction of the best M vectors, where M << N. 2 The second method, moving away from bad vectors. But both introduce more parameters to the algorithm Artificial Intelligence Lab 24

25 Appendix F Entropy Extension to the prototype vector the prototype vector is only adjusted based upon the single best solution vector generated in the current generation. Alternatives 1 The first is to move the probability vector in the direction of the best M vectors, where M << N. 2 The second method, moving away from bad vectors. But both introduce more parameters to the algorithm Artificial Intelligence Lab 25