Combining Dialectical Optimization and Gradient Descent Methods for Improving the Accuracy of Straight Line Segment Classifiers

Transcription

1 Cmbining Dialectical Optimizatin and Gradient Descent Methds fr Imprving the Accuracy f Straight Line Segment Classifiers Rsari A. Medina Rdriguez and Rnald Fumi Hashimt University f Sa Paul Institute f Mathematics and Statistics, IME-USP Department f Cmputer Science Rua d Mata, 00 - Sa Paul, Brazil rsarir@ime.usp.br and rnald@ime.usp.br Abstract A recent published pattern recgnitin technique called Straight Line Segment (SLS) uses tw sets f straight line segments t classify a set f pints frm tw different classes and it is based n distances between these pints and each set f straight line segments. It has been demnstrated that, using this technique, it is pssible t generate classifiers which can reach high accuracy rates fr supervised pattern classificatin. Hwever, a critical issue in this technique is t find the ptimal psitins f the straight line segments given a training data set. This paper prpses a cmbining methd f the dialectical ptimizatin methd (DOM) and the gradient descent technique fr slving this ptimizatin prblem. The main advantage f DOM, such as any evlutinary algrithm, is the capability f escaping frm lcal ptimum by multi-pint stchastic searching. On the ther hand, the strength f gradient descent methd is the ability f finding lcal ptimum by pinting the directin that maximizes the bjective functin. Our hybrid methd cmbines the main characteristics f these tw methds. We have applied ur cmbining apprach t several data sets btained frm artificial distributins and UCI databases. These experiments shw that the prpsed algrithm in mst cases has higher classificatin rates with respect t single gradient descent methd and the cmbinatin f gradient descent with genetic algrithms. Keywrds- straight line segments; gradient descent technique; dialectical ptimizatin; genetic algrithms; pattern recgnitin. I. INTRODUCTION In Image Prcessing and Cmputer Visin areas we frequently find many Pattern Recgnitin prblems such as ptical character recgnitin (OCR), handwritten recgnitin, human face image recgnitin, industrial inspectin and medicine diagnses [], [2]. Recently a new technique fr Pattern Recgnitin (fcused n supervised binary classificatin),called as Straight Line Segment (SLS) classifier [3], [4], [5] was published. The key issue in this technique is t find ptimal psitins f the straight line segments given a training data set. An algrithm fr finding these ptimal psitins is called training algrithm. In this paper, we cmbine the traditinal gradient descent methd with a nvel evlutinary algrithm [6], [7] called Dialectical Optimizatin Methd (DOM), fr slving this ptimizatin prblem. On ne hand, the main advantage f DOM is the capability f escaping frm lcal ptimum by multi-pint stchastic searching. On the ther hand, the strength f gradient descent methd [8] is the ability f finding lcal ptimum by pinting the directin that maximizes the bjective functin. Thus, based n these characteristics, in this paper we prpse a technique that cmbines the gradient descent and dialectical ptimizatin methds fr slving ptimizatin prblem in the training algrithm f the SLS classifier. In rder t verify the viability f ur algrithm, we have applied it t several data sets which were btained frm bth artificial prbability distributins (with knwn prbability density functins) and UCI databases. Based n the results f these experiments, the prpsed algrithm btained in mst f the cases higher classificatin rates when cmpared t: the slutins btained frm using just ne single gradient descent methd and the slutins btained frm the cmbinatin f gradient descent with genetic algrithms. As a cnclusin, this hybrid technique imprves the accuracy f the SLS classifiers in rder t prvide better slutins fr supervised Pattern Recgnitin prblems that appear in Image Prcessing r Cmputer Visin areas. Fllwing this brief intrductin, Sectins II and III briefly recall, respectively, the SLS classifier and the Dialectical Optimizatin Methd. In Sectin IV, we present ur hybrid apprach. Sectin V shws the experimental results we have cnducted. Finally, we give sme cnclusins and directins fr future wrk in Sectin VI. II. CLASSIFIER BASED ON STRAIGHT LINE SEGMENTS A recent publicatin n Pattern Recgnitin presents a new technique based n straight line segments [3], [4], [5]. Its main cntributin is t intrduce a new type f classifier based n distances between a set f pints and tw sets f straight line segments. S far, this technique is fcused n binary classificatin, s there is nly tw pssible classes in the data

2 set. Fr the sake f cmpleteness, we briefly describe in this sectin the SLS classifier. A. Basic Definitins fr SLS Classifiers Let p and q R d+. The straight line segment (SLS) with extremities p and q is defined as: L p,q = {x R d+ : x = p + λ (q p), 0 λ } () Given a pint x R d, let x e = (x, 0) dente the extensin f x t R d+. Given a pint x R d and L p,q R d+. The pseuddistance between x and L is given by: distp (x, L) = dist(x e, p) + dist(x e, q) dist(p, q) (2) 2 where dist(a, b) dentes the Euclidean distance between tw pints a, b R d+. Let L dente a set f SLSs, defined as: L = {L pi,q i : p i, q i R d+, i =,..., m} (3) where m represents the number f SLSs fr each class. Given a pint x R d, the discriminative functin is defined as: T L0,L (x) = distp (x, L) + ε distp (x, L) + ε L L L L 0 (4) where ε is a small psitive cnstant t avid divisin by zer. Cnsidering the discriminative functin, the classificatin functin is defined as: { 0, if SL0,L F L0,L (x) (x) < 0.5; (5), therwise where S L0,L (x) is a sigmid functin dented by: S L0,L (x) = (6) + e g(t L 0,L (x)) The fllwing relatinships can be fund amng the discriminative, sigmid and classificatin functins: If x has apprximately the same distance frm bth L 0 and L (that is, L L 0, L L : distp (x, L) distp (x, L ) and vice-versa) then, by Eq. 4, T L0,L (x) 0 and cnsequently, by Eq. 6, S L0,L (x) 0.5. S, the mre x is equally separated frm L 0 and L, the mre it is difficult t discriminate whether x belngs t class 0 r. The clser x is t L (that is, L L : distp (x, L) 0) and, at the same time, x is farther frm L 0 (that is, L L 0 : distp (x, L) + ) then, by Eq. 4, the bigger is T L0,L (x) (that is, it tends t + ) and, cnsequently, by Eq. 6, the clser S L0,L (x) is t, and, therefre, by Eq. 5, the clser is mre x can be discriminated as belnging t class (that is, the clser the classificatin functin F L0,L (x) is t ). Analgusly, the clser x is t L 0 and (at the same time) farther frm L, the clser the classificatin functin F L0,L (x) is t 0. B. Training Algrithm Given a set f n examples E n = {(x i, y i ) R d {0, } : i =, 2,.., n}, the main purpse f the training algrithm in the cntext f SLS classifiers, is t find tw sets f SLSs L 0 and L in rder t minimize the mean square errr functin defined as: E(F L0,L ) = n [F L0,L n (x i ) y i ] 2 (7) i= This algrithm can be divided int tw phases [4]: ) Placing: This phase cnsists f pre-allcating (finding initial psitins f) the SLSs (in L 0 and L ) based n the fact that pints x clser t L 0 (r L, respectively) and farther frm L (r L 0, respectively) lead the classificatin functin F L0,L (x) t 0 (r, respectively). T achieve this gal, the set f examples E n is split int tw grups, X i = {x R d : (x, y) E n and y = i} (fr i = 0, ), and then the clustering algrithm k-means is applied t each grup. Later, with the bjective t btain the initial extremities f the SLSs fr each cluster, the k-means algrithm (with k = 2) is applied again, but at this time t each cluster btained frm the previus k-means applicatin. 2) Tuning: The purpse f this phase is t minimize the mean square errr functin. One pssible way t accmplish this task is t use the gradient descent technique [8] t find the psitins f the SLSs in L 0 and L such that the mean square functin derivate is equal t zer. Despite f the gradient descent methd des nt guarantee the glbal minimum and the final slutin (psitins f the SLSs) depends n the initial placing phase, this methd was successfully applied in [4]. III. DIALECTICAL OPTIMIZATION METHOD The Dialectical Optimizatin Methd (DOM) was intrduced in [6], [7] and it is an evlutinary methd based n the materialist dialectics fr slving search and ptimizatin prblems. DOM is based n the dynamics f cntradictins between their integrating dialectical ples [7], where each ple is cnsidered as a pssible slutin fr the prblem. These ples are invlved in a ple struggle and they are affected by revlutinary crises where sme ples may disappear r may be absrbed by anther ples and new ples may cme up frm new perids f revlutinary crises. These tw prcesses make the system tend t stability [6]. Since this methd is based n philsphical cncepts, their definitins shwn in [6], are briefly described in this sectin. A ple is the fundamental integrating unit f a dialectical system and crrespnds a candidate slutin t the prblem and it is represented by a vectr f cnditins with n inputs. Given a set f ples Ω = {w, w 2,..., w m }, each ple w i is assciated t a vectr f weights defined as: w i = (w i,, w i,2,..., w i,n ) T (8)

3 where m is the number f ples and n is the system dimensinality. The scial frce f each ple w i is assciated t the bjective functin f the ptimizatin prblem and frm nw n, it will be represented by f(w i ). This is the fundamental idea f the dialectical methd. The cntradictin between tw ples w p and w q is defined as: δ p,q = dist(w p, w q ) (9) where dist(a, b) dentes the Euclidean distance between tw pints a and b. A histrical phase cnsists f tw stages: (i) evlutin and (ii) revlutinary crisis. These states crrespnd t the dynamics between ples thrugh cntradictin evaluatins. At the ple struggle stage, the ple w i has the hegemny at instant t when f P (t) = f(w i (t)) = max j m f(w j(t)) (0) and f P (t) is knwn as the present hegemny at time t. In the same way, the histric hegemny up t time t, dented by f H (t), and defined as the maximum scial frce value btained in each histrical phase up t time t. Given a ple w i such that a w i b, where a, b R, the ppsite r the abslute antithesis f w i is defined as: w i = b w i + a () Accrding t the dialectical methd, the synthesis f tw ples is the reslutin f the cntradictin between them (thesis and antithesis). As a result, it is btained ne new ple with characteristics inherited frm bth f them. The set f parameters that must be prvided by the user is: the number f ples (m), the number f histrical phases (η P ) and the duratin f each histrical phase (η H ). As can bee seen in the pseudcde presented n Algrithm, the parameters η P and η H wrk as threshlds fr the lps described at Lines 2 and 9. In additin, the tw phases f DOM (evlutin and revlutinary crisis) are represented at Lines 3 8 and Lines 0 5, respectively. IV. HYBRID OF DIALECTICAL OPTIMIZATION AND GRADIENT DESCENT METHODS On ne hand, the main advantage f DOM is the capability f escaping frm lcal ptimum by multi-pint stchastic searching. On the ther hand, the strength f gradient descent methd [8] is the ability f finding lcal ptimum by pinting the directin that maximizes the bjective functin. Thus, based n these prperties, the cmbinatin f gradient descent and dialectical ptimizatin methds may imprve the slutin quality f ptimizatin prblems. The idea t build a hybrid methd is shwed in Fig., where the main gal f DOM is t assist the gradient descent methd by prviding t it a new set f initial psitins (the utput f the dialectical ptimizatin methd) fr the tw sets f SLSs L 0 and L. The fllwing steps resume the idea f this hybrid methd: Algrithm Dialectical Optimizatin Methd Algrithm Require: m, η P, η H : set f initial parameters. Ensure: dglbalw inv alue: ptimized value. : Generatin f m initial ples (w[0 : m]) 2: while (inump hases < η P ) d 3: if (inump hases > 0) then 4: deletesimilarp les() 5: generatesynthesis() 6: generateantithesis() 7: addexternalef f ects() 8: end if 9: while (ip haseduratin < η H ) d 0: while (inump les < m) d : iw inp le 0 2: if (f(w inump les ) < f(w iw inp le )) then 3: iw inp le inump les 4: dglbalw inv alue f(w iw inp le ) 5: end if 6: end while 7: addcrisisef f ect() 8: end while 9: end while ) Generate initial ples as described in DOM. 2) Fr each phase f DOM, apply the gradient descent t each ple in the ppulatin in rder t btain ne ptimum lcal fr each ple. 3) Prceed with the next steps f DOM (i.e.ple struggle, synthesis, etc.) as referred in Sectin III. Our apprach uses an evlutinary methd t scape frm the lcal ptimal prvided by the gradient descent technique t find anther value (even better t the previus ne) which may be the new glbal ptimum. Then, at the mment when the gradient descent stps at the minimum lcal, the DOM prvides a new start pint helping the gradient t scape frm that valley. It is imprtant t emphasize that the initial ppulatin (set f ples) includes the slutin btained frm ne single applicatin f gradient descent s that ur hybrid apprach can generate a slutin that is equal t r better than the ne btained by just using the gradient descent methd. In this sense, ur apprach is cnservative. A. Adaptatins t DOM cncepts Sme mdificatins have been dne t the previus DOM cncepts shwn in Sectin III t adapt the prblem f finding the best the psitins f the SLSs in L 0 and L. Since a ple crrespnds t a pssible slutin, we define a ple as a vectr cnsisting f the extremities f the SLSs belnging t bth L 0 and L. Thus, a ple can be represented by a vectr btained frm the cncatenatin f L 0 with L (i.e. [L 0 L ]). The initial ppulatin is built in the fllwing way: half f the initial ppulatin is randmly generated with data between the range f the minimum and maximum values frm the

4 Training data set Training Dialectical Optimizatin Gradient Descent N Nrmalizatin frm ples int SLSs Generatin f ples (thesis/antithesis) N End f Phase Yes Remval & Synthesis f Ples End f Phases Yes Nrmalizatin frm ples int SLSs Applicatin f Grad. Descent Find the Hegemnies Evaluatin f cntradictins Nrmalizatin frm SLSs int ples Update ples Add crisis effect SLSs_0 and SLSs_ Testing Testing data set Estimatin f distances frm pints t SLSs Estimatin f Objective Functin Value Calculatin f Mean Square Errr Fig.. Data flw representatin f the prpsed hybrid methd, where each blck represents a prcess. The system input is a training data set and the system utput is a set f tw SLSs and the classified data set. training data set. The ther half part f the initial ppulatin is generated with the ppsite ples (antithesis). The antithesis f a ple [L 0 L ] (described in Sectin III) is redefined as [L L 0 ]. Since ur prblem deals with a set f SLSs which divides tw classes f pints, the ppsite vectr is equivalent t invert classes f the SLSs psitins, meaning that the SLSs in L 0 and L, nw represents class e 0, respectively. The scial frce is represented by the MSE defined as R En (L 0, L ) = n (y i F L0,L n (x i )) 2 (2) i= where F L0,L (x i ) is the classificatin functin frm the SLS classifier (defined in Eq. 5) and y i is the class value. The cntradictin f tw ples was defined in Sectin III as their Euclidian distance. In ur wrk, the cntradictin is redefined as the difference in abslute value between the MSE btained fr each invlved ple. Therefre, the cntradictin means the difference f classificatin errrs between tw ples. The pseudcde fr this hybrid methd is the same as presented in Algrithm, with the nly difference that right befre Line 3, the gradient descent methd is applied t each ple befre executing the steps f DOM s methd. By lking at Fig., it is easier t understand this hybrid methd, flwing the data flw which is divided int tw prcess. First, given a training data set, the ppulatin (thesis and antithesis) is generated. Then, it is applied the gradient descent methd t each member f the ppulatin. Finally the remaining DOM functins, gruped as Evlutin (first lp) and Revlutinary Crisis (secnd lp), are applied. At the end f the training phase, tw sets f SLSs are btained, which are the nes with minimum MSE errr. Secnd, given a test data set, the distances between the new examples and the btained SLSs frm the training set are cmputed t btain the MSE errr. As a final result, an image f the examples classificatin is btained, shwing the SLSs psitins. V. EXPERIMENTAL RESULTS AND DISCUSSION In rder t evaluate the prpsed hybrid methd fr training SLS classifiers, we cnducted tw experiments. In the first ne, using artificial data, we have applied ur hybrid methd n data sets drawn frm fur artificial prbability density functins named here as F, S, Simple and X and described in [4]. A. Artificial Data Sets In these data sets, each regin has a unifrm prbability density functin. This prperty makes pssible t apply the Bayes classifier [] t btain the ideal classificatin rate and cmpare it with ur result as it was prpsed in [4]. Since the prbability density functin is knwn, it is pssible t use numerical integratin t calculate the classificatin rate f the SLS classifier that was btained by applying ur hybrid methd in the training phase. Fr this experiments, the parameters used fr the gradient descent methd are the same used in [4]: Number f iteratins = 000,

5 Initial Value = 0., Displacement decrement = 0.5, Displacement increment = 0., Minimum value = 0 5, and, the parameters fr DOM are: Number f ples = 30, Number f phases = 20, Number f iteratins = 5, Minimum Value = 0 3, Learn Rate = 0.99, Crisis Effect value = 0.2. We have generated fur samples with 00, 200, 400 and 800 examples, respectively. Fr each sample, we have applied ur hybrid methd t, 2, 3 and 4 SLSs fr each class. T shw the results, we use a graphic representatin (a grayscale image) f the classificatin regins cmputed by the classificatin functin F L0,L (x i ) f the SLS classifier. Fr each image, the black and white regins represent the regins fr classes 0 and, respectively. The SLSs in L 0 and L are als indicated in the ppsite clr (i.e. black regins with white SLSs and white regins with black SLSs). The SLSs are in R d+. Fr the sake f visualizatin, when the SLSs are in R 3, they are prjected int R 2 with the purpse f shwing them in the image results. In rder t evaluate the hybrid methd, we have cmputed the classificatin rate based n the Bayes classifier fr each sample. The bld numbers indicates the best classificatin rate fr each sample (i.e. the best classifier with n SLSs per class). It is imprtant t emphasize that all the results are clse t the ideal classificatin rate and, in mst cases, they are better than the results btained in [4]. B. Discussin The prbability density functin F is represented n the first rw in Fig. 2. By lking at it, we can bserve that regins and SLSs psitins are better fitted n the prbability density functin, it is pssible t see the imprvement frm in secnd clumn, with genetic algrithms (-AG) in third clumn and ur methd in furth clumn. In the last three clumns we present the best results btained fr each methd, -AG and -DOM. It can be seen that and -DOM results are very similar but uses 3-SLS and -DOM nly 2-SLS which is better. We can als bserve that amng the classificatins rate presented at Table I, the results fr larger amunt f examples, such as 400 and 800, the best classificatin rate is btained frm a SLS classifier with 3 SLSs fr each class. The classificatin rate is abut 87.4% fr 800 examples which is very clse t the ideal classificatin rate (with Bayes classifier) 87.65%. In the same way, the prbability density functin S is represented n the secnd rw in Fig. 2, and nce again the best representatin f the data distributin is in clumn d which is the result frm ur hybrid methd. Frm Table I, we can say that the best classificatin rates were reached using 3 SLSs fr samples with 200 and 400 examples (clumns d and e frm Fig. 2); while fr samples with 800 examples, the best classifier uses 2 SLSs (clumn f ). In all results, the classificatin rates are clse t the ideal classificatin rate. The Simple-distributin is the ne with the best result amng the fur distributins prpsed in [4] and as can be seen n Fig. 2 and cmpared with the classificatin rates in Table I. We can highlight that all percentages f classificatin rate are very clse t the ideal classificatin rate (93.90%) and even ne f them is nly 0.40% percent lwer than the ideal ne. This crrespnds t the case f the SLS classifier with 3 SLSs btained frm a sample with 800 examples. It can als be seen that fr samples with 00 and 400 examples, the percentage f crrect classificatin is abut 93%. In the ppsite way, the X-distributin presents the weakest classificatin rate amng the thers fr ur prpsed methd as can be seen in Table I and it is the well knwn XOR distributin. Fr this distributin, we did nt btain a big classificatin rates imprvement (smetimes they are equal t the gradient descent methd, and smetimes they are wrst than it). As can be seen n Fig. 2 (last three clumns) the best results fr this distributin were reached with just -SLS fr and -DOM, but using -AG were used 3-SLS t reach the best classificatin rate. C. Gradient Descent vs. Gradient Descent with DOM In this sectin, we present a line diagram in Fig. 3, in rder t cmpare the classificatin rates btained fr bth methds, and -DOM when a SLS classifier with 3- SLS is applied n the fur distributins F, S, X and Simple presented previusly and drawn in different clrs. As can be seen, the gradient descent methd is represented with pinted lines and ur hybrid methd with cntnuus lines. In the figure are represented the best classificatin rates btained in the experiment cnduced n this wrk with 00, 200, 400 and 800 examples. Using this graphic is much easier t cmpare the gradient descent with the prpsed methd. Thus, fr F-distributin (using plus signs), the imprvement by using ur apprach is between % e 4%. In the same way, fr S and Simple distributins (using circles and asterisk, respectively) the imprvement f the classificatin rates varies between 0.50% e 8%. Finally, fr X-distributin (using letter x), the mst cmplicated fr ur methd the percentage f imprvement is between % e 4%. Furthermre, the graph suggests that fr this wrk the distributin with the best classificatin rate is Simple-distributin, with percentages fr bth and -DOM ver 90%. Whereas the F-distributin has a percentage f crrect classificatin between 80% and 90%, and fr the ther tw distributins, S and X, the percentages f crrect classificatin are between 60% e 70%. It is wrth nting that fr all the distributins presented in this wrk, ur methd increases the percentage f crrect classificatin when the number f examples is greater, as can be seen in Fig. 3, where fr 400 and 800 examples per sample

6 (a) Distributin (b) (-SLS) (c) -AG (-SLS) (d) -DOM (-SLS) (e) (best) (f) -AG (best) (g) -DOM (best) Fig. 2. Graphic representatin f the results btained fr the distributins befre mentined. The first fur clumns were btained using 800 examples and represent: (a) Data distributin (F, S, Simple and X), (b),(c) and (d) Results with the three methds used in this wrk. The last clumns represent the best classificatin rate btained fr each distributin using (e), (f) -AG and (g) -DOM, respectively. TABLE I C LASSIFICATION RATES OBTAINED USING : (i) (G RADIENT D ESCENT ); (ii) -DOM ( WITH D IALECTICAL O PTIMIZATION ) AND (iii) -AG ( WITH G ENETIC A LGORITHMS ), FOR EACH DISTRIBUTION WITH AN IDEAL CLASSIFICATION RATE REPRESENTED IN PARENTHESIS. E XAMPLES F (87.65%) % 75.02% 74.7% 76.86% -DOM 83.32% 86.24% 86.64% 84.69% -AG 59.94% 73.29% 74.52% 76.77% N UMBER OF S TRAIGHT L INE S EGMENTS PER CLASS 2 3 -DOM -AG -DOM -AG 85.99% 84.84% 84.48% 86.75% 85.49% 73.63% 86.88% 87.7% 80.94% 87.06% 86.94% 8.76% 86.95% 86.72% 85.02% 82.23% 86.77% 79.30% 86.34% 86.99% 82.5% 86.7% 87.4% 75.70% 4 -DOM -AG 84.4% 86.46% 67.9% 83.30% 87.3% 8.7% 83.95% 86.5% 75,04% 86.36% 86.55% 78.6% S (68.78%) % 59.09% 59.0% 58.93% 6.79% 65.37% 66.38% 66.72% 59.72% 60.27% 60.6% 59.45% 66.03% 66.87% 66.82% 59.8% 60.3% 65.32% 66.86% 67.90% 56.27% 64.80% 60.94% 66.5% 65.48% 62.89% 66.4% 59.76% 62.84% 65.90% 66.99% 67.66% 60.3% 63.00% 62.52% 65.9% 62.22% 63.90% 60.54% 59.52% 59.0% 65.22% 66.90% 67.78% 62.74% 58.92% 58.82% 62.28% S IMPLE (93.90%) % 92.5% 92.53% 9.75% 92.92% 92.66% 93.9% 93.50% 86.92% 89.30% 89.60% 90.02% 92.78% 92.55% 92.6% 92.56% 92.99% 92.62% 92.77% 93.56% 9.98% 9.82% 93.59% 93.3% 92.8% 92.49% 92.68% 92.39% 93.0% 92.02% 93.46% 93.59% 92.48% 92.22% 9.8% 93.04% 92.78% 92.4% 92.68% 92.70% 92.96% 92.66% 93.45% 93.5% 88.92% 92.0% 93.08% 92.92% X (66.70%) % 66.26% 65.9% 66.50% 65.75% 65.92% 65.98% 66.46% 40.79% 37.4% 29.65% 9.96% 66.04% 66.29% 65.% 66.3% 64.99% 64.48% 64.57% 65.90% 38.53% 62.70% 62.08% 65.25% 65.33% 66.09% 63.85% 65.93% 64.03% 62.98% 64.06% 66.2% 64.83% 6.70% 65.56% 64.9% 66.24% 66.8% 63.66% 66.35% 63.69% 60.96% 63.36% 65.8% 58.52% 62.77% 60.76% 63.7%

7 ur methd btained a percentage equal t r greater than the ne btained by the gradient descent methd. Classificatin Rate (%) * * * * * x x + * x dist_f dist_s dist_simples dist_x x x DOM Number f Examples Fig. 3. Cmparisn between bth methds and -DOM using the classificatin rates btained frm applying the SLS classifier with 3-SLS n the fur distributins described befre. D. Gradient Descent with DOM vs. Gradient Descent with Genetic Algrithms Sme experiments were perfrmed t cmpare ur hybrid methd with anther ptimizatin algrithms, in this case we chse Genetic Algrithms [9], [0], []. These techniques are cmmnly applied t find apprximate slutins in ptimizatin prblems and they belng t a particular class f evlutinary algrithms which are inspired n evlutinary bilgy such as natural selectin, mutatin and inheritance. The algrithm perates by iteratively updating a set f individuals, called the ppulatin. The ppulatin cnsists f many individuals called chrmsmes. All members f the ppulatin are evaluated by the fitness functin n each iteratin. A new ppulatin is then generated by prbabilistically selecting the mst fit chrmsmes frm the current ppulatin. Sme f the selected chrmsmes are added t the new generatin while thers are selected as parent chrmsmes. Parent chrmsmes are used fr creating new ffsprings by applying genetic peratrs such as crssver and mutatin [2]. Here the ppulatin is generated in the same way as DOM, each chrmsme represents a ptential slutin t the ptimizatin prblem (i.e. [L 0 L ]). Cnsequently, a ppulatin is a set f pssible slutins and initially is randmly generated. Like in DOM methd, there are sme parameters in genetic algrithms that must be prperly tuned. In particular the + x x number f chrmsmes fr initial ppulatin (set t 200) and the number f iteratins (set t 00). In Table I, can be seen the classificatin rates btained frm applying the SLS classifier using in this case a cmbinatin f gradient descent and genetic algrithms. Fr F-distributin the rates have a difference f abut 5% r 7% percent frm the classificatin rates btained with -DOM. In S-distributin, this new cmbinatin get the highest rate amng all the ther methds, but nly fr 00 examples and 4-SLS. It was the same situatin with Simple-distributin, but this time using 400 examples and 2-SLS. In additin, fr Simple and X distributins the rates were clser t the ideal classificatin rate in sme cases. These methds were cmbined in the same way as gradient descent with DOM: (i) generate ppulatin; (ii) apply gradient descent t each chrmsme in ppulatin and (iii) the remaining AG functins. The results suggests that ur prpsed methd btained better classificatin rates than the cmbinatin f gradient descent with genetic algrithms, thus DOM has mre iteratins with the pssible slutins and the peratrs in DOM were redefined in rder t adapt them t ur prblem. E. Public Data Set The prpsed hybrid methd was als applied t ne public data set frm the UCI Machine Learning Repsitry [3]: the Breast Cancer Wiscnsin (Diagnstic) Dataset, with 2 classes (B fr Benign and M fr Malign) and 0 attributes (features). The experiment was dne using a sample with 400 examples fr training and a sample with 300 examples fr testing. The applied SLS classifiers have, 2, 3 and 4 SLSs per class. Table II presents and cmpares the results btained by ur methd with the nes btained by [4], fr the same data set, but with a sample with 682 examples when using the gradient descent methd. As we can see in Table II, ur hybrid methd had a better perfrmance using SLS classifiers with 3-SLSs and 2-SLSs. It was nt pssible t btain sme figures t represent the classificatin fr this data set, because it is 0-dimensinal data set. TABLE II CLASSIFICATION RATE FOR BREAST CANCER DATA SET METHOD NUMBER OF SLSS PER CLASS GradDesc 96.78% 96.92% 96.34% 96.78% GradDescDOM 96.66% 97.32% 96.99% 82.94% VI. CONCLUSIONS This paper presents a new methd fr training SLS classifiers, with a hybrid between gradient descent and dialectical ptimizatin methds. The gal is t find the best psitin f the SLSs in rder t minimize the MSE errr. Fr that, in [4], it was applied the gradient descent methd t ptimize the search f this ptimal psitins. In this wrk, ur main cntributin is t imprve the training phase (ptimizatin

8 f SLSs psitins), cmbining the gradient descent with an evlutinary methd fr ptimizatin based n philsphical cncepts such as dialectics. Cmparing the methds previusly mentined, all f them have a high classificatin rate, but n the hybrid methd case (-DOM), the distributins are better fitted with the btained SLSs psitins. In additin, we can say that the best SLS classifier fr the fur prpsed artificial distributins and the public data set, must have 3 SLS per class, because it has imprved the classificatin rate in an average f 2%. The prpsed hybrid methd has shwn a very gd classificatin fr F and S distributin, althugh the classificatin rate was nt imprved fr X-distributin, which has the lwest imprvement percentage abut 0.5%. While this methd imprves the classificatin rate, the cmputatin time fr the training algrithm increases because f the multiple iteratins f evlutinary DOM and gradient descent methd. In additin it has been studied the use f threads n the implementatin t reduce the training time. Althugh, in this paper we use sme predefined distributins and ne public dataset, the presented results indicate that the SLS classifier using the prpsed hybrid methd can be ptentially used in Cmputer Visin prblems. We plan t d this analysis fr future wrk and als extend the SLS binary classifier t a multiclass classifier. [2] R. Chandra and C. Omlin, The cmparisn and cmbinatin f genetic and gradient descent learning in recurrent neural netwrks: An applicatin t speech phneme classificatin, in Artificial Intelligence and Pattern Recgnitin, 2007, pp [3] A. Asuncin and D. J. Newman, UCI Machine Learning Repsitry, ACKNOWLEDGMENT Financial supprt fr this research has been prvided by FAPESP and CNPq. The authrs are grateful t annymus reviewers fr the critical cmments and valuable suggestins. REFERENCES [] R. Duda, P. Hart, and D. Strk, Pattern Classificatin, 2nd ed. New Yrk: Wiley, 200. [2] A. Jain, R. Duin, and J. Ma, Statistical pattern recgnitin: a review, Pattern Analysis and Machine Intelligence, IEEE Transactins n, vl. 22, n., pp. 4 37, Jan [3] J. Ribeir and R. Hashimt, A new machine learning technique based n straight line segments, in ICMLA 2006: 5th Internatinal Cnference n Machine Learning and Applicatins, Prceedings. IEEE Cmputer Sciety, 2006, pp [4], A new training algrithm fr pattern recgnitin technique based n straight line segments, in Cmputer Graphics and Image Prcessing, SIBGRAPI 08. XXI Brazilian Sympsium n, ct. 2008, pp [5], Pattern Recgnitin, Recent Advances. I-Tech, 200, ch. Pattern Recgnitin Based n Straight Line Segments, bk Chapter. [6] W. D. Sants and F. D. Assis, Optimizatin based n dialectics, in IJCNN, 2009, pp [7] W. D. Sants, F. D. Assis, R. D. Suza, P. Mendes, H. Mnteir, and H. Alves, Dialectical nn-supervised image classificatin, in Prceedings f the Eleventh cnference n Cngress n Evlutinary Cmputatin, ser. CEC 09. Piscataway, NJ, USA: IEEE Press, 2009, pp [Online]. Available: id= [8] D. Michie, D. Spiegelhater, and C. Taylr, Intrductin t ptimizatin thery, 973. [9] D. Gldberg, Genetic Algrithms in Search, Optimizatin and Machine Learning, st ed. Bstn, MA, USA: Addisn-Wesley Lngman Publishing C., Inc., 989. [0] R. Haupt and S. Haupt, Practical Genetic Algrithms. Wiley- Interscience, [] J. Hlland, Adaptatin in natural and artificial systems. Cambridge, MA, USA: MIT Press, 992.