Support Vector Machines versus Artificial Neural Networks for wood Dielectric Loss Factor estimation

Transcription

1 Suort Vector Machines versus Artificial Neural Networks for wood Dielectric Loss Factor estimation Lazaros Iliadis 1*, Stavros Tachos, Stavros Avramidis 3 Shawn Mansfield 3 1 Democritus University of Thrace, Deartment of Forestry & Management of the Environment & Natural Resources, 193 Pandazidou st., 6800 N Orestiada, Greece, Aristotle University of Thessaloniki, Greece 3 University of British Columbia, Deartment of Wood Science, Vancouver, Canada * 1 liliadis@fmenr.duth.gr staxos@gmail.com 3 stavros.avramidis@ubc.ca 4 Abstract: This research effort aims in the estimation of Wood Loss Factor by emloying Suort Vector Machines. For this urose exerimental data for two different wood secies were used. The estimation of the dielectric roerties of wood was done by using various Kernel algorithms as a function of both ambient electro-thermal conditions alied during drying of wood and basic wood chemistry. Actually the best fit neural models that were develoed in a revious effort of our research team were comared to the Kernels aroaches in order to determine the otimal ones. Keywords: Wood Loss Factor, Gaussian Suort Vector Machines, Polynomial Suort Vector Machines, Artificial Neural Networks 1. Introduction It has been shown in the literature that the knowledge of dielectric roerties of wood can be used to determine its density and moisture content nondestructively. On the other hand this knowledge can lead to the detection of hand knots, siral grain, and other defects [14]. From a theoretical ersective a better understanding of the molecular structure of wood and wood-water interactions can be obtained [9]. Various aroaches towards this direction have been reorted in the literature, namely the radio frequency vacuum drying [11][1][13] and the high frequency electric field heating, such as the veneer and finger-joint gluing and arallam manufacturing [19]. Thus, estimation of the dielectric constant (ε'), the loss tangent (tanδ) and the loss factor (ε") lays a catalytic role in the rocess of design, control, otimization and simulation. Recently Koumoutsakos [11] has shown that ε" is roortional to the thermal energy transferred to the wood, during radio frequency vacuum drying (RFV) rocess. RFV heating is a volumetric method where thermal energy is roduced simultaneously through a ile of lumber which is laced inside an electromagnetic field. In this rocess the moisture starts its transortation from the center to the surface as soon as the wood is exosed to the field [1]. In fact, the electric ower converted to thermal is given by the following formula 1.

2 PD = 5.56x10-11 E f ε" (1) It should be clarified that PD is the ower density in W/m 3, E is the field strength in V/m and f is the frequency measured in Hz Research efforts in the ast have shown that there exists a strong relationshi between the dielectric roerties and wood attributes studied lus field frequency alied [18][19]. It is a fact that desite all conducted research so far, there is very little resective knowledge regarding the effect of wood chemical comosition on the determination of ε". Actually, Norimoto [15][16] have investigated the dielectric roerties of some wood chemical constituents as a function of frequency and temerature, but no attemt was made towards the correlation of their ercent content in the cell-wall comosition to the gross wood ε" values. Another oint that motivated this study is the fact that the data that have been used so far to construct such models have originated from assorted secies, under different thermo-hysical conditions and variable frequencies and thus, they are not suitable to be emloyed in drying modeling []. This research effort aims in the develoment of Suort Vector Machines models (Gaussian, Fuzzy weighted and Polynomial) caable of determining reliably the value of ε" based on ambient electro-thermal conditions and also on basic wood chemistry. On the other hand a comarative study is erformed with a revious study of our research team that used Artificial Neural Networks (ANNs) for the same urose []. This comarison aims in the determination of the otimal aroaches that can be used to offer reliable and most of all cost and time effective aroximation.. Materials and methods.1. Obtaining the exerimental data This study emloyees ε" values and macro-hysical data that were already ublished by Zhou and Avramidis []. According to Zhou and Avramidis all-sawood and allheartwood western hemlock [Tsuga heterohylla (Raf.) Sarg.], and all-heartwood western red cedar [Thuja licata Donn] secimens were evaluated in the radial direction (thickness) and at various moisture contents and temerature levels were exosed to two levels of electric field voltage. After the exosure was terminated the ε" values were calculated indirectly by erforming heating studies at a MHz fixed frequency with a laboratory size RFV dryer [] []. Also the above described wood secies were analyzed by Zhou and Avramidis [] regarding their chemical comosition as follows: air dried wood samles were ground in a Wiley Mill to ass a 40-mesh screen and extracted in a Soxhlet aaratus with acetone for 1 hours, and extractives determined gravimetrically []. According to Zhou and Avramidis [], the content of Lignin was determined using a modified Klason aroach derived from the TAPPI standard method T om-98. For more details refer to [].

3 .. 5-Fold-Cross Validation Cross-validation is a methods for estimating generalization error based on "resamling" [1] [7] [8] [17]. The resulting estimates of generalization error are often used for choosing among various models. In k-fold cross-validation, the data are divided into k subsets of (aroximately) equal size. The model is trained k times, each time leaving out one of the subsets from training, but using only the omitted subset to comute whatever error criterion interests you. If k equals the samle size, this is called "leave-one-out" cross-validation. The division of the data set was done by the use of MATLAB s crossvalind function, which is included in the Bioinformatics Toolbox..3. ε-sv Regression (ε-svr) Suort Vector Machines (SVM) are keeing the training error foxed while at the same time they are minimizing the confidence interval [10]. Their rimary target was attern recognition [3]. Now they are used for both regression and classification. Vanik [0] [10] introduced the following loss function that ignores errors less than a redefined value 0 y f x max 0, y f x () It is a fact that the ε-svr algorithm offers in many cases the otimal function of the form: f x k w, x bw, x R N, b R (3). The whole idea is based on the determination of the function with the minimum testing error. The roblem in this case is that the minimization of the above function is not ossible due to the fact that the robability distribution P is unknown. Consequently, the actual solution can be reached by minimizing the following normalized risk function 4. R em f 1 SVR em w C R f (4) 1 is the function of emirical risk Rem f L f, xi, yi the loss function is L( f, xi, yi ) yi f xi Also it is clear that (5) whereas i 1 (6) and it ignores errors less than. w is related to the comlexity of the model, whereas C SVR is a constant value determining the oint that relates R f em to w [6]. Obviously, the objective which is the Minimization of the above risk function 4 can be hased as a secific otimization roblem which hases the constraints described below: The aim is the minimization of the following function: 1 1 w,, w CSVR i i (7) i1

4 The constraints in this case are: yi k w xi b i i i k w x b y (8) and, i i i (9) and of course, 0 (10). Obviously (and this can be seen in the following figure 1) i, i are the distances of the training data set oints from the e-zone. All of the errors that are smaller than are ignored. As it is clearly shown, stands for the distance of a oint that can be found above the e-tube i zone and i stands for the distance of a oint that is located below the e-tube zone. It is well known that in the case of the alication of Lagrange multiliers the roblem would corresond to a double otimization one as follows: Maximize 1 W a, a a i ai a i ai yi a i ai a j a j k xi, x j (11) i1 i1 i, j1 subject to ai a i CSVR 0 (1) and ai 0, i1 (13). Figure 1. ε-svr for Regression Based on Vanik and Kecman [0][10], the otimization roblem can be solved by the use of the following linear extension of the kernel functions (14) and (15) i i i (14) i1 i i i i1 w a a x,,, f x a a a a k x x b (15)

5 where b is estimated by the following function 16 b average i signai ai yk a j a j k x j, xi (16) j The training set vectors x i related to nonzero values of ai a i are called SVM. The following function 17 resents the Radial Basis function (RBF) kernel x x i f x, a, a ai aiex b (17) i1 RBF The involved arameters,, RBF are very significant and they determine to a great level of extend the good fit of the ε-svr. It should be secified that is the RBF RBF kernel s standard deviation, has a constant value that determines the oint where the emiric error is related to comlexity and finally the arameter is the width of the ε-zone. In statistics, when we consider the Gaussian robability density function σ RBF is called the standard deviation, and the square of it, the variance..4. Fuzzy weighted Suort Vector Regression The fuzzy weighted SVR with a fuzzy artition first emloys the fuzzy c-mean clustering algorithm to slit training data into several training subsets. Then, the localregression models (LRMs) are indeendently obtained by the SVR aroach for each training subset. Finally, those LRMs are combined by a fuzzy weighted mechanism to form the outut. Exerimental results show that the roosed aroach needs less comutational time than the local SVR aroach and can have more accurate results than the local/global SVR aroaches does [5]. 3. Alication In this section of the aer, the alication results of the Global SVR aroach for all existing data vectors will be resented thoroughly. The regression was done for different kernels, namely the Gaussian (RBF) the Fuzzy weight SVM and the Polynomial one. In order to erform Global SVR a global regression model was develoed. The regression was erformed by the use of the LIBSVM v.9 (htt:// [4] which is encoded in C++ and offers a Matlab Interface. In this secific alication the RBF-Kernel was alied Performing trial and error for the Gaussian Kernel The initial data set comrised of one hundred forty-four (144) sets of exerimental data, where the ε" was measured under different temerature and moisture conditions and for various tyes of chemical comosition.

6 Minimum values of % During and, several hundreds of exeriments were erformed in order to determine the otimal arameters values. More secifically in the case of the Gaussian (RBF) Kernel, 400 exerimentation cases (alications of the 5-fold cross validation) were constructed with the integer value of σ RBF ranging from 1 to 0 and with the value of the arameter γ ranging from 1 to 14. In the same case the e arameter took real values from the following set {0.003, 0.004, 0.005, 0.006, 0.007, 0.008, 0.009, 0.1, 0.1, 0.14, 0.16, 0.18,0.0, 0.} This means that totally 400*14=5600 trials were erformed during the 5-fold cross validation rocess. The Root Square Errors () roduced in each or cycle were stored and then they were averaged each time 5-fold cross validation was emloyed. In each trial the initial data set was divided into five subsets (4 training and one testing) where five training and testing cycles were erformed (each time changing the testing set in a round robin manner). Totally 5600 Average Root Square Errors were obtained (A) one for each 5-fold cross validation cycle and for each unique combination of the arameters, together with the corresonding % Average Percent Error (). In the case of the Gaussian Kernel the otimal SVM model was roduced for the σ RBF = and for the value of the arameter γ=1 in the case with e=0.04 as it is shown in the following table 1. Table 1: Performance of the Otimal Gaussian Kernel Otimal Gaussian Kernel SVM (5-fold cross validation) of σ RBF of γ of e The following figures and 3 resent the evolution of the and of the resectively, according to the values of the e arameter for the Gaussian Kernel. It is clearly shown that the best model should have an e value equal to Performing Gaussian SVM with 5-fold cross validation s of Parameter e Parameter e Minimum % Figure. Evolution of the for different values of the arameter

7 Minimum Gaussian SVM using 5-fold cross validation Parameter e s of Parameter e 0 Figure 3. Evolution of the for different values of the e arameter For the case of the Polynomial Kernel, the σ RBF again took integer values from 1 to 0 and the arameter γ from 1 to 14. However the e arameter took real values from the following set {0.005, 0.1, 0.15, 0.0, 0.5}. The erformance of the best fit Polynomial Kernel is shown in the following table and it was achieved for σ RBF= 3 γ=18 and e=0.1 Table : Performance of the Otimal Polynomial Kernel Otimal Polynomial Kernel SVM (5-fold cross validation) of σ RBF of γ of e For the Fuzzy weight Suort Vector Machine the σ RBF again took integer values from 1 to 0 and the arameter γ from 1 to 14. Also the e arameter took real values from the following set {0.005, 0.1, 0.15, 0.0, 0.5}. The erformance of the best fit Fuzzy weight SVM is shown in the following table 3 and it was achieved for σ RBF= 3 γ=16 and e=0.05 Table 3: Performance of the Otimal Fuzzy weight SVM Otimal Fuzzy weight SVM (5-fold cross validation) of σ RBF of γ of e

8 Loss Factor values Fuzzy Weighted SVMs Cases RealOutut EstimatedOutut Figure 4. Actual versus estimated values of Loss Factor for Fuzzy Weighted SVMs 4. Discussion Comarative analysis The results of the e-svm regression were comared to the results of the Artificial Neural Network that was develoed by our research team in a revious study [] and also the results of the three SVM kernels were comared to each other. In [] the otimal ANN that estimated wood loss factor for the same inut variables was found to be a multi layer Back roagation one, with Sigmoid transfer function. It was found to have eleven (11) rocessing elements in the inut layer, nine (9) neurons in the hidden layer and of course one neuron in the outut layer []. Table 4: Performance of the otimal ANN for the estimation of loss factor in wood Otimal ANN Otimal ANN R Otimal ANN Otimal ANN R R = R = The above table 4 resents the erformance of the best fit ANN. It is clearly shown that the Gaussian Kernel aroach has by far much lower RMS error than the Polynomial and the Fuzzy weight SVM and also than the best fit ANN. Thus it can be considered the most reliable modeling aroach.

9 This research effort is quite new and rocesses a high degree of innovation due to the fact that similar research has not been erformed in wood science literature before. It is the final art of a wider effort to roduce rational and useful Soft Comuting regression models towards the estimation of wood dielectric roerties. It has been clearly shown that all three Kernel aroaches and the ANN model are reliable and they can be used alternatively in wood industry alications. However the Radial Basis function (Gaussian) Kernel has roven to roduce a much more romising model. Further exeriments will be erformed to roduce more data vectors and comlimentary modeling efforts will be erformed in the future. References: 1. Avramidis S., (1999). Radio Frequency Vacuum Drying of wood. Proceedings of the International conference of cost Action E1 5 Wood Drying. Theme State of the Art for Kiln Drying. Avramidis S., Iliadis L., Mansfield S. (006) Wood Dielectric Loss Factor Prediction with Artificial Neural Networks Wood Science and Technology ISSN: Volume 40, Number 7, Journal of the International Academy of wood Science. Sringer Berlin Heidelberg 3. Boser B., Guyon I., and Vanik V.: A training algorithm for otimal margin classifiers In Fifth Annual Worksho on Comutational Learning Theory, ages , Pittsburgh, ACM (199) 4. Chang C.C., Lin C. J.(009). LIBSVM A Library for Suort Vector Machines, on line at: htt:// 5. Chuang C. C. (007). Fuzzy Weighted Suort Vector Regression with a Fuzzy Partition Systems,Man, and cybernetics, Part B: Cybernetics, IEEE Transaction on. Vol 37, Issue Davy M. (005). An Introduction to Suort Vector Machines and other kernel algorithms, Proc. of the 9 th Engineering Alications of Neural Networks Conference, Lille, France 7. Efron, B. and Tibshirani, R.J. (1993), An Introduction to the Bootstra, London: Chaman & Hall 8. Hjorth, J.S.U. (1994), Comuter Intensive Statistical Methods Validation, Model Selection, and Bootstra, London: Chaman & Hall 9. Kabir M. F., Daud W. M., Khalid K. b., Sidek H. A., (001). Temerature deendence of the dielectric roerties of rubber wood. Wood and Fiber Science. V. 33() 10. Kecman V., (001). Learning and Soft Comuting. MIT Press, USA 11. Koumoutsakos, A., Avramidis, S., and S. Hatzikiriakos. 001a. Radio frequency vacuum drying Part I. Theoretical model. Drying Technol. 19(1): Koumoutsakos, A., Avramidis, S., and S. Hatzikiriakos. 001b. Radio frequency vacuum drying Part II. Exerimental model evaluation Drying Technol. 19(1): Koumoutsakos, A., Avramidis, S., and S. Hatzikiriakos Radio frequency vacuum drying Part III. Two dimensional Model, Otimization and

10 Validation. Drying Technol. 1(8): Martin P., R. Collet, P. Barthelemy, G. Roussy. (1987). Evaluation of wood characteristics: Internal scanning of the material by microwaves. Wood Sci. Technol. 1: Norimoto, M Dielectric roerties of wood. Wood Research. No. 59/60: Norimoto, M., Hayashi, S. and T. Yamada Anisotroy of dielectric constants in coniferous wood. Holzforschung 3: Shao, J. and Tu, D. (1995), The Jackknife and Bootstra, New York: Sringer-Verlag 18. Siau, J.F Wood: influence of moisture on hysical roerties. Deartment of Wood Science and Forest Products, VPI&SU 19. Torgovnikov, G. I Dielectric roerties of wood and wood-based material. Sringer-Verlag, Berlin Heidelberg, New York 0. Vanik V.N., S. Golowich, Smola A (1997).: Suort Vector method for function aroximation regression estimation and signal rocessing, In: Advances in Neural Information Processing Systems, vol. 9, Cambridge, MA:MIT Press, (1997) 1. Weiss, S.M. and Kulikowski, C.A. (1991), Comuter Systems That Learn, Morgan Kaufmann. Zhou, B., and S. Avramidis On the loss factor of two B.C. softwoods. Wood Sci. and Technol., 33(4):99-310