Learning Matrix Quantization and Variants of Relevance Learning

Transcription

1 Learning Matrix Quantization and Variants of Relevance Learning K. Domaschke1, M. Kaden2, M. Lange2, and T. Villmann2 1 - Life Science Inkubator Dresden, Dresden - Germany 2- University of Al. Sciences Mittweida - Det. of Mathematics Mittweida, Saxonia - Germany Abstract. We roose an extension of the learning vector quantization framework for matrix data. Data in matrix form occur in several areas like gray-scale images, time deendent sectra or fmri data. If the matrix data are vectorized, imortant satial information may be lost. Thus, rocessing matrix data in matrix form seems to be more aroriate. However, it requires matrix dissimilarities for data comarison. Here Schatten--norms come into lay. We show that they can be used in a natural way relacing the vector dissimilarities in the learning framework. Moreover, we transfer the concet of vectorial relevance learning also to this new matrix variant. We aly the resulting learning matrix quantization aroach to the classification of time-deendent fluorescence sectra as an exemlary real world alication. 1 Introduction Classification of comlex data is still a challenging task in machine learning. Many methods were develoed for rocessing vectorial data ranging from rototye based classifiers like Suort Vector Machines (SVM, [17]) or the family of Learning Vector Quantizers (LVQ, [12]) to classification trees [5]. LVQ rovides an intuitive and robust algorithmic aroach based on Euclidean distance learning, frequently achieving romising results. One of the key extensions of LVQ is relevance learning, which weights the data dimensions erformance imrovement [8]. It can be further imroved taking the data dimension correlations into account [18]. These methods can also be alied to data in matrix form (matrix data), if those data are simly vectorized. However, vectorization may destroy satial relations within a matrix, such that imortant information is lost. For this reason, we roose an extension of the LVQ framework for matrix data using dissimilarities exlicitly based on matrix norms. A standard norm alied for distances between matrices is the Schatten--norm, as a natural counterart of the l -norms for vectorial data. In the following we exlain the modification of the LVQ required to rocess matrix data and denote this new variant as Learning Matrix Quantization (LMQ). The automatic weighting of the dimensions the classification erformance imrovement is a vitally extension of the LVQ known as relevance learning [8]. In this contribution we also discuss concets of relevance learning in LMQ. An exemlary real world alication illustrates the usefulness of the LMQ with relevance learning. 2 Learning Vector Quantization based on l -norms LVQ was introduced by KOHONEN as an intuitive rototye based learning classifier for vector data heuristically aroximating a Bayes-classifier [11]. The generalized learn M. Kaden is suorted by a grant of the Euroean Social Fund, Saxony (ESF). 13

2 ing vector quantization (GLVQ) model [15] is a cost function based modification of the intuitive LVQ aroximating the classification error as the objective to be minimized. The rototyes of the GLVQ model are the set W = {wk Rn, k = 1... M }. Each data vector v V Rn of the training data belongs to a class xv C = {1,..., C}. The rototyes are labeled by yw C such that there is at least one rototye er class. Thereby, the dissimilarities between data oints v and an arbitrary rototye w are P n judged in terms of a measure d based on the l -norm kv wk = i=1 vi wi. + In articular, we consider d (v, w) = kv wk. Further d+ (v) = d (v, w ) denotes the dissimilarity between the data vector v and the closest rototye w+ with the same class label yw+ = xv, and d (v) = d (v, w ) is the dissimilarity degree for the best matching rototye w with a class label yw different from xv. In stan2 dard GLVQ, the squared Euclidean distance d2 (v, w) = (kv wk2 ) is used. GLVQ maximizes the hyothesis margin d+ (v) d (v) [2, 7]. The resective cost function minimized by GLVQ is EGLV Q (W ) = 1 X f (µ (v)) 2 (1) v V where f is a monotonically increasing squashing function usually chosen as a sigmoid or the identity function and µ (v) = d+ (v) d (v) d+ (v) + d (v) (2) is the classifier function with µ (v) [1, 1]. Learning in GLVQ is erformed by the stochastic gradient descent learning for the cost function EGLV Q. For the more general l -norm, the rototye udates of w+ and w become w± µ (v) d± (v) ± µ (v) d (v) w± (3) d± with w± are the formal derivatives of d (v, w) [14]. Several extensions were roosed to adat this basic scheme according to secific classification tasks. A recent overview can be found in [10]. One of the most successful modifications is relevance learning in GLVQ (GRLVQ, [3, 8]). In the GRLVQ, the dissimilarity measure d (v, w) is relaced by d,r (v, w) = kr (v w)k (4) with Pn the relevance vector r consisting of the relevances ri with normalization i=1 ri = 1. Here, r x denotes the Hadamard roduct. The relevances ri weight each data dimension indeendently to imrove the classifier erformance and can also be adated by stochastic gradient learning according to µ (v) d+ µ (v) d,r (v),r (v) ri. (5) µ (v) ri ri d+ d,r (v),r (v) 14

3 The GRLVQ can be further generalized when using d,ω (v, w) = kω (v w)k (6) where Ω Rm n is a maing matrix. Then Λ = ΩT Ω can be interreted after stochastic gradient learning as a classification correlation matrix combining those data dimensions, which suorts the class searabilities [18]. 3 Learning Matrix Quantization based on Schatten--norms Matrix data are the obvious extension of vector data. However, those data are frequently rocessed alying a vectorization scheme and then utilizing of vector methods. Thus, structural information can be destroyed by the vectorization. Therefore, we roose to modify LVQ for matrix data V Vm,n Rm n. In this case, the rototye set W consists of matrices Wk Rm n. In the next ste, we have to relace the l -norm kvk determining the dissimilarity measure d (v, w) by a resective matrix norm. Mathematically, the set Rm n of matrices is a vector sace, which can easily be equied with a resective matrix norm fulfilling the usual norm axioms. Examles are the maximum norm defined as the maximum absolute value of the matrix entries or the Ky-Fan-Norm as the sum of the first K singular values of the matrix [9]. More sohisticated norms are those, which additionally are consistent with the matrix multilication, i.e. satisfying the Cauchy-Schwarz-inequality kx Yk kxk kyk [6]. One rominent examle for these so-called sub-multilicative norms is the Schatten--norm q s (A) = tr( A ) (7) where tr( ) is the trace oerator [16]. For = 2, the Schatten--norm reduces to the Frobenius norm s2 (A) = tr(aat ). Schatten--norms are closely related to l norms according to s (A) = kσ (A)k, where σ (A) denotes the vector of the singular values of A. Based on (7), we take δ (V, W) = (s (V W)) (8) as a dissimilarity measure comarable to d (v, w), which can be lugged into the cost function (1) reading now as EGLM Q (W ) = 1 2 X f (µ ). (9) V Vm,n Alying the same formalism as for GLVQ, we obtain the formal udate rules W± µ δ± ± µ δ W± δ ± (10) = 2W±. Accordingly, the algorithm is for W±. For = 2, we simly get W ± denoted as Generalized Learning Matrix Quantization (GLMQ). 15

4 4 Relevance Learning in GLMQ In the following we will discuss variants of relevance learning for LMQ. The obvious counterart to the vector variant (4) for Schatten--norms would be δ,r (V, W) = (s (R (V W))) (11) with the relevance matrix R weighting indeendently each entry of a data matrix V by the Hadamard roduct. For = 2, this aroach yields δ2,r (V, W) = T tr (R (V W)) (R (V W)) tic gradient of (9) becomes R µ δ. The formal relevance udate as the stochas- + δ,r δ,r µ µ + δ,r δ,r! (12) where 2,R = 2R (V W) (V W) is obtained for = 2. It is ac δ = comanied by the resective rototye derivatives involving the term 2,R W± 2R R (V W± ) for = 2. We denote this variant as Hadamard-RelevanceLearning (HRL). We notice that the HRL learning alying the Frobenius-norm can be transfered to the vectorial counterart GRLVQ with the Euclidean norm. Alternatively to the Hadamard roduct based relevance weighting introduced in (11) we can think about the weighting δ,r (V, W) = (s (R (V W))) (13) using the ordinary matrix multilication. Here, a weighted linear relevance mixing is alied, taking artially linear combinations of matrix entries into account. The relevance udate is structurally equivalent as in (12) aying attention to the new derivative δ,r δ T. For = 2 we get 2,R = 2R (V W) (V W). The resective δ2,r rototye udate involves the term W± = 2RT R (V W± ) and the method is referred here as Multilicative Relevance Learning (MRL). At this oint, alying the MRL with the Frobenius norm the satial matrix information is taken into account. 5 Alication in Time Resolved Laser induced Fluorescence Sectroscoy In the last 20 years it turned out that one imortant way to distinguish between different substances or to characterize a comosite samle is the Time Resolved Laser induced Fluorescence Sectroscoy (TRLFS). This method selectively excites the molecules of a desired substance to a secified state of higher energy by a laser beam. In the relaxing hase the molecules temorally dissiate this energy thermally or otically in terms of auto-fluorescence. The latter leads to time-deendent emission and can be measured as a signal reflecting the substance secific satio-temoral characteristics of this rocess. The obtained two-dimensional signal (matrix) shows the fluorescence intensity [a.u.] deendent on the emission energy [nm] and the time [ns] after fluorescence initialization. These sectra are called Time Resolved Fluorescence Sectra (TRFS). Deending on the exeriment in use, the emission sectra of different biological comounds can 16

5 acc. in % std. dev. GLVQ GRLVQ GLMQ GRLMQ (HRL) GRLMQ (MRL) Table 1: Classification test results for the TRFLS dataset averaged over 10 reetitions of 10-fold cross-validatations. be quite similar and difficult to distinguish, so the imortant information is the time deendence [13,.578]. For the roblem to be investigated in this study, the given data includes only TRFS of a single biological comound at two different excitation energies (energy level/class). The underlying assumtion is that the substrate resonse secifically and this behavior is reflected in the measured signal matrices [4]. However, the secificity of the signal may be overlayed by noise. Therefore, the time-deendent logarithmic signals are integrated with resect to time to reduce the noise. However, according to this vectorization the time-deendent information is lost, which may contain substantial information. Thus we obtain for an exeriment both vector and matrix data for GLVQ and GLMQ analysis, resectively, for comarison. In articular, we have 60 data for each energy class with a matrix resolution of for emission energy and time, resectively. For both algorithms we used only one rototye er class. All test accuracies resented as the results are obtained as average of 10 random reetitions of 10-fold crossvalidations. Further, we alied relevance learning for comarison, whereby for GLMQ we considered both introduced variants. The results are collected in Tab. 1. We observe that the vector variant GLVQ is slightly imroved by relevance learning, as exected. However, a similar imrovement is obtained by GLMQ without relevance learning. If relevance learning is included, a further imrovement may be achieved. The imrovement amount deends on the kind of the relevance learning method. While alication of the HRL, which is mathematically comarable to GRLVQ, does not delivers an imrovement, the rogress becomes moderate if MRL is used. This can be dedicated to the fact that MRL takes more structured matrix relations into account than HRL. 6 Conclusion and Future Work In the contribution we roose the Learning Matrix Quantization framework as extension of GLVQ for matrix data. We emhasize the fact that otherwise vectorization of matrix data may destroy structured information coded in the matrix. Further, we discuss ossibilities of relevance learning for GLMQ. So far we considered Hadamardrelevance and (left) multiicative learning. The latter one delivers better results, which may dedicated to the artially considered correlations within the data matrices. An obvious otion for future research would be to consider the right multilicative variant or, as a more sohisticated ossibility, combining both. This leads to the QR-norms as introduced in fmri-analysis to relate satio-temoral deendencies [1], but causing substantially increased numerical comlexity. Another, way could be to aly also the Kronecker-matrix-multilication offering another kind of structural combination of matrix entries [6, 9]. Finally, the vectorial matrix learning by GMLVQ would lead to tensor-based relevance learning for matrix data as adequate counterart. 17

6 References [1] G. Allen, L. Grosenick, and J. Taylor. A generalized least squares matrix decomosition. Journal of the American Statistical Association, Theory & Methods, 109(505): , [2] M. Biehl, B. Hammer, P. Schneider, and T. Villmann. Metric learning for rototye-based classification. In M. Bianchini, M. Maggini, F. Scarselli, and L. Jain, editors, Innovations in Neural Information Paradigms and Alications, volume 247 of Studies in Comutational Intelligence, ages Sringer, Berlin, [3] T. Bojer, B. Hammer, D. Schunk, and T. von Toschanowitz K. Relevance determination in learning vector quantization. In 9th Euroean Symosium on Artificial Neural Networks. ESANN Proceedings. D-Facto, Evere, Belgium, ages 271 6, [4] P. Clark, Z.-P. Liu, J. Zhang, and L. Gierasch. Intrinsic trytohans of CRABPI as robes of structure and folding. Protein Science, 5(6): , [5] R. Duda and P. Hart. Pattern Classification and Scene Analysis. Wiley, New York, [6] G. Golub and C. V. Loan. Matrix Comutations. Johns Hokins Studies in the Mathematical Sciences. John Hokins University Press, Baltimore, Maryland, 4th edition, [7] B. Hammer, M. Strickert, and T. Villmann. Relevance LVQ versus SVM. In L. Rutkowski, J. Siekmann, R. Tadeusiewicz, and L. Zadeh, editors, Artificial Intelligence and Soft Comuting (ICAISC 2004), Lecture Notes in Artificial Intelligence 3070, ages Sringer Verlag, Berlin-Heidelberg, [8] B. Hammer and T. Villmann. Generalized relevance learning vector quantization. Neural Networks, 15(8-9): , [9] R. Horn and C. Johnson. Matrix Analysis. Cambridge University Press, 2nd edition, [10] M. Kaden, M. Lange, D. Nebel, M. Riedel, T. Geweniger, and T. Villmann. Asects in classification learning - Review of recent develoments in Learning Vector Quantization. Foundations of Comuting and Decision Sciences, 39(2):79 105, [11] T. Kohonen. Learning Vector Quantization. Neural Networks, 1(Sulement 1):303, [12] T. Kohonen. Self-Organizing Mas, volume 30 of Sringer Series in Information Sciences. Sringer, Berlin, Heidelberg, (Second Extended Edition 1997). [13] J. Lakowicz. Princiles of Fluorescence Sectroscoy. Sringer, 3rd edition, [14] M. Lange and T. Villmann. Derivatives of l -norms and their aroximations. Machine Learning Reorts, 7(MLR ):43 59, ISSN: , htt:// fschleif/mlr/mlr_04_2013.df. [15] A. Sato and K. Yamada. Generalized learning vector quantization. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Advances in Neural Information Processing Systems 8. Proceedings of the 1995 Conference, ages MIT Press, Cambridge, MA, USA, [16] R. Schatten. A Theory of Cross-Saces, volume 26 of Annals of Mathematics Studies. Princeton University Press, [17] B. Schölkof and A. Smola. Learning with Kernels. MIT Press, Cambridge, [18] P. Schneider, B. Hammer, and M. Biehl. Adative relevance matrices in learning vector quantization. Neural Comutation, 21: ,