Learning with Tensor Representation

Transcription

1 Report No. UIUCDCS-R UILU-ENG Learnng wth Tensor Representaton by Deng Ca, Xaofe He, and Jawe Han Aprl 2006

2 Learnng wth Tensor Representaton Deng Ca Xaofe He Jawe Han Department of Computer Scence, Unversty of Illnos at Urbana-Champagn Yahoo! Research Labs Abstract Most of the exstng learnng algorthms take vectors as ther nput data. A functon s then learned n such a vector space for classfcaton, clusterng, or dmensonalty reducton. However, n some stuatons, there s reason to consder data as tensors. For example, an mage can be consdered as a second order tensor and a vdeo can be consdered as a thrd order tensor. In ths paper, we propose two novel algorthms called Support Tensor Machnes (STM) and Tensor Least Square (TLS). These two algorthms operate n the tesnor space. Specfcally, we represent data as the second order tensors (or, matrces) n R n R n2, where R n and R n2 are two vector spaces. STM ams at fndng a maxmum margn classfer n the tensor space, whle TLS ams at fndng a mnmum resdual sum-of-squares classfer. Wth tensor representaton, the number of parameters estmated by STM (TLS) can be greatly reduced. Therefore, our algorthms are especally sutable for small sample cases. We compare our proposed algorthms wth SVM and the ordnary Least Square method on sx databases. Expermental results show the effectveness of our algorthms. Introducton The problem of data representaton has been at the core of machne learnng. Most of the tradtonal learnng algorthms are based on the Vector Space Model. That s, the data are represented as vectors x R n. The learnng algorthms am at fndng a lnear (or nonlnear) functon f(x) = w T x accordng to some pre-defned crtera, where w = (w,, w n ) T are the parameters to estmate. However, n some stuatons, there mght be reason to consder data as tensors. For example, an mage s essentally a second order tensor, or matrx. It s reasonable to consder that pxels close to each other are correlated to some extent. Smlarly, a vdeo s essentally a thrd order tensor wth the thrd dmenson beng tme. Also, two consecutve frames n a vdeo are probably correlated. The work was supported n part by the U.S. Natonal Scence Foundaton NSF IIS /IIS Any opnons, fndngs, and conclusons or recommendatons expressed n ths paper are those of the authors and do not necessarly reflect the vews of the fundng agences.

3 In supervsed learnng settngs wth many nput features, overfttng s usually a potental problem unless there s ample tranng data. For example, t s well known that for unregularzed dscrmnatve models ft va tranng-error mnmzaton, sample complexty (.e., the number of tranng examples needed to learn well ) grows lnearly wth the Vapnk-Chernovenks (VC) dmenson. Further, the VC dmenson for most models grows about lnearly n the number of parameters (Vapnk, 982), whch typcally grows at least lnearly n the number of nput features. All these reasons lead us to consder new representatons and correspondng learnng algorthms wth less number of parameters. In ths paper, we propose two novel supervsed learnng algorthms for data classfcaton, whch are called Support Tensor Machnes (STM) and Tensor Least Square (TLS). Dfferent from most of prevous classfcaton algorthms whch take vectors n R n as nputs, STM and TLS take the second order tensors n R n R n 2 as nputs, where n n 2 n. For example, a vector x R n can be transformed by some means to a second order tensor X R n R n 2. A lnear classfer n R n can be represented as w T x + b n whch there are n + ( n n 2 + ) parameters (b, w, =,, n). Smlarly, a lnear classfer n the tensor space R n R n 2 can be represented as u T Xv + b where u R n and v R n 2. Thus, there are only n + n 2 + parameters. Ths property makes STM and TLS especally sutable for small sample cases. Recently there have been a lot of nterests n tensor based approaches to data analyss n hgh dmensonal spaces. Vaslescu and Terzopoulos (2003) have proposed a novel face representaton algrothm called Tensorface. Tensorface represents the set of face mages by a hgher-order tensor and extends Sngular Value Decomposton (SVD) to hgher-order tensor data. Some other researchers have also shown how to extend Prncpal Component Analyss, Lnear Dscrmnant Analyss, Localty Preservng Projecton, and Non-negatve Matrx Factorzaton to hgher order tensor data (Ca et al., 2005; He et al., 2005; Shashua & Hazan, 2005; Ye et al., 2004). Most of prevous tensor based learnng algorthms are focused on dmensonalty reducton. In ths paper, we extend SVM and Least Square based deas to tensor data for classfcaton. The rest of ths paper s organzed as follows: Secton 2 gves some descrptons on tensor space model for data representaton. The Tensor Least Square (TLS) and Support Tensor Machnes (STM) approaches for classfcaton n tensor space are descrbed n Secton 3. The expermental results on sx datasets from UCI Repostory are presented n Secton 4. In Secton 5, we provde a general algorthm for learnng functons n tensor space. Fnally, we provde some concludng remarks and suggestons for future work n Secton 6. 2 Tensor based Data Representaton Tradtonally, a data sample s represented by a vector n hgh dmensonal space. Some learnng algorthms are then appled n such a vector space for classfcaton. In ths secton, we ntroduce 2

4 Vector to Tensor Converson (a) x (b) Fgure : Vector to tensor converson. 9 denote the postons n the vector and tensor formats. (a) and (b) are two possble tensors. The x n tensor (b) s a paddng constant. a new data representaton model: Tensor Space Model (TSM). In Tensor Space Model, a data sample s represented as a tensor. Each element n the tensor corresponds to a feature. For a data sample x R n, we can convert t to the second order tensor (or matrx) X R n n 2, where n n 2 n. Fgure shows an example of convertng a vector to a tensor. There are two ssues about convertng a vector to a tensor. The frst one s how to choose the sze of the tensor,.e., how to select n and n 2. In fgure, we present two possble tensors for a 9-dmensonal vector. Suppose n n 2, n order to have at least n entres n the tensor whle mnmzng the sze of the tensor, we have (n ) n 2 < n n n 2. Wth such a requrement, there are stll many choces of n and n 2, especally when n s large. Generally all these (n, n 2 ) combnatons can be used. However, t s worth notcng that the number of parameters of a lnear functon n the tensor space s n + n 2. Therefore, one may try to mnmze n + n 2. In other words, n and n 2 should be as close as possble. The second ssue s how to sort the features n the tensor. In vector space model, we mplctly assume that the features are ndependent. A lnear functon n the vector space can be wrtten as g(x) = w T x + b. Clearly, the change of the order of the features has no mpact on the functon learnng. In tensor space model, a lnear functon can be wrtten as f(x) = u T Xv + b. Thus, the ndependency assumpton of the features no longer holds for the learnng algorthms n the tensor space model. Dfferent feature sortng wll lead to dfferent learnng result n the tensor space model. A possble approach for sortng the features s usng the w learned by a vector classfer. Each w n w, where =,, n, corresponds to a feature. Thus, the problem of sortng features n the tensor can be converted to fll these n elements w nto the n n 2 tensor. We dvde w nto three groups: postve, negatve and zero. The correspondng features n the same group tend to be correlated. Suppose each group has l k elements, where k =, 2, 3. For each group, we sort l k elements by ther absolute value and get ( w k wk l k ). We take the frst n elements to form the frst column of the tensor and the next n elements to form the second column of the tensor Note that, n ths paper our prmary nterest s focused on the second order tensors. However, the TSM presented here and the algorthms presented n the next secton can also be appled to hgher order tensors. 3

5 and so on. For each group, we wll fll l k /n columns. The remanders n each group wll be put together to fll the remanng entres n the tensor. In ths paper, we take n n 2 n whch we called square tensors. The features are flled nto the tensor entres by usng the way we descrbed above. The better ways of convertng a data vector to a data tensor wth theoretcal guarantee wll be left for our future work. 3 Classfcaton wth Tensor Representaton Gven a set of tranng samples {X, y }, =,, m, where X s the data pont n order-2 tensor space, X R n R n 2 and y {, } s the label assocated wth X. Let (u,,u n ) be a set of orthonormal bass functons of R n. Let (v,,v n2 ) be a set of orthonormal bass functons of R n 2. Thus, X can be unquely wrtten as: X = j (u T Xv j )u v T j A lnear classfer n the tensor space can be wrtten as f(x) = u T Xv + b, where u R n and v R n 2 are two vectors. The problem of lnear classfcaton n tensor space model s to fnd the u and v based on a specfc objectve functon. In the followng two subsectons, we ntroduce two novel classfers n tensor space model based on dfferent objectve functons, whch are Tensor Least Square classfer and Support Tensor Machnes. One s the tensor generalzaton of least square classfer and the other s the tensor generalzaton of support vector machnes. 3. Tensor Least Square Analyss The least square classfer mght be one of the most well known classfers (Duda et al., 2000; Haste et al., 200). In ths subsecton, we extend the least square dea to tensor space model and develop a Tensor Least Square classfer (TLS). The objectve functon n tensor least square analyss s as follows 2 : mn u,v ( u T ) 2 X v y () 2 Note that, we need to add a constant column (or row) to each data tensor to ft the ntercept. 4

6 By smple algebra, we see that: ( u T ) 2 X v y Smlarly, we also have: = ( u T X vv T X T u 2y u T X v + y 2 ) ( m ( = u T X vv T X )u T u T 2 y X )v + y 2 (2) ( u T ) 2 X v y = ( v T X T uu T X v 2y v T X T u + y 2 ) ( m ) ) = v T X T uu T X v v (2 T y X T u + y 2 (3) Requrng the dervatve of Eqn (2) wth respect to u to be zero, we get: ( ) ( X vv T X T u = 2 y X )v Thus, we get: ( ) ( u = X vv T X T 2 y X )v (4) Smlarly, we can get: ( ) ( v = X T uu T X 2 ) y X T u (5) Notce that u and v are dependent on each other, and can not be solved ndependently. We can use an teratve approach to compute the optmal u and v,.e., we frst fx u, and compute v by Equaton (5); Then we fx v, and compute u by Equaton (4). The convergence proof of such an teratve approach s gven n Secton Support Tensor Machnes Support Vector Machnes are a famly of pattern classfcaton algorthms developed by Vapnk (995) and collaborators. SVM tranng algorthms are based on the dea of structural rsk mnmzaton rather than emprcal rsk mnmzaton, and gve rse to new ways of tranng polynomal, 5

7 neural network, and radal bass functon (RBF) classfers. SVM has proven to be effectve for many classfcaton tasks (Joachms, 998; Ronfard et al., 2002). Specfcally, SVM try to fnd a decson surface that maxmzes the margn between the data ponts n a tranng set. The objectve functon of lnear SVM can be stated as: mn w,b,ξ 2 wt w + C ξ subject to y (w T x + b) ξ, (6) ξ 0, =,, m. Our Support Tensor Machnes (STM) s fundamentally based on the same dea. As we dscussed before, A lnear classfer n the tensor space can be naturally represented as follows: f(x) = u T Xv + b, u R n,v R n 2 (7) Equaton (7) can be rewrtten through matrx nner product as follows: f(x) = < X,uv T > +b, u R n,v R n 2 (8) Thus, the large margn optmzaton problem n the tensor space s reduced to the followng: mn u,v,b,ξ 2 uvt 2 + C ξ subject to y (u T X v + b) ξ, (9) ξ 0, =,, m. We wll now swtch to a Lagrangan formulaton of the problem. We ntroduce postve Lagrange multplers α, µ, =,, m, one for each of the nequalty constrants (9). Ths gves Lagrangan: L P = 2 uvt 2 + C ξ ( α y u T X v + b ) + α α ξ µ ξ Note that 2 uvt 2 = 2 trace( uv T vu T) = ( v T v ) trace ( uu T) 2 = ( v T v ) ( u T u ) 2 6

8 Thus, we have: L P = ( v T v ) ( u T u ) + C 2 ( α y u T X v + b ) ξ + α α ξ µ ξ Requrng that the gradent of L P wth respect to u, v, b and ξ vansh gve the condtons: u = α y X v v T (0) v v = α y u T X u T () u α y = 0 (2) C α µ = 0, =,, m (3) From Equatons (0) and (), we see that u and v are dependent on each other, and can not be solved ndependently. In the followng, we descrbe a smple yet effectve computatonal method to solve ths optmzaton problem. We frst fx u. Let β = u 2 and x = X T u. Thus, the optmzaton problem (9) can be rewrtten as follows: mn v,b,ξ 2 β v 2 + C ξ subject to y (v T x + b) ξ, (4) ξ 0, =,, m. It s clear that the new optmzaton problem (4) s dentcal to the standard SVM optmzaton problem. Thus, we can use the same computatonal methods of SVM to solve (4), such as (Graf et al., 2004; Platt, 998; Platt, 999). Once v s obtaned, let β 2 = v 2 and x = Xv. Thus, u can be obtaned by solvng the followng optmzaton problem: mn u,b,ξ 2 β 2 u 2 + C ξ subject to y (u T x + b) ξ, (5) ξ 0, =,, m. Agan, we can use the standard SVM computatonal methods to solve ths optmzaton problem. Thus, v and u can be obtaned by teratvely solvng the optmzaton problems (4) and (5). In our experments, u s ntally set to the vector of all ones. 7

9 3.3 Convergence Proof In ths secton, we provde a convergence proof of the teratve computatonal method n STM and TLS algorthms. We have the followng theorem: Theorem The teratve procedure to solve the optmzaton problems (4) and (5) wll monotoncally decreases the objectve functon value n (9), and hence the STM algorthm converges. Proof Defne: f(u,v) = 2 uvt 2 + C Let u 0 be the ntal value. Fxng u 0, we get v 0 by solvng the optmzaton problem (4). Lkewse, fxng v 0, we get u by solvng the optmzaton problem (5). Notce that the optmzaton problem of SVM s convex, so the soluton of SVM s globally optmum (Burges, 998; Fletcher, 987). Specfcally, the solutons of equatons (4) and (5) are globally optmum. Thus, we have: f(u 0,v 0 ) f(u,v 0 ) ξ Fnally, we get: f(u 0,v 0 ) f(u,v 0 ) f(u,v ) f(u 2,v ) Snce f s bounded from below by 0, t converges. Smlarly, for Tensor Least Square algorthm, snce the soluton for least square s globally optmum, teratve procedure (4) and (5) wll monotoncally decreases the objectve functon value n (), and hence the TLS algorthm converges. 3.4 From Matrx to Hgh Order Tensor The TLS and STM algorthms descrbed above take order-2 tensors,.e., matrces, as nput data. However, these algorthms can also be extended to hgh order tensors. In ths secton, we brefly descrbe the extenson of these algorthms to hgh order (> 2) tensors. we take STM as an example. Let (T, y ), =,, m denote the tranng samples, where T R n R n k. The decson functon of STM s: f(t) = T(a,a 2,,a k ) + b a R n,a 2 R n 2,,a k R n k where T(a,a 2,,a k ) = T,, k a a k k n. k n k 8

10 As before, a,,a k can also be computed teratvely. We frst ntroduce the l-mode product of a tensor T and a vector a, whch we denote as T l a. The result of l-mode product of a tensor T R n R n k and a vector a R n l, l k wll be a new tensor B R n R n l R n l+ R n k, where n l B,, l, l+,, k = T,, l, l, l+,, k a l Thus, the decson functon n hgher order tensor space can also be wrtten as: f(t) = T a 2 a 2 k a k + b l = The optmzaton problem of STM n hgh order tensors s: mn a,,a k,b,ξ 2 a a k 2 + C ξ (6) subject to y (T (a,a 2,,a k ) + b) ξ, ξ 0, =,, m. Here a a k denotes the tensor norm of a a k (Lee, 2002). Frst, to compute a, we fx a 2,,a k. Let β 2 = a 2 2,, β k = a k 2. We then defne t = T 2 a 2 k a k. Thus, the optmzaton problem (6) can be reduced as follows: mn a,b,ξ 2 β 2 β k a 2 + C subject to y (a T t + b) ξ, (7) ξ 0, =,, m. Agan, we can use the standard SVM computatonal methods to solve ths optmzaton problem. Once a s computed, we can fx a,a 3,,a k to compute a 2. So on, all the a can be computed n such teratve manner. ξ 4 Experments To evaluate the performance of tensor based classfers, we performed experments on sx datasets from UCI Repostory and report results for all of them. These sx datasets nclude BREAST- CANCER, DIABETES, IONOSPHERE, MUSHROOMS, SONAR, as well as the ADULT data n a representaton produced by Platt (999). All of these datasets are bnary categores and the features were scaled to [, ]. The preprocessed datasets can also be downloaded from LIBSVM dataset webpage

11 Table : Classfcaton accuraces on varous data sets. n s the number of features and m s the total number of examples Data set n m Tran TLS LS TLS vs LS STM SVM STM vs SVM Adult % Breast-cancer % Dabetes % Ionosphere % > Mushrooms % Sonar % or means P-value 0.0 n the pared T-test on the 50 random splts > or < means 0.0 < P-value 0.05 means P-value > 0.05 All the datasets were randomly splt nto tranng and testng sets. Notce that the tranng sets are pretty small snce we are partcularly nterested n the performance on the small tranng sze cases. We averaged the results over 50 random splts and reported the average performance. 4. Expermental Results on TLS and LS We used the regress functon n statstcs toolbox of Matlab 7.04 to solve the least square problems n both Tensor Least Square classfer (TLS) and Least Square classfer (LS). Table summarzed the results. Besdes the average accuracy, a pared T-test on the 50 random splts s also reported, whch ndcates whether the dfference between two systems s sgnfcant. We can see that n 3 out of 6 datasets, TLS s better than LS. In the remanng 3 datasets, both algorthms are comparable. To get a more detaled pcture of the performance of TLS wth respect to the tranng set sze. We tested TLS and LS over varous tranng szes (from % to 0%) on BREAST-CANCER dataset. The results are shown n Fgure 2. As can be seen, when the tranng set s small (%, 2% and 3%), TLS outperforms LS. As the number of tranng samples ncreases, TLS tends to get same results wth LS. 4.2 Expermental Results on STM and SVM In ths experment, we compared the classfcaton performance of Support Tensor Machnes (STM) and tradtonal Support Vector Machnes (SVM). We used the LIBSVM system (Chang & Ln, 200) and tested t wth the lnear model. For both STM and SVM, there s a parameter C need to be set. We use cross-valdaton on the 0

12 96 94 Accuracy (%) Tensor Least Square Least Square Tranng sample rato (%) Fgure 2: Classfcaton accuracy wth respect to the tranng sample sze of TLS and LS on BREAST-CANCER dataset. As can be seen, when the tranng set s small (%, 2% and 3%), TLS outperforms LS. As the number of tranng samples ncreases, TLS tends to get same results wth LS. tranng set to fnd the best parameter C. Table summarzed the results. We can see that n 4 out of 6 datasets, STM s better than SVM. In the remanng 2 datasets, both algorthms are comparable. We also tested STM and SVM over varous tranng sze (from % to 0%) on BREAST-CANCER dataset. The results are shown n Fgure 3. As can be seen, when the tranng set s small (%, 2% and 3%), STM outperforms SVM. As the number of tranng samples ncreases, STM tends to get same results wth SVM. In all the cases, large margn classfers (STM and SVM) are better than least square classfers (TLS and LS). Both of these two experments suggest that classfers based on tensor representaton are partcularly sutable for small sample problems. Ths mght be due to the fact that the number of parameters need to be estmated n tensor classfers s n + n 2 whch can be much smaller than n n 2 n vector classfers. 5 The General Algorthm for Learnng Functons n Tensor Space In the above sectons, we have developed an teratve algorthm for solvng the optmzaton problems of STM and TLS. It can be notced that these two algorthms share the smlar teratve procedure. Actually, such teratve algorthm can be generalze to solve a broad famly of tensor verson of tradtonal vector based learnng algorthms. Suppose the lnear classfer n vector space s g(x) = w T x + b and ts objectve functon s mnv (w,x, y, =,, m) (8)

13 Accuracy (%) STM SVM Tranng sample rato (%) Fgure 3: Classfcaton accuracy wth respect to the tranng sample sze of STM and SVM on BREAST-CANCER dataset. As can be seen, when the tranng set s small (%, 2% and 3%), STM outperforms SVM. As the number of tranng samples ncreases, STM tends to get same results wth SVM. The correspondng lnear classfer n tensor space s f(x) = u T Xv + b wth objectve functon mnt(u,v,x, y, =,, m) (9) The algorthmc procedure for solvng the optmzaton problem (9) s formally stated below:. Intalzaton: Let u = (,, ) T. 2. Computng v: Let x = X T u. The tensor classfer can be rewrtten as f(x) = u T Xv + b = x T v + b = g (x) Thus, the optmzaton problem of (9) can be rewrtten as the optmzaton problem n vector space: mn V (v,x, y, =,, m) (20) Note: Any computatonal method for solvng (8) can also be used here. 3. Computng u: Once v s obtaned, let x = X v. Smlarly, The tensor classfer can be rewrtten as f(x) = u T Xv + b = u T x + b = g ( x) Thus, u can be computed by solvng the same vector space optmzaton problem: mn V (u, x, y, =,, m) (2) Note: As above, Any computatonal method for solvng (8) can also be used to solve (2). 2

14 4. Iteratvely computng u and v: By step 2 and 3, we can teratvely compute u and v untl they tend to converge. Note: As long as the soluton of optmzaton problem (8) s globally optmum, the teratve procedure descrbed here wll converge. The convergence proof wll be smlar to the proof we gven n Secton Conclusons In ths paper we have ntroduced a tensor framework for data representaton and classfcaton. In partcular, we have proposed two new classfcaton algorthms called Support Tensor Machnes (STM) and Tensor Least Square (TLS) for learnng a lnear classfer n tensor space. Our expermental results on 6 databases from UCI Repostory demonstrate that tensor based classfers are especally sutable for small sample cases. Ths s due to the fact that the number of parameters estmated by a tensor classfer s much less than that estmated by a tradtonal vector classfer. There are several nterestng problems that we are gong to explore n the future work:. In ths paper, we emprcally construct the tensor. The better ways of convertng a data vector to a data tensor wth theoretcal guarantee need to be studed. 2. Both STM and TLS are lnear methods. Thus, they fal to dscover the nonlnear structure of the data space. It remans unclear how to generalzed our algorthms to nonlnear case. A possble way of nonlnear generalzaton s to use kernel technques. 3. In ths paper, we use an teratve computatonal method for solvng the optmzaton problems of STM and TLS. We expect that there exst more effcent computatonal methods. References Burges, C. J. C. (998). A tutoral on support vector machnes for pattern recognton. Data Mnng and Knowledge Dscovery, 2, Ca, D., He, X., & Han, J. (2005). Subspace learnng based on tensor analyss (Techncal Report). Computer Scence Department, UIUC, UIUCDCS-R Chang, C.-C., & Ln, C.-J. (200). LIBSVM: a lbrary for support vector machnes. Software avalable at Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern classfcaton. Hoboken, NJ: Wley- Interscence. 2nd edton. Fletcher, R. (987). Practcal methods of optmzaton. John Wley and Sons. 2nd edton edton. 3

15 Graf, H. P., Cosatto, E., Bottou, L., Dourdanovc, I., & Vapnk, V. (2004). Parallel support vector machnes: The cascade SVM. Advances n Neural Informaton Processng Systems 7. Haste, T., Tbshran, R., & Fredman, J. H. (200). The elements of statstcal learnng. Sprnger- Verlag. He, X., Ca, D., & Nyog, P. (2005). Tensor subspace analyss. Advances n Neural Informaton Processng Systems 8. Joachms, T. (998). Text categorzaton wth support vector machnes: Learnng wth many relevant features. ECML 98. Lee, J. M. (2002). Introducton to smooth manfolds. Sprnger-Verlag New York. Platt, J. C. (998). Usng sparseness and analytc QP to speed tranng of support vector machnes. Advances n Neural Informaton Processng Systems. Platt, J. C. (999). Fast tranng of support vector machnes usng sequental mnmal optmzaton, Cambrdge, MA, USA: MIT Press. Ronfard, R., Schmd, C., & Trggs, B. (2002). Learnng to parse pctures of people. ECCV 02. Shashua, A., & Hazan, T. (2005). Non-negatve tensor factorzaton wth applcatons to statstcs and computer vson. Proc Int. Conf. Machne Learnng (ICML 05). Vapnk, V. N. (982). Estmaton of dependences based on emprcal data. Sprnger-Verlag. Vapnk, V. N. (995). The nature of statstcal learnng theory. Sprnger-Verlag. Vaslescu, M. A. O., & Terzopoulos, D. (2003). Multlnear subspace analyss for mage ensembles. IEEE Conference on Computer Vson and Pattern Recognton. Ye, J., Janardan, R., & L, Q. (2004). Two-dmensonal lnear dscrmnant analyss. Advances n Neural Informaton Processng Systems 7. 4