An efficient dictionary learning algorithm for sparse representation

Transcription

1 An efficient dictionary learning algorithm for pare repreentation Leyuan ang 1 and Shutao Li 1 1.College of Electrical and Information Engineering, Hunan Univerity, Changha, 41008, China fangleyuan@gmail.com, hutao_li@yahoo.com.cn Abtract Spare and redundant repreentation of data aume an ability to decribe ignal a linear combination of a few atom from a dictionary. If the model of the ignal i unknown, the dictionary can be learned from a et of training ignal. Like the K-SVD, many of the practical dictionary learning algorithm are compoed of two main part pare-coding and dictionary-update. hi paper firt propoe a Stagewie leat angle regreion (St-LARS) method for performing the pare-coding operation. he St-LARS applie a hard-threholding trategy into the original leat angle regreion (LARS) algorithm, which enable it to elect many atom at each iteration and thu reult in fat olution while till provide good reult. hen, a dictionary update method named approximated ingular value decompoition (ASVD) i ued on the dictionary update tage. It i a quick approximation of the exact SVD computation and can reduce the complexity of it. Experiment on both ynthetic data and 3-D image denoiing demontrate the advantage of the propoed algorithm over other dictionary learning method not only in term of better trained dictionary but alo in term of computation time. Key Word Dictionary learning, pare repreentation, leat angle regreion, hard threholding 1 INRODUCION Spare and redundant repreentation of a ignal y N K over a dictionary D (with K column denoted a atom) refer that one can find a linear combination of a few atom from D that i cloe to the ignal y. reviou work [1-4] have hown that modeling a ignal with uch a pare decompoition i very effective in many ignal and image proceing application. A fundamental conideration in employing the above pare model i the choice of the dictionary D. he majority of work on thi topic can be parted into two main categorie analytical-baed and learning-baed. he analytical approache contruct the dictionary, baing on variou type of wavelet [5], and it variant [6-7]. he learning approache recommend uing machine learning technique to infer the dictionary from a et of training example [8-10]. Advantage of thee approache are the finer dictionarie they produce compared to analytical approache, and the ignificantly better performance in application. However, the complexity contraint in thee algorithm often limit the ize of the dictionarie that can be trained, and the dimenion of ignal that can be proceed. So, reducing the complexity of thee learning algorithm i a tiff challenge for u. A we know, the main proce of many learning algorithm [8-10] can be divided into two tage pare coding and dictionary update. Spare coding i to find the paret olution of the training ignal, which dominate the complexity of the dictionary learning. Commonly ued trategie conducting the pare coding are typically baed on greedy puruit and convex relaxation. Greedy puruit employ ome greedy algorithm (e.g. the matching puruit (M) [11] and orthogonal matching puruit (OM) [1]) to get an approximate olution of the pare repreentation. Since the M and OM are that only one atom i elected at each N iteration, Donoho et al. have recently propoed the tagewie orthogonal matching puruit (StOM) [13] by applying a threhold trategy named fale dicover rate (DR) control into the atom election proce. So, the StOM can elect more than one atom for each iteration and thu ha low computational complexity than the M and OM, epecially for large-cale problem. However, the high non-convexity of thee greedy algorithm uually can not find the optimal olution and thi will enable the dictionary learning algorithm to get caught in local minimal or even addle point [, 14]. he convex relaxation approache (e.g. the bai puruit (B) [4] and the LASSO [15]) ue the 1 -norm a a parene meaure and have been proven to obtain the paret olution. But they run much lower than the above greedy algorithm, and thu will create heavy computational burden to the whole learning algorithm. A fat algorithm called leat angle regreion (LARS) introduced in [16] can make a mall modification to olve the LASSO problem and the computational complexity of it i very cloe to that of the greedy method. But, the LARS alo jut allow one atom to be choen in the atom election proce, and therefore there i a trong incentive to elect more atom for each iteration in order to accelerate the convergence, a in [13]. ollowing thi idea, we propoe a method named St-LARS by applying a hard-threholding trategy into the LARS. hi can enable it to elect more than one atom at each iteration while till keep good performance a the LARS. It i worthwhile to note that for the hard-threholding i le time conuming than the threholding controlled by the DR, the St-LARS generally can run much fater than the StOM, a will be demontrated in the experimental part. A a reult, the St-LARS greatly accelerate the tage of the pare coding and give a good pare olution for the dictionary update tage. Updating the dictionary, when the pare repreentation i found, i comparatively eaier. he SVD decompoition ued /10/$ IEEE

2 in the K-SVD ha been hown to be a wier method than many other approache [8-10, 14, 17]. In thi paper, we replace the exact SVD computation with a impler approximated one (ASVD) [18] and thu obtain further acceleration for the dictionary learning algorithm. he ret of thi paper i organized a follow. In Section, we introduce the propoed dictionary learning algorithm. Our experimental reult on both ynthetic data and 3-D image denoiing are preented in ection 3. Section 4 conclude thi paper and ugget future work. DICIONARY LEARNING Dictionary learning i the tak of learning or training a dictionary uch that it i well adapted to it training data. Uually the objective i to give pare repreentation of the training et, making the total error a mall a poible, i.e. minimizing the um of quared error. Let the training data contitute the column in a matrix Y and the pare coefficient vector are the column in matrix X. he objective function of the dictionary learning can be tated formally a a minimization problem min Y DX ubject to X, (1) D,X where the function denote the -norm. A practical optimization trategy can be found by plitting the problem into two part which are alternately olved within an iterative loop [, 8-10]. he two part are 1) Spare coding keeping D fixed, find X; ) Dictionary update keeping X fixed, find D. Since the propoed dictionary learning algorithm i alo baed on the two part, the following ubection will give the detailed decription of our improvement on thee part..1 Spare coding Conider olving the optimization problem (1) with -norm penalty over the pare matrix X while keeping the dictionary D fixed. hi problem can be olved by optimizing over each vector x of the pare matrix individually min y x ubjectto x t, x D () where the y repreent one ignal of the training matrix Y. Notice that the above optimization tak can be eaily changed to be 1 min y D x + λ x. (3) x or a proper choice of λ, the two problem are equivalent. If the p in the function i et to 0, the above problem i known to be N-hard in general [19] and ome greedy algorithm [11-13] can not get the optimal pare olution of it. he B [4] and LASSO [15] replace the combinatorial 0 -norm with the 1 -norm. A reported in [4], they achieve parer olution but are lower than the greedy approache for mot experiment. In [16], an algorithm called LARS i introduced to olve the LASSO problem with minor modification and it computational complexity i linear with the ize of input ignal a the greedy algorithm. However, the LARS compute the olution by only adding one atom to it active et at each iteration. herefore, we adapt the idea from [13], and propoe the St-LARS algorithm to olve the problem (3). y ˆx c I dx xi ig. 1. he cheme of the St-LARS algorithm. he cheme of St-LARS i preented in ig. 1, which conit of the following even tep. Step 1 Set initial olution x 0 = 0, initial reidual r=y, 0 initial etimate I 0 = φ, initial threhold λ 0 = D y and counter =1. Step Apply the D to the current reidual r -1, getting a vector of reidual correlation c c ( k) = dk r -1, k = 1,..., K, (4) for each correponding atom ( K i the number of atom in D, d k denote the k-th atom in D, and c ( k ) tand for the k-th element in c ). he reidual correlation c are uppoed to contain a mall fraction of ignificant non-zero. Step 3 erform hard threholding with threhold λ to find the ignificant non-zero in c I = { k c( k) > λ}. (5) he λ i calculated by λ = μλ 1, where the μ i a threhold decreae tep. r

3 Step 4 Compute the update direction dx by projecting the reidual r onto ubpace panned by the column of D belonging to I dx (I ) = ( D D ) D r. (6) 1 I I I Step 5 Update the olution x by augmenting x 1 in the direction dx with a tep ize ε (typically choen equally to the threhold decreae tep) x = x 1 + εdx. (7) Step 6 Contruct a new D x uing the x. hen, the current reidual can be calculated by r = y D x. (8) Step 7 Check the topping condition. he procedure top with the output of final olution ˆx when the -norm of current reidual r reache an error goal, or when the threhold λ i le than a pre-choen value. If it i not atified, we et = + 1 and go to the tep. hi algorithm i a fat tagewie approximation to LARS and provide a good approximation to the olution of (3). It i till note that the computation in the tep 4 will uually not be carried out explicitly for it high computation cot. So, we employ a progreive Choleky update proce to reduce the work involved in the matrix inverion, and thu further accelerate the St-LARS. or more detail about the Choleky proce, the reader can be found in [18, 0].. Dictionary update Given fixed pare matrix X, the dictionary D can be updated by olving the following problem min Y DX. (9) D In general, thi problem can be olved uing gradient decent with iterative projection [17] or other leat quare method [8]. However, it can be much more efficiently olved uing the SVD decompoition. or a given atom k, the quadratic term in (9) i rewritten a Y d g d g = E d g, (10) j k j j k k k k k where the g j are the row of pare matrix X, the d k denote the atom of the dictionary D and the E k tand for the reidual matrix. After the SVD decompoition of the matrix E k, both the atom d k and g k can be updated. o avoid the introduction of new non-zero in X, the update ue only the ignal vector whoe current repreentation ue the atom d k. Since the exact SVD computation i time conuming, adopting a much quicker approach to get a olution of (10) i a more exciting option. herefore, our implementation here ue the ASVD, which employ a ingle iteration of alternate-optimization over the atom d k and the pare matrix row g k, which i given by d k= Ekg k/ Ekg k, (11) g k = ( Ek) d k. hi operation ha been proven to both reduce the complexity and give very cloe reult to the full decompoition of E k [18]. 3 EXERIMENS We evaluate the propoed method with ynthetic and real data. Uing ynthetic data with random dictionarie help u to examine the ability of the propoed method to recover dictionarie exactly (to within an acceptable quared error). hi ynthetic experiment i very imilar to that in [10], except for the dimenion of the data and the dictionary we generate are lightly higher. o tet the performance on real data, we chooe the 3-D C volume, which need a comparatively larger dictionary. So thee experiment are to demontrate that our method i more applicable to large-cale dictionary learning problem than other approache. 3.1 Synthetic experiment In thee imulation, a dictionary D i firt generated by normalizing a matrix with i.i.d. uniform random entrie. hen, we produce 1500 data ignal of dimenion 50, each created by a linear combination of ten different dictionary atom, with independent location and uniformly ditributed i.i.d. coefficient in random. After that, white Gauian noie with varying ignal-to-noie ratio (SNR) i added to the reulting data ignal. he training dictionary i initialized with data ignal and the number of the training iteration in thi etting i et to 100. If the quared error between the learnt and the true dictionary element i below 0., it i claified a correctly recovery. o allow a fair comparion, the imulation are repeated five time and the average value are calculated. In ig., the dictionary recovery ucce rate of our method are compared with that of the K-SVD [10], StOM-ASVD, and LARS-ASVD. he StOM-ASVD and LARS-ASVD are the method which firt apply the StOM and LARS to perform the pare coding repectively, and then employ the ASVD to conduct the dictionary update. he topping criteria for the LARS in LARS-ASVD and St-LARS in our method are λ = 0.3 in (3). or the OM in the K-SVD and StOM in the StOM-ASVD, they are topped when the fixed number of ten coefficient are found. he threhold decreae tep μ and the tep ize ε in (7) are both choen to A can be een from the ig., the LARS-ASVD and our method recover nearly the ame number of atom and perform better than the other two method for all the teted cae. In ig. 3, we alo compare the computation time of the four algorithm for the above imulation. he imulation are done in the environment of an AMD Athlon CU.81 GHz with a.00 GB RAM C, operating under Matlab A we can ee, the peed advantage of our method i obviou and it i about two time fater than the other approache. It i till notice that the StOM-ASVD can not run very fat in thee

4 imulation and even lower than the K-SVD, though the StOM can alo elect many atom at each iteration on the pare coding tage. he main reaon for thi i that the DR In thi ubection, our method i teted on two 3-D C volume Viible Male-Head and Viible emale-ankle. Like the K-SVD denoiing algorithm in [1], we firt train an over-complete dictionary uing block from the noiy image, and then denoie the image uing thi dictionary. he peak-ignal-to-noie ratio (NSR) i ued a objective denoiing meaure. he ize of the training block i and the ize of the dictionary i he number of the training block i choen to and the number of the training iteration i et to 5. We hould mention that ome newly denoiing technique, e.g. multi-cale framework [] and non-local imultaneou pare-coding [1], can be ued here to further improve our performance. However, our work here only focue on the original denoiing formulation for implicity, and thu do not conduct a thorough comparion with ome of the tate-of-the-art work. ig.. Comparion of the dictionary recovery ucce rate uing different dictionary learning method (a) (b) ig. 3. Comparion of the computation cot of the dictionary learning method control which calculate the threhold for each iteration of the StOM conume large computation, and the computational complexity of it i far larger than the hard-threholding ued in our method. We gue that if the StOM applie in larger dictionary or higher dimenional ignal, the StOM-ASVD will become fater than the K-SVD D image denoiing experiment able 1 C denoiing reult uing the K-SVD, and our method. Value repreent SNR (db), and are averaged over 5 execution. he bet reult in thi table are labeled in bold. Noie level Vi.. Ankle Vi. M. Head σ K-SVD Our method K-SVD Our method (c) (d) ig. 4. Denoiing reult for Viible emale-ankle. Image are mainly provided for qualitative evaluation. (a) Original image ( σ = 50); (b) Noiy image; (c) Denoied uing the K-SVD; (d) Denoied uing our method. he denoiing reult are ummarized in able 1. We can ee from the table that the SNR gain of our method over the K-SVD i conitent, though not very large. Some actual denoiing reult are hown in ig. 4. It can be een that the viual quality of the K-SVD i very cloe to that of our method. Although the difference in viual quality are typically mall, the main appeal of our method i it ubtantially better complexity, a depicted in able. A can be een, the complexity advantage of our method tranlate to a 4 reduction in denoiing time compared to the K-SVD. So, it can be eaily concluded that our method i more uitable for large-cale dictionary learning problem.

5 able Running time (in econd) of the K-SVD, and our method for the reult in able 1. Method/σ Vi.. Ankle Vi. M. Head K-SVD Our method CONCLUSION In thi paper, we combine the St-LARS and ASVD into an efficient dictionary learning algorithm. he St-LARS give a better olution for the pare coding and greatly reduce the complexity of it by imply exploiting a hard-threholding trategy. he ASVD i a quick approximation way for updating dictionary, and thu further accelerate the whole dictionary learning algorithm. he experimental reult demontrate the uperior performance of the propoed method in term of training better dictionary and reducing the computational complexity. So thi learning algorithm i very uitable for ome large-cale ignal proceing application (like the 3-D C denoiing above). However, the full potential of thi algorithm i needed to be further explored, and other ignal proceing tak (involving color image denoiing, deblurring, and inpainting) are expected to benefit from it. ACKNOWLEDGEMENS hi paper i upported by the National Natural Science oundation of China (No ), the h.d. rogram oundation of Minitry of Education of China (No ), the Key roject of Chinee Minitry of Education (009-10), and the Open roject rogram of National Laboratory of attern Recognition, China. REERENCES [1] M. Elad and M. Aharon. Image denoiing via pare and redundant repreentation over learned dictionarie. IEEE ran. Image proce., 15(1) , 006. [] J. Mairal, G. Sapiro, and M. Elad. Learning multicale pare repreentation for image and video retoration. SIAM J. Multicale Model. Simul., 7(1) 14-41, 008. [3] J. Mairal,. Bach, J. once, G. Sapiro, and A. Zierman. Non-local pare model for image retoration. roc. ICCV 009, okyo, Japan, 009, pp [4] S. S. Chen, D. L. Donoho, and M. A. Saunder. Atomic decompoition by bai puruit. SIAM Rev., 43(1) , 001. [5] S. Mallat. A wavelet tour of ignal proceing, third edition. Academic re, 009. [6] M. N. Do and M. Vetterli. he contourlet tranform an efficient directional multireolution image repreentation. IEEE ran. Image roce., 14(1) , 005. [7] D. L. Donoho. Wedgelet nearly minimax etimation of edge. he Annal of Statitic, 7(3) , [8] K. Engan, S. O. Aae, and J. Hakon Huoy. Method of optimal direction for frame deign. roc. ICASS 1999, Wahington, USA, 1999, Vol.5, pp [9] S. Leage, R. Gribonval,. Bimbot, and L. Benaroya. Learning union of orthonormal bae with threhold threholded ingular value decompoition. roc. ICASS 005, hiladelphia, USA, 005, Vol.5, pp [10] M. Aharon, M. Elad, and A. M. Brucktein. he K-SVD An algorithm for deigning of overcomplete dictionarie for pare repreentation. IEEE ran. Signal roce., 54(11) , 006. [11] S. Mallat and Z. Zhang. Matching puruit with time-frequency dictionarie. IEEE ran. Signal roce., 41(1) , [1] Y. C. ati, R. Rezaiifar, and. S. Krihnapraad. Orthogonal matching puruit Recurive function approximation with application to wavelet decompoition. roc. AilomarSS, California, USA, 1993, Vol.1, pp [13] D. L. Donoho, Y. aig, I. Drori, and J.-L. Starck. Spare olution of underdetermined linear equation by tagewie orthogonal matching puruit. 006 [Online]. Available http//tat.tanford.edu/~idrori/stom.pdf, reprint. [14] R. Rubintein, A. M. Brucktein, and M. Elad. Dictionarie for pare repreentation modeling. roceeding of the IEEE, 98(6) , 010. [15] R. ibhirani. Regreion hrinkage and election via the lao. J. Royal. Statit. Soc B., 58(1) 67-88, [16] B. Efron,. Hatie, I. Johnton, and R. ibhirani. Leat angle regreion. Ann. Statit., 3() , 004. [17] Y. Cenor and S. A. Zenio. arallel optimization theory, Algorithm and Application, Oxford Univerity re, [18] R. Rubintein, M. Zibulevky, and M. Elad. Efficient implementation of the K-SVD algorithm uing batch orthogonal matching puruit. echnical Report-CS echnion, 008. [19] G. Davi, S. Mallat, and M. Avellaneda. Adaptive greedy approximation. Contructive approximation, 13(1) 57-58, [0]. Blumenath and M. Davie. Gradient puruit. IEEE ran Signal roce., 56(6) , 008. [1] K. Dabov, A. oi, V. Katkovnik, and K. Egiazarian. Image denoiing by pare 3-D tranform-domain collaborative filtering. IEEE ran. Image roce., 16(8) , 007.