Laplacian Regularized Subspace Learning for Interactive Image Reranking

Transcription

1 Lalacan Regularzed Subsace Learnng for Interactve Image Rerankng Lnng Zhang School of Electrcal and Electronc Engneerng Nanyang echnologcal Unversty Sngaore Lo Wang School of Electrcal and Electronc Engneerng Nanyang echnologcal Unversty Sngaore Wes Ln School of Comuter Engneerng Nanyang echnologcal Unversty Sngaore Abstract Content-based mage retreval (CBIR) has attracted substantal attenton durng the ast few years for ts otental alcaton. o brdge the ga between the low level vsual features and the hgh level semantc concets, varous relevance feedback (RF) or nteractve rerankng (IR) schemes have been desgned to mrove the erformance of a CBIR system. In ths aer, we roose a novel subsace learnng based IR scheme by usng a grah embeddng framework, termed Lalacan Regularzed Subsace Learnng (). he method can model both the wthn-class comactness and the between-class searaton by secally desgnng an ntrnsc grah and a enalty grah n the grah embeddng framework, resectvely. In addton, can share the oular assumton of the based dscrmnant analyss () for IR but avod the sngular roblem n. Extensve exermental results have shown that the roosed method s effectve for reducng the semantc ga and targetng the ntentons of users for an mage retreval task. Keywords-content based mage retreval; mage rerankng; subsace learnng; grah embeddng I. INRODUCION Durng the ast few years, mage retreval has attracted much attenton n the multmeda socety []. It s mractcally for users to manually gve all mages recse textual descrtons. Moreover, to clearly descrbe an mage usng lmted words s nfeasble. On the other hand, to automatcally annotate an mage record s actually far beyond the current technology. herefore, content based mage retreval (CBIR) has ganed tremendous nterests both from the academa and the ndustry. However, the ga between low-level vsual features and hgh-level semantc concets usually lead to the oor erformance of a CBIR system. In addton, dfferent users or even the same user at a dfferent tme may have dfferent erceton on the same mage []. herefore, nteracton schemes between the user and the system, e.g., relevance feedback (RF) or nteractve rerankng (IR), are actually requred to mrove the erformance of a CBIR system [2] [3]. In a CBIR system, mages are usually reresented by lowlevel vsual features n a hgh dmensonal sace. It s reasonable to assume that dfferent semantc concets lve n dfferent subsaces and each mage can lve n many dfferent subsaces. Consequently, t s necessary to reduce the dmensonalty of vsual features to acqure an effcent and effectve semantc subsace before conductng an mage retreval task. Based on a oular assumton that all ostve samles are alke whle each negatve examle s negatve n ts own way, based dscrmnant analyss () [3] was roosed for subsace learnng n mage retreval, n whch only ostve samles are requred to be clustered, and negatve feedback samles are requred to be searated from the centrod of ostve feedback samles. he kernel verson, termed kernel (.e., K for short), was also devsed to facltate nonlnear based dscrmnaton. o address the sngular roblem of the ostve wthn scatter n, D [4] was roosed by utlzng the drect method. However, ths aroach stll dscards the null sace of the negatve scatter wth resect to the ostve centrod, whch contans mortant dscrmnatve features. As a varant of margnal fsher analyss, margnal based analyss () [5] was also ntroduced to construct an RF aroach and has shown better erformance than ; however, t stll suffers from the ntrnsc sngular roblem n the orgnal. By recsely arametrzng ostve feedbacks, negatve feedbacks and unlabelled samles, Ban and ao roosed an RF method, whch can fnd the ntrnsc coordnate of mage low-level vsual features [6]. Nevertheless, t s generally beleved that more samles are actually requred to model the exquste geometry structure n a hgh dmensonal sace. Varous subsace learnng methods have been roosed to cature the ntrnsc geometry roerty of samles n a hgh dmensonal sace durng the ast few years. In [7], Yan et al. has shown that most of oular methods for subsace learnng or dmensonalty reducton methods can be mathematcally unfed nto a grah embeddng framework. For a subsace learnng roblem, the grah embeddng framework requres an ntrnsc grah, whch s used to descrbe the smlarty relaton of samle ars wthn a same class, and a enalty grah, whch characterzes the dfference relaton of samle ars between dfferent classes. In ths aer, we roosed a Lalacan Regularzed Subsace Learnng () method based on the grah embeddng framework [7] for IR n CBIR. he new algorthm can not only extract the most dscrmnatve nformaton from the nearest neghbor samles n dfferent classes but also share the oular assumton (.e., all ostve examles are alke; each negatve examle s negatve n ts own way) used n

2 [3], D [4] and [5]. In addton, by ntroducng a Lalacan regularzaton term [8], a locally smooth transformaton can also be learned, whch sgnfcantly reduce the over fttng roblem. Fnally, the new algorthm can never meet the sngular roblem encountered by and. Extensve exerments on a subset of Corel Image Database have shown the effectveness of the roosed for IR n comarng wth reresentatve methods. II. LAPLACIAN REGULARIZED SUBSPACE LEARNING FOR INERACIVE IMAGE RERANKING Let us denote the hgh-dmensonal sace as R h and the l low-dmensonal sace as R. In each round of feedback teraton, there are n feedback samles h X = { x, x2,, xn} R. For smlcty, we assume that the frst n + samles are ostve samles x ( n + ), the next n + + samles are negatve samles x ( n + n + n ). Let lx ( ) be the class label of samle x, we denote lx ( ) = for ostve samles, lx ( ) = for negatve samles. For smlcty, we restrct the embeddng transform to be lnear, whch can be defned by a transformaton matrx h l α R ( l < h ). Wth the observaton that all ostve examle are alke; each negatve samle s negatve n ts own way [3], two dfferent grahs should be formed accordngly [7]: ) an ntrnsc grah G, whch characterzes the smlarty of ostve samles; 2) a enalty grah G, whch characterzes the dfference between ostve and negatve samles. We frst construct an ntrnsc grah to characterze the smlarty relaton of ostve samles n the ostve class. Snce all ostve samles are alke, for each ostve samle x, we fnd all other ostve feedback samles n the ostve class, whch always share a common concet wth the ostve samle x, and ut an edge between x and all other ostve samles. he ntrnsc grah of the ostve class s characterzed as follows: 2 α = arg mn α x α xj * Wj j = argmn tr[ α X ( D W ) X α] () ( ) ( tr α XL X α ) = arg mn [ ] / Ν, f l( ) = and l( j) =, j Wj = (2) 0, f l( ) = and l( j) =, = j where D s a dagonal matrx whose dagonal elements are calculated by D = W ; j j Ν s the total number of ars of samles n the ostve class. Bascally, the ntrnsc grah measures the total average dstance of the ostve samle ars, and s used to characterze the wthn-class comactness for all the ostve feedback samles. And then, we construct a enalty grah to characterze the dfference relaton between ostve and negatve samles. For all feedback samles, we frst comute the ar-wse dstances between each ar of samles n dfferent classes. More strctly seakng, we exect that the total average dstance between the samle ars wth dfferent labels should be as large as ossble. For each feedback samle, we fnd ts k nearest neghbor feedbacks wth dfferent labels and corresondng ars of feedback samles wth weghts W j. he enalty grah can be formed as follows: 2 α = arg max α α x xj Wj j: j or j = arg max tr[ α X ( D W ) X α] (3) ( ) ( tr α XBX α ) = arg max [ ] / N, f l( ) = and l( j) =, jor j Wj = (4) 0, else where D s a dagonal matrx whose dagonal elements are calculated by D = W ; N denotes the total number j j of k nearest neghbor samle ars wth dfferent labels. Smlarly, the enalty grah measures the total average dstance of the N nearest neghbor samle ars n dfferent classes, and s used to characterze the between-class searaton. In the followng, we descrbe how to utlze the grah embeddng framework to develo algorthms based on the desgned ntrnsc and enalty grahs [7]. Dfferent from the orgnal trace rato formulaton of the grah embeddng framework n [7], the new algorthm otmzes the objectve functon n a trace dfference form nstead,.e., α = arg max ( tr( α XBX α) λ* tr( α XL X α) ) α (5) = arg max tr[ α X ( B λ* L ) X α] α where λ s a nonnegatve tunng arameter and s used to trade off the between-class searaton and wthn-class comactness. As gven n Equaton (5), we can notce that the objectve functon works n two ways, whch tres to maxmze tr( α XBX α ) and at the same tme mnmze tr( α XL X α ). By formulatng the objectve functon as a trace dfference form, we can regard t as the total average margn between the ostve and negatve class. herefore, Equaton (5) can be used as a crteron to dscrmnate the dfferent classes. o avod trval solutons, n ths aer, we mose a constrant,.e., α α = I, on Equaton (5). And thus, ths roblem can be solved by conductng the standard Egenvalue decomoston and the mang matrx α s formed by the l egenvectors assocated wth the frst l largest egenvalues. Moreover, t s reasonable to exect that ntegratng the essental manfold structure [8] of the ostve samles wll 2

3 further mrove the erformance of the IR algorthm for an mage retreval task. As a consequence, we can further 3 Fg. the framework of the CBIR system usng the Lalacan Regularzed Subsace Learnng for IR desgn another ntrnsc grah to characterze the local consstence roerty of ostve samles, whch can be used to regularze the searablty of dfferent classes. he regularzaton term mlements the local consstency rncle by reservng the local smlarty among the ostve neghborhood samles. For each ostve samle x, we fnd ts k2 nearest neghborhood ostve samles, whch can be s reresented as a samle set N, and ut an edge between x and ts neghborhood ostve samles. hen the local consstency for the ostve samles can be characterzed as follows: W L j α = α x α x W = argmn [ ( ) ] = arg mn [ ] 2 u arg mn j * j s s j: j or j L L ( tr α X D W X α ) ( tr α XL2 X α ) x x f l = l j = or j = 0, else 2 2 u u l nex( / δ ), ( ) ( ) 0, N j j 2 2 where ωj = ex( x xj / δ ) s the heat kernel accordng to Lalacan Egenmas [8], whch reflects the + + affnty of the samle ars; W = { w } n n j R s a symmetrc matrx, whch reflects the local geometry of the ostve samles n a hgh dmensonal sace; and D w s a dagonal + n j= matrx and ts -th entry s w j ; S () s the set of ndces of the neghborng samles n the ostve class for the l ostve samle x ; N s the total number of k 2 nearest ostve samles for all ostve samles. Accordng to [8] [9], a defnton n Equaton () 2 corresonds to the aroxmaton of f( x), the M manfold on whch the ostve samles resde. Mnmzng the objectve functon can encourage the consstent outut for the ostve samles n the hgh dmensonal sace and ths wll result n transformng wth hgh local smoothness and best (6) (7) local reservaton. Hence, a smooth transformaton that s exected to be less lkely to over ft the tranng samles can be learned by ths Lalacan regularzaton term. By ntegratng the Lalacan regularzer nto the searablty term, we can easly obtan the for IR,.e., * α = arg max ( tr[ α X ( B λ* L β* L2) X α] ) (8) he value of λ s used to balance the between-class searablty and the wthn-class comactness, whch can be set as n exerments for smlcty. And the value of β trades off the searablty of dfferent classes and the local consstency of ostve samles n the ostve class. In the followng exerments, we resent the senstvty of n relaton to the regularzaton arameter β, and then select the value that shows the best erformance. III. CONEN-BASED IMAGE RERIEVAL SYSEM Gven a query mage by the user, the CBIR system s exected to feedback more semantcally relevant mages after each feedback teraton []. We have mlemented a CBIR system to evaluate the referred IR algorthms. he framework of the CBIR system s llustrated n Fgure. From Fgure, we can notce that when a query mage s rovded by the user, the system frst extracts the low level features of the query. he all the mages n the database are sorted based on a certan smlarty metrc,.e., Eucldean dstance metrc. If the user s satsfed wth the results, the retreval rocess s ended, and the results are resented to the user. However, because of the semantc ga, most of the tme, the user s not satsfed wth the frst retreved results. hen she/he wll label the most semantcally relevant and rrelevant mages as ostve and negatve feedbacks n to retreved results, resectvely. Based on these feedback samles, an RF model can be traned based on certan machne learnng algorthms. he smlarty metrc can be udated as well as the RF model. hen, all the mages are resorted based on the recalculated smlarty metrc. If the user s not satsfed wth

4 the retreved results, the IR rocess wll be erformed teratvely. Average Precsons Retreval average recson n to 30 retreved mages Average Recalls 5 0. Retreval average recall n to 30 retreved mages Log(beta) Log(beta) (a) (b) Fg.2 Performance evaluaton at the to 30 reranked mages for dfferent value of β (a) average recsons. (b) average recalls we fnd that the regularzaton arameter β can sgnfcantly IV. EXPERIMENAL RESULS affect the results. However, t s stll oen roblem how to All exerments are mlemented on a Corel Image Database, tune the regularzaton arameter to balance the searablty whch ncludes 0,763 mages wth 80 dfferent concetual term and the local consstence term. From the results, we can grous, e.g., tger, castle, cloud, dog, elehant, ceberg, tran see that for ths roblem, the algorthm acheves best and waterfall. o reresent mages, we utlze the color 6 erformance when β s set as 2. herefore, n the followng hstogram [0], Webber s law descrtors [] and the edge 6 drectonal hstogram from Y comonent n YCrCb sace [2], exerments, we emrcally set the tradeoff arameter β = 2. each of whch can descrbe the semantc content of mages It s convnced that the arameter β can be further tuned to from dfferent asects, resectvely. All of these features are acheve better erformance. hs analyss above also combned nto a feature vector, whch results n a vector wth ndcates the mortant role of local consstency for mrovng 50 values, and then we normalze each feature to a normal the generalzaton ablty. dstrbuton. In exerments, 500 queres are randomly selected from the B. Statstcal Exermental Results mage database, and then RF s automatcally done by a In ths art, we comare the roosed wth some comuter. At each feedback teraton, the to 20 retreved reresentatve methods,.e., [3], D [4], [5]. result are serally examned from the to; the frst 5 query he fgures n Fg.3 show the average recsons and standard relevant mages are labeled as ostve feedbacks and the frst devatons of the 500 exerments for to 0, 30 and 50 results. 5 query rrelevant mages are marked as negatve feedbacks We can see that the can acheve better erformance unless fewer such mages are found among the to 20 mages, comarng wth the orgnal. n whch case the fewer number of samles found are used as he unreasonable Gaussan dstrbuton assumton for the the feedbacks. All the labeled mages n the feedback ostve samles n the orgnal and the regularzaton teratons are used to tran an IR model. method used are the man reasons whch usually result n oor In ths aer, average recson, standard devaton, and erformance of IR for CBIR. he D algorthm solves the average recall are utlzed to evaluate the erformance of sngular roblem by the drect method and can acheve better algorthms. In exerments, the arameter k and k2 are erformance than the orgnal. However, much emrcally set as 4 accordng to manfold learnng aroaches dscrmnatve nformaton contaned n the null sace of S b s [7] [8]. dscarded. he algorthm effectvely extracts the dscrmnatve nformaton from the margnal samles; A. Senstvty n relaton to the regularzaton arameter however, t stll suffers from the sngular roblem, whch hs exerment shows the senstvty of wth regard causes serous stablty roblems for. he roosed to dfferent values of the regularzaton arameter β. In method can extract the most dscrmnatve nformaton exerment, we emrcally set the arameter β as a value n a from the nearest neghborhood samles n dfferent classes, sequence,.e., {2, = 0, 9,, 9, 0}. Fg.2 shows the but never encounters the sngular roblem. By ntroducng the Lalacan regularzaton term, a locally smooth transformaton average recson and average recall curves n to 30 results of can also be learned whch s exected to be less vulnerable to the 7-th feedback teratons wth dfferent β values based on over ft the tranng samles and have good generalzaton 500 ndeendent exerments, resectvely. In exerments, roertes. he exermental results have shown that the

5 roosed consstently outerforms the, D and. Average Precsons Retreval average recson n to 0 retreved mages D Average Precsons Retreval average recson n to 30 retreved mages D Average Precsons Retreval average recson n to 50 retreved mages D 5 (a) (b) (c) Standard Devatons Retreval standard devaton n to 0 retreved mages 4 2 D Standard Devatons Retreval standard devaton n to 30 retreved mages D 0.8 Standard Devatons Retreval standard devaton n to 50 retreved mages D (d) (e) (f) Fg.3 Evaluaton exermental results based on the Corel database wth 0,768 mages wth 300 queres. (a), (b), and (c) dslay the retreval average recson n to 0, 30, and 50 retreved mages, resectvely. (d), (e), and (f) dslay the corresondng standard devson of the recson curve n to 0, 30, and 50 retreved mages, resectvely. V. CONCLUSIONS In ths aer, we have studed a new subsace learnng algorthm, Lalacan Regularzed Subsace Learnng () for Interactve Rerankng (IR) n CBIR. he roosed algorthm s under the grah embeddng framework, wth a artcular desgned ntrnsc grah and a enalty grah to smultaneously cature the wthn-class comactness and the between-class searaton, resectvely. By ntroducng a Lalacan regularzaton term, a smooth and locally consstent transformaton can also be learned for IR to effectvely reduce the rsk of over fttng. Extensve exermental results on a large real world mage database have shown that the roosed method has much mroved erformance than other reresentatve methods. REFERENCES [] R. Datta, D. Josh, Ja L, J.Z. Wang, Image retreval: deas, nfluences, and trends of the new age. ACM Comutng Surveys, Vol. 40, No.2,. 60, Ar., [2] X. Zhou and. Huang, Relevance feedback for mage retreval: a comrehensve revew, ACM Multmeda Systems J., Vol. 8, , [3] X. Zhou and. Huang, Small samle learnng durng multmeda retreval usng basma, In Proc. IEEE Int l Conf. Comuter Vson and Pattern Recognton,.-7,200. [4] D. ao, X. ang, X. L, and Y. Ru, Kernel drect based dscrmnant analyss: a new content-based mage retreval relevance feedback algorthm, IEEE rans.on Multmeda, Vol.8, No.4, , Aug [5] D. Xu, S. Yan, D. ao, S. Ln, H. Zhang, Margnal fsher analyss and Its varants for human gat recognton and content-based mage retreval, IEEE rans. on Image Processng Vol.6, No,, , Nov [6] W. Ban and D. ao, "Based Dscrmnant Eucldean Embeddng for Content-Based Image Retreval", IEEE rans. on Image Processng Vol.9, No,2, , Feb [7] S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, and S. Ln, Grah embeddng and extensons: A general framework for dmensonalty reducton, IEEE rans.on Pattern Anal. Mach. Intell., Vol. 29, No.,.40 5, Jan [8] X. He, S. Yan, Y.Hu, P. Nyog and H. Zhang, Face recognton usng lalacan faces, IEEE rans.on Pattern Anal. Mach. Intell., Vol. 27, No. 3, , Dec [9] X. He and P. Nyog, Localty reservng rojectons, In Advances n Neural Informaton Processng Systems, Vol. 6, [0] M. J. Swan and D. H. Ballard, Color ndexng, Internatonal. Journal of Comututer. Vson., Vol.7, No.,. 32, 99. [] J. Chen, S. Shan, G. Zhao, X. Chen, W. Gao, Matt Petkanen, WLD: A Robust Descrtor based on Weber s Law, IEEE rans.on Pattern Anal. Mach. Intell., Vol. 32, No.9, , Se. 200.

6 6 [2] Y. Rubner, J. Puzcha, C. omas, J. M. Buhmann, Emrcal evaluaton of dssmlarty measures for color and texture, Comuter Vson and Image Understandng, Vol.84,