Multi-Perspective Cost-Sensitive Context-Aware Multi-Instance Sparse Coding and Its Application to Sensitive Video Recognition 1

Transcription

1 Mul-Perspecve Cos-Sensve Conex-Aware Mul-Insance Sparse Codng and Is Applcaon o Sensve Vdeo Recognon Weng Hu, Xnao Dng, Bng L, Janchao Wang, Yan Gao, Fangsh Wang 3, and Sephen Maybank 4 whu@nlpr.a.ac.cn; dngxnao@6.co; bl@nlpr.a.ac.cn; janchao030@63.co; @qq.co; fshwang@bju.edu.cn; sjaybank@dcs.bbk.ac.uk aonal Laboraory of Paern Recognon, Insue of Auoaon, Chnese Acadey of Scences, Bejng 0090 Shandong Insue of Busness and Technology 3 Bejng Jaoong Unversy, Bejng 00044) 4 Deparen of Copuer Scence and Inforaon Syses, Brkbeck College, Male Sree, London WCE 7HX Absrac: Wh he developen of vdeo-sharng webses, PP, cro-blog, oble WAP webses, and so on, sensve vdeos can be ore easly accessed. Effecve sensve vdeo recognon s necessary for web conen secury. Aong web sensve vdeos, hs paper focuses on volen and horror vdeos. Based on color eoon and color harony heores, we exrac vsual eoonal feaures fro vdeos. A vdeo s vewed as a bag and each sho n he vdeo s represened by a key frae whch s reaed as an nsance n he bag. Then, we cobne ul-nsance learnng (MIL) wh sparse codng o recognze volen and horror vdeos. The resulng MIL-based odel can be updaed onlne o adap o changng web envronens. We propose a cos-sensve conex-aware ul-nsance sparse codng (MI-SC) ehod, n whch he conexual srucure of he key fraes s odeled usng a graph, and fuson beween audo and vsual feaures s carred ou by exendng he classc sparse codng no cos-sensve sparse codng. We hen propose a ul-perspecve ul-nsance jon sparse codng (MI-J-SC) ehod ha handles each bag of nsances fro an ndependen perspecve, a conexual perspecve, and a holsc perspecve. The experens deonsrae ha he feaures wh an eoonal eanng are effecve for volen and horror vdeo recognon, and our cos-sensve conex-aware MI-SC and ul-perspecve MI-J-SC ehods ouperfor he radonal MIL ehods and he radonal SVM and K-based ehods. Index ers: Cos-sensve conex-aware MI-SC, Mul-perspecve MI-J-SC, Horror vdeo recognon, Volen vdeo recognon, and Vdeo eoonal feaure exracon.. Inroducon The eergence and developen of vdeo-sharng webses, PP, cro-blog, podcasng, oble WAP webses, and 3GP webses faclae he dssenaon of sensve vdeos, such as adul, horror, volen, and Copyrgh (c) 03 IEEE. Personal use of hs aeral s pered. However, persson o use hs aeral for any oher purposes us be obaned fro he IEEE by sendng a reques o pubs-perssons@eee.org.

2 errors vdeos. Fg. shows soe exaples of volen vdeos and horror vdeos. Dffuson of sensve vdeos poses a ajor hrea o naonal secury, socal sably, and he physcal, psychologcal, and enal healh of vewers. Effecve recognon of sensve vdeos s necessary for web conen secury [54]. Recognon of sensve vdeos s a newly eergen research opc n he uleda and paern recognon counes, n he conex of uleda rereval [55, 56], uleda conen undersandng [59, 60], and ulodal fuson [56, 57], ec. In recen years a nuber of specfc aeps have been ade o deal wh he proble of sensve vdeo recognon, and os of he focus on adul vdeo recognon [,, 8, ]. In hs paper, we focus on recognon of horror vdeos and volen vdeos. (a) (b) Fg.. Exaples of fraes aken fro (a) volen vdeos and (b) horror vdeos... Relaed work Volen vdeos usually sulae psychc pulses by showng he use of force o njure ohers or oneself. The conens of volen vdeos [5] nclude fghs, gun shos, explosons, and self-ulaon. The curren recognon ehods usually use vsual feaures or audo feaures separaely or fuse vsual and audo feaures. The vsual feaures can be used o deec huan volence, such as kckng and fs fghng, n vdeos [5]. For nsance, Daa e al. [] adoped an acceleraed oon vecor o deec fgh scenes. Wang e al. [44] deeced volence n vdeos usng he accuulaed squared dervave feaures whch were exraced fro dense rajecores derved fro vdeos. Xu e al. [53] deeced volen vdeos by capurng dsncve local shape and oon paerns. Audo feaures can be used o deec volen speech or acons. For nsance, Cheng e al. [3] used a herarchcal audo-based ehod o denfy car racng and gunplay. Theodoros e al. [4, 3] exraced egh audo feaures fro he frequency and e doans o deec volen vdeos. Acar e al. [50] deeced volen vdeos usng d-level audo feaures n a bag-of-audo words ehod usng Mel-frequency

3 Cepsral coeffcens (MFCCs). Vsual and audo feaures can be cobned o ore accuraely locae volen scenes. a e al. [3] recognzed volen vdeos by deecng blood and flaes and explong represenave audo effecs, such as explosons and gunshos. Seaon e al. [35] cobned vsual and audo feaures o selec represenave shos n an acon vdeo. Gannakopoulos e al. [43] deeced volence usng he sascs of audo feaures and average oon and oon orenaon varance feaures. Ln and Wang [45] cobned audory and vsual classfers n a co-ranng way o deec volen shos n oves. Horror vdeos srve o elc he prary eoons of fear, horror, and error. The conens of horror vdeos nclude seral kllngs, ghoss, onsers, vapres, anal kllng, and rrelgon. Horror nforaon ay arouse fears n chldren and eenagers and even nduce phobas [46, 47]. The earler work [5, 6, 8] on horror vdeo recognon was carred ou as a par of a vdeo scene classfcaon based on huan eoons. Specfc work on horror vdeo recognon wh s own characerscs eerged [3, 4]. Xu e al. [4] deeced audo eoonal evens o locae horror vdeo segens n vdeos whch are known a pror o conan such segens. Wu e al. [3] represened each vdeo as a bag of ndependen fraes and appled ul-nsance learnng (MIL) o horror vdeo recognon. The curren ehods for volen and horror vdeo recognon have he followng laons: They focus on usng low level vsual, oon, and audo feaures, or hey only use affecve audo feaures. Research on affecve color and vsual seancs, ogeher wh affecve audo seancs n volen and horror vdeos, s sll exploraory, bu he resuls of hs research are avalable for applcaon o volen and horror vdeo recognon. The curren ehods only focus on ndependen fraes and do no consder he underlyng conexual cues whn volen and horror vdeos, even hough conexual cues beween fraes are useful for recognzng volence and horror. Whle conexs beween fraes are useful for recognzng volen and horror eoons, ndependen frae cues also have eoonal conen. The ndependen frae cues, conexual cues aong fraes, and holsc feaures of he enre vdeo are dfferen sources of nforaon for volen and horror vdeo recognon. Well-chosen feaures fro dfferen perspecves can ebody a varey of dscrnave nforaon. The curren volen and horror recognon algorhs do no nclude he fuson of ul-perspecves o prove her perforance. Web nforaon changes rapdly. The curren volen and horror vdeo algorhs, overall, are unable 3

4 o updae he classfers onlne when new ranng saples are obaned... Our work As a varan of supervsed learnng, each saple for ul-nsance learnng (MIL) s a bag of nsances nsead of a sngle nsance. Each bag s gven a dscree or real-valued label. In bnary classfcaon, a bag s consdered as posve f a leas one nsance n s posve, and consdered as negave f all s nsances are negave. As a pror, a volen or horror vdeo conans a leas one volen or horror sho, and all he shos n a non-volen or non-horror vdeo are necessarly non-volen or non-horror. If a vdeo s reaed as a bag and a sho n he vdeo s reaed an nsance n he bag, volen and horror vdeo recognon s conssen wh he fraework of MIL. So, we use MIL o recognze volen and horror vdeos. The os curren odels for MIL n coon use, such as axs-parallel conceps [5], he dverse densy (DD) ehod [5], he expecaon-axzaon verson of dverse densy (EM-DD) [7], he MI-kernel ehod [8], he -SVM and MI-SVM [9], he -Graph and MI-Graph [9], and he adapve p-poseror xure-odel (PP-MM) kernel [4], are raned n bach sengs, n whch he enre ranng se s avalable before each ranng procedure begns. Babenko e al. [48] proposed an onlne MIL algorh based on a boosng echnque. However, hs onlne ehod assues ha all he nsances n a posve bag are posve. Ths assupon s easly volaed n praccal applcaons. L e al [49] exended he MIL algorh based on ebedded nsance selecon [6, 7] o an onlne MIL algorh. However, a classfer sll needs o be reraned usng he new saples. The caon-k [6] s no par of he ranng process. I deernes he label of each es bag usng he labeled bag saples neares o he es bag and he bag saples whose neares bag saples conan he es bag. However, he caon-k s sensve o ouler saples. Sparse codng (SC) s ranng-free, and he odel can be updaed onlne each e he labeled saple se s updaed. Furherore, SC s no sensve o oulers, because he sparsy regularzaon can suppress oulers n he sparse represenaon. Therefore, we cobne MIL wh sparse codng o for a ul-nsance sparse codng (MI-SC) echnque for recognzng volen and horror vdeos. The conrbuons of our work are suarzed as follows: We exrac color eoonal feaures accordng o he resuls fro psychologcal experens. These color eoonal feaures brdge he affecve seanc gap o soe exen. The color eoonal feaures ogeher wh low-level vsual feaures, oon feaures, and audo feaures are used for A sho s a consecuve sequence of fraes capured by a caera acon whch akes place beween sar and sop operaons. 4

5 volen and horror vdeo recognon. We propose a cos-sensve conex-aware MI-SC ehod whch can ake use of he conex aong fraes n he sae vdeo and he conex beween vsual and audo cues for volen and horror vdeo recognon. A vdeo s dvded no a seres of shos va sho segenaon and a key frae fro each sho s seleced. The vsual feaure vecor of each key frae s exraced o represen he sho n whch he key frae exss. An audo feaure vecor s exraced for he enre vdeo. A vdeo s represened as a bag of nsances whch correspond o he vsual feaure vecors. A graph s consruced usng he key fraes as nodes o represen her conexual relaons. A cos-sensve sparse codng odel s consruced o represen he conex beween he bag of vsual feaure vecors and he audo feaure vecor. We solve he cos-sensve conex-aware MI-SC usng he exsng feaure sgn search algorh va a aheacal ransforaon. We propose a ul-perspecve ul-nsance jon sparse codng (MI-J-SC) ehod o cobne nforaon fro a conexual perspecve, an ndependen perspecve, and a holsc perspecve. The conexs beween key fraes for only a conexual perspecve for volen and horror vdeo recognon. A key frae also ncludes seanc eanng, so reang a vdeo as a bag of ndependen nsances can be consdered as an ndependen perspecve. The holsc feaures for he enre vdeo can be reaed as anoher perspecve. The nforaon fro dfferen perspecves ore fully descrbes a vdeo. The curren MIL lacks he ably o fuse ul-perspecves. We ncorporae he jon sparse codng no ul-nsance classfcaon o fuse he feaures fro ul-perspecves, n order o oban ore accurae recognon of volen and horror vdeos. The experenal resuls show he effecveness of he exraced vdeo eoon feaures. The resuls on he volen and horror vdeo daases show ha our ehods ouperfor he radonal MIL-based ehods and he radonal SVM and K-based ehods. The resuls on he general MIL daases show ha our ehods ay be effecve for oher general ul-nsance probles. The reander of hs paper s organzed as follows: Secon presens he MI-SC echnque. Secons 3 and 4 propose our cos-sensve conex-aware MI-SC and ul-perspecve MI-J-SC ehods, respecvely. Secon 5 presens our ehod for exracng eoonal feaures and our ehod for recognzng volen and horror vdeos. Secon 6 repors he experenal resuls. Secon 7 concludes hs paper. 5

6 . Mul-Insance Sparse Codng Mul-nsance sparse codng (MI-SC) carres ou MIL usng he sparse codng echnque. In he followng, we frs brefly nroduce sparse codng. Then, we descrbe he echans of MIL va sparse codng... Sparse codng The goal of sparse codng [0] s o represen each npu vecor approxaely as a weghed lnear cobnaon of bass vecors such ha a sall nuber of weghs are non-zero. Gven an h-densonal npu vecor h x and n bass vecors hn U [ u, u,..., u ], a sparse vecor n w n w, whose enry j ( j n) s he wegh of u j, s found such ha n j x Uw u w. () j j The objecve of sparse codng s usually forulaed as he nzaon of he reconsrucon error wh sparsy regularzaon: n w x Uw w () where he nor w of w s he sparsy er and λ s a regularzaon facor o conrol he sparsy of w... MIL va sparse codng For MIL, a ranng daase {( X, y),,( X, y ),,( X, y )} consss of bags { X } and her labels { y }. A bag X consss of n nsances: X { x,,, x,,, x, }, where each nsance x, j s a j n vecor. The ask of MI-SC s o sparsely cobne he ranng bags { X } o represen a es bag. Due o he se srucure of he bags, a es bag canno drecly be sparsely and lnearly reconsruced usng he ranng bags. We apply a appng funcon : d X o ap each bag X o a hgh densonal vecor space: X( X )(he descrpons and handlng of he appng funcons wll be dealed n Secon 3.3). Then, by appng he ranng bags o he hgh densonal vecor space, a bass arx B [ ( X), ( X),, ( X )] for sparse codng s obaned. Gven a es bag X, he sparse codng n he hgh densonal vecor space s defned as: n ( X ) Bw w. (3) w The label of X s deerned by he labels of he ranng saples whose weghs are nonzero for sparsely 6

7 represenng X. I s clear ha hs s a ranng-free onlne learnng odel whch s updaed only by changng he labeled saples. The laon of he above MI-SC s ha he conexs aong nsances are no odeled. 3. Cos-Sensve Conex-Aware MI-SC To handle he above laon, we forulae conex-aware MI-SC and cos-sensve sparse codng, and propose a ehod for opzng he coeffcens for he cos-sensve conex-aware MI-SC. 3.. Conex-aware MI-SC Tradonal MIL usually assues ha nsances n a bag are ndependen of each oher. Zhou e al. [9] bul a graph [33] n her SVM-based MIL ehod o odel he conexs beween nsances n each bag. Ths graph represenaon of conexs s ncorporaed no our MI-SC ehod. For a bag X, a graph G whose nodes are he nsances n he bag s consruced. The dsances beween nsances are copued. If he dsance beween wo nsances s saller han a prese hreshold, hen he wegh for he edge beween he correspondng wo nodes s se o, oherwse he wegh s se o 0. A nn arx E of he adjacency weghs for G s obaned, where Eaa, ( a,,..., n ). The ranng saples are represened as {( X, G, y ),...,( X, G, y ),...,( X, G, y )}, and a es bag s gven as d ( X, G, y ). We apply a appng funcon : G o ap each graph G o a hgh densonal vecor space: G ( G). Then, he bass arx for spare codng s replaced by C [ ( G), ( G),..., ( G n )]. The conex-aware MI-SC s forulaed as: 3.. Cos-sensve sparse codng n ( ) Cw w. (4) w G In real applcaons, each bag X ay be assocaed wh anoher knd of feaure. For exaple, an audo s usually assocaed wh a vdeo, and he holsc feaures of he audo can overall characerze he enre vdeo. We propose a cos-sensve sparse represenaon o ncorporae he assocaed feaures no he bags. For each bag X, s assocaed feaure vecor a s exraced. Then, he ranng se can be represened by ( a, X, G, y ),( a, X, G, y),...,( a, X, G, y ). Gven a es bag X, we defne a dagonal arx D whose dagonal enres are he Eucldean dsances beween he assocaed feaure vecor of he es bag and he assocaed feaure vecors of each ranng bag: 7

8 D dag( a a,, a a,, a a ). (5) To ncorporae he assocaed feaures no he MI-SC, we forulae cos-sensve conex-aware MI-SC n a hgh densonal feaure space as follows: n ( ) w G Cw Dw. (6) where he dagonal arx D s ncluded no he nor n (4). The enres n D are cos values for he dfferen ranng saples. In hs way, he ranng saples, whose assocaed feaure vecors have sall dsances o he assocaed feaure vecor of he es bag, are ore lkely o be seleced o reconsruc he es bag. In he sensve vdeo recognon applcaon, he vdeos whch have audo racks slar o he es vdeo are ore lkely o be chosen o represen he es vdeo Opzaon The radonal sparse codng opzaon ehods canno be drecly appled o he cos-sensve conex-aware MI-SC n (6). We ransfor he objecve funcon n (6) o a for o whch he radonal sparse codng opzaon can be appled. Then he feaure sgn search (FSS) algorh s used o solve for he coeffcen vecor w. Le q Dw, where q. In order o ensure ha D s nverble, we add a very sall value ε o he dagonal enres of D, and oban an nverse as follows:,,..., dag D a a a a a a (7) Subsung - w= D q no (6) yelds: n ( ) q G CD q q. (8) Le V CD, where V d. Forula (8) s rewren as: n ( ) Vq q. (9) q G The funcon () whch s used o ap bags no a hgh densonal space s dffcul o defne explcly. T Insead, he scalar produc ( G) ( G ) n he hgh densonal space s explcly defned va a kernel j T funcon. So, we ransfor he objecve of (9) no a for nvolvng scalar producs ( G) ( G ). I s clear ha j T T T T T T ( G ) Vq =[ ( G ) Vq] [ ( G ) Vq] [ ( G )] ( G ) q V Vq q V ( G ). (0) Then, we only need o consder T VV and V T ( G ). I s clear ha 8

9 T V V = CD CD ( D ) C CD T T T ( D ) T G G G G G G = ( D ) ( ), ( ),..., ( ) ( ), ( ),..., ( ) D T ( G ) ( G ) ( G ) ( G )... ( G ) ( G ) ( G ) ( G ) ( G ) ( G )... ( G ) ( G ) D T T T ( G ) ( G) ( G ) ( G)... ( G ) ( G ) T T T T T T T () and V CD D T T T T ( G ) ( ) ( G ) ( ) [ ( G), ( G),..., ( G )] ( G ) ( D ) ( G ) ( G ) ( ) ( G ) T ( G) ( G) T T T G. () T I reans o defne a graph kernel funcon K g () o represen he scalar produc ( G) ( Gj) of graphs G and G j n he hgh densonal feaure space. The defnon of a graph kernel funcon depends on a kernel funcon beween any wo nsances. The Gaussan radal bass funcon (RBF) kernel K( x, a, x j, b) beween an nsance x a, n bag and an nsance x jb, n bag j s defned as:, a j, b, a j, b K( x, x ) exp x x (3) where ρ s a scalng facor. Le a, be he wegh for he nsance x, a, n bag, whch s defned as: a, n u E au, (4) where u s he ndex for an nsance n bag and E s he adjacency wegh arx for bag. The kernel funcon g () [9] beween graphs G and G j s defned as: n n j K( x, x ), a j, b, a j, b a b g ( G, Gj ) n n, a j, b a b. (5) Usng he graph kernel funcon, he objecve funcon n (9) s explcly forulaed, and hen he opzaon n (9) s effcenly solved by he recenly proposed feaure-sgn search algorh (FSS) [30] Classfcaon Afer he opal coeffcen vecor q s obaned, we calculae he reconsrucon resdual of he es bag for each bag label, and he label wh he salles reconsrucon resdual s seleced as he label o whch 9

10 he es bag belongs. For each label, we defne a vecor δ whose l-h enry l s: ql, yl l 0, yl (6).e., hs vecor only selecs coeffcens assocaed wh labels. The reconsrucon resdual ( G ) of he es bag for label s defned as: T T T T G G G ( ) ( ) Vδ ( δ ) VVδ ( δ ) V ( ) (7) T where ( G) ( G). We assgn he es bag he fnal label c whch s defned as follows: c arg n( ( G )). (8) 4. Mul-Perspecve Mul-Insance Jon Sparse Codng Based on he srucured jon sparse represenaon [, 3, 34], we propose ul-perspecve cos-sensve MI-J-SC whch ncludes he above cos-sensve conex-aware MI-SC. 4.. Srucured jon sparse represenaon I s assued ha here are K dfferen ypes of feaure and M labels n he ranng daase. Le k hk Ψ be he arx of each feaure k (k=,,..,k) for he ranng saples wh label, where k h s he denson of he k-h ype of feaure and s he nuber of he ranng saples wh label : M. Then, he arx of he k-h ype of feaure for all he ranng saples s hk Ψ [ Ψ, Ψ,..., Ψ ]. The k-h ype s feaure vecor z k of a es saple Z s reconsruced fro he k k k k M k-h feaure vecors of he ranng saples: M k k k k z Ψ w (9) where, and k w s he reconsrucon coeffcen vecor for he k-h feaure vecors of he saples wh label k k k T k T k T T s he resdual er. Le ( ),( ),...,( M ) w w w w be he coeffcen vecor for he k-h ype of feaure. Le [,,..., K K W w w w ]. The, -xed nor of W s: M K M k, k W w W (0) where W w w w. Then, he reconsrucon n (9) can be represened by he leas square K K [,,... ] 0

11 regresson based on he, xed- nor regularzaon [, 3, 34]: K n W z Ψ w W. (), M k k k k The, xed-nor ncludes he nor of he vecor of he coeffcens of he K feaure vecors for each ranng saple and he nor of he vecor of he nor values for all he saples. The, xed-nor guaranees jon sparse represenaon. The reasons are suarzed as follows: The nor n he, xed-nor ensures ha he ranng saples chosen o represen a es saple are as few as possble. The nor n he, nor ensures ha when a ranng saple s no chosen o represen a es saple, all he K feaure vecors of he ranng saple are no chosen o represen he es saple. Ths srucured jon sparse codng can effecvely fuse nforaon fro ulple feaures. 4.. Mul-perspecves of ul-nsances We exend he above srucured jon sparse represenaon o MIL o fuse nforaon fro ul-perspecves. Dfferen perspecves can be defned accordng o dfferen applcaons. We defne he followng hree ul-perspecves n he conex of sensve vdeo recognon: ) Independen perspecve: As n radonal MIL, he nsances n a bag are reaed as ndependen. We defne a appng funcon d : X o ap he feaure space of he bags { X } o a d -densonal vecor space: X ( X ). Then, he ranng saples are ransfored o F [ ( X ),, ( X ),, ( X )]. In he d -densonal vecor space, we defne a kernel funcon () beween any wo bags X and as follows: X j ( X, X ) [ ( X )] ( X ) n n j K( x, x ) a jb T a b j j n n n j nj K( x, x ) K( x, x ) l l js js l l s s () where he kernel K() beween wo nsances s defned as n (3). ) Conexual perspecve: The graph consraned MI-SC n Secon 3. s nroduced o for a conexual perspecve for MIL. We defne a appng funcon d : G o ap he feaures of each bag wh a graph G o a d -densonal space: G ( G) ( s jus n (4)). The conex-aware ranng

12 bags are ransfored o F [ ( G ),, ( G),, ( G)]. In he d -densonal vecor space, he kernel funcon () s defned as n (5). 3) Holsc perspecve: Sascal hsogras of nsances n bags can be used for bag classfcaon. Fro a holsc perspecve, we consruc a feaure hsogra for a bag based on he bag-of-words odel [4]. Gven he se of he ranng bag saples, all he nsances are clusered o for a lexcon of R code words { d,, d,..., d }. Each nsance x j n a bag r R X s apped o a code word ( x ) whch s deerned by: j ( x ) arg n x d. (3) j j r rr In bag X, he nuber of occurrences hrx (, ) of each code word r ( r,,..., R ) s couned: h( r, X ) { x X : ( x ) r}, where s he nuber of enres n a se. Then, bag j j X s represened by a noralzed hsogra ξ : ξ h(, X ) h( r, X ) h( R, X ),,,, h( r, X ) h( r, X ) h( r, X ) R R R r r r. (4) Then, he se of he ranng saples s represened by {( X, ξ, y ),,( X, ξ, y ),,( X, ξ, y )}. We ap each hsogra feaure vecor o a hgh densonal feaure space usng a appng funcon 3 d3 : ξ. Then, he hsogras of he ranng saples are ransfored o F [ ( ξ ),..., ( ξ ),..., ( ξ )]. In hs hgh densonal space, we defne he kernel funcon beween any wo bags as follows: R R 3 T 3 3 j j r j r K r r rr ( ξ, ξ ) [ ( ξ )] ( ξ ) ξ [ ] ξ [ ] ( d, d ) (5) where K( dr, d ) r s he Gaussan kernel funcon beween wo code words d and d : r r r r r r K( d, d ) exp( d d ). (6) 4.3. Mul-perspecve cos-sensve MI-J-SC We use he srucured jon sparse represenaon n Secon 4. o fuse he nforaon fro ul-perspecves such as defned n Secon 4.. Also, cos-sensve sparse codng can be appled o he srucured jon sparse represenaon. Then, we propose a ul-perspecve cos-aware MIL ehod by negrang ul-perspecves no a unfed jon sparse codng fraework based on he, nor. Gven K perspecves (K s 3 n hs paper), he ranng saple se s represened by K arces K {,,..., } F F F, where

13 F k [ k ( X), k ( X),..., k ( X )]. Gven a es saple X, s feaure vecor n each perspecve k s k k represened by f ( X ). Le k w be he coeffcen vecor for he ranng saples a perspecve k and W be he arx of he coeffcen vecors of he K perspecves: [,,..., K K W w w w ]. Then, ul-perspecve cos-sensve MI-SC s represened by: K n W k k k k f F w DW, (7) where D s he cos arx defned n (5). In (7), he frs er s he su of he squared reconsrucon errors fro dfferen perspecves, and he second er s he regularzaon o conrol he sparsy of he coeffcens. We group he ranng feaure se k F of each perspecve k accordng o he class labels { } M of he k k k k ranng saples: F [ F,..., F,..., F M] where k F s he arx whch consss of he k-h feaure vecors of he ranng saples wh label. Accordngly, he k-h coeffcen vecor n W s also grouped as: k T k T k T ( w ),...,( w),...,( w M). Le T K K W [ w, w,..., w ] (=,,..,M), where s he nuber of he ranng saples wh class. Then, Equaon (7) s rewren as: n W f F w D W (8) K M M k k k, k where D s he dagonal arx whose enres are hose eleens n D correspondng o he ranng saples wh label Opzaon The, xed-nor acceleraed proxal graden (APG) algorh [34] s nroduced o opze he objec funcon n (8). The APG canno be drecly appled o (8). We ake a ransforaon o (8). Le Q D W where Q q q q. K K [,,..., ] W D Q. F w F ( D ) q. Le k k k k k k U F ( D ). I follows ha (8) s equvalen o: n W f U q Q. (9) K M M k k k k The APG algorh can be appled o (9). The APG algorh alernaely updaes a coeffcen arx k, Q [ q ] and an aggregaon arx k, V [ v ] a each eraon whch consss of a generalzed graden appng sep and an aggregaon sep. 3

14 In he generalzed graden appng sep, gven he curren aggregaon arx V ˆ, he coeffcen arx U [ U,..., U,..., U ]. I s clear ha k k k k Q s updaed. Le M ( X ) ( X ) ( X ) ( X )... ( X ) ( X ) k T k k T k k T k k T k k T k k T k T T ( ) ( ) ( ) ( )... ( ) ( ) k k ( ) X X X X X X U U D D k T k k T k k T k ( X ) ( X) ( X ) ( X)... ( X ) ( X ) (30) and ( X ) ( X ) ( ) ( )... k T k ( X) ( X) k T k k T k T k T k ( ) ( ) X X U X D (3) k T k where he scalar produc ( X) ( X j) beween bags X and X j s evaluaed usng a kernel funcon,,, whch s explcly defned n (5), (), or (5). A arx [,..., k,..., K ] P p p p K s defned as follows: p U X U U, k=,,..,k. (3) k, T, ( ) T v k k k k Then, k, k, k, q v q, k=,,,k (33) and q ax,0 q,,, M q (34) where s he sep sze paraeer. In he aggregaon sep, he aggregaon arx s updaed by consrucng a lnear cobnaon of Q and Q : ( ) V Q ( Q Q ) (35) where convenonally / ( ) [36] Classfcaon Usng he obaned opal coeffcen arx Q, he reconsrucon resdual ( X ) of he es bag for label {,, M} s defned as: 4

15 K K k k k k T k T k k k T k k K k k ( X ) ( X ) U q ( δ ( q )) ( U ) U δ ( q ) ( U ) ( X ) δ ( q ) (36) k where δ ( q ) s a coeffcen selecor ha only selecs coeffcens assocaed wh label,.e., he l-h k enry n δ ( q ) s defned as follows: k d ( q ) l ql, yl. (37) 0, yl Slar o (8), he label ha has he salles resdual s assgned o he es bag X. 5. Sensve Vdeo Recognon A vdeo Vdeo shos Key Fraes Audo Fraes Vsual-audo feaures... An audo feaure vecor A vdeo bag Fg.. Bag consrucon for each vdeo. We apply he proposed cos-sensve conex-aware MI-SC and ul-perspecve MI-J-SC o recognze sensve vdeos, especally horror vdeos and volen vdeos. Gven a se of vdeos { I, I,..., I }, hey are labeled as { y, y,..., y } ( y {,},.e., M=) where a sensve vdeo s labeled and a non-sensve... vdeo s labeled as. Each vdeo I s dvded no n shos { s,, s,,..., s, } by easurng uual n nforaon and jon enropy beween fraes [37]. In each sho, we selec he frae whch s closes o he ean of he color eoonal feaures n he sho as a key frae, and hen a key frae se { θ,, θ,,..., θ, } for n vdeo I s obaned. The vsual and audo feaure vecor, j f for each key frae θ, j s exraced. An audo feaure vecor a s exraced fro he enre audo assocaed wh I. A bag for each vdeo s consruced by reang he feaure vecor of each key frae as an nsance, as shown n Fg.. Then, he above MI-SC 5

16 ehods can be appled o { I, I,..., I }. The opal coeffcens obaned by he cos-sensve conex-aware MI-SC or he ul-perspecve MI-J-SC are used o classfy he es vdeos as sensve or non-sensve. In he followng, we descrbe he feaures exraced fro horror and volen vdeos. The feaures exraced fro vdeos are based on eoonal percepon heory. Dfferen colors, exures, and audo rhyhs ay produce dfferen eoons. So, we exrac he followng vdeo feaures ha produce eoons n he vewers: color eoonal feaures, vsual feaures, and audo feaures. These eoon-producng feaures are used for horror vdeo recognon. Addonal oon feaures are used for recognzng volen vdeos. 5.. Color eoonal feaures Ou e al. [38, 39] developed color eoon odels for sngle colors and harony odels for wo color cobnaons by psychophyscal experens. We exrac color eoonal feaures based on hese color eoon odels. Ou e al. found ha color eoons for sngle-colors depend on he followng hree facors: acvy, wegh, and hea, whch are defned as follows: Acvy ( L 50) ( a 3) (( b 7) /.4) * 0 Wegh (00 L ) 0.45cos( h 00 ) *.07 0 Hea ( C ) cos( h 50 ) * * * (38) where ( L *, a *, b * ) and ( L *, C *, h ) are he color coponens n he CIELAB and CIELCH color spaces, respecvely. Based on (38), we defne an eoonal nensy (EI) for each pxel (x,y) as follows: EI ( x, y) Acvy Wegh Hea. (39) Gven a frae n a vdeo, he EIs for all he pxels are copued. Based on he EIs, a color eoon hsogra s acqured and eployed as par of he color eoonal feaures. Ou and Luo [] developed a quanave wo-color harony odel whch consss of hree ndependen color harony facors: hue effec ( H ), lghness effec ( H ), and chroac effec ( H ). These hree H harony facors for wo colors are explcly esaed usng hues, sauraons, and lghness values of hese wo colors n he CIELAB color space (The deals can be found n []). The overall harony score CH beween hese wo colors s defned as he su of he hree facors: CH HH HC HL. Gven a frae, for each pxel we calculae he color harony score CH beween s color and he ean of he colors of s surroundng pxels and he color harony score CH beween s color and he ean of he colors of all he L C 6

17 pxels n he frae. The color harony score CH f of hs pxel s defned as he su of he wo scores: CH f =CH +CH. Based on he color harony scores n he frae, we consruc a color harony hsogra whch s used as anoher par of he eoonal feaures. 5.. Vsual feaures The vsual eoonal feaures nclude lghng feaures, color feaures, exure feaures, and Rhyh feaures. ) Lghng feaure: Lghng affecs vewers feelngs drecly [5, 7]. The lghng effec s deerned by wo facors: he general level of lgh and he proporon of shadow area. We use he edan of he L values of all he pxels n a frae n he Luv color space [7] o characerze he general level of he lgh n he frae. The proporon of he pxels, whose lghness values are below a ceran shadow hreshold, s used o esae he proporon of shadow area. ) Color feaure: The color values used n he HSV space are clearly dsngushable by huan percepon, so we use he eans and varances of coponens of he HSV color space n a frae o characerze he an cues of colors n he frae. Parcular colors have srong relaons wh ove genres [7]. The parcular colors n a frae can be represened by he covarance arx Θ of he L, u, v values of pxels n he frae: L L, u L, v Θ L, u L u, v. (40) L, v u, v v The deernan of (40), de( Θ ), s used as he feaure for he parcular colors. 3) Texure feaure: Texure s anoher poran facor relevan o age eoon, because dfferen exures gve people dfferen feelngs. Geusebroek and Seulders [40] proposed a sx-sulus bass for sochasc exure percepon: Texure dsrbuons n age scenes confor o a Webull dsrbuon assocaed wh a rando varable x: x wb( x) x e (4) where β represens he conras of he age (a hgher value for β ndcaes ore conras), and γ represens he gran sze of he age (a hgher value for γ ndcaes a saller gran sze,.e., ore fne exures). The paraeers β and γ copleely characerze he spaal srucure of he exure, and hey are used as he exure feaure for horror and volen vdeo recognon. 7

18 4) Rhyh feaure: In horror and volen vdeos, quck sho swchng and srong oons are ofen used o exce nervous oods n he vewers. We use he nverse lengh of a sho o represen he speed of sho swchng. For a frae, he ean and sandard devaon of oons beween fraes n a shor clp cenered a he frae are used o easure he quany of oon assocaed wh he frae [0] Audo feaures Specfc sounds and usc are ofen used o hghlgh eoonal aosphere and prooe draac effecs. The followng audo feaures [9] are exraced: The ean and varance of he MFCCs (Mel-frequency Cepsral coeffcens) of each frae and he MFCCS frs-order dfferenal, where he MFCCs are copued fro he fas Fourer ransfor (FFT) power coeffcens. Specral power whch s used o easure he energy nensy of an audo sgnal: For an audo sgnal s(), each frae s weghed wh a Hang wndow h(), where s he ndex of a saple n he frae. The specral power of an audo frae of he sgnal s() s calculaed as: T o 0log s( ) h( )exp j T 0 T (4) where T s he nuber of saples of each frae, and o s he ndex of an order of he DFT coeffcens. The ean and varance of he specral cenrods of he audo sgnal, whch are eployed as easures of usc brghness. Te doan zero crossngs rae whch provdes a easure of he nosness of an audo sgnal Moon feaures The followng oon feaures are exraced especally for volen vdeo recognon: ) Opcal flow: Corners n he prevous frae are deeced. Opcal flow s used o esae he posons of he corners n he curren frae. The dsance oved by each corner beween he prevous and curren fraes s calculaed. The su, ean, and sandard devaon of he dsances oved by he corners are used as oon feaures. ) Moon eplae: The oon eplae s consruced usng he oon hsory age obaned fro consecuve fraes. The oon eplae s segened no a nuber of regons. The global oon orenaon of he eplae and he ean and sandard devaon of he oon orenaons of all hese regons 8

19 are calculaed and used as addonal oon feaures. Eprcally, all he above color eoonal feaures, vsual feaures, and audo feaures are useful for boh horror vdeo recognon and volen vdeo recognon. The above oon feaures are useful only for volen vdeo recognon, raher han horror vdeo recognon, because volen vdeos use uch ore nense oons han horror vdeos o rgger srong eoons. In each nsance he used feaures are cobned no a feaure vecor. The value of each coponen n a feaure vecor s noralzed accordng o he axu of he values of hs coponen over all he saples n he daase. For he cos-sensve conex-aware MI-SC, all he audo feaures exraced fro an enre vdeo for a sngle audo feaure vecor for he vdeo. Ths audo feaure vecor s used o calculae he cos arx. For he ul-perspecve MI-J-SC, he sae feaures are used n all he hree perspecves. 6. Experens In he experens, he color eoon hsogra has 64 bns. The color harony hsogra has 5 bns. The shadow hreshold for lghng feaure exracon was experenally deerned as 0.8. For an audo sgnal, we exraced a sngle-channel audo srea a 44. KHz and copued MFCCs over 0s fraes. We used he precson (P), recall (R), and F-easure (F ) o evaluae he perforance of an algorh. Le HS be he horror or volen vdeos n a daase, and ES be he vdeos ha are recognzed as horror or volen by he algorh. The precson (P), recall (R), and F-easure (F ) are defned as: HS ES P ES HS ES R HS PR F P R (43) The proposed sensve vdeo recognon ehods were copared wh he followng ehods: EM-DD [7]: Ths s an MIL ehod whch cobnes he EM algorh wh he dverse densy (DD) axzaon [5]. -Graph [9]: Ths ehod uses a graph [33] o odel he conexs beween nsances n a bag. MI-kernel [8]: Ths ehod regards each bag as a se of feaure vecors and hen apples a se-based kernel drecly for bag classfcaon. MI-SVM: Ths ehod s exended fro SVM o deal wh MIL probles. I represens a posve bag by he nsance farhes fro he separang hyper-plane. 9

20 -SVM: I looks for he hyper-plane such ha for each posve bag here s a leas one nsance lyng n he posve half-space, and all he nsances belongng o negave bags le n he negave half-space. Caon-K: I s exended fro K o deal wh MIL probles. I consders no only he labels of he bags whch are neares o he es bag, bu also he labels of he bags whose neares saples conan he es bag. SVM: The feaure vecors of he key fraes n a vdeo were averaged no one vecor. These averaged feaure vecors of all he ranng saples were used o consruc a classcal SVM-based sensve vdeo classfer. K: The K, nsead of he SVM n he SVM-based classfer, was used o ran a classfer. In he followng, we repor frs he resuls of horror vdeo recognon, hen he resuls of volen vdeo recognon, and fnally he resuls on he general MIL daases for valdang he effec of proposed MI-SC ehods. 6.. Horror vdeo recognon We downloaded horror and non-horror vdeos fro he nerne. Ths daase consss of 400 horror vdeos and 400 non-horror vdeos. These vdeos coe fro dfferen counres, such as Chna, US, Japan, Souh Korea, and Thaland. The genres of he non-horror oves nclude coedy, acon, draa, and caroon. Half of he horror vdeos and half of he non-horror vdeos were used for ranng, and he reanng vdeos were used for esng. The average accuraces of en es 0-fold cross valdaon were used o easure he perforance of each ehod. Table shows he values of he average Precson (P), Recall (R) and F easure (F ) of our ehods based on cos-sensve conex-aware MI-SC and ul-perspecve cos-sensve conex-aware MI-J-SC, and also he values for he copeng ehods based on -Graph, MI-kernel, MI-SVM, Caon-K, EM-DD, SVM, and K. In order o valdae he effecveness of he audo cos n he ehod based on cos-sensve conex-aware MI-SC, he audo feaures were no ncluded n he feaure vecor n each nsance when esng he ehod based on cos-sensve conex-aware MI-SC. We also copared he ehod based on cos-sensve conex-aware MI-SC wh he ehod based on he pure conex-aware MI-SC obaned by reovng he audo cos fro hs ehod,.e., he dagonal arx D was fxed as D=dag(,,..., ). I s unfar o copare he ehod based on he pure conex-aware MI-SC wh he 0

21 copeng ehods n whch he audo feaures are used. Therefore, we reoved he audo feaures fro he feaure vecors and copared he ehod based on he pure conex-aware MI-SC wh he -Graph-based ehod whou audo feaures. Fro Table, he followng pons were revealed: Table. The experenal resuls on he horror vdeo daase (%) Algorhs Precson (P) Recall (R) F easure Mul-perspecve cos-sensve MI-J-SC 85.56(±0.5) 85.(±0.39) 85.38(±0.3) Cos-sensve conex-aware MI-SC 8.6(±0.7) 83.38(±0.87) 8.46(±0.9) Pure conex-aware MI-SC 80.0(±.08) 8.00(±0.76) 80.98(±0.53) Graph wh audo feaures 8.87(±.95) 8.4(±.5) 8.4(±.0) Graph whou audo feaures 80.0(±.59) 80.8(±0.9) 80.40(±.06) MI-kernel 80.70(±.4) 8.43(±0.9) 8.05(±0.5) MI-SVM Caon-K EM-DD SVM k Our ehod based on ul-perspecve cos-sensve MI-J-SC s uch ore accurae han all he oher ehods. Ths shows ha horror vdeo recognon benefs fro ul-perspecves. The lower sandard devaons ply ha our ehod s sable. The ehod based on he cos-sensve conex-aware MI-SC has a hgher ean F value and a uch lower sandard devaon han he ehod based on he pure conex-aware MI-SC. Ths ndcaes ha he vsual-audo conex s useful for horror vdeo recognon and he ehod based on he cos-sensve MI-SC effecvely fuses he vsual and audo feaures. Our ehod based on cos-sensve conex-aware MI-SC, our ehod based on ul-perspecve cos-sensve conex-aware MI-J-SC, he ehod based on he pure conex-aware MI-SC and he ehod based on he -Graph ehod, all of whch odel conexual cues aong nsances n a bag, ouperfor oher MIL-based ehods n whch he nsances are reaed ndependenly. Ths shows ha he conexual relaons beween nsances are useful for horror vdeo recognon. The resuls of he -Graph-based MIL ehod, n whch SVMs raher han sparse codng are used, are repored. I s seen ha our ehod based on cos-sensve conex-aware MI-SC yelds ore accurae resuls han he -Graph-based ehod wh audo feaures. Alhough he ehod based on he pure conex-aware MI-SC yelds less accurae resuls han he -Graph-based ehod wh audo feaures, yelds ore accurae resuls han he -Graph-based ehod whou audo

22 feaures. I s apparen ha he sparse codng-based MIL ehods ouperfor he SVM-based MIL ehod for horror vdeo recognon. The wo non-mil-based ehods, he K-based ehod and he pure SVM-based ehod, overall yeld less accurae resuls han he MIL-based ehods. Ths s because hey use holsc feaures n vdeos. If a horror vdeo conans only a sall nuber of horror fraes, hen he holsc feaures nevably weaken he feaures obaned fro he horror fraes. The pure SVM-based ehod ouperfors he K-based ehod, because SVM consders experenal rsk and srucural rsk. Furherore, he ranng free characersc of he sparse codng classfers akes feasble o exend our ehods based on cos-sensve conex-aware MI-SC and ul-perspecve cos-sensve conex-aware MI-J-SC o onlne classfers ha are necessary for nework vdeo analyss applcaons. The copuaonal effcency of he proposed odel s ensured by he effcen opzaon ehods for obanng he sparsy coeffcen vecor. The feaure sgn search (FSS) algorh n he cos-sensve conex-aware MI-SC produces a sgnfcan speedup for sparse codng. The APG algorh n he ul-perspecve cos-sensve MI-J-SC s a fas algorh for solvng he, nor-regularzed opzaon. Table copares he runes of he proposed ehods and oher represenave ehods on he horror vdeo daase esed on a copuer wh Inel(R) Core(TM) Quad CPU. I s seen ha he es speed of our sparse codng-based ehods s coparable o oher represenave ehods, no akng no accoun he fac ha our ehods have no ranng e. Table. Rune n seconds per vdeo for dfferen ehods Mul-perspecve cos-sensve MI-J-SC Cos-sensve conex-aware MI-SC Tranng Tes Graph kernel SVM Caon-k In he experens we fused dfferen feaures o show her dfferen conrbuons. Seven dfferen cobnaons of he vsual feaures (VF), he audo feaures (AF), and he color eoon feaures (EF) were obaned. Table 3 shows he precson, recall, and F easure for ul-perspecve MI-J-SC, -Graph, MI-SVM, SVM, and k usng hese seven feaure cobnaons on he horror vdeo daase. I s seen ha he bes one aong hree ypes of feaures s he audo feaure, whch has he hghes F easure. Generally,

23 he cobnaon of he vsual feaures, he audo feaures, and he color eoonal feaures can prove he recognon accuracy, whch shows he copleenary characerscs of he hree ypes of feaures. Table 3. The resuls for dfferen feaure cobnaons (%) Mulperspecve MI-J-SC -Graph MI-SVM SVM k VF EF AF VF+EF VF+AF EF+AF VF+EF+AF Precson Recall F easure Precson Recall F easure Precson Recall F easure Precson Recall F easure Precson Recall F easure Volen vdeo recognon Table 4. The experenal resuls on he volen vdeo daase (%) Algorhs Precson (P) Recall (R) F easure Mul-perspecve cos-sensve MI-J-SC 86.57(±0.48) 87.87(±0.87) 87.(±0.53) Cos-sensve conex-aware MI-SC 86.3(±0.98) 85.95(±0.95) 86.04(±0.87) -Graph 85.95(±.8) 85.8(±.69) 85.85(±.5) MI-kernel 84.77(±3.37) 84.99(±.84) 84.79(±.5) MI-SVM Caon-K EM-DD SVM k We downloaded volen and non-volen oves fro he nerne. Ths daase consss of 400 volen vdeos and 400 non-volen vdeos. Half of he volen vdeos and half of he non-volen vdeos were used for ranng, and he reanng vdeos were used for esng. The average accuraces of en es 0-fold cross valdaon were used as he fnal perforances for each ehod. Table 4 shows he recognon resuls of our ehods based on cos-sensve conex-aware MI-SC and ul-perspecve cos-sensve MI-J-SC, and he copeng ehods based on -Graph, MI-kernel, MI-SVM, Caon-K, EM-DD, SVM, and K. All he ehods use he sae feaures ncludng color eoonal feaures, vsual feaures, audo feaures, and oon feaures whch are all nroduced n Secon 5. I s seen ha our ehods yeld ore accurae resuls han he copeng ehods, and our ul-perspecve cos-sensve conex-aware MI-J-SC ehod yelds 3

24 ore accurae resuls han our cos-sensve conex-aware MI-SC ehod. The resuls have he sae characerscs as on he horror daase. We also esed perforance of volen vdeo recognon ehods on he VSD (volen scene deecon) 04 daase [6], whch bencharks volence deecon n Hollywood oves a he MedaEval bencharkng nave for uleda evaluaon. The ranng se n he daase has 4 Hollywood oves and conans bnary annoaons of all he volen scenes, where a scene was denfed by s sar and end fraes. A se of 7 Hollywood oves was used for esng. All he es volen segens were annoaed a vdeo frae level,.e., a volen segen was defned by s sarng and endng frae nubers. We segened he es vdeos no scenes and labeled he scenes as volen or non-volen usng he vdeos annoaons a he frae level. Table 5 copares he resuls of our ehods for deecng volen scenes wh he sae-of-he-ar resuls on he daase. I s seen ha he resuls of our ehods are beer han he sa-of-he-ar resuls. The effecveness of he exraced feaures and he MI-SC-based classfcaon n our ehods s clearly shown. Table 5. Coparson beween he resuls (%) of our ehods and he sae-of-he-ar resuls on he Hollywood ove es se and he YouTube ove es se, respecvely Tes subse Mehod Precson Recall F easure Mul-perspecve MI-J-SC Hollywood oves Conex-aware MI-SC FUDA [63] RECOD [6] VIVOLAB [58] MIL daases Alhough we focused our ul-perspecve conex-aware MI-J-SC ehod on applcaons o sensve vdeo recognon, our ehod can be used n oher applcaons. To verfy he generaly of our ul-perspecve conex-aware MI-J-SC ehod, we esed on he general daases whch were wdely used o evaluae he perforance of MIL ehods. They nclude fve benchark daases: Musk, Musk, Elephan, Fox, and Tger [5, 9]. The Musk and Musk daases are usk olecule daases. Each olecule whch corresponds o a bag has several shape srucures whch correspond o nsances. Each srucure was represened by a 66 densonal vecor. The Musk daase conans 47 posve and 45 negave bags. The Musk daase conans 39 posve and 63 negave bags. The Elephan, Fox and Tger daases are age daases. Each age whch corresponds o a bag was segened no several age paches whch correspond o nsances. A 30 densonal vecor was exraced fro each pach. Each of hese hree daases conans 4

25 00 posve and 00 negave bags. Table 6. Accuracy (%) on he MIL benchark daases Algorh Musk Musk Elephan Fox Tger Mul-perspecve MI-J-SC 9.(±.8) 90.6(±.3) 88.5(±.) 6.7(±.8) 86.8(±.) -Graph 88.9(±3.3) 90.3(±.6) 86.8(±0.7) 6.6(±.8) 86.0(±.6) MI-Graph 90.0(±3.8) 90.0(±.7) 85.(±.8) 6.(±.7) 8.9(±.5) MI-Kernel 88.0(±3.) 89.3(±.5) 84.3(±.6) 60.3(±.9) 84.(±.0) MI-SVM SVM Mss-SVM /A /A /A PP-MM DD /A /A /A EM-DD We copared our ul-perspecve conex-aware MI-J-SC ehod wh he ehods based on -Graph, MI-Graph, MI-Kernel, MI-SVM, -SVM [9], Mss-SVM [4], PP-MM kernel [4], he dverse densy (DD) [5], and EM-DD [7]. For all he ehods he sae feaures fro he benchark daases were used. The perforance of each ehod was evaluaed usng he accuracy whch s he proporon of he saples whch are correcly classfed. Our ul-perspecve conex-aware MI-J-SC ehod and he ehods based on -Graph, MI-Graph, and MI-Kernel were run by us. The 0-fold cross valdaons for en es were carred ou o yeld he average accuraces and sandard devaons. The resuls of he copeng ehods based on MI-SVM and -SVM [9], Mss-SVM [4], PP-MM kernel [4], DD [5], and EM-DD [7] were drecly aken fro [9]. All he resuls are shown n Table 6. I s seen ha our ul-perspecve MI-J-SC ehod acheves beer perforances han he ehods based on MI-Graph and -Graph on he Musk, Elephan and Fox daases. The perforances of he ehods based on ul-perspecve MI-J-SC, MI-Graph, -Graph, and MI-Kernel on he Musk and Tger daases are coparable. More poranly, our ul-perspecve MI-J-SC ehod yelds lower sandard devaons on all he benchark daases. Ths shows he sably of our ul-perspecve conex-aware MI-J-SC ehod. 7. Concluson In hs paper, we have proposed a cos-sensve conex-aware MI-SC ehod n whch a graph kernel has been used o odel he conexs aong fraes and cos-sensve sparse codng has been used o odel he conexs beween vsual cues and audo cues. We have also proposed a ul-perspecve MI-SC ehod whch can effecvely fuse nforaon fro he conexual perspecve, he ndependen nsance perspecve, and he holsc perspecve. Based on he color eoon and color harony heores, we have exraced each 5

26 vdeo s color eoonal feaures whch are hgher level feaures n conras wh he low-level color and vsual feaures. These color eoonal feaures ogeher wh he cos-sensve conex-aware MI-SC ehod and he ul-perspecve MI-J-SC ehod have been appled o recognze volen and horror vdeos. Experenal resuls have shown ha he exraced eoonal feaures are effecve for recognzng volen and horror vdeos. I has been shown ha our ehods no only are superor o radonal MIL-based ehods and radonal SVM and K-based ehods on he volen and horror vdeo daases bu also ay be effecve n oher general ul-nsance probles as esed on he general MIL daases. Alhough hs paper focuses on he recognon of volen and horror vdeos, our cos-sensve conex-aware MI-SC ehod and our ul-perspecve MI-J-SC ehod are avalable for recognzng oher ypes of web vdeos. References. L.C. Ou and M.R. Luo, A colour harony odel for wo-colour cobnaons, Color Research & Applcaon, vol. 3, no. 3, pp. 9-04, F. e, H. Huang, X. Ca, and C. Dng, Effcen and robus feaure selecon va jon l,-nors nzaon, n Proc. of Advances n eural Inforaon Processng Syses, pp. 83-8, J. Lu, S. J, and J. Ye, Mul-ask feaure learnng va effcen l,-nor nzaon, n Proc. of Conference on Uncerany n Arfcal Inellgence, pp , J. Wnn, A. Crns, and T. Mnka, Objec caegorzaon by learned unversal vsual dconary, n Proc. of IEEE Inernaonal Conference on Copuer Vson, vol., pp , H.L. Wang and L. Cheong, Affecve undersandng n fl, IEEE Trans. on Crcus and Syses for Vdeo Technology, vol. 6, no. 6, pp , A. Hanjalc and L.Q. Xu, Affecve vdeo conen represenaon and odelng, IEEE Trans. on Muleda, vol. 7, no., pp.43-54, Z. Rasheed, Y. Shekh, and M. Shah, On he use of copuable feaures for fl classfcaon, IEEE Trans. on Crcus and Syses for Vdeo Technology, vol. 5, no., pp. 5-64, H.B. Kang, Affecve conen deecon usng HMMs, n Proc. of ACM nernaonal conference on Muleda, pp. 59-6, G. Tzaneaks and P. Cook, Muscal genre classfcaon of audo sgnals, IEEE Trans. on speech and audo processng, vol. 0, no. 5, pp , S. Zhu and K.-K. Ma, A new daond search algorh for fas block-achng oon esaon, IEEE Trans. on Iage Processng, vol. 9, no., pp , Feb Y. Lu, X. Wang, Y. Zhang, and S. Tang, Fusng audo-words wh vsual feaures for pornographc vdeo deecon, n Proc. of IEEE Inernaonal Conference on Trus, Secury and Prvacy n Copung and Councaons, pp , 0.. C. Jansohn, A. Ulges, and T.M. Breuel, Deecng pornographc vdeo conen by cobnng age feaures wh oon nforaon, n Proc. of ACM nernaonal conference on Muleda, Bejng, pp , B. Wu, X. Jang, T. Sun, S. Zhang, X. Chu, C. Shen, and J. Fan. A novel horror scene deecon schee on revsed ulple nsance learnng odel, n Proc. of Inernaonal Conference on Advances n Muleda Modelng, pp , M. Xu, L.-T. Cha, and J. Jn, Affecve conen analyss n coedy and horror vdeos by audo eoonal even deecon, n Proc. of IEEE Inernaonal Conference on Muleda and Expo, pp. 6-65, July

27 5. T.G. Deerch, R.H. Lahrop, and T. Lozano-Perez, Solvng he ulple-nsance proble wh axs-parallel recangles, Arfcal Inellgence, vol. 89, no. -, pp. 3-7, Y. Chen, J. B, and J.Z. Wang, MILES: Mulple-nsance learnng va ebedded nsance selecon, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 8, pp , Y. Chen and J.Z. Wang, Iage caegorzaon by learnng and reasonng wh regons, Journal of Machne Learnng Research, vol. 5, pp , S. Lee, W. Sh, and S. K, Herarchcal syse for objeconable vdeo deecon, IEEE Trans. on Consuer Elecroncs, vol. 55, no., pp , May S. Andrews, I. Tsochanards, and T. Hofann, Suppor vecor achnes for ulple nsance learnng, n Proc. of Advances n eural Inforaon Processng Syses, pp , Cabrdge, MIT Press, J. Wrgh, A. Yang, A. Ganesh, S. Sasry, and Y. Ma, Robus face recognon va sparse represenaon, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 3, no., pp. 0-7, Feb T. Endeshaw, J. Garca, and A. Jakobsson, Classfcaon of ndecen vdeos by low coplexy repeve oon deecon, n Proc. of IEEE Appled Iagery Paern Recognon Workshop, Washngon DC, pp. -7, Oc A. Daa, M. Shah, and.d.v. Lobo, Person-on person volence deecon n vdeo daa, n Proc. of Inernaonal Conference on Paern Recognon, pp , W.-H. Cheng, W.-T. Chu, and J.-L. Wu, Seanc conex deecon based on herarchcal audo odels, n Proc. of ACM SIGMM Inernaonal Workshop on Muleda Inforaon Rereval, pp. 09-5, T. Gannakopoulos, D. Kosopoulos, A. Arsdou, and S. Theodords, Volence conen classfcaon usng audo feaures, Advances n Arfcal Inellgence, Lecure oes n Copuer Scence, vol. 3955, pp , O. Maron and T. Lozano-P érez, A fraework for ulple-nsance learnng, n Proc. of Advances n eural Inforaon Processng Syses, pp Cabrdge, MIT Press, J. Wang and J.-D. Zucker, Solvng he ul-nsance proble: a lazy learnng approach, n Proc. of Inernaonal Conference on Machne Learnng, pp. 9-5, Q. Zhang and S.A. Goldan, EM-DD: an proved ul-nsance learnng echnque, n Proc. of Advances n eural Inforaon Processng Syses, Cabrdge, MIT Press, pp , T. Garner, P.A. Flach, A. Kowalczyk, and A.J. Sola, Mul-nsance kernels, n Proc. of Inernaonal Conference on Machne Learnng, pp , Z.-H. Zhou, Y.-Y. Sun, and Y.-F. L, Mul-nsance learnng by reang nsances as non-i.i.d. saples, n Proc. of Inernaonal Conference on Machne Learnng, pp , H. Lee, A. Bale, R. Rana, and Y.. Andrew, Effcen sparse codng algorhs, n Proc. of Advances n eural Inforaon Processng Syses, pp , T. Gannakopoulos, A. Pkraks, and S. Theodords, A ul-class audo classfcaon ehod wh respec o volen conen n oves usng Bayesan neworks, n Proc. of IEEE Workshop on Muleda Sgnal Processng, pp , Oc J. a, M. Alghoney, and A.H. Tewfk, Audo-vsual conen-based volen scene characerzaon, n Proc. of IEEE Inernaonal Conference on Iage Processng, pp , J.B. Tenenbau, V. de Slva, and J.C. Langford, A global geoerc fraework for nonlnear densonaly reducon, Scence, vol. 90, pp , X.T. Yuan and S. Yan, Vsual classfcaon wh ul-ask jon sparse represenaon, n Proc. of IEEE Conference on Copuer Vson and Paern Recognon, pp , June A.F. Seaon, B. Lehane,.E. O Connor, C. Brady, and G. Crag, Auoacally selecng shos for acon ove ralers, n Proc. of ACM Inernaonal Workshop on Muleda Inforaon Rereval, pp. 3-38, P. Tseng, On acceleraed proxal graden ehods for convex-concave opzaon, SIAM Journal of Opzaon, May 008. hp:// 37. Z. Cernekova, I. Pas, and C. kou, Inforaon heory-based sho cu/fade deecon and vdeo suarzaon, IEEE 7

28 Trans. on crcus and syses for vdeo echnology, vol.6, no., pp. 8-9, L.C. Ou, M.R. Luo, A. Woodcock, and A. Wrgh, A sudy of colour eoon and colour preference. par I: colour eoons for sngle colours, Color Research & Applcaon, vol. 9, no. 3, pp. 3-40, L.C. Ou, M.R. Luo, A. Woodcock, and A. Wrgh, A sudy of colour eoon and colour preference. par III: colour preference odelng, Color Research & Applcaon, vol. 9, no. 5, pp , J.M. Geusebroek and A.W.M. Seulders, A sx-sulus heory for sochasc exure, Inernaonal Journal of Copuer Vson, vol. 6, no., pp. 7-6, Z.-H. Zhou and J.-M. Xu, On he relaon beween ul-nsance learnng and se-supervsed learnng, n Proc. of Inernaonal Conference on Machne Learnng, pp , H.Y. Wang, Q. Yang, and H. Zha, Adapve p-poseror xure odel kernels for ulple nsance learnng, n Proc. of Inernaonal Conference on Machne Learnng, pp , T. Gannakopoulos, A. Makrs, D. Kosopoulos, S. Peranons, and S. Theodords, Audo-vsual fuson for deecng volen scenes n vdeos, Arfcal Inellgence: Theores, Models and Applcaons, Lecure oes n Copuer Scence, vol. 6040, pp. 9-00, K. Wang, Z. Zhang, and L. Wang, Volence vdeo deecon by dscrnave slow feaure analyss, n Proc. of Chnese Conference on Paern Recognon, pp , Sep J. Ln and W. Wang, Weakly-supervsed volence deecon n oves wh audo and vdeo based co-ranng, Advances n Muleda Inforaon Processng, Lecure oes n Copuer Scence, vol. 5879, pp , A. Feld and J. Lawson, Fear nforaon and he developen of fears durng chldhood: effecs on plc fear responses and behavoural avodance, Behavour Research and Therapy, vol. 4, no., pp , ov J. Kng, G. Eleonora, and T.H. Ollendck, Eology of chldhood phobas: curren saus of Rachan's hree pahways heory, Behavour Research and Therapy, vol. 36, no. 3, pp , B. Babenko, M.-H. Yang, and S. Belonge, Vsual rackng wh onlne ulple nsance learnng, n Proc. of IEEE Conference on Copuer Vson and Paern Recognon, pp , M. L, J. Kwok, and B.L. Lu, Onlne ulple nsance learnng wh no regre, n Proc. of IEEE Conference on Copuer Vson and Paern Recognon, pp , E. Acar, F. Hopfgarner, and S. Albayrak, Deecng volen conen n Hollywood oves by d-level audo represenaons, n Proc. of Inernaonal Workshop on Conen-Based Muleda Indexng, pp , June X. Dng, B. L, W. Hu, W. Xong, and Z. Wang, Horror vdeo scene recognon based on ul-vew ul-nsance learnng, n Proc. of Asan Conference on Copuer Vson, pp , H.-D. K, S.-S. Ahn, K.-H. K, and J.-S. Cho, Sngle-channel parcular voce acvy deecon for onorng he volence suaons, n Proc. of IEEE Inernaonal Syposu on Robo and Huan Ineracve Councaon, Korea, pp. 4-47, Aug L. Xu, C. Gong, J. Yang, Q. Wu, and L. Yao, Volen vdeo deecon based on MoSIFT feaure and sparse codng, n Proc. of IEEE Inernaonal Conference on Acousc, Speech and Sgnal Processng, pp , P.Y. Lee, S.C. Hu, and A.C.M. Fong, An nellgen caegorzaon engne for blngual web conen flerng, IEEE Trans. on Muleda, vol. 7, no. 6, pp , M. Soleyan, M. Larson, T. Pun, and A. Hanjalc, Corpus developen for affecve vdeo ndexng, IEEE Trans. on Muleda, vol. 6, no. 4, pp , C. Kang, S. Xang, S. Lao, C. Xu, and C. Pan, Learnng conssen feaure represenaon for cross-odal uleda rereval, IEEE Trans. on Muleda, vol. 7, no. 3, pp , J. Geng, Z. Mao, and X. Zhang, Effcen heursc ehods for ulodal fuson and concep fuson n vdeo concep deecon, IEEE Trans. on Muleda, vol. 7, no. 4, pp , D. Casan, M. Rodrguez, A. Orega, C. Orre, and E. Lleda, VVoLab and CVLab-MedaEval 04: volen scenes deecon affec ask, n Workng oes Proc. MedaEval 04 Workshop, Barcelona, Caalunya, Span, Oc

29 59. C. Tekn and M. van der Schaar, Conexual onlne learnng for uleda conen aggregaon, IEEE Trans. on Muleda, vol.7, no. 4, pp , Z. Ma, Y. Yang,. Sebe, and K. Zheng, and A.G. Haupann, Muleda even deecon usng a classfer-specfc neredae represenaon, IEEE Trans. on Muleda, vol. 5, no. 7, pp , S. Avla, D. Morera, M. Perez, D. Moraes, I. Coa, V. Teson, E. Valle, S. Goldensen, and A. Rocha, RECOD a MedaEval 04: volen scenes deecon ask, n Workng oes Proc. MedaEval 04 Workshop, Barcelona, Caalunya, Span, Oc M. Schedl, M. Sjobergy, I. Mroncaz, B. Ionescuz, and V.L. Quangx, Y.-G. Jang, and C.-H. Dearyk, VSD04: a daase for volen scenes deecon n Hollywood oves and web vdeos, n Proc. of Inernaonal Workshop on Conen-Based Muleda Indexng, pp. -6, June Q. Da, Z. Wu, Y.-G. Jang, X. Xue, and J. Tang, Fudan-JUST a MedaEval 04: volen scenes deecon usng deep neural neworks, n Workng oes Proc. MedaEval 04 Workshop, Barcelona, Caalunya, Span, Oc. 04. Acknowledgen Ths work s parly suppored by he 973 basc research progra of Chna (Gran o. 04CB349303), he aural Scence Foundaon of Chna (Gran o. 6474, ), he Projec Suppored by CAS Cener for Excellence n Bran Scence and Inellgence Technology, and he Projec Suppored by Guangdong aural Scence Foundaon (Gran o. S000008) Weng Hu receved he Ph.D. degree fro he deparen of copuer scence and engneerng, Zhejang Unversy n 998. Fro Aprl 998 o March 000, he was a posdocoral research fellow wh he Insue of Copuer Scence and Technology, Pekng Unversy. ow he s a professor n he Insue of Auoaon, Chnese Acadey of Scences. Hs research neress are n vsual oon analyss, recognon of web objeconable nforaon, and nework nruson deecon. Xnao Dng receved he M.S. degree n elecronc engneerng fro Dalan Mare Unversy, Dalan, Chna, n 004, and he Ph. D. degree fro he School of Mechancal Elecronc and Inforaon Engneerng, Chna Unversy of Mnng and Technology, Bejng, Chna, n 03. She s currenly a vsng scholar n he Insue of Auoaon, Chnese Acadey of Scences. Her an research neress nclude Iage and vdeo analyss and undersandng, achne learnng and nerne secury. Bng L receved he PhD degree fro he Deparen of Copuer Scence and Engneerng, Bejng Jaoong Unversy, Chna, n 009. Currenly, he s an assocae professor n he aonal Laboraory of Paern Recognon (LPR), Insue of Auoaon, Chnese Acadey of Scences. Hs research neress nclude color consancy, vsual salency and web conen nng. Janchao Wang go he bachelor degree fro he Unversy of Scence and Technology of Chna n 008 and he aser degree fro he aonal Laboraory of Paern Recognon n 0. He s a an age processng engneer n he copany of euan. Hs research neress nclude age processng and copuer vson. Yan Gao receved he B.S. degree n elecrcal engneerng fro he orh Unversy of Chna, Tayuan, n 03. He s currenly pursung he M.S. degree n he Cvl Avaon Unversy of Chna, Tanjn. Hs research neress nclude objec recognon, age deecon, and age rereval. Fangsh Wang s a professor wh Sofware Engneerng School n Bejng Jaoong Unversy. She receved he PhD degree fro he School of Copuer Scence and Engneerng, Bejng Jaoong Unversy, Chna, n 007. Her research neress focus on Copuer vson, Vdeo analyss and seanc ag. Sephen Maybank receved a BA n Maheacs fro Kng's college Cabrdge n 976 and a PhD n copuer scence fro Brkbeck college, Unversy of London n 988. ow he s a professor n he School of Copuer Scence and Inforaon Syses, Brkbeck College. Hs research neress nclude he geoery of ulple ages, caera calbraon, vsual survellance ec. 9