Audio-based Classification of Video Genre Using Multivariate Adaptive Regression Splines

Transcription

1 Audo-basd Classfcato of Vdo Gr Usg ultvarat Adaptv Rgrsso Spls H E Latt, u War Abstract A larg umbr of rsarchrs ar attractd by vdo gr classfcato, vdo cotts rtrval ad smatcs rsarch vdo procssg ad aalyss doma. ay rsarchrs try to propos structur or framworks to classfy th vdo gr that s tgratg may algorthms usg low ad hgh lvl faturs. Faturs grally clud both usful ad uslss formato that ar dffcult to sparat. I ths papr, vdo gr classfcato s proposd by usg oly th audo chal. A dcomposto modl s basd o multvarat adaptv rgrsso spls to sparat usful ad uslss compots ad prform th gr dtfcato s prformd o ths low-lvl acoustc faturs such as FCC ad tmbral txtual faturs. Exprmts ar coductd o a corpus composd from cartoos, sports, ws, dahmas ad muscs o whch obta ovrall classfcato rat of 9.8%. Idx Trms ultvarat Adaptv Rgrsso Spls, l Frqucy Cpstral Coffcts, Factor Aalyss I. ITRODUCTIO Today, ffct tools ar d for usrs to crawl th larg collcto bcaus th avalabl vdo amout has largd sgfcatly o th Itrt. Whl most of th rsarch o vdo classfcato has th tt of classfyg a tr vdo, som authors hav focusd o classfyg sgmts of vdo such as dtfyg volt [] or scary [] scs a mov or dstgushg btw dffrt ws sgmts wth a tr ws broadcast []. From ths raso, may works ar forcd o structurg audovsual databass by cott aalyss, basd o txt-basd catgorzato []. For th purpos of vdo classfcato, faturs ar draw from thr modalts: txt, audo, ad vsual. ost of th proposd approachs rly o mag aalyss. I [], may works ar motvatd by th crtcal d of ffct tools for structurg audovsual databass ths last yars. I [], thy vstgat hghr lvl aalyss lk trackg of audovsual vts. Audo-basd approachs wr xplord by automatc trascrpto of spch cotts, or by low lvl audo stram aalyss. Howvr, ths systms grally hav poor prformacs o uxpctd lgustc domas ad advrs acoustc codtos. Acoustc-spac charactrzato s prstd by usg statstc classfr lk gaussa mxtur modl (G), ural ts or support vctor machs (SV) o cpstral doma faturs [,, 7]. Varous kds of acoustc faturs hav b valuatd th fld of vdo gr dtfcato. I [8, 7, ], tm-doma audo faturs ar proposd lk zro crossg rats or rgy dstrbutos. Thrfor, low-lvl approachs prst a bttr robustss to th hghly varabl ad uxpctd codtos that may b coutrd o vdos. I th cpstral doma, o of th ma dffcults gr dtfcato s du to th dvrsty of th acoustc pattrs that may b producd by ach vdo gr. I ths papr, ths problm s am to addrss th fld of dtfyg vdo grs by applyg multvarat adaptv rgrsso spls. Vdo gr classfcato framwork s focusd o by usg a audo-oly mthod. I th xt scto a ovrvw of th prstd systm s provdd frst. Th archtctur of th systm ad th basc udrlyg cocpts ar xplad. Scodly, th multvarat adaptv rgrsso spls algorthm s dscrbd. Fally, th xprmtal rsults ar also show. l_frq_cpstral _Coffct II. SYSTE ARCHITECTURE Vdo Fls Extract Audo Fls Fatur Extracto ZroCrossgRat ShortTmErgy SpctralFlux SpctralCtrod Spctralrolloff os_fram_rato Slc_Rato auscrpt rcvd ay, 0. H E Latt s wth th Faculty of Iformato ad Commucato Tchology, Uvrsty of Tchology (Yataarpo Cybr Cty), Py Oo Lw, yamar Dr. u War was wth th Faculty of Iformato ad Commucato Tchology, Uvrsty of Tchology (Yataarpo Cybr Cty), Py Oo Lw, yamar. ws Cartoos Sports uscs Fg.. Ovrvw of systm archtctur Dahmas Th ovrall procdur to xtract audo fl from a vdo clp has show fgur.. Frstly, bas audo faturs ar 80

2 xtractd from th audo sgal. Th FCC, zro crossg rat, short tm rgy, spctral flux, spctral ctrod, spctral rolloff, os fram rato ad slc rato ar usd as th bas audo faturs ths papr. Ad th, multvarat adaptv rgrsso spls dvlop th modl for ach gr typs by usg th bas audo faturs st. Th xt stp s a ffct mchasm for classfyg gr th databas ad masurg thr prformac. Th dtals of th proposd audo fgrprt ar xplad fgur. III. FEATURE EXTRACTIO ay of th audo-basd faturs ar chos to approxmat th huma prcpto of soud. I ths fram work uss low-lvl acoustc faturs that ar both tm-doma faturs ad frqucy-doma faturs. Th tmbral txtual faturs ar calculatd from th gv audo sgal. Tmbral txtual faturs ar thos usd to dffrtat mxtur of souds basd o thr strumtal compostos wh th mlody ad th ptch compots ar smlar. Th us of tmbral txtural faturs orgats from spch rcogto. Extractg tmbral faturs rqur prprocssg of th soud sgals. Th sgals ar dvdd to statstcally statoary frams, usually by applyg a wdow fucto at fxd trvals. Th applcato of a wdow fucto rmovs th so-calld dg ffcts. Popular wdow fuctos cludg th Hammg wdow fucto. Short-Trm Fourr Trasform Faturs: Ths s a st of faturs rlatd to tmbral txturs ad s also capturd usg FCC. It cossts of Spctral Ctrod, Spctral Rolloff, Spctral Flux ad Low Ergy, Zro Crossgs ad th computs th ma for all fv ad th varac for all but zro crossgs. So, thr ar a total of faturs. I th tm-doma, Zro crossg rat (ZCR) s th umbr of sgal ampltud sg chags th currt fram. Hghr frqucs rsult hghr zro crossg rats. Spch ormally has a hghr varablty of th ZCR tha musc. If th loudss ad ZCR ar both blow thrsholds, th ths fram may rprst slc. Th slc rato s th proporto of a fram wth ampltud valus blow som thrshold. Spch ormally has a hghr slc rato tha musc. ws has a hghr slc rato tha commrcals. I th frqucy-doma, th rgy dstrbuto (short tm rgy) s th sgal dstrbuto across frqucy compots. Th frqucy ctrod, whch approxmats brghtss, s th mdpot of th spctral rgy dstrbuto ad provds a masur of whr th frqucy compots ar coctratd. ormally brghtss s hghr musc tha spch, whos frqucy s ormally blow 7 khz. Badwdth s a masur of th frqucy rag of a sgal. Som typs of souds hav mor arrow frqucy rags tha othrs. Spch typcally has a lowr badwdth tha musc. Th fudamtal frqucy s th lowst frqucy a sampl ad approxmats ptch, whch s a subjctv masur. l-frqucy cpstral coffcts (FCC) ar producd by takg th logarthm of th spctral compots ad th placg thm to bs basd upo th l frqucy scal, whch s prcpto-basd. IV. ULTIVARIATE ADAPTIVE REGRESSIO SPLIES Aalyss wr prformd usg multvarat adaptv rgrsso spls, a tchqu that uss pc-ws lar sgmts to dscrb o-lar rlatoshps btw audo faturs ad vdo gr. Th thory of multvarat adaptv rgrsso spls (ARS) was dvlopd by Jrom Frdma [9] 99. Lt z b th dpdt rspos, whch ca b cotuous or bary, ad lt Y = (Y,...,Y ) D () b th st of pottal prdctv covarats. Th th systm assum that th data ar gratd from a ukow tru modl. I cas of a cotuous rspos ths would b z f Y, Y,..., ) ( Y () Th dstrbuto of th rror s mmbr of th xpotal famly []. f s approxmatd by applyg fuctos, whch clud tractos of at most scod ordr. That mas that us th modl f ( Y) g g ( Y ) g ( Y, Y 0 j j j, j j j) j j j () whras wth rror varac. Lar spls ad thr tsor products ar usd to modl th fucto g(.). A o-dmsoal spl ca b wrtt as g( y) b K b0 y bk ( y t k ) k () ad th kot tk th rag of th obsrvd valus of Y. For ths raso th fucto g s stuatd a lar spac wth th K + bass fuctos. Thus th followg modl rsults: g0 0 g, g j ) B ( Y ) j j j j j, j, Yj ) B, Yj ) () bcaus th tracto gj,j s modld by mas of tsor product spls as g( y, y) g( y) g( y) (7) Th rprst th umbr of bass fuctos th modl ad th Bs rprst spl bass fuctos as dscrbd abov ad th βs ar coffcts. I ths approach th coffcts ar stmatd by usg th Last Squars mthod. ow th coffct matrx ca b wrtt as ˆ * * T * T ( Y Y ) Y Z. (8) Y s rprstd as th dsg matrx of th slctd bass fuctos, ad Z rprsts th rspos vctor. Istad of Yj, ARS uss a collcto of w prdctors th form of pcws lar bass fuctos ar as { t),( t Yj) }, j,...,, t { y j,.., yj} Aftr that, th gralzd cross-valdato crtro s usd to masur th dgr of ft or lack of accuracy of th modl : 80

3 GCV ( ) [ z fˆ d. [ ] ( y )] (9) whras dots th fttd valus of th currt ARS modl ad d dots th palzg paramtr. Th umrator s th commo rsdual sum of squars, whch s palzd by th domator, whch accouts for th crasg varac th cas of crasg modl complxty. A smallr d grats a largr modl wth mor bass fuctos, a largr d crats a smallr modl wth lss bass fuctos. Accordg to th tabl., Forward Procss Stag s that th stpws addto procss bass fuctos ar addd utl th maxmal allowd modl sz s rachd. Th largst modl grally ovrfts th data. Th Backward prug Stag -th stpws dlto procss- s that all ucssary bass fuctos ar rmovd aga utl a fal modl s obtad whch s bst cosdrg th GCV that s th o wth th mmum GCV. I th frst stp of th addto procss a costat modl s fttd. Subsqutly th umbr of caddat bass fuctos dpds o th umbr of possbl kots pr prdctor varabl. To kp th procdur fast, th rsults robust th umbr of possbl kots pr prdctor ad also th possbl caddats pr stp ar lmtd. To dtrm th umbr of pottal kots of a spcfc covarat a ordr statstc s computd ad a subst of t s th chos as pottal kots. Commoly ths ar about 0 kots pr prdctor, at most vry thrd valu s chos yt. I th frst trato aftr th ft of th costat modl a lar bass fucto o o of th prdctor varabls s fttd. Th scod trato s accoutd for both lar bass fuctos o aothr covarat ad bass fuctos wth kots of th covarat alrady th modl. Th modl to choos vry stp durg th forward procss s th o out of all possbl modls whch mmzs th GCV. I th backward procss o bass fucto s dltd pr stp ad th GCV s computd for th rducd modl. Th modl whch ylds th smallst cras of GCV bcoms th w o. V. EXPERIETAL RESULTS Th trag data st cluds th formato about a tst rlatd wth ws,cartoo, sport, musc, dama vdo cludg 80 obsrvatos. Amog of thm, 0,, 7,, ad 9 clps ar ws, Dahma, Cartoo, Sport ad usc rspctvly. Th data of th dpdt varabl ar bary, ad t vstgatd whthr th sport or ot. Hr, a s trprtd as tstd postv. Ths data sampl rvals 0 xplaatory varabls whch ar gv as:y toy: FCC, Y: Zro Crossg Rat, Y: Short Tm Ergy, Y: Spctral Flux, Y7: Spctral Ctrod, Y8: Spctral rolloff, Y9: os Fram Rato, ad Y0: Slc Rato. Usg ths data st, th proposd framwork bult th modl for ach fv gr typs as show Fgur. I modl buldg stag, th paramtrs s st that maxmum umbr of bass fuctos s ad maxmum umbr of tractos s, trpolato typ s pcws-lar, palty pr kot s, th bst valu ca also b foud usg -fold Cross-Valdato. Largr valus wll lad to fwr kots bg placd. (a)odl for Dahmas (b)odl for usc TABLE.. ALGORITH Forward Procss Stag: Kpg coffcts th sam for varabl xstd th currt modl, Updat bass fuctos wth th updatd sd kots Add th w bass fuctos to th modl ad add th rflctd partr Slct a w bass fucto par that producs th largst dcras trag rror. Rpat th whol procss utl som trmato codto s mt: f rror s too small or f th umbr of modl's coffcts th xt trato s xpctd to b lss tha umbr of put varabls. Backward Prug Stag: Fd th subst whch gvs th lowst Cross Valdato rror, or GCV. Dlt o bass fucto pr stp ad rduc modl Yld th smallst cras of GCV bcom th w o (c)odl for Sport (d)odl for Cartoo ()odl for ws Fg.. s for ach gr Th varabl slcto rsults usg ach ARS modl ca b summarzd Tabl. It s obsrvd that FCC ad FCC do play mportat rols dcdg th ARS Dahma modls. For th ARS usc modls, FCC ad os Fram Rat do play major rols. For th ARS Sports modls, os Fram Rat, Slc Rato ad Spctral Flux 80

4 ar mor mportat. Short tm rgy ad FCC ar mportat varabl to dcd th Cartoo odl. os Fram Rato, Short Tm Ergy, FCC varabls ar mor mportat dcdg th w modls. From ths rsults, FCC, FCC, Short Tm Ergy, os Fram Rat, Slac Rato ad Spctral Flux ar mor usful faturs tha othr faturs. Gr Fu cto Dahma usc Sports Cartoo TABLE. DECOPOSITIO AALYSIS OF EACH ODEL ws STD GCV bass para ms varabl s 0,0, 9,,,7,9 0,0,9,0,0,0,0 9,0 7 9,,,9,7 8, For ths proposd systm, a tst st of 0 clps ar cratd to tst th accuracy rat. Classfcato accuracy rat s 97 prct for usc gr whch s bst prformac tha othr gr. It ca b s wth matlab program for our ow trag databas that cluds 80 vdos. As th pot of accuracy rol, ths algorthm has b proofd wth tru postv rat, tru gatv rat, fals postv rat, ad fals gatv wth th ow 0 vdos databas matlab. Accordg to th Tabl, tru postv rat ar 8 prct ad prct for ws ad Sport rspctvly. Fals postv rat ar also rportd prct ad, prct for ws ad Sport rspctvly. Tru gatv rat ar show ovr 9 prct for ach gr. From ths tabl, th proposd approach s ot optmzd for th Sport ad ws typs. Gr Dahma 9 usc Sport 0 Cartoo 7 Tru Postv () 0.9 (0) 0.7 (7) 0.9 () ws 0.8 () TABLE. CLASSIFICATIO Tru gatv 0.9 (80) 0.97 (8) 0.97 (0) 0.9 (8) 0.90 (9) Fals Postv 0. (8) 0.08 () 0. () () VI. COCLUSIO Fals gatv 0.07 (9) () 0.08 Accura cy 0.99 (0) (0) 0.90 (0) 0.9 (0) 0.8 (0) I ths papr, ultvarat Adaptv Rgrsso Spls s prstd for automatc vdo gr classfcato usg oly audo faturs. Th xprmts hav dcatd that t produc th classfcato rat whch wr 9%. As our studs maly us dmographc varabls as dpdt varabls, futur studs may am at collctg mor mportat varabls to mprov th classfcato accuracs. From th xprmtal rsults, FCC, FCC, Short Tm Ergy, os Fram Rat, Slc Rato ad Spctral Flux ar mor usful faturs tha othr faturs. Th xprmtal valuato of ths proposd systm cofrms th good prformac of vdo classfcato systm xcpt sport ad w but t s stll rasoabl rsults for both typs. Furthr xprmts o largr volum of audo, audo-vsual faturs wll b tstd ths framwork. Itgratg gtc algorthms ad/or gry thory, wth ural tworks ad/or support vctor machs ar possbl rsarch drctos furthr mprovg th classfcato accuracs. ACKOWLEDGET y Scr thaks to my suprvsor Dr. u War, for provdg m a opportuty to do my rsarch work. I xprss my thaks to my Isttuto amly Uvrsty of Tchology (Yataarpo Cybr Cty) for provdg m wth a good vromt ad faclts lk Itrt, books, computrs ad all that as my sourc to complt ths rsarch work. y hart-flt thaks to my famly, frds ad collagus who hav hlpd m for th complto of ths work. REFERECES [] J. am,. Alghomy, ad A. H. Twfk, Audo-vsual cott-basd volt sc charactrzato, Itratoal Cofrc o Imag Procssg (ICIP 98), vol., 998, pp. 7. [] S. ocrf, S. Vkatsh, ad C. Dora, Horror flm gr typg ad sc lablg va audo aalyss, ultmda ad Expo, 00, 00. [] W. Zhu, C. Toklu, ad S.-P. Lou, Automatc ws vdo sgmtato ad catgorzato basd o closd-captod txt, ultmda ad Expo, ICE, 00. pp

5 [] D. Brzal ad D. J. Cook, Automatc vdo classfcato : A survy of th ltratur, Systms, a, ad Cybrtcs, 008. []. Roach, L.-Q. Xu, ad J. aso, Classfcato of o-dtd broadcast vdo usg holstc low-lvl faturs, (IWDC 00), 00. [] R. Jassch ad J. Lou, Automatc tv program gr classfcato basd o audo pattrs, Euromcro Cofrc, 00, 00. [7] L.-Q. Xu ad Y. L, Vdo classfcato usg spatal-tmporal faturs ad pca, ultmda ad Expo, (ICE 0), 00. [8]. Roach ad J. aso, Classfcato of vdo gr usg audo, Europa Cofrc o Spch Commucato ad Tchology, 00. [9] Frdma, J.H., ultvarat adaptv rgrsso spls. A. Stat. 9, (wth dscusso) 99. Frst Author H E Latt has compltd astr of Egrg (Iformato Tchology) (.E-IT) from Wst Yago Tchology Uvrsty (WYTU). Currtly, sh s a PhD caddat from Uvrsty of Tchology (Yataarpo Cybr Cty). Sh s workg as a assstat- lcturr Tchology Uvrsty (yk) ad hr rsarch aras ar vdos from th hug amout of vdo collcto.. 80