An Integrated Statistical Model for Multimedia Evidence Combination

Transcription

1 An Integrated Statistial Model for Multiedia Evidene Cobination Sheng Gao, Joo-Hwee Li and Qibin Sun Institute for Infoo Researh, 21 Heng Mui Keng Terrae, Singapore, {gaosheng, joohwee, ABSTRACT Given rih ontent-based features of ultiedia (e.g., visual, text, or audio) followed by various detetors (e.g., SVM, Adaboost, HMM or GMM, et), an we find an effiient approah to obine these evidenes? In the paper, we address this issue by proposing an Integrated Statistial Model (ISM) to obine diverse evidenes extrated fro the doain knowledge of detetors, the intrinsi struture of odality distribution and interonept assoiation. The ISM provides a unified fraework for evidene fusion, owning the following unique advantages: 1) the intrinsi odes in the odality distribution are disovered and odeled by the generative odel; 2) eah ode is a partial desription of struture of the odality and the ode onfiguration, i.e. a set of odes, is a new representation of the douent ontent; 3) the ode disriination is autoatially learned; 4) prior knowledge suh as the detetor orrelation and inter-onept relation an be expliitly desribed and integrated. More iportantly, an effiient pseudo-em algorith is realized for training the statistial odel. The learning algorith relaxes the oputation ost due to the noralized fator and latent variables in graphial odel. We evaluate the syste perforane on ultiedia seanti onept detetion with the TRECVID 25 developent dataset, in ters of effiieny and apaity. Our experiental results deonstrate that the ISM fusion outperfors the SVM based disriinative fusion ethod. Categories and Subjet Desriptors H.3.3 [Inforation Systes]: INFORMATION STORAGE AND RETRIEVAL. General Ters Algoriths, Manageent, Theory. Keywords Seanti onept detetion, average preision, evidene fusion, odel-based fusion. 1. INTRODUCTION Multiedia douent ontains rih (e.g. inforation arried in Perission to ake digital or hard opies of all or part of this work for personal or lassroo use is granted without fee provided that opies are not ade or distributed for profit or oerial advantage and that opies bear this notie and the full itation on the first page. To opy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speifi perission and/or a fee. MM 7, Septeber 23 28, 27, Augsburg, Bavaria, Gerany. Copyright 27 ACM /7/9...$5.. ultiple hannels suh as visual, textual and audio) and diverse (e.g. visual appearane has a lot of variations for the sae seanti onept) inforation. It is far fro reahing the right features for ultiedia indexing, espeially for visual indexing [5]. Even if the features are rightly hosen, we still fae the proble of finding suitable ahine learning tools for ultiedia seanti onept detetion and inforation aess and retrieval. There are so any tools (e.g. paraetri or non-paraetri odels, generative or disriinative odels, et.) available to address the proble in hand. It is very hallenging to find the right ones. Thus, in pratie, the hoie is based on the experients or experienes learned fro other researhers. Rih and diverse inforation in ultiedia douent teahes us that no single solution would exist so far. Suessful systes in TRECVID always extrat various features fro the visual (e.g. olor, texture, edge, et), textual (e.g. tf-idf, nae entity, et.) or audio (e.g. Mel Frequeny Cepstral Coeffiients (MFCC), pith, Fast Fourier Transfor (FFT), et.) signals, and build various types of detetors (e.g. Support Vetor Mahine (SVM), AdaBoost, Hidden Markov Model (HMM), Gaussian Mixture Model (GMM), et.) 1. Then the outputs of these detetors are obined to obtain the final deision. For instane, 11 detetors are built based on the features extrated fro the visual and textual odalities and are obined to iprove seanti onept detetion in [4]. Therefore, the evidene obination is a ritial step in ultiedia ontent lassifiation. For sipliity, we disuss the evidene fusion in the ontext of seanti onept detetion in the paper. The evidenes ay be extrated using the detetors whih are trained on the different visual, textual or audio features using the suitable ahine learning algoriths or they ay be the prior knowledge on the feature disriination power, the assoiation strength aong the seanti onepts, et. For exaple, if we need to detet N seanti onepts and there are N d types of detetors trained for eah onept, the task of the fusion odel is to effiiently obine the N *N d detetor outputs to boost the perforane of onept detetion. Many different approahes, i.e. the non-paraetri ethod or the paraetri ethod, have been presented to address the issue. The non-paraetri ethod, e.g. CobSUM, CobMAX, does not need training saples to build a fusion odel [1, 14]. It is an adho ethod for easy usage. On the ontrary, the paraetri ethod needs training saples to estiate the fusion odel. It ay treat the N *N d outputs as a new representation of ultiedia douent. Then the supervised learning algoriths 1

2 are exploited, e.g. graphial odel [7], SVM [1, 4, 8, 13], MC MFoM [16], et. In TRECVID 1 [25], the SVM-based disriinative fusion odel is the doinant approah. In pratie, it ay be preferred to luster the detetor outputs into a few groups aording to soe prior knowledge. In this ase, the fusion will be opleted using ultiple stages. For exaple, if we have 3 detetors built on olor histogras in RGB, HSV and LUV spaes respetively and 2 detetors built on texture features suh as Gabor filter and gray-level o-ourrene, it ay be better to have the forer 3 outputs in one group and the others in another group. Then the fusion is first arried out in eah group and the outputs of the 2 groups are further obined. The doain knowledge an guide the design of groups. Soeties, the unsupervised learning approahes suh as PCA and ICA an be eployed to disover the groups [17]. Following [17], we use the ter odality to refer to eah group. Besides the evidenes fro the detetor outputs and the doain knowledge of detetors, another soure of evidene is the interonept assoiation, i.e. the perforane of deteting one onept an be boosted by deteting other onepts. For exaple, deteting the onept outdoor will help deteting the onept anial beause anial frequently plays in the outdoor. Anial is the boosted onept while outdoor is the boosting onept. Many works have been arried out to obine this ontextual inforation [7, 9, 11, 2]. To use the inter-onept relation in the fusion stage, graphial odel with various odel strutures (e.g. restrited Boltzann ahine, onditional rando field, arkov rando field, et.) is extensively eployed. The power of the ontextual evidene depends on any issues, e.g. the perforane of boosting onepts, the assoiation strength between the boosting onept and the boosted onept, et. To selet the strong boosting onepts, an ative ontext-based onept fusion is proposed in [2] and it is further inorporated into the boosted onditional rando fields in [9]. The experients on TRECVID dataset report obvious perforane iproveent due to the obination of the inter-onept relation. These works epirially show that the evidene of the onept assoiations an enhane the onept detetion. The issue is that the oputation ost is high for the graphial odel. In addition, the existing approahes lak the apaity to unify the various types of evidenes. For exaple, in [7, 9, 11, 2], the odels are designed to fuse the inter-onept assoiation where eah onept has one detetor output. It is an issue whether they would work well when eah onept has ultiple detetors. They also ignore the evidene of the orrelation aong detetors. Other works suh as [17] utilize the orrelation. However, the interonept relation is issed. All the approahes have no apaity to disover and inorporate the intrinsi statistial distribution of the odality, whose effiieny to iprove the obination of ultiple searh engines is deonstrated in [12] through odeling the sore distribution of the searh engine output. In [12], the odel is an exponential distribution for the non-relevant douents and a noral distribution for the relevant douents. In the paper, an Integrated Statistial Model (ISM) is presented to address the hallenging researh issue of obining ultiple evidenes extrated fro the detetor orrelation, the odality distribution and inter-onept assoiation. The ISM provides a unified fraework to obine evidenes with the following unique features: 1) the intrinsi odes in the odality distribution are disovered and odeled by the generative odel; 2) eah ode is a partial desription of struture in the odality distribution while the ode onfiguration, i.e. a set of odes, an be used to represent the douent; 3) the ode disriination is autoatially learned; 4) the prior knowledge suh as the odality orrelation and inter-onept relation is expliitly desribed and integrated. Further, we develop an effiient pseudo- EM algorith for training the statistial odel. It relaxes the oputation ost due to the noralized fator and latent variables in the graphial odel [7, 9, 11, 2]. We study and evaluate the proposed fusion odel on the task of seanti onept detetion using the developent set in TRECVID 25. The paper is organized as follows. In the next setion, the ajor oponents of ISM are disussed in detail. Then the pseudo-em algorith for estiating the ISM paraeters is presented in Setion 3. Experients and analyses are given in Setion 4. Finally, onluding rearks are presented in Setion OUTLINE OF ISM FRAMEWORK In this setion, we will first give a brief overview of the proposed ISM fraework. The learning algorith will be further disussed in Setion 3. Figure 1 depits the key oponents of the ISM odel. X is the deteted evidene, i.e. output sores of N *N d onept detetors and E is the prior knowledge inluding the orrelation in X and the inter-onept assoiation. In the following, we introdue eah oponent fro the botto to the top as shown in Figure 1. Doain knowledge of X P( X,E) Predit onept probability Extrat douent-level oourrene features of odes Predit latent odes Deteted evidene X Inter-onept assoiation Figure 1 key oponents of the Integrated Statistial Model 2.1 Predit Latent Modes Firstly we introdue a few ters whih will be used throughout the paper in the ontext of evidene obination. Modality: Assuing that a set of detetors are built for the onept detetion. The output value of the detetor is used to deterine whether the onept is present or absent. Usually the output is rando and its value is a real nuber. We refer to the odality as the rando variable as well as the orresponding detetor. Soeties we onatenate the output values fro soe detetors into a vetor for deision. In this senario the odality refers to the rando vetor as well as the orresponding vetors. We use odality value for a real value of rando variable.

3 Modality distribution: It refers to the statistial distribution of odality values. Mode: The odality ay ontain rih strutures, eah of whih ay be desribed by soe paraetri statistial distributions. The ode refers to one partial struture as well as its orresponding paraetri distribution. For exaple, the ode here is odeled by a single Gaussian distribution with the ean and ovariane and the odality with the 2-ixture Gaussian oponents onsists of 2 odes. Mode onfiguration: It is a vetor whose diension equals to the nuber of odalities. Eah eleent in the vetor is the ost representative ode identity for the observed odality value. The odality distribution ontains inforation that an iprove the ranking perforane. It is studied in [12], where the odality distribution is odeled by two oponent odels, one is Gaussian oponent for relevant douents and the other is Poisson oponent for irrelevant douents. Then the douents are resored using the learned odels. Rather than expliitly odeling the odality distribution as in [12], we odel the odes in the paper. All odes work together to render an approxiate iage of the orresponding odality. The ode odels are unknown and ode onfiguration is hidden. To learn the ode odels and ode onfiguration, the generative and disriinative approahes are eployed. Not liited to the Gaussian distribution for ode odels, other generative odels an also be used. However, it is not studied in the paper. When the ode odels are available, the observed odality values are apped to its orresponding ode onfiguration. The ode onfiguration is treated as a syboli representation of the odality values. The further deision an be arried out on it. It is uh different fro the traditional fusion odels, where only the original odality value is used while the deep struture of odality distribution is ignored. In the next setion, we will use a toy exaple to deonstrate the power of the ode odels to lassifiation and ranking Toy exaple Figure 2 illustrates a toy proble for 2 ategories, i.e. positive and negative lasses. 6 saples are used: 4 negative saples and 2 positive saples. One detetor is used to sore the 6 saples. The orresponding output sores are shown in the figure: irle points for negative saples and plus points for positive saples. The positive sores are loated in the iddle of the negative sores. With any threshold, there is always lassifiation error ourred. If the threshold is set to zero, the error rate is.33 with the 2 rightost negative saples, i.e..6 and.8, wrongly lassified. However, perfetly orret lassifiation ould be obtained if the ode odels were known. In this exaple, one ode is enough for haraterizing the odality distribution. The urve of the Gaussian ode odel (ean:.23, standard derivation:.42) is plotted in the figure (blue urve). Measured by the Eulidean distane, the distanes between the raw sores of saples and the ean of ode odel are.1 and.34 for the 2 positive saples, respetively. Correspondingly, they are.4,.19,.13 and.32 for the 4 negative saples (fro the left to the right), respetively. Now the 6 saples an be orretly lassified if the threshold (suh as.35) is used. Using the new sores, the 6 saples are orretly ranked. This exaple learly deonstrates the usefulness of the odality odes, despite that it is just a toy proble. With the statistis of the odes, the raw odality values will be transfored into a new spae where good ranking would be observed. Figure 2 a toy exaple to illustrate the ode iportane of the odality (Cirle points: negative saples. Plus points: positive saples. Blue urve: fitting Gaussian urve fro the saples) Predit odality odes To predit the ode identity, a set of ode odels are built. Eah odality will have K odes to haraterize its distribution. Like the toy exaple above, a single Gaussian distribution with the ean and variane is used for odeling the ode. The k-th ode is denoted as f ( ) (, ) k x = N x μk. Here x is the odality value. k The predited probability to assign the k-th ode to x is alulated as, η fk ( x) P( k x) = (1) Z ( x) K where Z ( x) f ( ) i 1 i x η = the ode with the axial probability, h( x ), is assigned to x as = and η is a soothing onstant. Then ( ) h x ( ) = arg ax P k x (2). k [ 1, K] However, the question is that the odes are unknown and they are hidden in the odality saples. There is no prior knowledge of the orret assignents between the odality value and the ode identity. Thus, the supervised learning approahes are infeasible. Fortunately, our ai is to use the ode as the interediate representation rather than to disover the eaningful odality odes. Therefore, the unsupervised learning algoriths, e.g. the k-eans lustering, are eployed. In the ISM fusion odel, the k-eans lustering algorith is used to initialize the ode odels. Then the ode odels are updated in the E-step in the iterative pseudo-em algorith developed for learning ISM odel (detailed in Setion 3). 2.2 Co-ourrene Mode Feature Extration When the ode odels of all odalities are available, the ode onfiguration an be found aording to Eq. (2). Assuing that there are M odalities eah having K odes, the odality values are X = x, i [ 1, M] and the orresponding ode onfiguration { i }

4 is H { hi, i [ 1, M] } =, where h i is the ode identity of x. This i onfiguration gives a syboli desription of the douent. Eah ode will funtion as a word likewise in a text douent. After apping the odality value using the ode odels, a douent represented in the ontinuous feature spae is tokenized using a set of odes. Thus, the douent-level features suh as tf-idf, unigra or bigra beoe available like in text ategorization and text inforation retrieval [22]. In this paper, the unigra feature, siilar to that adopted in text ategorization [21], is extrated. It is defined as, ( qi) ( ) #, w fq, ( I, y) Z q,, if = y, otherwise q, (3) = Here q is one ode identity of M*N*K odality odes (N: the nuber of onepts). I is a douent belonging to the onept y. w is a weight easuring the assoiation degree between the q, ode q and the onept (to be detailed in Setion 2.4). It oes fro the prior knowledge of the inter-onept assoiation. fq, ( ) is a feature extrator designed for the ode q and the onept. Z ( q, ) is a noralization fator so that the su of features is equal to 1, i.e. q, (, ) 1 (4) f I y = q, 2.3 Predit Conept Probability Fro the o-ourrene ode features (see Eq. (3)), we an train the onept odels to predit the probability assigned to a onept. The axiu entropy (ME) approah is applied to odel the onepts in the paper [2]. When the ME odels have been trained, they are used to predit the probability assigned to the onept aording to the observed evidene X. It is alulated as, (, θ ) ( λ,, ( ) q, q q ) 1 P ( I, θ) = exp f I, (5) Z I ( q, ) where Z ( I, θ) = exp λ f q, q, ( I, ) is the noralization fator and θ { λ q, } λ q, = is the paraeter set of onept odels. is a weight oeffiient of the feature extrated in Eq. (3). Eq. (5) is onept dependent. Hereafter, we use the ter onept odel to refer to it. In the ontext of lassifiation, the douent is assigned to the onept * whih has the axial probability aording to Eq. (5). * = arg ax P( I, θ ) (6) [ 1, N ] The odel paraeters θ an be trained through axiizing the likelihood on the training saples. Effiient algoriths suh as generalized iterative saling (GIS) or iproved iterative saling (IIS) are developed for estiating the odel paraeters. 2.4 Prior Knowledge The prior knowledge inludes the relations aong the detetors and the assoiation between the seanti onepts. The forer helps to luster the detetors into groups to obtain the odalities. In the paper, a group or odality only ontains the detetors built for one onept. For exaple, if a onept, saying A, has a set of detetors {A}. Siilarly, the onept B has a set of detetors {B}. Grouping the detetors is only arried out in {A} or {B} separately. And it is not allowed to luster the eleent in {A} and the eleents in {B} into one group. This onstraint keeps eah odality to have one unique onept identity, whih is shared by all its odes. It failitates the definition of the weights between the odality ode and the onept, i.e. w, in Eq. (3). The q, weight w is set to be equal to the assoiation strength between q, the onept identity assigned to q and the onept. The pair-wise assoiation strength is adopted in the paper. The degree of assoiation strength is estiated fro the training saples. For exaple, the assoiation strength, w, between the, ' onept and another onept is alulated as, w, ' ( ) #( ) #, ' = (7) where #(, ) is the nuber of douents relevant to both and in the training set and #() is the nuber of douents only relevant to. Eq. (7) is the easureent of onditional probability of on. Higher the value is, stronger the assoiation between and is. The strength of with itself is defined to be 1. For exaple, the assoiation strength between the onept airplane and outdoor is.84 and.67 between the airplane and sky. But it is zero between airplane and anial or building (estiated fro the training set based on TRECVID 5 developent set. See setion 4 for details.). 2.5 Disussions So far, the key oponents have been explained. We would like to stress that the ISM unifies these oponents rather than sequentially obining the. In the above, the ode onfiguration is deterinisti for siplifying the disussion. This indues a siple botto-up struture. One the ode odels are learned, they are not affeted by the onept odels estiated in the oponent predit onept probability. It is not optial. The good one is to integrate the botto-up and the top-down ethods, i.e. firstly, the ode odels (Eq. (1)) are estiated fro the observed odality values as well as the onept odel paraeters in Eq. (5) in the botto-up anner; seondly, the learned odels are used to predit the onept probability, whih are further feedbak to the botto so that the ode odels are updated using the top-down anner. These proedures ake it ipossible to learn the ISM odel using the traditional algoriths. In the next setion, we will present an effiient learning algorith to train the ISM and to use it to infer the onept identity assigned to the douent. 3. LEARNING AND INFERENCE We assue that there are M odalities aording to the doain knowledge of detetors, denoted by X = x, i [ 1, M]. Eah { i } odality gets the values in the ultidiensional spae. Correspondingly, the diensions for M odalities are denoted D = d, i 1, M. d is the nuber of detetors assigned to as { i [ ]} i

5 the i-th odality, i.e. the diension of the i-th odality. Thus, a douent I is represented in the M odality spae by a set of vetors, say I = ( x1, x2, L, xm ). Soeties a few odalities are issed due to any reasons, e.g. there are no detetor outputs for these odalities or the detetors are not used. In this ase, these odalities are skipped in learning and inferene. To learn the ISM for deteting the onept C, a training set, S = {( I, y), y { 1,} }, is given. y is the annotation for the douent I, whih is 1 if I is relevant to C ( i.e. the positive lass) and (i.e. the negative lass) otherwise. The odel paraeters to be estiated inlude 1) the ean and ovariane (diagonal here) of the ode odels, φ = { μ, }, with μ and being the paraeters q q q q for the ode q of the odality and 2) the ode weights, i.e. θ = λ, λ for the ode q of the odality and lass y. { } qy, qy, 3.1 Objetive Funtion In the ISM, there is a variable H ( h h h ) = 1, 2, L, to desribe the M apping between the observed odality values and the ode identities. If it is deterinisti, learning is easy. However, it is hidden and rando. In the next, we will derive an objetive funtion for effiient optiization. Firstly, we see the alulation of log-likelihood to predit the lass y, given the ISM. It is alulated as, ( P( y I φ θ) ) P( y H I φ θ) log,, = log,,, (8) It is the su over all possible ode onfigurations H. H M K For M odalities eah having K odes, there will be onfigurations. It is ipossible to opute Eq. (8) in pratie. Even if it were possible, there would be soe other hallenges to find a oputable odel for the joint distribution of the lass and the hidden variables, i.e. P( y, H I, φ, θ ). Here we seek an approxiate oputational odel to solve the proble. Aording to the Bayesian rule and Jensen s inequality, we an fatorize the joint distribution in Eq. (8) and find its lower bound, ( P( y I φ θ) ) = P( H I φ θ) P( y H φ θ) H P( H I, φ) log ( P( y H, θ) H ) log,, log,,,, The su in the seond line in Eq. (9) is the lower bound of Eq. (8) (note that P( H I, φ, θ ) is independent of θ and P( y H, φ, θ ) is independent of φ ). Rather than oputing Eq. (8), we use its lower-bound to approxiate it, i.e, ( P( y I φ θ) ) P( H I φ) ( P( y H θ) ) log,,, log, (1) H The first ter on the right hand side (RHS) is the predited probability of one ode onfiguration given the observed odality features and the ode odels. The seond ter explains how uh probability the lass y an be predited fro a fixed onfiguration given the onept odels. With the assuption that the odalities our independently, the first ter on the RHS in Eq. (1) is fatorized to be, (9) where ( i i, ) (, φ) P( hi x, φ) P H I = (11) P h x φ is the probability assigned to the ode i by the ode preditors. It is alulated fro Eq. (1). Substituting Eq. (5) into Eq. (1), the overall likelihood in the training set S is, Γ ( φθ, S) = P% ( I, y) P( y I, φθ, ) I, y = P% (12) ( I, y) P( H I, ),, (, ) I, y φ H λ q, qfq H y P% I P H I, φ log Z H, θ where P % ( I, y) and P ( I ) I ( ) ( ) ( ) H % are the epirial distributions in the training set. Eq. (12) is still diffiult for optiization due to the nonlinear ter, log Z I, θ. We further approxiate it using its upper bound, i.e., and, ( ) ( θ ) Z( H θ ) i log Z H, 1, (13) ( H, y) fq, Z ( H, θ) exp( λ, ) y q, q f (14) f where f f ( H, y) =. It is a onstant and is equal to 1 in q, q, the paper (see Eq. (4)). Substituting Eqs. (13-14) into Eq. (12), we an obtain the lower bound of Eq. (12), i.e., Γ low ( φθ, S) = P% ( I, y) (, ),, (, ) Iy, P H I φ H λ, q f q q H y (15) fq, ( H, y) + 1 P% ( I) P( H I, φ) exp( λ, ) I H y q, q f f In the equation, P ( HIφ, ) is fatorized as in Eq. (11), f ( H y) is a linear funtion that is alulated through siply q,, ounting the nuber of ourrenes of the ode in the douent, and exp( λq, f ) only depends on one ter. Thus, it an be effiiently optiized using the following pseudo-em algorith. This lower-bound funtion is the objetive funtion for learning the ISM. 3.2 Pseudo-EM Algorith The ISM paraeters are solved through axiizing the objetive funtion of Eq. (15). Sine the ode odel paraeters φ and onept odel paraeters θ are intertwined, we seek an iterative algorith, i.e. the pseudo-em algorith, to find the solution. In the M-step, the ode odels are fixed so that the onept odels,θ, are found for axiizing Eq. (15). By allowing the gradients of the objetive funtion over θ to be zero, we will find that θ has a losed solution (see Eqs. 16 (a-)). In the E-step, θ is fixed so that φ is solved by axiizing Eq. (15). However, φ is not analyti and the gradient desent algorith is applied to find a loal solution (see Eq. (17)).

6 1. Initialization a) k-eans lustering for initializing odality ode odels φ. b) θ is set to zero. 2. M-step: Calulate onept odels θ whenφ is fixed. 3. E-step: Update ode odels φ using the gradient desent algorith when θ is fixed. 4. Stop until the predefined riterion reahes, i.e. the axial iterative nuber or the relative inreent of objetive funtion is less than the threshold. Otherwise, go to (2). Figure 3 Pseudo-EM algorith to estiate the ISM In the M-step, the onept odel paraeters,θ, are alulated as, ( ) ( ) (, P% I q ) I P% I o δ y λ qy, = log P% ( I) oq where [1, M], q [1, K] and, ( y, ) o q Z I, { 1, } P( h, = q x φ ) I I y, (16a) = (16b) Z = o (16) q, q δ is an indiator funtion, whih is 1 if is equal to y and otherwise. In the E-step, φ is found using the gradient desent algorith, δγlow ( φ, θ S ) φt+ 1 = φt + α (17) δφ where φ is the estiate of t φ at the t-th iteration and α is a onstant to ontrol the learning rate. For a speifi ode odel, it is easy to dedue their partiular gradient funtions. Thus, the details are skipped here. Note that the varianes are updated in the log-doain to avoid overflow. 3.3 Ranking with ISM One the ISM is learned, we an use it for lassifiation or ranking. Thus, we need to alulate the log-likelihood in Eq. (8). Again its lower-bound is used for approxiation. The approxiated log-likelihood, L y, for the lass y is oputed as, where, T * T Ly =Λy O Λ O (18a) ( 1 1 M M y λ1, y,, λk, y,, λ1, y,, λk, y,, λ1, y,, λk, y) M M ( λ1,,, λk,,, λ1,,, λk,,, λ1,,, λk, ) * λq, exp( λq, ) Λ = L L L L L (18b) * 1 * 1 * * * * * T Λ = L L L L L (18) = (18d) ( 1 1 M M 1,, K,, 1,, K,, 1,, K ) O= o L o L o L o L o L o (18e) T T The oputation is trivial. When the likelihood is known for all lasses, the douent will be assigned to the lass having the axial value. To use the ISM for ranking, the likelihood ratio or the loglikelihood differene between the positive lass and the negative lass is used. The log-likelihood differene is alulated as, R= L1 L (19). The douents are ranked aording to the dereasing sore R. Higher the value R is, higher the rank is assigned to the orresponding douent. 4. RESULTS AND ANALYSES We evaluate the presented fusion odel on the task of seanti onept detetion using the developent dataset in TRECVID 25. The dataset has 137 MPEG news videos. We randoly split the videos into three sets, i.e. 7% (96 videos, ~3, shots) for training, 15% (2 videos, ~7, shots) for validation, and 15% (21 videos, ~6,7 shots) for evaluation. 39 seanti onepts, offiially used in TRECVID, are listed in Table 1. Table 1 Seanti onepts in TRECVID 5 ID Conept ID Conept ID Conept 1 Airplane 14 Explosion_Fire 27 Polie_Seurity 2 Anial 15 Fae 28 Prisoner 3 Boat_Ship 16 Flag-US 29 Road 4 Building 17 Governent-Leader 3 Sky 5 Bus 18 Maps 31 Snow 6 Car 19 Meeting 32 Sports 7 Charts 2 Military 33 Studio 8 Coputer_TVsreen 21 Mountain 34 Truk 9 Corporate-Leader 22 Natural-Disaster r 35 Urban 1 Court 23 Offie 36 Vegetation 11 Crowd 24 Outdoor 37 Walking_Running 12 Desert 25 People-Marhing 38 Watersape_Waterfront 13 Entertainent 26 Person 39 Weather The features used to build the onept detetors are shown below: Global olor orrelogra (GCC) in HSV spae: 324- diension. Co-ourrene texture extrated fro global gray-level o-ourrene atrix (GLCM): 64-diension. 3-D global olor histogra in HSV (HSV): 162-diension. 3-D global olor histogra in RGB (RGB): 125-diension. 3-D global olor histogra in LAB (LAB): 125-diension. For eah type of features, one SVM lassifier (SVM) [24] or one linear disriinative funtion (LDF) lassifier is trained. LDF is trained using the ROC optiization algorith [23]. Table 2 lists the details of 8 lassifiers, i.e. the identity nuber of a lassifier (Colun ID), feature type (Colun Feature) and lassifier type (Colun Classifier). For eah onept, these 8 lassifiers are trained. In total there are 312 detetor sores available for fusion. The perforane etri of the onept detetion is the average preision (AP) at the top-2 retrieved shots and the syste perforane is easured by the ean average preision (MAP) over 39 onepts. This is the offiial NIST evaluation etri.

7 Table 2 Detailed desription of lassifiers ID Feature Classifier ID Feature Classifier 1 GCC SVM 5 GLCM SVM 2 HSV SVM 6 GCC LDF 3 LAB SVM 7 HSV LDF 4 RGB SVM 8 GLCM LDF 4.2 Effet of Inter-onept Assoiation The seond experient evaluates the effet of the inter-onept assoiation. The assoiation strength between the onepts is estiated fro the training saples and alulated aording to Eq. (7). The 312 lassifier outputs are used, eah detetor being treated as one odality. As oparison, the benhark syste is still the SVM-based fusion odel trained on a 312-diensional feature vetor. Both systes are tuned using the sae way to the above experient. The two systes are denoted as SVM2 and ISM2, respetively. Figure 5 illustrates the AP values for all onepts SVM1 ISM Figure 4 Perforane oparison between the ISM and the benhark syste on the evaluation set (X-axis: the onept ID. Y-Axis: the AP value. Red bar: syste ISM1. White bar: syste SVM1.) 4.1 Coparison with SVM The benhark syste is based on the SVM disriinative odel fusion. It has been deonstrated suessful in TRECVID. In the first experient, we build the benhark syste, SVM1, by only obining the onept-speifi lassifiers, i.e. 8 lassifiers, for deteting a partiular onept, and do not onsider the effets of other onepts. We arefully tune the SVM onfiguration, i.e. the kernel type and paraeters of the kernel, for eah onept on the validation set. Then the onfiguration having the highest AP value on the validation set is used to train the final SVM odel and the learned SVM fusion odel is evaluated on the evaluation set. Correspondingly, we also train an ISM-based syste, ISM1, where eah lassifier is treated as one odality. The ISM is tuned as follows: first we train the ISM using 3 differene ode nubers, i.e. 2, 4 and 8, for eah odality and 1 iterations to selet the ode nuber having the highest AP value on the validation set. Then the ISM with the seleted ode nuber is trained in 3 iterations. Eah iteration generates an ISM odel, fro whih the odel having the highest AP value on the validation set is hosen for testing on the evaluation set. All other onstant paraeters, e.g. η,α, in the ISM learning algorith are epirially set based on one onept. Then they are used for all other onepts. These experient results are shown in Figure 4. The MAP value of ISM fusion is.239 over 39 onepts on the evaluation set. Coparing with.223, the MAP value of the SVM-based fusion, we have obtained a relative iproveent of 7.2%. The ISM syste outperfors the SVM syste aong 27 out of 39 onepts. Both systes are better than the perforane of the best individual detetor. In our experient, the best individual detetor is observed for the SVM syste trained on the HSV feature. Its MAP value is SVM2 ISM2 Figure 5 Effet of inter-onept assoiation on the perforane for the ISM and SVM systes on the evaluation set (X-axis: the onept ID. Y-Axis: the AP value. Red bar: ISM2 syste. White bar: SVM2 syste.) The MAP value of the SVM syste, SVM2, on 39 onepts is only.24, whih is worse than the SVM1 syste with the MAP value.223. In ontrast, the ISM syste, ISM2, obtains 5.4% relative iproveent of the MAP value when opared with the ISM1 syste. Its MAP value reahes.252. The further analysis on eah onept reveals that inorporating the inter-onept assoiation has indeed enhaned the detetion perforane for 26 out of 39 onepts. 4.3 Effet of Grouping Detetors The above experients treat eah detetor as one odality. Now we will study the effet of grouping soe detetors into one odality. Here only the results on ISM are reported. We base on the ISM2 syste and group soe detetors into one odality. For sipliity, we will only study one grouping ethod, i.e. grouping 8 detetors fro the sae onept into one odality. Thus there are 39 odalities to be used to train the third ISM syste, ISM3. The oparison of AP values between the ISM3 syste and the ISM2 syste is depited in Figure 6. The ISM3 syste has the MAP value.236. It is worse than the ISM2 syste with the MAP value.252. It suggests that this way of grouping detetors does not iprove syste perforane. Aong 39 onepts, the ISM3 syste only has 12 onepts whih perfor better than the ISM2. For soe onepts, grouping greatly deteriorates the ranking perforane, e.g. the onept ourt (ID=1) whose AP value is redued to.64 fro.244. Perhaps there are other grouping shees that ay perfor better, whih ay not be easy to identify. The knowledge of detetor orrelation does not see to be as powerful as that of the inter-onept assoiation.

8 ISM3 ISM2 Figure 6 Effet of grouping detetors on the perforane on the evaluation set for the ISM systes (X-axis: the onept ID. Y-Axis: the AP value. Red bar: ISM2 syste. White bar: ISM3 syste.) 4.4 Analysis of Modality Distribution As disussed in Setion 2, eah odality has its distint distribution haraterized by a set of ode odels in the ISM. In the setion, we study how the learned ode odels fit the epirial estiation of the odality distribution fro the training saples, and also epirially analyze and visualize the relation between the odes and the lasses. The experients of the ISM1 syste are hosen for illustration, where 8 odalities are used and eah lassifier is treated as one odality (see Table 2 for the IDs of the lassifiers or odalities). Due to the liited spae, we only selet 2 odalities, i.e. Modality 1 (GCC feature based SVM) and Modality 6 (GCC feature based LDF) for the onept airplane. Its Modality has 2 odes in the ISM1 syste. First, we opare the epirial histogra of odality values estiated fro the training saples with the predited histogra by the ode odels. They are shown in Figure 7 for the Modality 1 and Figure 8 for the Modality 6. It is found that the predition perfors better for the odality 6, i.e., the LDF output, than the Modality 1, i.e. SVM output. It ay be that the SVM sore has a uh saller variane than the LDF. In the future, we will seek other generative odels to fit the different odality distribution. In addition, fro Figure 7 and Figure 8, we observe that there is an obvious relation between the epirial histogra and the ode odel. The two ode odels respetively fit into two different kinds of epirial histogras. To build a link between the odes and the lasses, we draw the epirial histogras of the odality values for the positive and negative lass separately and depit the with the predited histogras by the two ode odels. The urves are illustrated in Figure 9 for Modality 1 and Figure 1 for Modality 6. Obviously, eah ode an be highly orrelated with one lass. For exaple, in Figure 9, Mode 2 fits well with the negative lass while Mode 1 fits with the positive lass. The siilar ase is found in Figure 1. It eans that for the Modality 1, its Mode 1 is disriinative for the positive lass while the Mode 2 is disriinative for the negative lass. Thus, the forer should have higher weight for the positive lass than that for the negative. The property is exeplified in Figure 11 through analyzing the ode weights. Siilarly, we an draw onlusions for other odality odes Epirial Predited Figure 7 Coparison between the epirial histogra (Blue urve arked with Epirial) and predited histogra (Red urve arked with Predited) by the ode odels on the training set (Conept: airplane. Modality: 1. X-axis: the odality values. Y-axis: the probability of saples whose odality values are in the interval.) Epirial -9.9 Predited Figure 8 Coparison between the epirial histogra (Blue urve arked with Epirial) and predited histogra (Red urve arked with Predited) by the ode odels on the training set (Conept: airplane. Modality: 6. X-axis: the odality values. Y-axis: the probability of saples whose odality values are in the interval.) Epirial_P Epirial_N Mode_1 Mode_2 Figure 9 Illustration of epirial histogras for the positive and negative lasses and predited histogras by the ode odels, respetively, on the training set (Conept: airplane.

9 Modality: 1. X-axis: the odality values. Y-axis: the probability of saples whose odality values are in the interval. Epirial_P: the epirial histogra of the positive lass. Epirial_N: the epirial histogra of the negative lass. Mode_1: the histogra predited by the Mode 1. Mode_2: the histogra predited by the Mode 2.) Epirial_P Epirial_N Mode_1 Mode_2 Figure 1 Illustration of epirial histogras for the positive and negative lasses and predited histogras by the ode odels, respetively, on the training set (Conept: airplane. Modality: 6. X-axis: the odality values. Y-axis: the probability of saples whose odality values are in the interval. Epirial_P: the epirial histogra of the positive lass. Epirial_N: the epirial histogra of the negative lass. Mode_1: the histogra predited by the Mode 1. Mode_2: the histogra predited by the Mode 2.) 4.5 Analysis of Mode Weights Now we analyze the learned weights in the onept odel of Eq. (5). These weights easure the ontribution of the odes to the onept. If its absolute value is high, the ode should be iportant and disriinative for the onept. Otherwise, the ode ontribution to predit the onept is sall. Here two onepts, airplane and flag-us, are seleted as exaples. The ISM odels used in the ISM1 are hosen for illustration. The weights are plotted in Figure 11 for airplane and Figure 12 for flag-us, respetively. In the seleted ISM odels, eah odality has 2 odes. Thus there are 16 weights. In the ode index in the 2 figures, Mode 1 and 2 belong to one odality. Siilarly, Mode 3 and 4 are in another odality. The sae rule is applied for others odes. Fro the two figures, the different patterns of ode weights are observed for the two lasses. For the two odes in one odality, it is often seen that one ode has a high weight for the positive lass while another has a high weight for the negative lass. It iplies that in eah odality, soe odes will doinate in the positive lass while others doinate in the negative lass. However, it is also found that the ode weights in a odality ay be alost equal. For the onept airplane (see Figure 11), the weights of Mode 9 are zeros for both the negative and positive lasses and are very lose for Mode 1. Siilarly, for the onept flag-us (see Figure 12), the weights of Mode 9 and 1 are also lose to eah other for both lasses. That iplies that the two odes have fewer disriinative inforation for predition and the orresponding odality ay have lower apaity to distinguish the positive lass fro the negative. We re-exaine the orresponding detetor perforane of the odality and find that it is the SVM lassifier using the GLCM feature. The AP value is.15 for airplane. It perfors worst in all 8 detetors for the onept. For flag-us, its AP value is.33 ranked at the iddle in all 8 detetors. The siilar observation is found for other onepts. These findings ay be used to predit and selet the disriinative detetors for fusion Positive Negative Figure 11 Learned ode weights for the positive and the negative lasses (Conept: airplane. X-axis: the ode ID. Y- axis: the weight oeffiient. Red bar: negative lass. White bar: positive lass.) Positive Negative Figure 12 Learned ode weights for the positive and the negative lasses (Conept: Flag-US. X-axis: the ode ID. Y- axis: the weight oeffiient. Red bar: negative lass. White bar: positive lass.) 5. CONCLUSION In the paper, a fraework, i.e. Integrated Statistial Model (ISM), is presented for obining rih evidenes extrated fro the doain knowledge of detetors, the intrinsi struture of odality distribution and inter-onept assoiation. The ISM provides a unified fraework for evidene fusion. Its effiieny and apaity are evaluated on seanti onept detetion using the developent dataset of TRECVID 25. We opare the ISM fusion with the SVM-based disriinative fusion. Signifiant iproveent is obtained. Through analyzing the histogra of odality values and the learned ode weights, we find that the

10 odes haraterize the struture of the odality distribution and they have different power to disriinate the positive lass fro the negative. However, we also find that the predited histogra by the learned ode odels does not fit well for soe odalities. In future, we will exploit other generative odels rather than the Gaussian distribution and study their effiieny. Referenes [1] Air, A., et al., IBM researh TRECVID-25 video retrieval syste. Pro. of TRECVID 5 Workshop. [2] Berger, A., et al., A axiu entropy approah to natural language proessing. Coputational Linguistis, 22(1), pp.39-71, Mar [3] Bradley, A.P., The use of the area under the ROC urve in the evaluation of ahine learning algoriths. Pattern Reognition, 3, pp , [4] Cao, J., et al., Intelligent ultiedia group of Tsinghua University at TRECVID 26. Pro. of TRECVID 26 Workshop. [5] Chang, S.-F., et al., Reent advanes and hallenges of seanti iage/video searh. Prof. of ICASSP 7. [6] M Donald, K. & Seaton, A. F., A oparison of sore, rank and probability-based fusion ethods for video shot retrieval. Prof. of CIVR 6. [7] Hauptann, A. G., et al., CMU Inforedia s TRECVID 25 skirishes. Pro. of TRECVID 5 Workshop. [8] Iyengar, G., et al., Disriinative odel fusion for seanti onept detetion and annotation in video. Pro. of ACM Multiedia 3. [9] Jiang, W., et al., Context-based onept fusion with boosted onditional rando fields. Pro. of ICASSP 7. [1] Lee, J.-H., Analyses of ultiple evidene obination. Pro. of SIGIR 97. [11] Naphade, M. R., et al., Probabilisti seanti video indexing. Pro. of NIPS. [12] Manatha, R., et al., Using odels of sore distributions in inforation retrieval. Pro. of Workshop on Language Modeling and Inforation Retrieval, 21. [13] Sith, J.R., et al., Multiedia seanti indexing using odel vetors. Pro. of ICME 3. [14] Tax, D. M. J. et al., Cobing ultiple lassifiers by averaging or by ultiplying. Pattern Reognition, 33(9), pp , 2. [15] Tseng, B. et al., Noralized lassifier fusion for seanti visual onept detetion. Pro. ICIP 3. [16] Wang, D. H., et al., Disriinative fusion approah for autoati iage annotation. Pro. of MMSP 5. [17] Wu, Y. & Chang, E.-Y., Optial ultiodal fusion for ultiedia data analysis. Pro. of ACM Multiedia 4. [18] Yan, R. & Hauptann, A. G., The obination liit in ultiedia retrieval. Pro. of ACM Multiedia 3. [19] Yavlinsky, A., et al., A oparative study of evidene obination strategies. Pro. of ICASSP 4. [2] Jiang, W., et al., Ative Context-based onept fusion with partial user labels. Pro. of ICIP 6. [21] Niga, K., et al., Using axiu entropy for text lassifiation. Pro. of IJCAI Workshop on Mahine Learning for Inforation Filtering, [22] Baeza-Yates, R. & Ribeiro-Neto, B.,, Modern inforation retrieval. New York, ACM Press, [23] Gao, S. & Sun, Q. B., Classifier optiization for ultiedia seanti onept detetion. Pro. of ICME 6. [24] Joahis, T., Learning to lassify text using support vetor ahines. Dissertation, Kluwer, 22. [25] Over, P., et al. TRECVID 25 overview. Pro. of TRECVID 5 Workshop.