Library-based coding: a representation for. ecient video compression and retrieval. MIT Media Lab,

Transcription

1 Lbrary-based codng: a representaton for ecent vdeo compresson and retreval Nuno Vasconcelos and Andrew Lppman MIT Meda Lab, fnuno,lpg@meda.mt.edu Abstract The ubquty ofnetworkng and computatonal capacty assocated wth the new communcatons meda unvel a unverse of new requrements for mage representatons. Among such requrements s the ablty of the representaton used for codng to support hgher-level tasks such as content-based retreval. In ths paper, we explore the relatonshps between probablstc modelng and data compresson to ntroduce a representaton - lbrary-based codng - whch, by enablng retreval n the compressed doman, satses ths requrement. Because t contans an embedded probablstc descrpton of the source, ths new representaton allows the constructon of good nference models wthout compromse of compresson ecency, leads to very ecent procedures for query and retreval, and provdes a framework for hgher level tasks such as the analyss and classcaton of vdeo shots. 1 Introducton The ntroducton of dgtal communcatons and nexpensve computaton orgnated a shft from mage representatons based on very smple prmtves (snusods) and operatons (lterng and modulaton) towards others based on more nvolved languages, capable of explotng the statstcal characterstcs of vdeo sources and the characterstcs of the Human Vsual System, or provdng hgher level descrptons of scene content. Whle ths shft allowed more ecent use of the avalable bandwdth, t has also uncovered a unverse of new requrements for mage representatons. Because dgtal decodng requres computatonal capacty at the recevng end of the channel, t leads to smart nformaton and entertanment applances capable of two way communcaton under the control of the user [8]. Dgtal decoders are ntellgent, and capable of searchng the network for the content that s \just rght" for ther users. Unfortunately, current dgtal representatons, such as JPEG or MPEG, desgned wth the sole goal of achevng compresson ecency, are not helpful for ths task. Perhaps due to ths, most of the recent eort n the area of content-based retreval consders ths task ndependently of the ssue of bandwdth ecency. Typcal

2 solutons consder a feature space for retreval whch does not overlap wth the representaton space used for compresson, e.g. whle compresson s based on DCT [6] or wavelet bass functons [1], retreval s based on color hstograms [5] or texture features [7]. Such solutons mply that ether the features are pre-computed and stored n addton to the compressed btstreams - a process whch s necent n terms of storage resources - or the bt-streams must be decoded and the features computed at the tme of query - a procedure whch s computatonally very necent snce all the work performed at the tme of mage encodng s useless for the task of retreval. Even when spaces used for retreval and compresson overlap and full mage reconstructon s not requred (e.g. when wavelet or DCT coecents are used for retreval), the process stll suers from necences: these coecents are n general a szeable porton of the btstream and ther decodng can, therefore, stll be an expensve task, and t s generally not clear that the feature space assocated wth them s the most approprate for statstcal dscrmnaton, or that t allows the constructon of good models for statstcal nference. The fundamental goal of ths work s to restore some of the producton optons or nteractve potental of vdeo by augmentng the representaton used n codng and by explotng the analyss used for compresson as a retreval ad. The noton s that the analyss used n makng an ecent coder can potentally provde useful cues to the underlyng acton n the scene that may facltate browsng, lterng, sortng or combnng sets of movng pcture sequences. For ths we ntroduce a representaton, lbrary-based codng, whch contans an embedded descrpton of the pcture content and allows the constructon of good nference models wthout compromse of compresson ecency. Ths embedded descrpton s compact and can be decoded ndependently of the bulk of the dgtal btstream, leadng to very ecent procedures for query and retreval. Furthermore, because the representaton allows the constructon of probablstc models for statstcal nference, t provdes a framework for performng hgher level tasks such as analyzng and classfyng vdeo shots. Fnally, because t s close to the representatons currently used for vdeo compresson, t requres only slght alteratons to the exstng standards. 2 Embedded probablstc descrptons A representaton capable of supportng content-based queres wthout full decodng should nclude a compact, decodable on ts own, descrpton of the statstcal propertes of the mage source. Ideally,onewould want a complete probablstc descrpton, such as the probablty densty functon (pdf) of the stochastc process from whch the mages were drawn, because that would allow retreval to be based on statstcal nferences. For example, gven the probablty denstes assocated wth M sources p(xjs ), =1 ::: M, ther relatve probabltes of occurrence P (S ), and a set of data x created from mages to be classed as belongng to one of the classes, the task could be performed optmally by usng an Maxmum a Posteror Probablty (MAP)

3 crteron and Bayes rule,.e. pck the class such that = arg max 2f1 ::: M g P (S jx) =arg max 2f1 ::: M g P (xjs )P (S ): (1) One possble way to acheve ths would be to use a non-parametrc descrpton of the source, such as a hstogram. However, non-parametrc descrptons are not compact, mplyng a degradaton of the ecency of the representaton n terms of compresson, and t s not clear how they could be used as a part of the encodng procedure,.e. not smply as overhead. Fortunately, the alternatve of usng parametrc descrptors, n partcular mxture denstes, satses these two requstes, provdng a much more ecent soluton. 2.1 Parametrc modelng, mxture denstes, and the EM algorthm A parametrc probablstc model capable of approxmatng any probablty densty s the class of mxture denstes [9]. A mxture densty has the form P (x) = CX =1 P (xj! )P (! ) (2) where C s the number of probablty classes, P (xj! ) are the class-condtonal denstes, andp (! ) =1 ::: C the class probabltes ( P C =1 P (! ) = 1). The classcondtonal denstes can be any vald probablty densty functons, even though they are n most of the applcatons (and n the rest of ths paper) assumed to be Gaussans. In ths case, the mxture densty becomes P (x) = CX =1 p e ; 1 (x j; 2 ) T ;1 (x j ; ) (3) where p = P (! ). The mxture s then completely characterzed by the parame- p(2) n jj ters L (s) = f (s) (s) p (s) =1 ::: Cg. The standard statstcal tool for the estmaton of mxture parameters s the Expectaton-Maxmzaton (EM) algorthm [3]. The EM algorthm treats the class assgnments (.e. whch class s responsble for each sample) as hdden (non-observed) varables and, gven a set of M ndependent and dentcally dstrbuted samples x =1 ::: M, nds the mxture parameters that maxmze the data lkelhood by teratng between the followng steps. E-step: h m = P (! jx m )= P (x m j! )p PN k=1 P (xm j! k )p k (4) M-step: new = Pm h m x P m new m h m = Pm h m (x m ; new P m h m )(x m ; new ) T p new = 1 M X m h m (5)

4 where m =1 ::: M and =1 ::: C. Gven a mage to compress, the parameter set L (s) can be estmated through the EM algorthm and ncluded n the compressed btstream leadng to a compact, stand-alone descrpton that can be used to perform statstcal nferences such as those mentoned above. However, whle ths would ncrease the retreval ablty of the representaton, t would also compromse ts compresson ecency, as ths descrpton would amounttopureoverhead. The nterestng mssng lnk s that mxture parameters are very closely related to the codebooks whch form the bass of a representaton whch s known to be optmal from a compresson standpont: vector quantzaton (VQ) [4]. 2.2 Vector quantzaton Avector quantzer Q s a mappng from a K-dmensonal vector space of nput samples to a nte set of reconstructon vectors, usually known as codevectors or codewords. The set of reconstructon vectors s generally desgnated by codebook. The N-dmensonal nput vector space s parttoned nto a set C of N K-dmensonal regons R,alsoknown as parttons or cells, and a reconstructon vector y assocated wth each regon. The non-lnearty nherent to the operaton of quantzaton makes t mpossble to acheve a sngle, closed-form soluton to the problem of optmal vector quantzaton. It s however possble to nd two necessary condtons for optmalty by decomposng the problem nto two smaller ones: ndng the optmal partton for a gven codebook, and the optmal codebook for a gven partton. The optmal partton (encoder) for a xed codebook (decoder) must satsfy the nearest-neghbor condton R fx : d(x y ) d(x y j ) 8j 6= g (6) whle the optmal codebook for a gven partton must satsfy the generalzed-centrod condton Q(x) =mnfe[d(x y )jx 2R ]g: (7) y The most popular algorthm for vector quantzer desgn - the LBG algorthm [4] - terates between these two condtons, whch, gven a tranng set T = t 1 ::: t M, and assumng the mean squared error dstorton metrc become, respectvely, R = ft 2T : jjt ; y jj < jjt ; y j jj 8j 6= g (8) and Y = E[xjx 2R ]= P M j=1 t j S (t j ) P M j=1 S (t j ) (9) where the S (t j )=1ft j 2R,andS (t j ) = 0 otherwse.

5 2.3 Relatonshp between mxture densty estmaton and vector quantzaton To understand the relatonshp between vector quantzaton and estmaton of mxture parameters we start by re-wrtng the EM equatons for the smpler case of equally lkely classes (p =1=C), and dentty covarances E-step: M-step: h m = P (x m j! ) P N k=1 P (x m j! k ) = e ; 1 ; 2 jj 2 P N 1 k=1 e; jjxm ; 2 k jj 2 (10) new = Pm h m x P m. (11) m h m The smlartes wth VQ desgn are made clear by the comparson of these expressons wth equatons 8 and 9. The only derence s that, n the VQ case, the h 's are thresholded after the E-step so that ( 1 h 0 = 0 f h >h j 8j 6= otherwse (12).e. the tranng sample s assgned to the mxture component that has maxmum a posteror probablty ofhavng generated t. Ths transforms equaton 10 nto equaton 8 1, and consequently equaton 11 nto equaton 9. Therefore, gven a sample to encode, the optmal codeword to represent t s the mean of the mxture component whch has maxmum a posteror probablty ofhavng generated the sample. The results above can be extended to the generc Gaussan case wth full covarances and class probabltes. In ths case, the relatonshps between mxture densty estmaton and vector quantzaton are the same, but the VQ s optmal under the Mahanolabs dstance, and has a constrant on ts output entropy. I.e. gven an nput vector t, the optmal partton s the one for whch where R j = ft 2T : jjt ; y j jj 2 j ; log p j < jjt ; y jj 2 ; log p 8g (13) jjt ; y j jj 2 =(t ; y j ) T ;1 (t ; y j ): (14) The man concluson s thus that vector quantzaton s smply EM estmaton wth MAP class assgnments and, n practce, ths means that the codebooks orgnated by VQ are a good approxmaton to the parameters of the mxture densty that best descrbes the data. Or, f one desgns the codebook wth the EM algorthm and uses MAP decsons only after the after tranng s completed, the codebook wll also provde an optmal estmate of the mxture parameters. 3 The lbrary-based coder The lbrary-based coder bulds on the smlartes between EM and VQ desgn to obtan a representaton that can be used to jontly address the ssues of compresson 1 Notce that the denomnator of equaton 10 s smply a normalzng constant, equal for all h 's.

6 and retreval. For each frame, a codebook s desgned and transmtted to the recever. The frame s then encoded usng VQ, and the quantzaton ndexes transmtted as well. Because the codebook provdes a probablstc descrpton of the source, t s all that needs to be decoded for the purposes of retreval - the bulk of the data beng decoded only when the frame s to be reconstructed. From the compresson pont of vew, the scheme can be seen as a unversal encoder, contnuously adapted to the source probabltes. Whle, n theory, VQ has long been known to be the optmal compresson scheme n practce, because optmalty sonlyacheved wth large vector szes and encodng complexty grows exponentally wth vector sze, vector quantzers have fallen short of provdng the theoretcally attanable performance. In fact, f block szes and encodng complexty are to reman compatble wth those of the current standards, codebooks wll be lmted to relatvely small szes, leadng to reduced rates and poor mage qualty. Rate control Lbrary desgn + DCT Q Q 1 M U X Lbrary update IDCT Lbrary predcton Moton compensated predcton Lbrary entres Lbrary ndces Moton estmaton Moton vectors Fgure 1: Block dagram of the lbrary-based coder. Due to ths lmtaton, and the desre to keep the codng model as close to that of the current standards as possble, our mplementaton of the lbrary-based coder s bascally an extended verson of MPEG, where the lbrary s used as an addtonal predctor. In ths settng, the lbrary-based coder can be seen as retreval-enabled MPEG, wth the addtonal benet of better predcton (through the lbrary) durng common events where block-matchng fals. A block dagram of the complete coder s presented n gure 1. Each nput frame s segmented nto square blocks, whch are then processed to mnmze both temporal and spatal correlaton. Two derent predcton structures are used for temporal processng: the lbrary-based predctor dscussed above, and a conventonal motoncompensated predctor. By usng the two predcton modes, t s possble to combne the hgher ecency of moton-compensated predcton n areas of translatonal or reduced ampltude moton, wth the ncreased performance of lbrary predcton n areas of non-translatonal moton, object occluson, or where new objects are revealed. The encodng of the predcton error sgnal s smlar to that used by MPEG-2 [6].

7 4 Content-based queres Consder the task of ndng the closest match n a database to a gven a query mage. Ths task can be solved n several ways f the lbrary coded representaton s used. The smplest of the solutons s probably to follow the route outlned n secton 2. Assumng that the blocks of the query mage x =1:::N, are ndependent samples of the same stochastc process, the mages n the database are samples from M mage sources S, and usng equaton 1, the source wth the hghest probablty of havng generated the query mage s s =arg max NY k2f1 ::: M g =1 P (x js k )P (S k ): (15) Gven the lbrary entres L (k) = f (k) (k) p (k) =1:::Cg, the condtonal probabltes of equaton 15 are computed through equaton 3. In the absence of any pror knowledge about the relatve source lkelhoods, the term P (S k ) can be dsregarded. In a more complete settng, pror probabltes can, however, be used to constran the search. If, for example, the mages n the database are annotated wth text and the user speces a preference for pctures contanng \people", the retreval engne can assgn a hgh pror to all the mages annotated wth the \people" keyword, and a low pror to the remanng mages, ncreasng the posteror lkelhood of the mages n the desred category. The pont s that, unlke other types of features, because lbrares are probablstc descrptons of the source, they allow statstcal reasonng and the constructon of powerful search paradgms n the compressed doman. A practcal lmtaton of a soluton based on equaton 15 s that computng all the N condtonal probabltes can stll be an expensve task, whch grows proportonally to the mage sze and mples decodng the query mage f ths s also orgnally compressed. An alternatve, less expensve, soluton conssts n substtutng the product of condtonal probabltes by a functon whch measures the smlarty between the probablty denstes assocated wth the query and the database mages s =arg max k2f1 ::: M g D[P (xjs q ) P(xjS k )] (16) where S q s the source of the query. Event thoughany of the tradtonal smlarty metrcs, such as the Kulback-Lebler dstance [2], could theoretcally be used n equaton 16, these metrcs typcally do not lead to smple closed-form expressons for Gaussan mxtures. We have, therefore, consdered the followng smpler metrc nspred by equaton 15 s = arg max CY k2f1 ::: M g =1 P ( (q) js k ) (17) where the mage blocks x are replaced by the means of the Gaussans (q) n the mxture assocated wth the query mage. Ths reduces the number of condtonal probabltes to be evaluated by approxmately an order of magntude, and allows searches that not even requre full decodng of the query mage.

8 The metrc above can be further smpled by notcng that, for a gven x, the contrbuton of most of the Gaussans n equaton 3 to P (x) wll be neglgble. In the extreme case of neglgble overlap between the Gaussans n the mxture, at most one of these Gaussans wll be responsble for the bulk of the probablty. One can, therefore, use the approxmaton obtanng ; log P ( (q) js k ) mn f 1 21:::C 2 jj(q) s =arg mn CX k2f1 ::: M g =1 mn f1 21:::C 2 jj(q) ; (k) jj2 (k) ; (k) jj2 (k) ; log p (k) g (18) ; log p (k) g: (19) Comparng equaton 18 wth the vector quantzaton expresson of 13 once agan makes explct the connecton between VQ and MAP probablty estmaton usng mxture denstes. Thus, under the assumpton of separated Gaussans and the approxmaton of equaton 17, the closest mage to that used as a query s the one whose lbrary s closest to the lbrary assocated wth the query mage n the tradtonal VQ sense. 5 Smulaton Results In ths secton we analyze the performance of the lbrary-based coder. Because the compresson ecency of the coder was already studed n [10, 11], wenow concentrate on ts retreval capabltes. Fgure 2 depcts the dstances between a query mage and the mages n a database of 700 frames contanng varous scenes taken from tralers of the moves \Termnal velocty" and \East of Eden". Image 200 was used as a query example, and s presented nsde a whte frame. Notce that the measured dstances agree wth what should be expected from the retreval system. The query mage obvously has a null dstance to tself. Next, the closest mages are those n the same vdeo shot and, for these mages, the query dstance ncreases gradually wth the temporal dstance to the query mage. The followng closest mages are those n scenes wth content smlar to that of the query mage, namely people oatng n md ar, and large areas of sky. Next come scenes whch contan some degree of sky or water and nally, at a sgncant dstance, are the artcally generated graphcs. Ths ndcates that, as a metrc, the nter-lbrary dstance s capable of ne dscrmnaton between derent types of content. Fgure 3 shows examples of content-based retreval on the same mage database. It contans four mages, each dsplayng the results of ve queres. In all cases, each row corresponds to a query, wth the query mage shown on the rght, followed by the four best matches (excludng tself) n the database whch are presented from left to rght accordng to ther smlarty rank - most smlar on the left, less smlar on the rght. The query mages were selected randomly from the 700 frames n the sequence. A total of 130 queres were performed. The retreval results were manually classed as good or bad matches. Ths s a relatvely easy task n ths settng, because t s easy to determne f the retreved

9 Fgure 2: Example of the retreval results acheved wth the lbrary-based coder. Graph shows the dstance to the query frame (frame 200) versus frame number for the 700 mages n the sequence. Key-frames are shown for most of the vdeo shots. Query frame s presented wth a whte border. mage belongs to the same vdeo shot as the query mage. Table 1 presents the percentage of good matches, as a functon of the rank of the retreved mage. References [1] M. Antonn, M. Barlaud, P. Matheu, and I. Daubeches. Image Codng Usng Wavelet Transform. IEEE Trans. on Image Processng, Vol. 1, Aprl [2] T. Cover and J. Thomas. Elements of Informaton Theory. John Wley, [3] A. Dempster, N. Lard, and D. Rubn. Maxmum-lkelhood from Incomplete Data va the EM Algorthm. J. of the Royal Statstcal Socety, B-39, [4] A. Gersho and R. Gray. Vector Quantzaton and Sgnal Compresson. Kluwer Academc Press, [5] Y. Gong, H. Zhang, H. Chuan, and M. Sakauch. An Image Database System wth Content Capturng and Fast Image Indexng Abltes. In Proc. Int. Conf. on Multmeda Computng and Systems, May 1994, Boston, USA. [6] ISO-IEC/JTC1/SC29/WG11. MPEG Test Model, MPEG93/457. [7] F. Lu and R. Pcard. Perodcty, drectonalty, and randomness: Wold features for mage modelng and retreval. Techncal Report 320, MIT Meda Laboratory Perceptual Computng Secton, 1995.

10 [8] N. Negroponte. Beng Dgtal. Alfred A. Knopf, Inc, [9] D. Ttterngton, A. Smth, and U. Makov. Statstcal Analyss of Fnte Mxture Dstrbutons. John Wley, [10] N. Vasconcelos. Lbrary-based Image Codng usng Vector Quantzaton of the Predcton Space. Master's thess, Massachusetts Insttute of Technology, [11] N. Vasconcelos and A. Lppman. Lbray-based Image Codng. In Proc. Int. Conf. Acoustcs, Speech, and Sgnal Processng, Adelade, Australa, Fgure 3: 20 examples of retreval from a database of 700 mages. Each row contans the results of a retreval, query mage shown on the left followed by the closest match, second best match, etc. Rank Good matches (%) Table 1: Percentage of good matches versus smlarty rank for 130 random retrevals from a sequence of 700 frames.