DISPLACEMENT ESTIMATION FOR IMAGE PREDICTIVE CODING AND FRAME MOTION-ADAPTIVE INTERPOLATION

Transcription

1 DSPLACEMENT ESTMATON FOR MAGE PREDCTVE CODNG AND FRAME MOTON-ADAPTVE NTERPOLATON Georges TZRTAS Laboraoire des Signa e Ssèmes (C.N.R.S.), Ecole Spériere d'elecricié, Plaea d Molon 9119 GF-sr-YVETTE, FRANCE ABSTRACT We presen and invesigae mehods of displacemen vecor esimaion, which can be sed in predicive image coding or moion-adapive frame inerpolaive coding. n boh cases, a differenial approach is adoped. This means ha he displacemen vecor esimaion is based on he measremen of he spaioemporal gradien of he image seqence. n he case of predicive coding he moion esimaor is composed of hree pars. The firs is a spaial predicor of he displacemen vecor, which is based on a spaial aoregressive relaion of he veloci field. The coefficiens of his relaion are made inensi-dependen. The presence of disconiniies inheren o he moion and o he insabiliies of he esimaion algorihm makes necessar a sage of deecion of all pe of disconiniies. Finall, an a poseriori esimaor achieves he ask of displacemen vecor esimaion. This las sage is of ieraive form. The applicaion of his algorihm in a ver nois image seqence has permied o obain a gain of abo 4% in he absole vale of he difference wih he prediced displacemen vecor and 65% wih he esimaed one afer wo ieraions. The same srcre of displacemen field esimaion can be sed o make a frame inerpolaion moionadapive. We presen sch an esimaor, which is slighl differen from ha proposed in predicive coding, knowing ha a non-casal a priori esimaion can be realized. We also propose a compleel differen algorihm, which ses a hierarchical represenaion of he displacemen field. A dnamic hierarchical 4-rees esimaion algorihm is presened. The moivaion for sch a proposiion comes from he fac ha large areas in a grea nmber of image seqences are no moving or have homogeneos wo-dimensional moion, ha is o sa ranslaional. Considering his redndanc in he displacemen field a significan gain in complei of calclaions can be obained, if he esimaion is carried o raher over blocks han pel-b-pel. We have applied he las algorihm in he case of a seqence wih moderae moion and some nmerical resls are given in his aricle. 1. NTRODUCTON Moion esimaion from a seqence of images arises in man applicaion areas, principall in scene analsis and image coding. We presen in his aricle an esimaor, which akes ino accon he consrains of predicive coding. We consider ha he predicion error afer movemen compensaion is ransmied. We also presen a emporal inerpolaor sing an hierarchical moion esimaor. n boh cases, ha is in frame predicion and inerpolaion, a differenial approach is adoped. The image inensi is considered as a coninos spaio-emporal fncion. The esimaion of he displacemen vecor is based on he spaial and emporal gradien of he inensi. The eqaion sed is ha of moion consrain, which is obained nder hpohesis ha he inensi variaion is niqel de o he moion. Hence sing a firs order Talor developmen, one can wrie = (+, + ; + ) - (, ; ) = ( (, ; ) + v (, ; ) + (, ; )) =,

2 (, ; ) being he inensi and (resp. v) he horizonal (resp. verical) componen of he veloci vecor. n real images his eqaion is no eacl verified and minimizing some qadraic error on his displaced frame difference and sing some ideas of spaial coherence of he displacemen vecor carr o he esimaion. A review of mehods proposed in he lierare o solve his problem, can be fond in 4. n he case of predicive coding he esimaor is composed of hree pars. The firs par is an a priori esimaor, which is a recrsive predicor of he veloci vecor, sing he esimaes of previos image poins. The second par is a disconini deecor. These disconiniies are de o eiher occlding edges of objecs nder differen 3D moions, or edges of differen srfaces of he same objec, or algorihmic disconiniies. The deecion crierion is based on he predicion error. The las par is an a poseriori esimaor, which is an adapive ieraive filer. Figre 1 gives he esimaor srcre. The same form of srcre is sed in 1,3,6. The displacemen esimaion in he cone of predicive coding is he objec of Secion. (.,.;k) Displacemen predicion (.,.;k) v (.,.;k) Disconiniies deecion Oi Non = v = A poseriori displacemen esimaion (.,.;k) v(.,.;k) e (.,.;k) Fig.1. Srcre of he displacemen esimaor We discss he frame moion-adapive inerpolaion in Secion 3. Two approaches are considered; he firs one, is a direc applicaion of he above srcre in he cone of inerpolaion, whereas he second one is a dnamic hierarchical esimaor of he displacemen field. We discss here briefl he idea, which moivaes his las approach. The compaional complei of pel-b-pel esimaion is ecessivel large for image seqences presening large areas of consan veloci vecor, when block esimaion cold give he same accrac. Some applicaions of he sdied mehods in real image seqences are presened in he las Secion.. MOVEMENT ESTMATON FOR PREDCTVE CODNG.1. A priori esimaion of he veloci vecor Le s consider he problem of esimaing he veloci vecor a poin A (i,j) (Fig. ), where i is he horizonal coordinae and j he verical coordinae. Le [ v ] T be he veloci vecor a his poin. We have o esimae his vecor knowing he veloci a "preceden" poins, ha is in he domain { (i-m,j) : m = 1,,i } U { (l,j-n) : n = 1,,j ; l} We are limied o a casal neighborhood, which conains hree poins { B (i-1, j), C (i, j-1), D (i-1, j-1) } given in Figre. Le [ B v B ] T = [(i-1,j) v(i-1,j)] T be he veloci vecor of poin B, and in he same wa for C and D. The esimaion is made b minimizing he following qadraic form

3 Q (,v) = [- B - C - D ]K -1 [- B - C - D ] T + [v-v B v-v C v-v D ]K -1 [v-v B v-v C v-v D ] T D B C A Figre. The casal domain of displacemen vecor predicion We can wrie Q (,v) = (1-a) T K -1 (1-a) + (v1-b) T K -1 (v1-b), where 1 = [ ] T, a = [ B C D ] T and b = [ v B v C v D ] T. Mari K ma be considered as he covariance of he vecor 1-a. This mari is spaiall varian and nknown. We have herefore o make a herisic choice. We give he solion of he minimizaion of he form Q (,v), before eposing or choice. We obain he following solion 1 K a 1 K b = and v =. 1 K 1 1 K 1 Mari K depends on he inensi, and pariclarl on he spaial gradien of he inensi, which measres he spaial acivi of he image. We aribe o he covariance a vale proporional o he inner prodc of he inensi gradiens in he respecive direcions and o he prodc of he disances of he corresponding poins. This conjecre condcs o he following mari µ + K = µ + µ + + (resp. ) being he horizonal gradien (resp. verical). Scanning he frame (.,.;k) of order k in he seqence, one disposes he spaial gradien a he preceden frame (.,.;k-1). A possible choice o calclae he gradien in he mari K is he gradien a poin (i-1-(i,j), j-v(i,j) ; k-1), which is he preceden poin in he scanning displaced in he previos frame. Afer inversing mari K, and afer some approimaions, we obain he following predicor = f B + f C - f f D and in he same wa for v, where he coefficiens f and f depend on he gradiens, and on µ ( < f, f < 1) in he following wa f = µ + µ + + and f µ + = µ + +

4 f + is small in comparison wih µ, hen f f = 1, and he predicion relaion becomes = = B + C - D This relaion can be conneced o he spaial relaion on he veloci field nder a hpohesis of orhographic projecion and locall planar srface of he 3D objec 5. n fac, nder his hpohesis he veloci componens are epressed locall as linear fncions of he coordinaes and herefore he above spaial relaion is obained... Deecion of disconiniies in he moion esimaion n naral images eis independen 3D rigid movemens, so he global moion of he scene is no rigid. There also eis edges beween differen srfaces, which are sbjec o he same 3D rigid moion, b he have differen characerisics, and herefore a disconini resls in he D veloci field. The esimaion mehod being based on he inensi, he algorihm ma cases false displacemens, so a new iniializaion is necessar. An efficien esimaion of he veloci vecor necessiaes he conjoin deecion of all hese disconiniies. This deecion have o be based on he correcness of he a priori esimaion and conseqenl on he predicion error of he inensi. ms be based on he a priori esimaion, becase his epresses he spaial coherence of he veloci vecor. Two approaches are possible in he cone of he ransmission of image seqences. Before presening hem, le s define he predicion error e (i,j) = (i,k) - (i-, j-v ;k-1) 1. The predicion error is compared wih he difference beween he wo frames. f a disconini is deeced, he difference beween he wo frames is ransmied in he place of he predicion error, as well as he informaion ha a disconini is deeced. The a priori esimaion is hen rese o zero ( = v = ).. A casal neighborhood is considered and he prediced displacemen is applied. This operaion gives he following predicion error e (i,j) = Σ (i-n,j-m;k) - (i-n-, j-m-v ;k-1) (m,n) D (for eample, D = {(,1),(1,)} ). This predicion error is compared wih he difference beween he wo frames in he same neighborhood. f a disconini is deeced, he a priori esimaion is rese o zero, as in he firs approach. B in his case he ransmission of he deecion informaion is no necessar, becase he decoder can also carr o he deecion..3. A poseriori esimaion of he veloci vecor The a priori esimaion iniializes he esimaion algorihm of he veloci vecor. The a poseriori esimaion ilizes an algorihm of ieraive form. The esimaion error is epressed b e(i,j) = (i,k) - (i-, j-v;k-1) Le [ n-1 v n-1 ] T he esimaion of he veloci vecor a n-1 ieraion. The opimizaion is carried o b minimizing a qadraic crierion Q(,v) = e + λ[(- n-1 ) +(v-v n-1 ) ] The second par of his form can be considered as a reglarizaion or sabilizaion consrains. The form o minimize is no qadraic in repor o he nknown variables. To obain a qadraic form, we have o linearize he esimaion error. The Talor developmen of order 1 of (i,k) - (i-, j-v;k-1) gives (i-, j-v;k-1) = (i- n-1, j-v n-1 ;k-1) - (i- n-1, j-v n-1 ;k-1) (- n-1 ) - (i- n-1, j-v n-1 ;k-1) (v-v n-1 ).

5 Ths we obain e(i,j) = e n-1 (i,j) + (i- n-1, j-v n-1 ;k-1) (- n-1 ) + (i- n-1, j-v n-1 ;k-1) (v-v n-1 ) Using his approimaion, we obain he following solion n (i,j) = n-1 (i,j) - λ + ( i ( i, j v, j v ; k 1) e ; k 1) + ( i j), j v ; k 1) and in he same wa for v. The nmber of ieraions is eiher fied in advance, or wih a sopping es on esimaion error. 3. MOTON-ADAPTVE MAGE NTERPOLATON A echniqe o redce he rae of ransmission is o skip frames a he ransmier and inerpolae he skipped frames a he receiver. To avoid eiher jerkiness or blrring of he inerpolaed images, we have o adap he inerpolaor o he moion. We shall sd here he case of inerpolaing one frame beween ever wo frames, ha is o sa inerpolaing he frame of order k beween he frames of order k-1 and k+1. f [ is he veloci vecor a he poin (i, j), hen his poin is inerpolaed b (i,k) =.5*((i+, j+ v ;k+1)+(i-, j-v ;k-1)) The displacemen vecor is esimaed sing hese same frames of order k-1 and k+1. The esimaion is based on he error e(i,j) = (i+,j+v;k+1) - (i-, j-v;k-1) = ( (i, k+1)+ (i, k-1)) + ( (i, k+1)+ (i, k-1)) v + (i,k+1) - (i, k-1) The esimaor ma have he same srcre, as in he case of predicive coding. We develop i in he Secion 3.1. n he Secion 3. we presen anoher approach, which is moivaed b he fac ha large areas in man image seqences are no changed for a long ime. A mliresolion hierarchical esimaor is hen proposed eraive esimaion The srcre of he esimaor of Figre 1 is also sed here. The difference is ha he a priori esimaor ma be non-casal. For he firs componen of he veloci vecor we propose he following a priori esimaor v ] T 1 α (i,j) = α n 1 α (i,j) +.5(1-α)( n (i-1,j) + (i+1,j) + (i,j-1) + (i,j+1) ) α α α α The a poseriori esimaion is given b n (i,j) = (i,j) - ( k + 1) + k 1)) eα j) α λ + ( k + 1) + k 1)) + ( k + 1) + k 1)) n he same wa we obain he oher veloci componen. 3.. Hierarchical esimaion For no-moving areas or for areas homogeneos in he sense of veloci vecor, one can significanl redce he amon of calclaions o esimae he moion b dividing he frame ino blocks and b realizing he

6 esimaion in an hierarchical wa. Ths we inrodce a dnamic mliresolion on he veloci field sing 4- rees. f, for eample, he firs resolion grid is composed from blocks of 16*16 poins, a block, which is no homogeneos in he sense of he moion, is divided ino for blocks of 8*8 poins. We need hen a crierion of homogenei. The crierion we se verifies if he moion of a block can be considered as a wo-dimensional ranslaion. This es is realized on he esimaion error. Considering ha he movemen of a block is ranslaional, we can se a linear regression esimaor for he displacemen vecor componens ( ) ( ) = = where = (i,k+1)+ (i,k-1), = (i,k+1)+ (i,k-1) and = (i,k+1)-(i,k-1), he smmaion carried o over a block. Having his esimaion we es he homogenei hpohesis. The crierion sed here is based on he power of he inensi esimaion error. The cos fncion is defined as follows Q(,v ) = ((i+,j+ v ;k+1) - (i-, j- v ;k-1)) ) The decision concerning homogenei is obained b comparing he above qani o a hreshold. f he block is classified non-homogeneos, i is divided ino for blocks and he esimaion-deecion procedre is recommenced. 4. APPLCATONS We have applied he algorihms presened in his aricle, in wo seqences of real images. The firs one, named "CAR", represens a car in movemen, when he camera pans he scene. A srong addiional noise disrbs he inensi and differen pes of spaial deails are presen in he scene. The second one, named "GRL", represens a girl speaking and moderael moving. Boh seqences conain onl he lminance, coded in 8 bis. The horizonal gradien is obained b a filer wih a finie implse response given in he following mari F = The ransposed mari gives he verical gradien. This filer is composed of a low-pass filer o smooh he derivaive, and of a bilaeral derivaion. We have applied he algorihm of Secion in he case of he "CAR" seqence, wih µ = 3 and λ =. These vales have o be compared o he spaial acivi of he image, which can be measred b he mean of he sqare of he spaial gradien, which is abo 15. For he eleven firs frames of he seqence he mean of he absole vale of he difference beween wo sccessive frames were The mean of he absole vale of he displaced frame difference sing he prediced displacemen vecor was 11.81; sing he a poseriori esimaed displacemen vecor afer wo ieraions i was The percenage of deeced disconini poins was.739 %. The algorihm of Secion 3. is applied in he case of "GRL" seqence. For a mean of he absole vale of he difference beween wo images eqal o 3.53, he mean of he absole vale of he displaced frame difference was 1.8 and he mean of he absole vale of he inerpolaion error was Three levels of resolion were sed, he iniial consising of 16*16 blocks.

7 5. REFERENCES 1. J.Biemond, L.Looijenga, D.E.Boekee A pel-recrsive Wiener-based displacemen esimaion algorihm, Signal Processing, pp , Dec C.Cafforio, F.Rocca Mehods for measring small displacemens of elevision images, EEE Trans. on nformaion Theor, Vol. T-, No. 5, pp , Sep R.Moorhead, S.Rajala Moion-compensaed inerframe coding, Proc. CASSP, pp , H.G.Msmann, P.Pirsch, H.-J.Graller Advances in Picre Coding, Proc. of he EEE, Vol. 73, No. 4, pp , Apr G.Tzirias Esimaion of moion and srcre of 3D objecs from a seqence of images,1s EEE Conf. on Comper Vision, pp , London, Jne D.Walker, K.Rao mproved pel-recrsive moion compensaion, EEE Trans. on Commnicaions, Vol. COM-3, No. 1, pp , Oc