Coes 1 Temporal filers 1 1.1 Modelig sequeces 1 1.2 Temporal filers 3 1.2.1 Temporal Gaussia 5 1.2.2 Temporal derivaives 6 1.2.3 Spaioemporal Gabor filers 8 1.3 Velociy-ued filers 9 Bibliography 13
1 Temporal filers Alhough addig ime migh seem like a rivial exesio from 2D sigals o 3D sigals, ad i may aspecs i is, here are some properies of how he world behaves ha make sequeces o be differe from arbirary 3D sigals. I 2D images mos objecs are bouded occupyig compac ad well defied image regios. However, i sequeces, objecs do o appear ad disappear isaaeously uless hey ge occluded behid oher objecs or eer or exi he scee hrough doors or he image boudaries. So, he behavior of objec across ime is very differe ha heir behavior across space, m. I ime, objecs move ad deform defiig coiuous rajecories ha have o begiig ad ever ed. 1.1 Modelig sequeces Sequeces will be represeed as fucios f (x, y, ), where x, y are he spaial coordiaes ad is ime. As before, whe processig sequeces we will work wih he discreized versio ha we will represe as f [, m, ], where, m are he pixel idices ad is he frame umber. Discree sequeces will be bouded i space ad ime, ad ca be sored as arrays of size N M P. Figure 1.1 illusraes his wih oe sequece show i fig. 1.1.a. This sequece has 90 frames ad shows people o he sree walkig parallel o he camera plae ad a differe disaces from he camera. Fig. 1.1.b shows he space-ime array f [, m, ]. Whe we look a a picure we are a lookig a a 2D secio, = cosa, of his cube. Bu i is ieresig o look a secios alog oher orieaios. Fig. 1.1.c ad d show secios for m = cosa ad = cosa respecively. Alhough hey are also 2D images, heir srucure looks very differe from he images we are used o seeig. Fig. 1.1.c shows a horizoal secio ha is parallel o he direcio of moio of he people walkig. Here we see sraigh bads wih differe orieaios. This bads appear o occlude each oher. Each bad correspods o oe perso ad is orieaio is give by he speed of walk ad he direcio of moio. Fig. 1.1.d looks like a foo-fiish phoograph as he oes used i sporig races. I boh images (d) ad (c), saic objecs appear as verical sripes i (b), ad horizoal sripes i (d).
LATEX Book Syle MITPress NewMah.cls Size: 7x9 Sepember 27, 2018 2 Chaper 1 12:04am Temporal filers a) m m b) c) d) Figure 1.1 a) 8 Frames from a sequece wih people walkig. The frames are show a regular ime iervals. The full sequece had 90 frames (correspodig o 3 secods of video). b) Space-ime array, f [, m, ] of size 128 128 90. c) Secio for m = 50, d) Secio for = 75. Saic objecs appear as sraigh lies. Oe special sequece is whe he image has a global moio wih cosa velociy (v x, vy ). I such a case we ca wrie: f (x, y, ) = f0 (x v x, y vy ) (1.1) where f0 (x, y) = f (x, y, 0) is he image beig raslaed, ad v x ad vy are cosas. We use coiuous fucios because i allows us o deal wih ay velociy values. This fucio assumes also ha he brighess of he pixels does o chage while he scee is movig (cosa brighess assumpio). The FT of his globally movig image is: F(w x, wy, w ) = F0 (w x, wy )δ(w v x w x vy wy ) (1.2) The coiuous FT of he sequece is equal o he produc of he 2D FT of he saic image f0 (x, y) ad a dela wall. To beer udersad his fucio le s look a a simple example o oly oe spaial dimesio, as show i figure 1.2. Figure 1.2 shows he FT for a sequece wih oe spaial dimesio, f (x, ), ha coais a blurry recagular pulse movig a hree differe speeds owards he lef. Fig. 1.2.a shows a sequece whe he pulse is saic ad fig. 1.2.d shows is FT. Across he spaial frequecy w x, he FT is approximaely a sic fucio. Across he emporal frequecy w, as he sigal is cosa, is FT is a dela fucio. Therefore he FT is a sic fucio coaied iside a dela wall i he lie w = 0. Fig. 1.2.b shows he same recagular
1.2 Temporal filers 3 vx = 0 vx = -0.5 vx = -1 x x a) b) c) w w wx wx w x wx d) e) f) Figure 1.2 a) A sequece wih oe spaial dimesio showig a saic recagular pulse. b) The recagular pulse moves o he lef a a speed v = 0.5 ad c) movig owards he lef, v = 1. As we work wih discreized sigals, speed uis are i pixels per frame. pulse movig owards he lef, v x = 0.5. Fig. 1.2.e shows he sic fucio bu skewed a log he frequecy lie w + 0.5w x = 0. Noe ha his is o a roaio of he sic fucio from fig. 1.2.d as he locaios of he zeros lie a he same w x locaios. Fig. 1.2.c shows he pulse movig a a faser speed resulig i a larger skewig of is FT, Fig. 1.2.f. I geeral, sequeces will be more complex, bu he properies of a globally movig image are helpful o udersad local properies i sequeces. We ca also wrie models for more complex sequeces. For isace, a sequece coaiig a movig objec over a saic backgroud ca be wrie as: f (x, y, ) = b(x, y)(1 m(x v x, y v y )) + o(x v x, y v y )m(x v x, y v y ) (1.3) where b(x, y) is he saic backgroud image, o(x, y) is he objec image movig wih speed (v x, v y ), ad m(x, y) is a biary mask ha moves wih he objec ad ha models he fac ha he objec occludes he backgroud. We le o he reader he work of compuig is FT ad visualizig he effec of he mask i he Fourier domai. 1.2 Temporal filers Liear spaio-emporal filers ca be wrie as spaio-emporal covoluios bewee he ipu sequece ad a covoluioal kerel (impulse respose). Discree spaioemporal filers have a impulse respose h [, m, ]. The exesio from 2D filer o spaio-emporal
4 Chaper 1 Temporal filers filers does o has ay addiioal complicaios. We ca also classify filers as low-pass, high-pass, ec. Bu i he case of ime, here is aoher aribue used o characerize filers: causaliy. Causal filers: hese are filers wih oupu variaios ha oly deped of he pas values of he ipu. This pus he followig cosrai: h [, m, ] = 0 for all < 0. This meas ha if he ipu is a impulse a = 0, he oupu will oly have o-zero values for > 0. If his codiio is saisfied, he he filer oupu will oly deped o he ipu s pas for all possible ipus. No-causal filers: whe he oupu has depedecy o fuure ipus. Ai-causal filers: his is he opposie, whe he oupu oly depeds o he fuure: h [, m, ] = 0 for all > 0. May filers are o-causal ad have boh causal ad ai-causal compoes (e.g., a Gaussia filer). Noe ha o-causal filers ca o be implemeed i pracice ad, herefore, ay filer wih a ai-causal compoe will have o be approximaed by a purely causal filer by boudig he emporal suppor ad shifig i ime he impulse respose. I his chaper, we have wrie all he filers as covoluios. However, some filers are beer described as differece equaios (his is specially impora i ime). A example of a differece equaio is: g [, m, ] = f [, m, ] + α h [, m, 1] (1.4) where he oupu g a ime depeds o he ipu a ime ad he oupu a he previous ime isa 1 muliplied by a cosa α. We ca easily evaluae he impulse respose, h [, m, ], of such a filer by replacig f [, m, ] wih a impulse, δ [, m, ]. The impulse respose is: h [, m, ] = α δ [, m] u [] (1.5) where u [], called he Heaviside sep fucio, is: 0 if < 0 u [] = 1 oherwise (1.6) Mos filers described by differeces equaios have a impulse respose wih ifiie suppor. They are called IIR (Ifiie Impulse Respose) filers. IIR filers ca be furher classified as sable ad usable. Sable filers are he oes ha give a bouded ipu, f [, m, ] < A, produce a bouded oupu, g [, m, ] < B. For his o happe, he impulse respose has o be bouded. I usable filers, he ampliude of he impulse respose diverges o ifiiy. I he previous example, he filer is sable if ad oly if α < 1. Le s ow describe some spaio-emporal filers.
1.2 Temporal filers 5 v=0 v=-1 v=1 5 5 5-5 -5-5 a) -5 5 b) -5 5 c) -5 5 Figure 1.3 a) Spaio-emporal Gaussia wih σ = 1 ad σ = 4. b) Same gaussia parameers bu skewed by he velociy vecor v x = 1, v y = 0 pixels/frame, c) ad v x = 1, v y = 0 pixel/frame. 1.2.1 Temporal Gaussia As wih he spaial case, we ca defie he same low-pass filers: he box filer, riagular filers, ec. As a example, le s focus o he Gaussia filer. The spaio-emporal gaussia is rivial exesio of he spaial gaussia filer we have see i secio??: g(x, y, ; σ x, σ ) = 1 exp x2 + y 2 exp 2 (2π) 3/2 σ 2 xσ 2σ 2 x 2σ 2 (1.7) Where σ x is he widh of he gaussia alog he wo spaial dimesios, ad σ is he widh o he emporal domai. As he uis for ad x, y are urelaed, i does o make sese o se all he σs o have he same value. We ca discreize he coiuous gaussia by akig samples ad buildig a 3D covoluioal kerel. We ca also use he biomial approximaio. The 3D Gaussia is separable so i ca be implemeed efficiely as a covoluioal cascade of 3 oe dimesioal kerels. Figure 1.3.a shows a spaio-emporal gaussia. The emporal gaussia is a o-causal filer, herefore i is o physically realizable. This is o a problem whe processig a video sored i memory. However, if we are processig a sreamed video, we will have o boud ad shif he filer o make i causal which will resul i a delay i he oupu. Figure 1.4.a shows oe sequece ad figure 1.4.b shows he sequece filered wih he gaussia from figure 1.3.a.This Gaussia has a small spaial widh, σ = 1, ad a large emporal widh, σ = 4 so he sequece is srogly blurred across ime. The movig objecs show moio blur ad are srogly affeced by he emporal blur, while he saic backgroud is oly affeced by he spaial widh of he gaussia. How could we creae a filer ha keeps sharp objecs ha move a some velociy (v x, v y ) while blurrig he res? Figure 1.4.c shows he desired oupu of such a filer. The boom image shows oe frame for a sequece filered wih a kerel ha keeps sharp objecs movig lef a 1 pixel/frame while blurrig he res. This filer ca be obaied by skewig he
6 Chaper 1 Temporal filers a) b) c) d) Figure 1.4 a) Oe frame from he ipu sequece ad he space-ime secio (o op). b) Oupu whe covolvig he he gaussia from fig. 1.3.a. c) Oupu of he covoluio wih fig. 1.3.b, ad d) oupu of he covoluio wih fig. 1.3.c. gaussia: g vx,v y (x, y, ) = g(x v x, y v y, ) (1.8) This direcioal blur is o a roaio of he origial Gaussia as he chage of variables is o uiary, bu he same effec could be obaied wih a roaio. Figure 1.4.c shows he effec whe v x = 1, v y = 0. The gaussia is show i fig. 1.3.b. The space-ime secio shows how he sequece is blurred everywhere excep oe orieed bad correspodig o he perso walkig lef. Figure 1.4.d shows he effec whe v x = 1, v y = 0. The oupu of his filer looks as if he camera was rackig oe of he objecs while he shuer was ope, producig a blurry image of all he oher objecs. 1.2.2 Temporal derivaives Spaial derivaives were useful o fid regios of image variaio such as objec boudaries. Temporal derivaives ca be used o locae movig objecs. We ca approximae a emporal derivaive for discree sigals as: f [m,, ] f [m,, 1] (1.9) As i he spaial case, i is useful o compue emporal derivaives of spaio-emporal gaussias: g = g(x, y, ) (1.10) σ 2
1.2 Temporal filers 7 a) b) c) d) =-3 =-2 =-1 =0 =1 =2 =3 Figure 1.5 Visualizaio of he space-ime Gaussia. The gaussia has a widh of σ 2 = σ 2 = 1.5, ad has bee discreized as a 3D array of size 7 7 7. Each image shows oe frame. a) Gaussia b) The parial derivaive of he Gaussia wih respec o. c) Derivaive alog v = (1, 0) pixels/frame. d) v = ( 1, 0) pixels/frame. where g(x, y, ) is he Gaussia as wrie i eq. 1.7. We ca compue he spaio-emporal gradie of a gaussia: g = ( g x (x, y, ), g y (x, y, ), g (x, y, ) ) = ( ) x/σ 2, y/σ 2, /σ 2 g(x, y, ) (1.11) Wha should we do if we wa o remove oly he objecs movig a a paricular velociy? I he case of a movig image wih velociy (v x, v y ), he sequece is f (x, y, ) = f 0 (x v x, y v y ), we ca compue he emporal derivaive of f (x, y, ) as: f = f 0 f 0 = v x x v f 0 y y If we compue he gradie of he gaussia alog he vecor [ 1, v x, v y ] : (1.12) h(x, y, ; v x, v y ) = g + v x g x + v y g y = g ( 1, v x, v y ) T (1.13) ad we covolve i wih f 0 (x v x, y v y ) we ge a zero oupu (usig eq. 1.12): f 0 (x v x, y v y ) h = f 0 (x v x, y v y ) ( g + v x g x + v y g y ) = ( f0 + v x f 0 x + v y f 0 y ) g = 0 (1.14)
8 Chaper 1 Temporal filers v=0 v=-1 v=1 a) b) c) d) Figure 1.6 Spaio-emporal Gaussia g [, ] ad derivaives. a) Gaussia wih σ 2 = 1.5. b) Parial derivaive wih respec o. c) Parial derivaive alog (1, 1), d) Parial derivaive alog (1, 1). a) b) c) d) Figure 1.7 a) Ipu sequece. b) v x = v y = 0, c) v x = 1 pixels/frame. d) v x = 1 pixel/frame. The filer h is show i figure 1.5 as a sequece for differe velociies. I he example show i he figure, he gaussia (fig. 1.5.a) has a widh of σ 2 = σ 2 = 1.5, ad has bee discreized as a 3D array of size 7 7 7. Figures b,c ad d show he filer h for differe velociies: (v x, v y ) = (0, 0), (1, 0) ad ( 1, 0). Fig. 1.6 shows he correspodig spaceime secios of he same spaio-emporal gaussia derivaives. Boh visualizaios are equivale ad help o udersad how he filer works. Such a filer will cacel ay objecs movig a he velociy (v x, v y ). By usig differe filers, each oe compuig derivaives a log differe space-ime orieaios, we 1.2.3 Spaioemporal Gabor filers Jus as we did wih Gaussia derivaives, exedig Gabor filer for moio aalysis is a direc geeralizaio of he x-y 2D Gabor fucio o a x-y- 3D Gabor fucio. Figure 1.8.a
1.3 Velociy-ued filers 9 w w w y x w x w x x a) b) c) Figure 1.8 Space-ime Gabor filers. a) Cosie ad sie x- Gabor filer, ad b) he skech of is rasfer fucio. c) Skech of he rasfer fucio of a spaioemporal Gabor filer i 2 spaial dimesios (x-y-). shows a x- (cosie ad sie) Gabor fucio i 1 spaial dimesio, ad Figure 1.8.b shows a skech of is Fourier rasform. This fucio is selecive o sigals raslaig o he righ wih a speed v = 1, i.e. i(x ). The red lie i Figure 1.8.b shows Dirac lie ha coais he eergy of he movig sigal. I 2 spaial dimesios, Figure 1.8.c shows he skech of he Gabor rasfer fucio. Noe ha he x-y- Gabor filer is o selecive o velociy. If we have a 2D movig sigal i(x v x, y v y ), he Fourier rasform is coaied iside a Dirac plae. Therefore, here are a ifiie umber of plaes ha will pass by he frequecies of he Gabor filer. All hose plaes iersec he red lie show i Figure 1.8.c. A sigle Gabor filer ca o disambiguae he ipu velociy. Equaio of he velociy as a fucio of he frequecy uig. 1.3 Velociy-ued filers How ca we measure ipu velociy? There are may differe approaches i he compuer visio commuiy for measurig moio. Here we show ha i is possible o measure moio eve wih he simple processig machiery ha we ve developed so far. We ca use quadraure pairs of orieed filers i space-ime o fid moio speed ad direcio i he video sigal. We jus eed o fid he space-ime orieaio of sroges respose. Figure 1.8.a shows a se of Gabor filers samplig he space-ime frequecy
10 Chaper 1 Temporal filers w w w y w y w x w x a) b) Figure 1.9 a) Space-ime Gabor filers iles. b) Se of Gabor filers selecive o a paricular velociy. domai. Whe he ipu coais a movig sigal, we ca use a se of filers o ideify he plae i he Fourier domai ha coais he ipu eergy. Figure 1.8.b shows he subse of filers ha have he sroges oupu for a paricular ipu moio. As a illusraio, Figure 1.10 shows oe possible archiecure o creae velociy-selecive uis. The firs layer is composed by Gabor filers (cosie ad sie) which are frequecyselecive uis. The oupus are combied accordig o differe plaes i he Fourier domai o creae velociy-selecive oupus. Give a ipu sequece, oe ca esimae velociy by lookig a he velociy-ued ui wih he sroges respose.
1.3 Velociy-ued filers 11 Ipu sequece y x Frequecy ued layer (.) 2 + (.) 2 (.) 2 (.) 2 (.) 2 + + (.) 2 (.) 2 (.) 2 (.) 2 (.) 2 (.) 2 + + + (.) 2 (.) 2 + (.) 2 (.) 2 + (.) 2 Velociy ued layer + + + + + + + + + + + + + + + + + + Figure 1.10 Archiecure o creae velociy-selecive uis. The firs layer is composed by space-ime Gabor filers (cosie ad sie) which are frequecy-selecive uis. Here we represe he impulse respose of each filer by a small x-y- cube. For each quadraure pair we compue he ampliude. The ampliude oupus are combied accordig o form differe plaes i he Fourier domai o creae velociy-selecive oupus. A ormalizaio layer ca be added o ormalize he oupus by dividig every oupu by he sum of all he ampliudes (o show). The full archiecure is o-liear.
Bibliography