CE 395 Special Topics in Machine Learning Assoc. Prof. Dr. Yuriy Mishchenko Fall 2017
DIGITAL FILTERS AND FILTERING
Why filers? Digial filering is he workhorse of digial signal processing Filering is a fundamenal operaion of Deep Neural Neworks Digial filers are used o pre-process daa in many ML algorihms Digial filers are everywhere in speech recogniion and compuer vision Many more in essence, DSP is filers
Wha are filers? Filer is any procedure for changing a signal essenially a ransformaion Filer may ransform a 1D signal ime, signal filer or image spaial filer. For clariy we focus on 1D signals firs. Filer: signal x y x Filer y
Linear filers Linear filer is such a filer ha ransform signal in linear manner. Tha is, he oupu for a superposiion of wo inpu signals will be he superposiion of he individual oupus of hose wo inpus. ' ' : ' and ' : z x z x F z z x x F
Linear filers Dela funcion or impulse signal is a signal which equals o zero a all imes excep 0=0: δ0=0 everywhere excep =0, and δ0=1 we wrie δ-0, o be precise δn
Linear filers Fac: Any inpu signal can be represened as a sum of impulses. 0 x x 0 0 Arbirary signal as a sum of impulses:
Linear filers Fac: Effec of filer on an impulse inpu signal a ime 0 is he response funcion h 0 ] [ ] [ 0 0 0 0 0 0 x h x x F h F
Linear filers Imagine a signal as a sum of impulses a =0,1,2 Each impulse is ransformed o independen ime series h 0=0, h 0=1, h 0=2, h: / -2-1 0 1 2 0 1 2 4 3 2 1 0 1 2 2 0 2 0 0 2 2 0 In he end, by lineariy hese are summed ogeher! x : 0 1 2 =0 [ 0 1 2 4 3 2 0 ]*x0 =1 [ 0 0 1 2 2 0 0 ]*x1 =2 [ 0 0 0 2 2 0 0 ]*x2 Σ: [ 0 1 3 8 7 2 0 ] = y : -3-2 -1 0 1 2 3 ie h 0 ie h 1 ie h 2
Linear filers as sum of responses: x, 1 5 The filer oupu hick line is he sum of individual responses o x0 a imes 0=1..5 hin lines. h ' sin 1 1 sin / ' y y x h 0 0 0
LTI filers Linear filer is called ime-invarian LTI if is response is : h 0 h 0
LTI filers The main propery of ime-invarian filers is F [ x ] F [ x ] 0 0 Reads: shifing inpu by 0 only resuls in idenical shif in he oupu signal
LTI filers Using he definiion propery of linear filers, hen we obain for ime-invarian filers he following formula: F [ x ] h x h x 0 0 0 0 0 LTI 0
LTI filers The oupu of LTI filer is a superposiion of single, ime-invarian response funcion h for each impulse x0 from he inpu F LTI [ x ] h x 0 0 0
Example of evaluaion of LTI filer: -2-1 0 1 2 h 0 1 2 3 4 LTI response funcion x : 0 1 2 =0 [ 0 0 1 2 3 4 0 ]*x0 h-0 =1 [ 0 0 0 1 2 3 4 ]*x1 h-1 =2 [ 0 0 0 0 1 2 3 ]*x2 h-2 Σ: [ 0 0 1 4 10 16 17] = y : -3-2 -1 0 1 2 3
Operaion of Convoluion The peculiar mahemaical operaion ha we saw in he ime-invarian filers has a general significance and is called in mahemaics convoluion: ' ' ' * y x y x 0 ] [ 0 0 LTI x h x F
Operaion of Convoluion Convoluion and ime-invarian filers are basically he same hing: You can visualize convoluion as summing up of h-0 responses riggered by each incoming pulse x0
Operaion of Convoluion This filer example from before is indeed a convoluion: -2-1 0 1 2 h 0 1 2 3 4 Convoluion h x : 0 1 2 =0 [ 0 0 1 2 3 4 0 ]*x0 h-0 =1 [ 0 0 0 1 2 3 4 ]*x1 h-1 =2 [ 0 0 0 0 1 2 3 ]*x2 h-2 Σ: [ 0 0 1 4 10 16 17] = y = h*x : -3-2 -1 0 1 2 3
Operaion of Convoluion There may exis differen mehods for acually evaluaing convoluion: Sliding response mask previous slide Sliding invered mask nex slide Diagonal mehod [consruc a able of xy for all and, hen sum he values on he couner-diagonals + =τ zτ verify a home ha ha indeed gives he convoluion]
Convoluion calculaion via diagonals: -4-3 -2-1 0 1 2 3 4 x 0 0 0 0 1 2 3 0 0 y 0 1 2 3 4 0 0 0 0 z 0 4 6 6 18 z 1 8 9 z 2 2 2 4 x\y =-3 =-2 =-1 =0 =0 1 2 3 4 =1 2 4 6 8 =2 3 6 9 12 τ=-2 τ=0 τ=1 z 0 1 4 10 16 17 12 0 0
Convoluion calculaion via invered mask: -4-3 -2-1 0 1 2 3 4 x 0 0 0 0 1 2 3 0 0 y 0 1 2 3 4 0 0 0 0 y- 0 0 0 0 4 3 2 1 0 z 0 1 4 10 16 17 12 0 0 Invered mask, slide lef righ z 0 x k y 0 k k x 1 y 1 x 0 y 0 k x k y k x 1 y 1 x 2 y 2 z 1 x k y 1 k x 1 y 0 x 2 y 1 k z 2 x k y 2 k x 2 y 0 k y-4-12 y- x y3- x 3 y 3 =2*4+3*3=17 x 4 y 4 =1*4+2*3+3*2=16 sliding invered mask
Main properies of convoluion For a DSP engineer he mos imporan hing is o know and undersand he properies of convoluion. All of hese can be shown direcly from he definiion exercise for home. * * x h h x Commuaive * * * g x h x g h x Disribuive * * * * g h x g h x Associaive * x x * 0 0 x x Ideniy Time-shif
LTI filers FIR filers are such filers whose response h is non-zero only in a small region around =0. Such filers are also ofenimes called masks. The resul of applying an FIR filer reduces o summing he signal x wih a sliding response mask h or convolving i wih invered mask h-, see previous slides for examples
LTI filers LTI filer is called causal if response h=0 for all <0. Tha is, he signal can only affec he filered signal y in he fuure F [ x ] h x 0 0 0 0 For example, he filer in slide 18 is no causal, because x affecs y-1. However, if we se h-1=0, hen ha filer would be causal.
Causal and non-causal filers -2-1 0 1 2 h 0 1 2 3 0 Response funcion Noe x0 affecs y-1 herefore his filer is no causal. Seing h-1=0 on he oher hand resuls in causal filer. x : 0 1 2 =0 [ 0 0 1 2 3 0 0 ]*x0 h-0 =1 [ 0 0 0 1 2 3 0 ]*x1 h-1 =2 [ 0 0 0 0 1 2 3 ]*x2 h-2 Σ: [ 0 0 1 4 10 12 9] = y : -3-2 -1 0 1 2 3
Spaial filers Typical filers used in image processing are FIR filers ha can be viewed as a 2D response mask ypically square being moved around he image, muliplied wih image pixel values, and hen summed o define he new pixel values
Spaial filers See more deails abou how a spaial filer works in hese grea animaions on youube! hps://www.youube.com/wach?v=gfelyrix010 hps://www.youube.com/wach?v=hrmsh2gf48
Sysem represenaions of filers Examples of sysems:
Sysem represenaions of filers Sysem can be simplisically viewed as definiions of sequences of processing seps for ransforming inpu signal x ino an oupu signal y, x y As we can see, his definiion is no much differen from wha we said abou filers - In fac, sysems are filers in many respecs!
Sysem represenaions of filers The difference beween sysems and filers is he poin of view: sysems are viewed primarily as sequences of operaions or machines, while filers are reaed more as mahemaical ransformaions Respecively, sysems are naurally represened by diagrams of signal processing seps, while filers are expressed by mahemaical formulas
Sysem represenaions of filers Sysem diagrams represen he sequence of operaions o be carried ou on an inpu signal o obain he oupu, in a diagram form Main elemens of sysem diagrams are elemenary operaions +,-,*, unary -x, signal branching copy, and ime delays labeled by Z -1 means x[-1] Addiionally, sysems may use oher sysems as building blocks
Example of reading a sysem diagram Copy x[n] ino wo branches h1 and h2 Signal in branch h2 is muliplied wih b 0 Signal in branch h1 is ime-delayed by 1 sep, x[n-1] Thus ime-delayed signal h1 is copied hree ways v1, v2 and v3 Signal v2 is furher ime delayed by 1 sep x[n-2] and hen copied wo ways u1 and u2 Signal in branch u1 is muliplied by a 2 Signal in branch u2 is muliplied by b 2 Signal in branch v1 is muliplied by a 1 and added o u1 Signal in branch v3 is muliplied by b 1 and added o u2 Signal in branch u1 is added back o inpu x[n] feedback Signal in branch u2 is added o h2, hus forming he resul y[n] o be served as oupu h2 h1 u1 v1 v2 v3 u2
Sysem represenaions of filers Sysems can be inerconneced o creae more complex sysems: y s 2 s1 x s 2 s1 x series: oupu of sysem 1 becomes inpu o sysem 2 y s1 x s 2 x s1 s 2 x parallel: afer processing signal over wo parallel branches resuls are plus ed ogeher y s 4 s 2 s1 x s 3 x NOW TRY AND OBTAIN THIS
Sysem represenaions of filers Exremely imporan is feedback inerconnecion: oupu of a sysem is used back as is inpu possibly afer addiional processing Shown feedbacks is x y =S 1 x+s 2 y
Sysem represenaions of filers Try o design and draw he sysem diagrams for following sysems or digial filers: Differeniaor filer D 1 xx-x-1=y Inegraor filer S 1 xx+y-1=y
Sysem represenaions of filers For us he imporan poin is ha one can use sysems language o design digial filers This in fac is a filer: In sysems language, we can combine muliple filers in a simple way, parallel, series, or feedbacks, o creae very complex filers!
Sysem represenaions of filers y F [ F [ x ]] 2 1 y F [ x ] F [ x ] 1 2 Series: A new filer is obained by applying filer F1 and hen F2 o F1 s resul Parallel: A new filer is obained by applying F1 and F2 and hen summing heir oupus Now ry and explain wha kind of filer his las diagram is
Sysem represenaions of filers Like a filer, a sysem is called linear if s x x s x s x 1 2 1 2 A sysem is called ime-invarian if is characerisics don change over ime, ha is s x y 0 0
Sysem represenaions of filers A sysem is LTI Linear Time-Invarian if i is linear and ime invarian a he same ime LTI sysems always can be described by convoluion jus like he LTI filers ' * ' ' h x h x x s
Helpful inerconnecion ideniies for LTI sysems/filers * * 1 2 2 1 s s s s Commuaive * * * 3 1 2 1 3 2 1 s s s s s s s Disribuive * * * * 3 2 1 3 2 1 s s s s s s Associaive * x x Ideniy Follow from he properies of convoluion now
Digial Signal Processing Original usage of filers: Modern usage of filers - Digial Signal Processing DSP
Digial Signal Processing
Banks of filers Digial Signal Processing Feaures for ML algorihms
Digial Signal Processing Deec edges using a FIR image filer
Digial Signal Processing Banks of muli-scale filers Across scales feaures for ML algorihm
Digial Signal Processing Muli-scale feaures for ML algorihms A Convoluional Neural Nework
Digial Signal Processing
Digial Signal Processing Problem: design a filer ha has required properies for example responds srongly o verical edges in an image There are powerful sofware ools for ha you can learn hose when/if you need hem
Digial filers represenaion Problem: how can we calculae a resul of filering a signal on a compuer? mos general form
Digial filers represenaion Direc form 1: 1 n a 1 n 1 a 2 Z... a n 1 Z y n b 1 b 2 Z... b n 1 Z b x n a b Z -1 indicaes ime-shif Z 1 x n x n 1 filer order
Digial filers represenaion Direc form 1: 2 nd order filer in direc form 1
Digial filers represenaion Direc form 2: Try and follow his sysem diagram o see how his ype of filers works. 2 nd order filer in direc form 2
Digial filers represenaion Digial filers in direc form may be unsable for evaluaion because of numerical overflows in he long chains of addiions of possibly very differen scale-numbers his problem is ypically exponenially bad in he filer order N
Digial filers represenaion SOS Second Order Secions biquadraic cascade filer represenaion form Theorem: Any digial filer can be represened as 2 nd order filers cascade. SOS Direc form 1
Digial filers represenaion SOS filers have much beer numerical sabiliy and are he preferred form of represenaion for digial filers
Digial filers represenaion Transfer funcion represenaion of filers we will come back o afer learning abou Fourier ransform
Nonlinear filers A nonlinear filer conains a nonlinear ransformaion in addiion o linear filering Very differen ways o implemen, acive area of research Some examples: ' ' ' x h f y ' ' ' x f h y Bx Ay f y
Some imporan DSP filers Low pass suppress high frequencies in signals or images resuling in generally smooher signal High pass enhance high frequencies in signals or images resuling in generally sharper looking feaures in signals wih emphasize on edges Band pass allow frequencies in signals or images in cerain range band o pass Band sop Noch remove frequencies in signals or images in cerain range
Some imporan DSP filers FILTER FILTER FILTER
Some imporan DSP filers Gaussian low pass or smoohing filer Band pass filer Comb filer Noch band sop filer
Some imporan DSP filers FIR Finie Impulse Response Filers FIR filers have finie span of he response funcion h and can conain no feedback, reduce o mask applicaion over signal IIR Infinie Impulse Response Filers IIR filers can have infinie span of response funcion h and are he general filers wih feedback
Some imporan DSP filers Mos filers used in spaial or image filering are FIR filers and can be described by a 2D, usually square filer mask being applied over local image paches o produce filered image s pixel values Gabor edge filer mask Gaussian smoohing filer mask
Some imporan DSP filers We have alked abou how such spaial masks are applied earlier. Gabor edge filer mask
Some imporan DSP filers Averaging, smoohing and denoising: 3x3 average/mean 3x3 median
Some imporan DSP filers Sharpening and edge deecors:
Some imporan DSP filers Simple edge deecors: Sobel, Prewi, Robers, Gradien, Laplacian, Gradien of Gaussian GoG Ridge deecors - Laplacian, Laplacian of Gaussian LoG Laplacian of Gaussian filer
Some imporan DSP filers Texure deecors Gabor filers, wavele filers Hough Transform used o deec shapes in images such as lines and circles Gabor
QUESTIONS FOR SELF-CONTROL
Define wha a filer is. Wha is linear filer? Wha do we mean when we say ha linear filer is a sum of filer responses? Define LTI filer. Define causal filer. Define he operaion of convoluion. Lis main properies of convoluion. Define he hree main sysem inerconnecion ypes. Give definiion of LTI sysem. Draw differeniaor and inegraor sysems, are hese LTI? In wha sense do we say ha LTI sysems and linear filers are equivalen? Wha is filer bank and how is i used for image processing?
Define direc forms 1 and 2. Define wha order of filer is. Wha would be he order of he filer corresponding o he inegraor sysem you consruced in a previous quesion? Define SOS filer. Explain why SOS filers are beer for digial signal processing han direc form filers? Define nonlinear filer. Define low pass, high pass, band pass, and band sop filers. Wha is he difference beween FIR and IIR filers? Explain how spaial filering by a filer mask works on he example of an edge deecor image filer.
ADVANCED
Ideal low pass filer Signal reconsrucion by ideal low pass filer can be viewed as a convoluion of sampled signal wih sinc funcion x x nt sinc nt x * sinc n
2D convoluion Alhough we considered only 1D convoluion, i can be generalized o any number of dimensions ' ' ', ' ', ', *, x y y y x x h y x f y x h y x f
Some general properies of sysems Memory sysems wih memory produce oupu depending on many pas values from x no jus he curren value Inverabiliy an inverse sysem g exiss for a sysem s if heir series inerconnecion produces ideniy g*s=δ, we say g=s -1 Causaliy a sysem is causal of is oupu a ime depends only on he pas values of x, < Sabiliy a sysem is sable if is oupu is bounded for any finie inpu x
Some properies of LTI sysems implies LTI mus have memory he only memory-less LTI sysem is he ideniy sysem δ ' * ' ' h x x h y
Some properies of LTI sysems LTI sysem h is inverible if here is anoher sysem g such ha g*h=δ he resoluion of his problem is obained in he heory of Fourier ransform nex lecure
Some properies of LTI sysems LTI sysem is sable if is oupu is bounded in magniude for every bounded inpu. This implies and requires sufficien and necessary: ' h '
Some properies of LTI sysems LTI is causal iff h=0 for all <0 0 ' * ' ' h x h x y h u LTI 0 ' ' ] [ Sep response of causal LTI sysem:
Some properies of LTI sysems Dela or impulse signal and uni sep signal 0 1, 0 0, 0 1, 0 0, u h y ' ' h y Response of LTI sysem o dela/impulse and sep signal
LTI sysems and dynamic sysems Many causal LTI can be described by differenial or finie difference equaions - such equaions describe effecively how he oupu of he sysem is compued: dy d Ay Bx y Ay Bx 1 s order LTI d 2 d y 2 A 1 dy d A 0 y Bx 2 nd order LTI y A y A y Bx 1 0
LTI sysems and dynamic sysems Quesion: Can any causal LTI sysem be described via a differenial equaion?
Low pass filers Buerworh smooh pass band, slow cuoff roll-off Chebyshev ype 1 smooh sop band, faser cuoff Chebyshev ype 2 smooh pass band, faser cuoff Bessel - no group delay, all smooh, very slow cuoff Gaussian no ripples in all bands, very slow Ellipic ripples in all bands, very fas roll-off ripples pass band sop band Gain or ransfer funcion describes by how much srengh of each frequency componen changes in he signal afer filering
High pass digial signal filers High pass filer removes low frequency componens from signal sharpening Essenially i is a 1 low pass filer ha is same design opions as in he las slide Sharp edges in signals and images generally are composed of very high frequencies needed o achieve fas rise! Therefore, high pass filers can be used o exrac or enhance he edges.
Band pass digial signal filers Band pass filer allows frequency componens in cerain bands o pass Band pass filer can be represened as a low and high pass filer in series, bu generally needs o be designed separaely per pass-band specificaions, as shown below!
Band sop digial signal filers Band sop filer sops frequency componens in cerain band - exac opposie of band pass Band sop filers are also called noch filers Comb filer
Some addiional ypes of image filers hiseq Poin ransformaions Conras adjusmen BW-hresholding Complemen inversion Hisogram equalizaion Denoising smoohing Average filer, median filer, Gaussian filer Sharpening Laplacian Geomeric ransforms Translaion, roaion, scaling, affine
Some addiional ypes of image filers Morphologic filers or operaions are nonlinear ransformaions of images used o implemen various miscellaneous ransformaions. Examples of morphological operaions include closing, opening, haopening, dilaion, erosion, and many oher.