Fas Space varyng Convoluon Fas Marx Vecor Mulplcaon l and FMRI Acvaon Deecon Janng We Advsors: Prof. Jan P. Allebach Prof. Ilya Pollak Prof. Charles A. Bouman Dr. Peer A. Jansson School of Elecrcal and Compuer Engneerng Purdue Unversy Augus 6 009
Fas Space-varyng Convoluon and Is Applcaon n Sray Lgh Reducon
Space-varyng Convoluon Space-varyng convoluon: convoluon wh a spaally varan pon spread funcon. Space-nvaran convoluon can be compued usng FFT. Space-varyng convoluon canno use FFT so s compuaonally expensve. Space-varyng convoluon fnds applcaons n sray lgh reducon Bls e. al. 007 and We e. al. 008 aberraon correcon Lam 003 mcroscopc magng Shaevz and Flecher 007 Our obecve: compue space-varyng convoluon much faser wh small error. 3
Sray Lgh Conamnaon Sray lgh lens flare n opcal magng sysems 1. Scaerng on lens surfaces. Scaerng whn ransparen glass or plasc lens elemens 3. Undesred reflecon beween opcal elemen surfaces y Ideal pon source Obec plane Dgal camera opcal sysem θ Image plane x Pon spread funcon capured resored 4
Model Formulaon Model of capured mage y = 1 β G + βs x x s he underlyng mage o be resored y s he capured mage S s he convoluon marx of sray lgh G accouns for dffracon and aberraon β represens he wegh of sray lgh 5
Sray Lgh Pon Spread Funcon Sray Lgh Pon Spread Funcon Feaures of sray lgh PSF y g Space-varyng Large suppor Model for sray lgh PSF α = 1 1 ; p p q q z s + + + + + + + + + 1 1 p p p q q p p p p p q p q p p p b c a c z z s a normalzng consan p p pon source locaon q q response pon locaon parameers a b c α are esmaed along wh β Bls e. al. 007 p g β and We e. al. 008 6
Sray Lgh Reducon Sray lgh reducon algorhm Van Cer s mehod xˆ = 1 + β y βsy Compuaonal problem x = Sy Drecly compung Sy s que expensve. For example for a 6M pxel mage akes 3.6x10 13 36000G mulples. 7
Theory of Lossy Marx Source Codng Bg Dense x S y Compress S lossy marx source codng Sparse marx Save sorage and compuaon Queson: How s error due o marx compresson relaed o error n oupu δ ~ x? δs F Answer: when he emprcal auocorrelaon of y s ~ deny hen δx s equal o δs F. 8
Marx Source Codng Approach x = Sy = W WST Λ Λ 1 1 1/ 1/ Ty W and T are boh wavele ransforms decorrelaon Ty approxmaely decorrelaes y Λ 1/ s dagonal marx whch normalzes auocorrelaon of Ty Compress S ~ 1/ y = Λ Ty R y = [ ~ yy ~ ~ E ] = I Transform o a space where s sparse ~ S = WST Quanze o save sorage and compuaon: 1 Λ 1/ 1 [ WST ] 1/ [ S ~ ] = WST Λ 9
Fas Convoluon Usng Marx Source Codng Onlne compuaon: % = % % 1 x x W S y wavele ransforms on mage + sparse marx vecor mulply complexy OP where P s he number of pxels Offlne compuaon: [ S ~ ] = Λ 1 [ WST ] 1/ wavele ransforms along rows and columns of huge marx complexy OP need o be mproved o OP. 10
Reducon of Offlne Compuaon Problem: Offlne compuaon oo expensve -- order OP. Soluon: Two sage reducon o sparse marx 1 1/ S% W ST Λ wavele ransform W on sparse daa ST 1 Λ 1/ saves me ST 1 Λ 1/ can be drecly compued wh order OP resulng formula for convoluon x = W 1 1/ [ W [ ST Λ ] 1/ ] Λ Ty ~ 1 [] 11
Fas Compuaon of Sparse Haar Wavele Coeffcens Obecve: drecly compue Wha s ST 1 Λ 1/? ST Λ 1 1/ wavele ransform along he rows of S hen scale ST Our sraegy for each row of S TS Λ = Λ 1 1/ 1/ 1. Locae mporan wavele coeffcens 1/ Imporan : nonzero afer mulplyng wh Λ and quanzaon. Compue hese mporan wavele coeffcens a op-down approach based on recurson reduce he compuaon from OP o O1 1/ 3. Apply scalng facors Λ 1
Locae Imporan Wavele Coeffcens For mage sze 104x104 ake 49 PSFs due o 49 equally spaced pon sources. Compue Haar wavele ransform of hem. Apply scalng facors and quanze he resul so ha 1000 coeffcens survve. Fnd ou he relaonshp beween locaon of mporan wavele coeffcens and he locaon of pon source and level of ransform. 13
Locae Imporan Wavele Coeffcens connued logpsf locaon of mporan wavele coeffcens q q k= k=1 1 q 1 q r 1 Predcng locaon of mporan wavele coeffcens a level k locaon of red crcles are relaed o pon source locaon and level of ransform: k q = q / k k q = q / k radus r k s no relaed o q q. 14
Compuaon of Sparse Haar Wavele Coeffcens Use Haar wavele for smple and fas compuaon. Wavele deal coeffcens can be compued from approxmaon coeffcens. We only need o compue necessary approxmaon coeffcens snce many deal coeffcens are zero. Horzonal band Vercal band Dagonal band Approxmaon coeffcens 15
Compuaon of Sparse Haar Approxmaon Coeffcens Recursve approach for compung sparse approxmaon coeffcens a[k][m][n] 1 ak [ ][ m][ n] = ak [ 1][ m][ n] + ak [ 1][ m][n+ 1] + ak [ 1][m+ 1][ n] + ak [ 1][m+ 1][n+ 1] Top-down approach When all correspondng deals coeffcens are zeros means approxmaon s perfec. So sop and reurn value k k k ak [ ][ m][ n] = a[0][ m][ n] k+1 Approxmaon coeffcens a dfferen levels k k-1 16
Smulaon Expermens We use Cohen-Daubeches-Feauveau 5-3 wavele for W Haar wavele for T. Three ranng mages are used o oban gan facors Λ ake Haar wavele ransform of each ranng mage compue varance of wavele coeffcen n each band and average over all ranng mages Error merc: normalzed roo mean square error x x% NRMSE = x where x = Sy x = W 1 1/ [ W [ ST Λ ] 1/ ] Λ Ty ~ 1 a ranng mage he es mage 17
Dsoron-Rae Curves Comparson beween wh and whou ransform for mage sze 56x56 Comparson beween wo dfferen resoluons No ransform: ~ x = [ S]y Noe ha for mage sze 104x104 we reduce he mulples per oupu pon from 104 o 10 wh only % error. The sorage s 10M for 104x104 mage. 18
Capured Image 19
Resored Image 0
Concluson We characerzed sray lgh pon spread funcon and used a deconvoluon algorhm o resore mages. We appled marx source codng approach o space-varyng convoluon n he resoraon algorhm o acheve reducon n complexy. We developed recursve approach for compung sparse Haar wavele coeffcens o save offlne compuaon. Expermenal resuls showed ha our algorhm can reduce he compuaon by a sgnfcan amoun e.g. for 104x104 mage we can acheve a 10 5 :1 reducon n compuaon wh a small amoun of error. 1
Fas Marx Vecor Mulplcaon Usng Sparse Marx Transform
y = Ax x npu vecor Px1 y oupu vecor Px1 Compuaonal Problem A ransform marx PxP bg and dense! Wdely used: sray lgh reducon large scale elecromagnec negral equaon Problem: canno consruc or sore A when P s large complexy OP s oo hgh Obecve: compue Ax much faser wh small error 3
Background on SMT SMT of order K T = K k = 1 Each T k s a sparse marx T k = T1T LT K Each T k operaes on wo coordnaes SMT for compung ML esmae of covarance marx gven M M<P sample daa vecors [ T Λ ] = SMTCovEs{ a1 a K am {a m } are sample vecors T s a SMT marx Λ s dagonal TΛT s an esmae of E[ a ] a m m } 4
Our Approach Basc dea: use N N<<P ranng npu X O and oupu Y Z vecors o fnd approxmae SVD of A. Y = AX Z = A O X O: mpulses a random locaons Jonly ran rows and columns of A Mnmze cos c U V Σ = U Y ΣV X F where U and V are orhonormal SMT marces Σ s dagonal Approxmae SVD of A: U Σ V Approxmae Ax wh UΣV x + V Z ΣU much faser when order of SMT s low <<P furher sudy needed: wha knd of marx A makes a good approxmaon? 5 O F
Deals of Our Algorhm Deals of Our Algorhm Two sage approach: g pp 1. Pre-processng: perform SMT on npu and oupu make second sage opmzaon easer. Fnd and o mnmze cos 0 0 0 0 ~ ~ ~ ~ ~ ~ ~ ~ O U Z V X V U Y V U c Σ + Σ = Σ U ~ V ~ Σ 0 0 0 0 F F O U Z V X V U Y V U c Σ + Σ Σ where and are orhonormal SMT marces are processed npu and oupu from frs sage U ~ V ~ 0 0 0 0 ~ ~ ~ ~ Z O Y X processed npu and oupu from frs sage. 6
SMT Pre-processng Sample M rows of marx A o ge A r of sze MxP. ~ E 0 Λ ] = SMTCovEs A r ~ E approxmaely decorrelaes he columns of A. [ 1 0 Sample M columns of marx A o ge A c of sze PxM. ~ [ F 0 Λ ] = SMTCovEs Ac ~ F approxmaely decorrelaes he rows of 0 Process ranng daa ~ ~ ~ X 0 = E0 X Y0 = ~ ~ ~ O0 = F0 O Z0 = ~ F0 Y ~ E Z 0 A. ~ ~ E A F 0 A 0 7
Ierave Approach for Mnmzng Cos Idea: only operae on wo coordnaes n each eraon o reduce he cos Keep updang ranng npu and oupu by orhonormal sparse ransforms n order o dagonalze he marx relang hem 8
Opmzaon Algorhm Inalzaon: frs esmae of Σ For each eraon k fnd a par of coordnaes wh maxmum cos reducon compue SVD of correspondng opmal ransform marx T k updae ranng vecors and dagonal marx a coordnaes updae cos reducon for relaed coordnaes 9
Opmal Transform for Two Coordnaes Opmal ransform for coordnaes a eraon k: Tk arg max where = k 1 c c k Soluon remove and k sub/sup scrps: where vec means sackng columns o form a vecor means Kronecker produc. 30
Updae Tranng Vecors Updae Tranng Vecors For coordnaes wh maxmum cos reducon Perform SVD of ransform marx k k k k V D U T = Updae npu and oupu vecors 1 1 ~ ~ ~ ~ ~ ~ k k k k k k Y U Y X V X = = Updae esmae of 1 1 ~ ~ ~ ~ k k k k k k Z V Z O U O = = Σ Updae esmae of Σ 11 k k k k D D = Σ = Σ 31
SVD Usng SMT SVD Usng SMT Afer K eraons we have O F U U Z E V V X E V V Y F U U K K K K K K 0 1 0 1 0 1 0 1 L L L L Σ Σ Approxmae SVD of A Fas marx vecor mulplcaon X E V V U F U A K K K 0 1 1 0 L L Σ Fas marx vecor mulplcaon x E V V U F U y K K K 0 1 1 0 ˆ L L Σ = 3
Compuaonal Complexy Searchng for wh maxmum cos reducon n each eraon s logp Neghborhood based search Red-Black ree daa srucure SVD usng SMT coss OK+K 0 logp 0 Regular SVD coss OP 3 Fas marx vecor mulplcaon coss OK+K 0 +P Rgorous mplemenaon coss OP 33
A Smple Expermen Image sze 56x56 whch means P=65536 Marx A s a convoluon marx A = a x y ; x y where ax y ;x y s he PSF due o pon source x y H y y W x x a x y ; x y = HW We se M=56 N=56. Tesng on a naural mage Error s measured n normalzed roo mean square error 34
Dsoron vs. Number of Roaons K 0 =8 K 0 =1 y ŷy NRMSE = y 35
Concluson and Fuure Work Concluson: We developed an algorhm o perform approxmae SVD of a huge dense marx usng SMT. We reduced he compuaon of marx vecor mulplcaon from OP o OK+K 0 +P wh a small amoun of error. Fuure work: Furher speed up by quanzng esmaed sngular values Expermen wh more marces PSFs Work wh larger mages 36
FMRI Acvaon Deecon
Problem Descrpon FMRI daa s 3D/4D daa whch conans a emporal dmenson and wo or hree spaal dmensons. Daa colleced whle he subec s presened wh a smulus n our case a vsual smulus. Obecve: deec acvaed regons n he bran from a sequence of fmri mages. =1 =5 =10 =15 38
Conrbuons We propose p a forward model whch smulaneously capures spaal and emporal dependences of he daa. We develop an effecve algorhm for esmang model parameers. We nroduce oal-varaon based resoraon as a very effecve pre-processng processng ool for fmri daa. We adop Markov random feld MRF model n fmri analyss whch resuls n spaally regularzed and robus esmae of he parameer map. 39
Model Formulaon y = g h + w + η k l k l k l k l y observed daa a locaon me. h hemodynamc response funcon HRF a locaon me. w physologcal nose a locaon me. Modeled by AR1 process w = ρ w + ε where ε s whe 1 nose wh mean zero and sandard devaon σ ε η scanner nose a locaon me modeled as whe nose wh mean zero and sandard devaon σ η. g kl Gaussan blurrng kernel characerzed by wdh σ. 40
HRF Models Gamma varae model Dale and Buckner 1997 h δ δ τ x e δ τ = 0 < δ parameer se θ = x δ τ x amplude parameer τ dsperson parameer { } Model wh undershoo Frson e al. 1998 a h = x 1 1 e c e a b a b parameer se 1 1 1 a b a a b 1 b b θ x amplude parameer 1 dsperson parameer b = { x a 1 b 1 a b c } 41
Overall Acvaon Deecon Sraegy Resore each frame usng consraned oal varaon mnmzaon Goldfarb and Yn 005 Esmae he HRF parameers usng spaal regularzaon assume he parameers follow generalzed Gaussan Markov Random Feld GGMRF dsrbuon perform maxmum a poseror esmaon Threshold he amplude parameers o oban he acvaon map 4
Consraned Toal Varaon Mnmzaon The opmzaon problem s formulaed as: mn h h + h h { h } + 1 + 1 subec o y = g h v and 1 N + k l k l k l v σ v Noaon: {h } mage o be resored {y } he observed mage {g kl } blurrng kernel {v } auxlary varables N number of pxels σ ε where σv = gk l + ση. kl 1 ρ σ ρ σ ε σ η are esmaed based on spaal and emporal correlaons of he daa pror o resoraon We Talavage and Pollak 007 Solved wh an neror pon mehod 43
Models for Parameers and Resored Daa We model he daa afer resoraon as θ h = h + e ˆ e --..d. Gaussan random varables Condonal dsrbuon: ˆ 1 p h θ = exp hˆ h θ T/ T π σe σe Pror dsrbuon of parameers GGMRF 1 1 p θ = exp K θ θ r r r p k l k l zr pσσ r and k l neghbors p r=1 R. E.g. for he Gamma varae HRF 1 3 θ = x θ = δ θ = τ assume he R parameers are muually ndependen 44
Regularzed HRF Parameer Esmaon Jon poseror dsrbuon p 1 R ˆ 1 R 1 R θ θ K θ h p h θ θ K θ p θ L p θ = Maxmum a poseror esmae R { K } 1 R θ θ θ argmax θ θ θ hˆ 1 1 R = p K θ θ K θ / pˆ h Cos funcon o mnmze c 1 1 θ θ θ h h θ K R 1 R ˆ r r = + K k l θ θk l p σe and k l neghbors r= 1 p σr erave coordnae descend ICD algorhm s used p 45
GLM Framework General lnear model GLM hˆ = Gx + e ˆ G s he regressor wh hree columns HRF fed by he averaged emporal daa HRF emporal and dsperson dervaves o accoun for emporal shf and change n shape of response orhonormalze hese hree vecors o buld G leas squares esmae of regresson coeffcens 1 ˆ = G' G G' hˆ x oban acvaon map by hresholdng he -sasc c xˆ = ˆ ˆ 1 h Gxˆ h Gx ˆ c G G c / T 3 c s a conras vecor se o 1 0 0 n our expermen 46
GLM Implemenaon Pre-processng: Gaussan smoohng Advanage: fas Problem: performance s very sgnfcanly affeced by changng he wdh of Gaussan kernel Summary: spaal regularzaon + lnear emporal analyss 47
Smulaed Expermen Acve regon Image a =5 peak me Resored mage Regularzed amplude esmae Deecon resul 48
Smulaon Resuls ROC curve for Gamma Varae model ROC curve for HRF model wh undershoo Noe ha n hese expermens: CNR=0.5 Daa sze 64x64x135 represenng mage sze 64x64 and 9 rals wh 15 me pons n each ral. 49
Robusness of Our Algorhm Cox waveform s used o generae he daa whereas he Gamma Varae model s used for deecon. If we change regularzaon parameers σ e σ 1 σ R separaely by 50% he maxmum change n correc deecon rae under he same false alarm rae s 1%. 50
Dfferen Acvaon Waveform a Dfferen Locaon Two acvaon regons Each wh dfferen HRF 51
Real Daa Expermen Two ypes of vsual checkerboard smul: lef hemfeld smulus and rgh hemfeld smulus. Deermne regularzaon parameers from lef hemfeld smulus daa ranng and use on rgh hemfeld smulus daa. Threshold he acvaon map such ha 16 pxels are declared acve for all mehods. 5
Real Daa Resuls proposed mehod Smulus 1 Smulus 53
Real Daa Resuls GLM mehod Smulus 1 Smulus 54
Benchmark Resuls Smulus 1 Smulus Benchmark resuls are obaned from block paradgm expermens whch are supposed o have hgher deecon power. 55
Thank you! 56
Backup Sldes 57
Appendx: Ideal Image Consrucon Capured mage Horzonal cross-secon of mage Ideal mage 58
Model Parameer Esmaon Expermen Take pcures of a lgh box a dfferen posons n he feld of vew Canon EOS 350D: focal lengh 55mm ISO100 f8.0 Olympus SP-510UZ: ISO 50 f8.0 59
Model parameer esmaon Model parameer esmaon Consruc deal mages from hese pcures Consruc deal mages from hese pcures Esmae parameers by mnmzng he cos funcon 9 9 9 1 ; ˆ ˆ = = = n I n n n y s x y y ξ where I n s he se of pxels where we compue he error 1 ; = = n I n p p p p n n p p y s x a poson n. 60
Wha f we furher ncrease K 0? 61