Correction and rectification of light field

Transcription

1 Correcton nd rectfcton of lgt feld Ke Deng, Lfeng Wng b, Zoucen Ln b, To Feng b, Zdong Deng, Bnng Guo b* Tsngu Unverst, b Mcrosoft Reserc As ABSTRACT Te lgt feld s well known mge-bsed renderng tecnolog. Te trdtonl w to cpture lgt feld s to move cmer on plne nd tke mges t ever grd ponts. However, due to devce defects t s rd to ensure tt te cptured lgt feld s del. For emple, te cmer s mgng plne s not on te mge plne of te lgt feld nd te cmer s not n te rgt poston. To solve ts problem we propose correcton nd rectfcton frmework tt uses onl four mges. Ts frmework nvolves un-dstorton, feture pont detecton, omogrp computton to correct te orentton of te cmer, nd postonl error correcton. It s te frst to tke postonl error nto consderton. Our eperments sow tt our metod s effectve. Kewords: Clbrton, Lgt feld. INTRODUCTION Te lgt feld [] s well known mge-bsed renderng tecnque. It uses two-plne prmeterzton to nde te lgt rs n regons of spce free of occluders. Ever lgt r cn be defned b connectng pont on te cmer plne to noter pont on te mge plne Fgure. Wt lgt feld, new vews cn be esl snteszed b nterpoltng mong pproprte rs. A lgt feld s usull cptured b movng cmer on cmer plne nd tkng pctures t ever grd ponts of te cmer plne Fgure b. However, n rel cses due to devce defects te cptured lgt feld usull not del. For emple, te cmer s not movng on plne, or te cmer s optcl s s not perpendculr to te cmer plne, or te cmer s not n te rgt poston precsel Fgure c. Moreover, lens dstorton cn lso mkes te r quer ncorrect. On te oter nd, correctng te lgt feld nd rectfng t,.e., mkng te eppolr lnes of te mges prllel to te coordnte es n te mge spce, fcltte furter tretment on te lgt feld, suc s renderng nd stereo mtcng mong te mges. Terefore, correcton nd rectfcton s necessr so tt te lgt feld s close to del... Prevous Work Tere s been some work on te rectfcton between stereo mges [][][], were te number of mges s usull two or te mges re on lne. In [], te eppolr lnes re lgned b estmtng two projectve prmeters. Te results of ts lgortm ve muc serng dstorton. Smlr dstorton ests n [] becuse t onl lgns te eppolr lnes nd does not consder te vsul effect of overll mge. In [], te estmton of ser trnsform s proposed nd tus te results re muc more stsfctor. It recovers te fundmentl mtr frst nd ten estmtes te rectfng omogrp see Secton.., wc reltes te coordntes of plnr object n two vews. But clcultng te fundmentl mtr s senstve to correspondence ccurc, so ts two-step lgortm cuses error to ccumulte. Toug tere re commons between te rectfcton of lgt felds nd tt of stereo mges, lgt feld rectfcton needs more constrnts nd conventonl metods used n stereo mge rectfcton cnnot be ppled drectl, becuse te lgt felds re tken b cmer movng n plne. Te requrement tt te eppolr lnes re lgned s not enoug []. Terefore, more constrnts must be mposed. Tere s lttle prevous work on te problem of rectfng lgt feld. In [], wrpng to common plne s mentoned s cmer pns nd tlts, but tere re few words bout ow to wrp nd te precson of te wrpng, wc re crtcl for te pplcton of lgt felds. Te wrpng s rectfng process. We fnd tt te pose of te cmer s usull fed wen movng, wc mens recoverng sngle rectfng omogrp s enoug for wole lgt feld. In our lgortm, we dopt novel metod b decomposton nd terton tt nsures te globll optml soluton nd lso vods error ccumulton. It s lso notewort tt our frmework s te frst to tke postonl error nto consderton. * Furter utor nformton: Ke Deng: E-ml: dengke@msrcn.reserc.mcrosoft.com

2 For te computton of omogrpes nd postonl error correcton, feture ponts re ndspensble. Terefore, t s mportnt to desgn mrkers to fcltte te detecton of feture ponts. In [], mrkers consstng of concentrc rngs were used for trckng from nd-eld cmer. Youngkwn Co proposed mult-rng color fducl sstem for ugmented relt n []. Concentrc rngs were lso used n [9]. We lso coose to use suc mrkers for te detecton of ndependent feture ponts. Wt s dfferent s tt we lso use dots between te concentrc-rngs s dependent feture ponts. Te mn dvntge of usng dependent fetures s tt te pttern of ll mrkers s smller so tt t s vsble n more vews smultneousl... Overvew of Our Metod Our frmework conssts of postonl error correcton nd rectfcton of te lgt feld, were te correcton lso depends on te omogrp computed n te rectfcton process. It nvolves te followng steps:. Un-dstortng ever mge n te lgt feld,. Detectng te ndependent nd dependent feture ponts of ec mrker,. Clcultng te ntermedte omogrp tt trnsforms te qudrngle of feture ponts nto rectngle,. Computng te ffne trnsform tt scles te mge trnsformed b te ntermedte omogrp pproprtel,. Itertng te steps nd to refne te omogrp,. Combnng te tertve results to compute te fnl omogrp nd rectfng te wole lgt feld, nd. Correctng te postonl error fter estmtng te error. Te rest of ts pper s orgnzed s follows. In Secton we present te detls of our frmework, ncludng feture desgn nd detecton, te computton of te omogrp, nd te correcton of postonl error. Fnll we sow our eperments n Secton nd conclude n Secton. b c Fgure. Te concept of lgt feld nd te correcton nd rectfcton problem. Te two-plne prmeterzton. b In te del lgt feld, te mgng plne of te cmer re n te rgt poston. c In prctce, t s rd to mke te mgng plne of te cmer on fed plne nd te cmer m not be on te cmer plne or t te grds precsel. Fgure. Te detecton of ndependent feture ponts. Te orgnl mult-rng mrker. b Te mrker fter projecton c Te contours re detected nd be ft b ellpses. d Te reltve rd r R.. FRAMEWORK.. Feture Desgn nd Detecton Our frmework s bsed on postons of severl feture ponts n dfferent vews. To smplf te process of feture detecton nd mtcng, fducl sstem s used. Te fducl sstem conssts of detectng two knds of fetures: ndependent ones nd dependent ones. Te postons of bot knds of fetures cn be detected ndependentl nd te fetures re prnted on plnr bord. However, te dentt of n ndependent feture cn lso be detected ndependentl, wle tt of dependent feture cn not untl ts negbors re recognzed.... Independent fetures

3 We coose concentrc rngs s mrkers snce te cn be detected ccurtel nd stbl. A tpcl mrker nd ts mge under projectve projecton re sown n Fgures nd b. Te centers of te mrkers re te ndependent fetures. Te detecton for suc feture ponts conssts of te followng steps:. All ellptc contours re recognzed nd ft b ellpses Fgure c usng te Open Source Computer Vson Lbrr b Intel Corporton [].. Snce te centers of ellpses nd te projected center of concentrc crcles re collner [], b fttng ts lne, te reltve rd of te crcles tt correspond to te smller ellpses cn be computed from te preservton of cross rto Fgure d. Tese reltve rd re used s fetures to dentf wc mrker t s.... Dependent fetures As sown n te bove fgures, n ndependent feture sould be reltvel lrge n sngle vew. Oterwse ts detls wll merge togeter. Moreover t needs to be crefull desgned. Oterwse tere m be smlr mrkers tt cn cuse trouble n recognton. Terefore, we cnnot use too mn ndependent fetures. In most cses, te number of ndependent fetures n sngle vew s less tn 9. So we mke use of dependent fetures tt eplot te relton between te fetures. Te poston of ec feture n te orgnl pttern s known. After projectve trnsform, f tree collner fetures re vsble n te mge, te postons of oter fetures on te sme lne cn be computed usng te cross rto. If four vsble fetures form qudrngle, te postons of ll fetures on te plne cn be estmted from te omogrp. Ten te postons of prevousl found mrkers re compred wt tese estmted postons. If te dstnce between te two s wtn tresold tpcll ~ pels, tese mrkers re dentfed. In our sstem, we use te pttern n Fgure, wc s 9 ndependent fetures concentrc-rngs nd dependent fetures dots.... Error nlss Our feture detecton lgortm s tested wt dfferent poses of te cmer. To obtn correct feture postons s te ground trut for error nlss, we use vrtul cmer so tt bot ntrnsc nd etrnsc prmeters re known. Te cmer s locted on emspere, wt te optcl s pontng to te center of te emspere Fgure. Te pttern s plced on te center of te bse of te emspere nd mges re cptured s cmer moves on te emspere Fgures bc. Fgure sows te relton of te errors n pel wt te poston of te cmer. We cn see tt our feture detect lgortm s robust nd ccurte wt te mml error less tn pel nd te verge error less tn. pels. Fgure. Te pttern wt ndependent fetures nd dependent fetures. Te left s te orgnl pttern. All fetures n ts pttern re numbered to, from left to rgt, top to bottom. Te center s te pttern fter perspectve trnsform. Te rgt s te pttern fter noter trnsform. Te fetures, nd re vsble ndependent fetures. Te fetures nd cn be found usng te cross rto. Fgure. Te sntetc eperment to test our feture detecton lgortm. Te vrtul cmer trnsltes on te emspere wt ts optcl s pontng to te center of te emspere. b c Ptterns tken t dfferent postons of te cmer. Te defntons of α nd β cn be found n... Homogrp Computton... Te omogrp A plnr object n two vews s relted b perspectve trnsform. Te omogrp determnes te relton between te coordntes of te plnr object n two vews b mtr:

4 H, 9 were omogeneous coordntes re used for D ponts n te mge spce: X w T. For λ, w T nd λ λ λw T represent te sme pont n te projectve spce In te sequel, λ stnds for n pproprte sclng prmeter.. Terefore, for w, w T s te ordnr pont w w T. As result, onl prmeters re ndependent n nd we m set 9. Ten te trnsform of X under te omogrp H cn be wrtten s: X ' HX. To determne te omogrp, onl four prs of correspondng ponts wt non-degenerte confgurton.e., te form conve qudrngle re enoug. Due to te fct tt te cmer s pose s usull fed wen movng, sngle omogrp s enoug to rectf ll mges. So we cpture four mges of te mrkers usng our lgt feld cpturng devce Fgure to estmte ts omogrp. All mges re undstorted usng te lgortm n [] before processng b tkng mges of te un-dstorton pttern sown n Fgure b. α b α 9 Fgure. Te relton between te errors n pels nd te poston of te cmer. Curves wt crcles denote te verge errors of ll detected feture ponts n. Tose wt crosses re te verge errors n. Dsed ones wt strs re te mml errors of ll detected fetures n, nd wt dots re te mml error n. All feture ponts re detected n ever mge tested ere. Fgure. Te lgt feld cpturng devce. It s vertcl XY-tble. Te object nsde te wte bo s CCD cmer. b Te pttern used for un-dstorton. Te mges re undstorted usng ts cessbord mge before rectfcton. Let us nvestgte one feture pont cptured t dfferent cmer postons. Te feture pont ppers n te upperleft, upper-rgt, bottom-left nd bottom-rgt mges s pels P, P, P, P, respectvel. And te coordntes of P s T, L,, respectvel. Ten te omogrp H must stsf: ', L,, H λ ' were T ' ' re te rectfed coordntes, nd ' ', ' ', ' ', ' '. A drect computton of H wll led to sstem of non-lner equtons. 9 9 B elmntng s nd s n usng relton. Suc metod s computtonll ntensve nd nccurte. ' ' So we coose n ndrect w s n []. On te oter nd, te constrnts n re too wek to compute ll te entres n H becuse te onl requre te rectfed qudrngle s rectngle. Te sze of te rectngle s not specfed. So we

5 dd two constrnts on H so tt t keeps te mge center nvrnt nd vods clppng. Consequentl, te computton of rectfng omogrp conssts of two prts:. Computton of te ntermedte omogrp H tt mps te qudrngle P P P P nto rectngle.. Computton of te ffne trnsform A tt keeps te mge center nvrnt nd vods clppng. For te computton of H, we decompose t nto n ffne trnsform nd perspectve trnsform s n []. Ten te estmted omogrp s H A H. As estmtng H onl once m not be ccurte enoug, we m terte te bove steps to mke t more ccurte.... Estmton of H From []: AP H ~ ~ 9 were ~ A ~ P, We cn estmte te ffne trnsform A ~ nd te perspectve trnsform P ~ seprtel. Let A ~ stsf nd, we get: Hence, we cn solve nd b te lest-squre metod. As we cn see lter, onl te rtos re necessr. Te two prmeters nd, wc re relted to te trnslton of te mge, re stll not solved et. However, te re not crtcl becuse te trnslton wll be solved n noter ffne trnsform A. On te oter nd, bndng, te two prmeters nd cn be clculted from te followng lner sstem:... Estmton of A Tll now, four constrnts on te omogrp re obtned, nmel,,, nd. Addtonl four constrnts must be ppled to solve ll entres of H. Tere re mn ws to mpose constrnts, s mentoned n []. Tose we coose re to keep bot te mge center nd te sze of mge nvrnt. Let: d s d s A, were s nd s ccount for te orzontl nd vertcl sclng, respectvel, d nd d ccount for te trnslton.

6 Suppose te mge s of sze w, nd on te mge plne, te -coordnte rnges from to w, nd te - coordnte rnges from to, ten te mge center s w,. Let Q be te qudrngle of four corners of te orgnl mge, nd Q ' HQ. Te constrnt of keepng te mge center nvrnt gves: w / w / AH / λ / T To vod clppng of te rectfed mge, let,, be te center of te mge fter pplng omogrp H : w / λ H /. As te mge center s nvrnt, s nd s sould be cosen s: m left, rgt s, m top, bottom s, w so tt te scled mge does not beond te re [ left, rgt] [ bottom, top] on te rectfed mge, were left s te mnmum of te -coordntes of Q ', rgt, top, nd bottom re defned lke. Combnng nd, we get d nd d from: s d / z w / s d / z /. ~ ~ Fnll, H AAP s computed.... Homogrp estmton b terton A sngle lner ppromton of non-lner problem s not ccurte enoug, but ec estmted H cn be used to trnsform ll coordntes of feture ponts. Ten we m estmte H gn b te trnsformed coordntes. Wen te errors re below preset tresold, te terton stops nd ll omogrpes re conctented togeter to get te fnl omogrp,.e., H H n LH H H, were H s te omogrp computed n t terton. Fgure sows te reducton of errors wt te terton, were te errors re defned s follows: N e ' ' ' ', N N e ' ' ' ', N N s te number of mges n te lgt feld nd ' ' T, L, re te trnsformed feture ponts fter pplng H. j.. Rectfcton Postonl Error Te mrkers sown n Fgure cn be put n te scene f te user does not cre te presence of te mrkers or te object of nterest s reltvel smll suc tt t does not occlude te mrkers lot. Oterwse we m cpture te lgt feld of te mrkers nd compute te omogrp. Ts omogrp re lso used to rectf te lgt feld of nterested object. Tll now, we onl nvestgte te problem wen te optcl s of te cmer s not perpendculr to te cmer plne, tere s possblt tt te cmer s not t te desred poston. Te use of mrkers cn lso solve ts problem. Te postonl error cn be computed becuse te postons of te feture ponts re computble. Ten we gn use te postonl error mesured from te lgt feld of mrkers to correct te nterested lgt feld. Te error tt te cmer s not on te cmer plne s neglgble. j

7 To demonstrte, our cpturng devce cptures two lgt felds wt te sme confgurton. We sow te verge error of te two lgt felds n Fgure 9. For one mge n te lgt feld, te error n drecton s defned s te verge devton n of ec feture pont from te topmost mge n te column; te error n drecton s defned s te verge devton n of ec feture pont from te leftmost mge n te row. Te verge errors n nd re te men error of ec column nd row, respectvel. From ts fgure, we cn see tt te errors re unbsed. Te errors of one lgt feld re used s te bencmrk. After subtrcton, te errors reduce n bot nd drectons, especll te mml errors Fgure. Error long -s b Error long -s Fgure. Errors over terton. Te vertcl es re te error n pel, nd orzontl ones re te tmes of terton. Curves wt crcles denote te errors n -coordnte, wle tose wt crosses denote te errors n -coordnte. Te left fgure s from sntetc dt. Te rgt fgure s from rel mges fter feture detecton. Due to te error of feture detecton, te errors do not converge to zero. Fgure. Postonl error correcton. Te vertcl es re te error n pel, nd te orzontl ones re te columns or rows of te mge. Curves wt crcles denote te errors n te frst lgt feld. Tose wt crosses denote te errors n te second lgt feld. Tose wt strs denote te errors fter subtrcton.. EXPERIMENTAL RESULT Prts of some lgt felds before nd fter rectfcton re sown n Fgure 9. Before rectfcton, postons of correspondences n lgt feld re not lgned vertcll nd orzontll. Tke Fgure 9 for emple, feture pont n te sttus on te top-left mge s pels lower tn ts correspondence on te top-rgt one, nd moves pels rgt compred to ts correspondence on te bottom-left one Fgure 9c. After rectfcton, te devton s less tn pel due to te error of mnull selectng feture ponts wtout postonl error correcton Fgures 9b nd d. After correctng te postonl error, te verge error over te entre lgt feld drops to less tn. pels. Te results fter postonl error correcton re not sown ere becuse te furter mprovement s too smll to be vsull detected. Tese emples sow tt our lgortm s rter effectve. Te run tme for detectng mrkers n four mges nd clcultng te omogrp s ~ ms on Pentum III wt M RAM.. CONCLUSION We ve proposed n effectve frmework tt bot corrects te postonl error of te lgt feld nd rectfes te lgt feld, n wc lgt feld of pttern s tken to compute te rectfcton omogrp nd postonl error. Oter lgt felds tken t te sme postons of te cmer use te sme omogrp nd te postonl error. As te core of our frmework, te rectfng omogrp s computed n dvde-nd-conquer mnner, nd te terton refned te omogrp. Our eperments sow tt our feture detecton, postonl error correcton nd rectfcton re ll robust nd ccurte. REFERENCES. M. Levo nd P. Hnrn, Lgt Feld Renderng, SIGGRAPH 9.. C. Loop nd Z. Zng, Computng Rectfng Homogrpes for Stereo Vson, CVPR F. Isgr nd E. Trucco, Projectve Rectfcton wtout Eppolr Geometr, CVPR 999.

8 . A., Fusello, E. Trucco nd A. Verr, Rectfcton wt unconstrned stereo geometr, Proceedngs of Brts Mcne Vson Conference, pges -9, 99.. S. Gortler, R. Grzeszczuk, R. Szelsk nd M. Coen, Te Lumgrp, SIGGRAPH 9.. Y. Co, J. Lee, nd U. Neumnn, A Mult-rng Color Fducl Sstem nd A Rule-Bsed Detecton Metod for Sclble Fducl-trckng Augmented Relt, IWAR 9.. O. Fugers, Tree-Dmensonl Computer Vson: A Geometrc Vewpont, MIT Press, 99.. Intel Corporton, Open Source Computer Vson Lbrr Reference Mnul, W.A. Hoff nd K. Nguen, Computer vson-bsed regstrton tecnques for ugmented relt, Proceedngs of Intellgent Robots nd Computer Vson XV, SPIE Vol. 9, Nov -, 99, Boston, MA, pp. -.. G. Xu nd Z. Zng, Eppolr Geometr n Stereo, Moton nd Object Recognton: A unfed pproc, Kluwer Acdemc Publsers, 99.. J.S. Km nd I.S. Kweon, Cmer clbrton usng projectve nvrnce of concentrc crcles, Proceedngs of Worksop on Imge Processng nd Imge Understndng IPIU, Jnur,