FAST EFFICIENT ALGORITHM FOR ENHANCEMENT OF LOW LIGHTING VIDEO. Xuan Dong, Guan Wang, Yi (Amy) Pang, Weixin Li, Jiangtao (Gene) Wen

Transcription

1 FAST EFFICIENT ALGORITHM FOR ENHANCEMENT OF LOW LIGHTING VIDEO Xuan Dong, Guan Wang, Y (Amy) Pang, Wexn L, Jangao (Gene) Wen ABSTRACT We desrbe a novel and effeve vdeo enhanemen algorhm for low lghng vdeo. The algorhm works by frs nverng an npu low-lghng vdeo and hen applyng an opmzed mage de-haze algorhm on he nvered vdeo. To falae faser ompuaon, emporal orrelaons beween subsequen frames are ulzed o expede he alulaon of key algorhm parameers. Smulaon resuls show exellen enhanemen resuls and 4x speed up as ompared wh he frame-wse enhanemen algorhms. Index Terms low lghng vdeo enhanemen, dehazng 1. INTRODUCTION As vdeo ameras beome nreasngly wdely deployed, he problem of vdeo enhanemen for low lghng vdeo has also beome nreasngly aue. Ths s beause alhough amera and vdeo survellane sysems are expeed o work n all lghng and weaher ondons, he majory of hese ameras are no desgned for low-lghng, and herefore he resuled poor apure qualy ofen renders he vdeo unusable for ral applaons. Alhough nfrared ameras are apable of enhanng low-lgh vsbly n [1] and [2], hey suffer from he ommon dsadvanage ha vsble objes mus have a emperaure hgher han s surroundngs. In many ases where he ral obje has a emperaure smlar o s surroundngs, e.g. a bg hole n he road, nfrared ameras are no very useful. On he oher hand, onvenonal mage proessng and rado proessng ehnques suh as hsogram equalzaon may no work well eher n many ases, espeally for real me proessng of vdeo sequenes. In hs paper, we propose a novel, smple and effeve enhanemen algorhm for low lghng vdeo. We show ha afer nverng he low-lghng vdeo, he pxels n he sky and dsan bakground regons of he nvered vdeo always have hgh nenses n all olor (RGB) hannels bu hose of non-sky regons have low nenses n a leas one olor hannel, smlar o he ase of vdeo apured n hazy weaher ondons. Therefore, we apply sae-of-he-ar mage de-hazng algorhms o he nvered vdeo frames and show ha he ombnaon of nverng low-lghng vdeos and applyng de-hazng on hem s an effeve algorhm for enhanng low-lghng vdeos. To falae faser mplemenaon, we furher ulze emporal orrelaons beween subsequen vdeo frames for he esmaon of key algorhm parameers, resulng n 4x speed-ups as ompared wh he onvenonal frame-byframe approah. 2. ENHANCEMENT OF LOW LIGHTING VISION VIDEO We noe ha low lghng vdeo enhanemen has muh n ommon wh vdeo haze removal, beause afer nverng he npu low-lghng vdeos, he resuled vdeos look very muh lke vdeos aqured n hazy lghng ondons. Fgures 1 and 2 provded some examples of low lghng vdeos, nvered low lghng vdeos and vdeos aqured n hazy lghng ondons. As menoned n [3], n hazy ondons, he nenses of bakground pxels are always hgh n all olor hannels, bu he nenses of he man objes, e.g. houses, vehles and people are usually low n a leas one hannel beause of he olor, shadow and e.. To es wheher he nvered low lghng vdeo has he same propery, we ompue he mnmum nensy of all olor (RGB) hannels for eah pxel of he nvered vdeos/mages for 30 randomly seleed (by Google) low lghng ones, some of whh are shown n he op row of Fgure 2, as well as 30 random haze vdeos/mages as shown n he boom row of Fgure 2. Fgure 3 shows he hsogram of he mnmum nenses of all olor hannels of all pxels for he 30 nvered low lghng vdeos/mages and Fgure 3 s he same hsogram of he 30 haze vdeos/mages. Comparng Fgures 3 and 3, we fnd ha he wo hsograms exhb grea smlares, and ha more han 80% of pxels n boh he nvered and he haze ases have hgh nenses n all olor hannels (ermed hgh nensy pxels ), nludng almos all of he pxels n sky regons. As for he pxels of non-sky regons, he nenses are always lower n a leas one of he olor hannels (ermed normal nensy pxels ). In Fgure 1(), we mark he normal nensy pxels n green. A pxel s desgnaed as a normal nensy pxel f one of s hree olor hannels nenses s lower han 180. Clearly, mos of he marked pxels are n a non-sky regon or obje, e.g. vehles, sdewalks, lghs and e.

2 () (d) (e) Fg. 1. low lghng vdeo npu I. nvered resul R from npu I. () marked mage: pxels wh low nensy n a leas one olor (RGB) hannel are n green. (d) de-haze oupu J usng equaon (7). (e) fnal oupu E afer nverng mage J. Fg. 2. Top: low lghng examples. Mddle: nvered oupus of he Top ones. Boom: haze examples. Fg. 3. hsogram of all pxels olor hannels mnmum nenses of he 30 nvered low lghng vdeos/mages and he 30 haze vdeos/mages. The above sass srongly suppor our nuon ha he nvered low lghng vdeos are smlar o haze vdeos. Based on he above observaon, we propose a novel low lghng vdeo enhanemen algorhm by applyng he nver operaon on low lghng vdeo frames, and hen performng haze removal on he nvered vdeo frames, before performng he nver operaon agan o oban he oupu vdeo frames. Thus, for he npu low lghng vdeo frame I, we nver usng: R ( = 255 I ( x ), (1) where s he olor hannel (RGB). I ( s he nensy of a olor hannel of pxel x of he low lghng vdeo npu I. R ( s he same nensy of nvered mage R. The haze removal algorhm we use s based on [3], n whh a hazy mage s modeled as shown n (2): R( = J ( + A(1 ), (2) where A s he global amospher lgh. R( s he nensy of pxel x he amera ahes. J( s he nensy of he orgnal objes or sene. desrbes how muh peren of he lgh emed from he objes or sene reahes he amera. The model s also menoned n [4-7]. The ral par of all haze removal algorhms s o esmae A and from he reoded mage nensy I( so ha hey an reover he J( from I(. When he amosphere s homogenous, he an be expressed as: βd ( = e, (3) where β s he saerng oeffen of he amosphere, and d( s pxel x s sene deph. In he same mage, where he β s onsan, s deermned by d(, he dsane beween he obje and he amera. In [3] s esmaed usng: R ( y) = 1 ω mn ( mn ( )), (4) { r, g, b} y Ω ( A whereω s 0.8 n hs paper. Ω( s a loal blok enered a x and he blok sze s 9 n hs paper. We ulze hs algorhm o esmae n hs paper. To esmae he global amosphere lgh A, we frs sele 100 pxels whose mnmum nenses n all olor (RGB) hannels are he hghes n he mage, and hen among he pxels, we hoose he sngle pxel whose sum of RGB values s he hghes. The RGB values of hs pxel RGB values are used for A. Thus, aordng o he equaon (2), we an of ourse reover he J( as follows: R( A J ( = + A, (5) however, we fnd ha dre applaon of equaon (5) mgh lead o under-enhanemen for low-lghng areas. To furher opmze he alulaon of, we fous on enhanng he Regon of Ineress suh as houses, vehles and e. whle avod proessng bakground e.g. sky regons n low lghng vdeos. To adjus adapvely whle mananng s spaal onnuy, so ha he resuled vdeo

3 beomes smooher vsually, we nrodue a mulpler P( no (5), and hrough exensve expermens, we fnd ha P( an be se as: 2, 0 < < 0.5 P ( =, (6) 1, 0.5 < < 1 hen he reovery equaon beomes: R( A J ( = + A, (7) P( he dea behnd equaon (7) s he followng. When s smaller han 0.5, whh means ha he orrespondng pxel needs boosng, we assgn P( a small value o make P( even smaller so as o nrease he RGB nenses of hs pxel. On he oher hand, when s greaer han 0.5, we refran from overly boosng he orrespondng pxel nensy. For low-lghng vdeo, one J( s obaned, he nver operaon of (1) s performed agan o produe he enhaned mage E of he orgnal npu low lgh vdeo frame. Ths proess s onepually shown n Fgure ACCELERATION OF PROPOSED VIDEO ENHANCEMENT PROCESSING ALGORITHM 3.1. Aeleraon Usng Moon Esmaon (ME) In hs seon we propose an approah o aelerae he proessng of he algorhm n he prevous seon. In expermen, we fnd alulang oupes nearly 70% of he ompuaon, whh means he fous of aeleraon ehnque beomes aelerang he alulaon of. Our ehnque s based on he orrelaons beween emporally suessve frames. Sne s deded by he nenses of pxels of eah frame, he values of of o-loaed pxels should also by emporally orrelaed n emporally neghborng frames. In parular, f he values of are sored from he prevous frame, and f he pxels n he urren frame are deermned o be suffenly smlar o he o-loaed pxels n he prevous frame, he sored values ould be used as he for he orrespondng pxels n he urren frame, hereby by-passng he alulaon of a sgnfan poron of he values. To deermne f a pxel s suffenly smlar o s oloaed ouner par n he prevous frame, we use he same ehnque n he moon searh proess of vdeo odng. We frs dvde he npu frames no Groups of Pures (GOPs), wh eah GOP sarng usng an Inra oded frame (I frame), n whh all values are alulaed. In he remanng frames,.e. P frames, ME s performed. In deal, eah frame s dvded no non-overlappng 16x16 maro bloks (MBs), for whh a moon searh usng he dsoron reron Sum of Absolue Dfferenes (SAD) are ondued. If he SAD s below he predefned hreshold, he MB wll be enoded n ner-mode and he alulaon of for he enre MB s skpped by assgnng he bes mah MB s drely. Oherwse, he MB wll be enoded n nra-mode and sll needs o be alulaed. In boh ases, he values for he urren frame are sored for possble use n he nex frame. We ake wo ME algorhms n hs paper. Frs, we propose a smple ME mehod named Co ME whh only alulaes he dsoron value of he (0, 0) moon veor for he urren MB. If he SAD value s larger han a predefned hreshold T, he of he urren MB s alulaed. Oherwse, he alulaon s by-passed. The Co Me algorhm s easy o mplemen and os lle sorage spae. The oher ME mehod s EPZS ME algorhm proposed n [12], whh s more dfful o mplemen and oupes more sorage. However, performs beer han Co ME n oal speedup beause an provde more aurae ME nformaon o mah he referene frames hus o redue more alulaon of. In EPZS ME algorhm, he frssage SAD hreshold T s fxed and predefned. The followng hreshold hanges as follow: T k = ak mn( MSAD1, MSAD2,, MSADn ) + bk, (8) where a k and b k are fxed value. Based on he above algorhm, we defne he speed up rao as: O oher + g, (9) Speedup = = g 1 A ( k) + + oher p k = 1 s he me of enhanemen proessng exep oher alulang. g s he GOP sze. p (k) s he me of k h alulang for he frame n a GOP. s he me of alulang for he I frame n a GOP. p (k) vares beause of dfferen onen of P frames. To analyze he equaon of speedup more easly, we make wo assumpons. Usng he average me p of alulang for he frame n a GOP subsues for p (k). Comparng he me of alulang, we gnore oher. (8) hen beomes: k h

4 Speedup Where = = p =. g + g For he same vdeo sequene where p 2, (10) g + s ons and s affeed by he SAD hreshold, frame onen and GOP sze, speedup s an nverse proporon funon of g. We run he proposed aeleraon algorhm for he same vdeo sequene wh dfferen GOP szes, and he resul s shown n Fgure 4. The nverse proporon funon fng urve mahes he expermenal resuls very well. Moreover, equaon (9) explans how he speedup hanges wh dfferen GOP szes and ondus us o se he approprae GOP sze for a needed speedup and qualy of he enhaned vdeo. In addon, our expermen shows afer adopng he above algorhm he ompuaonal me of dwndles a lo whle he ME me oupes muh of he whole proessng me. Thus, n nex subseon, we propose a fas SAD algorhm o aelerae ME. Fg. 4. Speedup of proposed fas enhanemen algorhm Aeleraon Usng Fas SAD Algorhm Expermenal resuls demonsrae he alulaon of SAD akes 60 peren of he whole ME me,.e. 40 peren of ompuaon os of he whole low lghng enhanemen algorhm. Thus, s desred for us o propose a fas SAD algorhm. Some exsng fas SAD algorhms are based on advaned hardware e.g. GPU. However, as dsussed n [8, 9, 10], n BBME, all he dsoron rerons nludng SAD are based on he nheren assumpon ha moon felds of vdeo sequenes are usually smooh and slowly varyng and eah MB onans only one rgd body and radonal moon. In mos ases where he assumpon s orre, he onvenonal dsoron reron seems heavy and unneessary. Moreover, n some oher ases, wll no be he rue moon veor when he SAD ges s mnmum f here s more han one rgd body or he moon s unradonal. In hese ases, he more pxels are sampled p he greaer error he SAD algorhm wll probably lead o. However, f we adop sub-samplng mehod n our fas SAD algorhm, no only he unneessary ompuaon n mos ases an be redued bu also he poor performane of BBME n omplex moon ondons an be mproved. Alhough he fewer samples wll redue he horzonal and veral soluon and have an effe on he auray, he error urns ou o be slgh and aepable n our algorhm f users do no have sr requremen of vdeo qualy bu need he algorhm as fas as possble. () Fg. 5. Fxed subsamplng paerns of fas SAD algorhm sandard 4:1 paern. Three-Coeffen paern. () proposed paern n hs paper There are many subsamplng paerns proposed e.g. he adapve pxel demaon n [9], he sandard 4:1 paern n [10], and e.. In expermens, we fnd fxed pxel subsamplng paern s smple and an gan he bes speedup n our algorhm. Some exsng fxed paerns are he sandard 4:1 paern n [10] shown n Fgure 5, and he Three- Coeffen paern n [11] shown n Fgure 5. We noe n hose nferor enhaned MBs, he drawbak pxels are always he edge pxels hus an unbalaned subsamplng paern fousng more on edge pxels suh as he one shown n Fgure 5 s more desrable. To make he subsamplng paern symmeral, we propose an nnovave paern shown n Fgure 5(). The sample we ake are he dagonal pxels and he edge pxels. The oal number s 60 pxels for MB. Expermen resuls show our algorhm performs as fas as he 4:1 paern whle he effes are no heavly affeed. Moreover, n omparson wh he exhausve pxel-wse SAD mehod, he proposed fas SAD mehod even gans beer enhanemen qualy wh only 20 peren of he ompuaon. 4. EXPERIMENTAL RESULTS In our expermens, we use Wndows PC wh he Inel Core 2 Duo proessor (2.0GHz) wh 3G of RAM. The vdeos resoluon s Examples of he enhanemen resuls of he proposed algorhm n Seon 2 are shown n Fgures 6, 7 and 8. The orgnal low lghng vdeo npus.e. Fgure 6, 7 and 8 exhb very lle sene nformaon due o he low llumnaon level. The enhaned resuls are shown n Fgure

5 6, 7 and 8 and he red arrows pon o areas wh noable enhanemen. In Fgure 6, he mprovemen n vsbly s obvous, nludng four vehles n he bakground and he lense plae of he vehle n he foreground. More enhanemen resuls are shown n Fgure 7 and 8. In addon, we perform he vdeo enhanemen algorhm usng boh he Co ME and he EPZS ME wh dfferen GOP sze, SAD hreshold parameers, a k and b k. In omparson wh he resul of he frame-wse enhanemen algorhm n whh all values of eah frame are alulaed, he speedup resuls are shown n Fgure 10, and he orrespondng PSNR resuls are shown n Fgure 11 (GOP sze s 10) and 11 (GOP sze s 30). From Fgure 11 we an fnd he PSNR of he EPZS ME wh T=1000, a k =0.8 and b k =40 s even hgher han he PSNR of Co ME wh T=500 n he same GOP sze. Meanwhle, as shown n Fgure 10, he speedup of EPZS ME s muh hgher han ha of Co ME oo. Ths demonsraes ha he EPZS ME helps fnd more aurae referene MBs, resulng n muh more of he alulaon of bypassed, alhough he EPZS ME akes more ompuaon oss han he Co ME. On he oher hand, we perform he vdeo enhanemen algorhm usng he Co and EPZS ME wh he fas SAD algorhm. The speedup resuls are shown n Fgure 10. The orrespondng PSNR resuls are shown n Fgure 12 (GOP sze s 10) and 12 (GOP sze s 30). From Fgure 6, we an learn he fas SAD algorhm enhanes he speedup furher for boh ME algorhms. Meanwhle, he Fgure 8 shows he PSNR of EPZS ME usng fas SAD algorhm s even enhaned somemes. To sum up, he fas SAD algorhm benefs he EPZS ME n boh speedup and vdeo qualy hus resuls n our low lghng vdeo enhanemen algorhm faser and hgher qualy. Examples of he resuls of he aeleraon algorhm are shown n Fgure 9. Fgure 9 s he 16 h frame of he npu vdeo sequene, Fgure 9 and 9() are he enhaned resuls of he same frame usng our aeleraon algorhm wh GOP sze 1, orrespondng o proessng eah frame ndependenly, and GOP sze 16. Alhough hs s he las frame of he group wh he GOP sze 16, he oupu of our algorhm wh GOP sze 16 n Fgure 9(), as ompared wh Fgure 9 n whh every frame s proessed ndependenly effevely, gans 4x speedups whou vsble nformaon loss. Fg. 6. Inpu low lgh mage 1. Enhaned mage 1 usng proposed low lghng enhanemen algorhm. Fg. 7 Inpu low lgh mage 2. Enhaned mage 2 usng proposed low lghng enhanemen algorhm. Fg. 8. Inpu low lgh mage 3. Enhaned mage 3 usng proposed low lghng enhanemen algorhm. () Fg. 9. The 16h frame of he npu vdeo sequene. Enhaned frame of wh GOP sze 1. () Enhaned frame of wh GOP sze 16.

6 Fg. 10. Speedup of proposed fas enhanemen algorhm usng Co ME and EPZS ME. Co ME and EPZS ME wh fas SAD algorhm. Fg. 11. PSNR of proposed low lghng enhanemen algorhm usng Co ME and EPZS ME. GOP sze s 10. GOP sze s 30. Fg. 12. PSNR of proposed low lghng enhanemen algorhm usng Co ME and EPZS ME wh fas SAD algorhm. GOP sze s 10. GOP sze s CONCLUSIONS AND FUTURE WORK In he paper, we desrbed a novel, effen and effen vdeo enhanemen algorhm for low lghng vdeo. The algorhm frs nvers he npu low lghng vdeo and hen apples de-hazng algorhms on he nvered vdeo before nverng he de-hazed mages agan o produe he oupu enhaned low-lghng vdeo. 4x speedups s aheved wh no vsble loss of ral nformaon by dvdng he npu sequene no GOPs and performng ME for P frames. Areas of furher mprovemen nlude faser deermnaon of unhanged MBs and mprovemens n he de-haze algorhm. Vson Drvers by Nonlnear Enhanemen of Fused Vsble and Infrared Vdeo, n Pro. IEEE Conf. Compuer Vson and Paern Reognon., San Dego, CA, Jun. 2005, pp.25. [2] Ngh Drver, Makng Drvng Safer a Ngh Rayheon Company, avalable a: hp:// [3] K. He, J. Sun, and X. Tang. Sngle Image Haze Removal Usng Dark Channel Pror, n Pro. IEEE Conf. Compuer Vson and Paern Reognon., Mam, FL, Jun. 2009, pp [4] R. Faal. Sngle Image Dehazng, n ACM SIGGRAPH 08, Los Angeles, CA, Aug. 2008, pp [5] S. G. Narasmhan, and S. K. Nayar. Chroma Framework for Vson n Bad Weaher, n Pro. IEEE Conf. Compuer Vson and Paern Reognon., Hlon Head, SC, Jun. 2000, vol. 1, pp [6] S. G. Narasmhan and S. K. Nayar. Vson and he Amosphere, n In. Journal Compuer Vson, vol. 48, no. 3, pp , [7] R. Tan. Vsbly n Bad Weaher from A Sngle Image, n Pro. IEEE Conf. Compuer Vson and Paern Reognon., Anhorage, Alaska, Jun. 2008, pp. 1-8 [8] A. Zaarn and B. Lu. Fas Algorhms for Blok Moon Esmaon, n Pro. IEEE In. Conf. Aouss, Speeh and Sgnal Proessng, San Franso, CA, 1992, pp [9] Y. L. Chan and W. C. Su. New Adapve Pxel Demaon for Blok Moon Veor Esmaon, n IEEE Trans. Crus Sys. Vdeo Tehnol., vol. 6, pp , [10] T. Koga, K. Inuma, A. Hrano, Y. Ijma, and T. Ishguro. Moon Compensaed Inerframe Codng for Vdeo Conferenng, n Pro. Nu. Teleommun. Conf., New Orleans, LA, Nov. 1981, pp. G G [11] B. Grod, and K. W. Suhlm uller. A Conen- Dependen Fas DCT for Low B-Rae Vdeo Codng, n Pro. IEEE In. Conf. Image Proess., Chago, Illnos, O. 1998, vol. 3, pp [12] A. M. Touraps. Enhaned Predve Zonal Searh for Sngle and Mulple Frame Moon Esmaon, n Pro. Vsual Communaons and Image Proessng., San Jose, CA, Jan. 2002, pp REFERENCES [1] H. Ngo, L. Tao, M. Zhang, A. Lvngson, and V. Asar. A Vsbly Improvemen Sysem for Low