EFFICIENT IMAGE RECONSTRUCTION FOR GIGAPIXEL QUANTUM IMAGE SENSORS. Stanley H. Chan and Yue M. Lu

Transcription

1 EFFICIENT IMAGE RECONSTRUCTION FOR GIGAPIXEL QUANTUM IMAGE SENSORS Stanley H. Chan and Yue M. Lu Shool of Engineering and Applied Siene Harvard Univerity, Cambridge, MA 02138, USA {han, ABSTRACT Reent advane in material, devie and fabriation tehnologie have motivated a trong momentum in developing olid-tate enor that an detet individual photon in pae and time. It ha been enviioned that uh enor an eventually ahieve very high patial reolution e.g., 10 9 pixel/hip) a well a high frame rate e.g., 10 6 frame/e). In thi paper, we preent an effiient algorithm to reontrut image from the maive binary bit-tream generated by thee enor. Baed on the onept of alternating diretion method of multiplier ADMM), we tranform the omputationally intenive optimization problem into a equene of ubproblem, eah of whih ha effiient implementation in the form of polyphae-domain filtering or pixel-wie nonlinear mapping. Moreover, we reformulate the original maximum likelihood etimation a maximum a poterior etimation by introduing a total variation prior. Numerial reult demontrate the trong performane of the propoed method, whih ahieve everal db of improvement in PSNR and require a horter runtime a ompared to tandard gradient-baed approahe. Index Term Image reontrution, quantum image enor, gigapixel imaging, ADMM 1. INTRODUCTION Thank to reent advane in material and fabriation tehnologie, there ha been an emerging la of olid-tate imaging enor that are apable of ahieving ingle-photon enitivity, ub-nanoeond time reolution and rapidly inreaing patial reolution [1, 2]. Thee devie, olletively referred to a quantum image enor QIS) in thi work, are enviioned a the next generation imaging tehnology after CMOS [3], with numerou appliation [4 7]. The operational priniple of a QIS i analogou to photographi film: Photon flux reahing a pixel of the ingle-photon detetor trigger a binary repone, thu generate a 1-bit ignal revealing the intenity of the flux during the expoure. With high frame rate e.g., 10 6 frame/e) and inreaing patial reolution e.g., 10 9 pixel/hip a enviioned in [3]), the QIS generate a maive tream of bit, whih mut be deoded to reover the underlying image. The 1-bit meaurement aquired by the QIS follow a quantized Poion proe. Therefore, image reontrution from thee meaurement an be olved via maximum likelihood etimation MLE). However, exept for imple ae where the photon flux i a pieewie ontant funtion, the MLE doe not have a loed-form olution in general, even though the problem i onvex [8]. Iterative algorithm have been propoed, e.g., [9 11], but how to improve the low onvergene i till an open problem. The work i upported, in part, by the U.S. National Siene Foundation under Grant CCF By reognizing the eparable truture of the MLE objetive funtion, we propoe a new algorithm baed on the onept of alternating diretion method of multiplier ADMM) [12, 13]. The ontribution of thi paper are two-fold. Firt, we preent fat olution to the ubproblem aoiated with the ADMM formulation. One key tehnique we ue i the polyphae repreentation, a powerful tool from multirate ignal proeing [14]. Seond, we introdue a total variation TV) prior and modify the MLE problem to a maximum a poterior MAP) problem. We how empirially that the MAP formulation ignifiantly improve the reontrution quality with only marginal inreae in omputational ot. The ret of the paper i organized a follow. We firt preent the modeling of QIS and elaborate on the MLE problem in Setion 2. In Setion 3 we diu the propoed ADMM algorithm. Experimental reult are hown in Setion 4 and we onlude in Setion MODEL AND PROBLEM FORMULATION The ignal proeing model of a QIS enor onit of two major omponent: 1) The light expoure tage whih model the propagation of light from ene to enor; ii) The ening tage whih model the onverion of the light field to the binary meaurement. A pitorial illutration i hown in Figure 1, and eah omponent i deribed a follow. Light expoure Binary ening n K g m m Photon Count y m Quantization Fig. 1: The two mathematial omponent of modeling the image formation of QIS.f. [10]) Modeling of Light Expoure Conider a light field intenity λx) of patial oordinate x. Without lo of generality we hall aume that λx) i a one-dimenional funtion and that the upport of the oordinate i the unit interval, i.e., 0 x 1. Furthermore, due to the preene of diffration limit of the len, we model λx) a a funtion in the hift-invariant pae panned by a non-negative interpolation kernel ϕx): λx) = N τ N 1 n=0 b m nϕnx n), 1) where τ i the expoure time, { n : n 0} i a et of variable enoding the ene, and N i the number of variable. The ontant N/τ i not eential, but it implifie the analyi below.

2 Suppoe that the enor onit of M pixel per unit pae and that the mth pixel over the area [m/m,m + 1)/M], then the total light expoure aumulated on the urfae area of the mth pixel within a time period[0, τ] i τ m+1)/m m = λx)dxdt = τ λx),βmx m), 2) 0 m/m where, repreent the tandard L 2 inner produt, and βx) i the box funtion ined a βx) = 1 for 0 x 1 and i zero otherwie. Subtituting 1) into 2) yield m = N 1 n=0 ng m Kn, 3) where K = M/N i the patial overampling ratio, and g m = ϕx),βkx m) 4) i the mth oeffiient of a direte filter. Uing matrix-vetor notation, the expreion in 3) an be equivalently written a = G, for appropriate R M 1, R N 1 and G R M N Modeling of Binary Sening Let y m be the number of photon impinging on the urfae of the mth pixel during an expoure period [0,τ]. We model the relationhip between m andy m a a Poion proe Y m: m e m P[Y m = y m; m] = ym y m!. 5) The binary ening of the QIS quantize the oberved photon ount y m through a threhold operation. In thi paper, we aume that the threhold level i et to unity o that the binary repone i B m = 1 if Y m 1 and B m = 0 if Y m = 0. 6) Hene, the probability of oberving B m i P[B m = 1] = 1 e m and P[B m = 0] = e m. 7) 2.3. Maximum Likelihood Etimation The image reontrution tak of the QIS an be formulated a the following maximum likelihood etimation MLE): = argmax =G = argmin =G M P[B m = b m; m] 8) M log 1 b m)e m +b m1 e m ) ). Defining F) = M log1 bm)e m + b m1 e m )) and introduing a prior p), we an further modify the MLE to a maximum a poterior MAP) etimation problem: = argmin =G In thi paper, we onider the total variation prior F) logp). 9) p) exp λ TV ), 10) where TV = D 1 and D i the firt order finite differene operator. Thu, 9) i equivalent to F)+λ D 1, ubjetto = G, 11) whih i a onvex optimization problem. In the following etion we preent an effiient algorithm to olve 11). 3. IMAGE RECONSTRUCTION ALGORITHM The propoed image reontrution algorithm i baed on the onept of alternating diretion method of multiplier ADMM) [13]. The idea i to onider the following equivalent ontrained problem F)+λ v 1, ubjetto = G, v = D, 12) by ining an auxiliary variable v R N 1. Then, we onider the augmented Lagrangian funtion: L,,v,z,r) = F)+λ v 1 z T G)+ ρ 2 G 2 r T v D)+ γ 2 v D 2, 13) where z R M 1 and r R N 1 are the Lagrange multiplier aoiated with the ontraint = G and v = D, repetively, andρ > 0 andγ > 0 are parameter that ontrol the emphai of the quadrati penalty funtion. ADMM tate that the addle point of L i the optimal olution of 12). Conequently, 12) an be olved by alternatingly olving the following equene of ubproblem: k+1) = argmin k+1) = argmin L, k),v k),z k),r k) ), L k+1),,v k),z k),r k) ), v k+1) = argmin L k+1), k+1),v,z k),r k) ), v z k+1) = z k) ρ k+1) G k+1)), r k+1) = r k) γ v k+1) D k+1)). 14a) 14b) 14) 14d) 14e) We now diu how eah of the above ubproblem i olved. For notational impliity we drop the iteration number ) k) Subproblem The -ubproblem i formulated by dropping term independent of in 14a), yielding z T G+r T D+ ρ 2 G 2 + γ 2 v D 2. 15) Setting the firt order derivative to zero, the olution of 15) i = P D [ρg T G+γD T D) 1 )] G T ρ z)+d T γv r), 16) where P D i the projetion onto the earh domain D = [0,1] N. We apply P D to enure that all etimation oflie ind. The hallenge of 16) i the matrix invere. By ontrution, D T D i a irulant matrix and hene it i diagonalizable uing the direte Fourier tranform DFT). Thu it remain to how thatg T G i alo a irulant matrix, for then the matrix invere an be implemented effiiently in the Fourier domain.

3 To how that G T G i irulant, it uffie to how that G T G i a onvolution matrix repreenting a finite impule repone FIR) filter. Expreing the operation of G and G T expliitly, we note that G i equivalent to applying an up-ampling operation followed by an FIR filter {g m}, where a G T i equivalent to applying the time reveral filter {g m} followed by a down-ampling operation. The onatenation of thee operation i the ytem hown in the middle of Figure 2. G T G K g m g m K ĝkm Fig. 2: The equivalene between the matrix operation G T G, the ytem, and the finite impule repone filterĝ Km. To implify the ytem hown in Figure 2, we denote ĝ m = g m g m 17) and plit ĝ m into K nonoverlapping polyphae omponent ĝ 0,m, ĝ 1,m,...,ĝ K 1,m, ined a ĝ k,m = ĝ Km+k, for 0 k < K. 18) Then, by uing the z-tranform and multirate identitie [14] one an how the following propoition. Propoition 1. G T G i a ingle onvolution matrix whoe impule repone i equal toĝ Km, where ĝ m i ined in 17). The impliation of Propoition 1 i that the matrix G T G ha a irulant truture given by the FIR filter ĝ Km. Thi ugget that diagonalizingg T G i equivalent to applying DFT to{ĝ Km}, whih an be done offline a {ĝ Km} i fully peified by the interpolation kernel ϕx) through 4) and 17) Subproblem Letting d = G in 14b) and eliminating term independent of, the -ubproblem an be written a SineF) i eparable, we expre 19) a F) z T + ρ 2 d 2. 19) M [ log 1 b m)e m +b m1 e m ) ) z m m + ρ 2 m dm)2]. 20) Thu, 20) i d if eah of individual term in the um i d. Conidering the mth term of 20), we need to olve two ae: if b m = 0 : min m m z m m + ρ 2 m dm)2, 21a) if b m = 1 : min m log1 e m ) z m m + ρ 2 m dm)2. 21b) We note that 21a) i a quadrati minimization. Thu it ha a unique olution given by m = 1/ρ)1 z m)+d m. 22) For 21b), the firt order optimality ondition implie that 1 zm +ρm dm) = 0. 23) m 1 e However, 23) i a tranendental equation. For arbitrary hoie of z m,d m,ρ), there i no loed form olution in general. To olve for a numerial olution of 23), we ontrut a nonlinear mapping for a fixedρ) a follow. We firt onider the equation 1 = ρx w, 24) e x 1 where we identify 24) a 23) by noting x = m and w = z m + ρd m. Let [w min, w max] be the interval in whih w i ined, we partition [w min, w max] into L equiditane ubinterval and onider a equene w l = w min +l w), 25) for l = 1,...,L, where w = w max w min)/l i the length of the ubinterval. For eah l, we numerially determine the olution of 24) uing fzero in MATLAB. Thi return a equene {xw l ) : l = 1,...,L}, whih are the olution of 24) for L partiular value ofw. For any otherw [w min, w max], we interpolate the reult uing a linear interpolation heme, given by ) xwl+1 ) xw l ) xw) = xw l )+ w w l ), 26) w l+1 w l where l i hoen uh thatw l w w l+1. We note that the omputational ot of olving 23) inlude the ontrution of a equene {xw l ) : l = 1,...,L} and a linear interpolation tep. The former an be determined offline beaue it i independent of the image data. The latter i evaluated uing 26), whih i a alar update and an be implemented in parallel v-ubproblem The v-ubproblem ined in 14) i v λ v 1 r T v D)+ γ 2 v D 2. 27) Applying the hrinkage formula [12], the olution i v = max D+r/γ λ/γ,0) ignd+r/γ). 28) The overall algorithm i ummarized in Algorithm 1. Algorithm 1 Image Reontrution for QIS while k+1) k) / k) tol do -ubproblem: for m = 1,...,M do Ifb m = 0, olve 1 z m +ρ m d m) = 0. Ifb m = 1, olve 1/1 e m ) z m +ρ m d m) = 0. end for -ubproblem: olve ρg T G+γD T D) = G T ρ z)+d T γv r). v-ubproblem: v = max D+r/γ λ/γ,0) ignd+r/γ). z-update: z = z ρ G). r-update: r = r γr D). end while

4 4. EXPERIMENTAL RESULTS 4.1. Convergene to Ideal MLE Solution The purpoe of the firt experiment i to verify the onvergene of the propoed algorithm without the TV penalty) toward the theoretial MLE olution. To thi end, we onider the ae where the photon ount m i pieewie ontant within eah ubinterval Kn m Kn+1) forn = 0,...,N 1. Thi implie that the direte filter g m i a box funtion of upport [0,K]. Conequently, the MLE ha a loed form olution [10] MLE) = P D log 1 G T b/k 2)), 29) where P D i the projetion onto the earh domain D = [0,1] N. A a omparion we alo onider a gradient deent algorithm preented in [15]. The gradient deent algorithm i applied to olve a implified ae of 11) where there i no total variation penalty i.e., λ = 0). In thi ae, we ubtitute the ontraint = G into the objetive of 11) to obtain an unontrained minimization problem in. We expet the gradient deent iterate to onverge to MLE) beaue the objetive F) of the unontrained problem i onvex. At the kth iteration, the deent update i given by k+1) = k) α F k) )/ F k) ), 30) where the gradient i with elementwie multipliation and diviion) )) F) = G 1 b)+b T e G, 31) 1 e G 4.2. Reontrution Quality To ompare the quality of the reontruted image, we onider an image of ize To imulate a realiti enario, we ine the interpolation kernel ϕx) a a Gauian kernel of ize 9 9 and unit variane. The overampling ratio i K = 16, and βx) i a box funtion of ize K K. Poion meaurement are generated and quantized at a threhold of 1 photon ount. The oberved binary data i fed into the propoed algorithm, and reult are obtained when the algorithm terminate. The internal parameter of the algorithm are ined a follow. We etλ = 10 to put moderate trength of the total variation penalty. The quadrati penalty parameter are et to ρ = 10 and γ = 1. We top the iteration of the algorithm when either the number of iteration exeed a maximum of 100, or the relative hange atifie k+1) k) / k) Figure 4 how the reult of the reontrution. In Figure 4a), we how the reontruted reult by running the propoed algorithm without total variation i.e., λ = 0). A illutrated in the previou experiment, thi reult onverge to the ideal MLE olution a we inreae the iteration number. For thi partiular image, the MLE olution ahieve a PSNR of db. In Figure 4b), we how the reult when λ = 10. Thi orrepond to the MAP olution with a total variation prior. Evidently, the PSNR value inreae to db, whih i ignifiantly higher than that of the MLE olution. In addition, we remark from Figure 3 that the runtime of the propoed ADMM algorithm i muh horter than that of the deent algorithm. Thee reult indiate that the propoed heme ahieve better reontrution quality with le omputation. and the tep ize α i determined by the tandard line earh. Figure 3 how the mean quared error MSE) a a funtion of runtime. It i evident from the figure that the propoed ADMM algorithm ha a ignifiantly fater onvergene than the deent algorithm. Here, runtime intead of iteration ount) i onidered beaue the ot per iteration of the deent algorithm i more than that of the ADMM algorithm due to the line earh, although the former require fewer iteration MSE = k) - MLE) 2 /N Gradient Deent Propoed ADMM a) MLE Solution [15] b) MAP Solution propoed) λ = 0 λ = 10 PSNR = db PSNR = db Fig. 4: Comparion of the propoed algorithm when total variation penalty i withed on/off. In both ae, we et K = runtime eond) Fig. 3: MSE = k) MLE) 2 /N a a funtion of runtime e). 5. CONCLUSION A fat image reontrution algorithm for quantum image enor i propoed. The new algorithm i baed on the onept of alternating diretion method of multiplier ADMM). We preented fat olution to eah ubproblem aoiated with the ADMM algorithm. Thee inlude a polyphae-domain filtering to ahieve effiient matrix inverion, and a nonlinear mapping to find olution of a tranendental equation. The new algorithm demontrate uperior performane and better onvergene peed than the tandard gradient deent algorithm. Future work will be foued on the analyi of olution in the large-ale limit.

5 6. REFERENCES [1] M. A. Itzler, S. Cova, M. Wahl, and A. Tomita, Introdution to the iue on ingle photon ounting: Detetor and appliation, IEEE J. Sel. Topi Quantum Eletron., vol. 13, no. 4, pp , Aug [2] W. H. P. Pernie, C. Shuk, O. Minaeva, M. Li, G. N. Goltman, A. V. Sergienko, and H. X. Tang, High-peed and higheffiieny travelling wave ingle-photon detetor embedded in nanophotoni iruit, Nature Communiation, vol. 3, pp. 1325, De [3] E. R. Foum, The Quantum Image Senor QIS): Conept and hallenge, in Pro. OSA Topial Mtg Computational Optial Sening and Imaging, Jul 2011, Paper JTuE1. [4] D. E. Shwartz, E. Charbon, and K. L. Shepard, A inglephoton avalanhe diode array for fluoreene lifetime imaging miroopy, IEEE J. Solid-State Ciruit, vol. 43, no. 11, pp , Nov [5] C. Nila, A. Roha, P. A. Bee, and E. Charbon, Deign and haraterization of a CMOS 3-D image enor baed on ingle photon avalanhe diode, IEEE J. Solid-State Ciruit, vol. 40, no. 9, pp , Sep [6] A. MCarthy, R. J. Collin, N. J. Krihel, V. Fernández, A. M. Wallae, and G. S. Buller, Long-range time-of-flight anning enor baed on high-peed time-orrelated ingle-photon ounting, Applied Opti, vol. 48, no. 32, pp , Nov [7] B. S. Robinon, A. J. Kerman, E. A. Dauler, R. J. Barron, D. O. Caplan, M. L. Steven, J. J. Carney, S. A. Hamilton, J. K. Yang, and K. K. Berggren, 781 Mbit/ photon-ounting optial ommuniation uing a uperonduting nanowire detetor, Opti Letter, vol. 31, no. 4, pp , [8] F. Yang, L. Sbaiz, E. Charbon, S. Sütrunk, and M. Vetterli, Image reontrution in the gigaviion amera, in Pro. Intl Conf. Computer Viion Workhop OMNIVIS), 2009, pp [9] M. Uner and M. Eden, Maximum likelihood etimation of linear ignal parameter for Poion proee, IEEE Tran. Aout., Speeh, Signal Proe., vol. 36, no. 6, pp , Jun [10] F. Yang, Y. M. Lu, L. Sbaiz, and M. Vetterli, Bit from photon: Overampled image aquiition uing binary poion tatiti, IEEE Tran. Image Proe., vol. 21, no. 4, pp , Apr [11] J. Zhang, F. Yang, T. Vogelang, D. G. Stork, and M. Vetterli, Multihannel ampling of low light level ene wth unknown hift, in Pro. Intl Conf. Image Proe., 2013, pp [12] S. H. Chan, R. Khohabeh, K. B. Gibon, P. E. Gill, and T. Q. Nguyen, An augmented Lagrangian method for total variation video retoration, IEEE Tran. Image Proe., vol. 20, no. 11, pp , Nov [13] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Ektein, Ditributed optimization and tatitial learning via the alternating diretion method of multiplier, Found. Trend Mah. Learn., vol. 3, no. 1, pp , Jan [14] P. P. Vaidyanathan, Multirate Sytem and Filter Bank, Prentie Hall, [15] F. Yang, Y. M. Lu, L. Sbaiz, and M. Vetterli, An optimal algorithm for reontruting image from binary meaurement, in Pro. IS&T/SPIE Conf. Computational Imaging VIII, 2010, vol. 7533, paper 75330K.