Noise, Image Reconstruction with Noise! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367! Lecture 10! Gordon Wetzstein! Stanford University!
What s a Pixel?! photon to electron converter!! photoelectric effect!! source: Molecular Expressions! wikipedia!
What s a Pixel?! source: Molecular Expressions! microlens: focus light on photodiode! color filter: select color channel! quantum efficiency: ~50%! fill factor: fraction of surface area used for light gathering! photon-to-charge conversion and ADC includes noise!
sensitivity! of sensor! to light! digital gain! ISO ( film speed )! bobatkins.com!
! Noise! noise is (usually) bad! many sources of noise: heat, electronics, amplifier gain, photon to electron conversion, pixel defects, read,! let s start with something simple: fixed pattern noise!
Fixed Pattern Noise!! dead or hot pixels, dust,! remove with dark frame calibration:!!! I = I! I captured dark I! white! I dark! on RAW image! not JPEG (nonlinear)! Emil Martinec!
Noise! other than that, different noise follows different statistical distributions, these two are crucial:! Gaussian! Poisson!
!!! Gaussian Noise!! thermal, read, amplifier! additive, signal-independent, zero-mean! + =
with i.i.d. Gaussian noise:! (independent and identically-distributed)! Gaussian Noise! b = x +! x! N ( x,0)!! N ( 0," 2 ) additive Gaussian noise!
with i.i.d. Gaussian noise:! Gaussian Noise! b = x + η x N ( x,0) ( ) (independent and identically-distributed)! η N 0,σ 2 b N ( x,σ 2 ) p( b x,σ ) = 1 2πσ 2 e b x ( ) 2 2σ 2
with i.i.d. Gaussian noise:! Gaussian Noise! b = x +! x! N ( x,0) ( ) (independent and identically-distributed)!!! N 0," 2 b! N ( x,! 2 ) Bayes rule:! p( b x,! ) = 1 2"! 2 e# ( b#x) 2 2! 2
with i.i.d. Gaussian noise:! Gaussian Noise - MAP! b = x +! x! N ( x,0) ( ) (independent and identically-distributed)!!! N 0," 2 b! N ( x,! 2 ) Bayes rule:! maximum-a-posteriori estimation:! p( b x,! ) = 1 2"! 2 e# ( b#x) 2 2! 2
Gaussian Noise MAP Flat Prior! trivial solution (not useful in practice):! p( x) = 1
Gaussian Noise MAP Self Similarity Prior! Gaussian denoisers like non-local means and other selfsimilarity priors actually solve this problem:!
General Self Similarity Prior! generic proximal operator for function f(x):! proximal operator for some image prior!
General Self Similarity Prior! can use self-similarity as general image prior (not just for denoising)! proximal operator for some image prior!
General Self Similarity Prior! can use self-similarity as general image prior (not just for denoising)!! 2 = " /!! = " /! (h parameter in most NLM! implementations is std. dev.)! proximal operator for some image prior!
Image Reconstruction with Gaussian Noise! x N Ax,0 with i.i.d. Gaussian noise:! b = Ax + η ( ) ( ) (independent and identically-distributed)! η N 0,σ 2
Image Reconstruction with Gaussian Noise! x N Ax,0 with i.i.d. Gaussian noise:! b = Ax + η ( ) ( ) (independent and identically-distributed)! η N 0,σ 2 b N ( Ax,σ 2 ) p( b x,σ ) = 1 2πσ 2 e b Ax ( ) 2 2σ 2
Image Reconstruction with Gaussian Noise! x! N Ax,0 with i.i.d. Gaussian noise:! b = Ax +! ( ) ( ) (independent and identically-distributed)!!! N 0," 2 b! N ( Ax,! 2 ) Bayes rule:! maximum-a-posteriori estimation:! p( b x,! ) = 1 2"! 2 e# ( b#ax) 2 2! 2 regularized least squares (use ADMM)!
Scientific Sensors! e.g., Andor ixon Ultra 897: cooled to -100 C! scientific CMOS & CCD! reduce pretty much all noise, except for photon noise!
What is Photon (or Shot) Noise?! noise caused by the fluctuation when the incident light is converted to charge (detection)! also observed in the light emitted by a source, i.e. due to particle nature of light (emission)! same for re-emission à cascading Poisson processes are also described by a Poisson process [Teich and Saleh 1998]!
Photon Noise! noise is signal dependent! for N measured photo-electrons! standard deviation is, variance is! mean is! N Poisson distribution! σ = N σ 2 = N f (k; N) = N k e N k!
Photon Noise - SNR! signal-to-noise ratio SNR = N N SNR in db N photons:!! = N 2N photons:! nonlinear!! = 2 N 1,000 10,000 N
Photon Noise - SNR! signal-to-noise ratio SNR = N N N photons:!! = N 2N photons:! nonlinear!! = 2 N wikipedia!
Maximum Likelihood Solution for Poisson Noise! image formation:! probability of measurement i:! joint probability of all M measurements (use notation trick ):!
Maximum Likelihood Solution for Poisson Noise!!!!! log-likelihood function:! gradient:!
Maximum Likelihood Solution for Poisson Noise! Richardson-Lucy algorithm: iterative approach to ML estimation for Poisson noise! simple idea:! 1.! at solution, gradient will be zero!! 2.! when converged, further iterations will not change, i.e.!
! Richardson-Lucy Algorithm! equate gradient to zero! rearrange so that 1 is on one side of equation! = 0
! Richardson-Lucy Algorithm! equate gradient to zero! rearrange so that 1 is on one side of equation! = 0 set equal to!
! Richardson-Lucy Algorithm! for any multiplicative update rule scheme:! start with positive initial guess (e.g. random values)! apply iteration scheme! future updates are guaranteed to remain positive! always get smaller residual! RL multiplicative update rules:!
Richardson-Lucy Algorithm - Deconvolution! Blurry & Noisy Measurement! RL Deconvolution!
Richardson-Lucy Algorithm! what went wrong?! Poisson deconvolution is a tough problem, without priors it s pretty much hopeless! let s try to incorporate the one prior we have learned: total variation!
!! Richardson-Lucy Algorithm + TV! log-likelihood function:! gradient:!
!!! Richardson-Lucy Algorithm + TV! gradient of anisotropic TV term! this is dirty : possible division by 0!!
!! Richardson-Lucy Algorithm + TV! follow the same logic as RL, RL+TV multiplicative update rules:! 2 major problems & solution hacks :! 1.! still possible division by 0 when gradient is zero!! set fraction to 0 if gradient is 0 2.! multiplicative update may become negative!! only work with (very) small values for lambda
(A Dirty but Easy Approach to) Richardson-Lucy with a TV Prior! Measurements! RL! RL+TV, lambda=0.005! RL+TV, lambda=0.025! Log Residual! Mean Squared Error!
Signal-to-Noise Ratio (SNR)! SNR = = mean pixel value standard deviation of pixel value = µ σ PQ e t PQ e t + Dt + N r 2 signal! noise! P = incident photon flux (photons/pixel/sec) Q e = quantum efficiency t = eposure time (sec) D = dark current (electroncs/pixel/sec), including hot pixels N r = read noise (rms electrons/pixel), including fixed pattern noise
Signal-to-Noise Ratio (SNR)! SNR = mean pixel value standard deviation of pixel value = µ! signal! noise! = PQ e t 2 PQ e t + Dt + N r signal-dependent! signal-independent! P = incident photon flux (photons/pixel/sec) Q e = quantum efficiency t = eposure time (sec) D = dark current (electroncs/pixel/sec), including hot pixels N r = read noise (rms electrons/pixel), including fixed pattern noise
Next: Compressive Imaging! single pixel camera! compressive hyperspectral imaging! compressive light field imaging! Marwah et al., 2013! Wakin et al. 2006!
References and Further Reading! https://en.wikipedia.org/wiki/shot_noise! L. Lucy, 1974 An iterative technique for the rectification of observed distributions. Astron. J. 79, 745 754.! W. Richardson, 1972 Bayesian-based iterative method of image restoration J. Opt. Soc. Am. 62, 1, 55 59.! N. Dey, L. Blanc-Feraud, C. Zimmer, Z. Kam, J. Olivo-Marin, J. Zerubia, 2004 A deconvolution method for confocal microscopy with total variation regularization, In IEEE Symposium on Biomedical Imaging: Nano to Macro, 1223 1226! M. Teich, E. Saleh, 1998, Cascaded stochastic processes in optics, Traitement du Signal 15(6)! please also read the lecture notes, especially for the clean ADMM derivation for solving the maximum likelihood estimation of Poisson reconstruction with TV prior!!