Super-Resolution Dr. Yossi Rubner yossi@rubner.co.il Many slides from Mii Elad - Technion 5/5/2007
53 images, ratio :4 Example - Video
40 images ratio :4 Example Surveillance
Example Enhance Mosaics
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Maximum lielihood Robust solution MAP using bilateral filter Special cases and related problems Translational Motion Interpolation / Denoising / Deblurring SR in time
Intuition For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough ( D), perfect reconstruction is possible. D D
Intuition Due to our limited camera resolution, we sample using an insufficient 2D grid 2D 2D
Intuition However, if we tae a second picture, shifting the camera slightly to the right we obtain: 2D 2D
Intuition Similarly, by shifting down we get a third image: 2D 2D
Intuition And finally, by shifting down and to the right we get the fourth image: 2D 2D
It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed. Intuition
What if the camera displacement is Arbitrary? What if the camera rotates? Gets closer to the object (zoom)? Rotation/Scale/Disp.
There is no sampling theorem covering this case Rotation/Scale/Disp.
Further Complications Complicated motion perspective, local motion, Blur sampling is not a point operation Different effect depending on geometrical warp Noise
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Maximum lielihood Robust solution MAP using bilateral filter Special cases and related problems Translational Motion Interpolation / Denoising / Deblurring SR in time
Image Formation Scene HR Geometric transformation F Optical Blur H Sampling D Noise LR Can we write these steps as linear operators? HR D H F LR
Geometric Transformation F Scene Geometric transformation Any appropriate motion model Every frame has different transformation Usually found by a separate registration algorithm
Geometric Transformation Can be modeled as a linear operation F F F F
Optical Blur H Geometric transformation Optical Blur Due to the lens PSF Usually H H
Optical Blur Can be modeled as a linear operation H H H H
Sampling D Optical Blur Sampling Pixel operation consists of area integration followed by decimation The integration can be part of H D is the decimation only Usually D D
H PSF * PIEL H
Fill Factor Real pixel area Geometrically, the fill-factor is less than 80% for circular microlenses since the area of an unit circle is π/4 0.7854 Square-shaped micro-lenses have poor focusing capabilities
Decimation Can be modeled as a linear operation D D D o 0 0 o 0... 0 D 0
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Maximum lielihood Robust solution MAP using bilateral filter Special cases and related problems Translational Motion Interpolation / Denoising / Deblurring SR in time
Super-Resolution - Model Geometric warp Blur Decimation High- Resolution Image F I H D V Y Low- Resolution Exposures Additive Noise F N H N D N Y N Y V N { 2 0, } D H F + V, V ~ N σ n N
Simplified Model Geometric warp Blur Decimation High- Resolution Image F I H D V Y Low- Resolution Exposures Additive Noise F N H D Y N Y V N { 2 0, } DHF + V, V ~ N σ n N
The Super-Resolution Problem Y DHF + V { 0, 2 } σ, V ~ N n Given Y The measured images (noisy, blurry, down-sampled..) H The blur can be extracted from the camera characteristics D The decimation is dictated by the required resolution ratio F The warp can be estimated using motion estimation σ n The noise can be extracted from the camera / image Recover HR image
The Model as One Equation 200,000 Y Y M Y 2 N DHF DHF2 M DHFN V V + M V 2 N r resolution factor 4 MM size of the frames 0000 N number of frames 20 F H D Y ofsize ofsize ofsize ofsize ofsize 2 2 2 2 [ r M r M ] 2 2 2 2 [ r M r M ] 2 2 2 [ M r M ] 2 2 [ r M ] 2 [ M ] [60,00060,000] [60,00060,000] [0,00060,000] [60,000] [0,000] Linear algebra notation is intended only to develop algorithm
SR - Solutions Maximum Lielihood (ML): argmin N N DHF Y 2 Often ill posed problem! Maximum Aposteriori Probability (MAP) argmin DHF Y 2 { } +λa Smoothness constraint regularization
ML Reconstruction (LS) ( ) N ML Y 2 2 DHF ε Minimize: ( ) ( ) 0 ˆ 2 2 N T T T ML Y DHF H D F ε Thus, require: N T T T N T T T Y ˆ D H F DHF D H F A B B A ˆ
LS - Iterative Solution Steepest descent ˆ N T T T n+ ˆ β n F H D n ( DHF ˆ Y ) Bac projection Simulated error All the above operations can be interpreted as operations performed on images. There is no actual need to use the Matrix-Vector notations as shown here.
LS - Iterative Solution Steepest descent ˆ N T T T n+ ˆ n β F H D n ( DHF ˆ Y ) For..N ˆ n Y geometry wrap convolve with H down sample - up sample convolve with H T inverse geometry wrap F H D T D T H T F -β ˆ n+
Example HR image LR + noise 4 Least squares Simulated example from Farisu at al. IEEE trans. On Image Processing, 04
Robust Reconstruction Cases of measurements outlier: Some of the images are irrelevant Error in motion estimation, Error in the blur function General model mismatch
Robust Reconstruction Minimize: ε 2 N ( ) DHF Y ˆ N T T T n+ ˆ β n F H D sign n ( DHF ˆ Y )
Robust Reconstruction Steepest descent For..N ˆ n ˆ N T T T n+ ˆ n β F H D sign n Y ( DHF ˆ Y ) geometry wrap convolve with H down sample - sign up sample convolve with H T inverse geometry wrap F H D T D T H T F -β ˆ n+
Example - Outliers Least squares HR image LR + noise 4 Simulated example from Farisu at al. IEEE trans. On Image Processing, 04 Robust Reconstruction
Example Registration Error L 2 norm based L norm based 20 images, ratio :4
MAP Reconstruction 2 MAP N ( ) DHF Y λa{ } ε + Regularization term: 2 Tihonov cost function A T { } Γ 2 Total variation Bilateral filter A TV { } A B P P { } l P m P α l + m S l x S m y
Robust Estimation + Regularization ( ) + + P P l P P m m y l x m l N S S Y 2 α λ ε DHF Minimize: ( ) [ ] ( ) + + + P P l P P m n m y l x n m y l x m l N n T T T n n S S S S I Y ˆ ˆ sign ˆ sign ˆ ˆ α λ β DHF H D F
Robust Estimation + Regularization ˆ β N P P l + m ( ) + [ ] ( ) l m l m DHF ˆ Y λ I S S sign ˆ S S ˆ T T T ˆ n F H D sign n n+ α l P m P x y n x y n For..N geometry wrap convolve with H down sample Y - sign up sample convolve with H T inverse geometry wrap ˆ n For l,m-p..p -β ˆ + n horizontal shift l vertical shift m - - m + l sign horizontal shift -l vertical shift -m λα From Farisu at al. IEEE trans. On Image Processing, 04
Example 8 frames Resolution factor of 4 From Farisu at al. IEEE trans. On Image Processing, 04
Images from Vigilant Ltd.
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Maximum lielihood Robust solution MAP using bilateral filter Special cases and related problems Translational Motion Interpolation / Denoising / Deblurring SR in time Limitations of Super-Resolution
Special Case Translational Motion In this case H and F commute: T T HF F H H F F T H T Y DHF + V DF H + V DF Z + V Z H SR is decomposed into 2 steps. Find blur HR image from LR images non-iterative 2. Deconvolve the result using H iterative
Intuition Y DF Z + V Z H PSF* ZPIEL*PSF* Using the samples can, at most, reconstruct Z To recover, need to deconvolve Z
Step I Find Blurred HR Minimize: ε 2 ML N ( Z) DF Z Y 2 L 2 For all frames, copy registered pixels to HR grid and average [Elad & Hel-Or, 0] L For all frames, copy registered pixels to HR grid and use median [Farisu, 04]
Solution for L 2 ( ) N ML Y Z Z 2 2 DF ε Minimize: ( ) 0 2 Z Z ε ML Thus, require: N T T N T T Y P D F DF D F R Z P ˆ R Average of HR grid Diagonal, number Of occurrences per HR grid
Step II - Deblur Minimize: ( ) H Z λa{ } 2 ε + ˆ ˆ β ˆ { ( ) + { } T H signhˆ Z A ˆ n+ n n λ n n
Example 6464 LR 256256 Before deblur 256256 After deblur From Pham at al. Proc. Of SPIE-IS&T, 05. Simulated.
Related Problems Denoising (multiple frames) Y + V, V { 0, σ 2 } ~ N n Denoising (single frame) Deblurring Y + V, Y H + V, { 0, σ 2 } V ~ N n Interpolation single-image super-resolution { 0, σ 2 } V ~ N n Y DH + V, { 0, σ 2 } V ~ N n
Super-Resolution - Agenda The basic idea Image formation process Formulation and solution Maximum lielihood Robust solution MAP using bilateral filter Special cases and related problems Translational Motion Interpolation / Denoising / Deblurring SR in time
Space-Time Super-Resolution Observing events faster than frame-rate Handles: Motion aliasing Motion blur Wor and slides by Michal Irani & Yaron Caspi (ECCV 02)
Motion Aliasing The Wagon wheel effect: Slow-motion: time time time Continuous signal Sub-sampled in time Slow motion
Motion Blur
Space-Time Super-Resolution Low-resolution images video sequences: time time time High space-time resolution sequence: time time Super-resolution in space and in time.
Space-Time Super-Resolution S h (x h,y h,t h ) y t T x x y Blur ernel: PSF l S l S n Exposure time t
Example: Motion-Aliasing Input Input 2 Input 3 Input 4 25 [frames/sec]
Example: Motion-Aliasing Input sequence in slow-motion (x3): Super-resolution resolution in time (x3): 75 [frames/sec] 75 [frames/sec]
Example: Motion-Blur (simulation) Simulated sequences of fast event Long exposure-time Low frame-rate Non-uniform distribution in time Overlay of frames One low-res sequence: Another low-res sequence: And another one...
Output sequence: Output trajectory: Deblurring: Input: (x5 frame-rate) Without estimating motion of the ball! Output: 3 out of 8 low-resolution input sequences (frame overlays; trajectories):
Example: Motion-Blur (real) Frames 4 input at sequences: collision: Video Video 2 Output frame at collision: Video 3 Video 4