Gradient-domain image processing
|
|
- Kerry Flynn
- 5 years ago
- Views:
Transcription
1 Gradient-domain image processing , , Computational Photography Fall 2018, Lecture 10
2 Course announcements Homework 3 is out. - (Much) smaller than homework 2, but you should still start early to take advantage of bonus questions. - Requires a camera with flash for the second part. Grades for homework 1 have been posted. Make-up lecture will be scheduled soon. - What day does majority of the class prefer? How was Ravi s lecture on Monday? Thoughts on homework 2?
3 Overview of today s lecture Gradient-domain image processing. Basics on images and gradients. Integrable vector fields. Poisson blending. A more efficient Poisson solver. Poisson image editing examples. Flash/no-flash photography.
4 Slide credits Many of these slides were adapted from: Kris Kitani (15-463, Fall 2016). Fredo Durand (MIT). James Hays (Georgia Tech). Amit Agrawal (MERL).
5 Gradient-domain image processing
6 Someone leaked season 8 of Game of Thrones or, more likely, they made some creative use of Poisson blending
7 Application: Poisson blending originals copy-paste Poisson blending
8 More applications Removing Glass Reflections Seamless Image Stitching
9 Yet more applications Fusing day and night photos Tonemapping
10 Entire suite of image editing tools
11 Main pipeline Estimation of Gradients Images/Videos/ Meshes/Surfaces Non-Integrable Gradient Fields Reconstruction from Gradients Images/Videos/ Meshes/Surfaces Manipulation of Gradients
12 Basics of images and gradients
13 Image representation We can treat images as scalar fields (i.e., two dimensional functions) I(x,y): R 2 R
14 Image gradients Convert the scalar field into a vector field through differentiation. scalar field I I I ( x, y ) : R 2 R vector field I = {, } : R 2 R 2 x y
15 Image gradients Convert the scalar field into a vector field through differentiation. scalar field I I I ( x, y ) : R 2 R vector field I = {, } : R 2 R 2 x y How do we do this differentiation in real discrete images?
16 Finite differences High-school reminder: definition of a derivative using forward difference
17 Finite differences High-school reminder: definition of a derivative using forward difference Alternative: use central difference For discrete signals: Remove limit and set h = 2 How do you efficiently compute this?
18 Finite differences High-school reminder: definition of a derivative using forward difference Alternative: use central difference For discrete signals: Remove limit and set h = 2 What convolution kernel does this correspond to?
19 Finite differences High-school reminder: definition of a derivative using forward difference Alternative: use central difference For discrete signals: Remove limit and set h = ? 1 0-1?
20 Finite differences High-school reminder: definition of a derivative using forward difference Alternative: use central difference For discrete signals: Remove limit and set h = 2 1D derivative filter 1 0-1
21 Image gradients Convert the scalar field into a vector field through differentiation. scalar field I I I ( x, y ) : R 2 R vector field I = {, } : R 2 R 2 x y How do we do this differentiation in real discrete images? Can we go in the opposite direction, from gradients to images?
22 Vector field integration Two core questions: When is integration of a vector field possible? How can integration of a vector field be performed?
23 Integrable vector fields
24 Integrable fields Given an arbitrary vector field (u, v), can we always integrate it into a scalar field I?? I x, y : R 2 R u x, y : R 2 R v x, y : R 2 R such that I x I y x, y x, y = u(x, y) = v(x, y)
25 Curl and divergence Curl: vector operator showing the rate of rotation of a vector field. Curl ( I ) = I Divergence: vector operator showing the isotropy of a vector field. Div ( I ) = I Do you know of some simpler versions of these operators?
26 yy xx y x y x I I y I x I I I div + = + = ), ( Curl and divergence Curl: vector operator showing the rate of rotation of a vector field. Divergence: vector operator showing the isotropy of a vector field. Can you use either of these operators to derive an integrability condition? I I Curl = ) ( xy yx x y y x I I y I x I I I y x = = det
27 Integrability condition Curl of the gradient field should be zero: Curl ( I ) = I I = yx xy 0 What does that mean intuitively?
28 Integrability condition Curl of the gradient field should be zero: Curl ( I ) = I I = yx xy 0 What does that mean intuitively? Same result independent of order of differentiation. I = yx I xy
29 Demonstration Image I x I y = How do we compute this? Div(I x, I y ) Curl(I x, I y ) I xy I yx
30 Laplace filter Basically a second derivative filter. We can use finite differences to derive it, as with first derivative filter. first-order 1D derivative filter finite difference second-order finite difference Laplace filter?
31 Laplace filter Basically a second derivative filter. We can use finite differences to derive it, as with first derivative filter. first-order 1D derivative filter finite difference second-order Laplace filter finite difference 1-2 1
32 Vector field integration Two core questions: When is integration of a vector field possible? - Use curl to check for equality of mixed partial second derivatives. How can integration of a vector field be performed?
33 Different types of integration problems Reconstructing height field from gradients Applications: shape from shading, photometric stereo Manipulating image gradients Applications: tonemapping, image editing, matting, fusion, mosaics Manipulation of 3D gradients Applications: mesh editing, video operations Key challenge: Most vector fields in applications are not integrable. Integration must be done approximately.
34 Poisson blending
35 Application: Poisson blending originals copy-paste Poisson blending
36 Key idea When blending, retain the gradient information as best as possible 3 6 source destination copy-paste Poisson blending
37 Poisson blending: 1D example bright dark two signals regular blending blending derivatives
38 Definitions and notation Notation g: source function S: destination Ω: destination domain f: interpolant function f*: destination function add image here Which one is the unknown?
39 Definitions and notation Notation g: source function S: destination Ω: destination domain f: interpolant function f*: destination function add image here How should we determine f? should it look like g? should it look like f*?
40 Interpolation criterion Variational means optimization where the unknown is an entire function Variational problem what does this term do? what does this term do? Recall... Image gradient is this known?
41 Interpolation criterion Variational means optimization where the unknown is an entire function Variational problem gradient of f looks like gradient of g f is equivalent to f* at the boundaries Recall... Image gradient Yes, since the source function g is known
42 This is where Poisson blending comes from Equivalently Poisson equation (with Dirichlet boundary conditions) what does this term do? Gradient Laplacian Divergence
43 Equivalently Poisson equation (with Dirichlet boundary conditions) Laplacian of f same as g Gradient Laplacian Divergence
44 Equivalently Poisson equation (with Dirichlet boundary conditions) so make these guys... the same How can we do this?
45 Equivalently Poisson equation (with Dirichlet boundary conditions) So for each pixel p, do: Or for discrete images: How did we compute the Laplacian?
46 Equivalently Poisson equation (with Dirichlet boundary conditions) So for each pixel p, do: Or for discrete images: Recall... Laplace filter What s known and what s unknown?
47 Equivalently Poisson equation (with Dirichlet boundary conditions) So for each pixel p, do: Or for discrete images: Recall... Laplace filter f is unknown except at the boundary g and its Laplacian are known
48 We can rewrite this as linear equation of N variables one for each pixel in destination In vector form: Linear system of equations What is this? (each pixel adds another sparse row here) How would you solve this? WARNING: requires special treatment at the borders (target boundary values are same as source )
49 Solving the linear system Convert the system to a linear least-squares problem: Expand the error: Minimize the error: Set derivative to 0 Solve for x
50 Solving the linear system Convert the system to a linear least-squares problem: In Matlab: f = A \ b Expand the error: Minimize the error: Set derivative to 0 Solve for x Note: You almost never want to compute the inverse of a matrix.
51 Integration procedures Poisson solver (i.e., least squares integration) + Generally applicable. - Matrices A can become very large. Acceleration techniques: + (Conjugate) gradient descent solvers. + Multi-grid approaches. + Pre-conditioning. + Quadtree decompositions. Alternative solvers: projection procedures. We will discuss one of these when we cover photometric stereo.
52 A more efficient Poisson solver
53 Let s look again at our optimization problem Variational problem gradient of f looks like gradient of g f is equivalent to f* at the boundaries Recall... Image gradient
54 Let s look again at our optimization problem Variational problem gradient of f looks like gradient of g f is equivalent to f* at the boundaries Recall... Image gradient And for discrete images: x y 1 0-1
55 Let s look again at our optimization problem We can use the gradient approximation to discretize the variational problem What are G, f, and v? Discrete problem min f Gf v 2 We will ignore the boundary conditions for now. Recall... Image gradient And for discrete images: x y 1 0-1
56 Let s look again at our optimization problem We can use the gradient approximation to discretize the variational problem Recall... Discrete problem matrix G formed by stacking together discrete gradients min Gf v 2 f vectorized version of the unknown image Image gradient vectorized version of the target gradient field And for discrete images: We will ignore the boundary conditions for now. x y 1 0-1
57 Let s look again at our optimization problem We can use the gradient approximation to discretize the variational problem Recall... Discrete problem matrix G formed by stacking together discrete gradients min Gf v 2 f vectorized version of the unknown image Image gradient vectorized version of the target gradient field And for discrete images: We will ignore the boundary conditions for now. x y 1 0-1
58 Let s look again at our optimization problem Recall... Discrete problem matrix G formed by stacking together discrete gradients min Gf v 2 f vectorized version of the unknown image Image gradient vectorized version of the target gradient field And for discrete images: How do we solve this optimization problem? x y 1 0-1
59 Approach 1: Compute stationary points Given the loss function: we compute its derivative: E f = Gf v 2 E f =?
60 Approach 1: Compute stationary points Given the loss function: we compute its derivative: E f = Gf v 2 E f = GT Gf v and we do what with it?
61 Approach 1: Compute stationary points Given the loss function: we compute its derivative: E f = Gf v 2 E f = GT Gf v and we set that to zero: E f = 0 GT Gf = v What is this matrix?
62 Approach 1: Compute stationary points Given the loss function: we compute its derivative: E f = Gf v 2 E f = GT Gf v and we set that to zero: E f = 0 GT Gf = v It is equal to the Laplacian matrix A we derived previously!
63 Reminder from variational case Poisson equation (with Dirichlet boundary conditions) So for each pixel p, do: Or for discrete images: Recall... Laplace filter What s known and what s unknown?
64 Reminder from variational case linear equation of N variables one for each pixel in destination In vector form: Linear system of equations What is this? (each pixel adds another sparse row here) Same system as: G T Gf = v We arrive at the same system, no matter whether we discretize the continuous Laplace equation or the variational optimization problem.
65 Approach 1: Compute stationary points Given the loss function: we compute its derivative: E f = Gf v 2 E f = GT Gf v and we set that to zero: E f = 0 GT Gf = v Solving this is exactly as expensive as what we had before.
66 Approach 2: Use gradient descent Given the loss function: E f = Gf v 2 we compute its derivative: E f = GT Gf v = Af v r We call this term the residual
67 Approach 2: Use gradient descent Given the loss function: E f = Gf v 2 we compute its derivative: E f = GT Gf v = Af v r We call this term the residual and then we iteratively compute a solution: f i+1 = f i η i r i for i = 0, 1,, N, where η i are positive step sizes
68 Selecting optimal step sizes Make derivative of loss function with respect to E f = Gf v 2 η i equal to zero: E f i+1 = G f i η i r i v 2 E f i+1 r i = v A f i η i r i T r i = 0 η i = ri T r i r i T Ar i
69 Gradient descent Given the loss function: E f = Gf v 2 Minimize by iteratively computing: f i+1 = f i η i r i, r i = v Af i, Is this cheaper than the pseudo-inverse approach? η i = ri T r i r i T Ar i for i = 0, 1,, N
70 Gradient descent Given the loss function: E f = Gf v 2 Minimize by iteratively computing: f i+1 = f i η i r i, r i = v Af i, η i = ri T r i r i T Ar i Is this cheaper than the pseudo-inverse approach? We never need to compute A, only its products with vectors f, r. for i = 0, 1,, N
71 Gradient descent Given the loss function: E f = Gf v 2 Minimize by iteratively computing: f i+1 = f i η i r i, r i = v Af i, η i = ri T r i r i T Ar i Is this cheaper than the pseudo-inverse approach? We never need to compute A, only its products with vectors f, r. Vectors f, r are images. for i = 0, 1,, N
72 Gradient descent Given the loss function: E f = Gf v 2 Minimize by iteratively computing: f i+1 = f i η i r i, r i = v Af i, η i = ri T r i r i T Ar i for i = 0, 1,, N Is this cheaper than the pseudo-inverse approach? We never need to compute A, only its products with vectors f, r. Vectors f, r are images. Because A is the Laplacian matrix, these matrix-vector products can be efficiently computed using convolutions with the Laplacian kernel.
73 In practice: conjugate gradient descent Given the loss function: E f = Gf v 2 Minimize by iteratively computing: f i+1 = f i + η i d i, r i = v Af i, for i = 0, 1,, N d i+1 = r i+1 + β i+1 d i, Smarter way for selecting update directions Everything can still be done β i+1 = ri+1 T r i+1 r i T r i η i = di T r i d i T Ad i using convolutions
74 Does the initialization f 0 matter? Note: initialization
75 Note: initialization Does the initialization f 0 matter? It doesn t matter in terms of what final f we converge to, because the loss function is convex. E f = Gf v 2
76 Note: initialization Does the initialization f 0 matter? It doesn t matter in terms of what final f we converge to, because the loss function is convex. E f = Gf v 2 It does matter in terms of convergence speed. We typically use a multi-grid approach: - Solve an initial problem for a very low-resolution f (e.g., 2x2). - Use the solution to initialize gradient descent for a higher resolution f (e.g., 4x4). - Use the solution to initialize gradient descent for a higher resolution f (e.g., 8x8). - Use the solution to initialize gradient descent for an f with the original resolution NxN.
77 Poisson image editing examples
78 Photoshop s healing brush Slightly more advanced version of what we covered here: Uses higher-order derivatives
79 Contrast problem Loss of contrast when pasting from dark to bright: Contrast is a multiplicative property. With Poisson blending we are matching linear differences.
80 Contrast problem Loss of contrast when pasting from dark to bright: Contrast is a multiplicative property. With Poisson blending we are matching linear differences. Solution: Do blending in log-domain.
81 More blending originals copy-paste Poisson blending
82 Blending transparent objects
83 Blending objects with holes
84 Editing
85 Concealment How would you do this with Poisson blending?
86 Concealment How would you do this with Poisson blending? Insert a copy of the background.
87 Texture swapping
88 How would you do this? Special case: membrane interpolation
89 Special case: membrane interpolation How would you do this? Poisson problem Laplacian problem
90 Flash/no-flash photography
91 Red Eye
92 Unflattering Lighting
93 Motion Blur
94 Noise
95 A lot of Noise
96 Ruined Ambiance
97 Flash No-Flash + Low Noise + Sharp - Artificial Light - Jarring Look - High Noise - Lacks Detail + Ambient Light + Natural Look
98 Image acquisition 1 Lock Focus & Aperture time
99 Image acquisition 1 Lock Focus & Aperture 2 No-Flash Image 1/30 s ISO 3200 time
100 Image acquisition 1 Lock Focus & Aperture 2 3 No-Flash Image 1/30 s ISO 3200 Flash Image 1/125 s ISO 200 time
101 Denoising Result
102 Show a larger result here No-Flash
103 Denoising Result
104 Key idea Denoise the no-flash image while maintaining the edge structure of the flash image How would you do this using the image editing techniques we ve learned about?
105 Denoising with bilateral filtering noisy input bilateral filtering median filtering
106 Denoising with bilateral filtering However, results still have noise or blur (or both) ambient flash Bilateral filter
107 Denoising with joint bilateral filtering In the flash image there are much more details Use the flash image F to find edges
108 Denoising with joint bilateral filtering Bilateral filter The difference Joint Bilateral filter
109 Not all edges in the flash image are real Can you think of any types of edges that may exist in the flash image but not the ambient one?
110 Not all edges in the flash image are real shadows specularities May cause over- or under-blur in joint bilateral filter We need to eliminate their effect
111 Detecting shadows Observation: the pixels in the flash shadow should be similar to the ambient image. Not identical: 1. Noise. 2. Inter-reflected flash. Compute a shadow mask. Take pixel p if is manually adjusted Mask is smoothed and dilated
112 Detecting specularities Take pixels where sensor input is close to maximum (very bright). Over fixed threshold Create a specularity mask. Also smoothed. M the combination of shadow and specularity masks: Where M p =1, we use A Base. For other pixels we use A NR.
113 Detail transfer Denoising cannot add details missing in the ambient image Exist in flash image because of high SNR We use a quotient image: Reduces the effect of noise in F Multiply with A NR to add the details Masked in the same way Bilateral filtered Why does this quotient image make sense for detail?
114 Detail transfer Denoising cannot add details missing in the ambient image Exist in flash image because of high SNR We use a quotient image: Reduces the effect of noise in F
115 Full pipeline
116 Demonstration ambient-only joint bilateral and detail transfer
117 Can we do similar flash/no-flash fusion tasks with gradient-domain processing?
118 Removing self-reflections and hot-spots Ambient Flash
119 Removing self-reflections and hot-spots Ambient Flash Face Hands Tripod
120 Removing self-reflections and hot-spots Ambient Result Reflection Layer Flash
121 Idea: look at how gradients are affected Same gradient vector direction Flash Gradient Vector Ambient Gradient Vector Ambient Flash No reflections
122 Idea: look at how gradients are affected Different gradient vector direction Reflection Ambient Gradient Vector Flash Gradient Vector Ambient Flash With reflections
123 Idea: look at how gradients are affected Different gradient vector direction Reflection Ambient Gradient Vector Flash Gradient Vector Ambient Flash With reflections
124 Gradient projections Residual Gradient Vector Flash Gradient Vector Result Gradient Vector Ambient Flash Result Residual
125 Flash/no-flash with gradient-domain processing X Y Flash Intensity Gradient Result X 2D Integration 2D Result Y Integration X Vector Projection Result Y Ambient
126 Flash
127 No-Flash
128 No-Flash
129 Result
130 Flash
131 No-Flash
132 No-Flash
133 Result
134 Flash
135 No-Flash
136 Flash
137 No-Flash
138 Result
139 References Basic reading: Szeliski textbook, Sections 3.13, 3.5.5, 9.3.4, Pérez et al., Poisson Image Editing, SIGGRAPH The original Poisson Image Editing paper. Agrawal and Raskar, Gradient Domain Manipulation Techniques in Vision and Graphics, ICCV 2007 course, A great resource (entire course!) for gradient-domain image processing. Petschnigg et al., Digital photography with flash and no-flash image pairs, SIGGRAPH Eisemann and Durand, Flash Photography Enhancement via Intrinsic Relighting, SIGGRAPH The first two papers exploring the idea of photography with flash and no-flash pairs, both using variants of the joint bilateral filter. Agrawal et al., Removing Photography Artifacts Using Gradient Projection and Flash-Exposure Sampling, SIGGRAPH A subsequent paper on photography with flash and no-flash pairs, using gradient-domain image processing. Additional reading: Georgiev, Covariant Derivatives and Vision, ECCV An paper from Adobe on the version of Poisson blending implemented in Photoshop s healing brush. Elder and Goldberg, Image editing in the contour domain, PAMI One of the very first papers discussing gradient-domain image processing. Szeliski, Locally adapted hierarchical basis preconditioning, SIGGRAPH A standard reference on multi-grid and preconditioning techniques for accelerating the Poisson solver. Bhat et al., Fourier Analysis of the 2D Screened Poisson Equation for Gradient Domain Problems, ECCV A paper discussing the (Fourier) basis projection approach for solving the Poisson integration problem. Bhat et al., GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering, ToG A paper describing gradient-domain processing as a general image processing paradigm, which can be used for a broad set of applications beyond blending, including tone-mapping, colorization, converting to grayscale, edge enhancement, image abstraction and non-photorealistic rendering. Krishnan and Fergus, Dark Flash Photography, SIGGRAPH A paper proposing doing flash/no-flash photography using infrared flash lights. Kettunen et al., Gradient-domain path tracing, SIGGRAPH In addition to editing images in the gradient-domain, we can also directly render them in the gradient-domain. Tumblin et al., Why I want a gradient camera? CVPR We can even directly measure images in the gradient domain, using so-called gradient cameras.
Subsampling and image pyramids
Subsampling and image pyramids http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 2018, Lecture 3 Course announcements Homework 0 and homework 1 will be posted tonight. - Homework 0 is not required
More informationComputational Photography
Computational Photography Si Lu Spring 208 http://web.cecs.pdx.edu/~lusi/cs50/cs50_computati onal_photography.htm 04/0/208 Last Time o Digital Camera History of Camera Controlling Camera o Photography
More informationShai Avidan Tel Aviv University
Image Editing in the Gradient Domain Shai Avidan Tel Aviv Universit Slide Credits (partial list) Rick Szeliski Steve Seitz Alosha Eros Yacov Hel-Or Marc Levo Bill Freeman Fredo Durand Slvain Paris Image
More informationTemplates, Image Pyramids, and Filter Banks
Templates, Image Pyramids, and Filter Banks 09/9/ Computer Vision James Hays, Brown Slides: Hoiem and others Review. Match the spatial domain image to the Fourier magnitude image 2 3 4 5 B A C D E Slide:
More informationMath (P)Review Part II:
Math (P)Review Part II: Vector Calculus Computer Graphics Assignment 0.5 (Out today!) Same story as last homework; second part on vector calculus. Slightly fewer questions Last Time: Linear Algebra Touched
More informationLecture 7: Edge Detection
#1 Lecture 7: Edge Detection Saad J Bedros sbedros@umn.edu Review From Last Lecture Definition of an Edge First Order Derivative Approximation as Edge Detector #2 This Lecture Examples of Edge Detection
More informationFourier Transform and Frequency Domain
Fourier Transform and Frequency Domain http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2017, Lecture 6 Course announcements Last call for responses to Doodle
More informationRandom walks and anisotropic interpolation on graphs. Filip Malmberg
Random walks and anisotropic interpolation on graphs. Filip Malmberg Interpolation of missing data Assume that we have a graph where we have defined some (real) values for a subset of the nodes, and that
More informationCSE 473/573 Computer Vision and Image Processing (CVIP)
CSE 473/573 Computer Vision and Image Processing (CVIP) Ifeoma Nwogu inwogu@buffalo.edu Lecture 11 Local Features 1 Schedule Last class We started local features Today More on local features Readings for
More informationImage Filtering. Slides, adapted from. Steve Seitz and Rick Szeliski, U.Washington
Image Filtering Slides, adapted from Steve Seitz and Rick Szeliski, U.Washington The power of blur All is Vanity by Charles Allen Gillbert (1873-1929) Harmon LD & JuleszB (1973) The recognition of faces.
More informationGradient Domain High Dynamic Range Compression
Gradient Domain High Dynamic Range Compression Raanan Fattal Dani Lischinski Michael Werman The Hebrew University of Jerusalem School of Computer Science & Engineering 1 In A Nutshell 2 Dynamic Range Quantity
More informationTRACKING and DETECTION in COMPUTER VISION Filtering and edge detection
Technischen Universität München Winter Semester 0/0 TRACKING and DETECTION in COMPUTER VISION Filtering and edge detection Slobodan Ilić Overview Image formation Convolution Non-liner filtering: Median
More informationLinear Diffusion. E9 242 STIP- R. Venkatesh Babu IISc
Linear Diffusion Derivation of Heat equation Consider a 2D hot plate with Initial temperature profile I 0 (x, y) Uniform (isotropic) conduction coefficient c Unit thickness (along z) Problem: What is temperature
More informationFast Local Laplacian Filters: Theory and Applications
Fast Local Laplacian Filters: Theory and Applications Mathieu Aubry (INRIA, ENPC), Sylvain Paris (Adobe), Sam Hasinoff (Google), Jan Kautz (UCL), and Frédo Durand (MIT) Input Unsharp Mask, not edge-aware
More informationGradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering
GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering Pravin Bhat C. Lawrence Zitnick Michael Cohen Brian Curless Presentation : Michael Tao Discussion : Sean Arietta Input
More informationEdge Detection in Computer Vision Systems
1 CS332 Visual Processing in Computer and Biological Vision Systems Edge Detection in Computer Vision Systems This handout summarizes much of the material on the detection and description of intensity
More informationLecture 04 Image Filtering
Institute of Informatics Institute of Neuroinformatics Lecture 04 Image Filtering Davide Scaramuzza 1 Lab Exercise 2 - Today afternoon Room ETH HG E 1.1 from 13:15 to 15:00 Work description: your first
More informationScreen-space processing Further Graphics
Screen-space processing Rafał Mantiuk Computer Laboratory, University of Cambridge Cornell Box and tone-mapping Rendering Photograph 2 Real-world scenes are more challenging } The match could not be achieved
More informationFace recognition Computer Vision Spring 2018, Lecture 21
Face recognition http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 2018, Lecture 21 Course announcements Homework 6 has been posted and is due on April 27 th. - Any questions about the homework?
More informationDigital Matting. Outline. Introduction to Digital Matting. Introduction to Digital Matting. Compositing equation: C = α * F + (1- α) * B
Digital Matting Outline. Introduction to Digital Matting. Bayesian Matting 3. Poisson Matting 4. A Closed Form Solution to Matting Presenting: Alon Gamliel,, Tel-Aviv University, May 006 Introduction to
More informationFourier Transform and Frequency Domain
Fourier Transform and Frequency Domain http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 2018, Lecture 3 (part 2) Overview of today s lecture Some history. Fourier series. Frequency domain. Fourier
More informationMotion Estimation (I)
Motion Estimation (I) Ce Liu celiu@microsoft.com Microsoft Research New England We live in a moving world Perceiving, understanding and predicting motion is an important part of our daily lives Motion
More informationImage preprocessing in spatial domain
Image preprocessing in spatial domain Sharpening, image derivatives, Laplacian, edges Revision: 1.2, dated: May 25, 2007 Tomáš Svoboda Czech Technical University, Faculty of Electrical Engineering Center
More informationImage Processing /6.865 Frédo Durand A bunch of slides by Bill Freeman (MIT) & Alyosha Efros (CMU)
Image Processing 6.815/6.865 Frédo Durand A bunch of slides by Bill Freeman (MIT) & Alyosha Efros (CMU) define cumulative histogram work on hist eq proof rearrange Fourier order discuss complex exponentials
More informationJustin Solomon MIT, Spring 2017
Fisher et al. Design of Tangent Vector Fields (SIGGRAPH 2007) Justin Solomon MIT, Spring 2017 Original version from Stanford CS 468, spring 2013 (Butscher & Solomon) Divergence Theorem Green s Theorem
More informationBlobs & Scale Invariance
Blobs & Scale Invariance Prof. Didier Stricker Doz. Gabriele Bleser Computer Vision: Object and People Tracking With slides from Bebis, S. Lazebnik & S. Seitz, D. Lowe, A. Efros 1 Apertizer: some videos
More informationMotion Estimation (I) Ce Liu Microsoft Research New England
Motion Estimation (I) Ce Liu celiu@microsoft.com Microsoft Research New England We live in a moving world Perceiving, understanding and predicting motion is an important part of our daily lives Motion
More informationEdges and Scale. Image Features. Detecting edges. Origin of Edges. Solution: smooth first. Effects of noise
Edges and Scale Image Features From Sandlot Science Slides revised from S. Seitz, R. Szeliski, S. Lazebnik, etc. Origin of Edges surface normal discontinuity depth discontinuity surface color discontinuity
More informationA Riemannian Framework for Denoising Diffusion Tensor Images
A Riemannian Framework for Denoising Diffusion Tensor Images Manasi Datar No Institute Given Abstract. Diffusion Tensor Imaging (DTI) is a relatively new imaging modality that has been extensively used
More informationImage enhancement. Why image enhancement? Why image enhancement? Why image enhancement? Example of artifacts caused by image encoding
13 Why image enhancement? Image enhancement Example of artifacts caused by image encoding Computer Vision, Lecture 14 Michael Felsberg Computer Vision Laboratory Department of Electrical Engineering 12
More informationPoisson Matting. Abstract. 1 Introduction. Jian Sun 1 Jiaya Jia 2 Chi-Keung Tang 2 Heung-Yeung Shum 1
Poisson Matting Jian Sun 1 Jiaya Jia 2 Chi-Keung Tang 2 Heung-Yeung Shum 1 1 Microsoft Research Asia 2 Hong Kong University of Science and Technology Figure 1: Pulling of matte from a complex scene. From
More informationDigital Image Processing: Sharpening Filtering in Spatial Domain CSC Biomedical Imaging and Analysis Dr. Kazunori Okada
Homework Exercise Start project coding work according to the project plan Adjust project plans according to my comments (reply ilearn threads) New Exercise: Install VTK & FLTK. Find a simple hello world
More informationEE 367 / CS 448I Computational Imaging and Display Notes: Image Deconvolution (lecture 6)
EE 367 / CS 448I Computational Imaging and Display Notes: Image Deconvolution (lecture 6) Gordon Wetzstein gordon.wetzstein@stanford.edu This document serves as a supplement to the material discussed in
More informationPoisson Image Editing
Poisson Image Editing Sébastien Boisgérault, Mines ParisTech, under CC BY-NC-SA 4.0 January 21, 2016 Contents Introduction 1 Modelling the Problem 2 Harmonic Functions 5 The irichlet Problem 8 Harmonic/Fourier
More informationPhotometric Stereo: Three recent contributions. Dipartimento di Matematica, La Sapienza
Photometric Stereo: Three recent contributions Dipartimento di Matematica, La Sapienza Jean-Denis DUROU IRIT, Toulouse Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 1 / 32 Outline 1 Shape-from-X techniques
More informationImage Alignment and Mosaicing Feature Tracking and the Kalman Filter
Image Alignment and Mosaicing Feature Tracking and the Kalman Filter Image Alignment Applications Local alignment: Tracking Stereo Global alignment: Camera jitter elimination Image enhancement Panoramic
More informationSingle Exposure Enhancement and Reconstruction. Some slides are from: J. Kosecka, Y. Chuang, A. Efros, C. B. Owen, W. Freeman
Single Exposure Enhancement and Reconstruction Some slides are from: J. Kosecka, Y. Chuang, A. Efros, C. B. Owen, W. Freeman 1 Reconstruction as an Inverse Problem Original image f Distortion & Sampling
More informationRecap of Monday. Linear filtering. Be aware of details for filter size, extrapolation, cropping
Recap of Monday Linear filtering h[ m, n] k, l f [ k, l] I[ m Not a matrix multiplication Sum over Hadamard product k, n l] Can smooth, sharpen, translate (among many other uses) 1 1 1 1 1 1 1 1 1 Be aware
More informationECE Digital Image Processing and Introduction to Computer Vision
ECE592-064 Digital Image Processing and Introduction to Computer Vision Depart. of ECE, NC State University Instructor: Tianfu (Matt) Wu Spring 2017 Outline Recap, image degradation / restoration Template
More informationPoisson Image Editing
Poisson Image Editing Sébastien Boisgérault, Mines ParisTech, under CC BY-NC-SA 4.0 November 28, 2017 Contents Introduction 1 Modelling the Problem 2 Harmonic Functions 5 The irichlet Problem 8 Harmonic/Fourier
More informationLecture 8: Interest Point Detection. Saad J Bedros
#1 Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu Last Lecture : Edge Detection Preprocessing of image is desired to eliminate or at least minimize noise effects There is always tradeoff
More informationCITS 4402 Computer Vision
CITS 4402 Computer Vision Prof Ajmal Mian Adj/A/Prof Mehdi Ravanbakhsh, CEO at Mapizy (www.mapizy.com) and InFarm (www.infarm.io) Lecture 04 Greyscale Image Analysis Lecture 03 Summary Images as 2-D signals
More informationLecture 8: Interest Point Detection. Saad J Bedros
#1 Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu Review of Edge Detectors #2 Today s Lecture Interest Points Detection What do we mean with Interest Point Detection in an Image Goal:
More informationPSET 0 Due Today by 11:59pm Any issues with submissions, post on Piazza. Fall 2018: T-R: Lopata 101. Median Filter / Order Statistics
CSE 559A: Computer Vision OFFICE HOURS This Friday (and this Friday only): Zhihao's Office Hours in Jolley 43 instead of 309. Monday Office Hours: 5:30-6:30pm, Collaboration Space @ Jolley 27. PSET 0 Due
More informationRobust flash denoising/deblurring by iterative guided filtering
RESEARCH Open Access Robust flash denoising/deblurring by iterative guided filtering Hae-Jong Seo 1* and Peyman Milanfar 2 Abstract A practical problem addressed recently in computational photography is
More informationNotes on Regularization and Robust Estimation Psych 267/CS 348D/EE 365 Prof. David J. Heeger September 15, 1998
Notes on Regularization and Robust Estimation Psych 67/CS 348D/EE 365 Prof. David J. Heeger September 5, 998 Regularization. Regularization is a class of techniques that have been widely used to solve
More informationGeometric Modeling Summer Semester 2012 Linear Algebra & Function Spaces
Geometric Modeling Summer Semester 2012 Linear Algebra & Function Spaces (Recap) Announcement Room change: On Thursday, April 26th, room 024 is occupied. The lecture will be moved to room 021, E1 4 (the
More informationGradient Descent and Implementation Solving the Euler-Lagrange Equations in Practice
1 Lecture Notes, HCI, 4.1.211 Chapter 2 Gradient Descent and Implementation Solving the Euler-Lagrange Equations in Practice Bastian Goldlücke Computer Vision Group Technical University of Munich 2 Bastian
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Jason E. Hicken Aerospace Design Lab Department of Aeronautics & Astronautics Stanford University 14 July 2011 Lecture Objectives describe when CG can be used to solve Ax
More informationI Chen Lin, Assistant Professor Dept. of CS, National Chiao Tung University. Computer Vision: 4. Filtering
I Chen Lin, Assistant Professor Dept. of CS, National Chiao Tung University Computer Vision: 4. Filtering Outline Impulse response and convolution. Linear filter and image pyramid. Textbook: David A. Forsyth
More informationImage pyramids and frequency domain
Image pyramids and frequency domain http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 2019, Lecture 3 Course announcements Homework 1 will be posted tonight. - Homework 1 is due on February 6
More informationReading. 3. Image processing. Pixel movement. Image processing Y R I G Q
Reading Jain, Kasturi, Schunck, Machine Vision. McGraw-Hill, 1995. Sections 4.-4.4, 4.5(intro), 4.5.5, 4.5.6, 5.1-5.4. 3. Image processing 1 Image processing An image processing operation typically defines
More informationScientific Computing II
Technische Universität München ST 008 Institut für Informatik Dr. Miriam Mehl Scientific Computing II Final Exam, July, 008 Iterative Solvers (3 pts + 4 extra pts, 60 min) a) Steepest Descent and Conjugate
More information2 Regularized Image Reconstruction for Compressive Imaging and Beyond
EE 367 / CS 448I Computational Imaging and Display Notes: Compressive Imaging and Regularized Image Reconstruction (lecture ) Gordon Wetzstein gordon.wetzstein@stanford.edu This document serves as a supplement
More informationIntroduction to Computer Vision. 2D Linear Systems
Introduction to Computer Vision D Linear Systems Review: Linear Systems We define a system as a unit that converts an input function into an output function Independent variable System operator or Transfer
More informationWhat is Image Deblurring?
What is Image Deblurring? When we use a camera, we want the recorded image to be a faithful representation of the scene that we see but every image is more or less blurry, depending on the circumstances.
More informationImage Alignment and Mosaicing
Image Alignment and Mosaicing Image Alignment Applications Local alignment: Tracking Stereo Global alignment: Camera jitter elimination Image enhancement Panoramic mosaicing Image Enhancement Original
More informationGaussian derivatives
Gaussian derivatives UCU Winter School 2017 James Pritts Czech Tecnical University January 16, 2017 1 Images taken from Noah Snavely s and Robert Collins s course notes Definition An image (grayscale)
More informationCS 4495 Computer Vision Binary images and Morphology
CS 4495 Computer Vision Binary images and Aaron Bobick School of Interactive Computing Administrivia PS6 should be working on it! Due Sunday Nov 24 th. Some issues with reading frames. Resolved? Exam:
More informationMultimedia Databases. Previous Lecture. 4.1 Multiresolution Analysis. 4 Shape-based Features. 4.1 Multiresolution Analysis
Previous Lecture Multimedia Databases Texture-Based Image Retrieval Low Level Features Tamura Measure, Random Field Model High-Level Features Fourier-Transform, Wavelets Wolf-Tilo Balke Silviu Homoceanu
More informationPDE Solvers for Fluid Flow
PDE Solvers for Fluid Flow issues and algorithms for the Streaming Supercomputer Eran Guendelman February 5, 2002 Topics Equations for incompressible fluid flow 3 model PDEs: Hyperbolic, Elliptic, Parabolic
More informationCSE 559A: Computer Vision EVERYONE needs to fill out survey.
CSE 559A: Computer Vision ADMINISRIVIA EVERYONE needs to fill out survey. Setup git and Anaconda, send us your public key, and do problem set 0. Do immediately: submit public key and make sure you can
More informationMultimedia Databases. Wolf-Tilo Balke Philipp Wille Institut für Informationssysteme Technische Universität Braunschweig
Multimedia Databases Wolf-Tilo Balke Philipp Wille Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 4 Previous Lecture Texture-Based Image Retrieval Low
More informationThe Frequency Domain, without tears. Many slides borrowed from Steve Seitz
The Frequency Domain, without tears Many slides borrowed from Steve Seitz Somewhere in Cinque Terre, May 2005 CS194: Image Manipulation & Computational Photography Alexei Efros, UC Berkeley, Fall 2016
More informationMultiscale Image Transforms
Multiscale Image Transforms Goal: Develop filter-based representations to decompose images into component parts, to extract features/structures of interest, and to attenuate noise. Motivation: extract
More informationMultimedia Databases. 4 Shape-based Features. 4.1 Multiresolution Analysis. 4.1 Multiresolution Analysis. 4.1 Multiresolution Analysis
4 Shape-based Features Multimedia Databases Wolf-Tilo Balke Silviu Homoceanu Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 4 Multiresolution Analysis
More informationErkut Erdem. Hacettepe University February 24 th, Linear Diffusion 1. 2 Appendix - The Calculus of Variations 5.
LINEAR DIFFUSION Erkut Erdem Hacettepe University February 24 th, 2012 CONTENTS 1 Linear Diffusion 1 2 Appendix - The Calculus of Variations 5 References 6 1 LINEAR DIFFUSION The linear diffusion (heat)
More informationADMINISTRIVIA SENSOR
CSE 559A: Computer Vision ADMINISRIVIA Everyone needs to fill out survey and join Piazza. We strongly encourage you to use Piazza over e-mail for questions (mark only to instructors if you wish). Also
More informationPDEs in Image Processing, Tutorials
PDEs in Image Processing, Tutorials Markus Grasmair Vienna, Winter Term 2010 2011 Direct Methods Let X be a topological space and R: X R {+ } some functional. following definitions: The mapping R is lower
More information6.869 Advances in Computer Vision. Bill Freeman, Antonio Torralba and Phillip Isola MIT Oct. 3, 2018
6.869 Advances in Computer Vision Bill Freeman, Antonio Torralba and Phillip Isola MIT Oct. 3, 2018 1 Sampling Sampling Pixels Continuous world 3 Sampling 4 Sampling 5 Continuous image f (x, y) Sampling
More informationCS4670: Computer Vision Kavita Bala. Lecture 7: Harris Corner Detec=on
CS4670: Computer Vision Kavita Bala Lecture 7: Harris Corner Detec=on Announcements HW 1 will be out soon Sign up for demo slots for PA 1 Remember that both partners have to be there We will ask you to
More informationFraunhofer Institute for Computer Graphics Research Interactive Graphics Systems Group, TU Darmstadt Fraunhoferstrasse 5, Darmstadt, Germany
Scale Space and PDE methods in image analysis and processing Arjan Kuijper Fraunhofer Institute for Computer Graphics Research Interactive Graphics Systems Group, TU Darmstadt Fraunhoferstrasse 5, 64283
More informationLecture 18 Classical Iterative Methods
Lecture 18 Classical Iterative Methods MIT 18.335J / 6.337J Introduction to Numerical Methods Per-Olof Persson November 14, 2006 1 Iterative Methods for Linear Systems Direct methods for solving Ax = b,
More informationAnnouncements. Filtering. Image Filtering. Linear Filters. Example: Smoothing by Averaging. Homework 2 is due Apr 26, 11:59 PM Reading:
Announcements Filtering Homework 2 is due Apr 26, :59 PM eading: Chapter 4: Linear Filters CSE 52 Lecture 6 mage Filtering nput Output Filter (From Bill Freeman) Example: Smoothing by Averaging Linear
More informationFeature extraction: Corners and blobs
Feature extraction: Corners and blobs Review: Linear filtering and edge detection Name two different kinds of image noise Name a non-linear smoothing filter What advantages does median filtering have over
More information3. Lecture. Fourier Transformation Sampling
3. Lecture Fourier Transformation Sampling Some slides taken from Digital Image Processing: An Algorithmic Introduction using Java, Wilhelm Burger and Mark James Burge Separability ² The 2D DFT can be
More informationImage Stitching II. Linda Shapiro CSE 455
Image Stitching II Linda Shapiro CSE 455 RANSAC for Homography Initial Matched Points RANSAC for Homography Final Matched Points RANSAC for Homography Image Blending What s wrong? Feathering + 1 0 1 0
More informationFrequency, Vibration, and Fourier
Lecture 22: Frequency, Vibration, and Fourier Computer Graphics CMU 15-462/15-662, Fall 2015 Last time: Numerical Linear Algebra Graphics via linear systems of equations Why linear? Have to solve BIG problems
More informationOptical Flow, Motion Segmentation, Feature Tracking
BIL 719 - Computer Vision May 21, 2014 Optical Flow, Motion Segmentation, Feature Tracking Aykut Erdem Dept. of Computer Engineering Hacettepe University Motion Optical Flow Motion Segmentation Feature
More informationLecture 6: Edge Detection. CAP 5415: Computer Vision Fall 2008
Lecture 6: Edge Detection CAP 5415: Computer Vision Fall 2008 Announcements PS 2 is available Please read it by Thursday During Thursday lecture, I will be going over it in some detail Monday - Computer
More informationCS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS LAST TIME Intro to cudnn Deep neural nets using cublas and cudnn TODAY Building a better model for image classification Overfitting
More informationIntelligent Visual Prosthesis
Orientation sensor (IMU) Intelligent Visual Prosthesis Depth image-based obstacle detection Depth camera Wideangle RGB camera Simultaneous object recognition, localization, and hand tracking New projects:
More informationEdge Detection. Introduction to Computer Vision. Useful Mathematics Funcs. The bad news
Edge Detection Introduction to Computer Vision CS / ECE 8B Thursday, April, 004 Edge detection (HO #5) Edge detection is a local area operator that seeks to find significant, meaningful changes in image
More informationIntroduction to Nonlinear Image Processing
Introduction to Nonlinear Image Processing 1 IPAM Summer School on Computer Vision July 22, 2013 Iasonas Kokkinos Center for Visual Computing Ecole Centrale Paris / INRIA Saclay Mean and median 2 Observations
More informationLecture Notes 5: Multiresolution Analysis
Optimization-based data analysis Fall 2017 Lecture Notes 5: Multiresolution Analysis 1 Frames A frame is a generalization of an orthonormal basis. The inner products between the vectors in a frame and
More informationCS 3710: Visual Recognition Describing Images with Features. Adriana Kovashka Department of Computer Science January 8, 2015
CS 3710: Visual Recognition Describing Images with Features Adriana Kovashka Department of Computer Science January 8, 2015 Plan for Today Presentation assignments + schedule changes Image filtering Feature
More informationPoisson composi-ng 3
1 naïve composi-ng 2 Poisson composi-ng 3 our result 4 Objec+ves Seamless composi+ng Robust to inaccurate selec+on Output Quality - Limit color bleeding Time- Performance - Efficient method 5 Related Work:
More informationImage Stitching II. Linda Shapiro EE/CSE 576
Image Stitching II Linda Shapiro EE/CSE 576 RANSAC for Homography Initial Matched Points RANSAC for Homography Final Matched Points RANSAC for Homography Image Blending What s wrong? Feathering + 1 0 ramp
More informationConjugate gradient acceleration of non-linear smoothing filters
MITSUBISHI ELECTRIC RESEARCH LABORATORIES http://www.merl.com Conjugate gradient acceleration of non-linear smoothing filters Knyazev, A.; Malyshev, A. TR25-43 December 25 Abstract The most efficient edge-preserving
More informationFinal Examination. CS 205A: Mathematical Methods for Robotics, Vision, and Graphics (Fall 2013), Stanford University
Final Examination CS 205A: Mathematical Methods for Robotics, Vision, and Graphics (Fall 2013), Stanford University The exam runs for 3 hours. The exam contains eight problems. You must complete the first
More informationMATH 333: Partial Differential Equations
MATH 333: Partial Differential Equations Problem Set 9, Final version Due Date: Tues., Nov. 29, 2011 Relevant sources: Farlow s book: Lessons 9, 37 39 MacCluer s book: Chapter 3 44 Show that the Poisson
More informationImage Reconstruction And Poisson s equation
Chapter 1, p. 1/58 Image Reconstruction And Poisson s equation School of Engineering Sciences Parallel s for Large-Scale Problems I Chapter 1, p. 2/58 Outline 1 2 3 4 Chapter 1, p. 3/58 Question What have
More informationIterative Image Registration: Lucas & Kanade Revisited. Kentaro Toyama Vision Technology Group Microsoft Research
Iterative Image Registration: Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group Microsoft Research Every writer creates his own precursors. His work modifies our conception of the past, as
More informationKazhdan, Bolitho and Hoppe. Poisson Surface Reconstruction 3/3
Kazhdan, Bolitho and Hoppe Poisson Surface Reconstruction 3/3 Siddhartha Chaudhuri http://www.cse.iitb.ac.in/~cs749 Recap of differential operators (in 3D) ( x y z) Gradient (of scalar-valued function):
More informationLifting Detail from Darkness
Lifting Detail from Darkness J.P.Lewis zilla@computer.org Disney The Secret Lab Lewis / Detail from Darkness p.1/38 Brightness-Detail Decomposition detail image image intensity Separate detail by Wiener
More informationComputer Vision & Digital Image Processing
Computer Vision & Digital Image Processing Image Restoration and Reconstruction I Dr. D. J. Jackson Lecture 11-1 Image restoration Restoration is an objective process that attempts to recover an image
More informationE27 SPRING 2015 ZUCKER PROJECT 2 LAPLACIAN PYRAMIDS AND IMAGE BLENDING
E27 SPRING 2015 ZUCKER PROJECT 2 PROJECT 2 LAPLACIAN PYRAMIDS AND IMAGE BLENDING OVERVIEW In this project, you will investigate two applications: Laplacian pyramid blending and hybrid images. You can use
More informationCSC321 Lecture 20: Reversible and Autoregressive Models
CSC321 Lecture 20: Reversible and Autoregressive Models Roger Grosse Roger Grosse CSC321 Lecture 20: Reversible and Autoregressive Models 1 / 23 Overview Four modern approaches to generative modeling:
More informationNonparametric Bayesian Methods (Gaussian Processes)
[70240413 Statistical Machine Learning, Spring, 2015] Nonparametric Bayesian Methods (Gaussian Processes) Jun Zhu dcszj@mail.tsinghua.edu.cn http://bigml.cs.tsinghua.edu.cn/~jun State Key Lab of Intelligent
More information