Photometric Stereo: Three recent contributions Dipartimento di Matematica, La Sapienza Jean-Denis DUROU IRIT, Toulouse Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 1 / 32
Outline 1 Shape-from-X techniques 2 Photometric techniques 3 Integration of a normal field 4 Uncalibrated Photometric Stereo 5 Conclusion and Perspective Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 2 / 32
Outline Shape-from-X techniques 1 Shape-from-X techniques 2 Photometric techniques 3 Integration of a normal field 4 Uncalibrated Photometric Stereo 5 Conclusion and Perspective Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 3 / 32
Shape-from-X techniques 3D-scanning 3D-reconstruction 3D-scanning 3D-reconstruction + Albedo ( color) estimation Different kinds of 3D-scanning Palpation Mechanical process Telemeters Time of flight of laser pulses Kinect Projection of an infra-red pattern Photographic techniques Shape-from-X Some applications of 3D-scanning Architecture Metrology Cultural heritage Augmented reality Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 4 / 32
Shape-from-X techniques 3D-scanning 3D-reconstruction 3D-scanning 3D-reconstruction + Albedo ( color) estimation Different kinds of 3D-scanning Palpation Mechanical process Telemeters Time of flight of laser pulses Kinect Projection of an infra-red pattern Photographic techniques Shape-from-X Some applications of 3D-scanning Architecture Metrology Cultural heritage Augmented reality Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 4 / 32
Shape-from-X techniques 3D-scanning 3D-reconstruction 3D-scanning 3D-reconstruction + Albedo ( color) estimation Different kinds of 3D-scanning Palpation Mechanical process Telemeters Time of flight of laser pulses Kinect Projection of an infra-red pattern Photographic techniques Shape-from-X Some applications of 3D-scanning Architecture Metrology Cultural heritage Augmented reality Current application in our laboratory Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 4 / 32
Shape-from-X techniques Main shape-from-x techniques Geometric techniques Photometric techniques Mono-view Shape-from-structured light Shape-from-shading techniques Shape-from-shadow (n = 1 image) Shape-from-contour Shape-from-texture Multi-view Stereoscopy Photometric Stereo techniques Structure-from-motion (n > 1 images) Shape-from-silhouettes Shape-from-defocus Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 5 / 32
Shape-from-X techniques Main shape-from-x techniques Geometric techniques Photometric techniques Mono-view Shape-from-structured light Shape-from-shading techniques Shape-from-shadow (n = 1 image) Shape-from-contour Shape-from-texture Multi-view Stereoscopy Photometric Stereo techniques Structure-from-motion (n > 1 images) Shape-from-silhouettes Shape-from-defocus Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 5 / 32
Outline Photometric techniques 1 Shape-from-X techniques 2 Photometric techniques 3 Integration of a normal field 4 Uncalibrated Photometric Stereo 5 Conclusion and Perspective Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 6 / 32
Photometric techniques Photometric techniques: Definition Data n 1 images of a scene under the same camera pose n light vectors S i, i [1, n] Results Always: Depth map Sometimes: Albedo map S 1 S 2 S 3 Depth map + Albedo map 3D-model Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 7 / 32
Photometric techniques Shape-from-shading (SFS) / Photometric Stereo (PS) 2 or 3 unknowns per pixel Always: Normal N such that N = 1 2 unknowns Sometimes: Albedo ρ [0, 1] 1 additional unknown n equations per pixel Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) If ρ unknown: M = ρ N R 3 Linear equations Known albedo Unknown albedo 2 unknowns per pixel 3 unknowns per pixel + Non-linear equations + Linear equations n = 1 SFS: Under-constrained SFSρ: Under-constrained++ n = 2 PS2: Well-constrained PS2ρ: Under-constrained n = 3 PS: Over-constrained PS: Well-constrained n > 3 PS: Over-constrained++ PS: Over-constrained Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 8 / 32
Photometric techniques Shape-from-shading (SFS) / Photometric Stereo (PS) 2 or 3 unknowns per pixel Always: Normal N such that N = 1 2 unknowns Sometimes: Albedo ρ [0, 1] 1 additional unknown n equations per pixel Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) If ρ unknown: M = ρ N R 3 Linear equations Known albedo Unknown albedo 2 unknowns per pixel 3 unknowns per pixel + Non-linear equations + Linear equations n = 1 SFS: Under-constrained SFSρ: Under-constrained++ n = 2 PS2: Well-constrained PS2ρ: Under-constrained n = 3 PS: Over-constrained PS: Well-constrained n > 3 PS: Over-constrained++ PS: Over-constrained Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 8 / 32
Photometric techniques Shape-from-shading (SFS) / Photometric Stereo (PS) 2 or 3 unknowns per pixel Always: Normal N such that N = 1 2 unknowns Sometimes: Albedo ρ [0, 1] 1 additional unknown n equations per pixel Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) If ρ unknown: M = ρ N R 3 Linear equations Known albedo Unknown albedo 2 unknowns per pixel 3 unknowns per pixel + Non-linear equations + Linear equations n = 1 SFS: Under-constrained SFSρ: Under-constrained++ n = 2 PS2: Well-constrained PS2ρ: Under-constrained n = 3 PS: Over-constrained PS: Well-constrained n > 3 PS: Over-constrained++ PS: Over-constrained Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 8 / 32
Photometric techniques Shape-from-shading (SFS) / Photometric Stereo (PS) 2 or 3 unknowns per pixel Always: Normal N such that N = 1 2 unknowns Sometimes: Albedo ρ [0, 1] 1 additional unknown n equations per pixel Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) If ρ unknown: M = ρ N R 3 Linear equations Known albedo Unknown albedo 2 unknowns per pixel 3 unknowns per pixel + Non-linear equations + Linear equations n = 1 SFS: Under-constrained SFSρ: Under-constrained++ n = 2 PS2: Well-constrained PS2ρ: Under-constrained n = 3 PS: Over-constrained PS: Well-constrained n > 3 PS: Over-constrained++ PS: Over-constrained Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 8 / 32
Photometric techniques Shape-from-shading (SFS) / Photometric Stereo (PS) 2 or 3 unknowns per pixel Always: Normal N such that N = 1 2 unknowns Sometimes: Albedo ρ [0, 1] 1 additional unknown n equations per pixel Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) If ρ unknown: M = ρ N R 3 Linear equations Known albedo Unknown albedo 2 unknowns per pixel 3 unknowns per pixel + Non-linear equations + Linear equations n = 1 SFS: Under-constrained SFSρ: Under-constrained++ n = 2 PS2: Well-constrained PS2ρ: Under-constrained n = 3 PS: Over-constrained PS: Well-constrained n > 3 PS: Over-constrained++ PS: Over-constrained Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 8 / 32
Photometric techniques Photometric techniques: Other variants More realistic models Camera projection: Orthographic or perspective Light beams: Parallel or point light sources Uncalibrated variant Additional unknowns S i, i [1, n] Worse constrained Direct search for the depth u Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) Relation between N and u: N = 1/ u 2 x + u 2 y + 1 [ u x, u y, 1] (2) Computer vision: Non-differential approach First resolution of (1), then resolution of (2) Mathematics: Differential approach Elimination of N from (1) + (2) n coupled PDEs in u Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 9 / 32
Photometric techniques Photometric techniques: Other variants More realistic models Camera projection: Orthographic or perspective Light beams: Parallel or point light sources Uncalibrated variant Additional unknowns S i, i [1, n] Worse constrained Direct search for the depth u Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) Relation between N and u: N = 1/ u 2 x + u 2 y + 1 [ u x, u y, 1] (2) Computer vision: Non-differential approach First resolution of (1), then resolution of (2) Mathematics: Differential approach Elimination of N from (1) + (2) n coupled PDEs in u Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 9 / 32
Photometric techniques Photometric techniques: Other variants More realistic models Camera projection: Orthographic or perspective Light beams: Parallel or point light sources Uncalibrated variant Additional unknowns S i, i [1, n] Worse constrained Direct search for the depth u Lambert s law: Greylevel I i = ρ N S i, i [1, n] (1) Relation between N and u: N = 1/ u 2 x + u 2 y + 1 [ u x, u y, 1] (2) Computer vision: Non-differential approach First resolution of (1), then resolution of (2) Mathematics: Differential approach Elimination of N from (1) + (2) n coupled PDEs in u Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 9 / 32
Photometric techniques Non-differential approach / Differential approach Non-differential approach Easier to handle (mainly for computer scientists!) Resolution of (2) required (called integration ): N = 1/ u 2 x + u 2 y + 1 [ u x, u y, 1] (2) Differential approach Better constrained since the elimination of N = [N 1, N 2, N 3 ] from (1) + (2) implicitely supposes: u xy = u yx y u x = x u y y (N 1 /N 3 ) = x (N 2 /N 3 ) Usually requires boundary conditions Problems which could benefit from the differential approach SFS Much studied PS2ρ Galileo project with La Sapienza? Uncalibrated PS with point light sources To be done Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 10 / 32
Photometric techniques Non-differential approach / Differential approach Non-differential approach Easier to handle (mainly for computer scientists!) Resolution of (2) required (called integration ): N = 1/ u 2 x + u 2 y + 1 [ u x, u y, 1] (2) Differential approach Better constrained since the elimination of N = [N 1, N 2, N 3 ] from (1) + (2) implicitely supposes: u xy = u yx y u x = x u y y (N 1 /N 3 ) = x (N 2 /N 3 ) Usually requires boundary conditions Problems which could benefit from the differential approach SFS Much studied PS2ρ Galileo project with La Sapienza? Uncalibrated PS with point light sources To be done Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 10 / 32
Photometric techniques Main advantages of Photometric Stereo Advantages of Photometric Stereo vs. Shape-from-shading Well-posed and linear if n 3 non-coplanar sources No assumption required on the albedo (uniform albedo, etc.) No need to restrict the 3D-reconstruction to a family of objects (applicable surfaces, human faces, etc.) Advantages of Photometric Stereo vs. Stereoscopy No external geometric calibration (single camera pose) No matching No problem such as occlusions Estimate of the shape and of the albedo 3D-model Dense and accurate 3D-reconstruction (applications in metrology) Limits of Photometric Stereo Limited to images taken under very controlled conditions 2.5D-reconstruction (depth map) instead of real 3D-reconstruction Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 11 / 32
Photometric techniques Photometric Stereo: Three recent contributions Integration of a normal field cf. Section 3 A new fast and edge-preserving method of integration Jean-François AUJOL (IMB, Bordeaux) Yvain QUÉAU (IRIT) Uncalibrated Photometric Stereo cf. Section 4 Resolution of the Uncalibrated PS problem using Total Variation François LAUZE (DIKU, Copenhagen) Yvain QUÉAU (IRIT) Photometric Stereo with n = 2 images cf. Perspective Theoretical study and practical resolution of the PS2 problem Roberto MECCA (IIT, Genova) Xavier DESCOMBES (INRIA, Sophia Antipolis) Yvain QUÉAU (IRIT) Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 12 / 32
Outline Integration of a normal field 1 Shape-from-X techniques 2 Photometric techniques 3 Integration of a normal field 4 Uncalibrated Photometric Stereo 5 Conclusion and Perspective Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 13 / 32
Integration of a normal field Integration of a normal field: Equations Data: Unit normal vector N = [N 1, N 2, N 3 ] Unknown: Depth u Relation between N and u: N = 1/ u 2 x + u 2 y + 1 [ u x, u y, 1] (2) On the other hand: N 3 0 N = N 3 [N 1 /N 3, N 2 /N 3, 1] Identification of both expressions of N: u = [N 1 /N 3, N 2 /N 3 ] Standard notations: p = N 1 /N 3, q = N 2 /N 3 Linear PDE in u: u = [p, q] (3) Integration of a normal field N such that y (N 1 /N 3 ) = x (N 2 /N 3 ) Integration of a gradient field [p, q] = [N 1 /N 3, N 2 /N 3 ] Equation (3) is invalid when N 3 = 0 (contours of occultation) Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 14 / 32
Integration of a normal field Closed-form solution Direct resolution of the PDE (3) Equation to solve: u(x, y) = [p(x, y), q(x, y)], for (x, y) Ω Partially ill-posed: u solution u + c Closed-form solution: u(x, y) = u(x 0, y 0 ) + Constraint of integrability (x,y) (x 0,y 0 ) On any closed contour, we should have: (x0,y 0 ) (x 0,y 0 ) [p(u, v) du + q(u, v) dv] = 0 solution, c R [p(u, v) du + q(u, v) dv] This occurs iff: curl [p, q, 1] = 0 p y = q x Integrability constraint Never rigorously satisfied in practice Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 15 / 32
Integration of a normal field Quadratic minimization Least-square resolution of the PDE (3) F L2 (u) = u(x, y) [p(x, y), q(x, y)] 2 dx dy (x,y) Ω Euler-Lagrange equation: F L2 (u) = 2 div ( u [p, q]) u = p x + q y (4) Poisson equation Natural boundary condition: ( u [p, q]) η = 0 (5) η outer normal to Ω in the image plane Minimization of F L2 (u) Resolution of the Poisson equation (4) Eq. (4) needs to be constrained, since any solution to the Laplace equation u = 0 (harmonic function) can be added to a solution Constrained boundary (Dirichlet or Neumann) or free boundary (natural boundary condition (5)) Unique solution to (4) Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 16 / 32
Integration of a normal field Quadratic minimization: Discretization Discretization of the functional F L2 (u) E(u) = [ ] ((i,j),(i+1,j)) Ω 2 u i+1,j u i,j p i+1,j +p 2 i,j 2 + [ ((i,j),(i,j+1)) Ω 2 u i,j+1 u i,j q i,j+1+q i,j 2 ] 2 with { u = [ui,j ] (i,j) Ω\ Ω if constrained boundary u = [u i,j ] (i,j) Ω if free boundary Discretization of the Poisson equation (4) For (i, j) Ω\ Ω: E u i,j = 0 u i+1,j + u i,j+1 + u i 1,j + u i,j 1 4 u i,j = p i+1,j p i 1,j +q i,j+1 q i,j 1 2 Discretization of the natural boundary condition (5) For (i, j) Ω, such that (i + 1, j) Ω and (i, j + 1) Ω: = 0 u i+1,j + u i,j+1 2 u i,j = p i+1,j p i,j +q i,j+1 q i,j 2 E u i,j Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 17 / 32
Integration of a normal field Quadratic minimization: Numerical resolution System to solve: (4) + boundary condition In any case (constrained boundary or free boundary): One linear equation per pixel Square linear system A u = B (6) Resolution of (6) using a Jacobi iteration Constrained boundary: [Horn, Brooks 1986] Free boundary: [Durou, Courteille 2007] Reconstruction domain Ω not necessarily rectangular Very slow algorithms Resolution of (4) using the Fourier transform Constrained boundary: [Frankot, Chellappa 1988] Free boundary: [Simchony et al. 1990] Fast algorithms thanks to the FFT Reconstruction domain Ω necessarily rectangular Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 18 / 32
Integration of a normal field Application to Photometric Stereo (first step) S 1 S 2 S 3 Reconstruction domain Ω Estimated normal field N Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 19 / 32
Integration of a normal field Application to Photometric Stereo (second step) Non-rectangular Ω + Free boundary [Durou, Courteille 2007] Images of size 256 256: CPU = 8 s Very slow! Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 20 / 32
Integration of a normal field A simple (but wrong) idea to accelerate the integration Embed Ω in the whole image domain Ω The normals in Ω \Ω are fixed to N = [0, 0, 1] (flat ground) Rectangular Ω + Free boundary [Simchony et al. 1990] [Durou, Courteille 2007]: CPU = 8 s [Simchony et al. 1990]: CPU 0 s The integration is much accelerated But quadratic minimization distorts the surface... Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 21 / 32
Integration of a normal field Non-quadratic minimization Weighted least-square resolution of the PDE (3) (Perona-Malik like) u(x, y) [p(x, y), q(x, y)] 2 F PM (u) = dx dy 1 + a p y (x, y) q x (x, y) (x,y) Ω The denominator is minimal when p y = q x (integrability constraint) Euler-Lagrange equation: u = p x + q y + ln(1 + a p y q x ) ( u [p, q]) (7) Linear PDE in u Resolution of the PDE (7) But (7) has non-constant coefficients FFT discarded Discretization of (7) New square linear system A u = B Matrix A very sparse LU decomposition very fast New method of normal integration: u = A \B Fast, edge-preserving, Ω not necessarily rectangular Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 22 / 32
Results Integration of a normal field Discontinuous surface Direct quadratic minimization Size 128 128 [Simchony et al. 1990] RMSE = 9.47, CPU = 0.23 s Iterative non-quadratic minimization [Durou et al. 2009] RMSE = 2.87, CPU = 81.3 s Direct non-quadratic minimization Proposed method RMSE = 0.49, CPU = 0.32 s Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 23 / 32
Outline Uncalibrated Photometric Stereo 1 Shape-from-X techniques 2 Photometric techniques 3 Integration of a normal field 4 Uncalibrated Photometric Stereo 5 Conclusion and Perspective Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 24 / 32
Uncalibrated Photometric Stereo Uncalibrated Photometric Stereo: Preprocessing Lambert s law In each pixel px Ω: [I 1 px... I n px] = M px S M px = ρ px N px R 1 3, S = [S 1... S n ] R 3 n Stacking such equalities for all the pixels px Ω: I = M S (8) I R m n, M R m 3, m = card(ω) Departures from the Lambert s law From (8), we should have rank(i) 3 In practice, due to departures from the Lambert s law: n > 3 real images rank(i) = min(m, n) > 3 Preprocessing of the real images: I = arg min J of rank 3 J I F The shadows and specularities are removed: see demo UPS Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 25 / 32
Uncalibrated Photometric Stereo Uncalibrated Photometric Stereo: Ambiguities Problem formulation Find M R m 3 and S R 3 n such that I = M S (10) For any solution (M, S ) of (10), and any G GL(3): I = M G G 1 S (M G, G 1 S ) other solution of (10) Bas-relief ambiguity [Belhumeur et al. 1999] Imposing the integrability constraint on N = M/ M 1 0 0 The integrability constraint imposes G = 0 1 0 µ ν λ Generalized bas-relief ambiguity: (µ, ν) R 2 and λ > 0 Resort to the differential approach? Not useful, since the integrability constraint is already used Could be useful with point light sources To be done Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 26 / 32
Uncalibrated Photometric Stereo Generalized bas-relief ambiguity: Example Two surfaces giving the same 3 images under different ligths How to find the correct one : compare the estimated albedos Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 27 / 32
Uncalibrated Photometric Stereo Uncalibrated Photometric Stereo: Resolution 3000 2500 2000 1500 1000 500 0 0 0.2 0.4 0.6 0.8 1 1000 800 600 400 200 0 0 0.2 0.4 0.6 0.8 1 A priori knowledge: Uniform albedo First solution better Minimizing the variance of the albedo: see demo UPS More general resolution: TV-regularization of the albedo Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 28 / 32
Outline Conclusion and Perspective 1 Shape-from-X techniques 2 Photometric techniques 3 Integration of a normal field 4 Uncalibrated Photometric Stereo 5 Conclusion and Perspective Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 29 / 32
Conclusion Conclusion and Perspective Three recent contributions to Photometric Stereo Contribution 1: Integration of a normal field A new fast and edge-preserving method of integration Contribution 2: Uncalibrated Photometric Stereo Resolution of the Uncalibrated PS problem using Total Variation Contribution 3: Photometric Stereo with n = 2 images Theoretical study and practical resolution of the PS2 problem Application of Contributions 1 and 2 Society Fitting Box (Toulouse) Interactive solutions to eyewear professionals : see video Technological transfer project Photometric stereo for the 3D-scanning of faces Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 30 / 32
Conclusion and Perspective Contribution 3: Photometric Stereo with n = 2 images PS2 problem Known albedo n = 2 equations and 2 unknowns per pixel But not so well-constrained, since non-linear equations 2 solutions N in each pixel: 2 m solutions? The solution N must satisfy: y (N 1 /N 3 ) = x (N 2 /N 3 ) Differential approach Better constrained: proof of unicity [Mecca and Falcone 2013] Dirichlet boundary condition required: how to do in practice? Practical resolution I(p, q) = (i,j) Ω p i,j+1 p i,j (q i+1,j q i,j ) Minimization of I(p, q) using simulated annealing: see demo PS2 Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 31 / 32
Perspective Conclusion and Perspective Outdoors Photometric Stereo There are more and more webcams (around 2000 in Toulouse!) Huge amount of images: big data (very trendy) An urban webcam is fixed + The Sun makes a parallel beam + It moves PS Continuation of Contribution 3 All the S i are coplanar (S of rank 2) Problem PS2 Unknown albedo ρ PS2ρ Linear, but under-constrained Ideas for a possible collaboration (Galileo project) Big data Robust PS2 (shadows, specularities, moving objects) TV-regularization of the albedo Jean-Denis DUROU (IRIT, Toulouse) 17 December 2013 32 / 32