Outline Camera calibration Camera projection models Camera calibration i Nonlinear least square methods A camera calibration tool Applications Digital Visual Effects Yung-Yu Chuang with slides b Richard Szeliski, Steve Seitz,, Fred Pighin and Marc Pollefes Pinhole camera Camera projection models
Pinhole camera model P (X,Y,Z origin P p (, principal point (optical center Pinhole camera model Z fx Z fy Z principal X Z principal point Y X f f fy fx ~ Z f Z fy Pinhole camera model principal X principal point Y X f f fy fx ~ Z f Z fy Principal point offset principal point intrinsic matri X X K I ~ onl related to camera projection Y X f f fy fx ~ Z f Z fy
Intrinsic matri Distortion Is this form of K good enough? f K f non-square piels (digital video skew fa s radial distortion K f Radial distortion of the image No distortion Pin cushion Barrel Caused b imperfect lenses Deviations are most noticeable for ras that pass through the edge of the lens Camera rotation and translation X ' X Y ' R3 3 Y t Z' Z X f ~ KR tx Y ~ f R t Z etrinsic matri wo kinds of parameters internal or intrinsic parameters such as focal length, optical center, aspect ratio: what kind of camera? eternal or etrinsic (pose parameters including rotation and translation: where is the camera?
Other projection models Orthographic projection Special case of perspective projection Distance from the COP to the PP is infinite it Image World Also called parallel projection : (,, z (, Other tpes of projections Illusion Scaled orthographic Also called weak k perspective Affine projection Also called paraperspective
Illusion Fun with perspective Perspective cues Perspective cues
Fun with perspective Forced perspective in LOR Ames room Ames video BBC stor Camera calibration Camera calibration Estimate both intrinsic and etrinsic parameters. wo main categories:. Photometric calibration: uses reference objects with known geometr 2. Self calibration: onl assumes static scene, e.g. structure from motion
Camera calibration approaches Chromaglphs (HP research. linear regression (least squares 2. nonlinear optimization i i Linear regression ~ K R t X MX Camera calibration
Linear regression Linear regression Directl estimate unknowns in the M matri using known 3D points (X i,yy i,zz i and measured feature positions (u i,v i Linear regression Linear regression Solve for Projection Matri M using least-square techniques
Normal equation Given an overdetermined sstem A b the normal equation is that which minimizes the sum of the square differences between left and right sides A A A b Linear regression Advantages: All specifics of the camera summarized in one matri Can predict where an world point will map to in the image Disadvantages: Doesn t tell us about particular parameters Mies up internal and eternal parameters pose specific: move the camera and everthing breaks More unknowns than true degrees of freedom Nonlinear optimization A probabilistic view of least square Feature measurement equations Probabilit of M given {(u i,v i } P Optimal estimation Likelihood of M given {(u i,v i } L P It is a least square problem (but not necessaril linear least square How do we minimize L?
Optimal estimation Non-linear regression (least squares, because the relations between û i and u i are non-linear functions of M unknown parameters We could have terms like f cos in this u uˆ ~ u K R t X Nonlinear least square methods known constant We can use Levenberg-Marquardt method to minimize it Least square fitting Linear least square fitting t number of data points number of parameters
Linear least square fitting Linear least square fitting model parameters model parameters ( t M ( t ; t ( t M ( t ; t t t Linear least square fitting Linear least square fitting model parameters model parameters ( t M ( t ; t t fi ( i M ( ti ; ( t M ( t ; t t fi ( i M ( ti ; residual prediction residual prediction M ( t ; t 3 t 2 i li t is linear, too.
Nonlinear least square fitting Function minimization Least square is related to function minimization. model parameters residuals t 2t M ( t; 3e 4e i [ 2 4 4,,, ] f ( M ( t ; i t 2t i 3 e 4 e i It is ver hard to solve in general. Here, we onl consider a simpler problem of finding local minimum. Function minimization Quadratic functions Approimate the function with a quadratic function within a small neighborhood
Quadratic functions A is positive definite. All eigenvalues are positive. For all, A>. negative definite Function minimization Wh? * B definition, if is a local minimizer, h is small enough * * F( h F( F( * * * h F( h F'( O( h 2 A is singular A is indefinite Function minimization Function minimization
Descent methods Descent direction Steepest descent method Line search Find so that ( F( αh is minimum the decrease of F( per unit along h direction h sd is a descent direction because h sd F ( = -F ( 2 < ( F( αh F h F' ( αh h F'(
Line search Steepest descent method h F' ( αh h F'( h h F'( h h αh ' '' F ( α F ( h h αh h h h Hh Hh isocontour gradient Steepest descent method Newton s method It has good performance in the initial stage of the iterative process. Converge ver slow with a linear rate.
Newton s method Another view E( h F( h F( h g Minimizer satisfies * E'( h E' ( h g Hh h 2 Hh Newton s method h H It requires solving a linear sstem and H is not alwas positive definite. It has good performance in the final stage of the iterative process, where is close to *. g h H g Gauss-Newton method Hbrid method Use the approimate Hessian H J J No need for second derivative H is positive semi-definite his needs to calculate second-order derivative which might not be available.
Levenberg-Marquardt method LM can be thought of as a combination of steepest descent and the Newton method. When the current solution is far from the correct one, the algorithm behaves like a steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton s method. Nonlinear least square Given a set of measurements, tr to find the best parameter vector p so that the squared distance is minimal. Here, ˆ, with ˆ f ( p. Levenberg-Marquardt method Levenberg-Marquardt method (J J Ih g μ= Newton s method μ steepest descent method Strateg for choosing μ Start t with some small μ If F is not reduced, keep tring larger μ until it does If F is reduced, d accept it and reduce μ for the net iteration
Recap (the Rosenbrock function Steepest descent k k g h h h Hh z f 2 2 2 (, ( ( 2 Global l minimum i at (, k k F' ' k g g 2 min k min 2 k
In the plane of the steepest descent direction Steepest descent ( iterations h hh h Hh k k Regularized Least- Gauss-Newton method k k H g With the approimate Hessian J H J No need for second derivative H is positive semi-definite min 2 k - H g k
Newton s method (48 evaluations Levenberg-Marquardt Blends steepest descent and Gauss-Newton At each step, solve for the descent direction i h (J J Ih g h gg If μ large,, steepest descent h (J J If μ small,, Gauss-Newton g Regularized Least- Levenberg-Marquardt (9 evaluations A popular calibration tool Regularized Least-
Multi-plane calibration Step : data acquisition Images courtes Jean-Yves Bouguet, Intel Corp. Advantage Onl requires a plane Don t have to know positions/orientations Good code available online! Intel s OpenCV librar: http://www.intel.com/research/mrl/research/opencv/ Matlab version b Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/inde.html Zhengou Zhang s web site: http://research.microsoft.com/~zhang/calib/ Step 2: specif corner order Step 3: corner etraction
Step 3: corner etraction Step 4: minimize projection error Step 4: camera calibration Step 4: camera calibration
Step 5: refinement Optimized parameters How is calibration used? Applications Good for recovering intrinsic parameters; It is thus useful for man vision applications Since it requires a calibration pattern, it is often necessar to remove or replace the pattern from the footage or utilize it in some was
Eample of calibration Eample of calibration Eample of calibration PhotoBook Videos from Gaech Dasatoo, MakeOf P!NG, MakeOf Work, MakeOf LifeInPaints, MakeOf PhotoBook MakeOf