Image Processing 1 (IP1) Bildverarbeitung 1

Transcription

1 MIN-Fakultät Fachbereich Informatik Arbeitsbereich SAV/BV (KOGS Image Processing 1 (IP1 Bildverarbeitung 1 Lecture 5 Perspec<ve Transforma<ons and Interpola<on Winter Semester 2014/15 Slides: Prof. Bernd Neumann Slightly revised by: Dr. Benjamin Seppke Prof. Siegfried S<ehl

2 Perspecve Projecon Transformaon Where does a point of a scene appear in an image? Transforma<on in 3 steps: x y z? 1. scene coordinates => camera coordinates 2. projec<on of camera coordinates into image plane 3. camera coordinates => image coordinates x p y p Perspecve projecon equaons are essenal for Computer Graphics. For Image Understanding we will need the inverse: What are possible scene coordinates of a point visible in the image? This will follow later University of Hamburg, Dept. Informatics 2

3 Perspecve Projecon in Independent Coordinate Systems It is oren useful to describe real-world points, camera geometry and image points in separate coordinate systems. The formal descrip<on of projec<on involves transforma<ons between these coordinate systems. y Camera coordinates Image coordinates Scene (world coordinates x y z x Op<cal axis Scene point v = x y z Op<cal center x v 0 y z Image point v p = x p y p z p v p = x p y p z p v p = x p y p University of Hamburg, Dept. Informatics 3

4 3D Coordinate Transformaon I The new coordinate system is specified by a transla<on and rota<on with respect to the old coordinate system: v v Op<cal center = R ( v v0 0 R Rota<on matrix R may be decomposed into 3 rotaons about the coordinate axes: R = R x R y R z If rota<ons are performed in the above order: 1 γ = rota<on angle about z-axis <lt 2 β = rota<on angle about (new y-axis pan 3 α = rota<on angle about (new x-axis nick Note that these matrices describe coord. transforms for posive rotaons of the coord. system. R x = R y = R z = cosα sinα 0 sinα cosα cosβ 0 sin β sin β 0 cosβ cosγ sinγ 0 sinγ cosγ University of Hamburg, Dept. Informatics 4

5 3D Coordinate Transformaon II By mul<plying the 3 matrices R x, R y and R z one gets R = cosβ cosγ cosβ sinγ sin β sinα sin β cosγ cosα sinγ sinα sin β sinγ + cosα cosγ sinα cosβ cosα sin β cosγ + sinα sinγ cosα sin β sinγ sinα cosγ cosα cosβ For formula manipula<ons, one tries to avoid the trigonometric func<ons and takes R = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 Note that the coefficients of R are constrained: A rotaon matrix is orthonormal: R R T = I (unit matrix University of Hamburg, Dept. Informatics 5

6 Example for Coordinate Transformaon Camera coordinate system : Displacement by v 0 Rota<on by pan angle β = Rota<on by nick angle α = 45 0 Applica<on of: v = R ( v v 0 with R = R x R y and: R x = y v 0 x y x z R y = z University of Hamburg, Dept. Informatics 6

7 Perspecve Projecon Geometry Projec<ve geometry relates the coordinates of a point in a scene to the coordinates of its projec<on onto an image plane. Perspec<ve projec<on is an adequate model for most cameras. op<cal center y focal length f image point v p = x y p x p z p = f image plane x p y p z p scene point v = x y z z = op<cal axis Projecon equaons: x p = x f z y p = y f z University of Hamburg, Dept. Informatics 7

8 Perspecve and Orthographic Projecon Perspecve Projekon: Projec<on equa<ons x p = x f y z p = y f z z p = f Nonlinear transforma<on Loss of informa<on (f = focal length If all objects are far away (z is large, f/z is approximately constant. à Orthographic projecon: x p = s x y p = s y z p = f (s = scaling factor can be viewed as projec<on with parallel rays + scaling has some linear proper<es, commonly used for formal analysis University of Hamburg, Dept. Informatics 8

9 From Camera Coordinates to Image Coordinates Transform may be necessary because op<cal axis may not penetrate image plane at origin of desired coordinate system transi<on to discrete coordinates may require scaling. x p = ( x p x a p0 ( b x p0 y p = y p y p0 Example: Image boundaries in camera coordinates: x max = c 1 x min = c 2 a, b scaling parameter, y p0 origin of the image coordinate system d 1 y x y max = d 1 y min = d 2 Discrete image coordinates: x = y = x y c 1 c 2 d 2 Transformaon parameters: x p0 = c 1, y p0 = d 1, a = 512, b = 576 c 2 c 1 d 2 d University of Hamburg, Dept. Informatics 9

10 Complete Perspecve Projecon Equaons Combina<on of 3 transforma<on steps: 1. Scene coordinates à camera coordinates 2. Projec<on of camera coordinates into image plane 3. Camera coordinates à image coordinates x p = y p = ( x p0 f cos(βcos(γ(x x z 0 + cos(βsin(γ( y y 0 + sin(β(z z 0 ( sin(αsin(βcos(γ cos(αsin(γ(x x 0 + f + + ( sin(αsin(βsin(γ + cos(αcos(γ( y y z 0 * + + sin(αcos(β(z z 0, (. ( y p0. b ( -. a with: z = ( cos(αsin(βcos(γ + sin(αsin(γ(x x 0 + ( cos(αsin(βsin(γ sin(αcos(γ( y y 0 + cos(αcos(β(z z 0 ( ( ( University of Hamburg, Dept. Informatics 10

11 Homogeneous Coordinates I 4D nota<on for 3D coordinates which allows to express nonlinear 3D transforma<ons as linear 4D transforma<ons. R 4 4 T 4 4 = Normal (3D: v = R 3x3 ( v v 0 Homogeneous coordinates: v = R 4 4 T 4 4v = A4 4 v r 11 r 12 r 13 0 r 21 r 22 r 23 0 r 31 r 32 r x y z Transi<on to homogeneous coordinates: v T affin = [x y z] v 4 Return to normal coordinates: (i Divide components 1 to 3 by 4th component (ii Omit 4th component ( T = [wx wy wz w] w 0 is arbitrary constant University of Hamburg, Dept. Informatics 11

12 Homogeneous Coordinates II Perspec<ve Projec<on in homogeous coordinates: v p,4 = P 4 4 v f 0 with P 4 4 = and gives: Return to normal coordinates gives: Transforma<on from camera- to image coordinates: v p,4 = B 4 4v p,4 a 0 0 x 0 a 0 b 0 y 0 b v 4 = wx wy with B 4 4 = and gives: ( ( ( ( ( wz w v p = v p,4 = x f z y f z f wx p wy p 0 w v p,4 = w x w y w z w z f Compare with earlier slide v p,4 = wa(x p x 0 wa( y p y University of Hamburg, Dept. Informatics 12 0 w ( ( ( ( (

13 Homogeneous Coordinates III Perspec<ve projec<on can be completely described in terms of a linear transforma<on in homogeneous coordinates: v p,4 = B 4 4 P 4 4 R 4 4 T 4 4 v4 B 4 4 P 4 4 R 4 4 T 4 4 may be combined into a single 4x4-Matrix C : v p,4 = C v4 4 4 In the literature the parameters of these equa<ons may vary because of different choices of coordinate systems, different order of transla<on and rota<on, different camera models, etc University of Hamburg, Dept. Informatics 13

14 Inverse Perspecve Equaons Which points in a scene correspond to a point in the image? Each image point defines a projec<on ray as the locus of possible scene points (for simplicity in camera coordinates: v p v λ = λ v p (λ: free parameter v = v 0 + R T λ v v p p Result: 3 equa<ons with the 4 unknowns x, y, z, λ and camera parameters R and Applica<ons of inverse perspec<ve mapping for e.g. distance measurements binocular stereo, mo<on stereo, camera calibra<on v 0 x p y p Ursprung? x y z v λ University of Hamburg, Dept. Informatics 14

15 Binocular Stereo I v y x o 1 l 1 u 1 b l 2 u 2 o 1 l 1, l 2 b o 1, o 2 f 1, f 2 v u 1, u 2 Camera posi<ons (op<cal centers Stereo base (baseline Camera orienta<ons (unit vectors Focal lengths Scene points Projec<on rays of scene point (unit vectors z University of Hamburg, Dept. Informatics 15

16 Binocular Stereo II Determine the distance to v by measuring and. Formally: α and β are overconstrained by the vector equa<on. In prac<ce, measurements are inexact, no exact solu<on exists (rays do not intersect. BeVer approach: Solve for the point of closest approxima<on of both rays: Minimizaon: α 0 α u 1 = b + β u 2 v = α u 1 + l 1 ( 2 v = α u b + β u2 0 u 1 u 2 ( 2 + l minimize: α u1 0 + b + β u2 0 1 ( T α 0 u1 + b ( + β 0 u2 α 0 u1 + b ( + β 0 u2 2 = α 0 u1 + b ( + β 0 u2 ( α 0 u1 + b ( ( + β 0 u2 T α 0 u1 + b ( ( + β 0 u2 = u 1T α 0 u1 b ( ( + β 0 u2 + α ( u b + β 0 ( T 1 0 u2 u1 ( ( ( = 0 = 2 u 1T α 0 u1 b + β 0 u2 = 2 α 0 u 1 T b β 0 u1 T u University of Hamburg, Dept. Informatics 16

17 Binocular Stereo III 2 α 0 u T T ( 1 b β 0 u1 u2 = 0 α 0 = u T T 1 b + β 0 u1 u2 (1 β 0 α 0 u1 + b ( ( + β 0 u2 T α 0 u1 + b ( + β 0 u2 ( = u 2T α 0 u1 b ( ( + β 0 u2 α ( u b + β 0 ( T 1 0 u2 u2 ( ( ( = 2 u 2T α 0 u1 b + β 0 u2 = 2 α 0 u2 T u1 u 2 T b β 0 T 2 α 0 u2 u1 u T ( 2 b β 0 = 0 β 0 = u T T 1 b +α 0 u2 u1 (2 Insert (2 into (1 gives: α 0 = u 1 T b u 1 T u2 ( u T ( 2 b ( 1 u 1 T u2 β 0 = T ( u u2 1 u T 1 b ( u T ( 2 b ( 2 1 u 1 T u University of Hamburg, Dept. Informatics 17

18 Distance in Digital Images Intui<ve concepts of con<nuous images do not always carry over to digital images. Several methods for measuring distance between pixels: Eucledian distance D E ((i, j,(h, k = (i h 2 + ( j k 2 costly computa<on of square root, can be avoided for distance comparisons City-block distance D 4 ((i, j,(h, k = i h + j k Chessboard distance { } D 8 ((i, j,(h, k = max i h, j k number of horizontal and ver<cal steps in a rectangular grid number of steps in a rectangular grid if diagonal steps are allowed (number of moves of a king on a chessboard University of Hamburg, Dept. Informatics 18

19 Connecvity in Digital Images Connec<vity is an important property of subsets of pixels. It is based on adjacency (or neighbourhood: Pixels are 4-neighbours if their distance is D 4 = 1 Pixels are 8-neighbours if their distance is D 8 = 1 all 4-neighbours of center pixel all 8-neighbours of center pixel A path from pixel P to pixel Q is a sequence of pixels beginning at Q and ending at P, where consecu<ve pixels are neighbours. In a set of pixels, two pixels P and Q are connected, if there is a path between P and Q with pixels belonging to the set. A region is a set of pixels where each pair of pixels is connected University of Hamburg, Dept. Informatics 19

20 Closed Curve Paradoxon Solid lines if 8-pixel neighbourhood is used line 1 line 2 A similar paradoxon arises if 4-pixel neighbourhoods are used Line 2 does not intersect line 1 although it crosses from the outside to the inside University of Hamburg, Dept. Informatics 20

21 Various applica<ons: Geometric Transformaons change of view point elimina<on of geometric distor<ons from image capturing registra<on of corresponding images ar<ficial distor<ons, Computer Graphics applica<ons Step 1: Determine mapping T(x, y from old to new coordinate system Step 2: Compute new coordinates (x, y for (x, y Step 3: Interpolate greyvalues at grid posions from greyvalues at transformed posions greyvalue must be interpolated University of Hamburg, Dept. Informatics 21

22 Polynomial Coordinate Transformaons General format of transforma<on: x = m m i a ik x i y k y = i=0 k=0 b ik x i y k Assume polynomial mapping between (x, y and (x, y of degree m Determine corresponding points a Solve linear equa<ons for a ik, b ik (i, k = 1... m m m i i=0 k=0 b Minimize mean square error (MSE for point correspondences Approximation by biquadratic transformation: x = a 00 + a 10 x + a 01 y + a 11 xy + a 20 x 2 + a 02 y 2 y = b 00 + b 10 x + b 01 y + b 11 xy + b 20 x 2 + b 02 y 2 Approximation by affine transformation: x = a 00 + a 10 x + a 01 y y = b 00 + b 10 x + b 01 y at least 6 corresponding pairs needed at least 3 corresponding pairs needed University of Hamburg, Dept. Informatics 22

23 Translaon, Rotaon, Scaling, Skewing Translaon by vector : v = v + t with v = x y Rotaon of image coordinates by angle α: v = R v with R = cosα sinα Scaling by factor a in x-direc<on and factor b in y-direc<on: v = S v with S = a 0 0 b Skewing by angle β: t, v = x y sinα cosα v = W v with W = 1 tan β 0 1 and t = t x t y University of Hamburg, Dept. Informatics 23

24 Example of Geometry Correcon by Scaling Distor<ons of electron-tube cameras may be 1-2 => more than 5 lines for TV images Correc<on procedure may be based on fiducial marks engraved into op<cal system a test image with regularly spaced marks Ideal mark posi<ons: Actual mark posi<ons: ideal image actual image x mn = a + mb, y mn = c + nd with m = 0 M 1 and n = 0 N 1 x mn, y mn Determine a, b, c, d such that MSE (mean square error of deviaons is minimized c y d x x x x x x x x x x x x x x x x b x a University of Hamburg, Dept. Informatics 24

25 Minimize From a = b = c = d = M 1 N 1 Minimizing the MSE E = (x mn x mn 2 + (y mn y mn 2 m=0 M 1 n=0 N 1 = (a+mb x mn 2 + (c + nd y mn 2 m=0 2 MN(M +1 6 MN(M 2 1 n=0 E a = E = b E = c E = 0 d 2 MN(N +1 6 MN(N 2 1 m n m n n m m n we get: (2M 1 3m x mn (2m M +1 x mn (2N 1 3n y mn (2n N +1 y mn Special case M=N=2: a = 1 ( x + x b = 1 ( x 2 10 x 00 + x 11 x 01 c = 1 ( y y 01 d = 1 ( y y + y y University of Hamburg, Dept. Informatics 25

26 Principle of Greyvalue Interpolaon Greyvalue interpolaon = computaon of unknown greyvalues at locaons (u v from known greyvalues at locaons (x y Two ways of viewing interpola<on in the context of geometric transforma<ons: A Greyvalues at grid loca<ons (x y in old image are placed at corresponding loca<ons (x y in new image: g(x y = g(t(x y à interpola<on in new image B Grid loca<ons (u v in new image are transformed into corresponding loca<ons (u v in old image: g(u v = g(t -1 (u v à interpola<on in old image We will take view B: Compute greyvalues between grid from greyvalues at grid loca<ons University of Hamburg, Dept. Informatics 26

27 Nearest Neighbour Greyvalue Interpolaon (x i y j (x i+1 y j Assign (x y to greyvalue of nearest grid locaon (x i y j (x i+1 y j (x i y j+1 (x i+1 y j+1 grid loca<ons (x y loca<on between grid with x i x x i+1 and y j y y j+1 (x y (x i y j+1 (x i+1 y j+1 Each grid loca<on represents the greyvalues in a rectangle centered around this loca<on: Straight lines or edges may appear step-like arer this transforma<on: University of Hamburg, Dept. Informatics 27

28 Bilinear Greyvalue Interpolaon The greyvalue at loca<on (x y between 4 grid points (x i y j (x i+1 y j (x i y j+1 (x i+1 y j+1 is computed by linear interpola<on in both direc<ons: g(x, y = 1 { (x i+1 x i (y j+1 y i (x i+1 x(y j+1 yg(x i, y j + (x x i (y j+1 yg(x i+1, y j + (x i+1 x(y y j g(x i, y j+1 + (x x i (y y j g(x i+1, y j+1 } Simple idea behind long formula: 1. Compute g 12 = linear interpola<on of g 1 and g 2 2. Compute g 34 = linear interpola<on of g 3 and g 4 3. Compute g = linear interpola<on of g 12 and g 34 g 1 g 12 g 2 g The step-like boundary effect is reduced. But bilear interpola<on may blur sharp edges. g 3 g 34 g University of Hamburg, Dept. Informatics 28

29 Bicubic Interpolaon Each greyvalue at a grid point is taken to represent the center value of a local bicubic interpola<on surface with cross sec<on h 3. h 3 = 1 2 x 2 + x x + 5 x 2 x 3 0 for 0 < x 1 for 1< x < 2 else The greyvalue at an arbitrary point (u v (black dot in figure can be computed by four horizontal interpola<ons to obtain greyvalues at points (u j-1... (u j+2 (red dots, followed by one ver<cal interpola<on (between red dots to obtain greyvalue at (u v. Note: For an image with constant greyvalues g 0 the interpolated greyvalues at all points between the grid lines are also g j-1 1 cross sec<on of interpola<on kernel j j+1 j+2 i-1 i i+1 i University of Hamburg, Dept. Informatics 29