Camera: optical system

Transcription

1 Camera: optical system Y dαα α ρ ρ curvature radius Z thin lens small angles: α Y ρ α Y ρ

2 lens refraction inde: n Y incident light beam deviated beam θ '' α θ ' α sin( θ '' α ) sin( θ ' α ) n θ α θ α deviation angle? Δθ θ θ Δθ ( n ) Y ( + ) ρ ρ sin( θ α) sin( θ ' α ) ' n Z

3 a) Y Δθ b) f Δθ Y hin lens rules f ( n )( ρ + ) ρ parallel rays converge onto a focal plane beams through lens center: undeviated independent of y Y Δθ f

4 Hp: Z >> a r f the image of a point P belongs to the line (P,O) P image plane O p p image of P image plane line(o,p) interpretation line of p: line(o,p) locus of the scene points projecting onto image point p

5 Projective D geometry Notes based on di R.Hartley e A.Zisserman Multiple view geometry

6 Projective D Geometry Points, lines & conics ransformations & invariants D projective geometry and the Cross-ratio

7 Homogeneous coordinates Homogeneous representation of lines a + by + c ( a,b,c) ( ka ) + ( kb) y + kc, k ( a,b,c ) ~ k( a,b,c) equivalence class of vectors, any vector is representative Set of all equivalence classes in R 3 (,,) forms P Homogeneous representation of points, y on l a,b,c if and only if a + by + c ( ) ( ) (,y, )( a,b,c) (,y, ) l (, y,) ~ k(, y,), k he point lies on the line l if and only if ll ( ) Homogeneous coordinates,, Inhomogeneous coordinates (, y) 3 but only DOF

8 Points from lines and vice-versa Intersections of lines he intersection of two lines l and l' is l l' Line joining two points he line through two points and ' is Eample l Line joining two points: parametric equation A point on the line through two points and is y + θ ' y

9 Ideal points and the line at infinity Intersections of parallel lines l ( a, b, c) and l' ( a, b, c' ) l l' ( b, a,) Eample ( b, a) ( a,b) tangent vector normal direction Ideal points (,, ) Line at infinity l (,,) P R l Note that in P there is no distinction between ideal points and others

10 Duality l l l l l' l ' Duality principle: o any theorem of -dimensional projective geometry there corresponds a dual theorem, which may be derived by interchanging the role of points and lines in the original theorem

11 Conics Curve described by nd -degree equation in the plane a + by + cy + d + ey + f or homogenized a α, 3 y α + b + c + d3 + e3 + f3 or in matri form a b / d / C with C b / c e / d / e / f 3 5DOF: { a : b : c : d : e : f }

12 Five points define a conic For each point the conic passes through f ey d cy y b a i i i i i i or ( ),,,,, c f y y y i i i i i i ( ) f e d c b a,,,,, c c y y y y y y y y y y y y y y y stacking constraints yields

13 Pole-polar relationship he polar line lc of the point with respect to the conic C intersects the conic in two points. he two lines tangent to C at these points intersect at

14 Polarity: cross ratio Cross ratio of 4 colinear points y + θ (with i,..,4) ratio of ratios θ θ -θ -θ 3 4 θ θ i -θ -θ 3 4 i Harmonic 4-tuple of colinear points: such that CR-

15 angent lines to conics he line l tangent to C at point on C is given by lc l C

16 Dual conics A line tangent to the conic C satisfies l C * l C * In general (C full rank): in fact Dual conics line conics conic envelopes C Line l is the polar line of y : y C l, but since y Cy - - * - - l C C C l C C C -

17 Degenerate conics A conic is degenerate if matri C is not of full rank e.g. two lines (rank ) C lm + ml e.g. repeated line (rank ) m l C ll l Degenerate line conics: points (rank ), double point (rank) Note that for degenerate conics ( * C ) * C

18 Projective transformations Definition: A projectivity is an invertible mapping h from P to itself such that three points,, 3 lie on the same line if and only if h( ),h( ),h( 3 ) do. heorem: A mapping h:p P is a projectivity if and only if there eist a non-singular 33 matri H such that for any point in P represented by a vector it is true that h()h Definition: Projective transformation ' h ' h ' 3 h 3 h h h 3 h h h or ' H 8DOF projectivitycollineationprojective transformationhomography

19 Mapping between planes central projection may be epressed by H (application of theorem)

20 Removing projective distortion select four points in a plane with known coordinates ' ' ' 3 h h 3 + h + h 3 y + h y + h 3 33 y' ' ' 3 h h 3 + h + h 3 y + h y + h 3 33 ( h3 + h3 y + h33 ) h + h y h3 ( h3 + h3 y + h33 ) h + h y h3 ' + y ' + (linear in h ij ) ( constraints/point, 8DOF 4 points needed) Remark: no calibration at all necessary, better ways to compute (see later)

21 More eamples

22 ransformation of lines and conics For a point transformation ' H ransformation for lines l' H - ransformation for conics l C ' H - CH - ransformation for dual conics * C' * HC H

23 A hierarchy of transformations Projective linear group Affine group (last row (,,)) Euclidean group (upper left orthogonal) Oriented Euclidean group (upper left det ) Alternative, characterize transformation in terms of elements or quantities that are preserved or invariant e.g. Euclidean transformations leave distances unchanged

24 Class I: Isometries (isosame, metricmeasure) ' y' ε cosθ ε sinθ sinθ cosθ t t y y ε ± orientation preserving: orientation reversing: ε ε ' H E R t 3DOF ( rotation, translation) R R special cases: pure rotation, pure translation I Invariants: length, angle, area

25 Class II: Similarities (isometry + scale) cos sin sin cos ' ' y t s s t s s y y θ θ θ θ ' t R H s S I R R also know as equi-form (shape preserving) metric structure structure up to similarity (in literature) 4DOF ( scale, rotation, translation) Invariants: ratios of length, angle, ratios of areas, parallel lines

26 Class III: Affine transformations ' ' y t a a t a a y y ' t A H A non-isotropic scaling! (DOF: scale ratio and orientation) 6DOF ( scale, rotation, translation) Invariants: parallel lines, ratios of parallel segment lengths, ratios of areas ( ) ( ) ( ) φ φ θ DR R R A λ λ D ) )( ( VDV UV UDV A where

27 Action of affinities and projectivities on line at infinity + v v v v A A t v A A t Line at infinity becomes finite, allows to observe vanishing points, horizon, Line at infinity stays at infinity, but points move along line

28 Class VI: Projective transformations ' H P A v t v v ( v v ), 8DOF ( scale, rotation, translation, line at infinity) Action: non-homogeneous over the plane Invariants: cross-ratio of four points on a line (ratio of ratios)

29 Projective geometry of D ( ), ' H 3DOF (-) he cross ratio Cross (,,, ) 3 4,, 3 3,, 4 4, i j i det i j j Invariant under projective transformations

30 Overview transformations Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof h h h3 a a sr sr r r h h h a a r r 3 sr sr h3 h 3 h 33 t t y t t y t t y Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). he line at infinity l Ratios of lengths, angles. he circular points I,J lengths, areas.

31 Number of invariants? he number of functional invariants is equal to, or greater than, the number of degrees of freedom of the configuration less the number of degrees of freedom of the transformation e.g. configuration of 4 points in general position has 8 dof (/pt) and so 4 similarity, affinity and zero projective invariants

32 Recovering metric and affine properties from images Parallelism Parallel length ratios Angles Length ratios

33 he line at infinity l l l A t A H A he line at infinity l is a fied line under a projective transformation H if and only if H is an affinity Note: not fied pointwise

34 Affine properties from images projection (affine) rectification A P l l l H H 3 ' [ ], l' 3 3 l l l l in fact, any point on l is mapped to a point at the

35 Affine rectification v v l l l 3 l l 4 v l3 l4 v l l l v v

36 he circular points I i J i I cos sin sin cos I I i se i t s s t s s i y S θ θ θ θ θ H he circular points I, J are fied points under the projective transformation H iff H is a similarity

37 he circular points circular points l + + d3 + e3 + f3 + 3 I (, i,) Intersection points between any circle and l J, i, Algebraically, encodes orthogonal directions I (,, ) + i(,, ) ( )

38 Circular points invariance {I,J} l any circumference Similarity: circ circ Similarity: circ l circ l Similarity: {I,J} {I,J} circular points: invariant under similarity

39 * C Conic dual to the circular points * C IJ + JI : line conic set of lines through any of the circular points C * C * H C * S H S C he dual conic * is fied conic under the projective transformation H iff H is a similarity * C Note: has 4DOF l is the null vector

40 Angles ( )( ) cos m m l l m l m l θ ( ) 3,, l l l l ( ) 3,, m m m m Euclidean: Projective: ( )( ) m m l l m l cos * * * C C C θ m l * C (orthogonal)

41 Metric properties from images ( ) ( ) ( ) ( ) ( ) ( ) v v v v ' * * * * K K KK H H C H H H H H C H H H H H H C H H H C A P A P A P S S A P S A P S A P U U C ' * Rectifying transformation from SVD H U

42 Normally: SVD (Singular Value Decomposition) U V C c b a ' * But is symmetric ' * C ' ' * * C VDU UDV C and SVD is unique V U with U and V orthogonal U U C ' * Why? Observation : HU orthogonal (33): not a P isometry

43 Once the image has been affinely rectified A H A C H C * * ' ' * t t KK K K C Metric from affine

44 Metric from affine ( ) 3 3 m m m l l l KK ( )( ),,,, + + k k k k k m l m l m l m l

45 Metric from projective ( ) ( ) ( ) ( ) c,,,,, m l m l m l m l m l m l m l m l m l ( ) v v v v 3 3 m m m l l l K K KK

46 Fied points and lines H e λ e (eigenvectors H fied points) (λ λ pointwise fied line) H l λ l (eigenvectors H - fied lines)

47 Projective 3D geometry

48 Singular Value Decomposition n n n m m m n m V Σ U A I U U n σ σ σ Λ I V V m n X V X V Σ X V UΣ n n U n V V U V U A σ σ σ Λ Σ Λ Μ Μ Μ Λ Μ Ο Μ Μ Λ Λ σ n σ σ

49 Singular Value Decomposition Homogeneous least-squares min AX subject to X solution X Vn Span and null-space S L [ U U ]; N [ U ] L 3U4 V V ; R V3V 4 Closest rank r approimation A ~ UΣ ~ V [ ] [ ] S R N Pseudo inverse + A VΣ U A UΣ V ~ Σ diag Σ σ ( σ, σ, Λ, σ,,, ) r Λ + ( Σ + diag σ, σ, Λ, σ,,, ) σ r Λ

50 Projective 3D Geometry Points, lines, planes and quadrics ransformations П, ω and Ω

51 3D points 3D point ( X, Y, Z ) in R 3 X ( X, X, X X ) 3, 4 in P 3 X X X X 3,,, Z X 4 X 4 X 4 ( X, Y,,) ( X 4 ) projective transformation X' H X (44-5 dof)

52 3D plane Planes π X + πy + π3z + π4 π X + πx + π3x 3 + π4x 4 ransformation X' H X - π' H π π X Euclidean n.x ~ representation + d ( π ), π, π3 d n ( ) π 4 X ~ X, Y, Z X 4 d / n Dual: points planes, lines lines

53 Solve X X X Planes from points 3 π from 3 π [ X X X X ] det 3 X π, X π and X π X (solve π as right nullspace of X ) X3 Or implicitly from coplanarity condition X det X X X 3 4 ( X) ( X ) ( X ) 3 ( X) ( X ) ( X ) 3 ( X) ( X ) ( X 3) 3 3 ( X ) ( X ) ( X ) X D 34 X D 34 + X 3 D 4 X π ( D ) 4D3 34, D34, D4, D

54 Points from planes X π π π 3 X M [ ] 3 X X X M π M M I p ( ) d c b a,,, π a d a c a b p,, X π X π, X π X 3 and from Solve (solve as right nullspace of ) X 3 π π π Representing a plane by its span

55 Lines W A B λa+ μb (4dof) W * P Q λp+ μq two points A and B two planes P and Q * * W W WW Eample: X-ais W W *

56 Points, lines and planes W M M π X X W W * M M X π * W π

57 Quadrics and dual quadrics X QX. 9 d.o.f. (Q : 44 symmetric matri). in general 9 points define quadric 3. det Q degenerate quadric 4. pole polar 5. (plane quadric)conic 6. transformation π * Q π π QX Q' H Q ο ο ο ο ο ο C M QM π : X M - QH - * - Q Q * * HQ H. relation to quadric (non-degenerate). transformation Q'

58 Quadric classification Rank Sign. Diagonal Equation Realization 4 4 (,,,) X + Y + Z + No real points (,,,-) X + Y + Z Sphere (,,-,-) X + Y Z + Hyperboloid (S) 3 3 (,,,) X + Y + Z Single point (,,-,) X + Y Z Cone (,,,) X + Y Single line (,-,,) X Y wo planes (,,,) X Single plane

59 Quadric classification Projectively equivalent to sphere: sphere Ruled quadrics: ellipsoid hyperboloid of two sheets paraboloid hyperboloids of one sheet Degenerate ruled quadrics: cone two planes

60 Hierarchy of transformations Projective 5dof A v t v Intersection and tangency Affine dof A t Parallellism of planes, Volume ratios, centroids, he plane at infinity π Similarity 7dof s R t he absolute conic Ω Euclidean 6dof R t Volume

61 Screw decomposition Any particular translation and rotation is equivalent to a rotation about a screw ais and a translation along the screw ais. screw ais // rotation ais t + t t //

62 he plane at infinity A π H A π π A t he plane at infinity π is a fied plane under a projective transformation H iff H is an affinity π (,,,). canical position. contains directions D ( X ), X, X 3, 3. two planes are parallel line of intersection in π 4. line // line (or plane) point of intersection in π

63 he absolute conic he absolute conic Ω is a (point) conic on π. In a metric frame: X or conic for directions: (with no real points) + X X 4 + X he absolute conic Ω is a fied conic under the projective transformation H iff H is a similarity 3 ( X, X, X ) I( X, X X ) 3, 3. Ω is only fied as a set. Circle intersect Ω in two points 3. Spheres intersect π in Ω

64 Absolute conic invariance Ω π any sphere Similarity: sphere sphere Similarity: sphere π sphere π Similarity: Ω Ω Ω : invariant under similarity

65 he absolute conic Euclidean: Projective: cosθ cosθ ( d ) d ( )( d ) d d d ( d ) Ω d ( d d )( Ω d Ω d ) d Ω d (orthogonalityconjugacy) normal plane

66 he absolute dual quadric * I Ω he absolute conic Ω * is a fied conic under the projective transformation H iff H is a similarity. 8 dof. plane at infinity π is the nullvector of Ω 3. Angles: * cosθ ( * )( * π Ω π π Ω π ) π Ω π

67 Y P X p c y f O Z y f f X Z Y Z perspective projection -nonlinear -not shape-preserving -not length-ratio preserving

68 Homogeneous coordinates In D: add a third coordinate, w Point [,y] epanded to [u,v,w] Any two sets of points [u,v,w ] and [u,v,w ] represent the same point if one is multiple of the other [u,v,w] [,y] with u/w, and yv/w [u,v,] is the point at the infinite along direction (u,v)

69 ransformations translation by vector [d,d y ] scaling (by different factors in and y) rotation by angle θ

70 Homogeneous coordinates In 3D: add a fourth coordinate, t Point [X,Y,Z] epanded to [,y,z,t] Any two sets of points [,y,z,t ] and [,y,z,t ] represent the same point if one is multiple of the other [,y,z,t] [X,Y,Z] with X/t, Yy/t, and Zz/t [,y,z,] is the point at the infinite along direction (,y,z)

71 ransformations translation scaling rotation Obs: rotation matri is an orthogonal matri i.e.: R - R

72 Pinhole camera model Z fy Z fx Z Y X ) /, / ( ),, ( α Z Y X f f Z fy fx Z Y X α

73 Scene->Image mapping: perspective transformation u v w f with f X Y Z y u w v w With ad hoc reference frames, for both image and scene

74 Let us recall them O image reference - centered on principal point - - and y-aes parallel to the sensor rows and columns - Euclidean reference c y f Y X scene reference - centered on lens center - Z-ais orthogonal to image plane - X- and Y-aes opposite to image - and y-aes Z

75 principal point c Actual references are generic y f O principal ais Y X image reference - centered on upper left corner - nonsquare piels (aspect ratio) noneuclidean reference Z scene reference - not attached to the camera

76 Principal point offset o o v Z fy u Z fx Z Y X ) /, / ( ),, ( + + α principal point o v o u ), ( + + Z Y X v f u f Z Zv fy Zu fx Z Y X o o o o α

77 CCD camera o y o v f u f K o o v f u f a K

78 Scene-image relationship wrt actual reference frames image u A u v w ( s a) a u v o o u v w normally, s X scene Y Z R t X

79 X t R X P u f f v a u o o t K R K t R I P o o v af u f K upper triangular: intrinsic camera parameters scene-camera tranformation etrinsic camera parameters orthogonal (3D rotation) matri P: - degrees of freedom ( if s)

80 z y o I M X o I M u ) ( o M o I M u i.e., defining [, y, z] o I M o M M m M t K R K P with R K M and m M o

81 Interpretation of o: u is image of if ( o) λm u i.e., if o + λm u he locus of the points whose image is u is a straight line through o having direction d M u o M m is independent of u o is the camera viewpoint (perspective projection center) line(o, d) Interpretation line of image point u

82 M K and R Intrinsic and etrinsic parameters from P M K R M R K R K RQ-decomposition of a matri: as the product between an orthogonal matri and an upper triangular matri M and m t o M m ( KR) Kt R K Kt R t t Ro

83 Camera center null-space camera projection matri P O X λa + ( λ)o PX λpa + ( λ)po For all A all points on AO project on image of A, therefore O is camera center Image of camera center is (,,), i.e. undefined o M m Finite cameras: O d Infinite cameras: o,md

84 Action of projective camera on point Forward projection PX [ M m] D Md PD Back-projection P O + X P P + P ( PP ) ( λ) P + λo X + - d M M X( λ) μ D - (pseudo-inverse) - M + O - m M - + PP I ( μ - m)

85 Camera matri decomposition Finding the camera center P O (use SVD to find null-space) X det( [ p,p3,m] ) Y det( [ p,p3,m] ) Z det( [ p,p,m] ) ([ p,p,m]) det Finding the camera orientation and internal parameters M KR (use RQ decomposition ~QR) (if only QR, invert) ( Q R ) - R - - Q

86 Radial distortion

87 Data normalization 3 X ~ i UX i u~ i u i (i) translate origin to gravity center (ii) (an)isotropic scaling

88 Eterior orientation Calibrated camera, position and orientation unkown Pose estimation 6 dof 3 points minimal (4 solutions in general)

89 Properties of perspective transformations ) vanishing points V image of the point at the along direction d u V M m d M d d M u V the interpretation line of V is parallel to d

90 d P O V he images of parallel lines are concurrent lines

91 Properties of perspective transformations ctd. ) cross ratio invariance Given four colinear points ( p, p, p p ) let (,, ) 3, 4 3, be their abscissae 4 CR ( p, p, p p ) 3,

92 Cross ratio invariance under perspective transformation a point on the line yz X [, y, z, t] [,,, t ] its image its coordinate u det u, u CR( u, u, u3, u4 ) det u, u det u [ u, v, w] 3 3 det u det u P X p u [ u, w] P4 p t 3, u, u detp detp det det,, 3 4 CR(,, 3, 4 ) det,, det det,, 3 belongs to a line detp detp 4 4 det det 3,, 4 4

93 Object localization : three colinear points geometric model of an object a perspective image of the object position and orientation of the object? calibrated camera: known P M known interpretation lines m C A B B A O C B A π C

94 a) orientation C A B C V O A B π Cross ratio invariance: a c solve CR( A', B', C', V ) CR( A, B, C, ) for V (image of ) b c V: vanishing point of the direction of (A,B,C) interpretation line of V parallel to (A,B,C) direction M uv

95 b) position (e.g., distance(o,a)) M u C γ C A B C V O α M u A A B M u V π interpretation lines OA AC sinγ sinα angles α and γ

96 Object localization : four coplanar points D C O E B A (i) orientation of (A,E,C) (ii) orientation of (B,E,D) (iii) distance (O,A)

97 Off-side Find vanishing point of the field-bottom line direction b b a a images of symmetric segments

98 a c b d a and b: c(a+b)/: d : abscissae of the endpoints of a segment abscissa of segment midpoint, point at the infinite along the segment direction CR a c a d b d a c b c ( a, b, c, d ) b c Harmonic 4-tuple (a,b,c,d) (a,b ) and (a, b ) are image of symmetric segments same image of the midpoint c, same vanishing point d

99 solve a' c' ( a', b', c', d' ) b' c' { CR for CR a' ', b'', c', d' a'' c' a' d' b' d' ( ) b'' c' a'' d' b'' d' c, d system of two linear equations in (c d ) and (c +d ) two degree equation, whose solutions are c and d among the two solutions, the one for d is the value eternal to the range [a,b ]

100 What can be told from a single image?

101 Action of projective camera on planes [ ] [ ] p p p p p p p PX Y X Y X he most general transformation that can occur between a scene plane and an image plane under perspective imaging is a plane projective transformation

102 Action of projective camera on lines forward projection ( μ) P(A + μb) PA + μpb a μb X + back-projection l PX l PX with Π P l Π X Interpretation plane of line l

103 [ ] [ ] [ ] P Y X Y X p p p p p p p p p p PX Image of a conic C C - P P therefore - C C' P P

104 Action of projective camera on conics back-projection of a conic C to cone Q co C Q co

105 back-projection of a conic C to cone Qco C PX C X P CPX with X Q co X Interpretation cone of a conic C Q co P CP eample: Q co K C [ K ] K CK

106 Images of smooth surfaces he contour generator Γ is the set of points X on S at which rays are tangent to the surface. he corresponding apparent contour γ is the set of points which are the image of X, i.e. γ is the image of Γ he contour generator Γ depends only on position of projection center, γ depends also on rest of P

107 Action of projective camera on quadrics apparent contour of a quadric Q dual quadric Q * Q is a plane quadric: the set of planes tangent to Q Π Q * Π Let us consider only those planes that are backprojection of image lines Π P l Π Q * Π l * PQ P l with C * * PQ P its dual is C C *

108 he plane containing the apparent contour Γ of a quadric Q from a camera center O follows from pole-polar relationship ΠQO he cone with verte V and tangent to the quadric Q is Q CO (V QV)Q - (QV)(QV) Q CO V back-projection to cone

109 What does calibration give? K[I ] d d K d d cos θ ( )( ) ( )( ) d d d d (K K ) (K K ) (K - K - ) An image line l defines a plane through the camera center with normal nk l measured in the camera s Euclidean frame. In fact the backprojection of l is P l nk l

110 he image of the absolute conic Ω d PX KR[I O] KRd mapping between π to an image is given by the planar homogaphy Hd, with HKR absolute conic (IAC), represented by I 3 within π (w) its image (IAC) ( ) - - KK K K ω ( C α H CH ) (i) IAC depends only on intrinsics (ii) angle between two rays cos θ (iii) DIACω * KK (iv) ω K (Cholesky factorization) (v) image of circular points belong to ω (image of absolute conic) ( ω )( ω ) ω

111 A simple calibration device (i) compute H i for each square (corners (,),(,),(,),(,)) (ii) compute the imaged circular points H i [,±i,] (iii) fit a conic ω to 6 imaged circular points (iv) compute K from ωk - K - through Cholesky factorization ( Zhang s calibration method)

112

113 Orthogonality relation cos θ ( )( ) v ωv v ωv v ωv v ωv * l ω l

114 Calibration from vanishing points and lines

115 Calibration from vanishing points and lines

116

117 wo-view geometry Epipolar geometry F-matri comp. 3D reconstruction Structure comp.

118 hree questions: (i) Correspondence geometry: Given an image point in the first view, how does this constrain the position of the corresponding point in the second image? (ii) Camera geometry (motion): Given a set of corresponding image points { i i }, i,,n, what are the cameras P and P for the two views? (iii) Scene geometry (structure): Given corresponding image points i i and cameras P, P, what is the position of (their pre-image) X in space?

119 he epipolar geometry C,C,, and X are coplanar

120 he epipolar geometry What if only C,C, are known?

121 he epipolar geometry All points on π project on l and l

122 he epipolar geometry Family of planes π and lines l and l Intersection in e and e

123 he epipolar geometry epipoles e,e intersection of baseline with image plane projection of projection center in other image vanishing point of camera motion direction an epipolar plane plane containing baseline (-D family) an epipolar line intersection of epipolar plane with image (always come in corresponding pairs)

124 Eample: converging cameras

125 Eample: motion parallel with image plane

126 Eample: forward motion e e

127 he fundamental matri F algebraic representation of epipolar geometry α l' we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matri F

128 he fundamental matri F geometric derivation ' H π [ e' ] Hπ F l' e' ' mapping from -D to -D family (rank )

129 he fundamental matri F algebraic derivation ( λ) P λc X + + ( + P P I) l P'C P' P + P + X( λ) F [ ] + e' P' P (note: doesn t work for CC F)

130 he fundamental matri F correspondence condition he fundamental matri satisfies the condition that for any pair of corresponding points in the two images ' F ( ' l' )

131 he fundamental matri F F is the unique 33 rank matri that satisfies F for all (i) ranspose: if F is fundamental matri for (P,P ), then F is fundamental matri for (P,P) (ii) Epipolar lines: l F & lf (iii) Epipoles: on all epipolar lines, thus e F, e F, similarly Fe (iv) F has 7 d.o.f., i.e. 33-(homogeneous)-(rank) (v) F is a correlation, projective mapping from a point to a line l F (not a proper correlation, i.e. not invertible)

132 he epipolar line geometry l,l epipolar lines, k line not through e l F[k] l and symmetrically lf [k ] l l l k k e e' l Fk (pick ke, since e e ) F[ e] l l F [ e' ] l' l'

133 Fundamental matri for pure translation

134 Fundamental matri for pure translation

135 Fundamental matri for pure translation [ e' ] H [ ] F e' ( H K RK) eample: e' (,,) F - ' F y y' PX ' P' X K[I ]X K - K[I t] Z ( X,Y,Z) - K /Z ' + Kt/Z motion starts at and moves towards e, faster depending on Z pure translation: F only d.o.f., [e] auto-epipolar

136 General motion [ e' ] H [ e' ] ˆ ' ' ' K'RK - + K' t/z

137 Geometric representation of F F ( F F )/ ( F F F )/ S + A ( F + ) F S F A F ( F ) A F F s : Steiner conic, 5 d.o.f. F a [ a ] : pole of line ee w.r.t. F s, d.o.f. S

138 Geometric representation of F

139 Pure planar motion Steiner conic F s is degenerate (two lines)

140 Projective transformation and invariance Derivation based purely on projective concepts ˆ H, ˆ' H' ' Fˆ PX ( )( - PH H X) Pˆ Xˆ ( )( - P' H H X) Pˆ' Xˆ H' - FH - F invariant to transformations of projective 3-space ' P' X ( P, P' ) α F F α ( P,P' ) canonical form P' [I ] [M m] unique not unique P [ m] M F

141 Projective ambiguity of cameras given F previous slide: at least projective ambiguity this slide: not more! ~ ~ Show that if F is same for (P,P ) and (P,P ), there eists a projective transformation H so that ~ ~ PHP and P HP lemma: H P' H ~ P [I ] P' [A a] P [I ] P ~ ' F [ a] A [ ~ a] A ~ ~ A ~ a ka k A + av ( ) rank [A ~ ~ a] af a[] a A ~ af ~ a ka a A ~ a A ~ a ka ~ - A ka ~ - A av [ ] [ ] [ ] ( ) ( ) k I k v k [A a] k I k v k [ k ( A - av ) ka] P ~ ' (-57, ok)

142 Canonical cameras given F F matri corresponds to P,P iff P FP is skew-symmetric ( X P' FPX, X) F matri, S skew-symmetric matri P [I ] P' [SF e'] [SF e'] (fund.matrif) F[I ] F S F e' F F S F Possible choice: P [I ] P' [[e'] F e'] Canonical representation: P [I ] P' [[e'] F + e' v λe']

143 he essential matri ~fundamental matri for calibrated cameras (remove K) E ˆ ' [ t] R R[R t] Eˆ - - ( ˆ K ; ˆ' K ' ) E K' FK 5 d.o.f. (3 for R; for t up to scale) E is essential matri if and only if two singularvalues are equal (and third) E Udiag(,,)V

144 Four possible reconstructions from E (only one solution where points is in front of both cameras)

145 wo-view geometry Epipolar geometry F-matri comp. 3D reconstruction Structure comp.

146 hree questions: (i) Correspondence geometry: Given an image point in the first view, how does this constrain the position of the corresponding point in the second image? (ii) Camera geometry (motion): Given a set of corresponding image points { i i }, i,,n, what are the cameras P and P for the two views? (iii) Scene geometry (structure): Given corresponding image points i i and cameras P, P, what is the position of (their pre-image) X in space?

147 he epipolar geometry C,C,, and X are coplanar

148 he epipolar geometry What if only C,C, are known?

149 he epipolar geometry All points on π project on l and l

150 he epipolar geometry Family of planes π and lines l and l Intersection in e and e

151 he epipolar geometry epipoles e,e intersection of baseline with image plane projection of projection center in other image vanishing point of camera motion direction an epipolar plane plane containing baseline (-D family) an epipolar line intersection of epipolar plane with image (always come in corresponding pairs)

152 Eample: converging cameras

153 Eample: motion parallel with image plane

154 Eample: forward motion e e

155 he fundamental matri F algebraic representation of epipolar geometry α l' we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matri F

156 he fundamental matri F geometric derivation ' H π [ e' ] Hπ F l' e' ' mapping from -D to -D family (rank )

157 l' he fundamental matri F algebraic derivation M + ( ) λc X λ - M [ ] [ ] [ ] M' m' C M' m' M' m' C M' M ' l' ' l' ' F P + X( λ) F [ ] e' M' M (note: doesn t work for CC F)

158 he fundamental matri F correspondence condition he fundamental matri satisfies the condition that for any pair of corresponding points in the two images ' F ( ' l' )

159 he fundamental matri F F is the unique 33 rank matri that satisfies F for all (i) ranspose: if F is fundamental matri for (P,P ), then F is fundamental matri for (P,P) (ii) Epipolar lines: l F & lf (iii) Epipoles: on all epipolar lines, thus e F, e F, similarly Fe (iv) F has 7 d.o.f., i.e. 33-(homogeneous)-(rank) (v) F is a correlation, projective mapping from a point to a line l F (not a proper correlation, i.e. not invertible)

160 he epipolar line geometry l,l epipolar lines, k line not through e l F[k] l and symmetrically lf [k ] l l l k k e e' l Fk (pick ke, since e e ) F[ e] l l F [ e' ] l' l'

161 Fundamental matri for pure translation

162 Fundamental matri for pure translation

163 Fundamental matri for pure translation [ e' ] H [ ] F e' ( H K RK) eample: e' (,,) F - ' F y y' PX ' P' X K[I ]X - K[I t] K Z ( X,Y,Z) - K /Z ' + Kt/Z motion starts at and moves towards e, faster depending on Z pure translation: F only d.o.f., [e] auto-epipolar

164 General motion [ e' ] H [ e' ] ˆ ' ' ' K'RK - + K' t/z

165 Geometric representation of F F ( F F )/ ( F F F )/ S + A ( F + ) F S F A F ( F ) A F F s : Steiner conic, 5 d.o.f. F a [ a ] : pole of line ee w.r.t. F s, d.o.f. S

166 Geometric representation of F

167 Pure planar motion Steiner conic F s is degenerate (two lines)

168 Projective transformation and invariance Derivation based purely on projective concepts ˆ H, ˆ' H' ' Fˆ PX ( )( - PH H X) Pˆ Xˆ ( )( - P' H H X) Pˆ' Xˆ H' - FH - F invariant to transformations of projective 3-space ' P' X ( P, P' ) α F F α ( P,P' ) canonical form P' [I ] [M m] unique not unique P [ m] M F

169 Projective ambiguity of cameras given F previous slide: at least projective ambiguity this slide: not more! ~ ~ Show that if F is same for (P,P ) and (P,P ), there eists a projective transformation H so that ~ ~ PHP and P HP lemma: H P' H ~ P [I ] P' [A a] P ~ [I ] P' F [ a] A [ ~ a] A ~ ~ A ~ a ka k A + av ( ) rank [A ~ ~ a] af a[] a A ~ af ~ a ka a A ~ a A ~ a ka ~ - A ka ~ - A av [ ] [ ] [ ] ( ) ( ) k I k v k [A a] k I k v k [ k ( A - av ) ka] P ~ ' (-57, ok)

170 Canonical cameras given F F matri corresponds to P,P iff P FP is skew-symmetric ( X P' FPX, X) F matri, S skew-symmetric matri P [I ] P' [SF e'] [SF e'] (fund.matrif) F[I ] F S F e' F F S F Possible choice: P [I ] P' [[e'] F e'] Canonical representation: P [I ] P' [[e'] F + e' v λe']

171 he essential matri ~fundamental matri for calibrated cameras (remove K) E ˆ ' [ t] R R[R t] Eˆ - - ( ˆ K ; ˆ' K ' ) E K' FK 5 d.o.f. (3 for R; for t up to scale) E is essential matri if and only if two singularvalues are equal (and third) SVD E Udiag(,,)V

172 Motion from E Given W Four solutions R' UW V and t' U 3 R' ' UWV t' ' U 3

173 Four possible reconstructions from E (only one solution where points is in front of both cameras)