EECS 442 Computer visio Multiple view geometry Affie structure from Motio - Affie structure from motio problem - Algebraic methods - Factorizatio methods Readig: [HZ] Chapters: 6,4,8 [FP] Chapter: 2 Some slides of this lectures are courtesy of prof. J. Poce, prof FF Li, prof S. Lazebik & prof. M. Hebert
Structure from motio problem X j j M m M mj 2j M 2 Give m images of fied 3D poits ij M i X j, i,, m, j,,
Structure from motio problem X j j M m M mj 2j M 2 From the m correspodeces ij, estimate: m projectio matrices M i 3D poits X j motio structure
Applicatios Courtesy of Oford Visual Geometry Group
Affie structure from motio (simpler problem) Image World Image From the m correspodeces ij, estimate: m projectio matrices M i (affie cameras) 3D poits X j
Fiite cameras p q r R Q P O [ ]X T K R T R K M 3 3 M Caoical perspective projectio matri Affie homography (i 3D) Affie Homography (i 2D) y s K o y o α α
Trasformatio i 2D Affiities: y H y t A y' ' a -Preserve: - Parallel lies - Ratio of areas - Ratio of legths o colliear lies - others - 6 DOF
Weak perspective projectio Whe the relative scee depth is small compared to its distace from the camera P ~ ' m y' my where m f z ' is the magificatio.
Weak perspective projectio Whe the relative scee depth is small compared to its distace from the camera ' m y' my Scalig fuctio of the distace (magificatio)
Orthographic (affie) projectio Whe the camera is at a (roughly costat) distace from the scee ' y' y Distace from ceter of projectio to image plae is ifiite
Orthographic (affie) projectio Whe the camera is at a (roughly costat) distace from the scee
Affie cameras [ ]X T K R T R K M y s K o y o α α Projective case Affie case
Trasformatio i 2D Projective: y H y b v t A y' ' p Affiities: y H y t A y' ' a
Affie cameras [ ]X T K R y α s α o y o K T R K M T R K M y s K o y o α α Projective case Affie case Parallel projectio matri (poits at ifiity are mapped as poits at ifiity) Magificatio (scalig term)
Weak perspective projectio Qigmig Festival by the Riverside Zhag Zedua ~9 AD
Affie cameras [ ]X T K R y K α α T R K M b A 4affie] [4 3affie] 3 [ 2 23 22 2 3 2 b a a a b a a a M + + 2 23 22 2 3 2 X b AX Euc M b b Z Y X a a a a a a y [Homogeeous] [o-homogeeous image coordiates] [ ] b A M M Euc ; P M Euc
Affie cameras p P p M camera matri To recap: from ow o we defie M as the camera matri for the affie case p u v AP + b M P ; M [ A b]
The Affie Structure-from-Motio Problem Give m images of fied poits P j (X i ) we ca write N of cameras N of poits Problem: estimate the m 2 4 matrices M i ad the positios P j from the m correspodeces p ij. How may equatios ad how may ukow? 2m equatios i 8m+3 ukows Two approaches: - Algebraic approach (affie epipolar geometry; estimate F; cameras; poits) - Factorizatio method
Algebraic aalysis (2-view case) - Derive the fudametal matri F A for the affie case - Compute F A - Use F A to estimate projectio matrices - Use projectio matrices to estimate 3D poits
. Derivig the fudametal matri F A p P v p u Homogeeous system Dim? 44
Derivig the fudametal matri F A where The Affie Fudametal Matri!
Affie Epipolar Geometry Note: the epipolar lies are parallel.
Estimatig F A From at least 4 correspodeces, we obtai a liear system o the ukow alpha, beta, etc Measuremets: u, u, v, v v u v u v u v u f M M M M M Computed by least square ad by eforcig f SVD
Estimatig projectio matrices from epipolar costraits p P p
Affie ambiguity Affie PX ( PQ - )( Q X) A A
Estimatig projectio matrices from epipolar costraits p P p
Estimatig projectio matrices from epipolar costraits Choose Q such that A ~ ~ b [ ] T A ~ a ~ b b [ d] T c Caoical affie cameras Fuctio of the parameters of F
Estimatig projectio matrices from epipolar costraits Choose Q such that By re-eforcig the epipolar costrait, we ca compute a, b, c, d directly from the measuremets
Remider: epipolar costrait p P v p u Homogeeous system
Estimatig projectio matrices from epipolar costraits Choose Q such that ~ A ~ b Re-eforce the Epipolar costrait
Estimatig projectio matrices from epipolar costraits Choose Q such that A b
Estimatig projectio matrices from epipolar costraits Liear relatioship betwee measuremets ad ukow Ukow: a, b, c, d Measuremets: u, u, v, v From at least 4 correspodeces, we ca solve this liear system ad compute a, b, c, d (via least square) The cameras ca be computed How about the structure?
4. Estimatig the structure from epipolar costraits A b Ca be solved by least square agai
A factorizatio method Tomasi & Kaade algorithm C. Tomasi ad T. Kaade. Shape ad motio from image streams uder orthography: A factorizatio method. IJCV, 9(2):37-54, November 992. Ceterig the data Factorizatio
Ceterig: subtract the cetroid of the image poits ( ) j i k k j i k i k i i j i k ik ij ij ˆ ˆ A X X X A b A X b A X + + A factorizatio method - Ceterig the data X k ik i ^
Ceterig: subtract the cetroid of the image poits ( ) j i k k j i k i k i i j i k ik ij ij ˆ ˆ A X X X A b A X b A X + + A factorizatio method - Ceterig the data
Ceterig: subtract the cetroid of the image poits ( ) + + k k j i k i k i i j i k ik ij ij ˆ X X A b A X b A X j i ij X A ˆ A factorizatio method - Ceterig the data Assume that the origi of the world coordiate system is at the cetroid of the 3D poits After ceterig, each ormalized poit ij is related to the 3D poit X i by
A factorizatio method - Ceterig the data X ˆ A ij i X j
Let s create a 2m data (measuremet) matri: m m m D ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ 2 2 22 2 2 L O L L cameras (2m ) poits ( ) A factorizatio method - factorizatio
Let s create a 2m data (measuremet) matri: [ ] m m m m X X X A A A D L M L O L L 2 2 2 2 22 2 2 ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ cameras (2m 3) poits (3 ) The measuremet matri D M S has rak 3 (it s a product of a 2m3 matri ad 3 matri) A factorizatio method - factorizatio (2m ) M S
Factorizig the measuremet matri Source: M. Hebert
Factorizig the measuremet matri Sigular value decompositio of D: Source: M. Hebert
Factorizig the measuremet matri Sigular value decompositio of D: Sice rak (D)3, there are oly 3 o-zero sigular values Source: M. Hebert
Factorizig the measuremet matri Obtaiig a factorizatio from SVD: Motio (cameras) structure What is the issue here? D has rak>3 because of - measuremet oise - affie approimatio
Factorizig the measuremet matri Obtaiig a factorizatio from SVD: structure D D
Affie ambiguity The decompositio is ot uique. We get the same D by usig ay 3 3 matri C ad applyig the trasformatios M MC, S C - S We ca eforce some Euclidea costraits to resolve this ambiguity (more o et lecture!)
Algorithm summary. Give: m images ad features ij 2. For each image i, ceter the feature coordiates 3. Costruct a 2m measuremet matri D: Colum j cotais the projectio of poit j i all views Row i cotais oe coordiate of the projectios of all the poits i image i 4. Factorize D: Compute SVD: D U W V T Create U 3 by takig the first 3 colums of U Create V 3 by takig the first 3 colums of V Create W 3 by takig the upper left 3 3 block of W 5. Create the motio ad shape matrices: M M U 3 ad S W 3 V 3 T (or U 3 W 3½ ad S W 3½ V 3T ) 6. Elimiate affie ambiguity
Recostructio results C. Tomasi ad T. Kaade. Shape ad motio from image streams uder orthography: A factorizatio method. IJCV, 9(2):37-54, November 992.
Net lecture Multiple view geometry Perspective structure from Motio