CS376 Compuer Vision Lecure 6: Opical Flow Qiing Huang Feb. 11 h 2019 Slides Credi: Krisen Grauman and Sebasian Thrun, Michael Black, Marc Pollefeys
Opical Flow mage racking 3D compuaion mage sequence (single camera) Tracked sequence 3D srucure + 3D rajecory 2
Wha is Opical Flow? p 2 p 3 v 2 v 3 p 1 Opical Flow v 1 p 4 v 4 ( i ),{ p } ( +1) Velociy vecors { v i } Opical flow is he 2D projecion of he physical movemen of poins relaive o he observer A common assumpion is brighness consancy: ( p, ) = ( p + v, + 1) i i i 3
When does Brighness Assumpion Break down? TV is based on illusory moion he se is saionary ye hings seem o move A uniform roaing sphere nohing seems o move, ye i is roaing Changing direcions or inensiies of lighing can make hings seem o move for eample, if he specular highligh on a roaing sphere moves Muscle movemen can make some spos on a cheeah move opposie direcion of moion
Opical Flow Assumpions: Brighness Consancy 5 * Slide from Michael Black, CS143 2003
Opical Flow Assumpions: Neighboring piels end o have similar moions When does his break down? 6 * Slide from Michael Black, CS143 2003
Opical Flow Assumpions: The image moion of a surface pah changes gradually over ime 7 * Slide from Michael Black, CS143 2003
1D Opical Flow
9 Opical Flow: 1D Case Brighness Consancy Assumpion: ) ), ( ( ) ), ( ( ) ( d d f + + = 0 ) ( = + v v = { 0 ) ( = f Because no change in brighness wih ime
2D Opical Flow
11 From 1D o 2D racking 0 ) ( = + 1D: 0 ) ( = + + y y 2D: 0 ) ( = + + v y u One equaion bu wo velociy (u,v) unknowns
How does his show up visually? Known as he Aperure Problem 12
Aperure Problem in Real Life 13
From 1D o 2D racking y (, y, + 1) (, ) (, + 1) v v 1 v 2 v 3 v 4 (, y, ) The Mah is very similar: v Aperure problem v G = b = Window size here ~ 1111 G 1 window around p window around p b 2 y y y 2 y 14
More Deail: Solving he aperure problem How o ge more equaions for a piel? -- impose addiional consrains mos common is o assume ha he flow field is smooh locally one mehod: preend he piel s neighbors have he same (u,v) Suppose a 55 window 15 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Lukas-Kanade flow Prob: we have more equaions han unknowns Soluion: solve leas squares problem minimum leas squares soluion given by soluion (in d) of: The summaions are over all piels in he K K window This echnique was firs proposed by Lukas & Kanade (1981) described in Trucco & Verri reading 16 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Condiions for solvabiliy Opimal (u, v) saisfies Lucas-Kanade equaion When is This Solvable? A T A should be inverible A T A should no be oo small due o noise eigenvalues l 1 and l 2 of A T A should no be oo small A T A should be well-condiioned l 1 / l 2 should no be oo large (l 1 = larger eigenvalue) 17 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Eigenvecors of A T A Suppose (,y) is on an edge. Wha is A T A? gradiens along edge all poin he same direcion gradiens away from edge have small magniude is an eigenvecor wih eigenvalue Wha s he oher eigenvecor of A T A? le N be perpendicular o N is he second eigenvecor wih eigenvalue 0 The eigenvecors of A T A relae o edge direcion and magniude 18 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Edge large gradiens, all he same large l 1, small l 2 19 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Low eure region gradiens have small magniude small l 1, small l 2 20 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
High eured region gradiens are differen, large magniudes large l 1, large l 2 21 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Observaion This is a wo image problem BUT Can measure sensiiviy by jus looking a one of he images! This ells us which piels are easy o rack, which are hard very useful laer on when we do feaure racking... 22 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Errors in Lukas-Kanade Wha are he poenial causes of errors in his procedure? Suppose A T A is easily inverible Suppose here is no much noise in he image When our assumpions are violaed Brighness consancy is no saisfied The moion is no small A poin does no move like is neighbors window size is oo large wha is he ideal window size? 23 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
mproving accuracy Recall our small moion assumpion -1 (,y) -1 (,y) This is no eac To do beer, we need o add higher order erms back in: -1 (,y) This is a polynomial roo finding problem Can solve using Newon s mehod Also known as Newon-Raphson mehod Lukas-Kanade mehod does one ieraion of Newon s mehod Beer resuls are obained via more ieraions 24 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
eraive Refinemen eraive Lukas-Kanade Algorihm 1. Esimae velociy a each piel by solving Lucas- Kanade equaions 2. Warp (-1) owards () using he esimaed flow field - use image warping echniques 3. Repea unil convergence 25 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Revisiing he small moion assumpion s his moion small enough? Probably no i s much larger han one piel (2 nd order erms dominae) How migh we solve his problem? 26 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Reduce he resoluion! 27 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Opical Flow Resuls 28 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
Opical Flow Resuls 29 * From Khurram Hassan-Shafique CAP5415 Compuer Vision 2003
30
31 * From Marc Pollefeys COMP 256 2003
Affine Flow 32 * Slide from Michael Black, CS143 2003
Horn & Schunck algorihm Addiional smoohness consrain : 2 2 2 es (( u + uy ) + ( v + = v 2 y )) ddy, besides Op. Flow consrain equaion erm ec ( u + yv + = ) 2 ddy, minimize es+aec 33 * From Marc Pollefeys COMP 256 2003
Horn & Schunck algorihm n simpler erms: f we wan dense flow, we need o regularize wha happens in ill condiioned (rank deficien) areas of he image. We ake he old cos funcion: d = arg min d ( (, ) ( + d, + 1) ) N And add a regularizaion erm o he cos: 2 d = arg min d N 2 ( (, ) ( + d, + 1) ) + a d Conve Program! We will see a lo of such formulaions in in robus regression! 34
Discussion: Wha are he oher mehods o improve opical flows?