Active Appearances. Statistical Appearance Models

Size: px

Start display at page:

Download "Active Appearances. Statistical Appearance Models"

Daniel Stevenson
5 years ago
Views:

Active Appearances The material following is based on T.F. Cootes, G.J. Edwards, and C.J. Taylor, Active Appearance Models, Proc. Fifth European Conf. Computer Vision, H. Burkhardt and B.

1 Active Appearances The material following is based on T.F. Cootes, G.J. Edwards, and C.J. Taylor, Active Appearance Models, Proc. Fifth European Conf. Computer Vision, H. Burkhardt and B. Neumann, eds., vol. 2, pp , T.F. Cootes, G.J. Edwards, and C.J. Taylor, "Active appearance models," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp , June Authors focus was development of method for matching statistical models of appearance to [2D] images Applied to faces, 2D medical images Basic idea has since been extended to many applications in 2D & 3D medical imaging /645 Fall 2015 Statistical Appearance Models Shape In this case, 2D locations of key feature points Texture I.e., patterns of intensities or colors across image patches Method to build: Identify key points; do deformable warp of points to common coordinate system; normalize intensities; read intensities into an intensity vector G G = 1 G k = /645 Fall

2 Statistical Appearance Models How might we do this? Shape In this case, 2D locations of key feature points Texture I.e., patterns of intensities or colors across image patches Method to build: Identify key points; do deformable warp of points to common coordinate system; normalize intensities; read intensities into an intensity vector G G = 1 G k = /645 Fall 2015 Deformable warping from point cloud matches One answer might make use of what we learned in programming assignments E.g., Determine some nominal location for each landmark point. E.g., pick some reference image or average multiple samples or do something else x (nom) k = 1 ( j ) x N j k Then fit Bernstein polynomials to determine distortion. x (nom) k = c s,t B s (u k ) B t (v k ) s,t Note: In this case, the coefficients will also parameterize the shape /645 Fall

Deformable warping from point cloud matches Another answer might use something like thin plate splines (e.g. Bookstein) TPS( v; a,b,c,p) = a +B v + U( v p i ) where U(r) = r 2 log( r) Thin plate splines are multidimensional analogues of 1- dimensional spline curves.

3 Deformable warping from point cloud matches Another answer might use something like thin plate splines (e.g. Bookstein) TPS( v; a,b,c,p) = a +B v + U( v p i ) where U(r) = r 2 log( r) Thin plate splines are multidimensional analogues of 1- dimensional spline curves. NOTE: One might also use other radial basis functions. For compact support, one example* could be c i i Ψ(r,σ) = I 1 r σ k+1+ d /2 0 otherwise if 0 r σ * See: M. Fornefett, K. Rohr, and H. S. Stiehl, "Radial basis functions with compact support for elastic registration of medical images", Image and Vision Computing, vol Äì2, pp , article/pii/s S (00) /645 Fall 2015 Thin Plate Splines Digression Some citations (from G. Donato and S. Belongie, Approximation Methods for Thin Plate Spline Mappings and Principal Warps, 2002; ) /645 Fall

4 M-dimensional Thin Plate Spline Summary Given TPS( v; a,b,c,p) = a +B v + U( v p i ) where U(r) = r 2 log r = r 2 log r 2 v = v 1,",v M p i = p 1,",p M ( ) for 2D ( ) for 3D T i P = p 1,", p N C = c 1,", c N B = b 1,", b M T T c i i /645 Fall 2015 M-dimensional Thin Plate Spline Fitting Given /645 Fall 2015 V = v 1,", v N F = f 1,", f N find a, B,C such that fi =TPS( v i ; a, B,C,V) To do this, solve the linear system K [NxN] 1[N 1] V 1 [1 N] 0 0 V T 0 0 [M M ] C T a T B T = F T 0 0 [M 1] where K i,j = K j,i = U v i v j K i,j = ( v i v j ) v i v ( j )log( ( v v ) ( v i j i v j )) ( ) with U(r) = r 2 logr or U(r) = r 2 logr 2 4

5 TPS 2D case Given a set of points p i = x i,y i and corresponding points p i * = x i *,y i *, we want to find TPS parameters such that p i * = TPS( p i ; a,b,c,p) To do this, we solve the least squares problem 0 " U 1,k " U 1,N 1 x 1 y 1 c 1 p 1 * # $ U ij # # # # # U k,1 " 0 " U k,n 1 x k y k # pk * # U ij $ # # # # # cn # = pn U N,1 " U N,k " 0 1 x N y N * a 1 " 1 " bx x 1 " x k " x N y 1 " y k " y N b y 0 where U i,j =U j,i =U( p i p j ) /645 Fall 2015 Define /645 Fall 2015 M-dimensional Thin Plate Spline Fitting L [M+N+1 M+N+1] = K 1[N 1] [NxN] V 1 [1 N] 0 0 V T 0 0 [M M ] If there are many points, this matrix may be expensive to invert or even pseudo-invert. There are various methods to deal with this problem. These include See Use a random sample of the v i to approximate the solution Use a random sample of the basis functions & all data to solve problem in least squares sense Use matrix approximation methods 5

6 Further Digression: Radial Basis Functions Note that the function U(r) in the previous discussion is a an example of a more general class of "radial basis functions". These functions can be used in deformable registration in much the same way as the TPS function used above. Other commonly used radial basis functions include U(r) = (r 2 + c 2 ) µ for µ + U(r) = (r 2 + c 2 ) µ for µ + U(e) = e r 2 /2σ 2 The last one is probably the most popular /645 Fall 2015 Appearance models, con d Appearance model is defined by an instance parameter vector λ, mean shape and texture X (avg) and G (avg), and variation mode matrices M X and M G. Thus, an instance ( j) would be G ( j ) = G (avg) +M G λ ( j ) = G (avg) + X ( j ) = X (avg) +M X λ ( j ) = X (avg) + N G M (k ) k=1 N X k=1 ( j ) λ G k M (k ) ( j ) λ X k In fact, they created a multi-resolution hierarchy with models similar to the above at different resolutions. Used PCA to determine the statistical parameters /645 Fall

7 Digression: PCA Suppose that you have a set of N vectors a i in an M dimensional space? Is there a natural "coordinate system" for these vectors? /645 Fall 2015 We proceed as follows a i a (avg) = i N /645 Fall 2015 Digression: PCA ; bi = a i a (avg) ; B= b 1," b N ; Then form the singular value decomposition B = UΣV T = U Σ(N) 0 V T where Σ (N) = diag(σ 1,",σ N ) Then we note that M = UΣ 2 U T. Of course U is huge, but we have the following useful fact. We note that B = u 1,", u N, u N+1,", u M σ 1 # σ N $ $ $ V T = u 1,", u N Σ(N) V T = U (N) Σ (N) V T 7

8 Digression: PCA This means that any column b k of B may be expressed as a linear combination of the first N columns of U where So B = u 1,", u N Σ(N) V T = U (N) Σ (N) V T b k = λ (k ) 1 u 1 +"+λ (k ) N u N = U (N) Λ (k ) Λ (k ) = transpose(u (N) ) b k a k = a (avg) + b k = a (avg) +λ 1 (k ) u 1 +"+λ N (k ) u N But often the last few values of the λ k are small. If we ignore all but the first D values, we have a k a (avg) +λ 1 (k ) u 1 +"+λ D (k ) u D /645 Fall 2015 Digression: PCA Suppose now that we have an arbitrary a (arb). We can approximate a (arb) as follows: b (arb) = a (arb) a (avg) Λ (arb) = transpose(u (D) ) b (arb) a (arb) a (avg) +λ 1 (arb) u 1 +"+λ D (arb) u D a (arb) a (approx) /645 Fall

9 400 faces 68 points Training Set for 2001 paper intensity values /645 Fall 2015 Complication How do you do PCA if shape and intensity may covary? Answer : Form combined vector of shape and intensity variation ( ) Y = W X X X(avg) G G (avg) where W X is a diagonal matrix of weights. Then do PCA on Y /645 Fall

Further complication How do you find the right weights to use? Answer (from Cootes et al.

Then find an optimal λ ( j ) for each training sample ( j).

and determine the corresponding texture vectors G ( j,k ).

10 Further complication How do you find the right weights to use? Answer (from Cootes et al. 1998): I.e., do PCA first on shape only and determine an appropriate V X. Then find an optimal λ ( j ) for each training sample ( j). Then vary the values of λ ( j,k ) = λ ( j ) +α e k to create new shape models X ( j,k ) and determine the corresponding texture vectors G ( j,k ). Then the weight w k = 1 G ( j,k ) G ( j ) 2 / α. j N /645 Fall 2015 Face modes Shape Intensity Source: Cootes et al /645 Fall

Face modes Combined Source: Cootes et al. 1998 21 600.

11 Face modes Combined Source: Cootes et al /645 Fall 2015 Basic Algorithm Make an initial guess at model weights Create a model from weights Evaluate error Iteratively improve /645 Fall

12 Basic Iteration of the Method Source: Cootes et al /645 Fall 2015 Basic Iteration of the Method Source: Cootes et al /645 Fall 2015 Note: simple sum of differences. What are some alternatives? 12

13 Results Source: Cootes et al /645 Fall 2015 Results Source: Cootes et al /645 Fall

14 Results: Knee Example Trained on 30 knee MRI images With 42 landmark points Source: Cootes et al /645 Fall 2015 Results: Knee Example Source: Cootes et al /645 Fall

Machine learning for pervasive systems Classification in high-dimensional spaces

Machine learning for pervasive systems Classification in high-dimensional spaces Department of Communications and Networking Aalto University, School of Electrical Engineering stephan.sigg@aalto.fi Version