Empirical Bayesian EM-based Motion Segmentation

Transcription

1 Emirical Bayesian EM-based Motion Segmentation Nuno Vasconcelos Andrew Liman MIT Media Laboratory 0 Ames St, E5-0M, Cambridge, MA 09 fnuno,lig@media.mit.edu Abstract A recent trend in motion-based segmentation has been to rely on statistical rocedures derived from eectationmaimization (EM) rinciles. EM-based aroaches have various attractives for segmentation, such as roceeding by taking non-greedy soft decisions with regards to the assignment of iels to regions, or allowing the use of sohisticated riors caable of imosing satial coherence on the segmentation. A ractical difficulty with such riors is, however, the determination of aroriate values for their arameters. In this work, we eloit the fact that the EM framework is itself suited for emirical Bayesian data analysis to develo an algorithm that finds the estimates of the rior arameters which best elain the observed data. Such an aroach maintains the Bayesian aeal of incororating rior beliefs, but requires only a qualitative descrition of the rior, avoiding the requirement of a quantitative secification of its arameters. This eliminates the need for trial-and-error strategies for the determination of these arameters and leads to better segmentations in less iterations.. Introduction A digital world of ubiquitous comuting, ultra-fast networking, and on-line communities unveils a new set of requirements for digital video reresentations. New reresentations should be fleible enough to suort interactivity, rovide sufficient content clues for automated retrieval and classification of content, and emhasize key features for scene understanding. Since these goals can be most naturally addressed in an object-domain (where scenes are characterized as comositions of object shaes, tetures, and motion), the ability to decomose a video sequence into the set of objects that comose it and accurately describe their motion becomes imortant. Image segmentation and motion (or otical flow) estimation have been widely studied in the fields of machine vision and image rocessing. Due to the difficulty of the segmentation roblem, early aroaches to otical flow comutation simly disregarded this comonent of the roblem, relying on smoothness assumtions and regularization to overcome the ill-osed nature of otical flow estimation [9, ]. This, however, resulted in oor motion estimates and imosed strong constraints (such as the assumtion of static scenes with camera motion, or simle scenes with a single moving object) on image analysis. It has been realized more recently that the roblem can be solved only by rocedures caable of jointly addressing the two comonents [, 5, 4]. This has led to a new generation of algorithms which iterate between otic flow estimation and segmentation. The idea is, for a given set of motion arameters and observed otic flow, to find the maimum a osteriori robability (MAP) estimate of the segmentation; and, given this segmentation, to find the set of motion arameters which maimizes the likelihood of the measured flow. Because a hard-decision (regarding the membershi of each iel in the image to each of the segmentation classes) is erformed for each iteration of these algorithms, they are sometimes referred to as clustering, or hard-decision algorithms. From a statistical ersective, such algorithms can be seen as variations of a stochastic otimization rocedure known as the Eectation-Maimization (EM) algorithm [6]. EM-based motion segmentation treats the roblem as one of region-wise regression. The observed motion field is seen as a realization of a stochastic rocess characterized by a Gaussian miture density with as many comonents as the number of distinct regions in the video sequence. Segmentation masks (i.e. which region is resonsible for each samle) are seen as hidden (non-observed) variables and the algorithm finds the values of the motion arameters that maimize the likelihood of the observed data by iterating between two stes. The E-ste estimates the eected values of the hidden variables given the current values of the motion arameters and the observed data. The M-ste then uses these eected values to find the set of arameters that maimize the likelihood of the data. Because the region-assignment variables are binary, and eectations of binary values are equal to the robabilities

2 of the variables being on ; the estimates comuted in the E-ste are nothing more than the osterior robability of the region-assignments given the observed otical flow. I.e., EM is similar to the hard-decision algorithms above, but roceeds by taking soft-decisions, the MAP estimate of the segmentation being taken only uon the convergence of the iterative rocedure. Even though soft-decisions can lead to significantly better erformance than hard-decisions [7], there are additional attractives in using EM for segmentation. In articular, because it rovides an elegant statistical framework for the segmentation roblem, EM allows the use of sohisticated riors, such as Markov Random Fields (MRFs) to enforce satial coherence on the segmentation [5, 6]. However, such riors are tyically characterized by arameters whose values are difficult to determine a riori. In ractice, these arameters are commonly set to arbitrary values, or adated to the observed data through heuristic rocedures. In this work, we eloit the fact that the EM framework is itself suited for emirical Bayesian data analysis [], and a well known aroimation to the likelihood of MRF rocesses to develo an algorithm that finds the estimates of the rior arameters which best elain the observed data. This eliminates the need for trial-and-error strategies for the determination of these arameters and leads to better segmentations in less EM iterations. Section rovides a brief review of Bayesian data analysis, and introduces the emirical Bayesian ideas that motivate our algorithm. The algorithm itself is then elained in detail in the remainder of the aer. Section resents the doubly stochastic motion model on which all the statistical inferences are based. Section 4 discusses the arameter initialization stage which rovides a rough segmentation guess used to bootstra the EM algorithm. The EM rocedure for emirical Bayesian motion segmentation is resented in section 5. Finally, section 6 illustrates the erformance of the algorithm with some simulation eamles and discusses the obtained results.. Bayesian and emirical Bayesian data analysis In this section, we briefly review Bayesian and emirical Bayesian rocedures [, ] for making inferences about the world, given observed image data. Assume that we are trying to make inferences about the world roerty Ω, given the image feature!. Under the Bayesian framework, all inferences are based on the osteriori density function P P (!jω)p (Ωj 0) (Ωj!) R P (!jω)p (Ωj0 )dω ; () where 0 is a arameter (or set of arameters) which controls the shae of the rior density for the world roerty. Under the Bayesian hilosohy, roerties in the world are not unknown deterministic quantities, but random variables characterized by robability densities that eress our degree of rior belief in their ossible configurations. The ratio between the osterior likelihoods of two configurations is roortional to the ratio of the resective rior likelihoods, the roortionality factor being deendent on the data. I.e. observation of the data merely re-scales rior beliefs [0]. It is therefore imortant, in Bayesian analysis, to get the rior beliefs right, a task which is generally difficult in ractice. Tyically one does not have absolute certainty about the shae of the rior density and the arameters that characterize it which, unless known with certainty, must be regarded as random variables. That is, unless there is absolute certainty regarding the value of 0, inferences should be based on P (Ωj!) R P (!jω)p (Ωj0 )P ( 0 )d 0 R R P (!jω)p (Ωj0 )P ( 0 )dωd 0 ; () instead of on equation. While from a ercetual standoint such a hierarchical structure has the aeal of modeling changes of rior belief according to contet (different contets lead to different values of 0, altering the shae of the density which characterizes rior beliefs), from a comutational standoint it significantly increases the comleity of the roblem. After all, the arameters of P ( 0 ) are themselves random variables, as well as the arameters of their density functions, and so on. We are therefore caught on a endless chain of conditional robabilities which is comutationally intractable. These issues are generally ignored in ractice, where riors are tyically chosen in order to minimize comutational comleity, or set to arbitrary values. The latter solution is revalent in the MRF literature, where arameters are commonly set in arbitrary fashion or adated using heuristics. The alternative suggested by the emirical Bayesian hilosohy is to relace 0 by an estimate ˆ 0 obtained as the value which maimizes the marginal distribution P (!j 0 ) as a function of 0. Inferences are then based on equation using this estimated value. While, strictly seaking, this aroach violates the fundamental Bayesian rincile that riors should not be estimated from data, in ractice it leads to more sensible solutions than setting riors arbitrarily, or using riors whose main justification comes from comutational simlicity (the so-called conjugate riors). More imortantly, it rovides a way to break the infinite chain of conditional robabilities mentioned above, while still allowing for different riors deending on contet. Consider, for eamle, the task of, given ictures of a tree, to determine the robability of the world roerty color (C) from the image feature iel color (c). The standard Bayesian solution would be to

3 erform inferences based on equation or, in this case, P (Cjc) / P (cjc)p (Cjs); where P (cjc), which is determined by the camera otics and sensor noise, relates world and iel colors, and P (Cjs) eresses rior beliefs in tree colors according to the arameters s. The main limitation of such model is that it fails to cature many factors that have an influence on tree colors, such as geograhy (leaf colors vary from region to region), seasonality (leaves are green in the Sring and yellow in the Fall), etc. Even though a simle rior may be aroriate to describe the colors of a given tye of tree, at a given time of the year, in a given geograhical location, no rior will be able to describe the colors of all trees, at all locations, for the entire year. Better models are obviously ossible by taking the route of equation, i.e. by considering hyerriors for all these factors, at the cost of enduring a significant increase in comleity. The emirical Bayesian ersective is to avoid this increase by keeing the simle model P (Cjs), but choosing the arameters s that best elain the data. In this way, even though not directly, the model can account for the variations above, as the estimated s will be different for ictures taken in different seasons, locations, etc. Choosing the s which maimizes P (cjs) will originate a rior which favors green colors for ictures taken in the Sring, and yellow colors for ictures taken in the Fall. In a sense, the emirical Bayesian aroach allows the observer to concentrate on the secification the qualitative shae of the rior, letting the quantitative comutation of rior arameters be inferred from the data. Comutationally, the bulk of work associated with emirical Bayesian rocedures relies on the search for the rior arameters that maimize the marginal likelihood P (!j 0 ). Because these arameters are related to the observed image features by the hidden world roerties, P (!j 0 ) Z P (!jω)p (Ωj 0 )dω; the roblem fits naturally into an EM framework. Thus, in ractice, emirical Bayesian estimates are commonly obtained through EM rocedures, that iterate between the comutation of the eected values for the world roerties, and the maimization over rior arameters. Therefore, the emirical Bayesian ersective not only suorts the recent trend towards the alication of EM for motion (and teture) segmentation, but etends it by roviding a meaningful way to tune the riors to the observed data.. Doubly stochastic motion model Our aroach to image segmentation is based on linear arametric motion models, according to which the motion of the iels associated with a given object is related to their image coordinates by () () ; () where (; y) T is the vector of iel coordinates in the image lane, () ( (); y ()) T the iel s motion, and (a ; : : : ; a P ) T the arameter vector which characterizes the motion of the entire object. In this work, we consider the articular case of affine motion where P 6, () y y ; (4) and equation models each of the comonents of the motion vector field as a lane in velocity sace. To account for uncertainties due to the imaging rocess, this motion model is embedded in a robabilistic framework, where iels are associated with classes that have a one-toone relationshi with the objects in the scene. We assume that, conditional on the knowledge of image I t? () and the class of iel in image I t (), the observed value of this iel is the outcome of an indeendent identically distributed Gaussian random rocess characterized by P (I t ()jz() e i ; i ; I t? ()) (5) e f? i i [I t ()? I t? (? i ())] g; where i () is the rediction of the motion of iel according to the class s model, i the variance of the iels in the class, z() is a vector of binary indicator variables, and z() e i (where e i is the i th vector of the standard unitary basis) if and only if iel belongs to object i. Deendencies between the class-assignment robabilities of adjacent iels are modeled by introducing a secondorder MRF as a segmentation rior P (z() e i jz) P (z() e i jz ()) Z e [ i + u i ()]; (6) where z is the random field of indicator vectors z(), z () is the second order neighborhood of iel (comosed by the eight adjacent iels), u i () is the number of neighbors of iel that belong to class i, and Z is a normalizing constant or artition function. This leads to a doubly stochastic motion model. Doubly stochastic random fields using MRFs are the -D etension of Hidden Markov Models (HMMs), and have long been used for teture modeling and segmentation [7, 4, ]. In articular, the rior of equation 6 has been shown to be a good model for segmentation masks (see for eamle figure 5 of [7]) and etensively used in the teture analysis literature. It is arameterized by the scalar and the vector

4 ( ; ; : : :) T. controls the degree of clustering, i.e. the likelihood of more or less class transitions between neighboring iels, while the s control the relative likelihood of each of the segmentation classes. 4. Parameter Initialization T T T For a tyical video sequence, the likelihood of the observed image data is a comlicated function of the segmentation and motion arameters. This resents a significant challenge to EM-based algorithms since, given a oor initial estimate, EM will get traed in undesirable local minima. The common aroach to the roblem of generating an initial estimate is to generate a large number of ossible models and use clustering techniques to reduce the cardinality of this set to the number of regions (classes) into which the sequence is to be segmented [4]. To initialize the EM algorithm, we start by comuting the otic flow between successive images using any of the conventional otical estimation techniques []. We then slit each image into M rectangular tiles and, for each tile, find the set of arameters ^ i ; i : : : M which achieves the least squares fit between the motion model of equation and the measured otic flow, according to the equations derived in aendi A. Even though the oulation of motion models obtained through this fied segmentation contains models which are close to the true R models, it also contains a significant number of outliers because many regions contain occlusion boundaries. These outliers are, tyically, characterized by motion models associated with affine lanes of large sloe and intercet, which can ruin the erformance of traditional clustering techniques, such as k-means. This is illustrated by figure. This figure illustrates a simle D eamle consisting of an occlusion boundary between two translating objects. In a), the solid line reresents the true otical flow originated by two translating objects, and the dashed lines the best affine fits i for each of three image tiles T i. As can be seen in b), if standard clustering is used to find the two arameter vectors associated with objects, the outlying model will ull one of the estimates away from its correct value. Even a robust clustering technique will fail in this eamle, as Here, there are tyically two sub-roblems: ) to determine how many classes are necessary to elain the observed data, and ) to estimate initial arameters for the motion of each of those regions. In this work we consider that the first roblem is solved, i.e. the number of image regions R is known in advance. If this is not the case, our aroach can be combined with most of the standard solutions to the roblem, which consist in solving the roblem for a range of values of R, and choosing the one which minimizes a cost function that includes the likelihood of the data and a enalty for solutions with large R. The number of tiles M is limited above by a minimum region size necessary for a reliable least squares estimate, and below by a region size which will minimize the number of regions containing motion boundaries. In our eeriments we have used region sizes of 0 0 iels. a) b) Figure. D eamle of the effect of outliers on clustering.,, and are the models suggested by each of the tiles. The resence of the outlier model will ull the centroid of one of the clusters away from its correct location. it will grou and in one cluster, leaving in the second. In order to obtain an initial estimate that is robust against these outliers, we rely on a rocedure which iterates between two stes: ) merging of models which are likely to be associated with the same object, and ) elimination of bad models by cross validation. 4.. Model merging As is shown in aendi A, the least squares estimates of i are Gaussian random vectors with mean i and covariances given by equation 9. The roblem of determining if two models are associated with the same image object can, therefore, be seen as a roblem of determining if the two associated arameter estimates are realizations of the same Gaussian rocess. For this, we use the well know result that [t i ] j [^ i ] j? [ i ] j [ i ] jj ; j : : : P (7) has a Student distribution with N? P degrees of freedom, where [a] j is the j th comonent of vector a, N is the region size in iels, and P the number of arameters in the motion model, and design the following sequential hyothesis test. Given two estimates ^ i and ^ k we use the hyothesis H 0 : [ i ] j [^ k ] j H : [ i ] j 6 [^ k ] j and the test statistic (null hyothesis) (alternative hyothesis) [T] j [^ i ] j? [^ k ] j [ i ] jj : (8) The null hyothesis is rejected at the level when j[t] j j > t N?P;? where t N?P;? is the (? )-quantile of the Student distribution above. If this haens, we know at the level that [^ k ] j is not the true value for the arameters 4

5 of region i, and we do not merge the regions. If not, we reverse the roles of i and k and reeat the test. If the null hyothesis is rejected for any of the j : : : P comonents of the motion arameter vector, the regions are not merged. I.e., regions are only merged when there is strong evidence that they are not distinct. Region merging is erformed by assigning all the initial square tiles associated with the air of models under analysis to the model that best elains their motion in the mean square sense. 4.. Outlier elimination A conservative region-merging strategy such as the above is required to avoid imroer merges, where a good model is combined with a model corruted by outliers. The other ste of our iterative rocedure aims to detect these outlier models and eliminate them. For this, we rely on the following crossvalidation rocedure. First, we start by considering a grid similar to the tilegrid used for the initial segmentation into fied size regions but dislaced by half of the tile dimensions. Each of the tiles R i in this dislaced grid is then wared according to all the current motion model candidates. The model that rovides the best fit between the wared tile and the the net image is then marked as a valid model. Finally, models that are not marked as valid for any of the tiles are eliminated from the list of candidates. The reasoning behind this cross-validation rocedure is illustrated by figure, which is a relica of figure with the tile grid dislaced. Because the grid is dislaced, model no longer rovides the best aroimation to the otical flow of the second tile, which is better elained by model. As a result, is identified as an outlier, and deleted from the list of candidate models, allowing any clustering technique to find the correct solution. aroimation to the true motion arameters of the various image regions. The second stage of our algorithm uses the EM-based emirical Bayesian learning aroach of section and the doubly stochastic motion model of section to: ) refine these initial estimates, ) find the MRF rior arameters which best elain the observed motion, and ) comute the MAP class assignment for each image iel. As mentioned in section, the fundamental comutational roblem osed by the emirical Bayesian framework is that of maimizing the marginal likelihood of the observed data as a function of the motion and MRF arameters P (I t j; I t? ) z P (I t jz; ; I t? )P (zj; I t? ); where the summation is over all ossible configurations of the hidden assignment variables vector z, is the vector of all motion and MRF arameters, and I t and I t? are the observed images. The air (I t ; z) is usually referred to as the comlete data and has log-likelihood l c log P (zji t? ; )+ ;i z i () log [P (I t ()jz() e i ; ; I t? )]; where z i () is the i th comonent of the vector z(), and where we have used the class conditional robabilities of equation 5, the conditional indeendence of the observations given the indicator variables, and the binary nature of z i (). The EM algorithm maimizes the likelihood of the incomlete, observed, data by iterating between two stes which act on the log-likelihood of the comlete data. 5.. The E-ste The E-ste comutes the so-called Q function defined by Q( 0 j () ) E[l c ji t ; () ] E[log P (zji t ; () )] + ;i E[z i ()ji t ; () ] log [P (I t ()jz() e i )] (9) R R R a) b) Figure. Elimination of outliers by cross-validation. 5. EM-based arameter estimation Because they are comuted over sets of tiles of arbitrary shae and granularity, the initial estimates are only a rough where () are the arameters obtained in the revious iteration and, for simlicity, we have droed the deendence on I t?. Under the MRF assumtion for the rior class robabilities, the comutation of E[z i ()ji t ; () ] and E[log P (zji t ; () )] becomes analytically intractable, and can only be addressed through Monte Carlo rocedures such as Gibbs samling [8]. Such rocedures are, however, eensive from a comutational ersective, and nesting a Gibbs samler inside the EM iteration would lead to a rohibitive amount of comutation. In order to simlify the roblem, we rely on the well known aroimation first roosed by Besag in his iterated coding mode (ICM) rocedure for MAP estimation of MRF arameters [], and later 5

6 used by Zhang et al. in the contet of EM-based segmentation [7]. This aroimation consists of relacing the true likelihood by the seudo-likelihood P (z) Y P (z()jz ()): (0) and is an etension of the Markov roerties of one dimensional chains, in which case the equation holds eactly. Assuming, further, that the configuration of the MRF does not change drastically from one iteration of the EM algorithm to the net, the seudo-likelihood can be aroimated by P (z) Y P (z()jz () ()) Y [P (z() e i jz () ())]zi() : ;i It is straightforward to show [7] that, under such aroimation, from which P (z i () j () ) P (z i () jz (); () ) () i (); () h i () E[z i ()ji t ; () ] P (z i () ji t ; () ) P (I t()jz() e i ; () )P (z i () j () ) P k P (I t()jz() e k ; () )P (z k () j () ) P (I t()jz() e i ; () ) () i () Pk P (I t()jz() e k ; () ) () ; () k () where we also used the binary nature of the indicator variables, and Bayes rule. Notice that the h i () are the osterior class assignment robabilities given the observed images. Given the current estimate of the rior robabilities () ( () (); () (); : : :)T, and the motion model arameters in (), they are comuted by substituting equation 5 in equation. One ossible roblem with this comutation is that a iel whose motion is oorly elained by all the models in () will originate zero class-conditional likelihoods and the corresonding h i () will be undefined. To avoid this roblem, we rely on the fact that a iel which cannot be elained by any of the models is an outlier, and set the corresonding h i () to zero. Such a solution has the additional benefit of roducing robust estimates without increasing the comleity of the M-ste. Once outliers are eliminated, equation 0, and the comuted h i s are substituted in 9, and the Q function becomes Q( 0 j () ) + ;i ;i h i () log P (I t ()jz() e i ) h i () log P (z()jz (); () ): () 5.. The M-ste In the emirical Bayesian framework, the M-ste maimizes the Q function obtained in the E-ste with resect to both the motion and MRF arameters. Substituting equations 5 and 6 in equation, we obtain Q( 0 j () )?? + ;i ;i ;i h i() log ( i ) h i () i [I t ()? I t? (? i ())] h i ()[ i + u i ()? log Z]: Since the first two terms on the right hand side of this equation do not deend on i or and the third term does not deend on i or i, the maimization can be searated into two sub-roblems. The first - maimization of Q with resect to the arameters of the class conditional df s - is a variation of the non-linear least-squares roblem found in otical flow estimation, and is solvable by non-linear otimization techniques. In our imlementation, we use a simlified version of Newton s method, where the terms which deend on second order image derivatives are disregarded, leading to the following iteration (k+) i " (k) i? ( (k+) i ) h i () () T r I 0 t? ()r I 0 t? ()T h i ()[I t ()? I 0 t? ()] ()T r I 0 t? (); () P hi ()[I t ()? I t? (? () (k+) P hi () #? i )] where r I is the satial gradient of I, and I 0 t? () I t? (? () (k) i ). The second sub-roblem - maimization of Q with resect to MRF arameters - deends only on the third term, and can also be solved through standard non-linear rogramming methods. In our imlementation we have used the simlest of such methods, gradient ascent, under which the MRF arameters are udated in the direction of the gradients of the likelihood function with resect @ [h i ()? i ()]; (4) [ i h i ()u i ()? i i ()u i ()] [E[u()] ost? E[u()] rior ]; (5) ; 6

7 where E[u()] is the eected number of neighbors of iel that belong to the same class. Once the new values of the MRF arameters are comuted, the rior robabilities (+) i are obtained by alying a single cycle of Besag s ICM rocedure: each iel is visited in a raster scan order and, given the configuration of its neighborhood, the corresonding i () are comuted using equation 6. It can be shown that Q is a concave function of i and, guaranteeing the eistence of a single global maima, and allowing fast convergence to the otimal value. It is interesting to analyze the meaning of the equations above. The new motion arameters are what one would obtain by erforming a weighted non-linear least squares-fit to the motion field that best aligns the two images. The arameter udate does not, however, rely on a greedy binary segmentation mask which is instead relaced by the osterior class assignment robabilities. I.e. the influence of a iel on the least-squares fit to the motion arameters of a given class is roortional to the likelihood of the iel belonging to that class. The gradient udate equations also have a nice intuitive meaning. A ste in the direction of equation 4 changes the MRF arameter so that, at each iel, the rior classassignment robabilities move towards the osterior assignment robabilities obtained from the observed motion. Similarly, a ste in the direction of equation 5 changes so that, at each iel, the eected number of neighbors in the same state as the iel is equal under both the rior and the osterior distributions. I.e. the EM algorithm sets the model arameters to the values that best elain the observed data, both in terms of class assignment robabilities and average number of neighbors in the same state as the neighborhood s central iel. 6. Eerimental results and conclusions In this section, we reort on simulation results obtained with the flower garden sequence. Figure resents a frame from the sequence, and the estimate of the segmentation obtained by the arameter initialization algorithm of section 4. While this segmentation mask is only a rough estimate of the true segmentation, it is able to cature the overall structure of the scene, discriminating the four regions that comose it. Figure 4 illustrates the benefits of the emirical Bayesian solution to the motion segmentation roblem that is now roosed. It resents three segmentations obtained after twenty iterations of the EM algorithm described in section 5, the to two originated by setting the MRF arameters to arbitrary values, and the bottom one roduced by the comlete EM rocedure (i.e. using equations 4 and 5 to comute these arameters). When the MRF arameters are set arbitrarily, the segmentation deends critically on the choice of the Figure. A frame from the inut video sequence (left), and the segmentation (right) originated by the arameter initialization algorithm of section 4. clustering arameter. Small values of clustering, lead to noisy segmentations such as the one on the to of the figure, while large values of originate segmentations with weakly defined region boundaries (notice the leakage between the house and sky regions and between the areas of tree detail and sky in the middle icture). While it may be ossible to obtain better results by a trial-and-error strategy for the determination of MRF arameters, we were not able to obtain, in this way, a better segmentation than the originated by the emirical Bayesian aroach, which is shown at the bottom of the figure. The better erformance of emirical Bayesian estimates can be understood by considering figure 5, which resents the evolution of the clustering arameter estimate as a function of the iteration number (for two different starting oints). Once again, the result of emirical Bayesian arameter udating makes intuitive sense: while in early iterations (where uncertainty is high) clustering is small and iels are free to wonder between regions, the clustering arameter increases as the EM rocedure aroaches convergence, and the segmentation freezes when this haens. Even if such gradual evolution were not required for a good segmentation, it is not clear that the best trial-and-error estimate for a given sequence would be a good estimate for a different one. In fact, a review of the teture segmentation literature reveals a wide range of roosals for the value of, which did not include the values that worked best for us. The oint is that using emirical Bayesian estimates eliminates the need for tedious trial-and-error rocedures that are not always guaranteed to rovide the best results. References [] J. Barron, D. Fleet, and S. Beauchemin. Performance of Otical Flow Techniques. International Journal of Comuter Vision, vol., 994. [] J. Besag. On the Statistic Analysis of Dirty Pictures. J. R. Statistical Society B, 48():59 0, 986. In a short survey we encountered values ranging from.5 to.5 7

8 Figure 4. Three motion based EM segmentations. For the to two, the MRF arameters were set to arbitrary values (to: 0:, middle: 0:7). The bottom one was obtained with the emirical Bayesian arameter estimates discussed in the tet. White iels are outliers. Beta Initial Beta 0 Initial Beta iteration Figure 5. Evolution of the clustering arameter as a function of iteration number. The two curves corresond to two different initial estimates of the arameter value. Notice that the evolution of is very insensitive to the initial estimate. [] B. Carlin and T. Louis. Bayes and Emirical Bayes Methods for Data Analysis. Chaman Hall, 996. [4] M. Comer and E. Del. Parameter Estimation and Segmentation of Noisy or Tetured Images Using the EM Algorithm and MPM Estimation. In Proc. Int. Conf. on Image Processing, Austin, Teas, 994. [5] T. Darrel and A. Pentland. Cooerative Robust Estimation Using Layers of Suort. Technical Reort 6, MIT Media Laboratory Percetual Comuting Grou, June 99. [6] A. Demster, N. Laird, and D. Rubin. Maimum-likelihood from Incomlete Data via the EM Algorithm. J. of the Royal Statistical Society, B-9, 977. [7] H. Derin and H. Elliott. Modeling and Segmentation of Noisy and Tetured Images Using Gibbs Random Fields. IEEE Trans. on Pattern. Analysis and Machine Intelligence, vol. PAMI-9, January 987. [8] S. Geman and D. Geman. Stochastic Relaation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Trans. on Pattern. Analysis and Machine Intelligence, Vol. PAMI-6, November 984. [9] B. Horn and B. Schunk. Determining Otical Flow. Artificial Intelligence, Vol. 7, 98. [0] A. Jeson, W. Richards, and D. Knill. Modal Structure and Reliable Inference. In D. Knill and W. Richars, editors, Percetion as Bayesian Inference. Cambridge Univ. Press, 996. [] B. Lucas and T. Kanade. An Iterative Image Registration Technique with an Alication to Stereo Vision. In Proc. DARPA Image Understanding Worksho, 98. [] D. Murray and B. Buton. Scene Segmentation from Visual Motion Using Global Otimization. IEEE Trans. on Pattern. Analysis and Machine Intelligence, Vol. PAMI-9, March 987. [] H. Robbins. An Emirical Bayes Aroach to Statistics. In Proc. Third Berkley Symosium Math. Statist., ages 57 6, 956. [4] J. Wang and E. Adelson. Reresenting Moving Images with Layers. IEEE Trans. on Image Processing, Vol., Setember 994. [5] Y. Weiss and E. Adelson. Percetually Organized EM: A Framework for Motion Segmentation that Combines Information about Form and Motion. In Proc. Int. Conf. on Comuter Vision, 995. [6] Y. Weiss and E. Adelson. A Unified Miture Framework for Motion Segmentation: IncororatingSatial Coherence and Estimating the Number of Models. In Proc. Comuter Vision and Pattern Recognition Conf., 996. [7] J. Zhang, J. Modestino, and D. Langan. Maimum- Likelihood Parameter Estimation for Unsuervised Stochastic Model-Based Image Segmentation. IEEE Trans. on Image Processing, Vol., July 994. A. Least squares otical flow fit It is well known that: ) modeling the otical flow v() as v() () + (); (6) 8

9 where () is a set of iid zero-mean Gaussian random variables and () defined by the motion model of equation, leads to a Gaussian likelihood function for the observed flow; and ) the maimization of this likelihood with resect to the motion arameters is equivalent to the minimization of the mean squared error between the observed flow and its rediction according to the model E (v()? ()) T? (v()? ()); (7) where is the covariance matri associated with (). Substituting equation into equation 7, comuting the gradient of E with resect to, and setting it to zero, we find that the estimate of which rovides the least squares fit 4 is!? ^ () T? () () T? v() : (8) Combining this equation with equation 6, it can be easily shown that ^ is a Gaussian random vector with mean and covariance ^! () T? ()!? : (9) 4 Also known as the maimum likelihood estimate of. 9