A total least squares framework for low-level analysis of dynamic scenes and processes

Transcription

1 A total least squares framework for low-level analysis of dynamic scenes and processes Horst Haußecker 1,2, Christoph Garbe 1, Hagen Spies 1,3, and Bernd Jähne 1 1 Interdisciplinary Center for Scientific Computing (IWR), University of Heidelberg Im Neuenheimer Feld 368, D Heidelberg, Germany {horst.haussecker,christoph.garbe,bernd.jaehne}@iwr.uni-heidelberg.de 2 Xerox Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, CA Dept. of Comp. Science, University of Western Ontario London, Ontario, N6G 5B7 Canada, hagen.spies@iwr.uni-heidelberg.de Abstract. We present a new method to simultaneously estimate optical flow fields and parameters of dynamic processes, violating the standard brightness change constraint equation. This technique constitutes a straightforward generalization of the standard brightness constancy assumption. Using TLS estimation the spatiotemporal brightness structure is analyzed in an entirely symmetric way with respect to the spatial and temporal coordinates. We directly incorporate nonlinear brightness changes based upon differential equations of the underlying processes. Keywords. optical flow, total least squares, brightness change model 1 Introduction The role of total least squares (TLS) parameter estimation ([13]) in motion analysis has been increasingly appreciated [8]. Although it is not always explicitly referred to, many recent approaches to optical flow computation (e. g. [1], [2], [12], [4], [10]) are based on TLS or related techniques. In differential-based optical flow techniques spatial gradients are related to temporal changes. Optical flow estimates are obtained by pooling local constraints over a small neighborhood in a least squares sense. Using standard least squares (LS) techniques the temporal derivatives are treated as (measured) observables and the spatial gradients are assumed to be error-free. However, both are prone to discretization errors and noise which leads to biased estimates in case of standard least squares techniques ([5]). As TLS estimation is symmetric in the spatial and temporal coordinates (see Appendix 1) it is unbiased in case of isotropic noise. Most optical flow methods are in fact model-based approaches. However, the underlying model is commonly the most simple assumption on the spatiotemporal brightness distribution, namely constant optical flow within a small spatial neighborhood, and conservation of image brightness along the path of objects. Our approach weakens these strong conservation laws and allows parameterized variations of both the motion field and the image brightness. In contrast to other approaches (e. g. [9],[11],[3]), temporal brightness variations of higher than linear order are allowed. Hence, the spatiotemporal image structure is analyzed in

2 an entirely symmetric way with respect to spatial and temporal coordinates. A major contribution of this paper is the generalized framework for incorporating brightness changes into motion analysis, based upon physical constraints in terms of differential equations of the underlying transport processes. It is capable of dealing with dynamically changing objects and constitutes a generalization of the brightness constancy assumption in a straightforward way. The work of this paper has been triggered by the need to quantitatively measure transport, exchange, and growth processes in scientific applications [7]. Here objects will change their shape and brightness, possibly within a few images. Standard brightness constancy assumptions inevitably lead to biased estimates of both the object motion and radiometric parameters. The new approach does not only allow to improve the accuracy of motion estimation but simultaneously yields physical parameters of the underlying transport processes. 2 Brightness constancy assumption Before we turn towards varying object brightness, we recall the basic brightness constancy assumption generally used in optical flow techniques. It requires the total time derivative of the object brightness g to equal zero, i.e. dg dt = g dx x dt + g dy y dt + g t =0, or, [g x,g y,g t ] f 1 f 2 1 =( g) T u=0, (1) where f =[f 1,f 2 ] T =[dx/dt, dy/dt] T ist the optical flow to be estimated, and u =[f 1,f 2,1] T is a 3-D vector in the spatiotemporal domain. With g i we denote the partial derivative of g with respect to the coordinate i {x, y, t}. All three partial derivatives are combined to the spatiotemporal gradient g. Equation (1) is known as the brightness change constraint equation (BCCE) [6]. It is ill-posed, as it constitutes only one equation in two unknowns. A common method to find a solution is to further constrain (1) by assuming that the optical flow f is constant within a small spatial neighborhood surrounding the pixel of interest. In order to further regularize the result, the neighborhood can as well be extended into the time domain 1. Given a set of N neighboring pixels, we get the following linear equation system: Du = 0 with D =[ g 1,..., g N ] T, (2) where D is a N 3 matrix made up from the gradients g n at the position n. The TLS solution of (2) is outlined in Appendix 1 and detailed in [5]. Although the BCCE model poses strict constraints that will be rarely fulfilled, the TLS estimation of (2) yields good results in a wide variety of applications with typically slow variations of both the object brightness and the spatial distribution of f. Such results are reported in [12], [4], [10], and [5]. 1 For enhanced brightness change models (Sect. 4) it is an important prerequisite to extend the neighborhood into the temporal domain. Otherwise no temporal brightness changes of higher than linear order can be estimated.

3 a b t x u u u u u u f f f f f f = 0 Fig. 1. A 1-D sinusoidal pattern is moving along the x-axis with a constant brightness and b linear brightness change. The optical flow f is the projection of the spatiotemporal parameter vector u onto the spatial coordinates (1). The dashed lines indicate the spatiotemporal path of the object, i. e. the correct direction of u. The influence of brightness changes is illustrated in Fig. 1 for 1-D constant motion of a sinusoidal pattern. In case of constant brightness (Fig. 1 a), all vectors u are pointing parallel and yield the correct optical flow f. For changing object brightness (Fig. 1 b), the algorithm still estimates u as the direction of constant brightness within the spatiotemporal neighborhood, although this deviates from the correct spatiotemporal path. This results in a biased estimate of f. 3 Parameterized flow model In a first step we allow the optical flow field to locally vary according to some parameterized function. Following [1] and [2], we replace the optical flow (translation) vector u by a generalized transformation S(r, a), where S =[S x,s y,s t ] T defines a 3-D invertible transformation acting on the spatiotemporal position r =[x, y, t] T : r = S(r, a), and r = S 1 (r, a). (3) With a =[a 1,...,a P ] T we denote the P -dimensional parameter vector of the transformation. For the moment we still require the brightness g to remain constant along the path of the object, i. e. g(r) =g(r )=g ( S 1 (r,a) ). (4) Successive application of S defines a trajectory through the sequence along which the brightness of g remains constant. If S is infinitely differentiable in r, and analytic in a, it forms a Lie group of transformations (LGOT). In this case, the vector r can be expanded about a =0: r=r + P i=1 a i S(r, a), with r = S(r, a =0), (5)

4 i. e. a = 0 is the identity element of the transformation. The dependence of the brightness function g on the transformation parameters a i can be derived as g(r) = g x + g x y y + g t t = g S x + g S y + g S t = L i g(r), (6) x y t using (5). The operator L i, i {1,...,P}, is called an infinitesimal generator of the Lie group of transformations in a i and defined as: L i = S x x + S y y + S t t. (7) In a final step, we expand g about r with respect to the parameters a i : g(r) =g(r )+ P i=1 a i g = g(r )+ P a i L i g, (8) where (6) was used. With the initial assumption of brightness conservation (4), i. e. g(r) =g(r ), (8) reduces to: i=1 P a i L i g =0, or, (Lg) T a =0, (9) i=1 which constitutes a generalized brightness change constraint equation (GBCCE), where the 3-D spatiotemporal gradient is replaced by the P -dimensional vector of Lie-derivatives Lg =[L 1 g,...,l P g] T. In order to find a best estimate of a we assume a to be constant within a small neighborhood containing N pixels and get the following linear equation system La =0, with L =[Lg 1,...,Lg N ] T. (10) Minimization of (10) with respect to a can again be carried out by the general framework of TLS estimation (Appendix 1). Examples of the coordinate transformation S are affine flow or higher-order polynomials in the spatial coordinates [5]. As this paper focuses on parameterized brightness changes, we only illustrate a trivial example for a parameterized flow, and refer to the literature for further examples (e. g. [1],[2],[5]). Example 1 (Constant translation). For constant translation, the coordinate transformation S reads S(r, t) =r+t, (11) where t =[δx, δy, δt] T denotes the translation vector to be estimated. Letting a = t, the infinitesimal generators are given by (7) L 1 = L x = x, L 2 = L y = y, and L 3 = L t = t. (12) Thus, (9) automatically yields the standard BCCE (1): δx g x + δy g y + δt g =0. (13) t

5 4 Dynamic spatiotemporal brightness model It has already been emphasized that a constant brightness model has to be refined in order to account for changing object brightness. Along the correct direction in the spatiotemporal domain, the brightness does not remain constant but changes according to the underlying physical process (Fig. 1 b). Hence, we need to find the directions of brightness changes following the corresponding functional relationship, instead of searching iso-brightness lines in the spatiotemporal neighborhood. In the most simple case, the brightness change is known to be linear, consisting of an offset and multiplier field. In [3] and [11] it is shown, that a variety of radiometric transformations can be cast into a linear relationship. It is also evident, that slow brightness variations can be expressed as linear changes in a first-order approximation. In scientific applications, however, dynamic transport processes can lead to fast brightness changes of higher than linear order, such as e. g. exponential decay. Furthermore, some of the transport models are not known as analytical functions but rather as physical laws in terms of differential equations, combining the spatial structure of objects with temporal brightness changes (e. g. diffusion processes). In addition to the spatiotemporal transformation (4), we allow the brightness of moving patterns to change according to a parameterized analytical function h(g) along the spatiotemporal path, where h defines a scalar invertible transformation acting on the image brightness g: g(r )=h(g (r ),b), and g (r )=h 1 (g(r ),b). (14) With b =[b 1,...,b Q ] T we denote the Q-dimensional parameter vector of the brightness change. The primed brightness g (r ) equals the initial brightness g(r) g (r )=g(r), (15) shifted to the position r. Hence, the entire transformation of the spatiotemporal structure can be virtually separated into two steps: 1. transformation without brightness change, g(r) g (r ), according to (4), 2. adjustment of the brightness, g (r ) g(r ), according to (14). If h is analytic in b, we can expand the brightness variation about b =0: g(r )=g (r )+ Q k=1 b k h b k, with h (g (r ), b = 0) =g (r ), (16) where b = 0 is the identity element of the transformation. Let further the spatiotemporal transformation r r be given by a LGOT, i. e. the flow field can be modeled according to (4)-(10). Then, we can express g(r ) in (16) by (8) and get, using (15): g(r) g (r )= P Q h a i L i g b k =0, (17) b k i=1 k=1

6 which constitutes a further generalization of the BCCE for parameterized brightness changes. In case the object brightness remains constant (b = 0), (17) reduces to (9). Again, we can express (17) as a scalar product d T p =0, with d = [ (Lg) T, ( b h) T ] T and p = [ a T, b T ] T, (18) where p denotes the augmented (P + Q)-dimensional parameter vector, containing parameters of both, the spatiotemporal transformation S and the brightness transformation h. Correspondingly, the (P + Q)-dimensional vector d combines the Lie-derivatives in (9) and the gradient of h with respect to the parameters b k. The same way as (1) and (9), (18) is ill-posed and needs to be further constrained in order to estimate the parameter vector p. Assuming p to be constant within a neighborhood U, we can apply (18) to all N pixels of the neighborhood and get the following linear equation system: Gp =0, with G =[d 1,...,d N ] T. (19) Once more, minimization of (19) with respect to p can be carried out by the general framework of TLS estimation (Appendix 1). Example 2 (Translation with exponential decay). If g(r) can be modeled as an exponential decay the brightness variation has the analytical form h(g(r),κ)=g(r )=g(r) exp ( κδt), (20) where the parameter vector b reduces to the scalar decay constant κ. Hence, h(g(r),κ) = δt g(r) exp ( κδt)= δt g(r ). (21) κ Assuming the optical flow to be a constant translation, the Lie-derivatives are given by (12) and (18) reduces to: d T p =0, with d =[g x,g y,g t,g] T, and p =[δx,δy,δt,κδt] T. (22) Example 3 (Translation with diffusion). For patterns that are subject to isotropic diffusion, the temporal brightness variation depends on the spatial structure according to the following differential equation: dg dt = D g=d (g xx + g yy ), (23) where D is the diffusion constant, and g xx and g yy denote the second-order derivatives in the spatial coordinates. In order to turn (23) into an analytical function, we approximate the total differential dg/dt by the discrete difference dg/dt = δt 1 (g(r ) g(r)). Solving for g(r ) yields h(g(r),d)=g(r )=g(r)+dδt g, (24)

7 a b c d frame #0 frame #6 (exponential) frame #6 (diffusion) #0 # 6, exponential # 6, diffusion normalized grey value pixel position Fig. 2. Test sequences for brightness changes. A Gaussian pattern is moving with the velocity [0.8, 0.8] T pixels/frame: a first frame of all sequences b exponential decay, c diffusion d cross sections through the maximum of the Gaussian at the actual position. and h(g(r),d) = δt g. (25) D Together with (12), assuming constant translation, we get from (18): d T p =0, with d =[g x,g y,g t, g] T, and p =[δx, δy, δt, Dδt] T. (26) 5 Results In order to illustrate the performance of the new technique, we applied it to both, computer-generated test sequences with ground-truth, as well as to application examples. The generated test sequences consist of a Gaussian pattern moving with constant translation of f =[0.8,0.8] T pixels/frame (Fig. 2). In addition to the constant brightness case, we created two sequences with the Gaussian being subject to exponential decay and diffusion, respectively. Figure 2 shows the resulting changes of the Gaussian pattern. Table 1 summarizes the results of three different parameterized models applied to all three sequences, namely constant brightness, exponential decay, and diffusion, respectively. It is obvious, that both the optical flow and the model parameters are estimated with high accuracy if the used model fits the underlying transport process. It can also be observed, that all models yield accurate results in case the object remains at constant brightness (first row). In this case, all models correctly estimate the model parameters to equal zero, as this was chosen to be the identity element of the dynamic brightness change. Figure 3 shows an application example from oceanography. The scientific task was to estimate the decay constant of an exponentially decaying heat spot on the ocean surface. In addition to the exponential decay the pattern is subject to deformation according to the underlying turbulent flow field. If the brightess is assumed to remain constant (Fig. 3 c), the estimated flow field is entirely unrealistic. However, using an exponential model for the dynamic changes (Fig. 3 d), the flow field can be accurately estimated together with the physically relevant decay constant κ.

8 Fig. 3. Exponentially decaying heat spot at a wavy water surface. a and b first and last frame of the sequence. c Optical flow field f estimated with the constant brightness assumption. d Optical flow field estimated with an exponential decay model. The decay rate κ is averaged over the area thresholded by the confidence measure. 6 Conclusions We presented a new approach to quantitatively analyze dynamic scenes and processes with high accuracy. The new technique is a straightforward extension of the standard brightness change constraint equation (BCCE) incorporating the spatiotemporal signature of dynamic processes. The presented framework allows to directly incorporate the underlying physical parameters in terms of differential equations. It is not restricted to physical transport processes but can be applied to all scenes that violate the brightness constancy assumption according to some known parametric model. Acknowledgements. We gratefully acknowledge financial support by the Deutsche Forschungsgemeinschaft, DFG, within the frame of the research unit Image Sequence Analysis to Investigate Dynamic Processes. References 1. Duc, B.: Feature design: applications to motion analysis and identity verification. PhD thesis, École Polytechnique Fédérale de Lausanne (1997) 2. Florac, L., W. Niessen, and M. Nielsen: The intrinsic structure of optical flow incorporating measurement duality. Intl. J. Comp. Vis., 27(3), (1998) Hager, G. D., and P. N. Belhumeur: Efficient region tracking with parametric models of geometry and illumination. IEEE PAMI, 20(10), (1998) Haußecker, H., and B. Jähne: A tensor approach for precise computation of dense displacement vector fields. In Informatik aktuell, Paulus, E., Wahl, F. M. (Hrsg.), Mustererkennung 1997, Springer-Verlag: Berlin, Heidelberg (1997) Haußecker, H., and H. Spies: Motion. In Handbook of Computer Vision and Applications, Jähne, B., Haußecker, H., and Geißler, P. (Eds.), Academic Press, (1999) 6. Horn, B. K., and B. G. Schunk: Determining optical flow. AI, 17, (1999) Jähne, B., H. Haußecker, H. Scharr, H. Spies, D. Schmundt, and U. Schurr: Study of dynamical processes with tensor-based spatiotemporal image processing techniques. Proc. ECCV 98 (Vol. 2), Burkhardt, H., and Neumann, B. (Eds.), Springer-Verlag: Berlin, Heidelberg (1998)

9 8. Mühlich, M., and R. Mester: The role of Total Least Squares in motion analysis. Proc. ECCV 98 (Vol. 2), Burkhardt, H., and Neumann, B. (Eds.), Springer-Verlag: Berlin, Heidelberg (1998) Nagel, H.-H.: On a constraint equation for the estimation of displacement rates in image sequences. IEEE PAMI, 11(1), (1989) Nagel, H.-H., and A. Gehrke: Bildbereichsbasierte Verfolgung von Straßenfahrzeugen durch adaptive Schätzung und Segmentierung von Optischen-Flußfeldern. In Informatik aktuell, Levi, P., Ahlers, R.-J., May, F., Schanz, M. (Hrsg.), Mustererkennung 1998, Springer-Verlag: Berlin, Heidelberg (1998) Negahdaripour, S.: Revised definition of optical flow: integration of radiometric and geometric clues for dynamic scene analysis. IEEE PAMI, 20(9), (1998) Ohta, N.: Optical flow detection using a general noise model. IEICE Trans. Inf. & Syst., Vol. E79-D, No. 7 July (1996) Van Huffel, S., and S. Vandewalle: The Total Least Squares Problem: Computational aspects and analysis. SIAM, (1991) Appendix 1: Total Least Squares Method Given the linear equation system Mp = 0, with an N P model Matrix M = [m 1,...,m N ] T, and a P -dimensional parameter vector p, N P, the total least squares method (TLS) seeks to minimize Mp 2. In order to avoid the trivial solution p = 0 we additionally require p T p = 1. Carrying out the minimization by the method of Lagrange multipliers we get p = arg min L(p,λ), L = Mp 2 +λ ( 1 p T p ) = p T Jp+λ ( 1 p T p ), (27) where J = M T M. For the constant brightness model (2), i. e. M = D, the matrix J is called structure tensor [4]. Eq. (27) is solved by forcing the partial derivatives of L with respect to all components of p to equal zero, which leads to the following eigenvalue problem: Jr = λr. It can be shown, that the vector r minimizing (27) is given by the eigenvector of J to the minimum eigenvalue [5]. This corresponds to the right singular vector to the smallest singular value of M ([13],[8]). Hence, the TLS solution can be found by an eigenvalue analysis of J or, alternatively by a singular value decomposition (SVD) of M. Without going further into detail, we want to point out, that confidence measures can be found by analyzing the relative size of the eigenvalues, or singular values, respectively ([4],[5],[10],[12]), which have been used to threshold the results in Fig. 3 and Tab. 1. The components of J are given by J pq = N 1 i=0 m ipm iq. In practical applications the summation can be replaced by a weighted average, i. e. J pq = m p m q = B (m p m q ), where B denotes a (typically binomial) smoothing operator applied to the pointwise product (m p m q ).

10 Table 1. Results of different parameterized brightness constraint models (columns) used to estimate motion and brightness change parameters of test sequences (rows) with constant brightness, exponential decay, and diffusion, respectively. Every sequence is evaluated with all three models. Errors are given for the estimated optical flow field (E f ) and for the model parameters (exponential decay constant E k, and diffusion constant E D; no BC parameter indicates that no additional brightness change parameter is estimated for the constant model). All errors are relative errors (in per cent) and show the mean and standard deviation across an area thresholded by the confidence measure. Dynamic model used for motion and brightness change estimation constant exponential diffusion constant E f =(6.5±7.9) 10 4 no BC parameter E f =(1.4±1.7) 10 2 E k =(6.7±8.2) 10 4 E f =(1.1±1.1) 10 3 E D =(4.8±5.9) 10 3 exponential E f =(4.7±2.6) 10 1 no BC parameter E f =(3.7±3.3) 10 2 E f =(1.9±1.0) 10 1 E k =(2.1±1.1) 10 1 E D =(5.3±0.9) 10 4 diffusion E f =(0.5±1.2) 10 3 no BC parameter E f =(5.8±1.5) 10 2 E k =(1.0±0.0) 10 2 E f =(1.2±0.5) 10 1 E D =(6.7±3.5) 10 2