Tracking an Unknown Time-Varying Number of Speakers using TDOA Measurements: A Random Finite Set Approach

Transcription

1 Tracing an Unnown Time-Varying Number of Speaers using TDOA Measurements: A Random Finite Set Approach Wing-Kin Ma, Ba-Ngu Vo, S. Singh, and A. Baddeley Abstract Speaer location estimation techniques based on time-difference-of-arrival (TDOA) measurements have attracted much attention recently. Many existing localization ideas assume that only one speaer is active at a time. In this paper, we focus on a more realistic assumption that the number of active speaers is unnown and time-varying. Such an assumption results in a more complex localization problem, and we employ the random finite set (RFS) theory to deal with that problem. The RFS concepts provide us with an effective, solid foundation where the multispeaer locations and the number of speaers are integrated to form a single set-valued variable. By applying a sequential Monte Carlo (SMC) implementation, we develop a Bayesian RFS filter that simultaneously tracs the time-varying speaer locations and number of speaers. The tracing capability of the proposed filter is demonstrated in simulated reverberant environments. I. INTRODUCTION Speaer localization using voice activity is an important problem in microphone array processing, driven by applications such as automatic camera steering in video-conferencing. By localization, one can consider estimating the directions of the speaer sources, or estimating the Cartesian coordinates of the sources. In this paper we are interested in the Cartesian coordinate localization, with particular emphasis on the time-difference-of-arrival (TDOA) approach. Readers who are interested in the direction finding approach (which presents a rather different signal processing framewor compared to the TDOA) are referred to the literature such as [] [4] and the references therein. Fig. depicts a microphone placement for the TDOA approach. Essentially microphones are grouped into pairs and those pairs are distributed in the room (note that one can also choose to place those pairs in proximity to each other; see the setting in [5] for example). For each pair the TDOA is measured independently. Assuming single speaer activity and no reverberation, the TDOAs can be reliably measured using a generalized cross correlation (GCC) method [6], [7]. The measured TDOAs, which embed location information relative to the microphone pair locations, are then fused to estimate the Cartesian coordinate of the speaer. This second stage process can be done by some simple, effective algorithms such as [5], [8] []. This wor was supported in part by a research grant awarded by the Australian Research Council. Fig.. Microphone pair Speaer source Microphone pair TDOA microphone array system. The TDOA single-speaer localization approach mentioned above is vulnerable to reverberation, a problem that is quite common in room environments. The rich multipaths resulting from reverberation can lead to anomalous GCC TDOA estimates. Under such circumstances it is suggested to employ the phase transform (PHAT) GCC method for reducing TDOA estimation error [7]. To further suppress the effect of anomalous TDOA measurements, blind channel identification methods have been introduced to replace the role of GCC [5], [], []. An alternative that has emerged more recently is to apply the Bayesian object tracing framewor [3], [4] in the source localization stage. In this approach the possibility of GCC TDOA outliers is incorporated in the problem formulation, thereby maing the resultant speaer location estimator less prone to reverberation (compared to localization methods that do not assume the presence of TDOA outliers). Another salient feature of the Bayesian approach is that the speaer location is sequentially traced with respect to (w.r.t.) time, by following a Marov speaer motion model. In other words, the Bayesian approach exploits the correlation of speaer motions w.r.t. time, which can help improve localization accuracy. The Bayesian tracing approach has been numerically demonstrated [3], [4] to be robust against the effects of reverberation. Moreover, performance comparisons of the Bayesian approach with the blind channel identification based localization approach have been examined in [4]. The objective of this paper is to deal with TDOA based localization with unnown time-varying number of speaers and with reverberation, by applying a Bayesian tracing framewor based on the random finite set (RFS) formulation [5] [8]. It is interesting to mention that conceptually, the Bayesian tracing idea can be applied to the blind channel identification methods to further improve localization accuracy. However, no such wor has yet appeared in the speaer localization literature.

2 This multi-speaer localization problem presents a challenge in signal processing, and very recently some attempts [9], [] have been made to tacle that problem for nown number of speaers. Our attempt for locating unnown time-varying numbers of speaers is based on the multi-object tracing approach, a generalization of the single-object tracing approach. There is a variety of techniques for multi-object tracing; see the reviews in [6], [], []. The RFS approach employed here is a recently emerged framewor that has been found to be promising for multi-object tracing [5] [8]. RFS is a rigorous mathematical discipline for dealing with random spatial patterns [3] [6] that has long been used by statisticians in many diverse applications including agriculture, geology, and epidemiology; see [5] and the references therein for further details. In essence, an RFS is a finite collection of elements where not only each RFS constituent element is random, but the number of elements is also random. The RFS approach to multi-object tracing is elegant in that the multiple object states and the number of objects are integrated to a single RFS. More importantly, RFS provides a solid foundation for Bayesian multi-object tracing, that is not found in traditional multi-object tracing approaches. For further discussions of the differences between the RFS and traditional approaches, please read [5], [6], [7]. An exposition of RFS theory is rather involved particularly when it comes to the constructions of probability densities for RFSs; see [5], [8], [5] for the details. Fortunately, for most engineering applications it suffices to now how to apply several ey concepts and results, which in our opinion are presently not well publicized for the generally nowledgeable readers and therefore will be demonstrated in this paper. The summary of this paper, together with the organization, are as follows. After a bacground review in Section II, in Section III we propose an RFS model for the multi-speaer tracing application. Section III also lays several reasonable assumptions for the application, that turn out to greatly facilitate the RFS tracing implementation. Those assumptions include i) At each time instant, at most one speaer source can be born. ii) The number of simultaneously active speaer sources is small. The assumption in ii) is particularly true in applications such as video-conferencing, in which the most frequently encountered events are no voice activity, one speaer voice activity, and one speaer interrupting another. Section IV describes the Bayes RFS filter (or tracer) and its implementation using the sequential Monte Carlo (SMC) method in [8]. The Bayes RFS SMC filter is nown to be computationally expensive for large number of objects [8], []. For those cases it would be appropriate to consider computationally efficient approximations such as the probability hypothesis density method [6], [8], [8]. Fortunately for small number of objects it is still computationally affordable to employ the Bayes RFS SMC filter (see, for example, []), and it appears that the multi-speaer tracing problem considered here falls into such case. Section V deals with trac association in the RFS framewor. The presently available trac association resolutions wor by combining the RFS module with some other tracing modules [9], [3], but in this wor we exploit the at-most-one-birth assumption to come up with a simple trac association method. The idea is to augment the speaer state vector by a discrete variable that records trac association information. It is interesting to point out that such an idea has been alluded to in [3]. It is further shown that by using the proposed trac association method, the respective RFS state estimation process can be greatly simplified. To demonstrate the performance of the Bayes RFS SMC filter, in Section VI we provide two sets of simulation results based on relatively realistic room simulations. In particular, in one of the simulation examples we tested the robustness of the Bayes RFS SMC filter against the effects of model mismatch. This wor is a more complete version of the conference paper [3]. In particular, [3] did not consider the trac association method and the respective RFS state estimation method in Section V. II. BACKGROUND This section provides a brief review on TDOA speaer tracing, by considering the simple case of single speaer activity and no reverberation. We should point out that the method reviewed in the following subsections is a simplified version of the TDOA single-speaer location tracing method in [3], [4], in which the reverberation problem was also addressed. In the first subsection, some aspects regarding TDOA measurements are described. Then, in the second subsection we consider some basic concepts of Bayesian tracing. A. TDOA Measurement In the scenario of a single speaer without reverberation, the received signals at a microphone pair can be modeled as [7] y (t) = s(t) + ν (t), y (t) = s(t τ) + ν (t) () where s(t) is the signal due to the source, ν i (t), i =, are bacground noise, and τ is the TDOA between the st and nd microphones. Let α R denote the (x, y) position vector of the speaer source. The TDOA τ is dependent on α through the following nonlinear relation: τ = c ( α u α u ) () where u and u are the positions of the two microphones,. is the -norm, and c is the speed of sound (note that extension to the 3-dimensional coordinate is straightforward). The TDOA can be measured by a generalized cross-correlation (GCC) estimator [6], given as follows: R gcc (τ) = ˆτ = arg R gcc(τ) (3) τ [ τ max,τ max] Φ(ω)S y,y (ω)e jωτ dω (4)

3 Here, R gcc (τ) is called the GCC function, τ max = u u /c is the maximum admissible TDOA value, S y,y (ω) is the cross spectral density of y (t) and y (t), and Φ(ω) is a weighting function; see [6], [7] for details regarding the choices of Φ(ω). A popular choice of Φ(ω) is the phase transform (PHAT), where Φ(ω) = / S y,y (ω). In this wor we will employ the PHAT. In practice the speaer position α can change over time, in which case it is appropriate to estimate τ from a relatively short time frame so that α is (almost) static over each frame. Thus, we replace S y,y (ω) in (4) by a short-time estimate Ŝ y,y (ω; ) = Y (ω; )Y (ω; ) (5) Y i (ω; ) = T ( )T y i (t)e jωt dt, i =, (6) where T is the time frame length, and is the time frame index. B. Sequential State Estimation We consider a standard state space model [3], [4] for the single-speaer TDOA problem mentioned above. We use the notation α to represent the speaer location at the th time frame. By defining a state vector x = [ α T, φt ]T R n where n is the state dimension and φ contains some inematic variables for the speaer motion (e.g., velocity), we model x by a dynamic process: x = Ax + Bw (7) where A and B are some pre-specified matrices, and w is a time-uncorrelated random Gaussian vector with zero mean and covariance I. In speaer location tracing, it is popular to employ the Langevin model [3], [4] in which φ consists of the (x, y) velocities. The state space equations for the Langevin model are given by α = α + Tφ (8) φ = e βt φ + ν e βt w (9) Here, β and ν are model parameters called the rate constant and the steady-state root-mean-square velocity, respectively. Next, we consider the TDOA measurements. We denote by the TDOA measured from the qth microphone pair at time frame. The measured TDOAs are modeled by: = τ q(cx ) + v [q], q =,...,Q. () Here, C = [ I ] so that Cx = α, τ q (α ) = c ( α u,q α u,q ) () is the true TDOA value, {u,q,u,q } are the position vectors of the qth microphone pair, and v [q] is time-uncorrelated noise. We assume that v [q] is independent of v[p] for any q p, and that each v [q] follows a Gaussian distribution with zero mean and variance σv. Our goal is to estimate x over time. In the sequential Bayesian framewor, we assume nowledge of the probability density function (p.d.f.) of x conditioned on x, which we denote by and the p.d.f. of f(x x ), () given x, which we denote by g q ( x ). (3) Eqs. () and (3) are called the state transition density and the lielihood function, respectively. By letting N(.; µ,p) denotes the Gaussian density function with mean µ and covariance P, the expressions () and (3) are f(x x ) = N(x ;Ax,BB T ) and g q ( x ) = N( ; τ q(cx ), σv), respectively. Let z [:Q] : define the sequence containing i for i =,..., and for q =,...,Q. The Bayesian approach considers finding the posterior p.d.f. p(x z [:Q] : ), (4) which then allows us to estimate x using some optimal criterion such as the expected a posteriori (EAP). The posterior p.d.f. obeys the following recursion [33], [34]: p(x z [:Q] : ) = f(x x )p(x z [:Q] : )dx (5) Q p(x z [:Q] : ) = q= g q( x )p(x z [:Q] : ) Q q= g q( x )p(x z [:Q] : )dx (6) To solve (5) and (6) exactly is not easy due to the nonlinearity of τ q (.). Presently, in TDOA single-speaer tracing, a promising approach to approximating (5) and (6) is the sequential Monte Carlo methods [33], [34]; see [3], [4] for the details. Our proposed RFS method follows the same paradigm as the above Bayesian tracing framewor. This is illustrated in the following sections. Moreover, we should point out that in [3], [4], the above Bayesian framewor has been extended to handle single-speaer tracing in the presence of reverberation. In those wors, the GCC method was slightly modified to cater for the possibility of false TDOA peas caused by reverberation. Since this wor will employ the same modified GCC method (which will be described in the next section), the proposed method may be considered as a multispeaer generalization of [3], [4]. III. RFS FORMULATION FOR MULTI-SPEAKER LOCALIZATION This section describes our problem formulation for TDOA multi-speaer tracing in the presence of reverberation, using the random finite set (RFS) framewor. In the first subsection, we outline the characteristics of our multi-speaer problem. An RFS formulation for the problem is then presented in the second subsection.

4 A. The Multi-Speaer Problem The multi-speaer scenario considered here has the following characteristics: i) each speaer follows the state space motion model described in Section II-B, but his/her own voice activity interval is unnown to the system; and ii) each speaer s voice undergoes a multipath propagation, due to room reverberation. In the proposed RFS treatment, we represent the state vectors of the speaers by a single finite set, given by: X = {x,,...,x N,}, (7) where N = X (. stands for the cardinality) is the number of active speaers at time, and each x i, represents a distinct speaer state vector. We assume that X N max for some given N max, and that X is unnown. In the next subsection we will develop a statistical finite set model for X, which describes not only the state space motion mechanism, but also the appearance and disappearance events for each speaer. GCC function 3 True TDOA of Speaer True TDOA of Speaer TDOA (in sec.) Fig.. GCC function response in the presence of multi-speaer activity and reverberation. The response was obtained by using recorded real speech, and by simulating a room environment where reverberation is present. In the presence of multi-speaer, reverberation-induced multipath signal propagations, the GCC function in (4) is composed of the cross-correlations of the various paths. Hence, some of the peas of the GCC function are expected to be contributed by the direct path components of the speaer sources. This can be seen from the illustration in Fig., which shows a GCC function response in the presence of multi-speaer multipath propagation. We follow the TDOA extraction scheme in [3], [4] where multiple TDOAs are measured from one GCC function by picing more than one pea (or locally maximum point) in (4). By collecting those TDOAs to form a finite set, we have the following finite-setvalued measurement at time for the qth microphone pair: Z [q] = {,,...,z[q] M [q], x 3 }, (8) where M [q] = Z [q] is the number of measured TDOAs. We are now faced with the following problems: i) Given an x i, in X we expect that one of the measured TDOAs in Z [q] is generated by x i,, but we do not now which element in Z [q] is due to x i,. ii) It may occasionally turn out that x i, does not contribute a measurement to Z [q]. This measurement miss situation can occur when one speaer crosscorrelation response mass that of another, and/or when the speech signal powers are too wea in certain time frames due to the nonstationarity of speech signals. iii) Z [q] may contain false TDOAs; i.e., TDOAs that are not generated by the direct paths. Such an effect can also be seen in Fig.. B. The RFS Formulation We consider an RFS formulation that models the multispeaer, multi-measurement problems described in the last subsection. The multi-speaer finite set X is modeled by X = B (b ) S (x i,,w i, ) (9),..., X where B (b ) contains state vectors of speaers born at time, S (x i,,w i, ) is a finite set associated with the previous speaer state x i,, and the vectors w i, and b are random variables independent of one other. For S, we have the following hypotheses: {, Hdeath S (x i,,w i, ) = {Ax i, + Bw i, }, Hdeath () where H death and H death are respectively the death and nodeath hypotheses. Note that for the no-death hypothesis, the state space process is exactly the same as that of the simple single-speaer case in (7). The hypothesis H death occurs with probability P death. For the birth process, we assume that at most speaer is born at a time. If X = N max then we have B =. Otherwise, the following hypotheses apply: {, Hbirth B (b ) = () {b }, H birth where H birth and H birth are respectively the birth and no-birth hypotheses, and b is an initial state vector under the birth hypothesis. We denote the probability of H birth by P birth. Moreover, b is assumed to follow an initial state distribution in which the (x, y) position is uniformly distributed within the room enclosure and the other inematic variables are zero. is inde- is modeled by C[q] () For the measurement model, we assume that Z [q] pendent of Z [p] for any q p. Each Z [q] where C [q] by Z [q] =,..., X T [q] (x i,, v [q] i, ) is the finite set of false TDOAs, and T [q] T [q] (x i,, v [q] i, ) = {, { Hmiss τ q (Cx i, ) + v [q] i, is given }, Hmiss (3) with v [q] i, N(, σ v). Here, τ q (Cx ) is given in (), and H miss and H miss are respectively the detection and miss hypotheses. The hypothesis H miss happens with probability P miss. For the false TDOAs, we follow the standard assumption in [3], [4] that each c [q] C[q] independently follows a uniform distribution over the admissible TDOA interval [ τ max, τ max ], where τ max = u,q u,q /c (For simplicity the inter-sensor distance u,q u,q for every microphone

5 pair is assumed to be the same). In addition, the number of false TDOAs C [q] is assumed to follow a Poisson distribution with an average rate of λ c. Some remars are now in order: Remar : The above described RFS model is applicable to any N max ; that is, the maximum number of simultaneously active speaers. However, the performance of the GCC TDOA measurement method in practice incurs a limitation on the choice of N max. GCC benefits from its simplicity, but it is not a super resolution method in the multiple TDOA estimation context. When there are many speaers or when the TDOAs of two speaers are close, GCC may only be able to obtain a few true TDOAs that are associated with the dominant sources. Fortunately, for speech applications it is generally true that the number of simultaneously active speaers is small, such as (some justification for this has been presented in Section I). Hence, in this TDOA speaer localization application, it is pertinent to focus on a small N max. Remar : The probability of birth P birth and the probability of death P death are not nown in reality. In practice it is reasonable to mae a guess on P birth and P death, and such parameter adjustments generally lead to some tradeoffs on the performance of multi-speaer tracing (presented in the next sections). Increasing P birth is expected to improve the chance of identifying a newly born speaer source. Liewise, to quicly identify speaer source death is advised to increase P death. However, increasing P birth and/or P death may also increase the chance of over-estimation and under-estimation on the number of speaers, especially in the presence of false measurements. In other words, the tuning of P birth and P death is a tradeoff between sensitivity and robustness. Remar 3: Lie P birth and P death, in practice the probability of miss P miss is decided by some rough guess. Increasing P miss is expected to improve the robustness against the measurement miss situations. However, increasing P miss may also reduce the accuracy of speaer state estimation. IV. BAYESIAN RFS FILTER With the above RFS problem formulation, we can develop a Bayesian framewor for estimating X ; i.e., sequentially estimating both the multi-speaer locations and the number of active speaers. In the first subsection, we examine some probabilistic results that are essential to the Bayesian RFS framewor. Then, the second subsection proposes an implementation for Bayesian RFS estimation using the sequential Monte Carlo (SMC) technique, also nown as the particle filter. A. Bayes Recursion for RFSs The RFS theory provides two important tools for the multispeaer tracing application. First, we can construct p.d.f.s for the RFS X and Z [q] according to the model outlined in the previous section. In particular, we can determine a multispeaer RFS state transition density, denoted by f(x X ), (4) and RFS lielihood functions, denoted by g q (Z [q] X ) (5) for q =,...,Q. To construct these p.d.f.s, some involved mathematical concepts are required and the details are beyond the scope of this application paper. Readers are referred to [5], [8], [5] for complete descriptions of the RFS p.d.f. concepts. From an application viewpoint, we are more interested in the results, particularly the expressions for (4) and (5). The derivations of (4) and (5) are given in the Appendix. Second, many ideas in RFS Bayes estimation are essentially the same as those of the standard Bayes framewor (c.f., Section II-B). To explain this, let Z [:Q] : define a sequence consisting of the finite sets Z [q] i for all i =,..., and q =,...,Q. In an RFS Bayesian framewor, we consider determining the posterior p.d.f. p(x Z [:Q] : ) thereby estimating X over time. Moreover, p(x Z [:Q] : ) has a recursive relation reminiscent of the prediction and update formulae in (5) and (6), given as follows: p(x Z [:Q] : ) = F(R n ) f(x X )p(x Z [:Q] : )µ(dx ) (6) Q p(x Z [:Q] : ) = q= g q(z [q] X )p(x Z [:Q] : ) Q q= g q(z [q] X )p(x Z [:Q] : )µ(dx ). F(R n ) (7) where F(R n ) is the class of all finite subsets of R n, and µ is a measure on F(R n ); see [8], [35], [36] for the details. The next section considers the implementation of (6) and (7) using SMC. B. Sequential Monte Carlo Implementation The Bayes recursion in (6) and (7) can be computed, in an approximate manner, by applying an RFS SMC method [8], [37]. In the single-speaer scenario, SMC has been shown [3], [4] to be effective in handling the nonlinearity of the TDOA function. In this multi-speaer extension where the p.d.f.s. exhibit even more complicated structures (c.f., the Appendix), the SMC implementations are particularly favorable. The implementation employed here is the RFS bootstrap SMC method, which is a special case of the generic RFS SMC method in [8] but is particularly easy to use (note that the RFS bootstrap SMC method here is not related to the method in [37]). The RFS boostrap SMC method is briefly described as follows. We use a random measure {X (i), w(i) }L to approximate the posterior p.d.f.: Here, X (i) associated with X (i) L p(x Z [:Q] : ) w (i) δ X (i) (X ). (8) is the ith (finite set) particle, w (i) is the weight, L is the number of particles applied, and

6 δ (i) X is a set-valued version of the standard Dirac delta function. As an approximation to probability densities, the weights for all i and L w(i) =. The particles {X (i) }L are randomly drawn conditioned on the time random measure {X (i), w(i) }L. Specifically, for each i =,...,L we generate have the properties that w (i) X (i) f(. X (i) ). (9) By the notion of importance sampling [33], [34] and by applying the time counterpart of (8) to the Bayes recursion in (6) and (7), one can obtain p(x Z [:Q] : ) L Q q= g q(z [q] L j= Q q= g q(z [q] (i) X )w(i) (j) X )w(j) δ (i) X (X ). (3) Table I summarizes the RFS bootstrap SMC filter. It should be noted that in Table I there is an additional step called resampling. This step is used to reduce the degeneracy problem commonly encountered in SMC approximations; see [33], [34] for more details. The essential ingredients for the bootstrap SMC filter are the particle generation at Step of Table I, and the expressions for g q (Z X ). In Table II we show the particle generation algorithm. The general expression for g q (Z X ) is shown in the Appendix. Some useful equations of g q (Z X ) for N max = are shown at the top of the page. TABLE I RFS BOOTSTRAP SMC FILTER FOR MULTI-SPEAKER TRACKING. RFS Bootstrap SMC Filter Given a particle size L. for =,,... Step. Sampling: For i =,..., L, generate X (i) f(. X (i) ) and compute end w (i) = Q q= g q(z [q] (i) X )w(i). Then, apply normalization w (i) all i. Step. Resampling: Apply a resampling algorithm [33],[34] on {X (i), w(i) }L to obtain a resampled set {, w(i) }L (i) := { X Then, update {X (i) /(È L := w(i) l= w(l) ) for (i) X }L., w(i), w(i) }L. We should point out that Eqs. (3), (3), (33) represent the lielihood functions for no speaer, one speaer, and two speaers, respectively. Also, recall that the parameter λ c in (3) to (33) represents the average number of false TDOA measurement. Some further remars are now in order: A set-valued Dirac delta function δ Y (X) is a function such that given every A F(R n ), we have Ê A δ Y(X)µ(dX) = A (Y). Here, A (Y) is an indicator function where A (Y) = if Y A, and A (Y) = otherwise. TABLE II PARTICLE GENERATION. Particle Generation Algorithm for X f(. X ) Set X =. Step. Source Death and Survival: for each x X Draw a random number u uniformly distributed over [, ). if u > P death draw a random vector w according to the state space model assumed; compute x := Ax + Bw ; and set X := X {x }. end end Step. Source Birth if X (i) < Nmax Draw a random number u uniformly distributed over [, ). if u P birth draw an initial state b according to the initial state distribution assumed; and set X := X {b }. end end Remar 4: An asymptotic convergence property for the RFS SMC filter, such as the above described bootstrap filter has been considered in [8]. Specifically, it has been proven that for sufficiently large L, the mean square approximation error of the RFS SMC filter is inversely proportional to L α for some constant < α. This implies that the RFS bootstrap SMC filter is an accurate approximation for large L. Remar 5: If we choose N max =, P death =, and P birth =, the RFS SMC filter reduces to a form very similar to the single-speaer SMC filter in [3], [4]. Remar 6: The computational complexity of the RFS bootstrap SMC filter is linearly dependent on the particle size L. Moreover, for each particle, the complexity depends on the evaluation of the lielihood function g q (Z X ). It can be seen in the Appendix that the computations of g q (Z X ) are exponential in X. In this application where X are small (see the argument in Section I and Remar ), this computational issue is insignificant. V. REFINEMENT OF THE BAYES RFS FILTER In this section we present some additional ideas that can further enhance the effectiveness of the proposed RFS multispeaer tracing method. The first subsection describes the trac association problem arising in the RFS framewor. A simple method, called trac labeling, is proposed to handle that problem. Then, in the second subsection, we propose a state estimation scheme that taes advantage of trac label information to simplify the estimation process.

7 g q (Z [q] ) = e λc ( λc g q (Z [q] {x }) = g q (Z [q] ) [q] Z τ max { g q (Z [q] {x,,x, }) = g q (Z [q] ) (6) ) ( ) P miss + ( P miss ) τmax λ c g q ( x ) (7) ( P miss ), Z[q] P miss + ( P miss ) Z[q] ( τmax λ c Z[q] ) gq ( x,)g q ( x,) ( ) τmax λ c g q ( } x i,) (8) A. Trac Association using Trac Labeling A problem with the RFS state formulation described in the previous sections is that it gives no information on the trac association between X and X. That is, given an element x X, we do not now which element in X is originated from x. It follows that a Bayesian RFS filter based on this model will not provide such information. For the general RFS multi-object tracing scenario in which target birth can be quite complex 3, handling trac association is non-trivial; see, for example, [9], [3]. In this multispeaer tracing problem where at most one speaer source is allowed to be born at one time, trac association can be quite easily handled by considering the following idea. To avoid notational inconsistency, let us re-define the state vector used in Sections II and III to be ξ = [ α T, φt ]T R n, in place of x. We define a new state vector x = [ ξ T, γ ] T (3) where we augment the state vector by a variable γ to indicate the trac identity of the speaer state. The variable γ is set to the birth time of the speaer source. Since no two speaers share the same birth time, γ will provide adequate information for resolving trac association when x in (34) is used in the RFS framewor. We call γ a trac label of a speaer source. Moreover, we refer to a state vector as the trac-l speaer state if its trac label γ taes the value l. To incorporate trac labels into the previously developed RFS framewor, we only need minor modifications on the state space equations. For the birth hypothesis process in (), the birth state vector is modified as b = [ ξ T init,, ]T (3) where ξ init, R n is the random initial state vector described in Section III-B. For the survival process in (), we have [ ] [ A B x i, = x i, + w ] i, (33) 3 In a general RFS multi-object tracing framewor, multiple targets can be born at one time. In addition, one target can split to form two or more. in which the trac label at time is directly carried forward to time. Trac labeling not only helps identify speaer tracs, it also simplifies the state estimation process as shown in the following subsection. B. State Estimation Incorporating Trac Labels Our algorithm development in the last section has focused on the particle posterior density approximation: L p(x Z [:Q] : ) w (i) δ X (i) (X ) (34) for some weights w (i) (i) and for some (set-valued) particles X. This subsection describes our proposed method for estimating X from (37), in which the trac labeling idea in the previous subsection is exploited to simplify the estimation process. In the current RFS framewor (i.e., without trac labeling), a number of Bayesian estimation criteria have been proposed [5], [7]. Here we are interested in the intensity measure [6], [8]. 4 The following quantity N (S) E { X S [:Q]} Z : (35) = X S p(x Z [:Q] : )µ(dx ) (36) defined for any set S R n, is called an intensity measure of X conditioned on Z [:Q] :. The intensity measure is the first-order moment of X. Physically, N (S) describes the expected number of state vectors lying in S; e.g., N (R n ) is the expected total number of speaer sources at time, given the measurements Z [:Q] :. From (37), N (S) can be approximated by N (S) L w (i) (i) X S (37) Roughly speaing, an intensity-measure-based state estimation method [8], [38], [39] consists of two steps: i) Obtain an 4 The density of the intensity measure is called the probability hypothesis density (PHD). It is worth mentioning that PHD is an important concept in RFS multi-object tracing; see [6], [8] for the details.

8 estimate of the number of speaers ˆN = N (R n ) where. is the rounding operation. ii) Determine a number of sets S i, for i =,..., ˆN, such that the intensity N (S i, ) shows good response for each i whilst S i, S j, = for i j and ˆN N (S i, ) = N (R n ). iii) For each i, determine the center of S i,, denoted by ˆx i,. The centers {ˆx i, } ˆN are then taen as the state estimates. The challenge of this approach lies in Step ii), where some clustering algorithm is usually used to numerically determine those sets. Since clustering is a nonlinear nonconvex optimization problem, poor data fitting could occur. The state estimation process can become simpler when trac label information is available. Recall that a state vector with trac labeling is in the form of x = [ ξ T, γ ] T R n Z. Hence, we can define the intensity measure for the trac-l speaer state: N (A; l) N (A {l}) (38) = {ξ } A p(x Z [:Q] : )µ(dx ) [ ξ T,γ ] T X γ =l (39) for any A R n, and for l. This trac-label-dependent intensity measure allows us to perform state estimation on a speaer-by-speaer basis. First, we note that Interpretation The quantity N (R n ; l) is the expected number of times that the trac-l speaer source is present at time, given the measurements Z [:Q] :. In other words, we can detect the trac-l source by testing whether N (R n ; l) is above certain threshold, say.5. Interpretation The vector ˆξ (l) = N (R n ; l) R n ξ N (dξ ; l) (4) is the expected state vector of the trac-l source at time, conditioned on the hypothesis that the trac-l speaer source is present at time, and conditioned on Z [:Q] :. It is interesting to note that (43) is reminiscent of the expected a posteriori (EAP) estimate in the single-object tracing scenario. Based on Interpretations and, we propose an RFS state estimation procedure in Table III. VI. SIMULATION RESULTS Two room simulation examples are used to test the tracing performance of the proposed multi-speaer RFS SMC filter. A. Example Fig. 3 illustrates the room settings for this example. The dimensions of the enclosure are 3m 3m.5m. We employ four microphone pairs, each of which has an intersensor spacing of.5m (which corresponds to τ max =.5ms). TABLE III STATE ESTIMATION ALGORITHM WITH TRACK LABELING. RFS state estimation algorithm Given a random measure {X (i), w(i) }L at time. Set ˆX =. Step. Extract the trac label set I = L [ ξ T,γ ] T X (i) {γ } Step. for each l I Obtain a particle approximation to N (R n ; l), denoted by ˆN (l), by summing the weights associated with the trac- l speaer source: end ˆN (l) = L w (i) [ ξ T,γ ] T X (i) {γ = l} If ˆN (l).5, compute a particle approximation to ξ (l), denoted by ˆξ (l), by maing a particle weighted average ˆξ (l) = ˆN (l) L w (i) [ ξ T,γ ] T X (i) and then update ˆX := ˆX [ ˆξ T (l),l ] T. {γ = l}ξ, Fig. 3 also shows the trajectories and birth/death times of the speaer sources. The speaer sources are all female. The acoustic image method [4] was used to simulate the room impulse responses. The reverberation time of the room impulse responses is about T 6 =.5s (see the literature such as [7], [4] for the definition of T 6 ). The speech-signal-to-noise ratio is about db. The time frame length for measuring TDOAs is 8ms, and the time frames are non-overlapping. Fig. 4 plots the measured TDOAs against the time frame index (we only displayed the measured TDOAs for two of the microphone pairs due to page limitation). We can see that the measured data is not very informative: For each time frame the largest GCC pea does not always represent one of the true TDOAs. Moreover, in the presence of two active speaers (from time to 3), the accuracy of the measured TDOAs tend to deteriorate due to mutual interference between the two speech signals. The parameter settings for the RFS SMC filter are as follows. The state space model is the Langevin model [cf., Eqs. (8) and (9)], with the model parameters β = s and ν = ms. The standard deviation of the TDOA measurement error is σ v = 5µs (which is also the sampling period). The other parameters are N max =, P birth =.5, P death =., P miss =.5, λ c = 3, and L = 5. Fig. 5 illustrates the tracing performance of the multi-speaer RFS

9 sensor pair sensor pair 3 =3 =45 speaer trajectory = speaer trajectory sensor pair 4 No. of active speaers True Multi-speaer RFS SMC (a) TDOA (in sec.) TDOA (in sec.) sensor pair Fig. 3. Geometric settings for the room simulation in Example..5 x x = True TDOA tracs ( speaers in total) GCC peas (or measured TDOAs) Largest GCC pea Fig. 4. Measured TDOAs at (a) sensor pair, and (b) sensor pair 3. SMC filter. The figures show that the RFS SMC filter is able to determine the two speaers locations and their respective activity intervals. Recall that in the legend of Fig. 5(b), the term trac label also represents the birth time of the estimated speaer trac. From Figs. 5(b) (c) we can see that the RFS SMC filter produces two tracs with trac labels and, but these two tracs actually correspond to the same speaer. This is because the RFS SMC filter can have estimation error on the birth time variables. For the readers interest, Figs. 5(b) (c) also show the performance of the existing single-speaer SMC filter [3], [4]. B. Example This example considers a situation where some model assumptions are not well satisfied. In other words, we are interested in testing the robustness of the proposed method against model mismatch. The room setting is shown in Fig. 6, where the room dimensions are 5m 3.5m.5m. The reverberation time is about T 6 =.35s, and the rest of the simulation parameters are the same as those in the previous example. In this example all the speaers are stationary, which violates the assumption that the speaers are moving following the Langevin motion model. Another model mismatch is with (a) (b) x (in m) y (in m) True ( speaers in total) Multi-speaer RFS SMC trac label= 3 trac label= trac label= Single-speaer SMC Fig. 5. Location tracing performance in Example. (a) RFS SMC filter estimates of the number of active speaers. (b) (c) Position estimates of the RFS SMC filter and the conventional single-speaer SMC filter. sensor pair 3 sensor pair sensor pair 4 sensor pair speaer active from = to 3 speaer 3 active from =5 to 75 sensor pair 6 speaer active from = to 59 sensor pair 8 Fig. 6. Geometric settings for the room simulation in Example. the measured TDOAs, which are illustrated in Fig. 7. In Fig. 7(b) we observe that from time to 6, there is a false TDOA that persistently appears with a value of about 3 second. One can also find a few other persistent false TDOA tracs in the figures. Those false TDOAs are caused by room reverberation. Since the speaer positions are fixed, so do those reverberation-induced false TDOAs. This phenomenon violates the assumption that false TDOAs are time uncorrelated. In this example we increase the number of microphone pairs to 8. The rationale is that the effect of model mismatch might be reduced when more sensors are available. Fig. 8 shows the localization results of the proposed multi-speaer RFS SMC filter. The figures indicate that inaccurate position estimates do sensor pair 5 sensor pair 7 (b) (c)

10 TDOA (in sec.).5 x True TDOA tracs (3 speaers in total) GCC peas (or measured TDOAs) Largest GCC pea (a) No. of active speaers True Multi-speaer RFS SMC (a).5 x 3 4 TDOA (in sec.) Fig. 7. Measured TDOAs at (a) sensor pair 3, and (b) sensor pair 7. happen sometimes; e.g., the trac from time 59 to 6 with trac label. But it is also seen from the figures that the proposed method provides reasonable tracing performance on average. C. Average Performance The above two examples show the localization performance for one trial. In this subsection we are interested in the localization performance averaged over many trials. To do so it is important to consider measures for comparing the differences between the true finite set state X and the estimated state set ˆX. First, it is useful to evaluate the probability of correct speaer number estimation: (b) P[ ˆX = X ] (4) Second, we are concerned with the location errors for the state vectors in ˆX. When the speaer number estimate is incorrect such that X ˆX, defining a localization error is a problem on its own; see [4]. Now, let us suppose that X = ˆX = n, and that X = {x,,...,x n, }, ˆX = {ˆx,,..., ˆx n, }. We consider the following multispeaer distance error: d(x, ˆX ) = min n Cx i, Cˆx ji, n j i {,...,n},,...,n j i j, i = (4) where C = [I ] is such that given a state x, Cx outputs the (x, y) position of that state. The idea of the minimization in (45) is to find a proper assignment between elements in X and ˆX. Moreover, we should mention that theoretically, (45) is a special case of the Wasserstein distance [4]. With (45), we can measure a conditional mean distance error, given by E ˆX {d(x, ˆX ) correct speaer number estimate} (43) The performance measures (44) and (46) were evaluated for Examples and with, trials. The results for Examples and are shown in Figs. 9 and, respectively. x (in m) y (in m) True (3 speaers in total) Multi-speaer RFS SMC trac label= trac label= trac label= 5 trac label= Fig. 8. Location tracing performance of the multi-speaer RFS SMC filter in Example. (a) Estimates of the number of active speaers. (b) (c) Position estimates. The figures illustrate that at the time instants where source birth/death occurs, the RFS method yields a transient behavior: At those birth/death time instants, the probability of correct speaer number estimation decreases and the conditional mean distance error increases. Then, the localization performance improves gradually with time. VII. CONCLUSION AND DISCUSSION Using the RFS theory and the SMC implementation technique, we have developed a TDOA multi-speaer location tracing algorithm that can handle unnown, time-varying number of active speaers. We have used simulations to show that the proposed algorithm can correctly determine not only the speaer locations, but also the voice activity interval for each speaer. The proposed RFS algorithm is suitable for many speech applications where the number of active speaers is usually small. As a technical challenge, it will be worthwhile to examine the case of large number of speaers. This direction leads to several open questions. First, our method (as well as the other methods in [3], [4], []) has been relying on the GCC TDOA measurement scheme, which has a modest resolution that generally cannot handle a large number of active speaers. To deal with the case of large number of speaers, it appears that we need to employ some more sophisticated microphone array structures and signal processing methods, such as those in the direction-of-estimation (DOA) estimation context []. Second, the RFS multi-speaer tracing principle is applicable to any number of speaers. However, (b) (c)

11 P[ ˆX = X ] Cond. Mean Distance Error (in m) spr sprs spr Fig. 9. Average location tracing performance of the multi-speaer RFS SMC filter in Example. P[ ˆX = X ] Cond. Mean Distance Error (in m) spr sprs spr sprs spr Fig.. Average location tracing performance of the multi-speaer RFS SMC filter in Example. the RFS Bayesian filter becomes more expensive to implement as the number of speaer increases. In those cases it might be appropriate to apply approximations such as the first-order moment method [6], [8]. Third, it will be interesting to extend the present method to deal with more complicated situations, such as when multiple source births are allowed at one time instant. APPENDIX The purpose of this section is to illustrate, in a concise manner, the derivations of the set-valued state transition density f(x X ) and the set-valued lielihood g q (Z X ). The principles of the derivations essentially follow those described in [5]. Readers are referred to [5] for further details. The following lemma will be frequently used: (a) (b) (a) (b) Lemma [5] Consider C = A B (44) where A and B are two independent RFSs. Then, the p.d.f. of C is p(c) = C C p(a = C)p(B = C C) (45) A. The State Transition Density Consider the finite set state structure in (9). By applying Lemma to (9), the state transition density is given by: f(x X ) = X X f b ( X X )f s (X X X ) (46) where f b (X X ) p(b (b ) = X X ) (47) is the p.d.f. for the birth states, and ( ) f s (X X ) p S (x i,,w i, ) = X X,..., X (48) is the transition density for the previous states. The expression for the birth state p.d.f. is as follows. For X = N max where no speaer birth is allowed, we have that {, X = f b (X X ) = (49), otherwise As for the case of X < N max, it can be shown from () that P birth, X = f b (X X ) = P birth β(x ), X = {x } (5), otherwise where β(x ) p(b = x ) is the initial state distribution. To construct the density f s (X X ), it is instructive to consider a one-speaer set-valued state transition density f s,i (X X ) p(s (x i,,w i, ) = X X ) = p(s (x i,,w i, ) = X x i, ) (5) where x i, is an element in X with x i, x j, for i j. From (), it is shown that f s,i (X X ) = P death, X = ( P death )f(x x i, ), X = {x }, otherwise (5) where f(x x ) is the single-speaer, vector-valued p.d.f. considered in Section II-B. Let X = {x,,...,x m, }, X = {x,,...,x n, } with m n. By applying Lemma to (5) repeatedly and by exploiting (55), it can be shown that f s (X X ) = P n m death ( P m death) m f(x j, x ij, ) i i m n j= (53)

12 where the summation term in the above equation means that n n n =... i... i m n i = i = i i i... i m n j= (58) where g q ( x ) = N( ; τ q(cx ), σv) is the singlespeaer lielihood function described in Section II-B. As for the false TDOA p.d.f., it is shown that ( ) m c q ({,,..., z[q] m, }) = P Z [q] (m) m! κ( i, ) (59) where P Z [q] (m) = P[ Z[q] = m] is the probability of the number of false TDOAs, and κ(z) is a uniform density with an interval [ τ max, τ max ]. Under the assumption that the number of false TDOAs is Poisson distributed with an average rate λ c, we have P [q] Z (m) = e λc λ m c /m! and Eq. (6) can be re-expressed as c q (Z [q] ) = e λc λ c κ( ) (6) Z[q] Substituting (63) into (58), the lielihood function can be simplified as g q (Z [q] X ) = c q (Z [q] ) Z [q] Z[q] g true,q ( [q] Z Z [q] X ) λ c κ( ) (6) REFERENCES [] H. Krim and M. Viberg, Two decades of array signal processing i m= i m i m... i vol. 3, no. 4, pp , 996. [] H. Wang and M. Kaveh, Coherent signal-subspace processing for the (54) research: the parametric approach, IEEE Signal Processing Mag., detection and estimation of angles of arrivals of multiple wide-band B. The Lielihood Functions The ideas behind deriving the lielihood functions are similar to those in the previous subsection. By applying Lemma sources, IEEE Trans. Acoust., Speech, and Signal Processing, vol. 33, no. 4, pp , 985. [3] B. Friedlander and A. Weiss, Direction finding for wide-band signals using an interpolated array, IEEE Trans. Signal Processing, vol. 4, no. 4, pp , 993. to the measurement model in (), the lielihood function for [4] F. Asano, S. Hayamizu, T. Yamada, and S. Naamura, Speech enhancement based on the subspace method, IEEE Trans. Speech Audio the TDOAs of the qth microphone pair is shown to be g q (Z [q] X ) = [q] g true,q ( Z X )c q (Z [q] Processing, vol. 8, no. 5, pp ,. [q] Z ) (55) [5] Y. Huang, J. Benesty, and G. Elo, Microphone arrays for video camera steering, in Acoustic Signal Processing for Telecommunications, S.L. Z [q] Z[q] Gay and J. Benesty, Eds., Kluwer Academic,. [6] C. Knapp and G. Carter, The generalized correlation method of estimation of time delay, IEEE Trans. Acoust., Speech, Signal Processing, where ( ) g true,q (Z [q] X vol. ASSP-4, no. 4, pp. 3 37, 976. ) p T [q] (x i,, v [q] i, ) = Z[q] X [7] T. Gustafsson, B. Rao, and M. Trivedi, Source localization in reverberant environments: modeling and statistical analysis, IEEE Trans. Speech,..., X Audio Processing, vol., no. 6, pp , 3. (56) [8] Y. Chan and K. Ho, A simple and efficient estimator for hyperbolic is the lielihood function of the true TDOAs, and location, IEEE Trans. Signal Processing, vol. 4, no. 8, pp , 994. c q (Z [q] ) p(c[q] = Z[q] ) (57) [9] M. Brandstein, J. Adcoc, and H. Silverman, A closed-form location estimator for use with room environment microphone arrays, IEEE Trans. Speech Audio Processing, vol. 5, no., pp. 45 5, 997. is the p.d.f. of the false TDOAs. It can be shown, in a way [] Y. Huang, J. Benesty, G. Elo, and R. Mersereau, Real-time passive source localization: an unbiased linear-correction least-squares similar to that for the state transition density in (55) to (56), that for approach, IEEE Trans. Speech Audio Processing, vol. 9, no. 8, pp ,. Z [q] = {z[q],,...,z[q] m, }, X [] J. Benesty, Adaptive eigenvalue decomposition algorithm for passive = {x,,...,x n, } acoustic source localization, J. Acoust. Soc. Amer., vol. 7, no., pp. with n m, the true TDOA lielihood function is given by ,. [] Y. Huang and J. Benesty, A class of frequency-domain adaptive g true,q (Z [q] X ) = P n m miss ( P miss) m approaches to blind multi-channel identification, IEEE Trans. Signal m g q ( j, x Processing, vol. 5, no., pp. 4, 3. [3] i j,). J. Vermaa and A. Blae, Nonlinear filtering for speaer tracing in noisy and reverberant environments, in Proc. IEEE Intl. Conf. Acoust., Speech, Signal Processing, May. [4] D. Ward, E. Lehmann, and R. Williamson, Particle filtering algorithms for tracing an acoustic source in a reverberant environment, IEEE Trans. Speech Audio Processing, vol., no. 6, pp , 3. [5] R. Mahler, An introduction to Multisource-Multitarget Statistics and Applications. Locheed Martin Technical Monograph,. [6], Multi-target Bayes filtering via first-order multi-target moments, IEEE Trans. Aerosp. Electron. Syst., vol. 39, no. 4, pp. 5 78, 3. [7] B.-N. Vo, S. Singh, and A. Doucet, Sequential Monte Carlo implementation of the PHD filter for multi-target tracing, in Proc. Int. Conf. Information Fusion, Cairns, Australia, pp , 3, also: [8], Sequential Monte Carlo methods for Bayesian multi-target filtering with random finite sets, to appear in IEEE Trans. Aerosp. Electron. Syst., July 5, also: [9] D. Zotin and R. Duraiswami, Accelerated speech source localization via hierarchical search of steer response power, IEEE Trans. Speech Audio Processing, vol., no. 5, pp , 4. [] I. Potamitis, H. Chen, and G. Tremoulis, Tracing of multiple moving speaers with multiple microphone arrays, IEEE Trans. Speech Audio Processing, vol., no. 5, pp. 5 59, 3. [] C. Hue, J.-P. L. Cadre, and P. Pérez, Sequential Monte Carlo methods for multiple target tracing and data fusion, IEEE Trans. Signal Processing, vol. 5, no., pp ,. [] M. Vihola, Random Sets for Multitarget Tracing and Data Association. Licentiate thesis, Dept. Inform. Tech. & Inst. Math., Tampere Univ. Technology, Finland, Aug. 4. [3] G. Mathéron, Random Sets and Integral Geometry. J. Wiley, 975. [4] D. Daley and D. Vere-Jones, An Introduction to the Theory of Point Processes. Springer-Verlag, 988. [5] D. Stoyan, D. Kendall, and J. Mece, Stochastic Geometry and its Applications. John Wiley & Sons, 995.