= = n b. p bs. n s = 2 = = = = = n b

Transcription

1 Computing Budget Allocation for Optimization of Sensor Processing Order in Sequential Multi-sensor Fusion Algorithms Lidija Trailović and Lucy Y. Pao Department of Electrical and Computer Engineering University of Colorado Boulder, CO USA and Abstract Because target tracking is a stochastic process, comparison of different tracking algorithms or parameter sets within one algorithm relies on time-consuming and computationally demanding simulations. In this paper we present a method to minimize simulation time, yet to achieve a desirable condence of the obtained results by applying ordinal optimization and computing budget allocation ideas and techniques. The developed method is applied to optimization of sensor processing order in sequential multi-sensor fusion algorithms. Introduction Target tracking problems involve processing position measurements (observed by a sensor) from a target of interest, and producing, at each time step, an estimate of the target's current state (position and velocity). Uncertainties in the target motion and in the measured values, usually modeled as additive random noise, lead to corresponding uncertainties in the target state. In many tracking problems there is an additional uncertainty regarding the origin of the received data, which may or may not include measurement(s) from the targets or random clutter (false alarms). This leads to the problem of data association or data correlation []. In this paper we consider a system of N s (not necessarily identical) sensors tracking T targets by performing a sequential multi-sensor joint probability data association (MSJPDA) algorithm [2, 8, 9, 0, 6]. A centralized processing architecture is assumed, where all sensor measurements are received by a central processor prior to data fusion. We are interested in optimizing tracking performance (here, root meaquare (RMS) position error) of the MSJPDA algorithm. The optimization consists of nding the order of processing sensor information that results in the smallest RMS position error. For the special case of two sensors tracking two This work was supported in part by an Ofce of Naval Research Young Investigator Award (Grant N ) and a National Science Foundation Early Faculty CAREER Award (Grant CMS ). The authors would also like to thank Professor Y. C. Ho of Harvard University for motivating this research. targets, this problem was addressed in [7]. In the general case of N s sensors tracking T targets, this is a difcult optimization problem: () the desigpace cae large (there are N s! possible sensor orders for a system with N s different sensors), (2) a computationally demanding simulation is needed to obtain a sample of the performance metric for a given sensor processing order, and (3) since target tracking is a stochastic process, many samples (and therefore many simulation runs) are required to achieve high condence levels in comparing, ranking, and selecting the best performing designs. Two complementary approaches have emerged to handle optimization problems such as the one considered in this paper. If the goal is to nd one or more among the best designs rather than accurate estimates of the performance metrics, ordinal comparisons (as opposed to cardinal evaluations) cae used to achieve faster convergence rates and signicantly reduce the computational effort. The idea of ordinal optimization [2,3,4] has been applied to a number of optimization problems [3, 6]. The second, complementary idea is to minimize the computational cost of achieving desired condence levels in ordinal comparisons and selection by intelligently allocating the computing budget among the designs [4, 5,6,7]. A distinguishing feature of the problem considered in this paper is that the performance metric is the RMS error (i.e., standard deviation or variance) rather than the mean of a random process. Ordinal ranking and selection procedures therefore must take into account statistical properties of the sampled process variance. In order statistics literature, the problem of selecting the population with the smallest variance was addressed in [], but the problem of computing budget allocation has not been addressed. The paper is organized as follows. The sequential implementation of the MSJPDA algorithm is briefly reviewed in Section 2, and the optimization problem is dened in Section 3. An algorithm for nding the best design with a desired (pre-dened) probability of correct selection or within a prescribed total computing budget

2 is presented in Section 4, as well as a discussion of convergence and performance properties of the algorithm. Results of applications of the developed algorithm to the problem of optimization of sensor processing order in the sequential MSJPDA target tracking algorithm are presented in Section 5, and concluding remarks are given in Section 6. 2 MSJPDA Tracking Algorithm The multi-sensor multi-target tracking problem is to track T targets using N s sensors in a cluttered environment, where measurements from the sensors are received by a central processor at discrete time intervals. Each measurement can originate from at most one target, and some of the measurements arise from targets while others arise from clutter. Some targets may not yield any measurements in a particular time interval or for a particular sensor, and measurement errors due to measurements from one sensor are assumed to be independent of those from another sensor. Assume the state x t (k),» t» T,ofeach target t is determined by known F t (k) and G t (k) as follows [, 2]: x t (k +)=F t (k)x t (k)+g t (k)w t (k) () where w t (k) are independent Gaussian random noise vectors with zero mean and known covariances Q t (k). With N s sensors, let M s;k,» s» N s,bethenumber of measurements from sensor s at time k. Assuming a pre-correlation gating process is used to eliminate some returns [], let m s;k be the number of validated measurements. The volume of the gate is choseuch that with probability P G;s the target originated measurements, if any, fall into the gate for sensor s. The target originated measurements are determined by z t s;`s(k) =H s (k)x t (k)+v t s(k); (2) where» t» T,» s» N s,and» `s» M s;k. The H s (k) matrices are known, and each v t s(k) is a zeromean Gaussian noise vector, with known covariances R s (k), and uncorrelated with other noise vectors. Given a target t and a sensor s, it is not known which measurement `s originates from the target. That is the problem of data association whereby it is necessary to determine which measurements originate from which targets. Further background on target tracking and data association methods cae found in [, 2,8,9,0,6,7]. This paper investigates the sequential implementation of the MSJPDA algorithm, where the measurements from each sensor are processed one sensor at a time [0, 7]: the measurements of the rst sensor are used to compute an intermediate state estimate and the corresponding covariance for each target; the measurements of each of the next sensors are then used sequentially to further update each state estimate and covariance until all the sensors are exhausted. 2. Simulation of the Sequential MSJPDA The MSJPDA simulator from [7] which was set up in Matlab [5] was expanded for N s sensors tracking T targets, with the matrices F t, G t, H s, Q t, and R s assumed known and time-invariant in () and (2). The initial state is assumed Gaussian with mean ^x(0j0) and covariance P(0j0). The state vectors x t (k) = [x _x y _y] T (k) represent the positions and velocities of the targets at time k, and the targets nominally movein a straight line with constant velocity, though corrupted by process noise. The process noise and measurement noise covariances Q t and R s, respectively, and the measurement matrices H s are as in [7]:»» Q t q 0 rs 0 = ; R 0 q s = ; (3) 0 r s H s =» ;» s» N s : The method developed in this paper is applicable to general R s, but we consider only diagonal R s because it allows us to more easily compare sensor qualities (better sensor has smaller r s ). The measurements corresponding to the sensor with covariance R are processed rst, and measurements from the sensor with covariance R Ns are processed last in the sequential MSJPDA. The position error ", which is the difference between the true target position and the target position estimate, is a normal (Gaussian) random variable with zero mean and standard deviation ff, i.e., " ο N[0;ff]. The standard deviation ff is the true RMS position error for agiveet of sensors and system parameters. For each time step k, the simulator computes a sample " k of the position error. The samples " k produce a discrete random variable ^". After a simulation run thatp generates n pts data points, the mean of ^" is μ" = n "k pts, and the standard deviation is ^ff = measured RMS error = s n pts X " 2 k : (4) where we have assumed that μ" ß 0. The standard deviation ^ff equals the measured RMS position error that the simulator produces as a sample of the performance metric for the given design and the set of parameters. If we perform imulation runs and update ^ff after each run,by the strong law of large numbers, ff = lim ^ff (5) n! with probability. This means that the measured standard deviation of the position error approaches the true RMS position error as the number n of simulation runs increases. 3 Notation and Problem Formulation In this section we establish the notation used and formulate the optimization problem.

3 S sensor set, S = fr ;r 2 ;:::;r Ns g; r s is the parameter in the sensor noise covariance dened in (3) i ith design, an ordered subset of S, i =[r i ;r i2 ;:::;r ini ], N i» N s desigpace, = f ;:::; Nd g; N d = j j is the size of the desigpace ff 2 i variance of the position error of the design i ; ff i is the true RMS position error of the design i n i ^ff i P CS number of simulation runs for the design i standard deviation estimate, i.e., RMS position error after n i simulation runs for the design i b selected best ranking design, ^ff b < ^ff i, for all i 6= b s second best ranking design, ^ff b < ^ff s < ^ff i, for all i 6= b, i 6= s P? N runs N max aposteriori probability of correct selection, P CS = P fff b <ff i ; for all i 6= b j ^ff i obtained after n i simulation runs for design i g desired probability of correct selection, i.e., desired condence level (predened, e.g., 90%) total numberofsimulation runs, N runs = P N d i= n i maximum allowed number of simulation runs, N runs» N max The optimization problem of nding the best design on the entire desigpace cae redened in two ways:. Select the best ranking design b s.t. P CS P?, while minimizing N runs. 2. Select the best ranking design b s.t. N runs» N max, while maximizing P CS. 4 Determining Smallest RMS Design In this sectiotatistical properties of the RMS position error performance metric are summarized and optimization algorithms are described, accompanied with a discussion of convergence and performance properties. 4. Statistics of RMS Position Error The position error " is a normal random variable with zero mean and standard deviation ff i. The random variable (n i ) ^f (6) ff i 2 has a χ 2 n i (Chi-squared) distribution with (n i ) degrees of freedom [8]. The probability of correct selection P CS cae computed as described in []. For an arbitrary numberofsimulation runs n i, the numerical computation of P CS is quite involved. We instead make use of the approximate probability of correct selection (AP CS) introduced in [4]. Consider two designs, i and j,with^ff i < ^ff j. Let us nd the aposteriori probability p ij that ff i <ff j or equivalently, p ij = P fff i <ff j j ^ff i ; ^ff j g (7) p ij = P ( ffj ff i 2 > The random variable x dened by x = ^ff2 j ff 2 j ff 2 i ^ff 2 i ) ^ff i; ^ff j : (8) (9) has a F ni ;n j distribution, with (n i ) degrees of freedom in the numerator and (n j ) degrees of freedom in the denominator [8]. Given n i,^ff i, n j,^ff j, the probability p ij that the design i gives a lower RMS position error than the design j cae found as the probability that x is greater than (^ff i =^ff j ) 2, p ij = CDF ff; ; ο 2 ij = where, to simplify notation, Z =ο 2 ij f(ff; ; x)dx (0) ο ij = ^ff j ; ff = ; = ; () ^ff i 2 2 and CDF( ) is the cumulative density function of the random variable x. The probability p ij in (0) cae computed easily, and the AP CS is then found as [4]: AP CS = YN d i=;i6=b Since AP CS is a lower bound for P CS [6]: p bi : (2) P CS AP CS; (3) AP CS can replace P CS in the optimization algorithm. 4.2 Uniform Computing Budget Allocation A simple approach to solve the optimization problem is to perform a sufciently large, equal number of simulation runs for all designs. This uniform computing budget allocation (UCBA) algorithm is UCBA algorithm Perform 2 simulation runs for each ofn d designs i ; Update ^ff i for all i; compute AP CS; While (AP CS < P? and N runs <N max ) Perform one simulation run for each ofn d designs; Update ^ff i for all i; computeap CS; End-while The algorithm starts with performing two simulation runs, because p ij in (0) is undened for n i ;n j < 2. The algorithm terminates when either the desired probability of correct selection P? or the maximum computing budget N max is reached.

4 p bs = 00 = 2 = 40 = 4 p bs = 00 5 = = = Figure : Probability p bs as a function of for several xed values of and ο bs =: Figure 2: Probability p bs as a function of for several xed values of and ο bs =: Optimized Computing Budget Allocation To minimize the total numberofsimulation runs, we propose an algorithm where only one simulation run is performed in each iteration, as opposed to simulation of all designs. The design m selected for simulation is the design for which increasing the number of runs n m should result in the largest AP CS. Let us dene p ij Λ = CDF n i ;n j +;ο 2 ij p i Λ j = CDF n i +;n j ;ο 2 ij EPCS u = 8 >< >: p bu Λ Q Nd i=;i6=b;i6=u p bi; Q Nd i=;i6=b p b Λ i; u 6= b u = b (4) (5) (6) where EPCS u is the estimated AP CS if the number of runs n u for the design u is incremented by. This denition of EPCS u is similar to the denition of EPCS in [6]. EPCS u cae easily computed for all designs and the desigelected for simulation is the one that maximizes EPCS. This steepest ascent approach is simpler than the one presented in [6] and the simplication is well justied by the fact that in our problem the cost of one simulation run is much higher than the cost of all other computations in one iteration. The algorithm with optimized computing budget allocation (OCBA) is OCBA algorithm Perform 2 simulation runs for each ofn d designs i ; Update ^ff i for all i; compute AP CS and p bs ; While (AP CS < P? and N runs <N max ) If (p bs <P init and <N init ) Perform one simulation run for each ofn d designs; Update ^ff i for all i; Else Find m such that EPCS m =max u (EPCS u ); Perform one simulation run for the design m ; Update ^ff m ; End-if Compute AP CS and p bs ; End-while The steepest ascent step is performed in the else" part of the iteration, while the if " part ensures convergence of the algorithm, as discussed below. The probability p bs is p ij in (0) for i = b and j = s, which isthe condence that design b is better than design s. 4.4 Convergence If the same number of simulation runs is allocated to each design, as in the UCBA algorithm, n = n 2 = = n Nd = n, itcanbeshown that [8] lim n! p bi =; for all i 6= b: (7) Therefore, AP CS in (2) for the UCBA algorithm converges to as the number ofsimulation runs increases, and AP CS P? in a nite number of simulation runs. In the OCBA algorithm, the number of simulation runs n i for different designs is not the same, and the proof of convergence requires a more detailed analysis. Figure illustrates how p bs depends on for several xed values of. For a xed, p bs increases monotonically with, but the asymptote for! is less than. Therefore, increasing the number of runs only for b does not guarantee convergence. Figure 2 shows how p bs depends on for several xed values of. Foragiven and small, p bs initially decreases with. Depending on the value of, p bs may continue to decrease with,oritmay reach a minimum after which it starts increasing with. This counterintuitive behavior of p bs is observed for ο bs close to one, i.e., when ^ff b and ^ff s do not differ alot. It is a result of comparing standard deviations (RMS position error) that leads to the F ni ;n j distribution. To avoid nonconvergence that would result from this behavior of p bs, our algorithm performs a certain number of iterations with UCBA until conditions are met for p bs to become monotonically increasing with. Since AP CS is dened by (2), it is only necessary to consider these conditions for p bs, because ο bi >ο bs and therefore p bi >p bs for all other designs i.

5 p s (,ξ) ξ = (.02) 2 APCS 0.8 OCBA 0.59 UCBA ξ = (.0) N init 8 N init 36 Figure 3: Probability p s(;ο) as a function of ns for ο = ο 2 bs =(:0) 2 and (:02) 2. Given ο bs >, and a large enough, a sufcient condition for convergence is that there exists N init such that p bs is monotonically increasing with for N init. To derive how to determine N init, let us nd how p bs behaves for large by dening p s ( ;ο)= To further simplify notation, let and from (0), we have Z p s ( ;ο)= lim! p bs = lim! lim! p bs: (8) ο = ο 2 bs (9) =ο The F -distribution PDF cae written as f(ff; ; x) = ( + ff x x ) ( + ff x)ff f(ff; ; x)dx: (20) (ff + ) (ff) () : (2) The rst factor in the denominator of (2) approaches e ff=x as! and the second denominator factor can be approximated with (x=ff) ff as!. Thus, we have ff (ff + ) ff f(ff; ; x)! (ff) () e ff=x x (ff+) (22) and hence p bs! (ff + ) (ff) () ff ffz =ο e ff=x x (ff+) dx: (23) The integral above cae solved as [9]: Z ff+ e ff=x ο x (ff+) dx = (ff +) ff ff; ff ; =ο ff where ff; ο ff is the incomplete Gamma function. For a xed ff, and with fl ff, and using the property (ff +)=ff (ff), p bs! (ff+) (ff) () ff ff ff ff+ ff (ff) ff ff; ο ff (24) Total number of simulation runs, N runs Figure 4: Comparison of AP CS as a function of N runs for the UCBA and OCBA algorithms. The number of designs is N d =0withο 9 bs =2orο bs =:08. Since (ff + ) () this nally yields! as! ; (25) ff p s ( ;ο)= ff; ο ff : (26) (ff) Figure 3 shows p s ( ;ο) in (26) versus for ο = ο 2 bs =(:0)2 and (:02) 2, and we can conclude that convergence of AP CS is guaranteed by selecting N init 36 ( 8) provided that the measured RMS position errors differ by at least % (2%). In the algorithm, UCBA iterations are performed until the number of simulation runs for each design exceeds N init. If the designs differ by more than the lower limiting value (ο bs > :0 or :02), the number of runs in the UCBA iteration can use a smaller N init. The initial iterations can also be terminated if p bs >P init. The value of P init = 68% is found as the value of p bs for = = 2 and ο bs =:334 where N init 2 is sufcient to guarantee convergence. With P init = 68% and N init = 36, the convergence conditions guarantee that AP CS P? in a nite number of simulation runs, provided that the designs differ by more than % (i.e., ο bs :0), but there is no guarantee about the rate of convergence. Generally, it is not of interest to labor over distinguishing between designs that are less than % different in performance. However, if necessary, (26) cae used to determine the minimum N init to use for ο bs less than : Performance Comparison To illustrate the improvement of the OCBA algorithm (Section 4.3) over the UCBA algorithm (Section 4.2) before actually applying it to sequential MSJPDA tracking, we raeveral tests in which we modeled performance metrics (^ff i ) for some hypothetical designs. Figure 4 presents a sample of our results when ff i = ^ff i for design i is held constant while n i is increased ioth the OCBA and UCBA algorithms. As

6 design n i RMS i = f0:0; 0:0g 6 0: = f0:0; 0:0; 0:0g 0:0655 N runs = 7, AP CS =90:04%, b = 2 = f0:0; 0:0; 0:0g 6 0: = f0:0; 0:0; 0:0; 0:0g 25 0:02622 N runs = 4, AP CS =90:2%, b = 2 = f0:0; 0:0; 0:0; 0:0g 73 0: = f0:0; 0:0; 0:0; 0:0; 0:0g 9 0:00893 N runs =64,AP CS =90:09%, b = 2 = f0:0; 0:0; 0:0; 0:0; 0:0g 84 0: = f0:0; 0:0; 0:0; 0:0; 0:0; 0:0g 6 0: N runs =200,AP CS =75:25%, b = 2 Table : Application of the OCBA algorithm: P? = 90%, N max =200,T = 3, and N s =2; 3; 4; 5; or 6. expected, the achieved AP CS increases as N runs increases. Figure 4 shows the improvement of the OCBA over the UCBA when there are N d = 0 competing designs with standard deviations between ff min = and ff max =2,and ff max ff min =(ο i;i+ ) N d =2; for» i» N d ; (27) that is, any two consecutive designs differ in performance metrics by 8% (ο i;i+ = :08). Note that the designs have ff's that are very close, which would make it difcult to distinguish the best one for any optimization algorithm. Improvement of the OCBA is signicant (e.g., from Figure 4, to achieve AP CS = 90%, the OCBA algorithm requires N runs = 000 while the UCBA requires N runs = 2800). 5 Sensor Processing Order Examples The algorithms from Sections 4.2 and 4.3 were applied to several examples of optimization of sensor processing order in the MSJPDA tracking simulator. In all simulations, the system noise was q =0:0 and the clutter density was = :4. This leads to 0:08 3:2 expected clutter measurements per target gate. Different designs correspond to different numbers of identical sensors or to different orders of non-identical sensors. Better sensors have a smaller r i in (3). Tables through 3 summarize a sampling of our results using the OCBA algorithm applied to the sequential MSJPDA. The choseest design is marked ioldface. Table presents a comparison of designs consisting of different numbers of identical sensors. As may be expected, performance improves as the number of sensors increases (N s = 5 has better performance than N s =4), design n i RMS i = f0:0; 0:g 5 0: = f0:; 0:0g 0 0:36 N runs =5,AP CS =95:2%, b = 2 = f0:02; 0:02g 8 0: = f0:0; 0:g 3 0: = f0:; 0:0g 2 0:3396 N runs =3,AP CS =95:66%, b = Table 2: Application of the OCBA algorithm: P? = 90%, N max = 00, T = 2, and N s =2. design n i RMS i = f0:0; 0:; 0:; :0g 42 0: = f0:0; 0:; :0; 0:g 55 0: = f0:0; :0; 0:; 0:g 39 0: = f:0; 0:; 0:; 0:0g 294 0: = f:0; 0:; 0:0; 0:g 43 0: = f:0; 0:0; 0:; 0:g 53 0: = f0:; :0; 0:0; 0:g 0 0: = f0:; 0:0; :0; 0:g 6 0: N runs =9,AP CS =93:6%, b = Table 3: Application of the OCBA algorithm: P? = 85%, N max = 000, T = 5, and N s =4. AP CS as a function of N runs for the listed designs is presented in Figure 5. but for some N s, improvement sometimes can not be veried in a predened N max (comparing designs when N s = 5 and N s =6,N max =200was exhausted before the AP CS achieved the desired P? = 90%). Table 2 compares orders of sensor processing of a two-sensor system where the overall quality of the two sensors is the same in all cases, that is, =r e = =r +=r 2,asin[7], and we arrived at the same conclusion, that processing the best sensor last yields the smallest RMS position error. Further, we came to the conclusion in only N runs = 5 (in [7] we used 200 Monte Carlo runs) and with AP CS > 95%. We also conrmed that two identical sensors outperform two very different quality sensors in any order. Results obtained when more diverse sensors are used in the sequential MSJPDA are shown in Table 3, and the increase of AP CS with N runs is shown in Figure 5 for the listed eight designs. The conclusion about processing the best available sensor last still holds. For these design choices, improvement over the UCBA is even more signicant: the UCBA achieved an AP CS = 85:7% in N runs = 472, that is, n i = 84, while the OCBA achieved an AP CS =85:03% in N runs = 897.

7 APCS θ = {0.0, 0., 0.,.0} θ 2 = {0.0, 0.,.0, 0.} θ 3 = {0.0,.0, 0., 0.} θ 4 = {.0, 0., 0., 0.0} θ 5 = {.0, 0., 0.0, 0.} θ 6 = {.0, 0.0, 0., 0.} θ 7 = {0.,.0, 0.0, 0.} θ 8 = {0., 0.0,.0, 0.} Number of simulation runs, N runs Figure 5: Achieved APCS as a function of N runs. Tracking T = 5 targets with clutter density = :4; b = 4 with AP CS =85:03% in N runs = Conclusion In this paper we analyzed sequential multi-sensor joint probabilistic data association algorithms, searching for the best order of processing sensor information when given non-identical sensors, so that the RMS position error performance metric is minimized. We also developed an algorithm that attempts to minimize simulation time, that is, achieves a predened condence level within a dened computatioudget, or returns the condence level of the achieved results if the computation budget must be exhausted. This optimization algorithm is constructed so that it addresses the statistical properties of the RMS position error performance metric by solving the problem of ranking and selection of standard deviations, as opposed to ranking and selection of process means which has been frequently addressed in literature. Results that we obtained with high condence levels and in reduced simulation times conrm the ndings from our previous research (for only 2 sensors) that processing the best available sensor the last in the sequential data association algorithm produces the smallest RMS position error on average. References [] Y. Bar-Shalom and T. Fortmann, Tracking and Data Association, Academic Press Inc., 988. [2] Y. Bar-Shalom and E. Tse, Tracking in a Cluttered Environment With Probabilistic Data Association," Automatica, (5): , Sept [3] C. G. Cassandras, L. Dai, and C. G. Panayiotou, Ordinal Optimization for a Class of Deterministic and Stochastic Discrete Resource Allocation Problems," IEEE Trans. Automatic Control, 43(7): , July 998. [4] C. H. Chen, A Lower Bound for the Correct Subset-Selection Probability and Its Application to Discrete-Event System Simulations," IEEE Trans. Automatic Control, 4(8): , Aug [5] C. H. Chen, H. C. Chen, and L. Dai, A Gradient Approach for Smartly Allocating Computing Budget for Discrete Event Simulation," Proc. Winter Simulation Conf., pp , Dec [6] C. H. Chen, S. D. Wu, and L. Dai, Ordinal Comparison of Heuristic Algorithms Using Stochastic Optimization," IEEE Trans. Robotics and Automation, 5(): 44 56, February 999. [7] H. C. Chen, C. H. Chen, and E. Yucesan, Computing Efforts Allocation for Ordinal Optimization and Discrete Event Simulation," IEEE Trans. Automatic Control, 45(5): , May [8] T. E. Fortmann, Y. Bar-Shalom, M. Scheffe, and S. Gelfand, Detection Thresholds for Tracking in Clutter A Connection Between Estimation and Signal Processing," IEEE Trans. Automatic Control, 30(3): , Mar [9] T. E. Fortmann, Y. Bar-Shalom, and M. Scheffe, Sonar Tracking of Multiple Targets Using Joint Probabilistic Data Association," IEEE J. Oceanic Engineering, 8(3): 73 83, July 983. [0] C. W. Frei and L. Y. Pao, Alternatives to Monte-Carlo Simulation Evaluations of Two Multisensor Fusion Algorithms," Automatica, 34(): 03 0, Jan [] S. S. Gupta and M. Sobel, On Selecting a Subset Containing the Population with the Smallest Variance," Biometrika, 49(3/4): , Dec [2] Y. C. Ho, An Explanation of Ordinal Optimization: Soft Computing for Hard Problems," Information Sciences, 3(3/4): 69 92, Feb [3] Y. C. Ho, C. G. Cassandras, C. H. Chen, and L. Dai, Ordinal Optimisation and Simulation," J. Operational Research Society, 5(4): , Apr [4] T. W. E. Lau and Y. C. Ho, Universal alignment probabilities and subset selection for ordinal optimization," J. Optimization Theory and Applications, 93(3): , June 997. [5] MATLAB: User's Guide, The Math Works, Inc., Natick, MA, 993. [6] L. Y. Pao, Centralized Multi-sensor Fusion Algorithms for Tracking Applications," Control Eng. Practice, 2(5): , Oct [7] L. Y. Pao and L. Trailović, Optimal Order of Processing Sensor Information in Sequential Multisensor Fusion Algorithms," IEEE Trans. Automatic Control, 45(8): , Aug [8] S. B. Vardeman, Statistics for Engineering Problem Solving, IEEE Press, 994. [9] S. Wolfram, The Mathematica Book, 3rd edition, Wolfram Media/Cambridge University Press 996.