PaCcET: An Objective Space Transformation to Shape the Pareto Front and Eliminate Concavity

Transcription

1 PaCcET: An Objective Space Transformation to Shape the Pareto Front and Eliminate Concavity Logan Yliniemi and Kagan Tumer Oregon State University Corvallis, Oregon, USA Abstract. In multi-objective problems, it is desirable to use a simple algorithm that can be interpreted by a system designer or decision maker. The simplest multi-objective algorithm is a linear combination of objectives. However, it is a well-accepted fact that in a multi-objective problem, a simple linear combination of all objectives cannot ever converge to solutions on the concave areas of the Pareto front: one of the points on the convex parts of the Pareto front or an extreme solution is always more desirable to an optimizer. This is a significant drawback of the linear combination. In this work we provide the Pareto Concavity Elimination Transformation (PaCcET), a novel objective space transformation that allows a linear sum (in this transformed objective space) to find solutions on concave areas of the Pareto front (in the original objective space). The transformation ensures that an optimizer will always value a non-dominated solution over any dominated solution, and can be used by any singleobjective optimizer. We demonstrate the efficacy of this method in two multi-objective benchmark problems with known concave Pareto fronts. Instead of the extreme points discovered by a simple linear sum, the transform-and-sum method produces a superior spread across the Pareto front, including concave areas, similar to those discovered by more complex multi-objective algorithms like SPEA2 and NSGA-II. 1 Introduction Cooperative multiagent systems research focuses on producing a set of autonomous agents to achieve a system-level goal [26]. Multiagent frameworks have been used to study complex, real-world systems like air traffic [2, 23], teams of satellites [9], and extra-planetary rover exploration [3]. In each case, the goal is to optimize a single, well-defined objective function. But, in many of these cases, the problems lend themselves more naturally to multiple objectives: for example, air travel should be as safe and as expedient as possible. Satellites may need to make observations for multiple separate institutions. Extraplanetary rovers should acquire multiple different types of scientific data. However, most research in multiagent systems does not take a multi-objective viewpoint: they

2 2 L. Yliniemi and K. Tumer typically seek to find a single usable solution, without considering the tradeoffs between potential alternatives that would increase one objective s value at the cost of another. These tradeoff solutions, which form the Pareto front, are a key solution concept in multi-objective problems [8]. Multi-objective optimization also has deep roots in the real world [17]. Problems such as the design of high-speed transport planes [18], the design of trusses [7] job shop scheduling [27], Urban planning [5], and greywater reuse [21] have all been addressed from a multi-objective standpoint. This has led to the development of many multi-objective techniques [12, 13, 17] that are capable of tackling multi-objective problems with a wide variety of properties. The type of techniques that can be used are limited by the shape of the Pareto front, which is often unknown before optimization. One very simple method is to use a linear combination of all objectives. However, in the case of a concave Pareto front, the linear combination will never be able to find the concave areas of the Pareto front [1, 4, 6, 8, 1, 14 16, 19, 2, 22, 24, 25], caused by the fact that a convex part (or an extreme point) will evaluate to be more desirable than the concave region. This is a major drawback for linear combinations. A second major drawback is that choosing weights even with a convex Pareto front can be difficult. The primary contribution of this work is to present the Pareto Concavity Elimination Transformation (PaCcET), a novel objective space transformation which shapes the Pareto front to be non-concave in the transformed space, regardless of its previous convexity. This allows a linear combination (in the transformed objective space) to find concave areas of the Pareto front (in the original objective space). This removes both major drawbacks of a linear combination, regardless of the shape of the original Pareto front. In turn, this allows a simple linear combination to be used instead of more computationally expensive multi-objective evolutionary algorithms, and produce similar results. The transform is formulated so that it does not rely on a choice of weights; is independent of the chosen optimizer (Ξ), allowing techniques as varied as simulated annealing, particle swarm optimization, A* search and evolutionary algorithms to work in tandem with the transformation; and develops a good spread of solutions across the Pareto front. However, it is dependent on the quality of the optimizer with which it is paired, and it cannot focus the effort of the optimizer on a particular part of the Pareto front. This work is organized as follows: Section 2 discusses background on multi-objective problems. Section 3 introduces the multi-objective methods we compare against. Section 4 describes the PaCcET in detail. Section 5 describes a test domains and show the practical results of using the PaCcET. Section 6 discusses the benefits and drawbacks of the PaCcET and draws the conclusion to this work. 2 Background This work draws from many distinct concepts from within multi-objective research. For the ease of the reader and for consistency, all of our discussions, visualizations and results across the work assume pure minimization across all objectives. Any objective

3 Pareto Concavity Elimination Transformation: PaCcET 3 to be maximized can instead have its negative minimized and the definitions remain the same. Dominance A solution u dominates another solution v when u simultaneously scores lower on all criteria (objectives) than v: c[f c (u) f c (v)]. u weakly dominates v when v has scored the same as u on some objectives, but less on at least one objective [8]. Pareto optimal set A solution which is not dominated by any other feasible solution is part of the Pareto optimal set P. As an optimizer solves a problem, it will approximate P with a Pareto approximate set PI at iteration I. Multi-objective spaces There exist two spaces discussed in most multi-objective works. Ω is the parameter space. It is as many-dimensional as the number of variables that the optimizer Ξ is allowed to manipulate. A typical optimizer will search in areas of Ω that are nearby currently-known successful solutions. Λ is the objective space. It is as many-dimensional as the number of independent objectives that are to be optimized. The mapping from Ω Λ is unknown to Ξ, but is usually repeatable with some stochastic error. In this work we use two additional spaces, that are not usually formally defined. The first is Λ norm, which is simply a normalized version of Λ, in which no feasible vector can have any component value lower than. The second is Λ τ, which is the post-paccet analogue to Λ. Utopia and nadir vectors Two important concepts in multi-objective problems are the utopia and nadir points. The utopia point takes on the best possible value for each objective, minus some small amount so that it is always infeasible. This point is difficult to find, requiring an optimization for each objective individually. Instead in this work, we simply approximate it: û (c) = min(p I (c)) (1) where û (c) is the c th element in the estimated utopia vector, min(p I(c)) is the minimum c th element of any vector in PI, and is a small value [11]. The nadir point takes on the worst value for each objective seen in the Pareto optimal set, which we can estimate as: û nad (c) = max(p I (c)) (2) but it is very important to note that this is a distinct concept from the worst feasible vector; it is instead the upper bound of the objective values for solutions within P I [11]. 3 Multi-Objective Methods Many successful multi-objective algorithms have been developed. In this work we address linear combinations since they are a component of PaCcET, as well as NSGA-II and SPEA2, two multi-objective evolutionary algorithms which have been successful in a wide variety of multi-objective problems.

4 4 L. Yliniemi and K. Tumer Linear Combination A linear combination of objectives is a simple solution to multiobjective problems that is sufficient for finding Pareto optimal points [17]. This takes the form: LC(v) = w 1 v(1) + w 2 v(2) + + w n v(n) (3) where LC(v) is the linear combination evaluation of vector v, v(c) is the evaluation of vector v on the c th objective and w c is the weight for the c th objective. This method is computationally cheap [25], but presents three primary problems. First, as the number of objectives increases, the choice of weights can become difficult. Second, this method is incapable of finding certain areas of the Pareto Front, those that are non-convex. Third, incrementing the weights evenly to converge to different parts of the Pareto front does not necessarily lead to evenly-spaced solutions along the front [1]. NSGA-II NSGA-II sorts solutions into a series of successive fronts that have the same properties as a Pareto approximate front. Those solutions on the less-dominated fronts are more desirable and are kept. To break ties, a local density measure is used. Details can be found in [12]. SPEA2 SPEA2 gives each vector a strength equal to the number of vectors in the current population it dominates. Each vector then sums the strengths of all vectors which dominate it, and this forms a raw fitness evaluation. This is altered slightly by a local k- nearest neighbor density calculation, and the best solutions move forward. Full details can be found in [13]. 4 Pareto Concavity Elimination Transformation (PaCcET) All members in the currently discovered Pareto-approximate set, PI and those weakly dominated by PI form the border (Λ B) between the dominated space Λ D and nondominated space Λ N. If we can produce a solution that is non-dominated by the current set PI (resides in Λ N ), that is desirable. A solution that lies further and further within Λ D would be considered less and less desirable. Of the solutions in PI, we cannot directly say which of them will be the best solution until after the optimization process, at which point a human expert might make a judgement. The PaCcET makes each solution on Λ B equally valuable to a linear combination in Λ τ through a two step transformation, which first transforms from Λ to Λ norm, and then transforms from Λ norm to Λ τ, where the τ superscript on any space or set denotes the transformed space or set. This means that any Pareto-approximate solution will have a linear combination evaluation of 1 when all weights are set to 1. All solutions that lie in Λ τ N will have a linear combination evaluation < 1, and all solutions in Λτ D will have a linear combination evaluation > 1. Algorithm To determine the transformed evaluation for a given solution vector v, we require the current Pareto approximate set PI, from which we can calculate the approximate utopia point û based on PI (Equation 1), and the matching nadir approximation û nad (Equation 2).

5 Pareto Concavity Elimination Transformation: PaCcET 5 The first step is to normalize the target vector v such that each objective takes on a value not less than, transforming Λ to Λ norm [11]: v norm (c) = ûnad (c) v(c) û nad (c) û (c) (4) Because the normalization is conducted based on the utopia and nadir vectors, by definition û nad,norm 1 and û,norm. The second step is to perform the transformation from Λ norm to Λ τ. Within this process, we use the unit vector R that points from û,norm toward v norm, which is calculated as: R = vnorm v norm (5) All distance measurements in the transformation process are taken along the direction of R, so we define the function Rdist(, ), which returns the distance between two objects measured along the direction of vector R. We measure three for use in the PaCcET: The distance from û,norm to v norm The distance from û,norm to Λ norm B d = Rdist(, v norm ) (6) along R D = Rdist(, Λ norm B ) (7) The distance from û,norm to Λ τ B D τ = Rdist(, Λ τ B) (8) Algorithm 1 PaCcET for iteration I Require: Set of solutions V Require: Pareto Approximate Set PI 1: Find û, û nad (Eq. 1 2) 2: c w c = 1 3: for all Solutions i V do 4: Find v i 5: Find vi norm (Eq. 4) 6: Find R (Eq. 5) 7: Find Rdist(, vi norm ) [d] 8: Find Rdist(, Λ norm B ) [D] 9: Find Rdist(, Λ τ B) [D τ ] 1: Find d τ (Eq. 9) [d τ ] 11: Find vi τ (Eq. 1) 12: F it P accet (v i) = LC(vi τ ) (Eq. 3) 13: end for

6 6 L. Yliniemi and K. Tumer Obj. 2 (norm or ) Objective 1 (norm or ) Fig. 1. Visualization of quantities used in transformation. The vector v norm is represented by the hollow green X mark, and v τ by the solid red X mark. v norm lies outside of the dominated hypervolume, so is a desirable point to discover. Green dots correspond to vectors in P,norm I, and red to their transformations in PI τ. All measurements are along the vector R. Λ τ B is a user-defined Rn 1 space embedded in R n. In this work we use the space which maps to LC(x) = 1 R n (see Figure 1). We then calculate d τ, the distance away from û,norm at which v τ should be located: d τ = D τ d D (9) And finally we determine the location of v τ : We enclose this whole process by defining: v τ = d τ R (1) v τ = P accet (v) (11) This procedure ensures that non-dominated solutions will have a LC value of < 1, making it more desirable to an optimizer using a linear combination of transformed objectives than a point in the non dominated set (LC = 1) or a dominated point (LC 1). Thus, the optimizer will seek to incrementally push forward the Pareto Approximation PI until it is a good approximation of the Pareto front, depending on the limitations of the optimizer Ξ. Maintaining PI, the Pareto approximate set P I is maintained as with any other Pareto optimality calculation: each new vector v is compared to all members in PI. If any

7 Pareto Concavity Elimination Transformation: PaCcET 7 f f Fig. 2. A grid of points in Λ. Black points are non-dominated, while cyan points are dominated by the red points in the current Pareto-approximate set PI. f τ f.5 τ 1 1 Fig. 3. The grid of points from Figure 2 transformed into Λτ. Black points evaluate to LC>1, the Pareto-approximate red points to LC=1, and cyan points to LC 1

8 8 L. Yliniemi and K. Tumer members of PI are dominated, they are removed. If, after testing against each point in PI, v has not been dominated, it is added to P I. The size of PI is a user-defined parameter. Through the course of our experiments, we found that a size of 25 gives a clear approximation of the Pareto front in most 2- and 3-objective cases. We ran tests with a size as small as 5, in which the algorithm still functions, but provides a very coarse approximation of the true Pareto front. Once over the chosen size, the user must define a method for choosing which members of PI are to be eliminated. We tested nearest-neighbor elimination, k-nearest neighbor elimination, and random elimination (each maintaining the extreme points that contribute to û and û nad ). The performance of the method in approximating the Pareto front was not sensitive to these. 5 Experiment: KUR As a first experimental domain, we use a test problem (KUR) from multi-objective optimization [11]. It is defined by two functions to be minimized: f 1 (x) = f 2 (x) = with the decision variables 2 i=1 [ ( )] 1 exp (.2) x 2 i + x2 i+1 (12) 3 [ xi sin(x i ) 3] (13) i=1 5 < x i < 5 ; i = {1, 2, 3} (14) It has a known Pareto front that is discontinuous and locally concave. A vector x is evaluated: F it LC (x) = w 1 f 1 + w 2 f 2 (15) and for the PaCcET, the fitness is calculated as: F it P accet (x) = f τ 1 + f τ 2 (16) where f τ 1 and f τ 2 represent the transformed objectives, within Λ τ, calculated as: [f τ 1, f τ 2 ] = P accet ([f 1, f 2 ]) (17) As the optimizer Ξ, we use an evolutionary algorithm, in which the population members are vectors of length 3 that meet the criteria set forth in Equation 14, and the mutation operator changes each of the elements of the vector by a random number chosen by a normal distribution centered around with standard deviation.25. We maintain a population of 1 solutions, with the 5 worst-performing solutions removed after each generation, replaced by mutated copies of the winner of 5 binary tournaments.

9 Pareto Concavity Elimination Transformation: PaCcET 9 Figure 4 shows all points discovered by the linear combination over 1 statistical runs of varied w c values 1. The linear combination provides only two areas of consistent solutions in KUR. Almost no solutions are reliably discovered between the two extremes. Figure 5 shows all points discovered by the PaCcET over 1 statistical runs, which produces a wide variety of solutions all along the Pareto front, including two complete sweeping portions of the Pareto front that the LC skips over completely. The PaCcET avoids the problem of solution clumping, and instead produces a very even spread across all but one of the discontinuous portions of the Pareto front. The PaCcET successfully approximates this Pareto front. Figure 2, used for the visualization in Section 4, shows a different experimental run on KUR, in which the PaCcET was used with a simulated annealing algorithm for one hundred thousand iterations with the same mutation operators. Even using this nonpopulation-based optimizer, a good approximation of the Pareto front was discovered. 1 w c values for independent statistical runs were chosen in an evenly-spread array across all objectives. 2 2 f f1-14 Fig. 4. KUR Linear combination results; notice the clumping of solutions near the two extremes.

10 1 L. Yliniemi and K. Tumer Figure 6 shows the percent of dominated hypervolume by PaCcET and two successful multi-objective methods, SPEA2 and NSGA-II, as a function of number of individual fitness evaluations. PaCcET proceeds faster than the other two methods toward the Pareto front. All methods shown eventually converge to a good approximation of the Pareto front, and dominate a similar amount of hypervolume. Figure 7 shows the final produced approximations of the Pareto front for a single run of KUR by the PaCcET, the linear combination, and NSGA-II and SPEA2. SPEA2 and PaCcET produce similar fronts, though PaCcET s computation time is lesser. NSGA-II produces a front with good coverage, but is sparser near the edges of the front. The linear combination fails to cover the whole front, and converges to an extreme solution. 6 Discussion and Conclusion In this work we have presented a low computational cost way to improve the performance of a linear combination in multi-objective problems. PaCcET eliminates concave 2 f f1-14 Fig. 5. KUR solutions produced by PaCcET. Notice the even spread along the Pareto front, and the discovery of 2 sweeping curves that the linear combination misses.

11 Pareto Concavity Elimination Transformation: PaCcET 11 Dominated Hypervolume PaCcET NSGA-II SPEA Population Members Evaluated Fig. 6. Percent of hypervolume dominated by SPEA2, NSGA-II and PaCcET in KUR, calculated using the limits of Figures 4 and 5. 5 statistical runs were conducted, and we report error in the mean (σ/ N where N = 5), and is smaller than the symbols used to plot. regions of the Pareto front for the sake of training, and allows for solutions in these areas to be found by an optimizer. The primary benefits of PaCcET displayed in this work are: 1. It allows a linear combination of transformed objectives to find concave areas of the Pareto front in the original objective space. 2. It acts independently of the chosen optimizer. 3. It creates a wide spread of solutions along the Pareto front on concave or discontinuous fronts. 4. It removes the need for the system designer to choose weights. 5. It functions in higher-than-two objective problems. The first benefit (1) allows a simple linear combination to be applied to a much broader class of multi-objective problems than it could be otherwise. Benefit (2) means that optimizers like evolutionary algorithms, A* search, simulated annealing, or particle swarm optimization can be applied to multi-objective problems through PaCcET with

12 12 L. Yliniemi and K. Tumer F2 F F1 SPEA F1 NSGA II F2 F F1 PaCcET Linear Combination F1 Fig. 7. Final Pareto front approximations by SPEA2, PaCcET, NSGA-II and a linear combination in a single experimental run of KUR. little alteration; it also means that future developments in single-objective optimizers are immediately useful to a large class of multi-objective problem. Benefit (3) reinforces (1): Even on challenging Pareto fronts, PaCcET develops a desirable array of solutions to choose between. Benefits (4,5) remove one of the primary challenges in using a linear combination on more-than-two objective problems. The three drawbacks we have identified are: 1. The optimizer s effort is cannot be concentrated on a particular area of the Pareto front. 2. There is no explicit guarantee about the spacing or coverage of the solutions on the Pareto front. 3. It is reliant on the quality of the optimizer it is paired with. Drawback (1) is common among most Pareto-based methods. Likewise, (2) is a common occurrence, though most Pareto methods show strong empirical evidence of solution spread along the Pareto front, as does PaCcET. Drawback (3) is not a common one, but is a side effect of benefit (2). An intelligently-chosen optimizer for the problem at hand will allow PaCcET to perform to its greatest ability, while a poor choice will lead to poor results.

13 Pareto Concavity Elimination Transformation: PaCcET 13 PaCcET offers a fundamentally different possible avenue for multi-objective research: the elimination of concavity as opposed to the developments of methods that deal well with concave Pareto fronts. Future work in this area includes testing the PaCcET on a large testbed of multi-objective problems, and examining the possible benefits that the PaCcET might offer other when combined with other multi-objective methods. 7 Acknowledgements This work was partially supported by the National Energy Technology Laboratory under Grant No. DE-FE1232 References 1. M. A. Abido. Environmental/economic power dispatch using multiobjective evolutionary algorithms. IEEE Transactions on Power Systems, 18(4): , A. Agogino and K. Tumer. Evolving distributed agents for managing air traffic. In Proceedings of the Genetic and Evolutionary Computation Conference, A. K. Agogino and K. Tumer. Analyzing and visualizing multi-agent rewards in dynamic and stochastic domains. Journal of Autonomous Agents and Multiagent Systems, 17(2):32 338, T. W. Athan and P. Y. Papalambros. A note on weighted criteria methods for compromise solutions in multi-objective optimization. Engineering Optimization, 27( ), R. Balling, J. Taber, M. Brown, and K. Day. Multiobjective urban planning using genetic algorithm. Journal of Urban Planning and Development, 125(2):86 99, S. E. Cieniawski, J. Wayland Eheart, and S. Ranjuthan. Using genetic algorithms to solve a multiobjective groundwater monitoring problem. Water Resources Research, 31(2):399 49, C. A. Coello and Alan D. Christiansen. Multiobjective optimization of trusses using genetic algorithms. Computers and Structures, 75(6):647 66, C. A. C. Coello. A comprehensive survey of evolutionary-based multiobjective optimization techniques. Knowledge and Information Systems, 1(3):269 38, S. Damiani, G. Verfaillie, and M.-C. Charmeau. An earth watching satellite constellation: How to manage a team of watching agents with limited communications. Autonomous Agents and Multiagent Systems, I. Das and J. E. Dennis. A closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Structural Optimization, pages 63 69, K. Deb. Search Methodologies, chapter 1, pages Springer, Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T Meyarivan. A fast elitist nondominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. In Marc Schoenauer, Kalyanmoy Deb, Günther Rudolph, Xin Yao, Evelyne Lutton, Juan Merelo, and Hans-Paul Schwefel, editors, Parallel Problem Solving from Nature PPSN VI, volume 1917 of Lecture Notes in Computer Science, pages Springer Berlin / Heidelberg, Marco Laumanns Eckart Zitzler and Lothar Thiele. Spea2: Improving the strength pareto evolutionary algorithm. Computer Engineering, 3242(13), R. R. Egudo. Efficiency and generalized convex duality for multiobjective programs. Journal of Mathematical Analysis and Applications, 1989.

14 14 L. Yliniemi and K. Tumer 15. Gerald W. Evans. An overview of techniques for solving multiobjective mathematical programs. Management Science, 3(11): , R. T. Marler and J. S. Arora. The weighted sum method for multi-objective optimization: new insights. Structural and Multidisciplinary Optimization, R.T. Marler and J. S. Arora. Survey of multi-objective optimization methods for engineering. Structural and Multidisciplinary Optimization, 26: , A. Messac and P. D. Hattis. Physical programming design optimization for high speed civil transport (hsct). Journal of Aircraft, 33(2):446 44, March P. Ngatchou, A. Zarei, and M. A. El-Sharkawi. Pareto multi objective optimization. Intelligent Systems Application to Power Systems, pages 84 91, K.E. Parsopoulos and M. N. Vrahatis. Particle swarm optimization method in multiobjective problems. ACM Symposium on Applied Computing, R. Penn, E. Friedler, and A. Ostfeld. Multi-objective evolutionary optimization for greywater reuse in municipal sewer systems. Water research, 47(15): , W. Rosehart, C. A. Cañizares, and V. H. Quintana. Multi-objective optimal power flows to evaluate voltage security costs in power networks. IEEE Transcations on Power Systems, C. Tomlin, G. J. Pappas, and S. Sastry. Conflict resolution for air traffic management: A study in multiagent hybrid systems. IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 43(4):59 521, APRIL P. Vamplew, R. Dazeley, A. Berry, R. Issabekov, and E. Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning, December David A. Van Veldhuizen. Multiobjective Evolutionary Algorithms: Classifications Analyses and New Innovations. PhD thesis, Air Force Institute of Technology, M. Wooldridge. An Introduction to MultiAgent Systems. John Wiley and Sons, G. Zhang, X. Shao, P. Li, and L. Gao. An effective hybrid particle swarm optimization algorithm for multi-objective flexible job-shop scheduling problem. Computers and Industrial Engineering, 56: , 29.