2:k e-1x,- Y~ 12 /21(2,

Transcription

1 The Elastic et Apprach t the Traveling Salesman Prblem 349 C~~unicated by Andre~ Bart ~~~~~~~~~~~~~~~~~~~~ An Analysis f the Elastic et Apprach t the Traveling Salesman Prblem Richard Durbin* King's Cllege Rescnrcl: Centre, Cambridge CB 1ST, England Richard Szeliski Artificial Intelligence Center, SRI Internatinal, J\;fenl Park, CA 945 USA Alan Yuille Divisin f Applied Sciences, Harvard University, Cambridge, MA 138 USA This paper analyzes the elastic net apprach (Durbin and Willshaw 1987) t the traveling salesman prblem f finding the shrtest path thrugh a set f cities. The elastic net apprach intly minimizes the length f an arbitrary path in the plane and the distance between the path pints and the cities. The tradeff between these tw requirements is cntrlled by a scale parameter K. A glbal minimum is fund fr large K, and is then tracked t a small value. In this paper, we shw that (1) in the small K limit the elastic path passes arbitrarily clse t all the cities, but that nly ne path pint is attracted t each city, () in the large!< limit the net lies at the center f the set f cities, and (3) at a critical value f K the energy functin bifurcates. We als shw that this methd can be interpreted in terms f extremizing a prbability distributin cntrlled by K, The minimum at a given K crrespnds t the maximum a psteriri (MAP) Bayesian estimate f the tur under a natural statistical interpretatin. The analysis presented in this paper gives us a better understanding f the behavir f the elastic net, allws us t better chse the parameters fr the ptimizatin, and suggests hw t extend the underlying ideas t ther dmains. 1 Intrductin _ The traveling salesman prblem (Lawler et al. 1985) is a classical prblem in cmbinatrial ptimizatin. The task is t find the shrtest pssible tur thrugh a set f cities that passes thrugh each city exactly nce. This prblem is knwn t be P-cmplete, and it is generally believed "Current address: Department f Psychlgy, Stanfrd University, Stanfrd, CA 9435 USA. eural Cmputatin 1, (1989) 1989 Massachusetts Institute f Technlgy that the cmputatinal pwer needed t slve it grws expnentially with the number f cities. In this paper we analyze a recent parallel analg algrithm based n an elastic net. apprach (Durbin and Willshaw 1987) that generates gd slutins in much less time. A This apprach uses a fast heuristic methd with a strng gemetrical flavr that is based n the tea trader mdel f neural develpment (Willshaw and Vn der Malsburg 1979). It will wrk in a space f any dimensin, but fr simplicity we will assume the tw-dimensinal plane in this paper. Belw we briefly review the algrithm. Let {Xi}, i = 1 t, represent the psitins f the cities. The algrithm manipulates a path f pints in the plane, specified by [Y,}, =1 t M (AI larger than ), s that they eventually define a tur (that is, eventually each city Xi has sme path pint Y, cnverge t it). The path is updated each time step accrding t where ~ Y, = () L l1ji(xi - Y) + ;3!((Y Y-1 - Y ) i e-/x/- Y J 1 / 1( 71Ji =----- :k e-1x,- Y~ 1 /1(, () and (J are cnstants, and K is the scale parameter. Infrmally, the () term pulls the path tward the cities, s that fr each Xi there is at least ne Y, within distance apprximately K, The ;3 term pulls neighbring path pints tward each ther, and hence tries t make the path shrt. The update equatins are integrable, s that ~Y = -!(/DY fr an "energy" functin, E, given by E( {Y },!() = -(}1{ L lg L e- 1X,- Y J 1 / K + /JL {Y - y + } (1.1) 1 Fr fixed!\ the path will cnverge t a (pssibly lcal) minimum f E. At large values f K the energy functin is smthed and there is nly ne minimum. At small values f K, the energy functin cntains many lcal minima, all f which crrespnd t pssible turs f the cities (we prve this later in the paper), and the deepest minimum is the shrtest pssible tur. The algrithm. prceeds by starting at large K, and gradually reducing K, keeping t a lcal minimum f E (see Fig. 1). We wuld like this minimum that is tracked t remain the glbal minimum as ]\ becmes small. Unfrtunately, this can nt be guaranteed (see sectin 3). The elastic net apprach is similar t a number f previusly develped algrithms that use elastic matching (Burr 1981), energy-based matching (Terzpuls et al. 1987), r tpgraphic mapping (Khnen 1988) t slve visin, speech, and neural develpment prblems. Alternative parallel analg algrithms have als recently been prpsed fr slving the traveling salesman prblem (Hpfield and Tank 1985; Angenil et al. 1988). The methd f Angenil et al. is clsely related

2 35 Richard Durbin, Richard Szeliski, and Alan Yuille The Elastic et Apprach t the Traveling Salesman Prblem 351 t that discussed here, but is based n Khnen's self-rganizatin algrithm (Khnen 1988). It is faster, but fr large prblems it is marginally less accurate than the elastic methd. An imprtant cntributin f this paper is t analyze the behavir f the energy functin as the cnstant K changes and t describe the energy landscape. In particular, we prve results abut the behavir f the functin fr large and small K, cnfirming the assertins made abve abut hw' the algrithm wrks. First, hwever, we shw that (a) (b) (c) ~ ~ ~D (d) (e) (t) ~ {O Figure 1: The cnvergence f the netwrk as a functin f K, The white and black squares represent the data (1 cities) and the netwrk (5 path pints), respectively. The six figures (a-f) shw the cnfiguratin fund by the netwrk at values K =.61,.6,.1,.,.1,.4. The data set is centered n (.49,.46) and has secnd-rder mments (K;r;r,l(ry, K!J]) = (.75, -.3,.7). We use a =. and /3 =1.. The first bifurcatin, when the rigin becmes unstable, can be calculated t ccur at K =.66 (see Sectin 5), in agreement with the simulatin. The secnd break temperature is between K =.1 and K =. fr the simulatin when the line spreads int a lp. This crrespnds well t the pint at which the secnd eigenvalue becmes negative (K =.196). The crrespndence is nt exact because nnlinear terms becme significant after the first break. minimizing this cst functin can be interpreted in terms f maximizing a prbability distributin. The Prbabilistic Interpretatin _ The energy functin (1.1) can be related t a prbability distributin by expnentiatin. This is analgus t use f the Gibbs distributin in statistical mechanics. L({Y } }~) - 1 -E/K.r: - (7r)K Af e 1 A1 1 At II-{L ~e-ix'-yi/k}iie-nl;{{y-y+d i=l AI =l 7rK =l Observe that minimizing E with respect t {Y } crrespnds t maximizing L with respect t {Y }. We can interpret L in terms f Bayes' therem, which states that P(YIX) = P(XIY)P(Y) (.1) P(X) where P(YIX) is the prbability f a tur (Y) given a set f cities (X). Our algrithm maximizes P(YIX) ver all pssible turs (Y), s the value f P(X) is irrelevant. The distributin At P(Y) = IIp-11/K {Y - y./+1 } (.) )=1 is the a priri prbability f a given tur. This distributin is a crrelated gaussian that assigns greater prir prbability t shrter turs. The distributin 1 At 1 P(XIY) =II-{L ~e-ix'-yji/j{} (.3) ;=1 l~f )=1 7r1\ is the prbability f the cities being at (X) given that the tur pints are at (Y). P(XIY) is the prduct f IV independent prbability distributins P(XiIY) = ~ f= ~(,-IX'-YJI/1\ M )=1 7r1\ This equatin is equivalent t assuming that the measured psitin f city Xi was actually derived frm ne f the tur pints in {Y./} with a tw-dimensinal gaussian errr f variance 1(, but withut knwing which tur pint Xi crrespnds t. Thus equatin.1 shws that the elastic net algrithm is cmputing the mst prbable tur (finding the

3 35 Richard Durbin, Richard Szeliski, and Alan Yuille The Elastic et Apprach t the Traveling Salesman Prblem 353 Bayesian MAP estimate) given a prir mdel (.) that favrs shrt turs and a sensr mdel (.3) with tw-dimensinal psitin uncertainty. Our methd thus has an bvius extensin t surface interplatin and threedimensinal surface mdeling where the crrespndence between surface pints and measured data pints is unknwn. (a) 3 ~acking a ~ini~u~~~~~~~~~~~~~~~~~~~_ The algrithm devised by Durbin and Willshaw minimizes E at large!\ and then tracks the minimum energy slutin dwn t small K, At a lcal minimum (r any extremum), we have (J}/"./J =a 1 (e) (f) where II is 1 r and r? and ~ are the.1' and y cmpnents f the psitin vectr Y i As we fllw the extrema as K changes we get the equatin d () d!\ f)~/j =a (g) (i) which becmes f)e f)e drr./i D1\DYl' + Dy~fl in)" di~ = T btain the traectry we must slve this equatin fr d}~r [dl«, When we are at a true minimum, the Hessian D E DrT D1T is a psitive definite matrix and can be inverted, enabling us t cmpute rlrt [ili«, Bifurcatins ccur when the Hessian has zer eigenvalues. In this case dr~'l rl!\ is underdetermined and there are several pssible slutins. Cmputer simulatins and ur calculatins in the large!\ limit (see belw) shw that such a bifurcatin ccurs as the tur initially spreads ut frm a pint. After this, ur simulatins suggest that the minimum tracks smthly with /<...r. Other minima f the energy functin als appear as K is reduced. Fr the cnfiguratin shwn in Figure 1, the number f minima increases rapidly frm 1 at!{ =.1 t 3 at!{ =.1, 9 at!\ =.8 (shwn in Fig. ), and very many at!{ <.5. The minimum fund by tracking K frm large t small values is 11t necessarily the ptimal (glbal) minimum (Fig. ). evertheless, empirically the minima fund are within a few percent f ptimal (Durbin and Willshaw 1987). One pssible imprvement wuld be t pick up and track nearby minima by lcal randm perturbatin as in simulated annealing (Kirkpatrick et al. 1983). Figure : Pssible minima f the energy functin fr the data in Figure 1. T() investigate the energy minima we started 1 simulatins at /\r =.8 with randm initial cnfiguratins, hence 'withut using the slutins fr larger 1\ as initial cnditins. In cases that lead t a sensible tur n subsequent slw reductin f K, the lines ining the white bxes (data pints) shw the tur fund by the netwrk. We fund nine distinct grups f minima with the fllwing frequencies: (a) 6, (b) 183 (tur length 3.356), (c) 14 (tur length 3.88), (d) 13 (tur length 3.35), (e) 95, (f) 7, (g) 48, (h) 36 (tur length 3.4), and (i) 36. We have prbably fund all the mar frms f minima, since the least frequent happened 36 times. te that the mst frequent pattern (a) is neither the ne btained frm tracking /( (b) nr the ptimal ne (c). 4 The S~all K Li~it~~~~~~~~~~~~~~~~ We nw cnsider the behavir f the extrema and the Hessian at these extrema as 1\ -+ O. We will shw that the nly stable extrema ccur

4 354 Richard Durbin, Richard Szeliski, and Alan Yuille The Elastic et Apprach t the Traveling Salesman Prblem 355 when each {X;} has at least ne [Y} arbitrarily clse t it. Thus, the extrema all crrespnd t pssible turs. Fr any given i let B(!{) = min 1)[; - y~ I.J Then L e-ix/-yji//(! < J.~fe-1J(1\)/1\ and -O!\" lg ""'" e-ix'-yji/1\ > /"1 ~f B (!\")n G - -n \ g ~ +- K Thus, fr the energy t be bunded we must have. {B (!( ) - 1\"1 gai } C 11m = 1\ ----+!{ where (1 is a cnstant and hence B(I\") = ()(I\r 1 / ). Thus, in the limit as 1\" ~, cnfiguratins with unmatched Xs will have arbitrarily high energy, and s will nt be fund by the algrithm. This means that the minima will all crrespnd t pssible turs. Althugh all the Xs are matched there is n requirement that all the Ys are matched. Indeed, with crrect chice f parameters nand /1 it can be shwn that there will be nly ne tur pint at each city. The remaining tur pints space themselves evenly in the intercity intervals. A sufficient. requirement n the parameters fr this t happen is that J (\ <.4 where.4 is the shrtest distance frm a tur pint being attracted t sme city t any tur pint nt being attracted t that city. T derive this cnditin n the parameters cnsider a single city situated at the rigin. Define 1i'.J by Assume the Y, are at equilibrium. We wish t cnsider the stability f an equilibrium in which there are tw Ii'./ that stay significantly nnzer as!\" -----" (t have a single tur pint cnverge t each city we require instability). We can chse!\" sufficiently small that there is n significant interactin between these tw tur pints and any ther cities. Cnsider perturbatins Y = Y + Ei: We want t find eigenvectrs such that )..C = -/( DY' -nw~y + /11«Y~+l + Y~-l - Y~) ""'" DW..r -O{71'C + Y, G(-D. Ck)} + /11\O(f) + O(f ) k v, -a{1l'c + LY~(WWkY! - likw Y). cd + /J/{O(r) + ()«() k 1\ (11)" ""'".r = - ~ {Ii C + Y, G ldk'(y k. Ck) - Y(Y. C)} + /11i O(f) + ()«( ) 1\ k where f = max (I C I). In the limit f small 1{ we can ignre the /1 term as well as the higher rder f term. Clearly then fr each e = fy, fr sme scalar f' Let II = _),,1(/, and cnsider the case where nly il'l, 1i' are significant. This leads t the eigenvalue equatin 11 [/< - (1 - l1'l)r?] - II 11']11'1 ) ( f] ) = ( 1lJ]U)~ 11J [/( - (1 - l)}] - 11 f The criterin fr instability is that at least ne A is psitive, and hence that the crrespnding 11 is negative. Fr large 1( bth eigenvalues II] and 11 are clearly psitive. Hence fr instability we require that 1( shuld be belw the value at which ne eigenvalue (and hence their prduct) ges negative. The prduct is = 1111' [(1< - 11J }.~)(1( - r 71'1 Y'"l) - 11'11 y ] Y] n'1 11'1< [1( - Therefre we require that li r < 11' ~ + 11Jl y; (71' ~ + 11y;)] Since H'l + tr = 1 we will be safe if min IY I > 1(. At equilibrium at small!\" we have Y, = (;31\r/ u' )A where A = Y Y - 1. When, as will be usual, Y 1 and Y are neighbrs, then IA I is ust the distance t the next path pint nt cnverging n the city, and a sufficient requirement is that this distance must be greater than n /;1 Since an average tur n 1\T cities has length (/)1/ a safe estimate fr.a min wuld be.(i\t/)1/ / AI. Alternatively ne culd chse (\ t be a decreasing functin f 1(, such as 1{') where p is a fixed expnent between zer and ne. 5 At Large 1\" _ Fr large K, the energy functin (1.1) has a minimum crrespnding t the net lying at the center f the cities. At a critical value f 1\" this mini

5 356 Richard Durbin, Richard Szeliski, and Alan Yuille The Elastic et Apprach t the Traveling Salesman Prblem 357 mum becmes unstable and the system bifurcates. The initial mvement f the net depends nly n the secnd-rder mments f the cities. T shw this, we first calculate the first and secnd derivatives f E with respect t Y. () (Yl' - Xr)e-IX,-YA.1/J( = T'L ( At - DYl' \. i=l L=l e, + ;J{}TIJ _ vrt! _ VII } (5.1) k 1 k+1 1 k-1 D E DYl'DYt () 61IV6kle-IX,-YAI/J< KL M i=l (L=l e + n (Y{ - XJ')(}[V _ Xnf-IX,-Yd/J(f~IX'-YI1/J( }r3 L \. i=l : (}TI,J _ X!.J)(Y;v _ xr~)d-ix'-yki/j( ~,k 1 k 1 L U~'l K3 ~ (",~1 D-IXz-YJ I/J() 1,=1 L...,)=1 t + /J6 1W {6kl - 6~'+1l - 6k-1l} (5.) By substituting Y, = int (5.1) we find that _ (} (_Xf)e-IX,1/K _ (} IJ DYt - K L (L M e-1x,i/ K) - - K M LX, ~ 1=1 )=1 1=1 The rigin is thus always an extremum, prvided that it is chsen at the center f the Xs (i.e., Lt:1 Xi = ). T shw that the center is a minimum fr very large K, we must calculate the eigenvalues f the Hessian. As!( decreases, this minimum becmes unstable and a bifurcatin ccurs. Knwing the value f!( at. which this ccurs will give us a useful starting value fr!( when we are running the elastic net algrithm. At the rigin the Hessian can be written as D E _O_{)IJlJ{). +, XeX~) DYtDY~v AI!( U 1(3 A1 ~ ~{)kl L Xf'..X-r + ;1{)IIV {{)kl - {)k+ll - {)k-ll} (5.3) I\. ~.Al i=l Fr large K, the eigenvalues f the Hessian are clearly all psitive. By inspectin f equatin, we see that thrughut the regin IY I «!( the Hessian is psitive definite and s the rigin is a unique minimum. Fr small K, the dminant terms (the secnd and third terms n the right-hand side f 5.) have negative trace, s there are sme negative eigenvalues. Thus, the rigin is a stable state fr large K but then becmes unstable as K decreases. T see hw this ccurs we must explicitly calculate the eigenvectrs. If we cmpute the eigenvalues f the Hessian (Durbin et al. 1989), we find that smallest eigenvalue is. = _ ~ + J in' ~ (5 4 A mm tcai!<4ai 81 s AI. ) where A is the principal eigenvalue f the city cvariance matrix. The center then becmes unstable and breaks at!( s.t. Amin = O. This can be calculated frm equatin 5.4. Since the eigenvectrs depend nly n the secnd-rder mments f the distributin f the cities the glbal minimum fr K ust belw the critical value will als depend chiefly n the secnd-rder mments, As K decreases further the higher rder mments will becme imprtant. These theretical results are cnfirmed by cmputer simulatins (see Fig. 1). The net stays at the rigin until the critical value f K and then frms a line alng the principal axis f the city distributin. ear the secnd critical value f!( (when the eigenvalue determined by substituting the minr eigenvalue f the city cvariance matrix int equatin 5.4 becmes negative) the line spreads int a lp. 6 Cnclusin _ In summary, we have btained several theretical results cncerning the elastic net methd. First, we have shwn hw the elastic net slutin can be interpreted as a maximum a psteriri estimate f an unknwn tur (circular curve), where sme pints alng the tur have been measured with gaussian uncertainty in psitin. Secnd, we have prved that fr small K every pint Xi is matched, and that each pint must be within (!(1/) f a tur pint. Third, we have fund a cnditin n the parameters, IJ under which each city becmes matched by nly ne tur pint. Furth, we have shwn that at large K, a single minimum exists fr the energy functin, with all f the tur pints lying at the center f gravity f the cities. Fifth, we have shwn hw t calculate the bifurcatin pints fr the elastic net as K is reduced. The first result is particularly interesting since it suggests that this apprach can be applied t ther interplatin, apprximatin, and matching prblems (such as surface interplatin in cmputer visin). The imprtant feature here is that we d nt need t prespecify which mdel pint matches a data pint, allwing "slippery" matching. The secnd result prves the "crrectness" f the elastic net methd, in that any final slutin must be a valid tur. The third, furth, and fifth results can be used in selecting parameter values, a starting cnfiguratin fr the net, and a starting value fr Ii. The elastic net methd that we have analyzed prvides a simple, -effective, and intuitively satisfying algrithm fr generating gd traveling salesman turs. We believe that similar cntinuatin-based algrithms can be applied t a wide range f ptimizatin and apprximatin prblems.