A Reinforcement Learning for Freight Train with Augmented Collective Motions in a Marshaling

Transcription

1 Proceedngs of the Internatonal MltConference of Engneers and Compter Scentsts 207 Vol I, IMECS 207, March 5-7, 207, Hong Kong A Renforcement earnng for Freght Tran wth Agmented Collectve Motons n a Marshalng Yoch Hrashma Abstract Ths paper proposes a new renforcement learnng method to solve a tran marshalng problem for assemblng an otgong tran. In the problem, the arrangement of ncomng freght cars s assmed to be random. Then, the cars are rearranged to the desred layot n order to assemble an otgong tran. In the proposed method, each set of freght cars that have the same destnaton make a grop, and the desrable grop layot constttes the best otgong tran. Then, a rearrange operaton s condcted by sng several sb-tracks and an otgong tran s assembled n the man track. When a rearrangement operaton s condcted n the proposed method, several cars located on dfferent sb-tracks are collected by a locomotve so that the transfer dstance of locomotve s redced. In order to rearrange cars by the desrable order, cars are removed from a sb-track to one of other sb-tracks. Each marshalng plan that conssts of seres of removal and rearrangement operatons are generated by a renforcement learnng system based on the transfer dstance of a locomotve. The total transfer dstance of the locomotve reqred to assemble an otgong tran can be mnmzed by the proposed method. Index Terms Collectve acton, Freght tran, Marshalng, -earnng, Schedlng, Contaner transfer problem R I. INTRODUCTION alway freght transportaton has smaller envronmental load as compared to the transportaton by tracks. However, modal shfts are reqred for area that has no ralway n order to lnk dfferent modes of transportaton ncldng ral. A freght tran conssts of several freght cars, and each car has ts own destnaton. The tran drven by a locomotve travels several destnatons decoplng correspondng freght cars at each freght staton. In ntermodal transports ncldng ral, contaners carred nto the staton are loaded on freght cars and located at the freght yard n the arrvng order. The ntal layot of freght cars n the yard s determned by consderng both arrangement of ncomng tran and the arrvng order of the contaners carred by non-ral methods. Contaners carred nto the staton are loaded on freght cars and the ntal layot of freght cars s ths random. For effcent shft n assemblng otgong tran, freght cars mst be rearranged before coplng to the freght tran. In general, the rearrangement process s condcted n a freght yard that conssts of a man-track and several sbtracks. Freght cars ntally placed on sb-tracks are rearranged, and lned nto the man track. Ths seres of operaton s called marshalng, and several methods to solve the marshalng problem have been proposed []-[4]. However, n these methods, the marshalng s based on the layot generated by the classfcaton process that assgns each car nto a certan sb-track. Ths cases restrctons on the nmber of sb-tracks and the nmber of cars n a sb-track. On the other hands, n or research grop, several methods that can generate marshalng plans from the random arrangement of cars n sb-tracks have been proposed [5],[6]. Snce, n these methods, each rearrange operaton s condcted for cars n the same sb-track, the marshalng plan can be mproved by sng method proposed n [7] consderng collectve movements n a rearrange operaton. A rearrange operaton n the method n [7] s condcted after collectng cars n the same grop from several sb-tracks by the desrable order. The method s effectve for redcng the total transfer dstance of locomotve. In ths paper, the collectve movements s agmented and a new learnng algorthm for mprovng marshalng plan of freght cars n a tran s proposed. In the proposed method, the ntal arrangement of freght cars n sb-tracks are assmed to be random, and ths any arrangements as the reslt of classfcaton of ncomng freght cars can be managed. Then, marshalng plans based on the transfer dstance of locomotve are obtaned by a renforcement learnng method [8]. Freght cars to be rearranged are collected from several sb-tracks, copled to each other and moved nto the man track by a locomotve. Smltaneosly, the optmal seqence of carmovements as well as the nmber of freght cars that can acheve the desred layot of otgong tran s obtaned by atonomos learnng. In order to show the effectveness of the proposed method, compter smlatons are condcted for several methods. A. Freght Yard II. PROBEM DESCRIPTION A freght yard s assmed to have man track and m sb- Manscrpt receved Janary 7, 207. Y. Hrashma s wth Osaka Insttte of Technology, -79- Ktayama, Hrakata, Osaka, Japan (phone, fax: ; e-mal: yoch.hrashma@ot.ac.p. ISBN: ISSN: (Prnt; ISSN: (Onlne IMECS 207

2 Proceedngs of the Internatonal MltConference of Engneers and Compter Scentsts 207 Vol I, IMECS 207, March 5-7, 207, Hong Kong tracks. Defne k as the nmber of freght cars carred n and assgned to sb-tracks n a classfcaton stage. Then, they are moved to the man track by the desrable order based on ther destnaton n a marshalng stage. In the yard, a locomotve moves freght cars, and the movement of freght cars from sb-track to sb-track s called removal, and the carmovement from sb-track to man track s called rearrangement. For smplcty, the maxmm nmber of freght cars that each sb-track can have s assmed to be n, the th car s recognzed by an nqe symbol c, (=,, k. Fg. shows the otlne of freght yard n the case k=30, m=n=6. In the fgre, track Tm denotes the man track, and other tracks []-[6] are sb-tracks. The man track s lnked wth sb-tracks by a ont track, whch s sed for movng cars between sb-tracks, or for movng them from a sb-track to the man track. every track, and a newly placed car s copled wth the adacent freght car. B. Desred layot n the man track In the man track, freght cars that have the same destnaton are placed at the neghborng postons. In ths case, removal operatons of these cars are not reqred at the destnaton regardless of arrangements of these cars. In order to consder ths featre n the desred layot n the man track, a grop s organzed by cars that have the same destnaton, and these cars can be placed at any postons n the grop. Then, makng a grop correspondng to each destnaton, the order of grops lned n the man track s predetermned by destnatons. Ths featre yelds several desrable layots n the man track. 5 grop 3 (destnaton 3 Desrable layots for grop c c 8 grop 2 (destnaton 2 c 7 c grop Fg. : Freght yard When the locomotve moves a certan car, other cars locatng between the locomotve and the car to be moved mst be removed to other sb-tracks. Ths operaton s called removal. Then, f k nm-(n- s satsfed for keepng adeqate space to condct removal process, every car can be rearranged to the man track. In each sb-track, postons of cars are defned by n rows. Every poston has nqe poston nmber represented by mn ntegers, and the poston nmber for cars at the man track s 0. [f] [e] [d] [c] [b] [a] [] [2] [3] [4] [5] [6] [] [2] [3] [4] [5] [6] Poston ndex Marshalng yard Fg. 2 Example of poston ndex and yard arrangement Fg. 2 shows an example of poston ndex for k=30, m=n=6 and the layot of cars for Fg. -(b. In Fg. 2, the poston ``[a][]'' that s located at row ``[a]'' n the sb-track ``[]'' has the poston nmber. For nfed representaton of layot of cars n sb-tracks, the frst car s placed at the row ``[a]'' n c c c 0 c 8 c c c 8 c 7 c (destnaton F: Grops and arrangements of freght cars Fg. 3 depcts examples of desrable layots of cars and the desred layot of grops n the man track. In the fgre, freght cars c,, to the destnaton make grop, c 7,, c 8 to the destnaton2 make grop2,,, 5 to the destnaton3 make grop3. Grops,2,3 are lned by ascendng order n the man track, whch make a desrable layot. Also, n the fgre, examples of layot n grop are n the dashed sqare. The layot of grops lned by the reverse order does not yeld addtonal removal actons at the destnaton of each grop. Ths, n the proposed method, the layot lned grops by the reverse order and the layot lned by ascendng order c c step Case (a Case (b Case (c step2 step3 step step2 step3 step step2 step3 ISBN: ISSN: (Prnt; ISSN: (Onlne IMECS 207

3 Proceedngs of the Internatonal MltConference of Engneers and Compter Scentsts 207 Vol I, IMECS 207, March 5-7, 207, Hong Kong from both ends of the tran are regarded as desred layots. By defnng r as the nmber of grops, the total nmber of layots of grop s 2 r-. Fg. 4 depcts examples of materal handlng operaton for extended layot of grops at the destnaton of grop. In the fgre, step shows the layot of the ncomng tran. In case (a, cars n grop are separated at the man track, and moved to a sb-track by the locomotve at step 2. In cases (b,(c, cars n grop are carred n a sb-track, and grop s separated at the sb-track. In the cases, grop can be located wthot any removal actons for cars n each grop. Ths, these layots of grops are regarded as canddate for desred one n the learnng process of the proposed method. When there exsts a rearrangng car that has no car to be removed on t, ts rearrangement precedes any removals. In the case that several cars can be rearranged wthot a removal, rearrangements are repeated ntl all the canddates for rearrangement reqres at least one removal. If several canddates for rearrangement reqre no removal, the order of selecton s random, becase any orders satsfy the desrable layot of grops n the man track. In ths case, the arrangement of cars n sb-tracks obtaned after rearrangements s nqe, so that the movement cont of cars has no correlaton wth rearrangement order of cars that reqre no removal. Ths operaton s called drect rearrangement. When a car n a certan sb-track can be rearrange drectly to the man track and when several cars located adacent postons n the same sb-track satsfy the layot of grop n the man track, they are copled and appled drect rearrangement. Fg. 5 shows 2 cases of marshalng ncldng a removal and rearrangements, exstng canddates for rearrangng cars that reqre no removal. At the top of fgre, from the left sde, a desred layot of cars and grops, and the ntal layot of cars n sb-tracks followed by 2 cases n 2 colmns are depcted for m=4, n=5, k=9. In both cases, c,,, are n grop,,, c 7, c 8 are n grop2, and grop mst be rearranged frst to the man track. In each grop, any layots of cars can be acceptable. In case, n step, n step 3 and step 4 are appled the drect rearrangement. Also, n case2, steps 2 throgh 4 are drect rearrangements. Step2 n case and step n case2 are removals. Step 4 n case, 3 cars c,, located adacent postons are copled wth each other and moved to the man track by a drect rearrangement operaton. In addton, at steps 2 and 3 n case 2, cars n grop and grop2 are collected n sb-tracks to rearrange drectly, snce the arrangement of,, c,, can satsfy the desred layot of grops n the man track. Whereas, at steps,3,4 n case, 3 drect rearrangements are condcted separately. In addton, Fg.6 shows an example of agmented collectve movements. In case3 n the fgre, the transfer dstance of locomotve s redced n steps and 4 becase the freght cars are collected from throgh wthot a decoplng operaton, and becase a removal s nclded n the collectve movements. Whereas a decoplng operaton s condcted n each step n case and two decoplng n steps, 4 n case2. C. Marshalng process A marshalng process conssts of followng 6 operatons: (I selectons: grop-layot n the man track, ISBN: ISSN: (Prnt; ISSN: (Onlne (II rearrangement of freght cars to the man track, when they can be moved drectly, grop 3 ( grop 2 Desrable layot (Man track Case (,,c 7,c 8 m=4 n =5 c 8 grop c c 7 (c,,, Intal arrangement (Sb-tracks Case2 Fg. 5 Example of marshalng (III selecton: a freght car to be rearranged nto the man track, (IV selecton: removal destnatons of the cars n front of the car selected n (IV, (V selecton: the nmber of cars to be moved n (V, (VI removal of the cars determned n (VI to the selected sb-track n (V. c 8 c 8 c c 7 c c 7 Step c 8 c c 8 c c 7 c 7 Step2 c c 8 c 8 c c 7 c 7 Step3 c c c 8 c 8 c 7 c 7 Step4 IMECS 207

4 Proceedngs of the Internatonal MltConference of Engneers and Compter Scentsts 207 Vol I, IMECS 207, March 5-7, 207, Hong Kong After operatons (I,(II are fnshed, (III-(VII are repeated ntl one of desrable layots s acheved n the man track, and a seres of operatons from the ntal one n (I to the fnal one n (III achevng the desrable layot s defned as a tral. Now, defne Go as the desred layot and h as the nmber of canddates of Go. Each canddate n operaton (I s represented by ( h, h =2 r- In the operaton (IV, each grop has the predetermned poston n the man track. Then, the car to be rearranged s defned as c T, and canddates of c T s determned by the nmber of freght cars that have already rearranged to the man track and the grop layot n Go. h 2 s defned as the nmber of freght cars n grop to be rearranged, and 3 (h + 3 h +h 2 as canddates of c T. In the operaton (V, the removal destnaton of cars located on the car to be rearranged s defned as T R. Then, defnng 4 (h +h 2+ 4 h +h 2+m- as canddates of T R, excldng the sb-track that has the car to be removed, and the nmber of canddates s m-. In the operaton (VI, defnng n p as the nmber of removal cars reqred to rearrange c T, and defnng n q as the nmber of removal cars that can be located the sb-track selected n the operaton (V, the canddate nmbers of cars to be moved are determned by 5 ( 5 mn{n p,n q}, h +h 2+m 5 h +h 2+m +mn{n p, n q}. (III-(VI are repeated ntl all the cars are rearranged nto the man track. In the operaton (II, collectve movements are generated by the followng operatons (IIa-IId: (IIa Determne the cars that can be rearranged drectly to the man-track, and the sb-tracks ncldng those cars. Also, determne f sch sb-tracks can be yelded by a sngle removal operaton, and defne h 4 the nmber of canddates of removal-destnaton for the removal cars. (IIb Defne T C as the sb-track ncldng cars to be rearranged drectly, and h 3 as the nmber of canddates of T C. (IIc If h 3>, select a T C as the tal of the collected cars, f h 3=, rearrange T C wthot collectve movements, f (h 3=0, h 4<2, operaton (II s fnshed, and f (h 3=0, h 4>, select a removal operaton, (IId Condct rearrange or removal and go to (IIa. Then, defnng 2 (h +h 2+m+mn{n p, n q} 2 h +h 2+ h 3+h 4+m+mn{n p, n q} as canddates D. Transfer dstance of ocomotve When a locomotve transfers freght cars, the process of the nt transton s as follows: (E starts wthot freght cars, and reaches to the ont track, (E2 restarts n reverse drecton to the target car to be moved, (E3 onts them, (E4 plls ot them to the ont track, (E5 restarts n reverse drecton, and transfers them to the ndcated locaton, and (E6 dsonts them from the locomotve. Then, the transfer dstance of locomotve n (E, (E2, (E4 and (E5 s defned as D, D 2, D 3 and D 4, respectvely, and the dstance of the nt transton D s calclated by D= D + D 2+ D 3 +D 4. Also, defne the nt dstance of a movement for cars n each sb-track as D mnv, the length of ont track between adacent sb-tracks, or, sb-track and man track as D mnh. The locaton of the locomotve at the end of above process s the start locaton of the next movement process of the selected car. The ntal poston of the locomotve s located on the ont track nearest to the man track. Fg. 7 shows an example of transfer dstance. In the fgre, m=n=6, D mnv=d mnh=,k=8, (a s poston ndex, and (b depcts movements of locomotve and freght car. Also, the locomotve starts from poston 8, the target s located on the poston 8, the destnaton of the target s 4, and the nmber of cars to be moved s 2. Snce the locomotve moves wthot freght cars from 8 to 24, the transfer dstance s D+D2=2, D=5,D2=7, whereas t moves from 24 to 6 wth 2 freght cars, and the transfer dstance s D3+D4=3 (D3=7,D4=6. kd mnv Case3 c 8 c 7 c 8 c c 7 c Step Step3 c 8 c 7 c 8 c c 7 c Step2 Step4 Fg. 6 Agmented collectve movements Man track D mnh = nd mnh D mnv = 24 6 c 8 Sb-tracks (b movement of cars III. EARNING AGORITHM (a Poston ndex n sb-tracks Defnng as an evalaton vale for Go, (Go s pdated by the followng rle when one of desred layot s acheved n the man track: (G O max { (G O, ( α (G O + αr γ } md mnv Fg. 7 Transfer dstance l = ( ISBN: ISSN: (Prnt; ISSN: (Onlne IMECS 207

5 Proceedngs of the Internatonal MltConference of Engneers and Compter Scentsts 207 Vol I, IMECS 207, March 5-7, 207, Hong Kong where l denotes the total movement conts reqred to acheve the desred layot, s learnng rate, s dscont factor calclated for each movement, R s reward that s gven only when one of desred layot s acheved n the man track. Defne s(t as the state at tme t, Tc as the sb-track selected as the destnaton for the removed car, p C as the nmber of classfed grops, q M as the movement conts of freght cars by drect rearrangement, and s as the state that follows s. In the drect rearrangement, 2a s defned as evalaton vales for (s, 2a, where s =[s, Go], s 2=[s, T C], and 2b s defned as evalaton vales for (s, 2b. Then, 2a(s,T C s pdated by the followng rle: 2a ( s, T C ( 2a ( s, T C V R (allcars moved nto the man track max3 ( s3, (collectve movement sfnshed 3 V 3 max2a ( s, (collectve movement scontned 2 a 2 a (2 In removal operatons, defne p M as the nmber of cars moved. 3, 4 and 5 are defned as evalaton vales for (s, 3,(s 3, 4,(s 4, 5 respectvely, where s 3=s, s 4=[s 3, c T], s 5=[s 4, T R]. 2b(s 4,T R, 3(s 3,c T 4(s 4, T R, and 5(s 5,p M are pdated by followng rles: ( s, T ( ( s, T V 2b 2 4 R V max 2a 2a ( s, 2a 2b 3( 3 T R s,c max ( s, (3 4( 4 R s, T max ( s, (4 6 c c 7 c c 6 9 c c c c 8 2 Fg. 8 Intal arrangement of cars n sb-tracks ( s, p 5 5 M ( 5 ( s5, pm V3, V3 = max 2a ( s', 2a ( s a rearrangement 2a (- (, R, 5 s5 pm V4 V4 max4 ( s4', 4 ( s a removal 4 (5 where s the learnng rate, R s the reward that s gven when one of desrable layot s acheved, and s the dscont factor that s sed to reflect the transfer dstance of locomotve and calclated by the followng eqaton: Dmax D,(0,0, (6 Dmax where D max s the maxmm vale of D. 2 Propagatng -vales by sng eqs.(-(6, -vales are dsconted accordng to the transfer dstance of locomotve. In other words, by selectng the removal destnaton that has the largest -vale, the transfer dstance of locomotve can be redced. In the learnng stages, each, ( h +h 2+h 3+h 4+m +mn{n p, n q} s selected by the soft-max acton selecton method [8] In the addressed problem, 2, 3, 4, 5 become smaller when the nmber of dsconts becomes larger. Then, for complex problems, the dfference between probabltes n canddate selecton reman small at the ntal state and large at fnal state before achevng desred layot, even after repettve learnng. In ths case, obtaned evalaton does not contrbte to selectons n ntal stage of marshalng process, and search movements to redce the transfer dstance of locomotve s spoled n fnal stage. To conqer ths drawback, 2a, 2b, 3, 4, 5, 6 are normalzed, and probablty P for selecton of each canddate s calclated by ( s, mn (, ~ s ( s, max ( s, mn ( s, P( s, ~ exp( ( s, / ~, exp( ( s, / ( 2,3,4,5 ~ exp( ( / P( ~, exp( ( / wherer s a thermo constant. IV. COMPUTER SIMUATIONS, (7 (8 Compter smlatons are condcted for m=2,n=6,k=36 and learnng performances of follown methods are compared: (A Proposed method consderng wth agmented collectve movements n the drect rearrangement, (B Proposed method wth conventonal collectve movements [7] (C Proposed method wthot collectve movements n the drect rearrangement [6]. The ntal arrangement of freght cars n sb-tracks tran s descrbed n Fg.8. The orgnal grop-layot s grop, grop 2, grop 3, grop 4, the order that s depcted n Fg.9. Also, cars c,, are n grop, c 0,,c 8 are n grop 2,,,7 are n grop 3,8,,6 are n grop 4. Other parameters are set as =0.9, =0.2, =0.9, R=.0, =0.. Both methods consder extended layot of grops and derve desrable layots atonomosly n order to redce the total transfer dstance of the locomotve. The locomotve and freght cars assmed to have the same length, and D mnv=d mnh=20m. ISBN: ISSN: (Prnt; ISSN: (Onlne IMECS 207

6 Proceedngs of the Internatonal MltConference of Engneers and Compter Scentsts 207 Vol I, IMECS 207, March 5-7, 207, Hong Kong c c 7 c 8 c 7 c 6 c c 0 c grop grop 2 grop 3 grop 4 Fg. 9 Orgnal grop-layot n the man track The reslts are shown n Fg.0. In the fgre, horzontal axs expresses the nmber of trals and the vertcal axs expresses the mnmm processng tme fond n the past trals. A tral starts wth an ntal arrangement n sb-tracks, and fnshed when all the freght cars are rearranged nto the man track. Each reslt s averaged over 20 ndependent smlatons. In Fg.0, althogh the learnng performance of method (A s nder preparng, (A ncldes all the collectve movements n (B so that the marshalng plan can be mproved as compared to that n (B. Also, the learnng performance of method (B s better than that of (C, becase (B collects freght cars n sb-tracks for each drect rearrangement. Smlaton reslts obtaned by (A wll be shown at the presentaton. Table Total transfer dstance Transfer dstance [m] best average worst Method(B Method(C [km] based on the renforcement learnng that evalates the total transfer dstance of locomotve n a marshalng. In order to redce the total transfer dstance of locomotve n a marshalng, the proposed method obtan grop-layot of desrable arrangement of otgong tran and collectng order of freght cars n sb-tracks, so that the total transfer dstance n a marshalng plan derved by the proposed method has been redced by abot 40% as compared to the conventonal method n compter smlatons. In addton, the arrangement of freght cars n the man track, the rearrange order of cars, the poston of each removal car, the nmber of cars to be removed, and the grop layot n the otgong tran has been obtaned smltaneosly so that the learnng performance of the proposed method has been mproved. REFERENCES [] C. F. Daganzo, R. G. Dowlng, R. W. Hall: Ralroad classfcaton yard throghpt: The case of mltstage tranglar sortng, Transportaton Research, Part A, No.7, Vol. 2 (983, pp [2] U. Blasm, M. R. Bsseck, W. Hochstättler, C. Moll, H.-H. Scheel and T. Wnter: Schedlng Trams n the Mornng, Mathematcal Methods of Operatons Research Vol. 49, No. (2000, pp [3] R. Jacob, P. Marton, J. Mae and M. Nnkesser: Mltstage Methods for Freght Tran Classfcaton, Proceedngs of 7th Workshop on Algorthmc Approaches for Transportaton Modelng, Optmzaton, and Systems (2007, pp [4] T. Nonner and A. Soza: Optmal Algorthms for Tran Shntng and Relaxed st Update Problems, 2th Workshop on Algorthmc Approaches for Transportaton Modellng, Optmzaton, and Systems (202, pp [5] Y. Hrashma: An Intellgent Tran Marshalng Based on the Processng Tme Consderng Grop ayot of Freght Cars n IAENG Transactons on Electrcal Engneerng Volme, World Scentfc (203, pp [6] Y. Hrashma: A Renforcement earnng System for Generatng Tran Marshalng Plan of Freght Cars Based on the Processng Tme Consderng Grop ayot, Proceedngs of The Internatonal MltConference of Engneers and Compter Scentsts 203, (203, pp [7] Y. Hrashma: A Renforcement earnng for Marshalng of Freght Tran Consderng Collectve Motons, Proceedngs of The Internatonal MltConference of Engneers and Compter Scentsts 206, (206, pp. pp9-24. [8] R. Stton and A. Barto: Renforcement earnng, MIT Press (999. Transfer dstance of locomotve (C (B Trals Fg. 0 Performance comparson V. CONCUSIONS A new schedlng method has been proposed n order to rearrange and lne cars n the desrable order onto the man track consderng agmented collectve movements of the locomotve n the drect rearrangement of freght cars. The learnng algorthm of the proposed method s derved ISBN: ISSN: (Prnt; ISSN: (Onlne IMECS 207