Adaptive and Coordinated Traffic Signal Control Based on Q-Learning and MULTIBAND Model

Size: px

Start display at page:

Download "Adaptive and Coordinated Traffic Signal Control Based on Q-Learning and MULTIBAND Model"

Britton Evans
5 years ago
Views:

1 Adpve nd Coordned Trffc Sgnl Conrol Bsed on Q-Lernng nd MULTIBAND Model Shoufeng Lu,. Trffc nd Trnsporon College Chngsh Unversy of Scence nd Technology Chngsh, Chn Xmn Lu, nd Shqng D. Shngh Insue of Appled Mhemcs nd Mechncs, Shngh Unversy Shngh, Chn Asrc Adpve nd coordned sgnl seng hs een reserch emphss. Cycle, spl, nd offse re hree mporn prmeers. Lessons nd expermens ou dpve sgnl conrol hve shon h cycle nd phse sequence renel nervl should e less hn 0 mnues. So frequenly opmzed prmeers re spl nd offse. For Weser sgnl seng heory, he ro eeen flo re o suron flo re s he deermnng prmeer. Bu hs ro s no sensve o smll flo re chnge. Therefore, he pper negres Q-lernng h Mulnd model o relze dpve nd coordned sgnl seng, n hch he former opmzes spl, he ler opmzes offse. Bsed on hs negred model, dpve nd coordned sgnl seng for he hree-nersecons rery s done. Keyords dpve, coordned, sgnl mng, Q-lernng, mulnd model I. INTRODUCTION The chrcersc of rffc flo s me-dependen. Adpve nd coordned sgnl conrol hs een reserch emphss. Sgnl seng mehods sepre rodly no o clsses. The frs clss consss of mehods h mxmze nddh nd progresson. Ths group develops from sngle rery o rerl neork, nd from unform nddh o vrle nddh. MAXBAND [] mxmze eghed comnon of he nddhs n he o drecons of he rerl y solvng mxed-neger lner progrmmng. MAXBAND s he se of susequen nddh opmzon models. MAXBAND cn no consder crossng sree sgnl opmzon. Ths defcency s redeemed n n exended verson of MAXBAND, he MAXBAND- model [] h cn hndle closed grd neorks of rerl srees. These o models genere unform nddh. By ncorporng no he model rffc-dependen creron, MULTIBAND [] clcules ndvdul nddhs for ech dreconl lnk of he rerl hle sll mnnng mn sree ploon progresson. The ndvdul nddhs depend on he cul rffc volumes h ech lnk crres, nd he resulng sgnl mng pln s lored o he vryng rffc flos long he rerl. Ths mehod s vlle only for sngle rerl prolems. MULTIBAND- [] produces vrle-dh progressons long ech rerl of he neork. The second group conns mehods h seek o mnmze dely, sops, fuel consumpon or oher mesures of dsuly. Exmples re he comnon mehod, TRANSYT, SCOOT. Trffc engneers prefer mxml nddh mehod over dsuly orened mehods ecuse hey hve cern nheren dvnges. For one hng, nddh mehods use relvely lle npu, he sc requremens eng sree geomery, rffc speeds, nd green spls. Secondly, progresson sysems re operonlly rous. Tme-spce dgrms le he rffc engneer vsulze esly he quly of he resuls. The drvers expec sgnl progresson nd ke s mesure of sgnl seng quly. Green me opmzon of ech nersecon s noher mporn componen. Convenonl mehod, such s Weser sgnl seng heory, s sed on prespecfed models of he envronmen. For Weser sgnl seng heory, he ro eeen flo re o suron flo re s he deermnng prmeer. Bu hs ro s no sensve o smll flo re chnge. Anoher mehod s lernng mehod sed on rfcl nellgence. Adulh e l. [] doped renforcemen lernng o formule dpve rffc sgnl conrol. Werng [] sed on Q-lernng o sudy rffc lgh conrol, nd pu forrd cr-sed vlue funcon. Werng er l. [] developed Green Lgh Dsrc rffc smulor sed on cr-sed renforcemen lernng lgorhm. Morry e l. [] dop dsrued rfcl nellgence o formule rffc conrol nd coordne lne chnges o mnn desred speeds. Thorpe, Anderson [] used renforcemen lernng o mnmze he me requred o dschrge fxed volume of rffc hrough rod neork, u hs pproch does no pper o e drecly pplcle o rel me rffc sgnl conrol. Bnghm [0] ppled renforcemen lernng n he conex of neuro-fuzzy pproch o rffc sgnl conrol, u me h lmed success due o he nsensvy of he pproch, lmed exploron n h s sochsc envronmen, nd off-lne pproch o vlue updng. Gregore e l. [] sed on lernng gens o opmze rffc conrol polcy. The mos sgnfcn dvnge of lernng mehod s h lernng mehod does no requre prespecfed model of he envronmen on hch o se con selecon. The pper negres Q-lernng h Mulnd model o relze dpve nd coordned sgnl seng, n hch he former opmzes spl, he ler opmzes offse. II. Q-LEARNING Lernng mehods rodly sepres no supervsed ----/0 /$ IEEE CIS 00

2 lernng nd unsupervsed lernng. Supervsed mchne lernng lgorhms requre lrge numer of exmples for rnng purposes. For unsupervsed mchne lernng lgorhm, knoledge s lerned hrough dynmc nercon h he envronmen. Q-lernng s unsupervsed, he oucome ssoced h kng prculr con n ny se encounered s lerned hrough dynmc rl-nd-error exploron of lernve cons nd oservon of he relve oucomes. Rher hn eng presened h lrge se of rnng exmples, he generon of hch s chnllegng sk n mny cses, even for domn exper, Q-lernng gen essenlly generes s on rnng experences from s envronmen. Q-lernng s dpve, n he sense h hey re cple of respondng o dynmclly chngng envronmen hrough ongong lernng nd dpon. For TRANSYT nd SCOOT sysem, opmzon mehod s hllclmng mehod. Q-lernng mehod hs more lrger con spce hn TRANSYT nd SCOOT opmzon mechnsm. The nercon eeen gen nd envronmen s llusred s Fg. []. se s rerd r r + s + Agen Envronmen rerd Fgure. Agen-envronmen nercon n renforcemen lernng Q Lernng Algorhm [] goes s follo: () Se prmeer γ, nd envronmen rerd mrx R () Inlze mrx Q s zero mrx () For ech epsode: Selec rndom nl se Do hle no rech gol se Selec one mong ll possle cons for he curren se Usng hs possle con, consder o go o he nex se Ge mxmum Q vlue of hs nex se sed on ll possle cons Compue Q( se, con) = R( se, son) + γ Mx[ Q( nex se, ll cons)] Se he nex se s he curren se If Q vlue ncrese, or nervl ermne, lernng ermne. End Do End For III. MODELING ADAPTIVE SIGNAL CONTROL BASED ON Q-LEARNING A. Modelng he Envronmen For rffc sgnl conrol, he envronmen s rffc flo. Adulh e l. [] dop queue lengh s se nformon, nd dely s rerd, hch s cheved y vedo mgng echnology. Becuse no ll he nersecon ll nsll vedo mgng equpmen, so queue lengh nformon cn no e oned for ll nersecons. Lu Zhyong e l. [] dop operng speed s se, nd supposed funcon s rerd, hch hs no physcl menng. Dely heores hve een developed for mny yers, hch hve een mure. Three ypcl dely heores re sedy se dely heory, deermnsc dely heory, rnson curve dely heory. For rnson curve heory hs more dply, hs een ppled dely. For exmple, TRANSYT() used rnson curve heory o clcule dely. In hs pper, e dop rnson curve heory o esme dely. B. Se, Acon, nd Rerd Defnon The se nformon s ol dely of he nersecon. Accordng o he experence of dpve rffc sgnl conrol, frequenly updng cycle ll cuse he flucuon of rffc flo, nd he loss of rffc flo flucuon s lrger hn mprovemen of sgnl seng. So Q-lernng focuses on he opmzon of green me. For every cycle, sgnl conrol gen ll dop con. Acon ses re he comnon of ech phse green me chnge. Snce he ddon of green me chnge scle ll drmclly ncreses he sze of con spce, lnce hs o e sough eeen he enef of hs nformon nd s mpc on prolem rcly. For exmple, for four-phse nersecon f ech phse green me chnge s seconds, con ses hve ccc = cons. If ech phse green me chnge s seconds nd seconds, con ses hve ccc = cons. For convenence, ech phse green me chnge s seconds n hs pper. The defnon of rerd s ol dely of he nersecon. For rffc sgnl conrol, rerd s he penly. C. Exploron Polces For rffc sgnl conrol, here re no defne gol se. When here re rffc flo, sgnl conrol gen ll lys opmze sed on Q-lernng. For con selecon, e dop greedy selecon sregy. IV. INTEGRATED Q-LEARNING AND MULTIBAND MODEL Inegred MULTIBAND nd Q-lernng model s Mxed Ineger Lner Progrmmng, hch s s follong: Fnd,,,, z, m, δ, δ o mx α+ α, =,, n () sujec o q α =,,, S q α =,,, S z T T = n () = n () () + r,,, n + r,,, = () = n ()

3 r nd r re deermned y Q-lernng. ( + ) ( ) + ( + ) + δl δl δ+ l+ + δ+ l+ m, =, n = ( r r) + ( τ + τ ) + + n ( T ) ( ) ( ) = = TF F F (), =,, n () 0. = + () s rvel me of he fses vehcle. (0),, z,,,, 0, =, n () m = n eger () δ, δ re 0-vrles () q s lnk flo re. S s suron flo re. z s he recprocl of cycle lengh T. ( ) s ouound (nound) nddh. r( r ) s ouound (nound) red me. The un s cycles. ( ) s me from rgh (lef) sde of red o lef (rgh) edge of ouound (nound) green nd. ( ) s mhemcl expecon of rvel me from nersecon o +. l( l ) s ouound(nound) lef urn green me. τ( τ ) s queue clernce me, n dvnce of he ouound (nound) nddh. The un of he ove vrles s sgnl cycle. V. EXPERIMENT A. Bsc D The pper opmzes nddh for he rery of hree nersecons h negred MULTIBAND nd Q-lernng model. Lyou of nersecon s llusred y Fg.. Suron flo re of hrough lne s 0veh/hr, suron flo re of oher ype lne s 0veh/hr. Fgure. Lyou of nersecon. The deled rffc d of he frs -mnue me nervl s llusred n Tle []. For he second -mnue me nervl, flo re of ll pprochs ncrese 0 veh/hr hn frs -mnue me nervl. For he hrd -mnue me nervl, flo re of ll pproch ncrese 0 veh/hr hn he second -mnue me nervl. TABLE I. 0m 00m DETAILED TRAFFIC DATA AND LAYOUT OF EACH INTERSECTION Inerseco n Wesound Approch Esound Approch Norhound Approch Souhound Approch Lef Turn Srgh Rgh Turn 0 0 Tol 0 Lne Funcon Inerseco n One Lef Lne, One Srgh Lne, One Srgh nd Rgh Lne. Wesound Approch One Srgh nd Lef Lne, One Srgh Lne, One Rgh Lne Esound Approch One Lef Lne, One Srgh nd Rgh Lne Norhound Approch Lef Turn 0 0 Srgh 0 Rgh Turn 0 One Srgh, Lef nd Rgh Lne Souhound Approch Tol 00 Lne One Lef One Lef One One Srgh, Funcon Lne, One Lne, One Srgh, Lef nd Srgh Lne, Srgh Lef nd Rgh Lne One Srgh Lne, One Rgh Lne nd Rgh Rgh Lne Lne. Inerseco n Wesound Approch Esound Approch Norhound Approch Lef Turn 0 Srgh Rgh Turn 0 Souhound Approch Tol 0 Lne One Lef One Lef One One Srgh, Funcon Lne, One Lne, One Srgh nd Lef nd Srgh Lne, Srgh lef lne, Rgh Lne One Srgh Lne, One One nd Rgh Rgh Lne Srgh nd Lne. Rgh Lne B. Mhemcl Expecon of Trvel Tme Wh Trffc Flo Dsperson Prmeer seng: mnmum speed s km/hr, mxmum speed s 0km/hr. Mhemcl expecon of rvel me T eeen nersecon nd s ( ) TF F. s rvel me of he fses vehcle, = T = 0 ( T 0) 0m 0 0 km / hr = s. F = = =s. Mhemcl expecon of rvel me eeen T nersecon nd s ( ) TF F. s rvel me of he fses vehcle, = T = 0 00m 0 0 km / hr = s. F = =. ( T ) =s. C. Green Spls Clculon Green spls s compued y usng he heory of Weser. Weser hs shon h under cern crcumsnces, ol dely n nersecon s mnmzed y dvdng he vlle.

4 cycle me mong compeng srems of rffc proporonl o her volumes dvded y her cpces. Le TRAT()=hrough rffc ro of volume o cpcy n drecon. LRAT()=lef urn rffc ro of volume o cpcy n drecon. =OUT, IN, OUTC, INC=ouound mn, nound mn, ouound cross sree, nound cross sree. MAIN=mx{TRAT(OUT)+LRAT(IN),TRAT(IN)+LRAT( OUT)}=he lrger of hrough volume/cpcy plus oppose lef urn volume/cpcy for he o drecons on he mn sree. CROSS=mx{TRAT(OUTC)+LRAT(IN),TRAT(INC)+LR AT(OUTC)} The sc spl eeen mn sree nd cross sree s MAIN MM= =green spl lloced o mn sree. CC= MAIN + CROSS CROSS MAIN + CROSS =green spl lloced o cross sree. Le L(OUT) [L(IN)]=ouound [nound] lef spl. G(OUT) [G(IN)]=ouound [nound] hrough spl. LRAT ( OUT ) LRAT ( IN) Then L(OUT)= MM, L(IN)= MAIN G(OUT)=MM-L(IN), G(IN)=MM-L(OUT). For nersecon, MAIN = mx +, CROSS = mx{ +, + } = MM = = L(IN)= L(OUT)= MM MAIN, =0.. =0.0, =0.0, G(OUT)=MM-L(IN)=0., G(IN)=MM-L(OUT)=0.. For nersecon, 0 MAIN = mx{ +, + } = CROSS = mx{ +, + } = MM = = L(IN)= = G(IN)=0.-0.0=0.. For nersecon,, L(OUT)= MAIN = mx{ +, + } = CROSS = mx{ +, + } =, G(OUT)=0.-0.0=0., =0.,, 0. MM = = L(IN)= = G(IN)=MM-L(OUT)=0.., L(OUT)= =, G(OUT)=MM-L(IN)=0., D. Bnddh Opmzon h MULTIBAND Model For he Frs -mnues Tme Inervl MULTIBAND model s mxed neger lner progrmmng, hch cn e solved y Brnch nd Bound mehod. In hs pper, hs model s solved y Ml progrmmng. The correspondng mxed neger lner progrmmng model s mx{ } () + r = 0. () + r = 0. () + r = 0. () + r = 0. () + r = 0. () + r = 0. (0) ( + ) ( + ) + z+ 0.0δ 0.0δ 0.0δ + 0.0δ m = z+ (0.0 0.) () ( + ) ( + ) + z+ 0.0δ 0.0δ 0.δ + 0.0δ m = z+ (0. 0.0) () 0.00 z 0.0,, z,,,, 0,,,,,,, () δ δ = () m m δ δ re negers, =,, () δ δ () The resuls re = 0., = 0., = 0., = 0., = 0., = 0., = 0.00, z = 0.0, δ =, δ δ δ =, δ =, δ =, m =, m =. So cycle lengh s =s, = s, = s, = s, z = s, = 0 s, = s, = 0. Opmzed lef urn green phse of nersecon s ouound lef lgs nd nound leds. Rgh of y sch s llusred n Fg.. Fgure. Rgh of y of nersecon Opmzed lef urn green phse of nersecon s ouound lef leds, nound lgs. Rgh of y sch s llusred n Fg.., Fgure. Rgh of y of nersecon

5 Opmzed lef urn green phse of nersecon s ouound lef lgs, nound lgs. Rgh of y sch s llusred n Fg.. Fgure. Rgh of y of nersecon E. Spl nd Bnddh Opmzon Bsed on Inegred Q-lernng nd MULTIBAND Model For he Second nd Thrd -mnues Tme Inervl For he second -mnues me nervl, flo re of ll enrnce ncreses 0 veh/hr. Inl sgnl seng s solved n Pr D. For nersecon, nl green me of ech phse s (,,0), crcl flo re s veh/hr, veh/hr, veh/hr. Inl ol dely s veh-s/cycle. For nersecon, nl green me of ech phse s (,,), crcl flo re s 0veh/hr, veh/hr, veh/hr. Inl ol dely s 0veh-s/cycle. For nersecon, nl green me of ech phse s (0,0,), crcl flo re s veh/hr, veh/hr, veh/hr. Inl ol dely s veh-s/cycle. TABLE II. Q-LEARNING FOR ACTION SELECTION OF INTERSECTION FOR FIRST TIME INTERVAL Selecon (,,) TABLE III. Q-LEARNING FOR ACTION SELECTION OF INTERSECTION FOR FIRST TIME INTERVAL Selecon (,,0) TABLE IV. Q-LEARNING FOR ACTION SELECTION OF INTERSECTION FOR FIRST TIME INTERVAL Selecon (,,) (,,0) (,,) (,0,) 0 0 (,,) (0,,) (0,,) Bnddh opmzon for he frs greedy selecon s: = s, = s, = s, = s, = s, = s, = 0. For he hrd -mnues me nervl, flo re of ll enrnce ncreses 0veh/hr more. Inl sgnl seng s he resul of frs -mnues me nervl. For nersecon, nl green me of ech phse s (,,), crcl flo re s veh/hr, veh/hr, 0veh/hr. Inl ol dely s veh-s/cycle. For nersecon, nl green me of ech phse s (,,0), crcl flo re s 00veh/hr, veh/hr, veh/hr. Inl ol dely s veh-s/cycle. For nersecon, nl green me of ech phse s (,,), crcl flo re s veh/hr, veh/hr, veh/hr. Inl ol dely s 0veh-s/cycle. TABLE V. Q-LEARNING FOR ACTION SELECTION OF INTERSECTION FOR SECOND TIME INTERVAL Selecon (,,) TABLE VI. Q-LEARNING FOR ACTION SELECTION OF INTERSECTION FOR SECOND TIME INTERVAL Selecon (,,) TABLE VII. Q-LEARNING FOR ACTION SELECTION OF INTERSECTION FOR SECOND TIME INTERVAL Acon (,0,0) (,,) (,0,) (,,) (,,) (,,) Acon Dely(vehs) (,,) (,,0) (,,0) (,,) (,,) (,,). Ech phse green me. Acon Dely(vehs) (,,) (,,) (,,) 0 (,,) (,,) (,,) Acon Dely(vehs) (,,) (,,) (,,) 0 (,,) 0 (,,) (,,0) Acon Dely(vehs) (,,) 0 (,,0) 0 (,,0) 0 (,,) (,,) 0 (,,0) 0 Acon Dely(vehs) (,,) (,,) (,,) Acon Dely(vehs) (,0,) (,,) (,,) Acon Dely(vehs) (,,) (,,) (,,) (,,) (,,0) (,,) Acon Dely(vehs) (,,0) (,,) (,,) 0 (,,0) (,,0) (,,) Acon Dely(vehs) (,,) (,,0) (,,0) (,,) (,,) (,,) Acon Dely(vehs) (,,) (,,) (,,) 0 (,,) (,,) (,,) Dely(vehs) unfesle Acon Dely(vehs) (,,) (,,) (,,) 0 (0,0,) 0 (,,) (,0,) Selecon (,,) Bnddh opmzon for he second greedy selecon s:

6 = s, = s, = s, = s, = s, = s, =, m = m =. Tme nd spce dgrm of opmzed nddh s llusred n Fg.. 0 Fgure. Tme spce dgrm of nddh opmzon for he second -mnues nervl. VI. CONCLUSION The pper negres Q-lernng h MULTIBAND model o relze dpve nd coordned sgnl conrol. In hs negred model, Q-lernng s used o opmze spl, nd MULTIBAND model s used o opmze offse. In comprson o Weser sgnl seng heory, Q-lernng cn respond o grdul chnge of flo re. From he numercl expermen, e kno h sgnl mng genered y Q- lernng cn reduce dely s/cycle mos. The cycle n hs pper s s. Eqully, Q-lernng cn reduce mnues per hour n dely. Ths hs sgnfcn prccl menng for reducng rffc congeson. Bsed on sgnl mng genered y Q-lernng, MULTIBAND model opmzes offse. Therefore, hs negred model cheves dpve nd coordned sgnl conrol. ACKNOWLEDGMENT Ths ork s suppored y NSFC (Grn No.0000), Nonl Bsc Reserch Progrm of Chn (Grn No.00CB000), NSFC (Grn No.000), nd Tlen Recrumen Foundon of Chngsh Unversy of Scence nd Technology (Grn No.000). REFERENCES [] Lle, J. D. C., M. D. Kelson, nd N. H. Grner. MAXBAND: A Progrm for Seng Sgnls on Areres nd Trngulr Neorks, Trnsporon Reserch Record, TRB, Nonl Reserch Councl, Wshngon, D.C., pp. 0,. [] Chng E.C.P., S.L.Cohem,C. Lu, N.A. Chudhry, nd C.Messer. MAXBAND-: A Progrm for Opmzng Lef-Turn Phse Sequence n Mulrerl Closed Neorks, Trnsporon Reserch Record, TRB, Nonl Reserch Councl, Wshngon, D.C. pp.-,. [] Grner,N.H., S.F.Assmnn, F.Lsg, nd D.L.Hou. A Mul-Bnd Approch o Arerl Trffc Sgnl Opmzon, Trnsporon Reserch, Vol., No., pp.-,. [] Chrons Smds, Nhn H.Grner, Mulnd-: A Progrm for Vrle Bnddh Progresson Opmzon of Mulrerl Trffc Neorks, Trnsporon Reserch Record, TRB, Nonl Reserch Councl, Wshngon, D. C., pp.,. [] Bher Adulh, Ro Prngle, nd Grgors J.Krkouls., Renforcemen Lernng for True Adpve Trffc Sgnl Conrol, Journl of Trnsporon Engneerng, pp-, My/June, 00. [] Mrco Werng, Mul-Agen Renforcemen Lernng for Trffc Lgh Conrol, Proceedng of he h Inernonl Conference on Mchne Lernng, 000. [] Mrco Werng, Jelle vn Veenen, Jlles Vreeken, Arne Koopmn, Inellgen Trffc Lgh Conrol, echncl repor UU-CS-00-0, nsue of nformon nd compung scences, urech unversy. [] Dvd E.Morry, Smon Hndley, nd P Lngley, Lernng Dsrued Sreges for Trffc Conrol, Proceedngs of he Ffh Inernonl Conference of he Socey for Adpve Behvor, Zurch, Szerlnd, pp. -,. [] Thorpe,T., Anderson, C., Trffc Lgh Conrol Usng SARSA h Three Se Represenons, IBM Corporon, Boulder,. [0] Ell Bnghm, Renforcemen Lernng n Neurofuzzy Trffc Sgnl Conrol, Europen Journl of Operonl Reserch, pp.-,00. [] Perre-Luc Gregore, Chrles Desjrdns, Brhm Ch-dr, Julen Lumoner, Urn Trffc Conrol Bsed on Lernng Agens, The 0h Inernonl IEEE Conference on Inellgen Trnsporon Sysems, Sep. 0 - Oc., 00, Hlon Hoel, Sele, Wshngon, USA. [] Rchrd S.Suon nd Andre G.Bro, Renforcemen Lernng: An Inroducon, The MIT Press, Cmrdge, Msschuses, London, Englnd, pp-,. [] Teknomo, Krd. Q-Lernng y Exmples. hp://people.revoledu.com/krd/uorl/renforcemenlernng/ndex. hml, 00. [] Lu Zhyong, M Fenge, On-lne Renforcemen Lernng Conrol for Urn Trffc Sgnls. Proceedngs of he h Chnese Conrol Conference, July -, 00, Zhngjje, Hunn, Chn. [] Chng Yuno, Peng Guoxong, Urn Arerl Rod Coordne Conrol Bsed on Genec Algorhm, Journl of Trffc nd Trnsporon Engneerng, Vol., No., pp.0-, 00. [] Qun Yongshen, Urn Trffc Conrol, RenMn Communcon Press, Bejng, Chn,.

Hidden Markov Model. a ij. Observation : O1,O2,... States in time : q1, q2,... All states : s1, s2,..., sn

Hidden Markov Model. a ij. Observation : O1,O2,... States in time : q1, q2,... All states : s1, s2,..., sn Hdden Mrkov Model S S servon : 2... Ses n me : 2... All ses : s s2... s 2 3 2 3 2 Hdden Mrkov Model Con d Dscree Mrkov Model 2 z k s s s s s s Degree Mrkov Model Hdden Mrkov Model Con d : rnson roly from