An Optimal Control Approach to the Multi-agent Persistent Monitoring Problem

Transcription

1 An Opmal Conrol Approach o he Mul-agen Perssen Monorng Problem Chrsos.G. Cassandras, Xuchao Ln and Xu Chu Dng Dvson of Sysems Engneerng and Cener for Informaon and Sysems Engneerng Boson Unversy, cgc@bu.edu, mmxcln@bu.edu, xcdng@bu.edu Absrac We presen an opmal conrol framewor for perssen monorng problems where he objecve s o conrol he movemen of mulple cooperang agens o mnmze an uncerany merc n a gven msson space. In a one-dmensonal msson space, we show ha he opmal soluon s for each agen o move a maxmal speed from one swchng pon o he nex, possbly wang some me a each pon before reversng s drecon. Thus, he soluon s reduced o a smpler paramerc opmzaon problem: deermnng a sequence of swchng locaons and assocaed wang mes a hese swchng pons for each agen. Ths amouns o a hybrd sysem whch we analyze usng Infnesmal Perurbaon Analyss IPA o oban a complee on-lne soluon hrough a graden-based algorhm. We also show ha he soluon s robus wh respec o he uncerany model used. Ths esablshes he bass for exendng hs approach o a wo-dmensonal msson space. I. INTRODUCTION Enabled by recen echnologcal advances, he deploymen of auonomous agens ha can cooperavely perform complex ass s rapdly becomng a realy. In parcular, here has been consderable progress repored n he leraure on robocs and sensor newors regardng coverage conrol [], [2], [3], survellance [4], [5] and envronmenal samplng [6], [7] mssons. In hs paper, we are neresed n generang opmal conrol sraeges for perssen monorng ass; hese arse when agens mus monor a dynamcally changng envronmen whch canno be fully covered by a saonary eam of avalable agens. Perssen monorng dffers from radonal coverage ass due o he perpeual need o cover a changng envronmen,.e., all areas of he msson space mus be vsed nfnely ofen. The man challenge n desgnng conrol sraeges n hs case s n balancng he presence of agens n he changng envronmen so ha s covered over me opmally n some well-defned sense whle sll sasfyng sensng and moon consrans. Examples of perssen monorng mssons nclude survellance, parol mssons wh unmanned vehcles, and envronmenal applcaons where roune samplng of an area s nvolved. In hs paper, we address he perssen monorng problem by proposng an opmal conrol framewor o drve agens so as o mnmze a merc of uncerany over he envronmen. In coverage conrol [2], [3], s common o model nowledge The auhors wor s suppored n par by NSF under Grans EFRI and CNS-2392, by AFOSR under gran FA , by DOE under gran DE-FG52-6NA2749, by ONR under gran N and by ARO under gran W9NF of he envronmen as a non-negave densy funcon defned over he msson space, and usually assumed o be fxed over me. However, snce perssen monorng ass nvolve dynamcally changng envronmens, s naural o exend hs model o a funcon of boh space and me o capure uncerany n he envronmen. We assume ha uncerany a a pon grows n me f s no covered by any agen sensors. To model sensor coverage, we defne a probably of deecng evens a each pon of he msson space by agen sensors. Thus, he uncerany of he envronmen decreases wh a rae proporonal o he even deecon probably,.e., he hgher he sensng effecveness s, he faser he uncerany s reduced.. Whle s desrable o rac he value of uncerany over all pons n he envronmen, hs s generally nfeasble due o compuaonal complexy and memory consrans. Movaed by pollng models n queueng heory, e.g., spaal queueng [8],[9], and by sochasc flow models [], we assgn samplng pons of he envronmen o be monored perssenly hs s equvalen o paronng he envronmen no a dscree se of regons. We assocae o hese pons uncerany queues whch are vsed by one or more servers. The growh n uncerany a a samplng pon can hen be vewed as a flow no a queue, and he reducon n uncerany when covered by an agen can be vewed as he queue beng vsed by moble servers as n a pollng sysem. Moreover, he servce flow raes depend on he dsance of he samplng pon o nearby agens. From hs pon of vew, we am o conrol he movemen of he servers agens so ha he oal accumulaed uncerany queue conen s mnmzed. Conrol and moon plannng for agens performng perssen monorng ass have been suded n he leraure. In [] he focus s on sweep coverage problems, where agens are conrolled o sweep an area. In [6], [] a smlar merc of uncerany s used o model nowledge of a dynamc envronmen. In [], he samplng pons n a one-dmensonal envronmen are denoed as cells, and he opmal conrol polcy for a wo-cell problem s gven. Problems wh more han wo cells are addressed by a heursc polcy. In [6], he auhors proposed a sablzng speed conroller for a sngle agen so ha he accumulaed uncerany over a gven pah n he envronmen s bounded, along wh an opmal conroller ha mnmzes he maxmum seady-sae uncerany, assumng ha he agen ravels along a closed pah and does no change drecon. The perssen monorng problem s

2 2 also relaed o robo parol problems, where a eam of robos are requred o vs pons n he worspace wh frequency consrans [2], [3], [4]. Alhough one-dmensonal perssen monorng problems are of neres n her own rgh e.g., see [5], our ulmae goal s o opmally conrol a eam of cooperang agens n a wo or hree-dmensonal envronmen. The conrbuon of hs paper s o ae a frs sep oward hs goal by formulang and solvng an opmal conrol problem for a eam of agens movng n a one-dmensonal msson space descrbed by an nerval [, L] R n whch we mnmze he accumulaed uncerany over a gven me horzon and over an arbrary number of samplng pons. Even n hs smple case, deermnng a complee explc soluon s compuaonally hard, as seen n [6] where he sngle-agen case was frs consdered. However, we show ha he problem can be reduced o a paramerc opmzaon problem. In parcular, he opmal rajecory of each agen s o move a full speed unl reaches some swchng pon, dwell on he swchng pon for some me possbly zero, and hen swch drecons. In addon, we prove ha all agens should never reach he end pons of he msson space [, L]. Thus, each agen s opmal rajecory s fully descrbed by a se of swchng pons {θ,..., θ K } and assocaed wang mes a hese pons, {w,..., w K }. As a resul, we show ha he behavor of he agens operang under opmal conrol s descrbed by a hybrd sysem. Ths allows us o mae use of generalzed Infnesmal Perurbaon Analyss IPA, as presened n [7],[8], o deermne gradens of he objecve funcon wh respec o hese parameers and subsequenly oban opmal swchng locaons and wang mes ha fully characerze an opmal soluon. I also allows us o explo robusness properes of IPA o exend hs soluon approach o a sochasc uncerany model. Our analyss esablshes he bass for exendng hs approach o a wo-dmensonal msson space n ongong research. In a broader conex, our approach brngs ogeher opmal conrol, hybrd sysems, and perurbaon analyss echnques n solvng a class of problems whch, under opmal conrol, can be shown o behave le hybrd sysems characerzed by a se of parameers whose opmal values delver a complee opmal conrol soluon. The res of he paper s organzed as follows. Secon II formulaes he opmal conrol problem. Secon III characerzes he soluon of he problem n erms of wo parameer vecors specfyng swchng pons n he msson space and assocaed dwellng mes a hem. Usng IPA n conjuncon wh a graden-based algorhm, a complee soluon s also provded. Secon IV provdes some numercal resuls and Secon V concludes he paper. II. PERSISTENT MONITORING PROBLEM FORMULATION We consder N moble agens movng n a -dmensonal msson space of lengh L, for smplcy aen o be an nerval [, L] R. Le he poson of he agens a me be s n [, L], n =,..., N, followng he dynamcs: ṡ n = u n.e., we assume ha he agen can conrol s drecon and speed. Whou loss of generaly, afer some rescalng wh he sze of he msson space L, we furher assume ha he speed s consraned by u n, n =,..., N. For he sae of generaly, we nclude he addonal consran: a s b, a, b L 2 over all o allow for msson spaces where he agens may no reach he end pons of [, L], possbly due o he presence of obsacles. We also pon ou ha he agen dynamcs n can be replaced by a more general model of he form ṡ n = g n s n b n u n whou affecng he man resuls of our analyss see also Remar n he nex secon. Fnally, an addonal consran may be mposed f we assume ha he agens are nally locaed so ha s n < s n, n =,..., N, and we wsh o preven hem from subsequenly crossng each oher over all : s n s n 3 We assocae wh every pon x [, L] a funcon p n x, s n ha measures he probably ha an even a locaon x s deeced by agen n. We also assume ha p n x, s n = f x = s n, and ha p n x, s n s monooncally nonncreasng n he dsance xs n beween x and s n, hus capurng he reduced effecveness of a sensor over s range whch we consder o be fne and denoed by r n hs s he same as he concep of sensor fooprn found n he robocs leraure. Therefore, we se p n x, s n = when x s n > r n. Alhough our analyss s no affeced by he precse sensng model p n x, s n, we wll lm ourselves o a lnear decay model as follows: { xs n p n x, s n = r n, f x s n r n 4, f x s n > r n Nex, consder a se of pons {α }, =,..., M, α [, L], and assocae a me-varyng measure of uncerany wh each pon α, whch we denoe by R. Whou loss of generaly, we assume α α M L and, o smplfy noaon, we se p n, s n p n α, s n. Ths se may be seleced o conan pons of neres n he envronmen, or sampled pons from he msson space. Alernavely, we may consder a paron of [, L] no M nervals whose cener pons are α = 2L 2M, =,..., M. We can hen se p nx, s n = p n, s n for all x [α L 2M, α L 2M ]. Therefore, he jon probably of deecng an even a locaon x [α L 2M, α L 2M ] by all he N agens smulaneously assumng deecon ndependence s: P s = Q [ p n, s n ] 5 where we se s = [s,..., s N ] T. We defne uncerany funcons R assocaed wh he nervals [α L 2M, α L 2M ], =,..., M, so ha hey have he followng properes: R ncreases wh a prespecfed rae A f P s =, R decreases wh a fxed rae B f P s = and R for all. I s hen naural o model uncerany so ha s decrease s proporonal o he

3 3 Fgure. A s Bp s A queueng sysem analog of he perssen monorng problem. probably of deecon. In parcular, we model he dynamcs of R, =,..., M, as follows: { f R =, A Ṙ = BP s A BP s oherwse 6 where we assume ha nal condons R, =,..., M, are gven and ha B > A > hus, he uncerany srcly decreases when here s perfec sensng P s =. Vewng perssen monorng as a pollng sysem, each pon α equvalenly, h nerval n [, L] s assocaed wh a vrual queue where uncerany accumulaes wh nflow rae A smlar models have been used n some daa havesng problems, e.g., [9]. The servce rae of hs queue s mevaryng and gven by BP s, conrollable hrough he agen poson a me. Fgure llusraes hs pollng sysem when N =. Ths nerpreaon s convenen for characerzng he sably of such a sysem over a msson me T : For each queue, we may requre ha T A < T Bp sd. Alernavely, we may requre ha each queue becomes empy a leas once over [, T ]. We may also mpose condons such as R T R max for each queue as addonal consrans for our problem so as o provde bounded uncerany guaranees, alhough we wll no do so n hs paper. Noe ha hs analogy readly exends o wo or hree-dmensonal sengs. The goal of he opmal perssen monorng problem we consder s o conrol he movemen of he N agens hrough u n n so ha he cumulave uncerany over all sensng pons {α }, =,..., M s mnmzed over a fxed me horzon T. Thus, seng u = [u,..., u N ] we am o solve he followng opmal conrol problem P: mn u J = T T R d 7 = subjec o he agen dynamcs, uncerany dynamcs 6, conrol consran u n, [, T ], and sae consrans 2, [, T ]. Noe ha we requre a r n and b L r m, for a leas some n, m =,..., N; hs s o ensure ha here are no pons n [, L] whch can never be sensed,.e., any such ha α < a r n or α > b r n would always le ousde any agen s sensng range. We wll om he addonal consran 3 from our nal analyss, bu we wll show ha, when s ncluded, he opmal soluon never allows o be acve. III. OPTIMAL CONTROL SOLUTION We frs characerze he opmal conrol soluon of problem P and show ha can be reduced o a paramerc opmzaon problem. Ths allows us o ulze an Infnesmal Perurbaon Analyss IPA graden esmaon approach [7] o fnd a complee opmal soluon hrough a graden-based algorhm. We defne he sae vecor x = [s,..., s N, R,..., R M ] T and he assocaed cosae vecor λ = [λ s,..., λ sn, λ,..., λ M ] T. In vew of he dsconnuy n he dynamcs of R n 6, he opmal sae rajecory may conan a boundary arc when R = for some ; oherwse, he sae evolves n an neror arc. We frs analyze he sysem operang n such an neror arc and om he consran 2 as well. Usng and 6, he Hamlonan s N H x, λ, u = R λ sn u n λ Ṙ = = 8 and he cosae equaons λ = H x are λ = H =, R =,..., M 9 λ sn = H n = B r n [ p d, s d ] λ Ϝ n d n B [ p d, s d ] r n Ϝ n λ d n where we have used 4, and he ses Ϝ n and Ϝ n are defned as Ϝ n = { : s n r n α s n } Ϝ n = { : s n < α s n r n } for n =,..., N. Noe ha Ϝ n, Ϝ n denfy all pons α o he lef and rgh of s n respecvely ha are whn agen n s sensng range. Snce we mpose no ermnal sae consrans, he boundary condons are λ T =, =,..., M and λ sn T =, n =,..., N. Applyng he Ponryagn mnmum prncple o 8 wh u, [, T, denong an opmal conrol, we have H x, λ, u = mn H x, λ, u u n [,],,...,N and s mmedaely obvous ha s necessary for an opmal conrol o sasfy: { u f λsn < n = 2 f λ sn > Ths condon excludes he possbly ha λ sn = over some fne sngular nervals [2]. We wll show ha f s n = a > or s n = b < L, hen λ sn = for some n {,..., N} may n fac be possble for some fne arc; oherwse λ sn = can arse only when u n =. The mplcaon of 9 wh λ T = s ha λ = T for all [, T ] and all =,..., M and ha λ s

4 4 monooncally decreasng sarng wh λ = T. However, hs s only rue f he enre opmal rajecory s an neror arc,.e., all R consrans for all =,..., M reman nacve. On he oher hand, loong a, observe ha when he wo end pons, and L, are no whn he range of an agen, we have Fn = F n, snce he number of ndces sasfyng s n r n α s n s he same as ha sasfyng s n < α s n r n. Consequenly, for he one-agen case N =, becomes λ s = B B λ λ 3 r r F F and λ s = snce he wo erms n 3 wll cancel ou,.e., λ s remans consan as long as hs condon s sasfed and, n addon, none of he sae consrans R, =,..., M, s acve. Thus, for he one agen case, as long as he opmal rajecory s an neror arc and λ s <, he agen moves a maxmal speed u = n he posve drecon owards he pon s = b. If λ s swches sgn before any of he sae consrans R, =,..., M, becomes acve or he agen reaches he end pon s = b, hen u = and he agen reverses s drecon or, possbly, comes o res. In wha follows, we examne he effec of he sae consrans whch sgnfcanly complcaes he analyss, leadng o a challengng wo-pon-boundary-value problem. However, we wll esablsh he fac ha he complee soluon bols down o deermnng a se of swchng locaons over [a, b] and wang mes a hese swchng pons, wh he end pons, and L, beng always nfeasble on an opmal rajecory. Ths s a much smpler problem ha we are subsequenly able o solve. We begn by recallng ha he dynamcs n 6 ndcae a dsconnuy arsng when he condon R = s sasfed whle Ṙ = A BP s < for some =,..., M. Thus, R = defnes an neror boundary condon whch s no an explc funcon of me. Followng sandard opmal conrol analyss [2], f hs condon s sasfed a me for some j {,..., M}, H x, λ, u = H x, λ, u 4 where we noe ha one can choose o se he Hamlonan o be connuous a he enry pon of a boundary arc or a he ex pon. Usng 8 and 6, 4 mples: N λ s n u n λ j [A j BP j s] = N λ s n u n 5 In addon, λ s n = λ s n for all n =,..., N and λ = λ for all j, bu λ j may experence a dsconnuy so ha: λ j = λ j π j 6 where π j s a mulpler assocaed wh he consran R j. Recallng 2, snce λ s n remans unaffeced, so does he opmal conrol,.e., u n = u n. Moreover, snce hs s an enry pon of a boundary arc, follows from 6 ha A j BP j s <. Therefore, 5 and 6 mply ha λ j = and λ j = π j. Thus, λ always decreases wh consan rae unl R = s acve, a whch pon λ jumps o a non-negave value π and decreases wh rae agan. The value of π s deermned by how long aes for he agens o reduce R o once agan. Obvously, λ, =,..., M, [, T ] 7 wh equaly holdng only f = T, or = wh R =, R >, where [ δ,, δ >. The acual evaluaon of he cosae vecor over he nerval [, T ] requres solvng, whch n urn nvolves he deermnaon of all pons where he sae varables R reach her mnmum feasble values R =, =,..., M. Ths generally nvolves he soluon of a wo-pon-boundary-value problem. However, our analyss hus far has already esablshed he srucure of he opmal conrol 2 whch we have seen o reman unaffeced by he presence of boundary arcs when R = for one or more =,..., M. We wll nex prove some addonal srucural properes of an opmal rajecory, based on whch we show ha s fully characerzed by a se of non-negave scalar parameers. Deermnng he values of hese parameers s a much smpler problem ha does no requre he soluon of a wo-pon-boundary-value problem. Le us urn our aenon o he consrans s n a and s n b and consder frs he case where a =, b = L,.e., he agens can move over he enre [, L]. We shall mae use of he followng echncal condon: Assumpon : For any n =,..., N, =,..., M,, T, and any ɛ >, f s n =, s n ɛ >, hen eher R > for all [ ɛ, ] or R = for all [ɛ, ]; f s n = L, s n ɛ < L, hen eher R > for all [ ɛ, ] or R = for all [ ɛ, ]. Ths condon excludes he case where an agen reaches an endpon of he msson space a he exac same me ha any one of he uncerany funcons reaches s mnmal value of zero. Then, he followng proposon assers ha neher of he consrans s n and s n L can become acve on an opmal rajecory. The assumpon s used only n Proposon III. for echncal reasons and does no aler he srucure of he opmal conroller. Proposon III.: Under Assumpon, f a =, b = L, hen on an opmal rajecory: s n and s n L for all, T, n {,..., N}. Proof. Suppose a = < T an agen reaches he lef endpon,.e., s n =, s n >. We wll hen esablsh a conradcon. Thus, assumng s n =, we frs show ha λ s n = by a conradcon argumen. Assume ha λ s n, n whch case, snce he agen s movng oward s n =, we have u n = and λ sn > from 2. Then, λ s n may experence a dsconnuy so ha λ s n = λ sn πn 8 where π n s a scalar consan. I follows ha λ s n = λ s n πn >. Snce he consran s n = s no an

5 5 explc funcon of me, we have λ s n u n = λ sn u n 9 On he oher hand, u n, snce agen n mus eher come o res or reverse s moon a s n =, hence λ s n u n. Ths volaes 9, snce λ s n u n <. Ths conradcon mples ha λ s n =. Nex, consder and observe ha n we have Fn =, snce α > s n = for all =,..., M. Therefore, recallng 7, follows from ha λ sn = B r n Ϝ n [ ] λ pd, s d d n Under Assumpon, here exss δ > such ha durng he nerval δ, no R becomes acve, hence no λ encouners a jump for =,..., M. I follows ha λ > for F n and λ s n s connuous wh λ s n > for δ,. Agan, snce s n =, here exss some δ 2 δ such ha for δ 2,, we have u n < and λ s n. Thus, for δ 2,, we have λ s n and λ s n >. Ths conradcs he esablshed fac ha λ s n = and we conclude ha s n for all [, T ], n =,..., N. Usng a smlar lne of argumen, we can also show ha s n L. Proposon III.2: If a > and or b < L, hen on an opmal rajecory here exs fne lengh nervals [, ] such ha s n = a and or s n = b, for some n {,..., N}, [, ], < T. Proof. Proceedng as n he proof of Proposon III., when s n = a we can esablsh 9 and he fac ha λ s n =. On he oher hand, u n, snce he agen mus eher come o res or reverse s moon a s n = a. In oher words, when s n = a on an opmal rajecory, 9 s sasfed eher wh he agen reversng s drecon mmedaely n whch case = and λ s n = or sayng on he boundary arc for a fne me nerval n whch case > and u n = for [, ]. The exac same argumen can be appled o s n = b. The nex resul esablshes he fac ha on an opmal rajecory, every agen eher moves a full speed or s a res. Proposon III.3: On an opmal rajecory, eher u n = ± f λ s n, or u n = f λ s n = for [, T ], n =,..., N. Proof. When λ s n, we have shown n 2 ha u n = ±, dependng on he sgn of λ s n. Thus, remans o consder he case λ s n = for some [, 2 ], where < 2 T. Snce he sae s n a sngular arc, λ s n does no provde nformaon abou u n. On he oher hand, he Hamlonan n 8 s no a explc funcon of me, herefore, seng H x, λ, u H, we have dh d =, whch gves dh d N = Ṙ λ s n u n = N λ s n u n λ Ṙ = λ R = 2 = Defne S = {n λ sn =, n =,..., N} as he se of ndces of agens ha are n a sngular arc and S = {n λ sn, n =,..., N} as he se of ndces of all oher agens. Thus, λ s n =, λ sn = for [, 2 ], n S. In addon, agens move wh consan full speed, eher or, so ha u n =, n S. Then, 2 becomes dh d = [ λ ]Ṙ = n S λ s n u n λ R = 2 = From 9, λ =, =,..., M, so λ =, leavng only he las wo erms above. Noe ha λ sn = H n and wrng R = dṙ d we ge: n S u n H n M =,R λ dṙ d = Recall from 6 ha when R we have Ṙ = A N B[ [ p s n ]], so ha H n = B dṙ d = B =,R N λ p s n n u n p s n n N p s d 22 d n N p s d 23 d n

6 6 whch resuls n B λ = R B = B = R n S N λ λ = R u n p s n n u n p s n n n S N p s d d n N p s d d n u n p s n n N p s d = 24 d n Noe ha ps n = ± r or, dependng on he relave poson of s wh respec o α. Moreover, 24 s nvaran o M or he precse way n whch he msson space [, L] s paroned, whch mples ha λ n S u n p s n n N p s d = d n for all =,..., M, [, 2 ]. Snce λ =, =,..., M, s clear ha o sasfy hs equaly we mus have u n = for all [, 2 ], n S. In concluson, n a sngular arc wh λ s n = for some n {,..., N}, he opmal conrol s u n =. Nex, we consder he case where he addonal sae consran 3 s ncluded. We can hen prove ha hs consran s never acve on an opmal rajecory,.e., agens reverse her drecon before mang conac wh any oher agen. Proposon III.4: If he consran 3 s ncluded n problem P, hen on an opmal rajecory, s n s n for, T ], n =,..., N. Proof. Suppose a = < T we have s n = s n, for some n =,..., N. We wll hen esablsh a conradcon. Frs assumng ha boh agens are movng as opposed o one beng a res oward each oher, we have u n = and u n =. From 2 and Prop III.3, we now λ s n < and λ sn >. When he consran s n s n s acve, λ s n and λ s n may experence a dsconnuy so ha λ s n = λ sn π λ s n = λ sn π 25 where π s a scalar consan. I follows ha λ s n = λ s n π < and λ sn = λ sn π >. Snce he consran s n s n s no an explc funcon of me, we have λ s n u n λ sn u n = λ s n u n λ sn u n 26 On he oher hand, u n and u n, snce agens n and n mus eher come o res or reverse her moon afer mang conac, hence λ s n u n λ sn u n. Ths volaes 26, snce λ s n u n λ sn u n <. Ths conradcon mples ha s n s n = canno be acve and we conclude ha s n s n for [, T ], n =,..., N. Moreover, f one of he wo agens s a res when s n = s n, he same argumen sll holds snce s sll rue ha λ s n u n λ sn u n <. Based on hs analyss, he opmal conrol u n depends enrely on he sgn of λ s n and, n lgh of Proposons III.-III.3, he soluon of he problem reduces o deermnng: swchng pons n [, L] where an agen swches from u n = ± o eher or ; or from u n = o eher ±, and f an agen swches from u n = ± o, wang mes unl he agen swches bac o a speed u n = ±. In oher words, he full soluon s characerzed by wo parameer vecors for each agen n: θ n = [θ n,,..., θ n,γn ] T and w n = [w n,..., w n,γn ] T, where θ n,ξ, L denoes he ξh locaon where agen n changes s speed from ± o and w n,ξ denoes he me whch s possbly null ha agen n dwells on θ n,ξ. Noe ha Γ n s generally no nown a pror and depends on he me horzon T. In addon, we always assume ha agen n reverses s drecon afer leavng he swchng pon θ n,ξ wh respec o he one had when reachng θ n,ξ. Ths seemngly excludes he possbly of an agen s conrol followng a sequence,, or,,. However, hese wo moon behavors can be capured as wo adjacen swchng pons approachng each oher: when θ n,ξ θ n,ξ, he agen conrol follows he sequence,, or,,, and he wang me assocaed wh u n = s w n,ξ w n,ξ. For smplcy, we wll assume ha s n =, so ha follows from Proposon III. ha u n =, n =,..., N. Therefore, θ n, corresponds o he opmal conrol swchng from o. Furhermore, θ n,ξ wh ξ odd even always corresponds o u n swchng from o o. Thus, we have he followng consrans on he swchng locaons for all ξ = 2,..., Γ n : { θn,ξ θ n,ξ, f ξ s even 27 θ n,ξ θ n,ξ, f ξ s odd. I s now clear ha he behavor of each agen under he opmal conrol polcy s ha of a hybrd sysem whose dynamcs undergo swches when u n changes from ± o and from o or when R reaches or leaves he boundary value R =. As a resul, we are faced wh a paramerc opmzaon problem for a sysem wh hybrd dynamcs. Ths s a seng where one can apply he generalzed heory of Infnesmal Perurbaon Analyss IPA n [7],[8] o convenenly oban he graden of he objecve funcon J n 7 wh respec o he vecors θ and w, and herefore, deermne generally, locally opmal vecors θ and w hrough a graden-based opmzaon approach. Noe ha hs s done on lne,.e., he graden s evaluaed by observng a rajecory wh gven θ and w over [, T ] based on whch θ and w are adjused unl convergence s aaned usng sandard graden-based algorhms. Remar. If he agen dynamcs n are replaced by a model such as ṡ n = g n s n b n u n, observe ha

7 7 2 sll holds. The dfference les n whch would nvolve a dependence on dgnsn ds n and furher complcae he assocaed wo-pon-boundary-value problem. However, snce he opmal soluon s also defned by parameer vecors θ n = [θ n,,..., θ n,γn ] T and w n = [w n,..., w n,γn ] T for each agen n, we can sll apply he IPA approach presened n he nex secon. A. Infnesmal Perurbaon Analyss IPA Our analyss hus far has shown ha, on an opmal rajecory, he agen moves a full speed, dwells on a swchng pon possbly for zero me and never reaches eher boundary pon,.e., < s n < L. Thus, he nh agen s movemen can be parameerzed hrough θ n = [θ n,,..., θ n,γn ] T and w n = [w n,..., w n,γn ] T where θ n,ξ s he ξh conrol swchng pon and w n,ξ s he wang me for hs agen a he ξh swchng pon. Therefore, he soluon of problem P reduces o he deermnaon of opmal parameer vecors θ n and w n, n =,..., N. As we poned ou, he agen s opmal behavor defnes a hybrd sysem, and he swchng locaons ranslae o swchng mes beween parcular modes of hs sysem. Ths s smlar o swchng-me opmzaon problems, e.g., [2],[22],[23], excep ha we can only conrol a subse of mode swchng mes. We mae use of IPA n par o explo robusness properes ha he resulng gradens possess [24]; specfcally, we wll show ha hey do no depend on he uncerany model parameers A, =,..., M, and may herefore be used whou any dealed nowledge of how uncerany affecs he msson space. Sngle-agen soluon wh a = and b = L: To manan some noaonal smplcy, we begn wh a sngle agen who can move on he enre msson space [, L] and wll hen provde he naural exenson o mulple agens and a msson space lmed o [a, b] [, L]. We presen he assocaed hybrd auomaon model for hs sngle-agen sysem operang on an opmal rajecory. Our goal s o deermne Jθ, w, he graden of he objecve funcon J n 7 wh respec o θ and w, whch can hen be used n a graden-based algorhm o oban opmal parameer vecors θ n and w n, n =,..., N. We wll apply IPA, whch provdes a formal way o oban sae and even me dervaves wh respec o parameers of hybrd sysems, from whch we can subsequenly obanng Jθ, w. Hybrd auomaon model. We use a sandard defnon of a hybrd auomaon e.g., see [25] as he formalsm o model he sysem descrbed above. Thus, le q Q a counable se denoe he dscree sae or mode and x X R n denoe he connuous sae. Le υ Υ a counable se denoe a dscree conrol npu and u U R m a connuous conrol npu. Smlarly, le δ a counable se denoe a dscree dsurbance npu and d D R p a connuous dsurbance npu. The sae evoluon s deermned by means of a vecor feld f : Q X U D X, an nvaran or doman se Inv : Q Υ 2 X, a guard se Guard : Q Q Υ 2 X, and v a rese funcon r : Q Q X Υ X. The sysem remans a a dscree sae q as long as he connuous medrven sae x does no leave he se Invq, υ, δ. If x reaches a se Guardq, q, υ, δ for some q Q, a dscree ranson can ae place. If hs ranson does ae place, he sae nsananeously reses o q, x where x s deermned by he rese map rq, q, x, υ, δ. Changes n υ and δ are dscree evens ha eher enable a ranson from q o q by mang sure x Guardq, q, υ, δ or force a ranson ou of q by mang sure x / Invq, υ, δ. We wll classfy all evens ha cause dscree sae ransons n a manner ha sus he purposes of IPA. Snce our problem s se n a deermnsc framewor, δ and d wll no be used. We show n Fg. 2 a paral hybrd auomaon model of he sngle-agen sysem where a = and b = L. Snce here s only one agen, we se s = s, u = u and θ = θ for smplcy. Due o he sze of he overall model, Fg. 2 s lmed o he behavor of he agen wh respec o a sngle α, {,..., M} and gnores modes where he agen dwells on he swchng pons hese, however, are ncluded n our exended analyss n Secon III-A2. The model consss of 4 dscree saes modes and s symmerc n he sense ha saes 7 correspond o he agen operang wh u =, and saes 84 correspond o he agen operang wh u =. Saes where u = are omed snce we do no nclude he wang me parameer w = w here. The evens ha cause sae ransons can be placed n hree caegores: The value of R becomes and rggers a swch n he dynamcs of 6. Ths can only happen when R > and Ṙ = A Bp s < e.g., n saes 3 and 4, causng a ranson o sae 7 n whch he nvaran condon s R =. The agen reaches a swchng locaon, ndcaed by he guard condon s = θ ξ for any ξ =,..., Γ. In hese cases, a ranson resuls from a sae z o z7 f z =,..., 6 and o z7 oherwse. The agen poson reaches one of several crcal values ha affec he dynamcs of R whle R >. Specfcally, when s = α r, he value of p s becomes srcly posve and Ṙ = A Bp s >, as n he ranson 2. Subsequenly, when s = α r A /B, as n he ranson 2 3, he value of p s becomes suffcenly large o cause Ṙ = A Bp s < so ha a ranson due o R = becomes feasble a hs sae. Smlar ransons occur when s = α, s = α r A /B, and s = α r. The laer resuls n sae 6 where Ṙ = A > and he only feasble even s s = θ ξ, ξ odd, when a swch mus occur and a ranson o sae 3 aes place smlarly for sae 8. IPA revew. Before proceedng, we provde a bref revew of he IPA framewor for general sochasc hybrd sysems as presened n [7]. The purpose of IPA s o sudy he behavor of a hybrd sysem sae as a funcon of a parameer vecor θ Θ for a gven compac, convex se Θ R l. Le { θ}, =,..., K, denoe he occurrence mes of all evens n he sae rajecory. For convenence, we se = and K = T. Over an nerval [ θ, θ, he sysem s a some mode durng whch he me-drven sae sasfes ẋ = f x, θ,. An even a s classfed as Exogenous f causes a dscree sae ranson ndependen of θ and sasfes d dθ = ; Endogenous, f here exss a connuously dfferenable funcon g : R n Θ R such ha = mn{ > : g x θ,, θ = }; and

8 8 2 2 R f, s s r, R 5 s R f, s s, R 4 s R f, s s, R 3 s R f, s rs, R 2 s r R s = s= = R = 2n- 2n- s 2 R, 6 7 As R, s s r, R R s = 2n- s r R As, s r, R s = 2n- s = 2n- s = 2n- s= 2n- s= s= 2n 2n 3 4 R As, R, s s r, R R s = 2n R As, s r, R 8 s r s= 2n s = 2n s = 2n 2 R s 2 R s R f2, s f2, s f, s s r, R s2, R s, R 2 R = R = s s s r s = 2n R f, s rs, R 9 s s K K where f A B, f2 A B, r, 2 r r r K2 K2 Fgure 2. Hybrd auomaon for each α. Red arrows represen evens when he conrol swches beween and. Blue arrows represen evens when R becomes. Blac arrows represen all oher evens. Induced f s rggered by he occurrence of anoher even a me m. IPA specfes how changes n θ nfluence he sae xθ, and he even mes θ and, ulmaely, how hey nfluence neresng performance mercs whch are generally expressed n erms of hese varables. Gven θ = [θ,..., θ Γ ] T, we use he Jacoban marx noaon: x xθ, θ, θ θ, =,..., K, for all sae and even me dervaves. I s shown n [7] ha x sasfes: d d x = f x x f θ for [, wh boundary condon: 28 x = x [ f f ] 29 for =,..., K. In addon, n 29, he graden vecor for each s = f he even a s exogenous and [ ] g = x f g θ g x x 3 f he even a s endogenous.e., g x θ,, θ =, defned as long as g x f. IPA equaons. To clarfy he presenaon, we frs noe ha =,..., M s used o ndex he pons where uncerany s measured; ξ =,..., Γ ndexes he componens of he parameer vecor; and =,..., K ndexes even mes. In order o apply he hree fundamenal IPA equaons 28-3 o our sysem, we use he sae vecor x = [s, R,..., R M ] T and parameer vecor θ = [θ,..., θ Γ ] T. We hen denfy all evens ha can occur n Fg. 2 and consder nervals [ θ, θ over whch he sysem s n one of he 4 saes shown for each =,..., M. Applyng 28 o s wh f = or due o and 2, he soluon yelds he graden vecor s = [ θ,..., θ M ] T, where =, for [, 3 for all =,..., K,.e., for all saes z {,..., 4}. Smlarly, le R = [ R θ,..., R θ M ] T for =,..., M. We noe from 6 ha f = for saes z Z {7, 4}; f = A for saes z Z 2 {, 6, 8, 3}; and f = A Bp s for all oher saes whch we furher classfy no Z 3 {2, 3,, 2} and Z 4 {4, 5, 9, }. Thus, solvng 28 and usng 3 gves: R = R { f z Z Z 2 B s oherwse where ps ps 32 = ± r as evaluaed from 4 dependng on he sgn of α s a each assocaed auomaon sae. Deals on he dervaon of a smple recursve expresson for R above can be found n Appendx A. Objecve Funcon Graden Evaluaon. Based on our analyss, he objecve funcon 7 n problem P can now be wren as Jθ, a funcon of θ nsead of u and we can

9 9 rewre as Jθ = T K = = θ θ R, θ d where we have explcly ndcaed he dependence on θ. We hen oban: Jθ = K R d T == R R 33 Observng he cancelaon of all erms of he form R for all wh =, K = T fxed, we fnally ge Jθ = T K θ == θ R d. 34 The evaluaon of Jθ herefore depends enrely on R, whch s obaned from n Appendx A and he observable even mes, =,..., K, gven nal condons s =, R for =,..., M and R =. Snce R self depends only on he even mes, =,..., K, he graden Jθ s obaned by observng he swchng mes n a rajecory over [, T ] characerzed by he vecor θ. 2 Mul-agen soluon where a and b L: Nex, we exend he resuls obaned n he prevous secon o he general mul-agen problem where we also allow a and b L. Recall ha we requre a r n and Lr m b L, for a leas some n, m =,..., N snce, oherwse, conrollng agen movemen canno affec R for all α locaed ousde he sensng range of agens. We now nclude boh parameer vecors θ n = [θ n,,..., θ n,γn ] T and w n = [w n,,... w n,γn ] T for each agen n and, for noaonal smplcy, concaenae hem o consruc θ = [θ,..., θ N ] T and w = [w,..., w N ] T. The soluon of problem P reduces o he deermnaon of opmal parameer vecors θ and w and we wll use IPA o evaluae Jθ, w = [ djθ,w djθ,w dθ dw ]T. Smlar o 34, s [ ] T clear ha hs depends on R = R R θ w and he even mes, =,..., K, observed on a rajecory over [, T ] wh gven θ and w. IPA equaons. We begn by recallng he dynamcs of R n 6 whch depend on he relave posons of all agens wh respec o α and change a me nsans such ha eher R = wh R > or A > BP s wh R =. Moreover, usng and our earler Hamlonan analyss, he dynamcs of s n, n =,..., N, n an opmal rajecory can be expressed as follows. Defne Θ n,ξ = θ n,ξ, θ n,ξ f ξ s odd and Θ n,ξ = θ n,ξ, θ n,ξ f ξ s even o be he ξh nerval beween successve swchng pons for any n =,..., N, where θ n, = s n. Then, for ξ =, 2,..., s n Θ n,ξ, ξ odd ṡ n = s n Θ n,ξ, ξ even 35 oherwse where ransons for s n from ± o are ncorporaed by reang hem as cases where w n,ξ =,.e., no dwellng a a swchng pon θ n,ξ n whch case ṡ n =. We can now concenrae on all evens causng swches eher n he dynamcs of any R, =,..., M, or he dynamcs of any s n, n =,..., N. From 29, any oher even a some me n [ hs hybrd sysem ] canno modfy he values of T [ ] T R = R R θ w or sn = n n θ n w n a =. Frs, applyng 28 o s n wh f =, or due o 35, he soluon yelds s n = s n, for [, 36 for all =,..., K, n =,..., N. Smlarly, applyng 28 o R and usng 6 gves: R = R { f R =, A < BP s G n oherwse and R = R w n,ξ w n,ξ { f R =, A < BP s G n w n,ξ oherwse where G = B d n p s d ps n n Deals on he dervaon of smple recursve expressons for he componens of s n and R n can be found n Appendx B. Objecve Funcon Graden Evaluaon. Proceedng as n he evaluaon of Jθ n Secon III-A, we are now neresed n mnmzng he objecve funcon Jθ, w n 7 wh respec o θ and w and we can oban Jθ, w = [ djθ,w dθ djθ,w dw ]T as Jθ, w = T K = = θ,w θ,w R d Ths depends enrely on R, whch s obaned from 37 and 38 and he even mes, =,..., K, gven nal condons s n = a for n =,..., N, and R for R and 52 found n Appendx B, whereas n =,..., M. In 37, s obaned hrough 5 s obaned hrough 36 and 49, 57, 63 found n Appendx B. In 38, R w n,ξ s agan obaned hrough 5 and 52, whereas n w n,ξ s obaned hrough 49 and 67 whch are also found n Appendx B. Remar 2. Observe ha he evaluaon of R, hence Jθ, w, s ndependen of A, =,..., M,.e., he values n our uncerany model. In fac, he dependence of R on A, =,..., M, manfess self hrough he even mes, =,..., K, ha do affec hs evaluaon, bu hey, unle A whch may be unnown, are drecly observable durng he graden evaluaon process. Thus, he IPA approach possesses an nheren robusness propery: here s no need

10 o explcly model how uncerany affecs R n 6. Consequenly, we may rea A as unnown whou affecng he soluon approach he values of R are obvously affeced. We may also allow hs uncerany o be modeled hrough random processes {A }, =,..., M; n hs case, however, he resul of Proposon III.3 no longer apples whou some condons on he sascal characerscs of {A } and he resulng Jθ, w s an esmae of a sochasc graden. B. Objecve Funcon Opmzaon We now see o oban θ and w mnmzng Jθ, w hrough a sandard graden-based opmzaon scheme of he form [θ l, w l ] T = [θ l, w l ] T [η θ, η w ] Jθ l, w l 39 where {ηθ l }, {ηl w} are approprae sep sze sequences and Jθ l, w l s he projecon of he graden Jθ l, w l ono he feasble se he se of θ l sasfyng he consran 27, a θ l b, and w l. The opmzaon scheme ermnaes when Jθ, w < ε for a fxed hreshold ε for some θ and w. Our IPA-based algorhm o oban θ and w mnmzng Jθ, w s summarzed n Algorhm where we have adoped he Armjo mehod n sep-sze selecon see [26] for { [ ηθ l, w] ηl }. One of he unusual feaures n 39 s he fac ha he dmenson Γ n of θn and wn s a pror unnown depends on T. Thus, he algorhm mus mplcly deermne hs value along wh θn and wn. One can search over feasble values of Γ n {, 2,...} by sarng eher wh a lower bound Γ n = or an upper bound o be found. The laer approach resuls n much faser execuon and s followed n Algorhm. An upper bound s deermned by observng ha θ n,ξ s he swchng pon where agen n changes speed from o for ξ odd and from o for ξ even. By seng hese wo groups of swchng pons so ha her dsance s suffcenly small and wang mes w n = for each agen, we deermne an approxmae upper bound for Γ n as follows. Frs, we dvde he feasble space [a, b] evenly no N nervals: [a n N b a, a n N b a], n =,..., N. Defne D n = a 2n 2N b a o be he geomerc cener of each nerval and se θ n,ξ = D n σ f ξ s even and θ n,ξ = D n σ f ξ s odd, so ha he dsance beween swchng pons θ n,ξ for ξ odd and even s 2σ, where σ > s an arbrarly small number, n =,..., N. In addon, se w n =. Then, T mus sasfy θ n, s n 2σ Γ n T θ n, s n 2σΓ n 4 n =,..., N, where Γ n s he number of swchng pons agen n can reach durng, T ], gven θ n,ξ as defned above. From 4 and nong ha Γ n s an neger, we have Γ n = 2σ [T θ n, s n ] 4 where s he celng funcon. Clearly, reducng σ ncreases he nal number of swchng pons Γ n assgned o agen n and Γ n as σ. Therefore, σ s seleced suffcenly small whle ensurng ha he algorhm can be execued suffcenly fas. As Algorhm repeas seps 3-6, w n,ξ and dsances beween θ n,ξ for ξ odd and even generally ncrease, so ha he number of swchng pons agen n can acually reach whn T decreases. In oher words, as long as σ s suffcenly small hence, Γ n s suffcenly large, when he algorhm converges o a local mnmum and sops, here exss ζ n < Γ n, such ha θ n,ζn s he las swchng pon agen n can reach whn, T ], n =,..., N. Observe ha here generally exs ξ such ha ζ n < ξ Γ n whch correspond o pons θ n,ξ ha agen n canno reach whn, T ]; he assocaed dervaves of he cos wh respec o such θ n,ξ are obvously, snce perurbaons o hese θ n,ξ wll no affec s n,, T ] and hus he cos Jθ, w. When Jθ, w < ɛ, we acheve a local mnmum and sop, a whch pon he dmenson of θn and wn s ζ n. Algorhm : IPA-based opmzaon algorhm o fnd θ and w : Pc σ > and ɛ >. 2: Defne D n = a 2n b a, n =,..., N, and se { 2N θn,ξ = D n σ f ξ even θ n,ξ = D n σ f ξ odd. Se w = [w,..., w N ] =., where w n = [w n,,..., w n,ξn ] and Γ n = 2σ [T θ n, s n ] 3: repea 4: Compue s n, [, T ] usng s n, 2, θ and w for n =,..., N 5: Compue Jθ, w and updae θ, w hrough 39 6: unl Jθ, w < ɛ ] ] 7: Se θn = [θn,,..., θ and w n,ζn n = [w n,,..., w, n,ζn where ζ n s he ndex of θ n,ζn, whch s he las swchng pon agen n can reach whn, T ], n =,..., N IV. NUMERICAL EXAMPLES In hs secon we presen some examples of perssen monorng problems n whch agen rajecores are deermned usng Algorhm. The frs four are sngle-agen examples wh L = 2, M = 2, α =, α M = 2, and he remanng samplng pons are evenly spaced over [, 2]. The sensng range n 4 s se o r = 4, he nal values of he uncerany funcons n 6 are R = 4 for all, and he me horzon s T = 4. In Fg. 3a we show resuls where he agen s allowed o move over he enre space [, 2] and he uncerany model s seleced so ha B = 3 and A =. for all =,..., 2, whereas n Fg. 3b he feasble space s lmed o [a, b] wh a = r = 4 and b = L r = 6. The op plo n each example shows he opmal rajecory s obaned, whle he boom shows he cos Jθ l, w l as a funcon of eraon number. In Fg. 4, he rajecores n Fg. 3a,b are magnfed for he nerval [, 75] o emphasze he presence of srcly posve wang mes a he swchng pons. In Fg. 3c we show resuls for a case smlar o Fg. 3a excep ha he values of A are seleced so ha A = A 2 =

11 2 Agen poson vs. me 2 Agen poson vs. me Agen poson 5 5 Agen poson 5 5 Cos J Tme Cos J vs. Number of eraons Armjo sep 8 consan sep Number of eraons a a =, b = 2. A =., =,..., 2. J = Cos J Tme Cos J vs. Number of eraons Number of eraons b a = 4, b = 6. A =., =,..., 2. J = Agen poson vs. me 2 Agen poson vs. me Agen poson 5 5 Agen poson 5 5 Cos J Tme Cos J vs. Number of eraons 3 2 Cos J Tme Cos vs. Number of Ieraons 5 Armjo sep consan sep Number of eraons Number of Ieraons c a =, b = 2. A = A 2 =.5, A =., =,..., 9. J = d a =, b = 2. A U.75,.25,.e.. J = Fgure 3. One agen example. L = 2, T = 4. For each example, op plo: opmal rajecory; boom plo: J versus eraons..5, whle A =., =,..., 9. Noe ha he wang mes a he swchng pons are now longer and even hough seems ha he swchng pons are a he wo end pons, hey are acually very close bu no equal o hese end pons, conssen wh Proposon III.. In Fg. 3d, on he oher hand, he values of A are allowed o be random, hus dealng wh a perssen monorng problem n a sochasc msson space, where we can es he robusness of he proposed approach. In parcular, each A s reaed as a pecewse consan random process {A } such ha A aes on a fxed value sampled from an unform dsrbuon over.75,.25 for an exponenally dsrbued me nerval wh mean before swchng o a new value. Noe ha he behavor of he sysem n hs case s very smlar o Fg. 3a where A =. for all =,..., 2 whou any change n he way n whch Jθ l, w l s evaluaed n execung 39. As already poned ou, hs explos a robusness propery of IPA whch maes he evaluaon of Jθ l, w l ndependen of he values of A. In general, however, when A s mevaryng, Proposon III.3 may no longer apply, snce an exra erm A would be presen n 24. In such a case, u n may be nonzero when λ n = and he deermnaon of an opmal rajecory hrough swchng pons and wang mes alone may no longer be possble. In he case of 3d, A changes suffcenly slowly o manan he valdy of Proposon III.3 over relavely long me nervals, under he assumpon ha w.p. no even me concdes wh he jump mes n any {A }. In all cases, we nalze he algorhm wh σ = 5 and ε = 2. The runnng mes of Algorhm are approxmaely sec usng Armjo sep-szes. Noe ha alhough he number of eraons for he examples shown may vary subsanally, he acual algorhm runnng mes do no. Ths s smply because he Armjo sep-sze mehod may requre several rals per eraon o adjus he sep-sze n order o acheve an adequae decrease n cos. In Fg. 3a,d, he red lne shows he cos as a funcon of eraon number usng a consan sep sze and he wo lnes converge o he same approxmae opmal value. Non-smoohness n Fg. 3d comes from he fac ha s a sochasc process. Noe ha n all cases he nal cos s sgnfcanly reduced ndcang he mporance of opmally selecng he values of he swchng pons and assocaed wang mes f any. Fgure 5 shows wo wo-agen examples wh L = 4, M = 4 and evenly spaced samplng pons over [, L], A =., B = 3, r = 4, R = 4 for all and T = 4. In Fg. 5a he agens are allowed o move over he whole msson space [, L], whle n Fg. 5b hey are only allowed o move over

12 a a =, b = b a = 4, b = 6. Fgure 4. Magnfed rajecory for sub-fgure a and b n Fg. 3, [, 75]. [a, b] where a = r and b = L r. We nalze he algorhm wh he same σ and ε as before. The algorhm runnng me s approxmaely 5 sec usng Armjo sep-szes, and we observe once agan sgnfcan reducons n cos. V. CONCLUSION We have formulaed an opmal perssen monorng problem wh he objecve of conrollng he movemen of mulple cooperang agens o mnmze an uncerany merc n a gven msson space. In a one-dmensonal msson space, we have shown ha he opmal soluon s reduced o he deermnaon of wo parameer vecors for each agen: a sequence of swchng locaons and assocaed wang mes a hese swchng pons. We have used Infnesmal Perurbaon Analyss IPA o oban sensves of he objecve funcon wh respec o all he parameers and, herefore, oban a complee on-lne locally opmal soluon hrough a gradenbased algorhm. We have also shown ha he soluon s robus wh respec o he uncerany model used. Our ongong wor ams a ncorporang consrans such as R T R max o he problem formulaon, hus ensurng ha an opmal perssen monorng soluon provdes ceran performance guaranees. We are also nvesgang he use of recedng horzon conrollers ha provde compuaonally fas approxmae soluons. Fnally, our wor o dae has esablshed he bass for exendng hs approach o a wo-dmensonal msson space. Specfcally, one dea s o decompose such a wo-dmensonal msson space no regons each one of whch s monored by agens movng on a one-dmensonal rajecory, hus ang drec advanage of he resuls n hs paper. APPENDIX A IPA DERIVATION FOR SINGLE-AGENT SOLUTION In order o deermne s and R whch are needed o evaluae R n 32, we use 29, whch nvolves he even me graden vecors = [ θ,..., θ Γ ] T for =,..., K he value of K depends on T. Loong a Agen poson Cos J Agen poson Cos J Agen poson vs. me Tme Cos J vs. Number of eraons Number of eraons a a =, b = 2. J = Agen poson vs. me Tme Cos J vs. Number of eraons Number of eraons b a = 4, b = 6. J = Fgure 5. Two agen example. L = 4, T = 4. Top plo: opmal rajecory. Boom plo: J versus eraons. Fg. 2, here are hree readly dsngushable cases regardng he evens ha cause dscree sae ransons: Case : An even a me whch s neher R = nor s = θ ξ, for any ξ =,..., Γ. In hs case, s easy o see ha he dynamcs of boh s and R are connuous, so ha f = f n 29 appled o s and R, =,..., M gves: { s = s R = R, =,..., M 42 Case 2: An even R = a me. Ths corresponds o ransons 3 7, 4 7, 4 and 4 n Fg. 2 where he dynamcs of s are sll connuous, bu he dynamcs of R swch from f = A Bp s o f =. Thus, s = s, bu we need o evaluae o deermne R. Observng ha hs even s endogenous, 3 apples wh g = R = and we ge R θ = ξ A Bp s, ξ =,..., Γ, =,..., K I follows from 29 ha R R [A Bp s R = ] θ ξ A Bp s =

13 3 Thus, whenever an even occurs a such ha R R becomes zero, s always rese o regardless of. R Case 3: An even a me due o a conrol sgn change a s = θ ξ, ξ =,..., Γ. Ths corresponds o any ranson beween he upper and lower par of he hybrd auomaon n Fg. 2. In hs case, he dynamcs of R are connuous and = R for all, ξ,. On he oher = ±. Observng ha any such even s endogenous, 3 apples wh g = s θ ξ = for some ξ =,..., Γ and we ge = u 43 we have R hand, we have ṡ = u = u Combnng 43 wh 29 and recallng ha u = u, we have θ = ξ θ [u u θ ] ξ ξ u = 2 where = because = = for all [,, snce he poson of he agen canno be affeced by θ ξ pror o hs even. In hs case, we also need o consder he effec of perurbaons o θ j for j < ξ,.e., pror o he curren even me clearly, for j > ξ, = snce he curren poson of he agen canno be affeced by fuure evens. Observe ha snce g = s θ ξ =, we have g = for j ξ and 3 gves = u, so ha usng hs n 29 we ge: [ θ = u j θ u ] j u = 44 Combnng he above resuls, he componens of s where s he even me when s = θ ξ for some ξ, are gven by = 2 f j = ξ f j > ξ f j =,..., ξ 45 I follows from 3 and he analyss of all hree cases above ha for all ξ s consan hroughou an opmal rajecory excep a ransons caused by conrol swchng locaons Case 3. In parcular, for he h even correspondng o s = θ ξ, [, T ], f u =, hen = 2 f ξ s odd, and = 2 f ξ s even; smlarly, f u =, hen = 2 f ξ s odd and = 2 f ξ s even. In summary, we can wre: { ξ 2u =, ξ =,..., Γ 46 < Fnally, we can combne 46 wh our resuls for R n all hree cases above. Leng s l = θ ξ, we oban he followng expresson for R for all l, [, : R = R f z Z Z 2 ξ 2B r u f z Z 3 ξ 2B r u f z Z 4 47 wh boundary condon { R θ = f z Z R ξ oherwse 48 APPENDIX B IPA DERIVATION FOR MULTI-AGENT SOLUTION The evaluaon of he componens of s n and R n usng 29 nvolves he even me graden vecors = [ ] T θ w for =,..., K, whch wll be deermned hrough 3. There are hree possble cases regardng he evens ha cause swches n he dynamcs of R or s n as menoned above: Case : An even a me such ha Ṙ swches from Ṙ = o Ṙ = A BP s. In hs case, s easy o see ha he dynamcs of boh s n and R are connuous, so ha f = f n 29 appled o s n and R, =,..., M, n =,..., N, and we ge s n = sn, n =,..., N 49 R = R, =,..., M 5 Case 2: An even a me such ha Ṙ swches from Ṙ = A BP s o Ṙ =,.e., R becomes zero. In hs case, we need o frs evaluae from 3 n order o deermne R hrough 29. Observng ha hs even s endogenous, 3 apples wh g = R = and we ge R = A BP s 5 I follows from 29 ha R = R [A BP s] R A BP = 52 Thus, R s always rese o regardless of R. In addon, 49 holds, snce he he dynamcs of s n are connuous a me. Case 3: An even a me such ha he dynamcs of s n swch from ± o, or from o ±. Clearly, 5 holds snce he he dynamcs of R are connuous a hs me. However, deermnng s n s more elaborae and requres us o consder s componens separaely, frs n θ n and hen n w n. Case 3.: Evaluaon of n θ n. Case 3..: An even a me such ha he dynamcs of s n n 35 swch from ± o. Ths s an endogenous

14 4 even and 3 apples wh g = s n θ n,ξ = for some ξ =,..., Γ n and we have: = n u n 53 and 29 yelds n = n [u n n θ ] n,ξ u n = 54 As n Case 3 of Secon III-A, we also need o consder he effec of perurbaons o θ j for j < ξ,.e., pror o he curren even me clearly, for j > ξ, n = snce he curren poson of he agen canno be affeced by fuure evens. Observe ha g =, herefore, 3 becomes = and usng hs n 29 gves: n = n n u n 55 [ ] un n u n = 56 Thus, combnng he above resuls, when s q = θ q,ξ for some ξ and he agen swches from ± o, we have n = {, f j ξ, f j = ξ 57 Case 3..2: An even a me such ha he dynamcs of s n n 35 swch from o ±. Ths s an nduced even snce s rggered by he occurrence of some oher endogenous even when he agen swches from ± o see Case 3.. above. Suppose he agen sars from an nal poson s n = a wh u n = and s he me he agen swches from he o ± a he swchng pon θ n,ξ. If θ n,ξ s such ha u n =, hen ξ s even and can be calculaed as follows: = θ n, a w n, θ n, θ n,2 w n,2... θ n,ξ θ n,ξ w n,ξ = 2 ξ v=, v odd θ n,v ξ2 v=2, v even θ n,v ξ v= w n,v θ n,ξ 58 Smlarly, f θ n,ξ s he swchng pon such ha u n =, hen ξ s odd and we ge: = 2 ξ2 v=, v odd θ n,v We can hen drecly oban ξ v=2, v even as θ n,v ξ w n,v θ n,ξ v= 59 = sgnu 6 Usng 6 n 29 gves: n θ = n n,ξ θ n,ξ [ u ] [sgnu ] = n 6 Once agan, we need o consder he effec of perurbaons o θ j for j < ξ,.e., pror o he curren even me clearly, for j > ξ, n =. In hs case, from 58-59, we have { = 2, f j odd 62 = 2, f j even and follows from 29 ha for j < ξ: n θ n,j n θ n,j 2, = n θ n,j 2, f u n =, j even, or u n =, j odd f u n 63 =, j odd, or u n =, j even n Case 3.2: Evaluaon of w n. Case 3.2.: An even a me such ha he dynamcs of s n n 35 swch from ± o. Ths s an endogenous even and 3 apples wh g = s n θ n,ξ = for some ξ =,..., Γ n. Then, for any j ξ, we have: = n w n,j w n,j u n 64 Combnng 64 wh 29 and snce u n = ±, we have n w = n n,j w [u n n w ] n,j n,j u n = 65 Case 3.2.2: An even a me such ha he dynamcs of s n n 35 swch from o ±. As n Case 3..2, s gven by 58 or 59, dependng on he sgn of u q. Thus, we have w n,j =, for j ξ. Usng hs resul n 29 and observng ha n w n,j = from 65, we have n w n,j = n w [ u n ] n,j, for j ξ 66 = u n Combnng he above resuls, we have for Case 3.2: { n, f w = un = ±, un = n,j, f u n =, un = ± 67 Fnally, noe ha n w n,ξ = for [,, snce he poson of he agen n canno be affeced by w n,ξ pror o such an even.

15 5 REFERENCES [] I. Reles, V. Lee-Shue, A. New, and H. Chose, Lmed communcaon, mul-robo eam based coverage, n IEEE Inl. Conf. on Robocs and Auomaon, vol. 4, 24, pp [2] J. Cores, S. Marnez, T. Karaas, and F. Bullo, Coverage conrol for moble sensng newors, IEEE Trans. on Robocs and Auomaon, vol. 2, no. 2, pp , 24. [3] W. L and C. G. Cassandras, A cooperave recedng horzon conroller for mulvehcle unceran envronmens, IEEE Transacons on Auomac Conrol, vol. 5, no. 2, pp , 26. [4] A. Grard, A. Howell, and J. Hedrc, Border parol and survellance mssons usng mulple unmanned ar vehcles, n 43rd IEEE Conf. Decson and Conrol, vol., 25, pp [5] B. Grocholsy, J. Keller, V. Kumar, and G. Pappas, Cooperave ar and ground survellance, IEEE Robocs & Auomaon Magazne, vol. 3, no. 3, pp. 6 25, 26. [6] S. Smh, M. Schwager, and D. Rus, Perssen roboc ass: Monorng and sweepng n changng envromens, IEEE Transacons on Robocs, 22, o appear. [7] D. Paley, F. Zhang, and N. Leonard, Cooperave conrol for ocean samplng: The glder coordnaed conrol sysem, IEEE Trans. on Conrol Sysems Technology, vol. 6, no. 4, pp , 28. [8] D. Bersmas and G. Van Ryzn, Sochasc and dynamc vehcle roung n he Eucldean plane wh mulple capacaed vehcles, Operaons Research, pp. 6 76, 993. [9] R. Cooper, Inroducon o queung heory. Edward Arnold, 98. [] G. Sun, C. G. Cassandras, Y. Ward, C. G. Panayoou, and G. Rley, Perurbaon analyss and opmzaon of sochasc flow newors, Auomac Conrol, IEEE Transacons on, vol. 49, no. 2, pp , 24. [] N. Ngam and I. Kroo, Perssen survellance usng mulple unmanned ar vehcles, n IEEE Aerospace Conf., 28, pp. 4. [2] P. Hoayem, D. Spanovc, and M. Spong, On perssen coverage conrol, n 46h IEEE Conf. Decson and Conrol, 28, pp [3] Y. Elmalach, A. Shlon, and G. Kamna, A realsc model of frequency-based mul-robo polylne parollng, n Proceedngs of he 7h nernaonal jon conference on Auonomous agens and mulagen sysems-volume. Inernaonal Foundaon for Auonomous Agens and Mulagen Sysems, 28, pp [4] Y. Elmalach, N. Agmon, and G. Kamna, Mul-robo area parol under frequency consrans, n IEEE Inl. Conf. on Robocs and Auomaon, 27, pp [5] D. Kngson, R. Beard, and R. Hol, Decenralzed permeer survellance usng a eam of uavs, IEEE Transacons on Robocs, vol. 24, pp , 28. [6] C. G. Cassandras, X. C. Dng, and X. Ln, An opmal conrol approach for he perssen monorng problem, n Proc. of 5h IEEE Conf. Decson and Conrol, 2, pp [7] C. G. Cassandras, Y. Ward, C. G. Panayoou, and C. Yao, Perurbaon analyss and opmzaon of sochasc hybrd sysems, European Journal of Conrol, vol. 6, no. 6, pp , 2. [8] Y. Ward, R. Adams, and B. Melamed, A unfed approach o nfnesmal perurbaon analyss n sochasc flow models: he sngle-sage case, IEEE Trans. on Auomac Conrol, vol. 55, no., pp. 89 3, 29. [9] J. Le Ny, M. Dahleh, E. Feron, and E. Frazzol, Connuous pah plannng for a daa harvesng moble server, n 47h IEEE Conference on Decson and Conrol. IEEE, 28, pp [2] A. Bryson and Y. Ho, Appled opmal conrol. Wley N.Y., 975. [2] M. Egersed, Y. Ward, and H. Axelsson, Transon-me opmzaon for swched-mode dynamcal sysems, Auomac Conrol, IEEE Transacons on, vol. 5, no., pp. 5, 26. [22] M. Shah and P. Canes, On he hybrd opmal conrol problem: Theory and algorhms, IEEE Trans. on Auomac Conrol, vol. 52, no. 9, pp , 27. [23] X. Xu and P. Ansals, Opmal conrol of swched sysems based on parameerzaon of he swchng nsans, IEEE Trans. on Auomac Conrol, vol. 49, no., pp. 2 6, 24. [24] C. Yao and C. G. Cassandras, Perurbaon analyss of sochasc hybrd sysems and applcaons o resource conenon games, Froners of Elecr. and Elecr. Engn. n Chna, vol. 6,3, pp , 2. [25] C. G. Cassandras and J. E. Lygeros, Sochasc hybrd sysems. Taylor and Francs, 27. [26] E. Pola, Opmzaon: algorhms and conssen approxmaons. Sprnger Verlag, 997. Chrsos G. Cassandras receved he B.S. degree from Yale Unversy, he M.S.E.E degree from Sanford Unversy, and he S.M. and Ph.D. degrees from Harvard Unversy n 977, 978, 979, and 982, respecvely. From 982 o 984 he was wh ITP Boson, Inc. where he wored on he desgn of auomaed manufacurng sysems. From 984 o 996 he was a faculy member a he Deparmen of Elecrcal and Compuer Engneerng, Unversy of Massachuses, Amhers. Currenly, he s Head of he Dvson of Sysems Engneerng and Professor of Elecrcal and Compuer Engneerng a Boson Unversy, Boson, MA and a foundng member of he Cener for Informaon and Sysems Engneerng CISE. He specalzes n he areas of dscree even and hybrd sysems, cooperave conrol, sochasc opmzaon, and compuer smulaon, wh applcaons o compuer and sensor newors, manufacurng sysems, and ransporaon sysems. He has publshed over 3 papers n hese areas, and fve boos. Dr. Cassandras was Edor-n-Chef of he IEEE Transacons on Auomac Conrol from 998 hrough 29 and has served on several edoral boards and as Gues Edor for varous journals. He s he 22 Presden of he IEEE Conrol Sysems Socey and he recpen of several awards, ncludng he 2 IEEE Conrol Sysems Technology Award, he Dsngushed Member Award of he IEEE Conrol Sysems Socey 26, he 999 Harold Chesnu Prze IFAC Bes Conrol Engneerng Texboo for Dscree Even Sysems: Modelng and Performance Analyss, a 22 Kern Fellowshp, and a 99 Llly Fellowshp. He s a member of Ph Bea Kappa and Tau Bea P, a Fellow of he IEEE and a Fellow of he IFAC. perssen survellance. Xuchao Ln receved he B.E. degree n elecrcal engneerng from Zhejang Unversy, Hangzhou, Chna and he M.S. degree from Boson Unversy, Boson, MA n 29 and 22, respecvely. He s currenly a PhD canddae n Sysem Engneerng a Boson Unversy, Boson, MA. Hs research neress nclude dscree even sysems and opmal conrol of hybrd sysems, wh applcaons o perssen monorng, communcaon newors, sensor newors, nellgen vehcle sysems and robocs. He s a suden member of he IEEE. Xu Chu Denns Dng receved hs B.S., M.S. and Ph.D. degree n Elecrcal and Compuer Engneerng from he Georga Insue of Technology, Alana, n 24, 27 and 29, respecvely. Durng 2 o 2 he was a posdocoral research fellow a Boson Unversy. He has recenly joned Uned Technologes Research Cener as a Senor Research Engneer. Hs research neress nclude formal mehods n conrol synhess, opmal conrol of hybrd sysems, coordnaon and conrol of mul-agen newored sysems, and nellgen and