Automatica. Stochastic source seeking for nonholonomic unicycle. Shu-Jun Liu a,b, Miroslav Krstic b, Brief paper. abstract. 1.

Transcription

1 Auomaia 46 () Conens liss available a SieneDire Auomaia journal homepage: Brief paper Sohasi soure seeking for nonholonomi uniyle Shu-Jun Liu a,b, Miroslav Krsi b, a Deparmen of Mahemais, Souheas Universiy, Nanjing 96, China b Deparmen of Mehanial and Aerospae Engineering, Universiy of California, San Diego, La Jolla, CA 993-4, USA arile info absra Arile hisory: Reeived 9 July 9 Reeived in revised form 3 November 9 Aeped 3 November 9 Available online 6 June Keywords: Nonholonomi uniyle Sohasi averaging Exremum seeking We apply he reenly inrodued mehod of sohasi exremum seeking o navigae a nonholonomi uniyle owards he maximum of an unknown, spaially disribued signal field, using only he measuremen of he signal a he vehile s loaion bu wihou he measuremen of he vehile s posiion. Keeping he forward veloiy onsan and onrolling only he angular veloiy, we design a sohasi soure seeking onrol law whih employs exiaion based on filered whie noise, raher han sinusoidal perurbaions used in he exising work. We sudy sabiliy wih he help of sohasi averaging heorems ha we reenly developed for general nonlinear oninuous-ime sysems wih sohasi perurbaions. We prove loal exponenial onvergene, boh almos surely and in probabiliy, o a small neighborhood near he soure. We haraerize he onvergene speed expliily and provide design guidelines for maximizing i, as well as for minimizing he residual se near he soure. We presen a deailed simulaion sudy, inluding a sudy of he effe of sauraion on he seering inpu. Elsevier Ld. All righs reserved.. Inroduion Several soure seeking algorihms employing sinusoidal perurbaions for nonholonomi vehiles in posiion-denied environmens have been reenly proposed. In Zhang, Arnold, Ghods, Siranosian, and Krsi (7), he angular veloiy of he vehile ener is made onsan and he onrol unes he forward veloiy. In Cohran and Krsi (9), he forward veloiy is made onsan and he angular veloiy is onrolled. In his paper, we invesigae a sohasi version of soure seeking by navigaing he uniyle wih he help of a random perurbaion, ahieving a behavior ha mimis he hemoaxislike moion observed in he baerium Esherihia oli (E. oli). E. oli is a single elled organism onsising of a ell body wih muliple railing flagella used for propulsion. In he works Berg (3), and Berg and Brown (97), i is observed ha he baerium is able o move up hemial gradiens owards higher densiies of nuriens by swihing beween alernae behaviors known as run and umble. The behavior run means ha he baerium moves in essenially a sraigh line by roaing he flagella ouner-lokwise as viewed from behind he ell and he behavior umble means ha he baerium eases forward The maerial in his paper was no presened a any onferene. This paper was reommended for publiaion in revised form by Assoiae Edior George Yin under he direion of Edior Ian R. Peersen. Corresponding auhor. Tel.: ; fax: addresses: sjliu@seu.edu.n (S.-J. Liu), krsi@usd.edu (M. Krsi). moion and spins by urning some flagella in a lokwise direion. I is also observed ha he umble behavior displays apparen random naure, alhough he ne moion of he baerium is no ompleely random bu is in he direion of higher nurien onenraions. Moivaed by he hemoai behavior of E. oli, we onsider he problem of sohasi soure seeking for a nonholonomi uniyle. The analogy is appropriae sine neiher he uniyle nor E. oli an exhibi sideways moions, hough hey an be seered. Our vehile has no knowledge of is posiion, nor of of he disribuion of he signal field. Like E. oli, i an only sense he signal loally. To find he soure, we employ a sohasi exremum seeking approah and provide a sabiliy analysis based on sohasi averaging heorems ha we reenly developed in Liu and Krsi (in press). Wih a onroller ha we design in he paper, he vehile is driven o approah a small neighborhood of he soure in a manner ha seems parly random bu is onvergen in a suiable sense. We presen a sabiliy proof for he sheme wih a sai soure and simulaion resuls for boh sai and moving soures. Convergene is proved boh in he almos sure sense and in probabiliy. I is imporan o onsider he relaive meris of he deerminisi soluion o he soure seeking problem in Cohran and Krsi (9) and he sohasi soluion presened here. As expeed, he seering inpus in he sohasi approah are less smooh, whih is a disadvanage of he sohasi approah from he viewpoin of auaor wear. However, he nearly random moion of he sohasi seeker has is advanage in appliaions where he seeker iself 5-98/$ see fron maer Elsevier Ld. All righs reserved. doi:.6/j.auomaia..5.5

2 444 S.-J. Liu, M. Krsi / Auomaia 46 () may be pursued by anoher pursuer. A seeker, whih suessfully performs he soure finding ask bu wih an unprediable, nearly random rajeory, is a more hallenging arge, and is hene less vulnerable, han a deerminisi seeker. Moivaed by E. oli hemoaxis, Mesquia, Hespanha, and Åsröm (8) onsider a similar problem of seeking he maximum of a salar signal, using a swarm of auonomous vehiles, and propose a onrol design whih indues he vehiles o perform a biased random walk, wih a ne moion of he swarm owards he maximum, and ahieving higher vehile densiies near he maximum a he end of he searh. Besides he differene in he algorihms presened in Mesquia e al. (8) and in he presen paper, differen resuls are proved. Mesquia e al. (8) guaranee ha he probabiliy densiy funion of he posiions of he vehiles evolves owards a speified funion of he spaial profile of he measured signal, whereas in he presen paper we prove onvergene (in probabiliy and almos surely), for any single vehile, o a speifi small neighborhood of he soure. Anoher signifian differene is ha we esablish exponenial onvergene, and in fa haraerize he bes ahievable value, and he wors-ase value, of he exponenial onvergene rae, as a funion of he design parameers. In onras, in Mesquia e al. (8) exponenial onvergene is no shown, nor formally laimed. A onsiderable differene in performane is also observed in simulaions. The algorihm in Mesquia e al. (8) a bes mahes he onvergene of he deerminisi algorihm in Zhang e al. (7), whereas he presen algorihm has superior onvergene o ha in Zhang e al. (7) as i does no employ moions ha would, in he absene of a gradien, keep a vehile in plae on he average (suh as random walk, or he riangle and diamond-shaped gais in Zhang e al. (7)), bu employs a sraegy ha keeps he vehile moving in some average direion even when he gradien is zero, as is he ase wih he design in Cohran and Krsi (9). However, i is imporan o noe ha he resuls we prove here are only for signal fields ha have irular level ses, whereas in Mesquia e al. (8) suh a resriion is no presen. The presen paper adds a new ool o a sring of reen suessful developmens in he area of exremum seeking, boh in he deerminisi ase (Ariyur & Krsi, 3; Moase, Manzie, & Brear, 9a,b; Tan, Nešić, & Mareels, 6) and in he sohasi (disree ime) ase (Manzie & Krsi, 9; Sankovi & Sipanovi, 9a,b). The paper is organized as follows. In Seion we presen he vehile model and sae he problem. In Seion 3 we presen our sohasi soure seeking onroller. In Seion 4 we prove loal exponenial onvergene for irular level ses, namely, where he signal depends only on he disane from he soure and deays quadraially. In Seion 5 we alulae he onvergene speed, for pariular parameer hoies for whih i is possible o do so expliily, and haraerize he bes ahievable onvergene speed. In Seion 6 we presen simulaions and disussions abou dependene on design parameers. In Seion 7 we onsider signal fields wih ellipial level ses.. Vehile model and problem saemen As in Cohran and Krsi (9), we onsider a mobile agen modeled as a uniyle wih a sensor mouned a is fron end, a disane R from he ener. Fig. depis he posiion, heading, angular and forward veloiies for he ener and sensor. The equaions of moion for he vehile ener are ṙ = ve jθ, () θ = u, () Fig.. The noaion used in he model of vehile sensor and ener dynamis. Fig.. Blok diagram of sohasi soure seeking via uning of angular veloiy of he vehile. where r is he vehile ener, θ is he orienaion, v, u are he forward and angular veloiy inpus, respeively, and j is he imaginary uni. The sensor is loaed a r s = r + Re jθ. The ask of he vehile is o seek a soure ha emis a signal a spaially disribued signal J = f (r(x, y)), whih has an isolaed loal maximum f = f (r ), where r is he loaion of he loal maximum. We ahieve loal onvergene o r, in a pariular probabilisi sense, wihou he knowledge of he shape of f ( ), and wihou he measuremen of r, using only he measuremen of J() a he vehile sensor. 3. Sohasi soure seeking onroller We employ he sheme depied by he blok diagram in Fig.. The forward veloiy of he vehile is se o v() = V ons, whereas he angular veloiy θ is uned by he exremum seeking onrol law θ = a η + ξ sin(η) d ξ sin(η), (3) where ξ = s [J] is he oupu of he washou filer for he s+h sensor reading J,η = g [Ẇ]( (, s+ )) is olored noise used as a perurbaion in sohasi exremum seeking, and V, a,, d, g,,h > are design parameers whih (along wih parameer R) influene he performane. The signal (W(), ) is a sandard Brownian moion defined in a omplee probabiliy spae (Ω, F, P) wih he sample spae Ω, he σ -field F, and he probabiliy measure P. Wih he observaion ha he ransfer funion from whie noise Ẇ o η is relaive degree zero, giving η = g s s + [Ẇ] = gs + g g s + [Ẇ] = g Ẇ η, (4)

3 he onrol law is rewrien as dθ = a η + (ξ d ξ ) sin(η) + ag dw, (5) dη = η + g dw. (6) Compared wih he deerminisi ase in Cohran and Krsi (9), where sin(ω) was used as he probing signal, we use he sohasi signal sin(η()) o develop a gradien esimae. I is no essenial o hoose he sinusoidal nonlineariy sin(η) in he sohasi design. This hoie is primarily made for he ease of deriving he average sysem in he sabiliy analysis. We an replae sin(η) wih oher bounded and odd funions, suh as ηe η, however, he inegrals in alulaing he expeaions in he derivaion of he average sysem beome more ompliaed. In fa, he boundedness of he perurbaion (suh as sin η or ηe η ) is only needed in he analysis, whereas in he simulaions, suessful onvergene is ahieved even when sin(η) is replaed by η. We refer o he erm d ξ sin(η) as he d -erm or he damping erm. This erm is no needed in he basi sohasi exremum seeking algorihm for a sai map Liu and Krsi (in press). This erm is essenial for ahieving exponenial sabiliy in soure seeking problems wih a vehile employing onsan forward veloiy. 4. Sabiliy analysis We assume ha he nonlinear map defining he disribuion of he signal field is quadrai and akes he form J = f (r s ) = f q r r s r where r is he unknown maximizer, f = f (r ) is he unknown maximum and q r is an unknown posiive onsan. We define an oupu error variable e = h [J] f, whih allows s+h us o express he signal ξ afer he washou filer, as ξ = s [J] = s+h J h [J] =J f e, and hus we have ė = hξ. s+h Sohasi approximaion is a good mehod o find he exremum of a funion. However for our soure seeking problem, he ondiions of onvergene analysis of sohasi approximaion are hard o verify due o he presene of dynamis and nonholonomi onsrains. In his paper, we use our sohasi average heory presened in Appendix o analyze he sabiliy of he losedloop sysem. Theorem 4.. Consider he losed-loop sysem dr = V e jθ, (7) dθ = a η + (ξ d ξ ) sin(η) + ag dw, (8) de = hξ, (9) ξ = (q r r s r + e), () r s = r + Re jθ, () dη = η + g dw, () where, d, h, R, V, q r >, and he parameers h, V, a, g > are hosen suh ha h > R I, (3) V I I where I = e a g 4, I = (The ondiion (3) is saisfied for any h > and V S.-J. Liu, M. Krsi / Auomaia 46 () β g is hosen as g = and a is hosen as < a < a a (β) e (a ) g 4 e (a+) g 4 > provided. β ln e β e β (e β for any β >. For example, for β =, ) a ().4.) If he iniial ondiions r (), θ(), e() are suh ha he following quaniies are suffiienly small, r () r ρ, e() + qr (R + ρ ), (4) eiher θ() arg(r r ()) + π or (5) θ() arg(r r ()) π, (6) where ρ = V I q r RI, (7) hen here exis onsans C,γ > and a funion T() : (, ) N suh ha for any δ>, lim inf : r () r ρ and > C e γ + δ =, a.s. (8) lim P r () r ρ C e γ + δ, [, T()] = (9) wih lim T() =, where he onsan C is dependen on he iniial ondiion (r (), θ(), e()) and on he parameers a,, d, h, R, V, q r, g, and he onsan γ is dependen on he parameers a,, d, h, R, V, q r, g. Proof. We sar by defining he shifed variables ˆr = r r, () ˆθ = θ aη, () and a map beween ˆr and a new quaniy θ given by ˆr = ˆr e jθ () θ = arg( ˆr ) = arg(r r ) π j ˆr ln = j ˆr ln ˆr π j ˆr ln ˆr, if θ ˆr, if θ π, π, π, π, π, if θ,π, (3) where θ represens he heading angle owards he soure loaed a r when he vehile is a r. Using hese definiions, he expression for ξ is ξ = (q r (R + ˆr R ˆr os(ˆθ θ + aη)) + e). Sine dˆθ = dθ adη = (ξ d ξ ) sin(η), (4) we obain he dynamis of he shifed sysem as dˆr dˆθ = dr = V e j(ˆθ+aη), (5) = (ξ d ξ ) sin(η), (6) de = hq r R + ˆr R ˆr os(ˆθ θ + aη) he. (7)

4 446 S.-J. Liu, M. Krsi / Auomaia 46 () By () and he definiion of Io sohasi differenial equaion, we have η() = η() η(s)ds + g dw(s). Thus i holds ha η() = η() η(u)du + g dw(u). Define B() = W(), χ() = η(). Then we have dχ() = χ() + gdb(), where B() is a sandard Brownian moion and he proess χ() is an Ornsein Uhlenbek (OU) proess whih is ergodi wih invarian disribuion µ(dy) = e y g dy. Now we define error variables r and θ whih represen he disane o he soure, and he differene beween he vehile s heading and he opimal heading, respeively, r = ˆr = r r, (8) θ = ˆθ θ. (9) Thus we obain he following dynamis for he error variables d r = d ˆr = d ˆr ˆr = dˆr ˆr ˆr dˆr + ˆr d θ = V os( θ + aχ(/)), (3) = dˆθ dθ = dˆθ + j dˆr ˆr ˆr dˆr ˆr = ( d ξ)ξ sin(χ(/)) + V r sin( θ + aχ(/)), (3) de = hξ, (3) ξ = q r (R + r R r os( θ + aχ(/))) + e, (33) dχ() = χ() + gdb(). (34) We use general sohasi averaging given in Appendix o analyze his error sysem. Firs we alulae he average sysem of (3) (3) (3). Sine e y R sin(ay)µ(dy) = + sin(ay) g dy =, os(ay) sin R (ay)µ(dy) = os(ay) sin(ay)µ(dy) = R, os(ay)µ(dy) = R + os(ay) e y g dy = e a g 4 = I (a, g), and sin(ay) R sin(y)µ(dy) = + sin(ay) sin(y) e y g dy = e (a ) g 4 e (a+) g 4 = I, by (A.3), we obain ha he average error sysem is d r ave d θ ave = V I os( θ ave ), (35) = + 4d q r (R + r ave ) + e ave q r R r ave sin( θ ave )I + d q ave r R r sin( θ ave )I + V sin( θ ave )I r ave, (36) de ave = hq r R hq r r ave + hrq r r ave os( θ ave )I he ave. (37) The average error sysem has wo equilibria defined by r aveeq, θ aveeq, e aveeq = ρ,+ π, q r(r + ρ ), (38) r aveeq, θ aveeq, e aveeq = ρ, π, q r(r + ρ ), (39) where ρ is given by (7). The above wo equilibria have he following Jaobians, respeively, A eq = V I A eq 4d γ ρ I 4d γρi, (4) hq r ρ hγρi h and A eq = V I A eq 4d γ ρ I 4d γρi, (4) hq r ρ hγρi h where A eq = A eq = 4γ( + d q r ρ )I, γ q r R. The haraerisi polynomial for boh Jaobians is = λ 3 + hλ + V I λ + h V I ρ ρ + 4d ρ q r R[RI λ + (V I I + hr(i I I ))λ]. (4) Sine a >, we have I > and I >. For he roos of he polynomial (4) o be in he lef half-plane, all of is hree oeffiiens need o be posiive and he produ of he oeffiiens assoiaed wih λ and λ needs o be greaer han he oeffiien assoiaed wih λ. All of hese ondiions are saisfied whenever V I I + hr(i I I ) >, whih is equivalen o he ondiion (3). When he ondiion (3) is saisfied, he Jaobians (4) and (4) are Hurwiz, whih implies ha boh average equilibria (38) and (39) are exponenially sable. Thus by Theorems A.3 and A.4 in Appendix, here exis onsans (i) >, r (i) >,γ (i) > and funions T (i) () : (, ) N, i =,, suh ha for any δ >, and any iniial ondiion Λ (i) () < r (i), lim inf and : Λ (i) () > (i) Λ (i) () e γ (i) + δ =, a.s., (43) lim P Λ (i) () (i) Λ (i) () e γ (i) + δ, [, T (i) ()] = (44) wih lim T (i) () =, where Λ () () = ( r () ρ, θ() π, e()+q r (R +ρ )) and Λ () () = ( r () ρ, θ()+ π, e()+q r(r + ρ )). The resuls (43), (44), ogeher wih he fa r () ρ < r () ρ, θ() ± π, e() + q r(r + ρ ) and he definiion of r, we have lim inf : r () r ρ > C (i) (i) e γ + δ =, a.s., (45) and lim P r () r ρ C (i) (i) e γ + δ, [, T (i) ()] = (46) wih lim T (i) () =, where C () = () ( r () ρ, θ() π, e() + q r (R + ρ )) and C () = () ( r () ρ, θ() + π, e() + q r (R + ρ )). This omplees he proof.

5 S.-J. Liu, M. Krsi / Auomaia 46 () Convergene speed Theorem 4. esablishes exponenial onvergene, however, he onvergene rae is deermined by he ompliaed ubi polynomial (4), whose roos are hard o find analyially in general. However, for pariular parameer hoie, hey an be found expliily, as given in he nex proposiion. Proposiion 5.. Le he vehile speed V and he parameer h of he washou filer be hosen aording o he following relaion: V = hr. (47) Then he exponenial onvergene rae of he soure seeking sysem in Theorem 4. is deermined by he eigenvalues λ = h, (48) ψ, (49) λ = d q r R hi I I λ 3 = d q r R hi I I where ψ = + ψ, (5) 4 3 I 3 >, d q (5) rhr I I and he radius of he residual annulus is hi ρ = q r I. (5) Proof. Wih V = hr, he sabiliy ondiion (3) beomes I > I I, (53) whih is saisfied for all parameers a, g, h, R >. Thus he haraerisi polynomial (4) has all hree roos wih negaive real pars. Le H 4d ρ q r R I, (54) M V I, (55) ρ Q 4d ρ q r R[V I I + hr(i I I )]. (56) Then we wrie he haraerisi polynomial ompaly as λ 3 + (h + H)λ + (M + Q )λ + hm =. (57) Denoe by λ i, i =,, 3, he roos of he polynomial (57). Then by he relaion beween he roos and he oeffiiens in he polynomial, we have h + H = λ λ λ 3, (58) hm = λ λ λ 3, (59) M + Q = λ λ + λ λ 3 + λ 3 λ. (6) A his poin one an jus verify (48) (5) by dire subsiuion ino (58) (6), however, we explain how we have arrived a (48) (5). Le λ = h. We shall show ha his hoie saisfies (58) (6) by also finding λ and λ 3 whih saisfy (58) (6). Wih λ = h, (58) (6) beome H = λ λ 3, (6) M = λ λ 3, (6) Q = hh. (63) Wih subsiuion of V = hr ino (56), we immediaely see ha (63) is verified. From (6) and (6) we see ha we only need o solve he quadrai equaion λ + Hλ + M =. Applying he formula for he roos of a quadrai equaion, we arrive a λ = d ρ q r R I + 4d ρ ρ6 q 4 r R4 I V I I I = d q r RV I + R d q r V I (a, g)i I 4hq r 3 I I, (64) λ 3 = d ρ q r R I 4d ρ ρ6 q 4 r R4 I V I I I = d q r RV I R d q r V I (a, g)i I 4hq r 3 I I, (65) whih, wih some simplifiaions, gives (48) (5). Of he hree eigenvalues in Proposiion 5. one is real and an be plaed arbirarily far o he lef by hoosing h large, whereas he oher wo an eiher be real or onjugae omplex. The opimal hoie is where he eigenvalues λ and λ 3 are equal, beause oherwise, eiher one or boh of hese eigenvalues are loser o he imaginary axis han when λ = λ 3. Unforunaely, his opimal eigenvalue plaemen anno be ahieved by inen, sine he design parameers would have o depend on he unknown q r, however, in he nex orollary we sae his resul in order o noe wha he bes ahievable onvergene speed is. Corollary 5.. Le V = hr and le he damping parameer be hosen as d = 3 I qr RhI I. (66) Then he exponenial onvergene rae of he soure seeking sysem in Theorem 4. is deermined by he eigenvalues λ = h, (67) λ = λ 3 = R q r hi I, (68) whereas he residual annulus is as in (5). From Corollary 5. we noe ha he opimizing damping oeffiien d grows, whereas he onvergene rae λ = λ 3 deays, wih a derease of he parameer q r, namely, wih he flaening of he exremum, as should be expeed. No surprisingly, he residual annulus (5) also grows wih he flaening of he exremum. The onvergene speed grows, whereas he annulus size shrinks, wih he uning gain. Proposiion 5.. For a fixed a, he opimal onvergene speed (68) has a non-monooni dependene on he noise inensiy g, wih he maximal onvergene speed ahieved for g = a ln a + + a a + a. (69)

6 448 S.-J. Liu, M. Krsi / Auomaia 46 () Proof. By onsidering (68) and maximizing I I = g 4 e (a ) g 4 e (a+) g 4 wih respe o g. e a a The non-monooni dependene of he onvergene speed on he noise inensiy g is inuiive. If he noise is low, he gradien exploraion is insuffiien and he uning proess is ineffeive. Too muh noise, and he perurbaion akes he rajeories oo far from he average rajeory, slowing he approah o he annulus. Proposiion 5.3. For a / he annulus radius ρ defined in (5) is a dereasing funion of noise inensiy g. For a (, /) he radius ρ has a non-monooni dependene on g, wih he minimal ρ ahieved for b g = a ln + a a. (7) Proof. By onsidering (5) and minimizing I (a,g) I (a,g) = e 4 g e ag e ag wih respe o g. Sine we wan o operae wih a relaively small perurbaion parameer a, he annulus-minimizing value of g in (7) is of ineres. Boh very large and very low inensiy of perurbaion noise resul in a large annulus, whereas a medium range of g is opimal. I is worh omparing he opimizing g for onvergene speed in (69) wih he opimizing g for he annulus in (7). For small a hey are similar, whih is very forunae. Fig. 3. (a) The rajeory of he vehile ener for he ase of soure wih irular level ses. The rajeory onverges o an annulus; (b) A zoomed in seion of he vehile rajeory, displaying he vehile moion more learly. For boh simulaions: V =., =, d =, a =., g =, =., R =., f =, h =, q r =.5. The soure is a r = (, ). 6. Simulaions, dependene on design parameers, effe of onsrains of he angular veloiy, and design alernaives 6.. Basi simulaions Wihou loss of generaliy, we le he unknown loaion of he soure be a he origin r = (, ). We pik he design parameers as V =., =, d =, a =., h =, g =, =., R =. and ake he parameers of he map as f =, q r =.5. The simulaion resuls are given in Fig. 3. We observe ha he rajeories of he vehile ener go o a small neighborhood of he soure and he vehile moion involves a random perurbaion omponen, insead of a sinusoidal perurbaion employed in he deerminisi ase Cohran and Krsi (9). In he simulaions we use band-limied whie noise o approximae he whie noise. The sohasi soure seeking approah an also be used for pursui of non-saionary soures. For he ase where he soure is performing a figure eigh moion, unknown o he pursuing vehile, he simulaion resul is shown in Fig Dependene of annulus radius ρ on parameers From (7), we see he radius ρ of he araive annulus is dependen on he model parameers q r, R and design parameers V,, a, g, and ha i an be made as small as desired. Hene, by (8) and (9), by making ρ as small as desired, he vehile an onverge as losely o he soure as desired. The dependene of ρ on he noise inensiy is haraerized by Proposiion 5.3. Fig. 5 show some of his dependene. For a fixed small a =., he radius for g = is ρ =., whih is smaller han he radius ρ =.9 for g =. Fig. 4. Vehile following a moving soure wih irular level ses. The simulaion parameers are V =., =, d =, a =., g =, =., R =., f =, h =, q r =.5. The soure moves aording o x sr () =.5 sin(.3), y sr () =.5 sin(.6) Effe of onsrains of he angular veloiy A physial vehile always has a seering onsrain, namely, a limi on he angular veloiy θ. This ype of a uniyle model is ommonly referred o as he Dubins vehile. Fig. 6 depis he rajeories of he vehile ener when he angular veloiy is resried o a symmeri inerval, [ u max, +u max ], for several values of u max. We observe ha, for u max as small as, our onrol law suessfully seers he vehile o he annulus, and keeps he vehile near he soure, see Fig. 6(a). In addiion, he vehile moves more smoohly for smaller u max, see Fig. 6(b). However, if he auaor onsrain u max is oo small, for example, u max =, he algorihm anno keep he vehile very near he soure, as observed in Fig. 7.

7 S.-J. Liu, M. Krsi / Auomaia 46 () a b Fig. 7. The rajeories of he vehile ener under a severe onsrain on he angular veloiy inpu (umax = ). The simulaion parameers are V =., =, a =., g =, =., R =., f =, h =, qr =.5. The soure is a r = (, ). Fig. 5. The radius of he araive annulus of he vehile ener for he ase of soure wih irular level ses. (a) is for g = ; (b) is for g =. The oher simulaion parameers are V =., =, d =, a =., =., R =., f =, h =, qr =.5. The soure is a r = (, ). a Fig. 8. The rajeory of he vehile ener under he onrol law (7). The simulaion parameers are V =., =, d =, a =., g =, =., R =., f =, h =, qr =.5. The soure is a r = (, ). () and he demodulaion equaion (4) in he presen work, he reader should noe ha he probing and demodulaion signals are differen. They are η and sin(η), respeively. In his paper we make suh a hoie for he sake of simpliiy of alulaing he average error sysem in he sabiliy analysis he inegrals in he expeaions are easier o obain analyially wih suh a hoie. If η is replaed by sin(η) as he sohasi perurbaion in (), he exremum seeking onrol (3) is replaed by b θ = a os(η)η ag sin(η) + ξ sin(η) d ξ sin(η) (7) and hus (8) in he losed-loop sysem hanges o dθ = a ag os(η)η sin(η) ag + ( ξ d ξ ) sin(η) + os(η)dw, (7) ag Fig. 6. (a) The rajeories of he vehile ener for differen onsrains of angular veloiy; (b) A zoomed in seion of vehile moion for differen onsrains umax on angular veloiy. The simulaion parameers are V =., =, a =., g =, =., R =., f =, h =, qr =.5. The soure is a r = (, ) Alernaive designs In he sandard exremum seeking algorihm (see Ariyur and Krsi (3)), he probing signal and he demodulaion signal are he same, ypially sin(ω ). Looking a he probing equaion where he addiional erm sin(η) resuls from he Io formula. Consequenly, he wo erms os(θ +aχ ( /)) and sin(θ +aχ ( /)) in he error sysem (3) (3) (3) should be replaed by os(θ + a sin(χ ( /))) and sin(θ + a sin(χ ( /))), respeively. I is hard o obain he orresponding analyial average error sysem beause we need o alulae wo inegrals: + y g + os(a sin(y))e y g dy and sin(a sin(y)) sin(y)e dy and i is hard o obain he analyial resuls hough we an obain numerial resuls. Fig. 8 depis he rajeory of he vehile ener when he onrol law (7) is used. From he simulaion, here is no noieable differene relaive o he rajeory in Fig. 3(a).

8 45 S.-J. Liu, M. Krsi / Auomaia 46 () Fig. 9. The rajeory of he vehile ener under he onrol law (73). The simulaion parameers are V =., =, d =, a =., g =, =., R =., f =, h =, q r =.5. The soure is a r = (, ). Now we analyze he radius of he annulus for hree alernaive perurbaion signals. Le V =., =, d =, a =., g =, =., R =., f =, q r =.5. Then () For he probing signal η in () and demodulaion signal sin(η) in (4), we obain he radius of he annulus as ρ I =.93. () If we use sin(η) o replae η as he probing signal in (), he expressions I and I are replaed by I and I (a, g), where I os(a sin(y))µ(dy) = R + os(a sin(y)) e y g dy and I sin(a sin(y)) R g dy. By alu- sin(y)µ(dy) = + sin(a sin(y)) sin(y) laing he inegrals numerially, we obain I e y (., ) =.9984 and I (., ) =.36. Thus, we ge he radius of he annulus as ρ II V = I (a,g) q r RI (a,g) =.35, whih is a lile larger han ρi. (3) If we use he bounded funion ηe η o replae boh η as he probing signal in () and sin(η) as he demodulaing signal in (4), by numerial alulaion we ob- ain R os(.ye y )µ(dy) = + os(.ye y ) =.9995, R sin(.ye y )ye y µ(dy) = + ye y e y e y g dy sin(.ye y ) g dy =.96. Thus he radius is ρ III =.588, whih is onsiderably larger han boh ρ I and ρ II. Therefore, from he poin of view of he annulus radius, our hoie η as he probing signal in () and sin(η) as he demodulaion signal in (4), ahieves he bes performane, in addiion o failiaing he analysis. If OU proess (η(), ) is used no only as he probing signal, bu also as he demodulaion signal in (4), he exremum seeking onrol law (3) is replaed by θ = a η + (ξ d ξ )η. (73) Wih sin(η) replaed by η as a demodulaion signal, where he laer signal is no uniformly bounded, he loal Lipshiz ondiion (Assumpion A. in Appendix) is no saisfied uniformly in he perurbaion proess for he resuling losed-loop sysem. For his reason, we anno use he general sohasi averaging heorem o analyze sabiliy. However from simulaion resuls given by Fig. 9, we observe ha he vehile ahieves onvergene o a an annulus near he soure under he onrol law (73). 7. Sysem behavior for ellipial level ses Our analysis is limied o irular level ses, namely, o fields ha depend on he disane from he soure only. In his seion, Fig.. The rajeory of he vehile ener for signal field wih ellipial level ses. The simulaion parameers are V =., =, d =, a =., g =, =., R =., f =, h =, q r =.5, q p =.5. The soure lies in r = (, ). we presen simulaion resuls for ellipial level ses. Wihou loss of generaliy, we assume he soure is a r = (, ), and he signal disribuion in spae is given (a he sensor loaion) by J = f (r s ) = f q r r s q p (r s + r s ) = f (q r + q p )x s (q r q p )y s = f q r r + Re jθ q p (r + Re jθ ) + ( r + Re jθ ), (74) where q r >, q r ± q p >. Fig. depis he rajeory of he vehile ener for a signal field wih ellipial level ses. The vehile reahes a small neighborhood of he soure, however, he average moion is no irular revoluion around he soure, nor ellipial revoluion, bu a moion bias o one of he flaer sides of he ellipse. More han one suh araor exiss. I depends on he iniial ondiion and on he noise sequene whih of he average araors he rajeory will onverge o. Fig. depis he rajeories of he vehile ener wih differen d -values in he onrol law. From Fig. (a), we see ha for larger d he vehile undergoes a roundabou behavior and hen moves ino a small neighborhood of he soure. This is no differen han he siuaion for irular level ses, wih eiher sohasi or deerminisi soure seeking algorihms. However, from Fig. (b), we observe a differene relaive o he resuls obained for ellipial level ses in he deerminisi ase in Cohran and Krsi (9). The value of d does no affe he shape and size of he sysem araors he moion near he soure is limied o an ellipial shape. 8. Conluding remarks We have proposed and analyzed an algorihm for sohasi soure seeking, employing, in a suiable way, olored noise perurbaions, insead of periodi deerminisi perurbaions. We have adaped he general sohasi averaging heory for nonlinear oninuous-ime sysems wih sohasi perurbaion o esablish exponenial onvergene, boh almos surely and in probabiliy, o an annulus shaped region around he soure. For pariular values of design parameers we have alulaed he onvergene rae expliily. If he uoff frequeny of he washou filer is hosen as high, hen he dominan dependene of he bes ahievable onvergene speed, given in (68), shows an inreasing dependene on all of he relevan parameers he vehile lengh R, he uning gain, and he peak sharpness oeffiien q r. However, boh he onvergene speed and he annulus size show non-monooni dependene on he noise inensiy g, for whih we provide opimizing values.

9 S.-J. Liu, M. Krsi / Auomaia 46 () a Assumpion A.. The veor field a(x, y) is a oninuous funion of (x, y), and for any x D, i is a bounded funion of y. Furher i saisfies he loally Lipshiz ondiion in x D uniformly in y S Y, i.e., for any ompa subse D D, here is a onsan k D suh ha for all x, x D and all y S Y, a(x, y) a(x, y) k D x x. Assumpion A.. The perurbaion proess (Y, ) is ergodi wih invarian disribuion µ. b Under Assumpion A., we obain he average sysem of sysem (A.) as follows: d X = ā( X ), X = x (A.) where ā(x) = a(x, y)µ(dy). (A.3) S Y Fig.. Signal field wih ellipial level ses. (a) The rajeories of he vehile ener for differen d -values; (b) A zoomed in seion of araors for differen d -values. The simulaion parameers are V =., =, a =., g =, =., R =., f =, h =, q r =.5, q p =.5. The soure lies in r = (, ). The resuls of his paper would no be diffiul o exend o 3D soure seeking, as in Cohran, Ghods, Siranosian, and Krsi (9a), for underwaer vehile appliaions, or even o soure seeking for fish models, as in Cohran, Kanso, Kelly, Xiong, and Krsi (9b). Aknowledgemens The researh was suppored by he Naional Siene Foundaion (NSF) and he ONR Gran N Shu-Jun Liu was also suppored by he Naional Naural Siene Foundaion of China under gran Appendix. General sohasi averaging dx Consider he sysem = a(x, Y /), X = x, (A.) where X R n ; Y R m is a ime homogeneous oninuous Markov proess defined on a omplee probabiliy spae (Ω, F, P), where Ω is he sample spae, F is he σ -field, and P is he probabiliy measure. The iniial ondiion X = x is deerminisi. is a small parameer in (, ) wih fixed >. Le D R n be a domain (open onneed se) of R n and S Y be he living spae of he perurbaion proess (Y, ). Noie ha S Y may be a proper (e.g., ompa) subse of R m. Consider he following wo assumpions: For he ase D = R n, under Assumpions A. and A., and he assumpion of he exisene of he soluion, we have proved general sohasi averaging heorems in Liu and Krsi (in press). The exisene assumpion is formulaed in Liu and Krsi (in press) as follows: for any x R n and he perurbaion proess (Y, ), sysem (A.) has a unique (almos surely) oninuous soluion on [, ). When he domain D is a proper subse of R n, his ondiion is a srong resriion on sysem (A.), beause i is hard o resri he soluion of a sohasi sysem wihin suh a domain. In his paper, we an eliminae his ondiion. Before presening he main resuls, we give wo lemmas. To his end, for any poin x D, we define by d(x, D) he disane beween x and he boundary D of he domain D, i.e., d(x, D) = inf{ x y :y D}. By onvenion d(x, ) =. Sine D is a domain, for any x D, we have ha d(x, D) >. If A is a subse of D, we define by d(a, D) he disane beween A and D as follows: d(a, D) = inf x A d(x, D) = inf{ x y :x A, y y}. Throughou his par of he paper, we assume ha x D, where x is he iniial value of sysem (A.). Sysem (A.) is a sohasi ordinary differenial equaion (sohasi ODE), and is soluion an be defined for eah sample pah of he perurbaion proess (Y / : ). If sysem (A.) saisfies Assumpion A., hen for any ompa subse D D and he onsan k D saed in Assumpion A., i holds ha for any ω Ω, any, any (, ), and all x, x D, a(x, Y / (ω)) a(x, Y / (ω)) k D x x. Thus by he heorem on loal exisene and uniqueness of soluions of nonlinear sysems (see, e.g., Theorem 3. of Khalil ()), for any (, ) and any ω Ω, sysem (A.) has a unique soluion X (ω) wih he life ime l (ω) >, where l (ω) = inf : X (ω) D. For > l (ω), we define X (ω) = Xl(ω) (ω), i.e., as soon as he soluion reahes he boundary of he domain D, we fix i and mainain i a ha onsan value hereafer. Lemma A.. Consider sysem (A.) under Assumpions A. and A.. If d({ X, }, D) >, hen for any T >, we have ha lim sup X X =, a.s. (A.4) T Proof. Fix T > and define A T = { X : T}. Then by he assumpion ha d({ X, }, D) >, we have ha

10 45 S.-J. Liu, M. Krsi / Auomaia 46 () δ T := d(a T, D) >. For any (, ), define a sopping ime τ by τ = inf : X X > δ T. Noie ha X = X = x. Then by he oninuiy of he sample pahs of (X, ) and ( X, ), we know ha <τ l, and if τ < +, hen X τ Xτ = δ T, (A.5) and hus d(x τ, D) δ T >, and so in his ase τ < l. From (A.) and (A.), we have ha for any < l, X X = + a(x s, Y s/) a( Xs, Y s/ ) ds a( Xs, Y s/ ) ā( Xs ) ds. (A.6) Sine X is oninuous, A T is a ompa subse of D. Furher, by he assumpion ha d({ X, }, D) >, we know ha he se D T := x D : d(x, A T ) δ T is a ompa subse of D. Then by Assumpion A., we obain ha for any s τ T, a(x, s Y s/) a( Xs, Y s/ ) k T X s Xs, (A.7) where k T is he Lipshiz onsan of a(x, y) wih respe o he ompa subse D T of D. Thus by (A.6) and (A.7), we have ha if τ T, hen X X k T X s Xs ds + Define a( Xs, Y s/ ) ā( Xs ) ds.(a.8) = X X, (A.9) α() = sup a( Xs, Y s/ ) ā( Xs ) ds. T Then by (A.8) and Gronwall s inequaliy, we have sup α()e k T (τ T) α()e k T T. τ T (A.) Sine ( X, ) is a deerminisi oninuous funion, by Assumpion A. and Birkhoff ergodi heorem (see e.g. Chaper of Skorokhod, Hoppensea, and Salehi ()), we have ha lim α() =, a.s. (A.) I follows from (A.9), (A.) and (A.) ha lim sup sup τ T X X =, a.s. (A.) By (A.5) and (A.), we obain ha for a.e. ω Ω, here exiss an (ω) > suh ha for any << (ω), τ (ω) > T. (A.3) Thus by (A.) and (A.3), we obain ha lim sup sup T X X =, a.s. Hene (A.4) holds. Lemma A.. Consider sysem (A.) under Assumpions A. and A.. If d({ X, }, D) >, hen for any δ>, we have lim inf{ : X X >δ} = +, a.s. Proof. Define Ω = ω : lim sup sup T X (ω) X =, T N. (A.4) Then by Lemma A., we have P(Ω ) =. (A.5) Le δ >. Wihou loss of generaliy, we an assume ha δ < d({ X, }, D) sine if < a < b, we have inf{ : X X > b} inf{ : X X > a}. For (, ), define a sopping ime τ δ by τ δ = inf{ : X X >δ}. By he fa ha X X =, and he oninuiy of he sample pahs of (X, ) and ( X, ), we know ha <τ δ +, and if < +, hen τ δ X τ δ Xτ δ =δ. (A.6) For any ω Ω, by (A.4), (A.6) and δ< d({ X, }, D), we ge ha for any T N, here exiss (ω, δ, T) > suh ha for any << (ω, δ, T), τ δ (ω) > T, whih implies ha lim τ δ (ω) = +. (A.7) Thus i follows from (A.5) and (A.7) ha lim τ δ = +, a.s. This omplees he proof. Now, by Lemmas A. and A., following he orresponding proofs in Liu and Krsi (in press), we obain he following wo heorems. Theorem A.3. Consider sysem (A.) under Assumpions A. and A.. Then if he equilibrium X x D of he average sysem (A.) is exponenially sable, hen here exis onsans r >, > and γ> suh ha for any iniial ondiion x {x D : x x < r}, and any δ>, he soluion of sysem (A.) saisfies lim inf : X x > x e γ + δ = +, a.s. (A.8) Theorem A.4. Consider sysem (A.) under Assumpions A. and A.. If he equilibrium X x D of he average sysem (A.) is exponenially sable, hen here exis onsans r >, >,γ > and a funion T() : (, ) N suh ha for any iniial ondiion x {x D : x x < r}, and any δ>, lim P sup T() or equivalenly, X x x e γ >δ =, (A.9) lim P X x x e γ + δ, [, T()] = (A.) wih lim T() = +. Referenes Ariyur, K. B., & Krsi, M. (3). Real-ime opimizaion by exremum seeking onrol. Hoboken, NJ: Wiley-Inersiene. Berg, H. (3). E. oli in moion. New York: Springer. Berg, H., & Brown, D. A. (97). Chemoaxis in E. oli analyzed by hree-dimensional raking. Naure, 39(5374), Cohran, J., Ghods, N., Siranosian, A., & Krsi, M. (9a). 3D soure seeking for underauaed vehiles wihou posiion measuremen. IEEE Transaions on Robois, 5, 7 9. Cohran, J., & Krsi, M. (9). Nonholonomi soure seeking wih uning of angular veloiy. IEEE Transaions on Auomai Conrol, 54(4),

11 S.-J. Liu, M. Krsi / Auomaia 46 () Cohran, J., Kanso, E., Kelly, S. D., Xiong, H., & Krsi, M. (9b). Soure seeking for wo nonholonomi models of fish loomoion. IEEE Transaions on Robois, 5, Khalil, H. K. (). Nonlinear sysems. Prenie Hall. Liu, S. -J., & Krsi, M. (9). Sohasi averaging in oninuous ime and is appliaions o exremum seeking. In IEEE Transaions on Auomai Conrol (in press). Manzie, C., & Krsi, M. (9). Exremum seeking wih sohasi perurbaions. IEEE Transaions on Auomai Conrol, 54, Mesquia, A. R., Hespanha, J. P., & Åsröm, K. (8). Opimoaxis: a sohasi muliagen opimizaion proedure wih poin measuremens. In M. Egerse, & B. Mishra (Eds.), Leure noes in ompuer siene: Vol Hybrid sysems: ompuaion and onrol (pp ). Springer-Verlag. Moase, W. H., Manzie, C., & Brear, M. J. (9a). Newon-like exremum-seeking par I: heory. In 48h IEEE onferene on deision and onrol. Moase, W. H., Manzie, C., & Brear, M. J. (9b). Newon-like exremum-seeking par II: simulaion and experimens. In 48h IEEE onferene on deision and onrol. Skorokhod, A. V., Hoppensea, F. C., & Salehi, H. (). Random perurbaion mehods wih appliaions in siene and engineering. New York: Springer-Verlag. Sankovi, M. S., & Sipanovi, D. M. (9a). Sohasi exremum seeking wih appliaions o mobile sensor neworks. In Proeedings of he 9 Amerian onrol onferene. S. Louis, Missouri, USA, June (pp ). Sankovi, M. S., & Sipanovi, D. M. (9b). Disree ime exremum seeking for auonomous vehile arge raking in sohasi environmen. In 48h IEEE onf. deision and onrol (pp ). Tan, Y., Nešić, D., & Mareels, I. (6). On non-loal sabiliy properies of exremum seeking onrol. Auomaia, 4(6), Zhang, C., Arnold, D., Ghods, N., Siranosian, A., & Krsi, M. (7). Soure seeking wih nonholonomi uniyle wihou posiion measuremen and wih uning of forward veloiy. Sysems and Conrol Leers, 56, Shu-Jun Liu reeived a B.S. degree in Mahemais from Sihuan Universiy, Chengdu, China, in 999, a M.S. degree in Operaional Researh and Cyberneis from he same universiy, in, and a Ph.D. degree in Operaional Researh and Cyberneis from he Insiue of Sysems Siene (ISS), Chinese Aademy of Sienes (CAS), Beijing, China, in 7. From 8 o 9, she held a posdooral posiion in he Deparmen of Mehanial and Aerospae Engineering, Universiy of California, San Diego. Sine, she has been wih he Deparmen of Mahemais of Souheas Universiy, Nanjing, China, where she is now an assisan professor. Her urren researh ineress inlude sohasi sysems, adapive onrol and sohasi exremum seeking. Miroslav Krsi is he Daniel L. Alspah Professor and he founding Direor of he Cymer Cener for Conrol Sysems and Dynamis (CCSD) a UC San Diego. He reeived his Ph.D. in 994 from UC Sana Barbara and was Assisan Professor a he Universiy of Maryland unil 997. He is a oauhor of eigh books: Nonlinear and Adapive Conrol Design (Wiley, 995), Sabilizaion of Nonlinear Unerain Sysems (Springer, 998), Flow Conrol by Feedbak (Springer, ), Real-ime Opimizaion by Exremum Seeking Conrol (Wiley, 3), Conrol of Turbulen and Magneohydrodynami Channel Flows (Birkhauser, 7), Boundary Conrol of PDEs: A Course on Baksepping Designs (SIAM, 8), Delay Compensaion for Nonlinear, Adapive, and PDE Sysems (Birkhauser, 9), and Adapive Conrol of Paraboli PDEs (Prineon, ). Krsi is a Fellow of IEEE and IFAC and has reeived he Axelby and Shuk paper prizes, NSF Career, ONR Young Invesigaor, and PECASE award. He has held he appoinmen of Springer Disinguished Visiing Professor of Mehanial Engineering a UC Berkeley.