RL for Large State Spaces: Policy Gradient. Alan Fern

Size: px

Start display at page:

Download "RL for Large State Spaces: Policy Gradient. Alan Fern"

Charla Smith
6 years ago
Views:

1 RL for Lrge Se Spce: Polcy Grden Aln Fern

2 Movon for Polcy Serch So fr ll of our RL echnque hve red o lern n ec or pprome vlue funcon or Q-funcon Lern opml vlue of beng n e or kng n con from e. Vlue funcon cn ofen be much more comple o repreen hn he correpondng polcy Do we relly cre bou knowng Qlef = Qrgh = Or ju h rgh beer hn lef n e Move erchng drecly n prmeerzed polcy pce Byp lernng vlue funcon nd drecly opmze he vlue of polcy 2

Movon for Polcy Serch Conder pplyng RL o

3 Movon for Polcy Serch Conder pplyng RL o problem wh huge con pce Connou: Acon re vecor n ome ubpce of R d Combnorl: Acon re vecor of dcree vrble wh fne domn # con eponenl n # of vrble. Acon = Connuou Moor Torque Acon = Jon Dcree Trge ow would we repreen he Q-funcon Q? 3

4 Movon for Polcy Serch Conder pplyng RL o problem wh huge con pce Connou: Acon re vecor n ome ubpce of R d Combnorl: Acon re vecor of dcree vrble wh fne domn # con eponenl n # of vrble. Could ry o defne feure over e-con pr Qˆ 0 f 2 f2... n fn or ue non-lner ppromor In eher ce cn be chllengng o compue greedy con rg m Q θ Serch over n nfne or eponenlly lrge e of con 4

5 Movon for Polcy Serch Conder pplyng RL o problem wh huge con pce Connou: Acon re vecor n ome ubpce of R d Combnorl: Acon re vecor of dcree vrble wh fne domn # con eponenl n # of vrble. Ined drecly lern polcy h ke e npu nd oupu con e.g. neurl nework Robo Senor ory Torque moor... Torque moor n 5

6 6 Remnder: Grden Acen Gven funcon f n of n rel vlue = n uppoe we wn o mmze f wh repec o A common pproch o dong h grden cen The grden of f pon denoed by f n n-dmenonl vecor h pon n he drecon where f ncree mo eeply pon Vecor clculu ell u h f ju vecor of prl dervve where lm 0 f f f n n f f f

7 Ade: Grden Acen Grden cen ervely follow he grden drecon rng ome nl pon Inlze o rndom vlue Repe unl oppng condon f θ Locl opm of f Wh proper decy of lernng re grden decen gurneed o converge o locl opm. θ 2 7

8 RL v Polcy Grden Acen The polcy grden pproch h he followng chem:. Selec pce of prmeerzed polce 2. Compue he grden of he vlue of curren polcy wr prmeer 3. Move prmeer n he drecon of he grden 4. Repe hee ep unl we rech locl mm 5. Pobly lo dd n rck for delng wh bd locl mm e.g. rndom rer So we mu nwer he followng queon: ow hould we repreen prmeerzed polce? ow cn we compue he grden? 8

9 Prmeerzed Polce One emple of pce of prmerc polce : rg m Qˆ where ˆ Q my be lner funcon e.g. Qˆ 0 f 2 f2... n fn The gol o lern prmeer h gve good polcy Noe h no mporn h cul Q-funcon Rher we only requre goodne ˆ Q ˆ Q be cloe o he good rnkng con n order of 9

10 Polcy Grden Acen For mplcy we wll mke he followng umpon: Ech run/rjecory of polcy r from fed nl e Ech run/rjecory lwy reche ermnl e n fne number of ep lernvely we f he horzon o Le be epeced vlue of polcy nl e ju he epeced dcouned ol rewrd of rjecory of Our objecve o fnd h mmze 0

11 Polcy Grden Acen Polcy grden cen ell u o ervely upde prmeer v: Problem: generlly very comple nd rre h we cn compue cloed form for he grden of even f we hve n ec model of he yem. Key de: eme he grden bed on eperence

12 Grden Emon Concern: Compung or emng he grden of dconnuou funcon cn be problemc. For our emple prmerc polcy rg m Qˆ connuou? 2

13 Emple: Dconnou Sr Se 0.0 R=0 A0 R = 00 A.0 S2 R = -00 ˆ 0 rg mq 0 rg m f 0 2 f2 0 Conder fng θ 2 nd vryng θ Q θ 0 A0 = θ f 0 A0 + θ 2 f 2 0 A0 π θ 0 = A0 dconnuou 00 π θ 0 = A Q θ 0 A = θ f 0 A + θ 2 f 2 0 A -00 3

14 Grden Emon Concern: Compung or emng he grden of dconnuou funcon cn be problemc. For our emple prmerc polcy rg m Qˆ connuou? No. There re vlue of where rbrrly mll chnge cue he polcy o chnge. Snce dfferen polce cn hve dfferen vlue h men h chngng cn cue dconnuou jump of. 4

15 Probblc Polce We would lke o vod polce h drclly chnge wh mll prmeer chnge ledng o dconnue A probblc polcy ke e npu nd reurn drbuon over con Gven e reurn he probbly h elec con n Noe h ll well defned for probblc polce Now uncerny of rjecore come from envronmen nd polcy Impornly f connuou relve o chngng hen lo connuou relve o chngng A common form for probblc polce he ofm funcon or Bolzmnn eploron funcon ep Qˆ Pr ep Qˆ ' ' A 5

16 Emple: Dconnou Sr Se 0 Pr.0 R=0 A0 R = 00 A ' A.0 Q ˆ S2 ep ep Qˆ ' R = -00 ρ θ = 00 π θ A0 00 π θ = 00 π θ A0 π θ A = 00 2π θ A0 A 00 π θ 0 A0 π θ 0 A = 0.5 π F θ 2 nd vryng θ θ 0 A -00 6

17 7 Emprcl Grden Emon Our fr nïve pproch o emng o mply compue emprcl grden eme Recll h θ = θ θ n nd o we cn compue he grden by emprclly emng ech prl dervve So for mll we cn eme he prl dervve by Th requre emng n+ vlue: lm 0 n n n n... n

18 Emprcl Grden Emon ow do we eme he qune n n... For ech e of prmeer mply eecue he polcy for N rl/epode nd verge he vlue cheved cro he rl Trl of ρθ θ 2 θ 3 Trl of ρθ + ε θ 2 θ 3 Trl of ρθ θ 2 θ 3 + ε

19 Emprcl Grden Emon ow do we eme he qune n n... For ech e of prmeer mply eecue he polcy for N rl/epode nd verge he vlue cheved cro he rl Th requre ol of Nn+ epode o ge grden eme For ochc envronmen nd polce he vlue of N mu be relvely lrge o ge good eme of he rue vlue Ofen we wn o ue relvely lrge number of prmeer e.g. neurl nework Ofen epenve o run epode of he polcy So whle h cn work well n mny uon ofen no prccl pproch compuonlly 9

20 Lkelhood Ro Grden Emon The emprcl grden mehod cn be ppled even when he funconl form of he polcy blck bo.e. don know mppng from o con drbuon owever h men we re no ble o eplo knowledge bou he polcy form when we know If we know he funconl form of he polcy nd cn compue grden wh repec o we cn do beer. Eme drecly from rjecore of ju he curren polcy Trl of ρθ θ 2 θ 3 Trl of ρθ + ε θ 2 θ 3 Trl of ρθ θ 2 θ 3 + ε

21 Lkelhood Ro Grden Emon Our pproch bed on he generl de of lkelhood ro emon. We ll fr decrbe he generl de nd hen pply o polcy grden emon Trl of ρθ θ 2 θ 3 Trl of ρθ + ε θ 2 θ 3 Trl of ρθ θ 2 θ 3 + ε

22 Ade: Generl Lkelhood Ro Grden Eme Le F be rel-vlued funcon over fne domn D Everyhng generlze o connuou domn Le X be rndom vrble over D drbued ccordng o P he prmeer vecor of h drbuon Conder he epecon of FX condoned on F X P D E F We wh o eme gven by: P F D D P F Ofen no cloed form for um D cn be huge 22

23 23 Rewrng So ju he epeced vlue of z XFX Ge unbed eme of by vergng over N mple of X j he j h mple of X Only requre bly o mple X nd o compue z Doe no depend on how bg D! log X F X z E F P P F P P P F P D D D z j N j j F z N X P θ Ade: Generl Lkelhood Ro Grden Eme

24 Defne equence of e nd - con genered for ngle epode of In generl wll dffer cro epode. X rndom due o polcy nd envronmen drbued : Defne he ol rewrd of X epeced ol rewrd of We wn o eme he grden of ρθ Apply lkelhood ro mehod! 24 Applcon o Polcy Grden 2 2 X T P R X F R E X F E

25 Recll for rndom vrble X we hve unbed eme We cn genere mple of by runnng polcy from he r e unl ermnl e h mpled epode um of oberved rewrd durng j h epode 25 Applcon o Polcy Grden j N j j F z N 2 2 X 2 2 j j R F log log log log log T T P z Doe no depend on knowng model! Allow model-free mplemenon.

26 Recll for rndom vrble X we hve unbed eme Conder ngle erm Snce con me doe no nfluence rewrd before me + we cn derve he followng reul: h non-rvl o derve Th jufe ung modfed compuon for ech erm: 26 Applcon o Polcy Grden j N j j F z N Tol rewrd fer me. k k R F z log k k R E F z E log k k R log Eme ll unbed bu generlly h mller vrnce.

27 Applcon o Polcy Grden Pung everyhng ogeher we ge: lengh of rjecory j Oberved rewrd fer kng j n e j N j j log j j R j k N j k # of mpled rjecore of curren polcy Drecon o move prmeer n order o ncree he probbly h polcy elec j n e j Inerpreon: ech epode conrbue weghed um of grden drecon Grden drecon for ncreng probbly of j n j weghed by um of rewrd oberved fer kng j n j Inuvely h ncree/decree probbly of kng con h re ypclly followed by good/bd rewrd equence 27

28 Bc Polcy Grden Algorhm Repe unl oppng condon. Eecue for N epode o ge e of e con rewrd equence 2. N j j log j j R j k N j k 3. Unnecery o ore N epode ue onlne men upde n ep 2 Ddvnge: mll # of upde per # epode Alo no well defned for non-epodc nfne horzon problem Onlne polcy grden lgorhm perform upde fer ech ep n envronmen ofen lern fer 28

29 29 Towrd Onlne Algorhm Conder he compuon for ngle epode Noce h we cn compue z n n onlne wy We cn now ncremenlly compue Δ T for ech epode Sorge requremen depend only on # of polcy prmeer k k k z R R R 2 2 log log log ; 0 z z z 0 ; 0 z R Ju reorgnze erm

30 Towrd Onlne Algorhm So he overll grden eme cn be done by ncremenlly compung Δ for ech of N epode nd compung her men The men of he Δ cro epode cn be compued onlne Tol memory requremen ll depend only on # prmeer Independen of lengh nd number of epode! Bu wh f epode go on forever? We could connully mnn Δ T ung conn moun of memory bu we would never cully do prmeer upde Alo Δ T cn hve nfne vrnce n h eng we wll no how h Soluon: Upde polcy prmeer fer ech rewrd oberved rher hn mply upde grden eme Δ T Inroduce dcounng Reul n OLPOMDP lgorhm 30

31 Onlne Polcy Grden OLPOMDP Repe forever. Oberve e 2. Drw con ccordng o drbuon 3. Eecue nd oberve rewrd r 4. ;; dcouned um of ;; grden drecon 5. z z log r z Perform polcy upde ech me ep nd eecue ndefnely Th he OLPOMDP lgorhm [Ber & Brle 2000]

32 Inerpreon Repe forever. Oberve e 2. Drw con ccordng o drbuon 3. Eecue nd oberve rewrd r 4. ;; dcouned um of ;; grden drecon 5. z z log r z Sep 4 compue n elgbly rce z Dcouned um of grden over prevou e-con pr Pon n drecon of prmeer pce h ncree probbly of kng more recen con n more recen e For pove rewrd ep 5 wll ncree probbly of recen con nd decree for negve rewrd.

33 33 Compung he Grden of Polcy Boh lgorhm requre compuon of For he Bolzmnn drbuon wh lner ppromon we hve: where ere he prl dervve compong he grden re: log A Q Q ' ' ˆ ep ˆ ep... ˆ f f f Q n n ' ' ' log f f

34 Connou Acon Spce ow do we repreen ochc polcy over connuou con pce? ech con vecor R d Common Choce: Lern mppng φ from e/npu o d-dmenonl con vecor e.g. neurl nework Polcy d-dm Gun wh men φ nd covrnce Σ π = Gunφ Σ φ S Gunφ Σ... φ d nd Seleced or Lerned 34

35 Conrollng elcoper Polcy grden echnque hve been ued o cree conroller for dffcul helcoper mneuver For emple nvered helcoper flgh. 35

36 Qudruped Locomoon Opmze g of 4-legged robo over rough errn 36

37 Procve Secury Inellgen Bone Conroller Ued OLPOMDP o procvely dcover mmlly dmgng bone ck n peer-o-peer nework

38 Polcy Grden Recp When polce hve much mpler repreenon hn he correpondng vlue funcon drec erch n polcy pce cn be good de Or f we lredy hve comple prmerc conroller polcy grden llow u o focu on opmzng prmeer eng For belne lgorhm he grden eme re unbed.e. hey wll converge o he rgh vlue bu hve hgh vrnce for lrge T Cn requre lrge N o ge relble eme OLPOMDP cn rde-off b nd vrnce v he dcoun prmeer nd doe no requre noon of epode Cn be prone o fndng locl mm Mny wy of delng wh h e.g. rndom rer or nellgen nlzon. 38

39 Oher Trck Much recen work n deep lernng ue polcy grden ofen wh couple of oher rck Addng belne o reduce vrnce Only updng polcy ech ep whn ru regon o vod chngng he polcy oo drclly Tru Regon Polcy Opmzon very nce combnon of hee de. Tru Regon Polcy Opmzon John Schulmn Sergey Levne Phlpp Morz Mchel I. Jordn Peer Abbeel. Inernonl Conference on Mchne Lernng ICML 205. pdf rxv preprn Implemenon vlble. 39

RL for Large State Spaces: Policy Gradient. Alan Fern

RL for Large State Spaces: Policy Gradient. Alan Fern RL for Lrge Se Spce: Polcy Grden Aln Fern RL v Polcy Grden Serch So fr ll of our RL echnque hve red o lern n ec or pprome uly funcon or Q-funcon Lern opml vlue of beng n e or kng n con from e. Vlue funcon