Reinforcement Learning II. George Konidaris
|
|
|
- Daisy Richardson
- 5 years ago
- Views:
Transcription
1 Reinforcement Learning II George Konidaris Fall 2018
2 Reinforcement Learning π : S A max R = t=0 t r t
3 MDPs Agent interacts with an environment At each time t: Receives sensor signal Executes action Transition: a t new sensor signal reward r t s t s t+1 Goal: find policy of discounted future rewards): [ π that maximizes expected return (sum max π E R = ] γ t r t t=0
4 Markov Decision Processes S A γ : set of states : set of actions : discount factor <S,A,γ,R,T > R : reward function R(s, a, s ) is the reward received taking action s and transitioning to state. a from state s T : transition function T (s s, a) is the probability of transitioning to state s taking action a in state s. after RL: one or both of T, R unknown.
5 The World
6 Real-Valued States What if the states are real-valued? Cannot use table to represent Q. States may never repeat: must generalize vs
7 RL Example: States: (θ 1, θ 1,θ 2, θ 2 ) (real-valued vector) Actions: +1, -1, 0 units of torque added to elbow Transition function: physics! Reward function: -1 for every step
8 <latexit sha1_base64="tlkfosn5optzqbnl1osll2fs1ui=">aaacwxicbzddattaeixxatk67p/bxpzmiak4ja1sklq3hddc9dkboglyjhitv/bi/rg7owyj9hb9jnihyg36bl0rljq/bxy+zplhlpoxsnqk41+d6mng5tnn3ee9fy9fvx7tf/vuznvkcthivll3kamxshoxiklkxjroom6vom8xx6v8/idwxlrznzalmgicgvlijhssrd9otttzfas+0lktv8tnpyfuiykg4dije5dougue06ep+v7s7ap+g6tdga9tileubj3uydmcduhymosp4oo4ftyeza0dttzj1r9op5zxwhjicr0fj3fjkxodsa5e00srl0rkc5yjcucdwvhj3zbqwe5wplbyf54han3/n2ru3i91hiy10tzfz1bmy9m4oulzpjamregyfnuoqbsqhvwjmjvocfllamiddh8fpkehnelvd67k1i4ic9+ezpl7ptyes8odjpdpx8hr13vhxfaebbmhs9gndss+srm2ypz9znfshv3p/i46utfq3y5gnfxofrujaosvf0yyig==</latexit> <latexit sha1_base64="tlkfosn5optzqbnl1osll2fs1ui=">aaacwxicbzddattaeixxatk67p/bxpzmiak4ja1sklq3hddc9dkboglyjhitv/bi/rg7owyj9hb9jnihyg36bl0rljq/bxy+zplhlpoxsnqk41+d6mng5tnn3ee9fy9fvx7tf/vuznvkcthivll3kamxshoxiklkxjroom6vom8xx6v8/idwxlrznzalmgicgvlijhssrd9otttzfas+0lktv8tnpyfuiykg4dije5dougue06ep+v7s7ap+g6tdga9tileubj3uydmcduhymosp4oo4ftyeza0dttzj1r9op5zxwhjicr0fj3fjkxodsa5e00srl0rkc5yjcucdwvhj3zbqwe5wplbyf54han3/n2ru3i91hiy10tzfz1bmy9m4oulzpjamregyfnuoqbsqhvwjmjvocfllamiddh8fpkehnelvd67k1i4ic9+ezpl7ptyes8odjpdpx8hr13vhxfaebbmhs9gndss+srm2ypz9znfshv3p/i46utfq3y5gnfxofrujaosvf0yyig==</latexit> <latexit sha1_base64="tlkfosn5optzqbnl1osll2fs1ui=">aaacwxicbzddattaeixxatk67p/bxpzmiak4ja1sklq3hddc9dkboglyjhitv/bi/rg7owyj9hb9jnihyg36bl0rljq/bxy+zplhlpoxsnqk41+d6mng5tnn3ee9fy9fvx7tf/vuznvkcthivll3kamxshoxiklkxjroom6vom8xx6v8/idwxlrznzalmgicgvlijhssrd9otttzfas+0lktv8tnpyfuiykg4dije5dougue06ep+v7s7ap+g6tdga9tileubj3uydmcduhymosp4oo4ftyeza0dttzj1r9op5zxwhjicr0fj3fjkxodsa5e00srl0rkc5yjcucdwvhj3zbqwe5wplbyf54han3/n2ru3i91hiy10tzfz1bmy9m4oulzpjamregyfnuoqbsqhvwjmjvocfllamiddh8fpkehnelvd67k1i4ic9+ezpl7ptyes8odjpdpx8hr13vhxfaebbmhs9gndss+srm2ypz9znfshv3p/i46utfq3y5gnfxofrujaosvf0yyig==</latexit> <latexit sha1_base64="tlkfosn5optzqbnl1osll2fs1ui=">aaacwxicbzddattaeixxatk67p/bxpzmiak4ja1sklq3hddc9dkboglyjhitv/bi/rg7owyj9hb9jnihyg36bl0rljq/bxy+zplhlpoxsnqk41+d6mng5tnn3ee9fy9fvx7tf/vuznvkcthivll3kamxshoxiklkxjroom6vom8xx6v8/idwxlrznzalmgicgvlijhssrd9otttzfas+0lktv8tnpyfuiykg4dije5dougue06ep+v7s7ap+g6tdga9tileubj3uydmcduhymosp4oo4ftyeza0dttzj1r9op5zxwhjicr0fj3fjkxodsa5e00srl0rkc5yjcucdwvhj3zbqwe5wplbyf54han3/n2ru3i91hiy10tzfz1bmy9m4oulzpjamregyfnuoqbsqhvwjmjvocfllamiddh8fpkehnelvd67k1i4ic9+ezpl7ptyes8odjpdpx8hr13vhxfaebbmhs9gndss+srm2ypz9znfshv3p/i46utfq3y5gnfxofrujaosvf0yyig==</latexit> Value Function Approximation Represent Q function: Q(s, a, w) :R n! R Samples of form: parameter vector (s i,a i,r i,s i+1,a i+1 ) Minimize summed squared TD error: nx min w i=0 (r i + Q(s i+1,a i+1,w) Q(s i,a i,w)) 2
9 Value Function Approximation Given a function approximator, compute the gradient and descend it. Which function approximator to use? Simplest thing you can do: Linear value function approximation. Use set of basis functions Q is a linear function of them: 1,..., n ˆQ(s, a) =w (s, a) = nx w j j (s, a) j=1
10 Function Approximation One choice of basis functions: Just use state variables directly: [1,x,y] What can be represented this way? Q y x
![1st order: [1,x,y,xy] 2nd order:](/docs-images/88/115791805/images/11-4.jpg "[1,x,y,xy,x 2,y 2,x 2 y, y 2 x,")
![x 2 y 2 ] This is like a Taylor](/docs-images/88/115791805/images/11-5.jpg "expansion.")
11 Polynomial Basis More powerful: Polynomials in state variables. 1st order: [1,x,y,xy] 2nd order: [1,x,y,xy,x 2,y 2,x 2 y, y 2 x, x 2 y 2 ] This is like a Taylor expansion. What can be represented?
12 Function Approximation How to get the terms of the Taylor series? Each term has an exponent: c i 2 [0,...,d] c(x, y, z) =x c 1 y c 2 z c 3 all combinations generates basis c(x, y, z) =x = x 1 y 0 z 0 c =[1, 0, 0] c(x, y, z) =xy 2 = x 1 y 2 z 0 c =[1, 2, 0] c(x, y, z) =x 2 z 4 = x 2 y 0 z 4 c =[2, 0, 4] c(x, y, z) =y 3 z 1 = x 0 y 3 z 1 c =[0, 3, 1]
13 Function Approximation Another: Fourier terms on state variables. [1,cos(πx),cos(πy), cos(π[x + y])] cos( c [x, y, z]) coefficient vector
14 <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> Objective Function Minimization First, let s do stochastic gradient descent. As each data point (transition) comes in compute gradient of objective w.r.t. data point descend gradient a little bit ˆQ(s, a) =w (s, a) nx min w i=0 (r i + w (s i+1,a i+1 ) w (s i,a i )) 2
15 <latexit sha1_base64="bv18zouygfnxpknr08xsik+lcxy=">aaacgxicbzhbattaeizxsg9peojt3rt0zqgp2osafaojlejob3qzqp0elcng65w18uordkcxruiqt9knchmlrmxb66qdc9/+/wyz/juuslokgt+ev/xo8zon2892nr94+wq3t/f6wurkcdhiwmlzlaavshzirjkuucqnwdxr4jkzf2v9yxthrntft1qwypljrjcp5ehoinu/fnf9fqbyp2zgc3sxbvyhqlvmcjeskq3axllvzpjncaui+fqtrgumbzau2+edwdumaq5hsdegw1moitjyltfwjcfxf4241w+oglxbqwg76louzupebttvvmpfqvyhtemwkglsoyhjlwh2osqkevkcz2lssmbc2em9cqubj06zqqqnowxbsv13osbc2mweum4ckbp3vvb8nzeukd2d1liokxifxy9kkwwkou0eptiitmrpalmr7q3amztiyf3pxpze6zlhyhuxthg/h4dwcxwuov7xqx/2tctom71nh9iaheyenbhv7jynggd33q731nvnb/ldp/cp yrvlf/4dsb+9oq==</latexit> <latexit sha1_base64="bv18zouygfnxpknr08xsik+lcxy=">aaacgxicbzhbattaeizxsg9peojt3rt0zqgp2osafaojlejob3qzqp0elcng65w18uordkcxruiqt9knchmlrmxb66qdc9/+/wyz/juuslokgt+ev/xo8zon2892nr94+wq3t/f6wurkcdhiwmlzlaavshzirjkuucqnwdxr4jkzf2v9yxthrntft1qwypljrjcp5ehoinu/fnf9fqbyp2zgc3sxbvyhqlvmcjeskq3axllvzpjncaui+fqtrgumbzau2+edwdumaq5hsdegw1moitjyltfwjcfxf4241w+oglxbqwg76louzupebttvvmpfqvyhtemwkglsoyhjlwh2osqkevkcz2lssmbc2em9cqubj06zqqqnowxbsv13osbc2mweum4ckbp3vvb8nzeukd2d1liokxifxy9kkwwkou0eptiitmrpalmr7q3amztiyf3pxpze6zlhyhuxthg/h4dwcxwuov7xqx/2tctom71nh9iaheyenbhv7jynggd33q731nvnb/ldp/cp yrvlf/4dsb+9oq==</latexit> <latexit sha1_base64="bv18zouygfnxpknr08xsik+lcxy=">aaacgxicbzhbattaeizxsg9peojt3rt0zqgp2osafaojlejob3qzqp0elcng65w18uordkcxruiqt9knchmlrmxb66qdc9/+/wyz/juuslokgt+ev/xo8zon2892nr94+wq3t/f6wurkcdhiwmlzlaavshzirjkuucqnwdxr4jkzf2v9yxthrntft1qwypljrjcp5ehoinu/fnf9fqbyp2zgc3sxbvyhqlvmcjeskq3axllvzpjncaui+fqtrgumbzau2+edwdumaq5hsdegw1moitjyltfwjcfxf4241w+oglxbqwg76louzupebttvvmpfqvyhtemwkglsoyhjlwh2osqkevkcz2lssmbc2em9cqubj06zqqqnowxbsv13osbc2mweum4ckbp3vvb8nzeukd2d1liokxifxy9kkwwkou0eptiitmrpalmr7q3amztiyf3pxpze6zlhyhuxthg/h4dwcxwuov7xqx/2tctom71nh9iaheyenbhv7jynggd33q731nvnb/ldp/cp yrvlf/4dsb+9oq==</latexit> <latexit sha1_base64="bv18zouygfnxpknr08xsik+lcxy=">aaacgxicbzhbattaeizxsg9peojt3rt0zqgp2osafaojlejob3qzqp0elcng65w18uordkcxruiqt9knchmlrmxb66qdc9/+/wyz/juuslokgt+ev/xo8zon2892nr94+wq3t/f6wurkcdhiwmlzlaavshzirjkuucqnwdxr4jkzf2v9yxthrntft1qwypljrjcp5ehoinu/fnf9fqbyp2zgc3sxbvyhqlvmcjeskq3axllvzpjncaui+fqtrgumbzau2+edwdumaq5hsdegw1moitjyltfwjcfxf4241w+oglxbqwg76louzupebttvvmpfqvyhtemwkglsoyhjlwh2osqkevkcz2lssmbc2em9cqubj06zqqqnowxbsv13osbc2mweum4ckbp3vvb8nzeukd2d1liokxifxy9kkwwkou0eptiitmrpalmr7q3amztiyf3pxpze6zlhyhuxthg/h4dwcxwuov7xqx/2tctom71nh9iaheyenbhv7jynggd33q731nvnb/ldp/cp yrvlf/4dsb+9oq==</latexit> <latexit sha1_base64="1aqpcpgn7xrgenaumw7yl36aiuk=">aaacdxicbzhdaxnbfmvnt37u2mrurxeuriwhtu4wqv8krv98rgdaqjyudyezyzj5wgbugskyf6gppvsn+orkazstfwz+nhogo5wpkiu9jcmpkn65dfvo3d17e/f3dx487dx6foft7bgyckusuyrqcywngjakja4qj1axslwws49l//k7cf5a84uwlrhpnbhzso4uplwzp4wje4dm1zpv5gnsfjwqkvfsd1wu4rcycwqnmiemjy1bvk1lz4foydq+blxdh+ai5lsbutrlhzinj1pqr+t8218j73st42q1cbpsdxtzzs7zzq9sbhmthsgu0pthmlq0atcr5eq0e1ntryv8hhmxdghqcz9qvgw18dioyyitc8cqrnr/bzsovv/oiiq10trf95bi/7xhtex7usnnvzmwfl2orbwqhwxbmjzocfklamiddg8fpkwhnmkfbg0prj0rfr4nzatxe7gjfyfhaedpb7tnhzyd7bkn7dnrszs9y2fseztna8bzz2gn2o8oot/xs/hf/godjapnnsdsa+i3fwab5rnm</latexit> <latexit sha1_base64="1aqpcpgn7xrgenaumw7yl36aiuk=">aaacdxicbzhdaxnbfmvnt37u2mrurxeuriwhtu4wqv8krv98rgdaqjyudyezyzj5wgbugskyf6gppvsn+orkazstfwz+nhogo5wpkiu9jcmpkn65dfvo3d17e/f3dx487dx6foft7bgyckusuyrqcywngjakja4qj1axslwws49l//k7cf5a84uwlrhpnbhzso4uplwzp4wje4dm1zpv5gnsfjwqkvfsd1wu4rcycwqnmiemjy1bvk1lz4foydq+blxdh+ai5lsbutrlhzinj1pqr+t8218j73st42q1cbpsdxtzzs7zzq9sbhmthsgu0pthmlq0atcr5eq0e1ntryv8hhmxdghqcz9qvgw18dioyyitc8cqrnr/bzsovv/oiiq10trf95bi/7xhtex7usnnvzmwfl2orbwqhwxbmjzocfklamiddg8fpkwhnmkfbg0prj0rfr4nzatxe7gjfyfhaedpb7tnhzyd7bkn7dnrszs9y2fseztna8bzz2gn2o8oot/xs/hf/godjapnnsdsa+i3fwab5rnm</latexit> <latexit sha1_base64="1aqpcpgn7xrgenaumw7yl36aiuk=">aaacdxicbzhdaxnbfmvnt37u2mrurxeuriwhtu4wqv8krv98rgdaqjyudyezyzj5wgbugskyf6gppvsn+orkazstfwz+nhogo5wpkiu9jcmpkn65dfvo3d17e/f3dx487dx6foft7bgyckusuyrqcywngjakja4qj1axslwws49l//k7cf5a84uwlrhpnbhzso4uplwzp4wje4dm1zpv5gnsfjwqkvfsd1wu4rcycwqnmiemjy1bvk1lz4foydq+blxdh+ai5lsbutrlhzinj1pqr+t8218j73st42q1cbpsdxtzzs7zzq9sbhmthsgu0pthmlq0atcr5eq0e1ntryv8hhmxdghqcz9qvgw18dioyyitc8cqrnr/bzsovv/oiiq10trf95bi/7xhtex7usnnvzmwfl2orbwqhwxbmjzocfklamiddg8fpkwhnmkfbg0prj0rfr4nzatxe7gjfyfhaedpb7tnhzyd7bkn7dnrszs9y2fseztna8bzz2gn2o8oot/xs/hf/godjapnnsdsa+i3fwab5rnm</latexit> <latexit sha1_base64="1aqpcpgn7xrgenaumw7yl36aiuk=">aaacdxicbzhdaxnbfmvnt37u2mrurxeuriwhtu4wqv8krv98rgdaqjyudyezyzj5wgbugskyf6gppvsn+orkazstfwz+nhogo5wpkiu9jcmpkn65dfvo3d17e/f3dx487dx6foft7bgyckusuyrqcywngjakja4qj1axslwws49l//k7cf5a84uwlrhpnbhzso4uplwzp4wje4dm1zpv5gnsfjwqkvfsd1wu4rcycwqnmiemjy1bvk1lz4foydq+blxdh+ai5lsbutrlhzinj1pqr+t8218j73st42q1cbpsdxtzzs7zzq9sbhmthsgu0pthmlq0atcr5eq0e1ntryv8hhmxdghqcz9qvgw18dioyyitc8cqrnr/bzsovv/oiiq10trf95bi/7xhtex7usnnvzmwfl2orbwqhwxbmjzocfklamiddg8fpkwhnmkfbg0prj0rfr4nzatxe7gjfyfhaedpb7tnhzyd7bkn7dnrszs9y2fseztna8bzz2gn2o8oot/xs/hf/godjapnnsdsa+i3fwab5rnm</latexit> <latexit sha1_base64="ml8kp3bas5qyii3vkbyy49ey8ns=">aaacf3icbzfdaxnbfizn169aruyfvfdmybatwsnuedqloeinlxvmw8iky9njbdjmzmezowsiy174m/0f/rtophdtemdg4x3p4rzeysutpcxj7yi+dfvo3xt79/cfpdx49ljz5omzt7utciistu4iry+1kuwqfgl5utmjjtfypj9/wfnnp6xzypbfavnjscfpqqolkikudx7xwqfoeiwofor2l8ei+9ec97xjgvupas9l4fow1aoxktgepkvjebbaxcqs8gqmej60hqbteeag+gbvybhtofam6gn3ajqjplwez51umkjwbtch3ukxbes061zxirw1ksujjd6p0qsicbm6w2jz7vpaywrfhkdyflbei/24wufvwuugtkcwlrysyk3+o9gg8x5p8tbpkgb+urcs/+enaio+jbtvvjxjumwwfbugsrdkhsbksuf6gqcfu+fwedmm2vp4nz0tubvzwty3izn0eg434ex4kab+9q578nmb0r57yv6xhkvze3bcvrjtnmscxuuh0fporrzfb+jbngxa42g784ztvpzxdytpv3i=</latexit> <latexit sha1_base64="ml8kp3bas5qyii3vkbyy49ey8ns=">aaacf3icbzfdaxnbfizn169aruyfvfdmybatwsnuedqloeinlxvmw8iky9njbdjmzmezowsiy174m/0f/rtophdtemdg4x3p4rzeysutpcxj7yi+dfvo3xt79/cfpdx49ljz5omzt7utciistu4iry+1kuwqfgl5utmjjtfypj9/wfnnp6xzypbfavnjscfpqqolkikudx7xwqfoeiwofor2l8ei+9ec97xjgvupas9l4fow1aoxktgepkvjebbaxcqs8gqmej60hqbteeag+gbvybhtofam6gn3ajqjplwez51umkjwbtch3ukxbes061zxirw1ksujjd6p0qsicbm6w2jz7vpaywrfhkdyflbei/24wufvwuugtkcwlrysyk3+o9gg8x5p8tbpkgb+urcs/+enaio+jbtvvjxjumwwfbugsrdkhsbksuf6gqcfu+fwedmm2vp4nz0tubvzwty3izn0eg434ex4kab+9q578nmb0r57yv6xhkvze3bcvrjtnmscxuuh0fporrzfb+jbngxa42g784ztvpzxdytpv3i=</latexit> <latexit sha1_base64="ml8kp3bas5qyii3vkbyy49ey8ns=">aaacf3icbzfdaxnbfizn169aruyfvfdmybatwsnuedqloeinlxvmw8iky9njbdjmzmezowsiy174m/0f/rtophdtemdg4x3p4rzeysutpcxj7yi+dfvo3xt79/cfpdx49ljz5omzt7utciistu4iry+1kuwqfgl5utmjjtfypj9/wfnnp6xzypbfavnjscfpqqolkikudx7xwqfoeiwofor2l8ei+9ec97xjgvupas9l4fow1aoxktgepkvjebbaxcqs8gqmej60hqbteeag+gbvybhtofam6gn3ajqjplwez51umkjwbtch3ukxbes061zxirw1ksujjd6p0qsicbm6w2jz7vpaywrfhkdyflbei/24wufvwuugtkcwlrysyk3+o9gg8x5p8tbpkgb+urcs/+enaio+jbtvvjxjumwwfbugsrdkhsbksuf6gqcfu+fwedmm2vp4nz0tubvzwty3izn0eg434ex4kab+9q578nmb0r57yv6xhkvze3bcvrjtnmscxuuh0fporrzfb+jbngxa42g784ztvpzxdytpv3i=</latexit> <latexit sha1_base64="ml8kp3bas5qyii3vkbyy49ey8ns=">aaacf3icbzfdaxnbfizn169aruyfvfdmybatwsnuedqloeinlxvmw8iky9njbdjmzmezowsiy174m/0f/rtophdtemdg4x3p4rzeysutpcxj7yi+dfvo3xt79/cfpdx49ljz5omzt7utciistu4iry+1kuwqfgl5utmjjtfypj9/wfnnp6xzypbfavnjscfpqqolkikudx7xwqfoeiwofor2l8ei+9ec97xjgvupas9l4fow1aoxktgepkvjebbaxcqs8gqmej60hqbteeag+gbvybhtofam6gn3ajqjplwez51umkjwbtch3ukxbes061zxirw1ksujjd6p0qsicbm6w2jz7vpaywrfhkdyflbei/24wufvwuugtkcwlrysyk3+o9gg8x5p8tbpkgb+urcs/+enaio+jbtvvjxjumwwfbugsrdkhsbksuf6gqcfu+fwedmm2vp4nz0tubvzwty3izn0eg434ex4kab+9q578nmb0r57yv6xhkvze3bcvrjtnmscxuuh0fporrzfb+jbngxa42g784ztvpzxdytpv3i=</latexit> <latexit sha1_base64="ncbb1cg2m3ji28bcu57d1mszq7u=">aaacynicbzfnaxsxeibltduk7pedhnvdufowcrt2sya5hvasywp1yvcazvartvvlq0wajtgl/2d+qe+fxnttt5e/ohxsachd+84ww6u0vnjrgh5vbhuphj/zp3jafpb8xctxrfbhltov5wlajtj2mkitshziqjkugjzwoe6vue5nn1f+9y2wtpriky1kmdy4kwquozkxklywk5ftf2wioq/xblvgmepmm0mql1pzdukt+9hypeagegafyl7tifem7efs5wrkvbwcfptrjk1oebyucx5ctiuo29zl0vozz4zxwhtefto3isksxjvaklyjztouncirz3airh4l1mkn63uas3jnlqxyy/0rcnbqvxm1aucwovwdgmnq7nsr8x/eqkl8bfzloqxifhyzkk8ukifvtjbjkziphqfkvvpbgu/riif/attbumnmhklb+msi+zk8hkupx5hnlyed80/bja7ya/awdvnettk5u2cxbma4u2w/2g/2p/ejaabt4gjtgjs2m0dsp4i3dweftl8=</latexit> <latexit sha1_base64="ncbb1cg2m3ji28bcu57d1mszq7u=">aaacynicbzfnaxsxeibltduk7pedhnvdufowcrt2sya5hvasywp1yvcazvartvvlq0wajtgl/2d+qe+fxnttt5e/ohxsachd+84ww6u0vnjrgh5vbhuphj/zp3jafpb8xctxrfbhltov5wlajtj2mkitshziqjkugjzwoe6vue5nn1f+9y2wtpriky1kmdy4kwquozkxklywk5ftf2wioq/xblvgmepmm0mql1pzdukt+9hypeagegafyl7tifem7efs5wrkvbwcfptrjk1oebyucx5ctiuo29zl0vozz4zxwhtefto3isksxjvaklyjztouncirz3airh4l1mkn63uas3jnlqxyy/0rcnbqvxm1aucwovwdgmnq7nsr8x/eqkl8bfzloqxifhyzkk8ukifvtjbjkziphqfkvvpbgu/riif/attbumnmhklb+msi+zk8hkupx5hnlyed80/bja7ya/awdvnettk5u2cxbma4u2w/2g/2p/ejaabt4gjtgjs2m0dsp4i3dweftl8=</latexit> <latexit sha1_base64="ncbb1cg2m3ji28bcu57d1mszq7u=">aaacynicbzfnaxsxeibltduk7pedhnvdufowcrt2sya5hvasywp1yvcazvartvvlq0wajtgl/2d+qe+fxnttt5e/ohxsachd+84ww6u0vnjrgh5vbhuphj/zp3jafpb8xctxrfbhltov5wlajtj2mkitshziqjkugjzwoe6vue5nn1f+9y2wtpriky1kmdy4kwquozkxklywk5ftf2wioq/xblvgmepmm0mql1pzdukt+9hypeagegafyl7tifem7efs5wrkvbwcfptrjk1oebyucx5ctiuo29zl0vozz4zxwhtefto3isksxjvaklyjztouncirz3airh4l1mkn63uas3jnlqxyy/0rcnbqvxm1aucwovwdgmnq7nsr8x/eqkl8bfzloqxifhyzkk8ukifvtjbjkziphqfkvvpbgu/riif/attbumnmhklb+msi+zk8hkupx5hnlyed80/bja7ya/awdvnettk5u2cxbma4u2w/2g/2p/ejaabt4gjtgjs2m0dsp4i3dweftl8=</latexit> <latexit sha1_base64="ncbb1cg2m3ji28bcu57d1mszq7u=">aaacynicbzfnaxsxeibltduk7pedhnvdufowcrt2sya5hvasywp1yvcazvartvvlq0wajtgl/2d+qe+fxnttt5e/ohxsachd+84ww6u0vnjrgh5vbhuphj/zp3jafpb8xctxrfbhltov5wlajtj2mkitshziqjkugjzwoe6vue5nn1f+9y2wtpriky1kmdy4kwquozkxklywk5ftf2wioq/xblvgmepmm0mql1pzdukt+9hypeagegafyl7tifem7efs5wrkvbwcfptrjk1oebyucx5ctiuo29zl0vozz4zxwhtefto3isksxjvaklyjztouncirz3airh4l1mkn63uas3jnlqxyy/0rcnbqvxm1aucwovwdgmnq7nsr8x/eqkl8bfzloqxifhyzkk8ukifvtjbjkziphqfkvvpbgu/riif/attbumnmhklb+msi+zk8hkupx5hnlyed80/bja7ya/awdvnettk5u2cxbma4u2w/2g/2p/ejaabt4gjtgjs2m0dsp4i3dweftl8=</latexit> Gradient For each weight wj: (r i j w (s i+1,a i+1 ) w (s i,a i )) 2 = 2 i=0 nx i=0 (r i + w (s i+1,a i+1 ) w (s i,a i )) j (s i,a i ) so for time i the contribution for weight wj is: TD error w i+1 = w i + (r i + w (s i+1,a i+1 ) w (s i,a i )) j (s i,a i ) make a step: w j,i+1 = w j,i + (r i + w (s i+1,a i+1 ) w (s i,a i )) j (s i,a i ) (s i,a i ) vector
16 λ-gradient The same logic applies when using eligibility traces. w i+1 = w i + (s i,a i ) becomes w i+1 = w i + e where vectors e t = e t 1 + (s t,a t ) e 0 = 0 [Sutton and Barto, 1998]
17 Acrobot
18 Acrobot
19 <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> <latexit sha1_base64="urphozkm+rhihkwjpmrbnxhn3t4=">aaacahicbzflaxrbfivroj5iflw6ehfzysdmea3dqyibqncnywhoepienldrqmfkqudtdtthapo/zptfipgd3grny+ekxij4oocutzhvvep6spkbtrr17/6dh9updh4/efrsefzi5am3teniwk2y7rxal5q0ykcsldivnebdkhfwzl4s/lnl4by05jvnkzhsodgylbwpshn8i9ps5feq+vrnjtxk2gsdmril9cdlevygm6dwcchcx5ygq6ay50n0l23fa66gd/abrjyccmhkpmrotqbuh4udpo4m+8ly4c6ka+iy9zzk8c9sbhmthsgu0pthmlq0atcr5eq0o1ntryv8hhmxdghqcz9qlp208c4oyyitc8cqlnv/bzsovz/riiq10ttf9hbi/7xhtewnusnnvzmwflworbwqhuxbmjzocflzamiddg8fpkwhnmi3bgwprj0rfr4nzas3e7glpwf7aebvh7vhn9cdbbo3bjf1wmoo2th7yk7yghf2zx6zpx3w+rxf0evozsoaddz3xrgnixb/autmucu=</latexit> Least-Squares TD Minimize: min w nx i=0 Error function has a bowl shape, so unique minimum. Just go right there! (r i + w (s i+1,a i+1 ) w (s i,a i )) 2
20 Least-Squares TD Derivative set to zero: nx (w (s i,a i ) r i w (s i+1,a i+1 )) (s i,a i ) T =0 w T i=1 n X i=1 (w (s i,a i ) w (s i+1,a i+1 )) T (s i,a i )= w = A 1 b nx i=1 r i T (s i,a i ) A = b = nx ( (s i,a i ) (s i+1,a i+1 )) T (s i,a i ) i=1 nx i=1 r i T (s i,a i ) [Bradtke and Barto, 1996]
21 LSTD(λ) Can derive the least-squares version of LSTD(λ) in this way. Try it at home! Write down the objective function Sample ri replaced by complex reward estimate. You will get a trace vector if you do some clever algebra. Trace vector is the same size as w. [Boyan, 1999]
22 LSTD(λ) One inversion solves for w! But: Computationally expensive. A may not be invert-able. Least-squares behavior sometimes unstable outside of data. LSPI: Least Squares Policy Iteration Requires recomputing A over historical data. ai+1 changes with the policy [Lagoudakis and Parr, 2003]
23 Linear Methods Don t Scale Why not? They re complete. They have nice properties (bowl-shaped error). They are easy to use! How many basis functions in a complete nth order Taylor series of d variables? (n + 1) d
24 Function Approximation TD-Gammon: Tesauro (circa ) At or near best human level Learn to play Backgammon through self-play 1.5 million games Neural network function approximator TD(λ) Changed the way the best human players played.
25 Arcade Learning Environment [Bellemare 2013]
26 Deep Q-Networks [Mnih et al., 2015]
27 Atari [Mnih et al., 2015] video: Two Minute Papers
28 Atari [Mnih et al., 2015]
29 POLICY SEARCH
30 Policy Search Represent policy directly: (s, a, ) :R n, R m! [0, 1] Objective function: max parameter vector " # 1X E R = i r i i=0 Why?
31 Policy Search So far: improve policy via value function. Sometimes policies are simpler than value functions: Parametrized program π(s, a θ) Sometimes we wish to search in space of restricted policies. In such cases it makes sense to search directly in policy-space rather than trying to learn a value function.
32 Hill Climbing What if you can t differentiate? Sample-based optimization: Sample some values near your current best. Adjust your current best to the highest value.
33 Aibo Gait Optimization from Kohl and Stone, ICRA 2004.
34 PoWER and PI2 More recently, two closely related algorithms: Generate some sample values. Next is sum of prior samples weighted by reward. (Theodorou and Schaal 2010, Kober and Peters 2011)
35 Policy Search What if we can differentiate with respect to θ? Policy gradient methods. Compute and ascend R/ θ This is the gradient of return w.r.t policy parameters Policy gradient theorem: Therefore, one way is to learn Q and then ascend gradient. Q need only be defined using basis functions computed from θ.
36 Postural Recovery
37 Deep Policy Search [Levine et al., 2016]
38 Deep Policy Search [Levine et al., 2016]
39 Robotics [Levine et al., 2016]
40 Reinforcement Learning Very active area of current research, applications in: Robotics Operations Research Computer Games Theoretical Neuroscience AI The primary function of the brain is control.
Reinforcement Learning II. George Konidaris
Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2017 Reinforcement Learning π : S A max R = t=0 t r t MDPs Agent interacts with an environment At each time t: Receives sensor signal Executes
Reinforcement Learning. George Konidaris
Reinforcement Learning George Konidaris gdk@cs.brown.edu Fall 2017 Machine Learning Subfield of AI concerned with learning from data. Broadly, using: Experience To Improve Performance On Some Task (Tom
Lecture 23: Reinforcement Learning
Lecture 23: Reinforcement Learning MDPs revisited Model-based learning Monte Carlo value function estimation Temporal-difference (TD) learning Exploration November 23, 2006 1 COMP-424 Lecture 23 Recall:
Machine Learning. Machine Learning: Jordan Boyd-Graber University of Maryland REINFORCEMENT LEARNING. Slides adapted from Tom Mitchell and Peter Abeel
Machine Learning Machine Learning: Jordan Boyd-Graber University of Maryland REINFORCEMENT LEARNING Slides adapted from Tom Mitchell and Peter Abeel Machine Learning: Jordan Boyd-Graber UMD Machine Learning
15-780: ReinforcementLearning
15-780: ReinforcementLearning J. Zico Kolter March 2, 2016 1 Outline Challenge of RL Model-based methods Model-free methods Exploration and exploitation 2 Outline Challenge of RL Model-based methods Model-free
CS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
Reinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo Marc Toussaint University of
Lecture 1: March 7, 2018
Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights
CS 287: Advanced Robotics Fall Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study
CS 287: Advanced Robotics Fall 2009 Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study Pieter Abbeel UC Berkeley EECS Assignment #1 Roll-out: nice example paper: X.
Reinforcement Learning. Machine Learning, Fall 2010
Reinforcement Learning Machine Learning, Fall 2010 1 Administrativia This week: finish RL, most likely start graphical models LA2: due on Thursday LA3: comes out on Thursday TA Office hours: Today 1:30-2:30
CS 570: Machine Learning Seminar. Fall 2016
CS 570: Machine Learning Seminar Fall 2016 Class Information Class web page: http://web.cecs.pdx.edu/~mm/mlseminar2016-2017/fall2016/ Class mailing list: cs570@cs.pdx.edu My office hours: T,Th, 2-3pm or
Reinforcement Learning
Reinforcement Learning Function approximation Daniel Hennes 19.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns Forward and backward view Function
15-889e Policy Search: Gradient Methods Emma Brunskill. All slides from David Silver (with EB adding minor modificafons), unless otherwise noted
15-889e Policy Search: Gradient Methods Emma Brunskill All slides from David Silver (with EB adding minor modificafons), unless otherwise noted Outline 1 Introduction 2 Finite Difference Policy Gradient
Reinforcement Learning
Reinforcement Learning Ron Parr CompSci 7 Department of Computer Science Duke University With thanks to Kris Hauser for some content RL Highlights Everybody likes to learn from experience Use ML techniques
Multiagent (Deep) Reinforcement Learning
Multiagent (Deep) Reinforcement Learning MARTIN PILÁT (MARTIN.PILAT@MFF.CUNI.CZ) Reinforcement learning The agent needs to learn to perform tasks in environment No prior knowledge about the effects of
Q-Learning in Continuous State Action Spaces
Q-Learning in Continuous State Action Spaces Alex Irpan alexirpan@berkeley.edu December 5, 2015 Contents 1 Introduction 1 2 Background 1 3 Q-Learning 2 4 Q-Learning In Continuous Spaces 4 5 Experimental
Reinforcement Learning. Yishay Mansour Tel-Aviv University
Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak
ilstd: Eligibility Traces and Convergence Analysis
ilstd: Eligibility Traces and Convergence Analysis Alborz Geramifard Michael Bowling Martin Zinkevich Richard S. Sutton Department of Computing Science University of Alberta Edmonton, Alberta {alborz,bowling,maz,sutton}@cs.ualberta.ca
REINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning Ronen Tamari The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (#67679) February 28, 2016 Ronen Tamari
Decision Theory: Q-Learning
Decision Theory: Q-Learning CPSC 322 Decision Theory 5 Textbook 12.5 Decision Theory: Q-Learning CPSC 322 Decision Theory 5, Slide 1 Lecture Overview 1 Recap 2 Asynchronous Value Iteration 3 Q-Learning
CSC321 Lecture 22: Q-Learning
CSC321 Lecture 22: Q-Learning Roger Grosse Roger Grosse CSC321 Lecture 22: Q-Learning 1 / 21 Overview Second of 3 lectures on reinforcement learning Last time: policy gradient (e.g. REINFORCE) Optimize
CS230: Lecture 9 Deep Reinforcement Learning
CS230: Lecture 9 Deep Reinforcement Learning Kian Katanforoosh Menti code: 21 90 15 Today s outline I. Motivation II. Recycling is good: an introduction to RL III. Deep Q-Learning IV. Application of Deep
An online kernel-based clustering approach for value function approximation
An online kernel-based clustering approach for value function approximation N. Tziortziotis and K. Blekas Department of Computer Science, University of Ioannina P.O.Box 1186, Ioannina 45110 - Greece {ntziorzi,kblekas}@cs.uoi.gr
Reinforcement Learning
Reinforcement Learning Function approximation Mario Martin CS-UPC May 18, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 18, 2018 / 65 Recap Algorithms: MonteCarlo methods for Policy Evaluation
Deep Reinforcement Learning. STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017
Deep Reinforcement Learning STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017 Outline Introduction to Reinforcement Learning AlphaGo (Deep RL for Computer Go)
CS 598 Statistical Reinforcement Learning. Nan Jiang
CS 598 Statistical Reinforcement Learning Nan Jiang Overview What s this course about? A grad-level seminar course on theory of RL 3 What s this course about? A grad-level seminar course on theory of RL
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Michail G. Lagoudakis Department of Computer Science Duke University Durham, NC 2778 mgl@cs.duke.edu
Lecture 8: Policy Gradient
Lecture 8: Policy Gradient Hado van Hasselt Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient Introduction Vapnik s rule Never solve
The Reinforcement Learning Problem
The Reinforcement Learning Problem Slides based on the book Reinforcement Learning by Sutton and Barto Formalizing Reinforcement Learning Formally, the agent and environment interact at each of a sequence
Introduction to Reinforcement Learning
Introduction to Reinforcement Learning Rémi Munos SequeL project: Sequential Learning http://researchers.lille.inria.fr/ munos/ INRIA Lille - Nord Europe Machine Learning Summer School, September 2011,
Reinforcement Learning
Reinforcement Learning Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ Task Grasp the green cup. Output: Sequence of controller actions Setup from Lenz et. al.
Lecture 3: The Reinforcement Learning Problem
Lecture 3: The Reinforcement Learning Problem Objectives of this lecture: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning II Daniel Hennes 11.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns
Reinforcement learning
Reinforcement learning Stuart Russell, UC Berkeley Stuart Russell, UC Berkeley 1 Outline Sequential decision making Dynamic programming algorithms Reinforcement learning algorithms temporal difference
Reinforcement Learning II
Reinforcement Learning II Andrea Bonarini Artificial Intelligence and Robotics Lab Department of Electronics and Information Politecnico di Milano E-mail: bonarini@elet.polimi.it URL:http://www.dei.polimi.it/people/bonarini
Reinforcement Learning in a Nutshell
Reinforcement Learning in a Nutshell V. Heidrich-Meisner 1, M. Lauer 2, C. Igel 1 and M. Riedmiller 2 1- Institut für Neuroinformatik, Ruhr-Universität Bochum, Germany 2- Neuroinformatics Group, University
Reinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
INF 5860 Machine learning for image classification. Lecture 14: Reinforcement learning May 9, 2018
Machine learning for image classification Lecture 14: Reinforcement learning May 9, 2018 Page 3 Outline Motivation Introduction to reinforcement learning (RL) Value function based methods (Q-learning)
Temporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI
Temporal Difference Learning KENNETH TRAN Principal Research Engineer, MSR AI Temporal Difference Learning Policy Evaluation Intro to model-free learning Monte Carlo Learning Temporal Difference Learning
Introduction to Reinforcement Learning
CSCI-699: Advanced Topics in Deep Learning 01/16/2019 Nitin Kamra Spring 2019 Introduction to Reinforcement Learning 1 What is Reinforcement Learning? So far we have seen unsupervised and supervised learning.
MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
Lecture 7: Value Function Approximation
Lecture 7: Value Function Approximation Joseph Modayil Outline 1 Introduction 2 3 Batch Methods Introduction Large-Scale Reinforcement Learning Reinforcement learning can be used to solve large problems,
Partially observable Markov decision processes. Department of Computer Science, Czech Technical University in Prague
Partially observable Markov decision processes Jiří Kléma Department of Computer Science, Czech Technical University in Prague https://cw.fel.cvut.cz/wiki/courses/b4b36zui/prednasky pagenda Previous lecture:
Value Function Approximation in Reinforcement Learning using the Fourier Basis
Value Function Approximation in Reinforcement Learning using the Fourier Basis George Konidaris Sarah Osentoski Technical Report UM-CS-28-19 Autonomous Learning Laboratory Computer Science Department University
Probabilistic Planning. George Konidaris
Probabilistic Planning George Konidaris gdk@cs.brown.edu Fall 2017 The Planning Problem Finding a sequence of actions to achieve some goal. Plans It s great when a plan just works but the world doesn t
Reinforcement Learning. Introduction
Reinforcement Learning Introduction Reinforcement Learning Agent interacts and learns from a stochastic environment Science of sequential decision making Many faces of reinforcement learning Optimal control
Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
An Introduction to Reinforcement Learning
An Introduction to Reinforcement Learning Shivaram Kalyanakrishnan shivaram@cse.iitb.ac.in Department of Computer Science and Engineering Indian Institute of Technology Bombay April 2018 What is Reinforcement
Reinforcement Learning: An Introduction
Introduction Betreuer: Freek Stulp Hauptseminar Intelligente Autonome Systeme (WiSe 04/05) Forschungs- und Lehreinheit Informatik IX Technische Universität München November 24, 2004 Introduction What is
Least squares temporal difference learning
Least squares temporal difference learning TD(λ) Good properties of TD Easy to implement, traces achieve the forward view Linear complexity = fast Combines easily with linear function approximation Outperforms
Decision Theory: Markov Decision Processes
Decision Theory: Markov Decision Processes CPSC 322 Lecture 33 March 31, 2006 Textbook 12.5 Decision Theory: Markov Decision Processes CPSC 322 Lecture 33, Slide 1 Lecture Overview Recap Rewards and Policies
(Deep) Reinforcement Learning
Martin Matyášek Artificial Intelligence Center Czech Technical University in Prague October 27, 2016 Martin Matyášek VPD, 2016 1 / 17 Reinforcement Learning in a picture R. S. Sutton and A. G. Barto 2015
Laplacian Agent Learning: Representation Policy Iteration
Laplacian Agent Learning: Representation Policy Iteration Sridhar Mahadevan Example of a Markov Decision Process a1: $0 Heaven $1 Earth What should the agent do? a2: $100 Hell $-1 V a1 ( Earth ) = f(0,1,1,1,1,...)
Variance Reduction for Policy Gradient Methods. March 13, 2017
Variance Reduction for Policy Gradient Methods March 13, 2017 Reward Shaping Reward Shaping Reward Shaping Reward shaping: r(s, a, s ) = r(s, a, s ) + γφ(s ) Φ(s) for arbitrary potential Φ Theorem: r admits
Introduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
6 Reinforcement Learning
6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and
ROB 537: Learning-Based Control. Announcements: Project background due Today. HW 3 Due on 10/30 Midterm Exam on 11/6.
ROB 537: Learning-Based Control Week 5, Lecture 1 Policy Gradient, Eligibility Traces, Transfer Learning (MaC Taylor Announcements: Project background due Today HW 3 Due on 10/30 Midterm Exam on 11/6 Reading:
Lecture 9: Policy Gradient II 1
Lecture 9: Policy Gradient II 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 Additional reading: Sutton and Barto 2018 Chp. 13 1 With many slides from or derived from David Silver and John
ARTIFICIAL INTELLIGENCE. Reinforcement learning
INFOB2KI 2018-2019 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Reinforcement learning Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
Lecture 8: Policy Gradient I 2
Lecture 8: Policy Gradient I 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 Additional reading: Sutton and Barto 2018 Chp. 13 2 With many slides from or derived from David Silver and John Schulman
Sequential decision making under uncertainty. Department of Computer Science, Czech Technical University in Prague
Sequential decision making under uncertainty Jiří Kléma Department of Computer Science, Czech Technical University in Prague https://cw.fel.cvut.cz/wiki/courses/b4b36zui/prednasky pagenda Previous lecture:
An Introduction to Reinforcement Learning
An Introduction to Reinforcement Learning Shivaram Kalyanakrishnan shivaram@csa.iisc.ernet.in Department of Computer Science and Automation Indian Institute of Science August 2014 What is Reinforcement
Reinforcement Learning
Reinforcement Learning Function Approximation Continuous state/action space, mean-square error, gradient temporal difference learning, least-square temporal difference, least squares policy iteration Vien
Linear Least-squares Dyna-style Planning
Linear Least-squares Dyna-style Planning Hengshuai Yao Department of Computing Science University of Alberta Edmonton, AB, Canada T6G2E8 hengshua@cs.ualberta.ca Abstract World model is very important for
Chapter 3: The Reinforcement Learning Problem
Chapter 3: The Reinforcement Learning Problem Objectives of this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
Machine Learning I Reinforcement Learning
Machine Learning I Reinforcement Learning Thomas Rückstieß Technische Universität München December 17/18, 2009 Literature Book: Reinforcement Learning: An Introduction Sutton & Barto (free online version:
Lecture 18: Reinforcement Learning Sanjeev Arora Elad Hazan
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 18: Reinforcement Learning Sanjeev Arora Elad Hazan Some slides borrowed from Peter Bodik and David Silver Course progress Learning
Reinforcement Learning
CS7/CS7 Fall 005 Supervised Learning: Training examples: (x,y) Direct feedback y for each input x Sequence of decisions with eventual feedback No teacher that critiques individual actions Learn to act
Trust Region Policy Optimization
Trust Region Policy Optimization Yixin Lin Duke University yixin.lin@duke.edu March 28, 2017 Yixin Lin (Duke) TRPO March 28, 2017 1 / 21 Overview 1 Preliminaries Markov Decision Processes Policy iteration
A Convergent O(n) Algorithm for Off-policy Temporal-difference Learning with Linear Function Approximation
A Convergent O(n) Algorithm for Off-policy Temporal-difference Learning with Linear Function Approximation Richard S. Sutton, Csaba Szepesvári, Hamid Reza Maei Reinforcement Learning and Artificial Intelligence
Regularization and Feature Selection in. the Least-Squares Temporal Difference
Regularization and Feature Selection in Least-Squares Temporal Difference Learning J. Zico Kolter kolter@cs.stanford.edu Andrew Y. Ng ang@cs.stanford.edu Computer Science Department, Stanford University,
Lecture 9: Policy Gradient II (Post lecture) 2
Lecture 9: Policy Gradient II (Post lecture) 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 Additional reading: Sutton and Barto 2018 Chp. 13 2 With many slides from or derived from David Silver
Temporal difference learning
Temporal difference learning AI & Agents for IET Lecturer: S Luz http://www.scss.tcd.ie/~luzs/t/cs7032/ February 4, 2014 Recall background & assumptions Environment is a finite MDP (i.e. A and S are finite).
Deep Reinforcement Learning SISL. Jeremy Morton (jmorton2) November 7, Stanford Intelligent Systems Laboratory
Deep Reinforcement Learning Jeremy Morton (jmorton2) November 7, 2016 SISL Stanford Intelligent Systems Laboratory Overview 2 1 Motivation 2 Neural Networks 3 Deep Reinforcement Learning 4 Deep Learning
CS599 Lecture 2 Function Approximation in RL
CS599 Lecture 2 Function Approximation in RL Look at how experience with a limited part of the state set be used to produce good behavior over a much larger part. Overview of function approximation (FA)
Deep Reinforcement Learning: Policy Gradients and Q-Learning
Deep Reinforcement Learning: Policy Gradients and Q-Learning John Schulman Bay Area Deep Learning School September 24, 2016 Introduction and Overview Aim of This Talk What is deep RL, and should I use
Reinforcement Learning
Reinforcement Learning Cyber Rodent Project Some slides from: David Silver, Radford Neal CSC411: Machine Learning and Data Mining, Winter 2017 Michael Guerzhoy 1 Reinforcement Learning Supervised learning:
Reinforcement Learning with Function Approximation. Joseph Christian G. Noel
Reinforcement Learning with Function Approximation Joseph Christian G. Noel November 2011 Abstract Reinforcement learning (RL) is a key problem in the field of Artificial Intelligence. The main goal is
Today s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes
Today s s Lecture Lecture 20: Learning -4 Review of Neural Networks Markov-Decision Processes Victor Lesser CMPSCI 683 Fall 2004 Reinforcement learning 2 Back-propagation Applicability of Neural Networks
CMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 12: Reinforcement Learning Teacher: Gianni A. Di Caro REINFORCEMENT LEARNING Transition Model? State Action Reward model? Agent Goal: Maximize expected sum of future rewards 2 MDP PLANNING
Regularization and Feature Selection in. the Least-Squares Temporal Difference
Regularization and Feature Selection in Least-Squares Temporal Difference Learning J. Zico Kolter kolter@cs.stanford.edu Andrew Y. Ng ang@cs.stanford.edu Computer Science Department, Stanford University,
POMDPs and Policy Gradients
POMDPs and Policy Gradients MLSS 2006, Canberra Douglas Aberdeen Canberra Node, RSISE Building Australian National University 15th February 2006 Outline 1 Introduction What is Reinforcement Learning? Types
Active Policy Iteration: Efficient Exploration through Active Learning for Value Function Approximation in Reinforcement Learning
Active Policy Iteration: fficient xploration through Active Learning for Value Function Approximation in Reinforcement Learning Takayuki Akiyama, Hirotaka Hachiya, and Masashi Sugiyama Department of Computer
Reinforcement Learning
Reinforcement Learning Policy gradients Daniel Hennes 26.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Policy based reinforcement learning So far we approximated the action value
Sequential decision making under uncertainty. Department of Computer Science, Czech Technical University in Prague
Sequential decision making under uncertainty Jiří Kléma Department of Computer Science, Czech Technical University in Prague /doku.php/courses/a4b33zui/start pagenda Previous lecture: individual rational
15-780: Graduate Artificial Intelligence. Reinforcement learning (RL)
15-780: Graduate Artificial Intelligence Reinforcement learning (RL) From MDPs to RL We still use the same Markov model with rewards and actions But there are a few differences: 1. We do not assume we
A Gentle Introduction to Reinforcement Learning
A Gentle Introduction to Reinforcement Learning Alexander Jung 2018 1 Introduction and Motivation Consider the cleaning robot Rumba which has to clean the office room B329. In order to keep things simple,
Lecture 25: Learning 4. Victor R. Lesser. CMPSCI 683 Fall 2010
Lecture 25: Learning 4 Victor R. Lesser CMPSCI 683 Fall 2010 Final Exam Information Final EXAM on Th 12/16 at 4:00pm in Lederle Grad Res Ctr Rm A301 2 Hours but obviously you can leave early! Open Book
Reinforcement Learning for NLP
Reinforcement Learning for NLP Advanced Machine Learning for NLP Jordan Boyd-Graber REINFORCEMENT OVERVIEW, POLICY GRADIENT Adapted from slides by David Silver, Pieter Abbeel, and John Schulman Advanced
Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms *
Proceedings of the 8 th International Conference on Applied Informatics Eger, Hungary, January 27 30, 2010. Vol. 1. pp. 87 94. Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms
Administration. CSCI567 Machine Learning (Fall 2018) Outline. Outline. HW5 is available, due on 11/18. Practice final will also be available soon.
Administration CSCI567 Machine Learning Fall 2018 Prof. Haipeng Luo U of Southern California Nov 7, 2018 HW5 is available, due on 11/18. Practice final will also be available soon. Remaining weeks: 11/14,
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlver s Lecture 4: Model-Free
Actor-critic methods. Dialogue Systems Group, Cambridge University Engineering Department. February 21, 2017
Actor-critic methods Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department February 21, 2017 1 / 21 In this lecture... The actor-critic architecture Least-Squares Policy Iteration
Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]
![Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]](/thumbs/94/119187228.jpg "Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]")
Review: TD-Learning function TD-Learning(mdp) returns a policy Class #: Reinforcement Learning, II 8s S, U(s) =0 set start-state s s 0 choose action a, using -greedy policy based on U(s) U(s) U(s)+ [r
Reinforcement Learning as Classification Leveraging Modern Classifiers
Reinforcement Learning as Classification Leveraging Modern Classifiers Michail G. Lagoudakis and Ronald Parr Department of Computer Science Duke University Durham, NC 27708 Machine Learning Reductions
1 Introduction 2. 4 Q-Learning The Q-value The Temporal Difference The whole Q-Learning process... 5
Table of contents 1 Introduction 2 2 Markov Decision Processes 2 3 Future Cumulative Reward 3 4 Q-Learning 4 4.1 The Q-value.............................................. 4 4.2 The Temporal Difference.......................................
arxiv: v1 [cs.lg] 30 Dec 2016
![arxiv: v1 [cs.lg] 30 Dec 2016](/thumbs/96/127560530.jpg "arxiv: v1 [cs.lg] 30 Dec 2016")
Adaptive Least-Squares Temporal Difference Learning Timothy A. Mann and Hugo Penedones and Todd Hester Google DeepMind London, United Kingdom {kingtim, hugopen, toddhester}@google.com Shie Mannor Electrical
, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount