Learning to Control an Unstable System with Forward Modeling

Transcription

1 324 Jrdan and Jacbs Learning t Cntrl an Unstable System with Frward Mdeling Michael I. Jrdan Brain and Cgnitive Sciences MIT Cambridge, MA Rbert A. Jacbs Cmputer and Infrmatin Sciences University f Massachusetts Amherst, MA ABSTRACT The frward mdeling apprach is a methdlgy fr learning cntrl when data is available in distal crdinate systems. We extend previus wrk by cnsidering hw this methdlgy can be applied t the ptimizatin f quantities that are distal nt nly in space but als in time. In many learning cntrl prblems, the utput variables f the cntrller are nt the natural crdinates in which t specify tasks and evaluate perfrmance. Tasks are generally mre naturally specified in "distal" crdinate systems (e.g., endpint crdinates fr manipulatr mtin) than in the "prximal" crdinate system f the cntrller (e.g., jint angles r trques). Furthermre, the relatinship between prximal crdinates and distal crdinates is ften nt knwn a priri and, if knwn, nt easily inverted. The frward mdeling apprach is a methdlgy fr learning cntrl when training data is available in distal crdinate systems. A frward mdel is a netwrk that learns the transfrmatin frm prximal t distal crdinates s that distal specificatins can be used in training the cntrller (Jrdan & Rumelhart, 1990). The frward mdel can ften be learned separately frm the cntrller because it depends nly n the dynamics f the cntrlled system and nt n the clsed-lp dynamics. In previus wrk, we studied frward mdels f kinematic transfrmatins (Jrdan, 1988, 1990) and state transitins (Jrdan & Rumelhart, 1990). In the current paper,

2 Learning t Cntrl an Unstable System with Frward Mdeling 325 we g beynd the spatial credit assignment prblems studied in thse papers and braden the applicatin f frward mdeling t include cases f tempral credit assignment (cf. Bart, Suttn, & Andersn, 1983; Werbs, 1987). As discussed belw, the functin t be mdeled in such cases depends n a time integral f the clsed-lp dynamics. This fact has tw imprtant implicatins. First, the data needed fr learning the frward mdel can n lnger be btained slely by bserving the instantaneus state r utput f the plant. Secnd, the frward mdel is n lnger independent f the cntrller: If the parameters f the cntrller are changed by a learning algrithm, then the clsed-lp dynamics change and s des the mapping frm prximal t distal variables. Thus the learning f the frward mdel and the learning f the cntrller can n lnger be separated int different phases. 1 FORWARD MODELING In this sectin we briefly summarize ur previus wrk n frward mdeling (see als Nguyen & Widrw, 1989 and Werbs, 1987). 1.1 LEARNING A FORWARD MODEL Given a fixed cntrl law, the learning f a frward mdel is a system identificatin prblem. Let z = g(s, u) be a system t be mdeled, where z is the utput r the state-derivative, s is the state, and u is the cntrl. We require the frward mdel t minimize the cst functinal J Jm = ~ (z - z)t(z - z)dt. (1) where z = 9(s, u, v) is the parameterized functin cmputed by the mdel. Once the minimum is fund, backprpagatin thrugh the mdel prvides an estimate u f the system Jacbian matrix :~ (cf. Jrdan, 1988). 1.2 LEARNING A CONTROLLER Once the frward mdel is sufficiently accurate, it can be used in the training f the cntrller. Backprpagatin thrugh the mdel prvides derivatives that indicate hw t change the utputs f the cntrller. These derivatives can be used t change the parameters f the cntrller by a further applicatin f back prpagatin. Figure 1 illustrates the general prcedure. This prcedure minimizes the "distal" cst functinal where z is a reference signal. T see this, let the cntrller utput be given as a functin u = f(s, z, w) f the state s, the reference signal z, and a parameter vectr w. Differentiating J with respect t w yields J ut zt "wj = - w u (z - z)dt. (3) (2)

3 326 Jrdan and Jacbs z* ~ \ Feedfrward Cntrller x Plant z Frward + - -Mdel Figure 1: Learning a Cntrller. The Dashed Line Represents Backprpagatin. The Jacbian matrix u cannt be assumed t be available a priri, but can be estimated by backprpagatin thrugh the frward mdel. Thus the errr signal available fr learning the cntrller is the estimated gradient.. J u z T 0' T V'wJ = (z - z)dt. w OU (4) We nw cnsider a task in which the freging framewrk must be bradened t allw a mre general frm f distal task specificatin. 2 THE TASK The task is t learn t regulate an unstable nnminimum-phase plant. We have chsen the ft-studied (e.g., Bart, Suttn, & Andersn, 1983; \Vidrw & Smith, 1964) prblem f learning t balance an inverted pendulum n a mving cart. The plant dynamics are given by: [ M+m mlcs(j mlcs(j ] [ ~ ] + [ -mlsi~(j ] ip = [ F ] I (J -mglszn(j 0 where m is the mass f the ple, M is the mass f the cart, I is half the ple length, I is the inertia f the ple arund its base, and F is the frce applied t the cart. The task we studied is similar t that studied by Bart, Suttn, & Andersn (1983). A state-feedback cntrller prvides frces t the cart, and the system evlves until failure ccurs (the cart reaches the end f the track r the ple reaches a critical angle). The system learns frm failure; indeed, it is assumed that the nly teaching infrmatin prvided by the envirnment is the signal that failure has ccurred.

4 Learning t Cntrl an Unstable System with Frward Mdeling 327 sgn (x) 0 lielo sgn( x) 0 lei 0 sgn(e) 0 lei 0 sgn(e) 0 e Ii Frward Mdel Actin Unit -0 ~,.p.. nl Tempral Difference Unit -0 Cntrller Figure 2: The Netwrk Architecture There are several differences between ur task and that studied by Bart, Suttn, &. Andersn (1983). First, disturbances (white nise) are prvided by the envirnment rather than by the learning algrithm. This implies that in ur experiments the level f nise seen by the cntrller des nt diminish t zer ver the curse f learning. Secnd, we used real-valued frces rather than binary frces. Finally, we d nt assume the existence f a "reset buttn" that reinitializes the system t the rigin f state space; upn failure the system is restarted in a randm cnfiguratin. 3 OUR APPROACH In ur apprach, the cntrl system learns a mdel that relates the current state f the plant and the current cntrl signal t a predictin f future failure. We make use f a tempral difference algrithm (Suttn, 1988) t learn the transfrmatin frm (state, actin) pairs t an estimate f the inverse f the time until failure. This mapping is then used as a differentiable frward mdel in the learning f the cntrller-the cntrller is changed s as t minimize the utput f the mdel and thereby maximize the time until failure. The verall system architecture is shwn in Figure 2. We describe each cmpnent in detail in the fllwing sectins. An imprtant feature that distinguishes this architecture frm previus wrk (e.g.,

5 328 Jrdan and Jacbs Bart, Suttn, & Andersn, 1983) is the path frm the actin unit int the frward mdel. This path is necessary fr supervised learning algrithms t be used (see als Werbs, 1987). 3.1 LEARNING THE FORWARD MODEL Tempral difference algrithms learn t make lng term predictins by achieving lcal cnsistency between predictins at neighbring time steps, and by grunding the chain f predictins when infrmatin frm the envirnment is btained. In ur case, if z(t) is the inverse f the time until failure, then cnsistency is defined by the requirement that z-l(t) = z-l(t + 1) + 1. The chain is grunded by defining z(t) = 1, where T is the time step n which failure ccurs. T learn t estimate the inverse f the time until failure, the fllwing tempral difference errr terms are used. Fr time steps n which failure des nt ccur, ( ) 1 A( ) e t = (t + 1) - z t, where (t) dentes the utput f the frward mdel. When failure ccurs, the target fr the frward mdel is set t unity: e(t) = 1 -- (t) The errr signal e(t) is prpagated backwards at time t + 1 using activatins saved frm time t. Standard backprpagatin is used t cmpute the changes t the weights. 3.2 LEARNING THE CONTROLLER If the cntrller is perfrming as desired, then the utput f the frward mdel is zer (that is, the predicted time-until-failure is infinity). This suggests that an apprpriate distal errr signal fr the cntrller is zer minus the utput f the frward mdel. Given that the frward mdel has the cntrl actin as an input, the distal errr can be prpagated backward t the hidden units f the frward mdel, thrugh the actin unit, and int the cntrller where the weights are changed (see Figure 2). Thus the cntrller is changed in such a way as t minimize the utput f the frward mdel and thereby maximize the time until failure. 3.3 LEARNING THE FORWARD MODEL AND THE CONTROLLER SIMULTANEOUSLY As the cntrller varies, the mapping that the frward mdel must learn als varies. Thus, if the frward mdel is t prvide reasnable derivatives, it must be cntinuusly updated as the cntrller changes. We find that it is pssible t train the frward mdel and the cntrller simultaneusly, prvided that we use a larger learning rate fr the frward mdel than fr the cntrller.

6 Learning t Cntrl an Unstable System with Frward Mdeling MISCELLANY 4.1 RESET Althugh previus studies have assumed the existence f a "reset buttn" that can restart the system at the rigin f state space, we prefer nt t make such an assumptin. A reset buttn implies the existence f a cntrller that can stabilize the system, and the task f learning is t find such a cntrller. In ur simulatins, we restart the system at randm pints in state space after failure ccurs. 4.2 REDUNDANCY The mapping learned by the frward mdel depends n bth the state and the actin. The actin, hwever, is itself a functin f the state, s the actin unit prvides redundant infrmatin. This implies that the frward mdel culd have arbitrary weights in the path frm the actin unit and yet make reasnable predictins. Such a mdel, hwever, wuld yield meaningless derivatives fr learning the cntrller. Frtunately, backprpagatin tends t prduce meaningful weights fr a path that is crrelated with the utcme, even if that path cnveys redundant infrmatin. T further bias things in ur favr, we fund it useful t emply a larger learning rate in the path frm the actin unit t the hidden units f the frward mdel (0.9) than in the path frm the state units (0.3). 4.3 REPRESENTATION As seen in Figure 2, we chse input representatins that take advantage f symmetries in the dynamics f the cart-ple system. The frward mdel has even symmetry with respect t the state variables, whereas the cntrller has dd symmetry. 4.4 LONG-TERM BEHAVIOR There is never a need t "turn ff" the learning f the frward mdel. Once the ple is being successfully balanced in the presence f fluctuatins, the average time until failure ges t infinity. The frward mdel therefre learns t predict zer in the regin f state space arund the rigin, and the errr prpagated t the cntrller als ges t zer. 5 RESULTS We ran twenty simulatins starting with randm initial weights. The learning rate fr the cntrller was 0.05 and the learning rate fr the frward mdel was 0.3, except fr the cnnectin frm the actin unit where the learning rate was 0.9. Eighteen runs cnverged t cntrller cnfiguratins that balanced the ple, and tw runs cnverged n lcal minima. Figure 3 shws representative learning curves fr six f the successful runs. T btain sme idea f the size f the space f crrect slutins, we perfrmed an exhaustive search f a lattice in a rectangular regin f weight space that cntained

7 330 Jrdan and Jacbs Average time until failure Bins (1 bin., 20 fillur ) Figure 3: Learning Curves fr Six Runs all f the weight cnfiguratins fund by ur simulatins. As shwn in Figure 4, nly 15 ut f 10,000 weight cnfiguratins were able t balance the ple. 6 CONCLUSIONS Previus wr k within the frward mdeling paradigm fcused n mdels f fixed kinematic r dynamic prperties f the cntrlled plant (Jrdan, 1988,1990; Jrdan &, Rumelhart, 1990). In the current paper, the ntin f a frward mdel is brader. The functin that must be mdeled depends nt nly n prperties f the cntrlled plant, but als n prperties f the cntrller. Nnetheless, the mapping is welldefined, and the results demnstrate that it can be used t prvide apprpriate incremental changes fr the cntrller. These results prvide further demnstratin f the applicability f supervised learning algrithms t learning cntrl prblems in which explicit target infrmatin is nt available. Acknwledgments The first authr was supprted by BRSG 2 S07 RR awarded by the Bimedical Research Supprt Grant Prgram, Divisin f Research Resurces, Natinal Institutes f Health and by a grant frm Siemens Crpratin. The secnd authr was supprted by the Air Frce Office f Scientific Research, thrugh grant AFOSR

8 Learning t Cntrl an Unstable System with Frward Mdeling 331 Lg Frequency 3 2 ) ~--.-r_----r_--~ Median Time Steps Until Failure Figure 4: Perfrmance f Ppulatin f Cntrllers References Bart, A. G., Suttn, R. S., & Andersn, C. W. (1983). Neurnlike adaptive elements that can slve difficult learning cntrl prblems. IEEE Transactins n Systems, Man, and Cybernetics, SMC.19, Jrdan, M. I. (1988). Supervised learning and systems with excess degress f freedm. (COINS Tech. Rep ). Amherst, MA: University f Massachusetts, Cmputer and Infrmatin Sciences. Jrdan, M. I. (1990). Mtr learning and the degrees f freedm prblem. In M. Jeannerd, (Ed). Attentin and Perfrmance, XIII. Hillsdale, NJ: Erlbaum. Jrdan, M. I. & Rumelhart, D. E. (1990). Supervised learning with a distal teacher. Paper in preparatin. Nguyen, D. & Widrw, B. (1989). The truck backer-upper: An example f selflearning in neural netwrks. In: Prceedings f the Internatinal Jint Cnference n Neural Netwrks. Piscataway, NJ: IEEE Press. Suttn, R. S. (1987). Learning t predict by the methds f tempral differences. Machine Learning, 9, Werbs, P. (1987). Building and understanding adaptive systems: A statistical/numerical apprach t factry autmatin and brain research. IEEE Transactins n Systems, Man, and Cybernetics, 17, Widrw, B. & Smith, F. W. (1964). Pattern-recgnizing cntrl systems. In: Cmputer and Infrmatin Sciences Prceedings, Washingtn, D.C.: Spartan.