Building A Fuzzy Inference System By An Extended Rule Based Q-Learning

Transcription

1 Buldng A Fuzzy Inference System By An Extended Rule Based Q-Learnng Mn-Soeng Km, Sun-G Hong and Ju-Jang Lee * Dept. of Electrcal Engneerng and Computer Scence, KAIST 373- Kusung-Dong Yusong-Ku Taejon 35-7, Korea Tel : , {mbella,sghong}@odyssey.ast.ac.r, jjlee@ee.ast.ac.r Buldng a Fuzzy Inference System (FIS) generally requres experts nowledge. However, experts nowledge s not always avalable. When there s few experts nowledge, t becomes hard to buld a FIS usng one of supervsed learnng methods. Meanwhle, Q-learnng s a nd of renforcement learnng where an agent can acqure nowledge from ts experences even wthout the model of the envronment and experts nowledge. The Q-learnng, however, has weaness that the orgnal algorthm cannot deal wth the contnuous states and contnuous actons. In ths paper, we proposed a FIS that can do Q-learnng. The proposed FIS structure s made up of several extended rules. Based on these extended rules, Q-learnng algorthm for the proposed structure s developed. It s shown that ths combnaton results n a FIS that can learn through ts experence wthout experts nowledge. Also the proposed structure can resolve the contnuous state/acton problem n Q-learnng by vrtue of a FIS. The effectveness of the proposed structure s shown through smulaton on the cart-pole system Introducton A FIS conssts of several fuzzy f-then rules. An extracton of these fuzzy rules requres a pror experts nowledge. However, t s not always easy to obtan experts nowledge. Generally, determnng consequent parts of fuzzy rules are consdered to be more dffcult than determnng antecedent parts. There have been varous approaches to obtan fuzzy rules n a supervsed way.[][] Most of those methods needs a large set of expermental data, whch becomes expensve as necessary data ncrease or the complexty of a problem ncreases. Meanwhle, Q-learnng[3] s a nd of renforcement learnng where an agent can acqure nowledge from ts experences even wthout the model of the envronment and experts nowledge. Q-value represents a qualty value for a certan acton n a certan state. An agent can learn to select ts proper acton n each state based by usng and updatng these Q-values. However, most research done n the feld of Q-learnng has focused on dscrete domans, although the envronment wth whch the agent must nteract s usually contnuous. There has been much research to mae Q-learnng deal wth the contnuous state/acton spaces.[4-6] In these methods, however, the acton selecton method s based on a step search n acton space. That s, after the Q-value for all actons n the possble acton range s calculated, the acton of whch the Q-value s the maxmum s selected and ths acton s performed. The contnuous property of acton s dependent on the step sze. Moreover, the sequence of generated actons does not change contnuously, or smoothly. The contnuty of acton s mportant from the practcal vew because the control nput cannot change rapdly by an actuator. Meanwhle, a FIS usually has a fuzzfer that translates real-valued nput nto fuzzy values and a defuzzfer that translates fuzzy output values nto real-valued output. Thus, contnuous states or contnuous actons can be descrbed by a fnte set of fuzzy varables. Thus, the FIS seems to be an effectve model to overcome the contnuous state/acton problem n Q-learnng. In ths paper, we proposed a FIS that can do Q-learnng. The proposed FIS structure s made up of several extended rules. Based on these extended rules, Q-learnng algorthm for the proposed structure s developed. It s shown that ths combnaton results n a FIS that can learn through ts experence wthout experts nowledge. Also the proposed structure can resolve the contnuous state/acton problem n Q-learnng by vrtue of a FIS. The remander of ths paper s organzed as follows. In Secton, after the basc fuzzy rule s brefly revewed, the concept of an extended rule s developed. The representaton scheme of Q-value based on an extended rule s dscussed. The proposed structure and the learnng algorthm for ths structure are shown n Secton 3. In Secton 4, the smulaton results are presented to show the effectveness of the proposed methods. Secton 5 presents concluson about the characterstcs of the proposed methods.

2 An Extended Fuzzy Rule. A basc fuzzy rule In fuzzy rule, premses and conclusons are expressed by means of lngustc terms. A FIS rule base s made of N dfferent fuzzy rules of the followng general form: R : f s s A and s s A then y s Bl S and where s s an nput state varable. R represents - th rule of a rule base. A and B are fuzzy membershps characterzed by lngustc labels(e.g., small, large and bg etc) and by a functon s µ ( s) [,] and s µ ( s) [,] respectvely.. An extended fuzzy rule To ncorporate Q-learnng n FIS, an extended rule s proposed n ths paper. The extended rule s dfferent from the basc fuzzy rule and has several characterstcs as follows: There are several canddate consequent parts (.e., canddate actons), for one antecedent part of each rule. Each canddate acton s a fuzzy membershp functon and has ts own Q value. Rule acton s selected among canddate actons based on each canddate s Q-value. Rule acton becomes an actual consequent part of the rule. Fnal acton s obtaned by defuzzfcaton. Ths extended rule can be descrbed as follows: s j j ) R : f s s L and s L and s s L () then a s π (u, Q where s S s an nput state varable and L m s a lngustc label related to each nput state. a s a rule acton whch corresponds to the actual consequent part of -th fuzzy rule, R. π (, ) s a polcy that determnes the rule acton from a canddate acton set A. And A s a collecton of acton u j, whch s a canddate acton for rule acton. Every ndvdual u j has ts own Q value, Q j. Superscrpt and subscrpt represent the rule number and = n. In each extended fuzzy rule, one canddate acton from A s selected based on ts Q-value and the polcy. To apply Q-learnng, Q-value for each canddate acton must be updated. A Rule s canddate acton s currently selected acton, among canddate consequent parts, and become an actual consequent part. These selected rule actons generate fnal acton by defuzzfcaton. The fnal acton, s n s A M M () however, s generally dfferent from each rule s rule acton. Because Q value s the functonal value of the state and the acton, dfferent value of acton cannot be used n obtanng Q-value for the current state and the fnal acton. That s, we must have Q-value for the fnal acton n each rule to obtan Q(current state, fnal acton) by defuzzfcaton. Thus some nd of technque to obtan the Q-value correspondng to fnal acton n each rule have to be devsed, the nterpolaton technque s used to obtan a Q value for the fnal acton n each rule n ths paper..3 Q-value representaton n an extended rule Now let us consder how to represent an approprate Q- value n an extended rule. The objectve s to obtan a Q- value of whch acton parameter s equal a f n each rule. A Q ( s, a) value for a certan acton a whch s not ncluded n the dscrete acton set A can be calculated usng the nterpolaton technque. Once the fnal acton a f s determned from the fuzzy rule base, the Q value for the fnal acton n -th rule R can be calculated as follows. p Q( R, a f ) = K( u j, a f ) Q j (3) j= p h= K( uh, a f ) where K (, ) s a ernel functon whch determnes the degree of how much each ( u j, Q j ) par n -th rule wll contrbute to calculatng Q ( R, a f ) and has the followng form.: (a f u j ) K(u j,a f ) = exp( ) (4) σ where σ should be determned accordng to the dstrbuton of canddate actons. 3 Self-Organzng Fuzzy Inference System 3. The proposed structure In ths secton, the self-organzng fuzzy nference system that can perform Q-learnng s proposed. The term, self-organzng, means that the consequent part of the fuzzy rule s automatcally selected from the dscrete acton set based on the polcy and on the Q value for each canddate acton. The whole system s shown n Fg.. Ths structure performs the fuzzy nference based on the fuzzy rule base that conssts of several extended rules n (). Inputs for the networ are the states of the envronment or plant. The networ generate a fnal acton and also approxmate Q(s,a)-functon. The networ conssts of 5 layers n total. Layer to layer 3 acheves fuzzfcaton n the nput state, whch resolves the contnuous state representaton problem n Q-learnng. Layer 4 and layer 5

3 generate contnuous acton by fuzzy nference and generate the assocated Q value for that nferred acton by fuzzy nference mxed wth the nterpolaton technque. Layer The functon of the frst layer s to receve the value of the state and transmt ths value to the next layer. Each node n ths layer corresponds to one nput state varable s S. Layer Ths layer conssts of several lngustc labels for each nput varable. Lngustc label L m n () has ts own membershp functon denoted by µ L m ( ). Trangular and trapezodal membershp functons are used here. For these membershp functons, µ L m ( ) s defned by four parameters : C L, CR, SL, S R where C L, CR, S L and S R correspond to the left center, rght center, left spread and rght spread respectvely. For trangular membershp functon C L s equal to C R, that s, C L = CR. The output value from layer ndcates a membershp degree of the current state and can be calculated usng the followng equaton. s CR s [ CR, CR + S R S R s CL µ L m ( s) = s [ CL S L, CL ] (5) S L s [ CL, CR otherwse Layer 3 One ln from a node n layer to a node n layer 3 corresponds to one antecedent part of FIS. That s, t represents the f part of each rule R n (). The value of each node n layer 3 smply represents how much current state s belongs to rule R and can be obtaned by fuzzy and operator. Ths value s the frng strength of rule for state s and denoted as α R. Then α R can be calculated as the followng. M α ( s) = µ ( s) (6) R m= where the fuzzy and operator s mplemented by product operaton. Layer 4 Layer 4 corresponds to the then part of each fuzzy rule. Based on Q values and polcy, one canddate acton from the dscrete acton set s selected as a rule acton a as n (7) and those selected rule actons from each rule are combned to generate the fnal acton.(.e., defuzzfcaton process) a = π ( u, Q ) (7) j Lm j Layer 5 Layer 4 Layer 3 Layer Layer u u u j R R s Q( s, a ) f where j s the number of canddate actons and π (, ) represents the polcy used to select the acton. In ths paper, the ε -greedy polcy[7] s used. ε -greedy polcy selects an acton of whch Q value s the maxmum wth the probablty of ( ε ). Besdes the selecton of the rule acton, the Q value for the fnal acton s calculated by (3) after the fnal acton s calculated n layer 5. Layer 5 In layer 5, the fnal acton s obtaned by usng a weghted sum of each rule acton wth ther frng strength as follows. n a f = α n R a (8) = h= α Rh Once the value of Q ( R, a f ) s calculated for each rule usng (3), the Q ( s, a f ) value can be obtaned by the followng equaton. n Q( s, a f ) = α R Q( R, a f ) (9) n = h= α Rh Although the rule acton a s determned by the ε - greedy polcy, the fnal acton obtaned by combnng (defuzzfyng) each rule acton cannot be sad a ε -greedy acton because the fnal acton s generally not equal to the acton that maxmze the Q-value for the current state. Thus the polcy conducted n the proposed FIS can be vewed as SARSA( λ )[8]. 3. Learnng algorthm u 3 u u j u 3 3 u uj Update s performed through the change n the Q value of each canddate acton for each rule. To speed up the learnng, elgblty s adopted and calculated as R 3 fnal acton a f s n n n u u uj Rn Fg.. The structure of the proposed FIS.

4 Q j Q( s, a) /. The elgblty s as the followng equaton. α R K( u j, a f ) f R and u j e = n j j = α R h= K uh a (, f ) has used () λγej otherwse where λ s an elgblty rate whch s used to weght each rule and rule acton par accordng to ther proxmty to the occurrng tme step from the current state. The learnng procedure s shown below: Intalzaton of Q values of all canddate actons to zero n each rule Perceve the current state s curr and generate fnal acton a f by (8). curr 3. Calculate Q ( s curr, a fcurr ) by (3) and calculate elgblty traces by (). 4. Perform acton a f curr transton to the next state, receve reward r curr and s next. 5. Generate fnal acton for the next state, a f next and calculate Q( s next, a fnext ) by (3) and calculate temporal error δ. δ = rcurr + γq( snext, a f ) Q( s, ) next curr a f curr () 6. For each canddate acton of every rule, update ther Q values as follows: Q j = Q j + αδej () 7. If tral s not ended, then let a f = a curr f and go to next step. 4 Smulaton Study 4. Cart-pole balancng problem The cart-pole balancng problem s concerned wth how to balance an uprght pole. The pole has only one degree of freedom and the prmary control tas s to eep the pole vertcally balanced whle eepng the cart wthn the ral trac boundares. Four state varables are used n descrbng ths system and one varable represents the force appled to the cart. These are: θ : the angle of the pole from uprght poston(radan); θ : the angular velocty of the pole(radan/seconds); x : the poston of the cart s center (meters); x : the velocty of the cart(meter/seconds); f : force appled to the cart (Newton) The dynamcs model used for the smulaton s shown below: trals Fg.. Learnng curve wth 5 canddate actons f m plθ snθ + µ c sgn( x ) µ pθ g snθ + cosθ mc + m p m pl θ = 4 m p cosθ l 3 mc + m p [ θ snθ θ cosθ ] µ sgn( x ) f + m pl c x = mc + m p Each parameter value can be found n [9]. Smulaton s done by the Euler method usng a tme step of.s. Constrants on the varables are θ,.4m x. 4m and N f N. It s assumed that the dynamcs of the system s unnown to the FIS. The controller (FIS) can be nformed of only the values of the state varables and the reward sgnal at each tme step. The only avalable feedbac than the proposed FIS can be acqured s falure sgnal when θ > or x >. 4m. The reward sgnal gven to the controller s as follows: r = θ > or x >.4m (3) otherwse Parameter values used for Q-learnng are as follows: γ s.9, α.3 and λ s Smulaton results Fgure show learnng curves when fve canddate actons are used wth zero ntal condton. The curve conssts of ten consecutve runs where a run conssts of several trals. Each tral ends n two cases. Frst, f the pole has fallen (.e., θ > ) or the cart poston exceeds the lmts of the ral trac (.e., x >. 4m ). Second, f the tme step exceeds. tme step corresponds to

5 seconds n smulaton tme. When 5 dscrete actons are used as canddate actons n consequent part of the extended fuzzy rule, the proposed FIS can learn to balance the pole after about 3-4 falures. In other words, the proposed FIS can solve the gven tas by usng ts experence wthout any experts nowledge or any supervsor. Fgure 3 to Fg. 5 show resultng trajectores of θ, x and f for a learned FIS. After learnng s occurred, most preferable consequent canddate acton has the hghest Q-value than any other canddate actons n the correspondng rule. Thus we can now what value of consequent part s good for a certan state or a certan antecedent part. In other words, we can obtan a local soluton, whch can be used to understand the characterstcs of the system or the envronment. Moreover, because ths local soluton s n the form of a fuzzy lngustc label, the nformaton s physcally explct and easly understandable. 4.3 Exploraton vs explotaton n Q-learnng Fgure 6 to Fg. 8 show the resultng trajectores of θ, x and f wth 5 canddate actons wth the same condtons. Although the same number of canddate actons and the same parameters are used, the obtaned performance s somewhat dfferent. These results can be explaned by famous exploraton vs explotaton problem n renforcement learnng. Snce the agent cannot experence every state durng fnte tmes, t faces the problem: whether to explore unnown state/acton to acheve better performance or to explot nown state/acton based on already obtaned experence. As shown n Fg. 6, the results seem worse than those n Fg. 3. But both of the resultng FIS s enough to fulfll the condton. It s a very dffcult problem n renforcement learnng whether the agent performs exploraton to obtan a better soluton that mght exst or performs explotaton. The exploraton can result n better performance, but the more tme s necessary. Tradeoff between the exploraton and the explotaton scheme s mportant factor n renforcement learnng because the agent cannot lve forever and cannot vst every possble state/acton par. 5 Conclusons In ths paper, the self-organzng fuzzy nference system that can perform Q-learnng was proposed. By the extended rule and the nterpolaton technque, the proposed FIS behaves as a fuzzy nference system whle approxmatng the Q(s,a) functon. By ncorporatng the fuzzy nference system wth Q-learnng based on an extended fuzzy rule, wthout the model and nowledge about the envronment, the proposed FIS can organze ts fuzzy rule base automatcally. As n ths way, we can buld a FIS(or an agent) that can learn to solve the gven θ (deg) X (meter) Control (N) Fg. 3. Trajectory of the pole angle Fg. 4. Trajectory of the cart poston Fg. 5. Trajectory of the control force. problem by tself through the nteracton wth the gven problem, that s, through ts experence le an anmal. Also, by fuzzfcaton, the contnuous nput state problem n Q-learnng s easly resolved. By defuzzfcaton, the contnuous acton problem s also resolved.

6 References [] S. Abe, M. S. Lan, 995, 'Fuzzy Rules Extracton Drectly from Numercal Data for Functon Approxmaton, IEEE trans. on SMC, Vol. 5, No., pp [] F. Klawonn, R. Kruse, 997, Constructng A Fuzzy Controller From Data, Fuzzy Sets and Systems, Vol. 85, pp [3] C. J. C. H. Watns and P. Dayan, 99, Techncal Note: Q- learnng, Machne Learnng, Vol. 8, pp [4] T. Horuch, A. Fujno, O. Kata, T. Sawarag, 996, Fuzzy nterpolaton based Q-learnng wth contnuous states and acton, Proc. FUZZ-IEEE'96, 5th Int. Conf. Fuzzy Systems., New Orleans, pp [5] J. H. Km, I. H. Suh, S. R. Oh, Y. J. Cho, Y. K. Chung, 997, Regon-based Q-learnng usng Convex Clusterng Approach, Proc. IROS 97, pp [6] J. C. Santamara, Rchard. S. Sutton, and A. Ram, 996, Experments wth Renforcement Learnng n Problems wth Contnuous State and Acton Spaces, CONIS Techncal Report 96-88, Unversty of Massachusetts, December. [7] Rchard S. Sutton and Andrew G. Barto, Renforcement Learnng: An Introducton, MIT Press, Cambrdge, MA, 998. [8] G. A. Rummery, 995, ``Problem Solvng wth Renforcement Learnng,'' Ph. D. Thess, Cambrdge Unversty Engneerng Department. [9] C. T. Ln, C. S. George Lee, 994, Renforcement Structure/Parameter Learnng for Neural-Networ-Based Fuzzy Logc Control Systems, IEEE Transactons on Fuzzy Systems, Vol., No., pp , February. θ (deg) X (meter) Control (N) Fg. 6. Trajectory of the pole angle Fg. 7. Trajectory of the cart poston Fg. 8. Trajectory of the control force.