Higher-order Logic Description of MDPs to Support Meta-cognition in Artificial Agents

Transcription

1 Higher-order Logic escrition of MPs to Suort Meta-cognition in Artificial Agents Vincenzo Cannella, Antonio Chella, and Roberto Pirrone iartimento di Ingegneria Chimica, Gestionale, Informatica, Meccanica, Viale delle Scienze, Edificio 6, Palermo, Italy Abstract. An artificial agent acting in natural enironments needs metacognition to reconcile dynamically the goal requirements and its internal conditions, and re-use the same strategy directly when engaged in two instances of the same task and to recognize similar classes of tasks. In this work the authors start from their reious research on meta-cognitie architectures based on Marko ecision Processes (MPs), and roose a formalism to reresent factored MPs in er-order logic to achiee the objectie stated aboe. The two main reresentation of an MP are the numerical, and the roositional logic one. In this work we roose a mixed reresentation that combines both numerical and roositional formalism using first-, second- and third-order logic. In this way, the MP descrition and the lanning rocesses can be managed in a more abstract manner. The resented formalism als maniulating structures, which describe entire MP classes rather than a secific rocess. Keywords: Marko ecision Process, A, Higer-order logic, u-mp, meta-cognition 1 Introduction An artificial agent acting in natural enironments has to deal with uncertainty at different leels. In a changing enironment meta-cognitie abilities can be useful to recognize also when two tasks are instances of the same roblem with different arameters. The work resented in this aer tries to address some of the issues related to such an agent as exressed aboe. The rationale of the work deries from the reious research of the authors in the field of lanning in uncertain enironments [4] where the uncertainty based MP (u-mp) has been roosed. u-mp extends lain MP and can deal seamlessly with uncertainty exressed as robability, ossibility and fuzzy logic. u-mps hae been used as the constituents of the meta-cognitie architecture roosed in [3] where the meta-cognitie u-mp erceies the external enironment, and also the internal state of the cognitie u-mp that is the actual lanner inside the agent. The main drawbacks suffered by MP models are both memory and comutation oerhead. For this reason, many efforts hae been deoted to define a comact reresentation for MPs aimed at reducing the need for comutational

2 resources. The roblems mentioned aboe are due mainly to the need of enumerating the state sace reeatedly during the comutation. Classical aroaches to aoid enumerating the sace state are based on either numerical techniques or roositional logic. The first reresentations for the conditional robability functions and the reward functions in MPs were numerical, and they were based on decision trees and decision grahs. These aroaches hae been subsequently substituted by algebraic decision diagrams (A) [1][8]. Numerical descritions are suitable to model mathematically a MP but they fail to emhasize the underlying structure of the rocess, and the relations between the inoled asects. Proositional or relational reresentations of MPs [6] are ariants of the robabilistic STRIPS [5]; they are based on either first-order logic or situation calculus [2] [10][9][7]. In articular, a first-order logic definition of ecision iagrams has been roosed. In this work we roose a mixed reresentation that combines both numerical and roositional formalisms to describe As using first-, second- and third-order logic. The resented formalism als maniulating structures, which describe entire MP classes rather than a secific rocess. Besides the reresentation of a generic A as well as the imlementation of the main oerators as they re defined in the literature, our formalism defines MetaAs (MA) as suitable A abstractions. Moreoer, MetaMetaAs (MMA) hae been imlemented that are abstractions of MAs. The classic A oerators hae been abstracted in this resect to deal with both MAs and MMAs. Finally, a recursie scheme has been introduced in order to reduce both memory consumtion and comutational oerhead. 2 Algebraic ecision iagrams A Binary ecision iagram (B) is a directed acyclic grah intended for reresenting boolean functions. It reresents a comressed decision tree is. Gien a ariable ordering, any ath from the root to a leaf node in the tree can contain a ariable just once. An Algebraic ecision iagram (A) [8][1] generalizes B for reresenting real-alued functions f : {0, 1} n R (see figure 1). When used to model MPs, As describe robability distributions. The literature in this field reorts the definition of the most common oerators for maniulating As, such as addition, multilication, and maximization. A homomorhism exists between As and matrices. Sum and multilication of matrices can be exressed with corresonding oerators on AS, and suitable binary oerators hae been defined urosely in the ast. As hae been ery used to reresent matrices and functions in MPs. SPU is the most famous examle of alying As to MPs[8]. 3 Reresenting As in Higher-order logic In our work As hae been described in Prolog using first-order logic as a fact in a knowledge base to exloit the Prolog caabilities of managing er-order logic. An A can be regarded as a coule S, where S is the tree s structure,

3 which is made u by nodes, arcs, and labels, and is the ector containing the alues in terminal nodes of the A. Let s consider the A described in the reious section. Figure 1 shows its decomosition in the structure-ector air. Terminal nodes are substituted with ariables, and the A is transformed into its structure. Each element of the ector is a coule made u by a roer ariable inserted into the i-th leaf node of the structure, and a robability alue that was stored originally into the i-th leaf node. Fig. 1. The decomosition of an A in the corresonding structure-ector air S,. In general, As can be reresented comactly through a recursie definition due to the resence of isomorhic sub-grahs. At the same manner, a structure can be defined recursiely. The figure 2 shows an examle. S= A B C A 0.1 B 0.2 = = T1 VA VA VB VB C 0.3 S= T1 VA A VB B VA VB T1 C 0.4 Fig. 2. Each structure can be defined recursiely as comosed by its substructures. A structure can be decomosed into a collection of substructures. Each substructure can be defined searately, and the original structure can be defined as a combination of substructures. Each (sub)structure is described by the nodes, the labels and the ariables in its terminal nodes. Such ariables can be unified seamlessly with either another substructure or another ariable. Foling this logic, we can introduce the concet of MetaA (MA), which is a structure-ector air S, where S is the lain A structure, while is an array of ariables that are unified with no alue (see figure 3). A MA exresses the class of all the different instances of the same function, which inole the same ariables but can roduce different results. An oerator o can be alied to MAs just like in the case of As. In this way, the definition of the oerator is imlicitly extended. The actual

4 Fig. 3. The MetaA corresonding to the A introduced in the figure 1. imlementation of an oerator o alied to MAs can be deried by the corresonding oerator defined for As. Gien three As, add 1, add 2, and add 3, and their corresonding MAs madd 1, madd 2, and madd 3, then add 1 o add 2 = add 3 madd 1 o madd 2 = madd 3. The definition of the ariables in madd 3 deends on the oerator. We will start exlaining the imlementation of a generic oerator for As. Assume that the structure-ector airs for two A s are gien: add 1 = S 1, 1, add 2 = S 2, 2. Running the oerator will gie the foling result: add 1 o add 2 = S 3, 3 Actual execution is slit into two hases. At first, the oerator is alied to both structures and ectors of the inut As searately, then the resulting temorary A is simlified. S tem = S 1 o S 2 tem = 1 o 2 simlify(s tem, tem ) S 3, 3 Here o is the exanded form of the oerator where the structure-ector air is comuted lainly. The equations aboe show that S tem deends only on S 1 and S 2, and it is the same for tem with resect to 1 and 2.The simlify(, ) function reresents the runing rocess, which takes lace when all the leaf nodes with the same arent share the same alue. In this case, such leaes can be runed, and their alue is assigned to the arent itself. This rocess is reeated until there are no leaes with the same alue and the same arent in any location of the tree (see figure 4). Such a general formulation of the effects roduced by an oerator on a coule of MAs can be stored in memory as Abstract Result (ABR). ABRs are defined recursiely to sae both memory and comutation too. An ABR is a t-ule M 1, M 2, O, M 3, F, where M 1, M 2 and M 3 are MAs, O is the oerator that combines M 1, M 2 and returns M 3, while F is a list of relationshis between the ariables in M 1, M 2 and M 3, which in turn deend on O. We alied the abstraction rocess described so far, to MAs also by relacing its labels with non unified ariables. The resulting structure-ector airs hae been called MetaMetaAs (MMA) (see figure

5 = 4 w1 w2 w3 w4 s1 s2 s3 s w1 w2 w3 w4 s1 s2 s3 s4 Formulas = <s1=1+w1; s2=2+w2; s3=3+w3; s4=4+w4> Fig. 4. Two MAs are added, roducing a third MA. Results are comuted according to the formulas described inside the box. 5). We called such comutational entity meta-structure M S. It has neither alues nor labels: all its elements are ariables. M S is couled with a corresonding ector, which contains ariable-label coules (see Figure 5). The definition of MetaMetaStructure A B C A B C Fig. 5. The decomosition of a MA structure into the air MS,, and the corresonding MMA oerators, their abstraction, and the concet of ABR remain unchanged also at this leel of abstraction. 4 iscussion of the Presented Formalism and Conclusions The generalizations of As to second- and third-order logic that were introduced in the reious section, al managing MPs in a more efficient way than a lain first-order logic aroach. Most art of MPs used currently, share many regularities in either transition or reward function. As said before, such functions can be described by As. If these regularities aear, As are made u by sub-as sharing the same structures. In these case, comuting a lan inoles many times the same structure with the same elaboration in different stes of the rocess. Our formalism als to comute them only once and sae it in a second- and third-order ABR. Eery time the agent has to comute structures

6 that hae been used already, it can retriee the roer ABR to make the whole comutation faster. Comutation can be reduced also by comaring results in different MPs. In many cases, two MPs can share common descritions of the world, similar actions, or goals, so the results found for a MP could be suitable for the other one. A second-order descrition als comaring MPs that manage roblems defined in similar domains, with the same structure but different alues. Finally, a third-order descrition als to comare MPSs, which manage roblems defined in different domains but own homomorhic structures. In this case, eery second- and third- order ABR comuted for the first MP can be useful to the other one. This knowledge can be shared by different agents. Adding ABRs to the knowledge base enlarges the knowledge of the agent, and reduces the comutational effort but imlies a memory oerhead. Our future inestigation will be deoted to deise more efficient ways for storing and retrieing ABRs thus imroing the oerall erformances of the agent. References 1. R.I. Bahar, E.A. Frohm, C.M. Gaona, G.. Hachtel, E. Macii, A. Pardo, and F. Somenzi. Algebraic decision diagrams and their alications. In Comuter- Aided esign, ICCA-93. igest of Technical Paers., 1993 IEEE/ACM International Conference on, ages , Craig Boutilier, Raymond Reiter, and Bob Price. Symbolic ynamic Programming for First-Order MPs. In IJCAI, ages , Vincenzo Cannella, Antonio Chella, and Roberto Pirrone. A meta-cognitie architecture for lanning in uncertain enironments. Biologically Insired Cognitie Architectures, 5:1 9, Vincenzo Cannella, Roberto Pirrone, and Antonio Chella. Comrehensie Uncertainty Management in MPs. In Antonio Chella, Roberto Pirrone, Rosario Sorbello, and Kamilla R. Johannsdottir, editors, BICA, olume 196 of Adances in Intelligent Systems and Comuting, ages Sringer, Richard earden and Craig Boutilier. Abstraction and aroximate decisiontheoretic lanning. Artif. Intell., 89(1-2): , January Charles Gretton and Sylie Thiébaux. Exloiting first-order regression in inductie olicy selection. In Proceedings of the 20th UAI Conference, UAI 04, ages , Arlington, Virginia, United States, AUAI Press. 7. Jan Friso Groote and Olga Teretina. Binary decision diagrams for first-order redicate logic. J. Log. Algebr. Program., 57(1 2):1 22, Jesse Hoey, Robert St-aubin, Alan Hu, and Craig Boutilier. Sudd: Stochastic lanning using decision diagrams. In In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, ages Morgan Kaufmann, Saket Joshi, Kristian Kersting, and Roni Khardon. Generalized first order decision diagrams for first order marko decision rocesses. In Craig Boutilier, editor, IJCAI, ages , Saket Joshi and Roni Khardon. Stochastic lanning with first order decision diagrams. In Jussi Rintanen, Bernhard Nebel, J. Christoher Beck, and Eric A. Hansen, editors, ICAPS, ages AAAI, 2008.