Carnegie Melln University Research Shwcase @ CMU Department f Electrical and Cmputer Engineering Carnegie Institute f Technlgy 1992 NETSYN : a cnnectinist apprach t synthesis knwledge acquisitin and use Nenad Ivezic Carnegie Melln University James Henry. Garrett Carnegie Melln University.Engineering Design Research Center. Fllw this and additinal wrks at: http://repsitry.cmu.edu/ece This Technical Reprt is brught t yu fr free and pen access by the Carnegie Institute f Technlgy at Research Shwcase @ CMU. It has been accepted fr inclusin in Department f Electrical and Cmputer Engineering by an authrized administratr f Research Shwcase @ CMU. Fr mre infrmatin, please cntact research-shwcase@andrew.cmu.edu.
NOTICE WARNING CONCERNING COPYRIGHT RESTRICTIONS: The cpyright law f the United States (title 17, U.S. Cde) gverns the making f phtcpies r ther reprductins f cpyrighted material. Any cpying f this dcument withut permissin f its authr may be prhibited by law.
NETSYN - A Cnnectinist Apprach t Synthesis Knwledge Acquisitin and Use N. Ivezic, J. Garrett EDRC 12-54-92
NETSYN a Cnnectinist Apprach t Synthesis Knwledge Acquisitin and Use a Technical Reprt by Nenad Ivezic and James Garrett, Jr. July 1992 This wrk has been supprted by the Engineering Design Research Center, an NSF Engineering Research Center.
11
Abstract The gal f machine learning fr synthesis is the acquisitin f the relatinships between frm, functin, and behavir prperties that satisfy design requirements. The prpsed apprach creates a functin t estimate the prbability f each pssible value f each design prperty being used in a given design cntext. NETS YN uses a cnnectinist learning apprach t acquire and represent this prbability estimatinfunctin and exhibits gd perfrmance fr apsed artificial design prblem. The bjective f future wik is t apply NETS YN t realistic design dmains, specifically the dmain f cmputer system design as practiced by Ml. in
IV
Acknwledgments The authrs gratefully acknwledge the help prvided by Benit Julien and Yram Reich in evaluating the ILLS system and the ECOBWEB system fr the purpses f cmparisns presented in this reprt.
VI
Table f Cntents Chapter 1 Intrductin 1 1.1 SYNTHESIS AND THE ROLE FOR MACHINE LEARNING 1 1.1.1 Design Framewrk fr Synthesis 1 1.1.2 A Rle fr Machine Learning 1 1.2 OVERVIEW OF APPROACH 3 13 MOTIVATIONS FOR USING CONNECTIONIST APPROACH 4 1.4 ORGANIZATION OF REPORT 5 Chapter 2 Backgrund 7 2.1 CHARACTERISTICS OF THE SYNTHESIS PROCESS 7 2.2 REQUIREMENTS FOR A LEARNING SYSTEM 8 23 INDUCTIVE LEARNING APPROACHES FOR SKAU 8 Chapter 3 A Prbabilistic Apprach T Synthesis 10 3.1 ASSUMPTIONS ABOUT SYNTHESIS 10 3.2 a psteriri PROBABILITY ESTIMATION FOR SYNTHESIS 11 3.2.1 General Idea 11 3.2.2 Required Accuracy fr the Prbability Estimatin Functin 13 3.23 Appraches t Prbability Estimatin 13 33 EXAMPLE USAGE: MICON SYNTHESIZER VERSION 1 (Ml) 14 33.1 MICON Synthesizer Versin 1 (Ml) 14 33.2 Example 1 Cmpleting a Design fr CGA cntrller 15 333 Example 2 Estimating Behavir fr Partial Designs 15 33.4 Example 3 Recgnizing Incnsistent Specificatins 15 Chapter 4 NETSYN a Cnnectinist Apprach t SKAU 19 4.1 A CONNECTIONIST a psteriri PROBABILITY ESTIMATION 19 4.2 THEORETICAL BASIS FORNETSYN 19 4.2.1 Cmputatinal Thery Backgrund 19 4.2.2 Learning Thery Backgrund 21 4.3 CONNECTIONIST (NETSYN) REPRESENTATIONS 21 43.1 Design Representatin 21 43.2 Design Prcess Representatin 21 4.4 NETSYN ARCHITECTURE 22 4.5 NETSYN LEARNING 22 vu
Chapter 5 Evaluatin f NETSYN 25 5.1 TEST METHODOLOGY - AN ARTIFICIAL DESIGN PROBLEM (ADP) 25 5.1.1 Synthesis Space 25 5.1.2 Synthesis Knwledge 26 5.1.3 Training and Test Set Generatin 26 5.1.4 Perfrmance Evaluatin Methdlgy 1 27 5.1.5 Perfrmance Evaluatin Methdlgy 2 28 5.2 NETSYN PERFORMANCE ON ADP 29 5.2.1 TEST 1 Results 29 5.2.2 TEST2 Results 31 5.3 COMPARATIVE ANALYSIS: ILLS AND NETSYN 31 5.4 COMPARATIVE ANALYSIS: ECOBWEB AND NETSYN 33 55 DISCUSSION 33 Chapter 6 Future Wrk 35 Chapter 7 Summary 36 References 37 V1U
List f Tables Ikble 1 - Synthesis Rules fr Attribute E 27 Ikble 2 - Synthesis Rules fr Attribute F 27 Tkble 3 - Synthesis Rules fr Attribute G 27 Ikble 4 - Synthesis Rules fr Attribute H 27 IX
List f Figures Figure 1 Synthesis 2 Figure 2 Analysis 2 Figure 3 Evaluatin 2 Figure 4 Redesign 2 Figure 5 Relatinships Needed t Capture fr SKAU 3 Figure 6 Alternative Decisin Sequences fr a Synthesis Prcess 11 Figure 7 The Prbability Estimatin Functin 12 Figure 8 Iterative Design Cmpletin Using the Prbability Estimatin 13 Figure 9 Synthesis Hierarchy fr CGA Cntrller 16 Figure 10 Cmpletin f Design fr CGA cntrller 17 Figure 11 Estimating Behavir fr Partial Design 18 Figure 12 Recgnizing Incnsistent Specificatins fr CGA Cntrller 18 Figure 13 Mapping a Neural Netwrk n Prbability Estimatin 20 Figure 14 NETSYN Architecture 22 Figure 15 Mdular Cnstructin f a Neural Netwrk fr SKAU 23 Figure 16 Itaining f a Netwrk fr Synthesis Knwledge Acquisitin 24 Figure 17 NETSYN Perfrmance with Respect t "set-cverage" (TEST 1) 30 Figure 18 NETSYN Perfrmance with Respect t "max-crrect" (TEST 1) 30 Figure 19 NETSYN Perfrmance with Respect t "set-cverage" (TEST 2) 32 Figure 20 NETSYN Perfrmance with Respect t "max-crrect" (TEST 2) 32 Figure 21 Cmparisn f Perfrmance: NETSYN and ILLS 34 Figure 22 Cmparisn f Perfrmance: NETSYN and ECOBWEB 34
Chapter 1 Intrductin The bjectives f this chapter are t describe a general framewrk fr the synthesis prcesses cnsidered in this research and t identify a rle fr machine learning in design dmains with weak r nnexistent synthesis knwledge. In additin, this chapter includes an verview f, and the mtivatin fr, cnnectinist learning appraches fr synthesis knwledge acquisitin and use (SKAU). Finally, an utline f the cntents f the reprt is given. 1.1 SYNTHESIS AND THE ROLE FOR MACHINE LEARNING 1.1.1 Design Framewrk fr Synthesis The main bjective f the synthesis prcess is t generate a descriptin f the frm f an artifact such that this descriptin satisfies a cllectin f design requirements. Frm here is used in the same sense as in [Hemming 92] t mean gemetry, tplgy, material, etc. The synthesis prcess is the initial stage f a multi-stage artifact design prcess cnsisting f the fllwing fur stages: synthesis, analysis, evaluatin, and redesign. In the synthesis stage, several pssible frms f an artifact are generated based n a given set f specificatins. In the analysis stage, the behavir f these pssible frms are determined fr the specified functinal cntext. In the evaluatin stage, the predicted behavirs are cmpared with thse desired and the deficiencies f the design are identified. In the redesign stage, changes t riginal pssible frms are made t address the deficiencies fund during evaluatin, r new realizatins are generated. If we cnsider the artifact resulting frm a design prcess t be described by its frm (e.g., tplgy, shape, material),/wrtcri0rt (e.g., lad resistance, vibratin islatin), and behavir prperties (e.g., stress and strain states), then we can describe each f the fur stages in the multi-stage prcess as fllws [Flemming92]: Synthesis is the prcess f mapping frm the design specificatin space (requirements n functin, behavir, frm) t the frm space (Fig. 1). Analysis is the prcess f mapping frm the frm and specificatin spaces t the behavir space (Fig. 2). Evaluatin is the prcess f cmparing the behavir prperties t the behavir requirements and the frm prperties t the frm requirements (Fig. 3). Redesign is the prcess f mapping frm the specificatin space t the frm space, with the knwledge f subspaces that lead t infeasible designs (Fig. 4). 1.1.2 A Rle fr Machine Learning The specificatin fr an artifact used as input by the synthesis prcess may invlve any number f required functinalities, frm cnstraints, and behavir cnstraints. Fr prblems where all f the cnstraints and bjectives can be clearly articulated and expressed in algebraic frm, mathematical pti-
Functinal Specs Behaviral Specs Frm Specs Specificatin Space represents set f frm prperty values Frm Space Figure 1 Synthesis represents set f frm attribute values Frm Space represents a set f behaviral attribute values Functinal Specs Specificatin Space Figure 2 Analysis Behavir Space 1 + behavir frm functin i. ; IcmnareJ^ ^^.. -I cmpare!-. * ^ Specificatin Space Frm Space Figure 3 Evaluatin Behavir Space infeasible subspaces nal Specs iviral Specs represents set f frm attribute values Specificatin Space Frm Space Figure 4 Redesign
mizatin appraches can be emplyed t search ver the slutin space (i.e., the space f all pssible slutins) and select the "ptimal" slutin t the set f cnstraints and defined bjectives. Hwever, fr prblems where there are few cnstraints and the relatinships between frm and perfrmance are unknwn, a mathematical ptimizatin apprach is nt amenable. Hwever, if weak theries r heuristic knwledge abut synthesizing slutins t a prblem exist, knwledge-based synthesis appraches can be used t assist (thrugh the applicatin f heuristic knwledge) in efficiently searching thrugh the space f pssible design slutins, eliminating thse slutins that are nt prmising and fcussing n thse slutins that are. Hwever, frthse prblems in whichn significant dmain thery r cllectin f heuristics exists, the nly surce f synthesis knwledge tends t be past design experience [Reich 91]. One rle fr machine learning, identified in [Reich 91], lies within thse design dmains pssessing the fllwing characteristics: The design space is identified (i.e., all prperties and pssible values are knwn a priri). Synthesis knwledge is weak r nnexistent There exists a cllectin f design experiences. The rle f machine learning in dmains f this type is t acquire, frm past design cases, the relatinships between frm, functin, and behavir prperties that satisfy specified design requirements. These relatinships frm a cne f useful synthesis knwledge. Having such synthesis knwledge facilitates a mre direct mapping frm the specificatin space t the desired lcatins within the frm and behavir spaces. In ther wrds, less search is needed t find a gd slutin t the specificatin (Fig. 5). 1 behavir frm functin ; m t _^-~-~-H" satisfies l^-» *2> *** Specificatin Space Frm Space Behavir Space Figure 5 Relatinships Needed t Capture fr SKAU Hence, the gal f the research described in this reprt is t investigate and identify prmising learning appraches which are capable f "playing" the abve rle f acquiring synthesis knwledge. Next, we utline the apprach taken by this research in accmplishing this gal that will be presented in mre detail later in this reprt. 12 OVERVIEW OF APPROACH We prpse a prbabilistic apprach t synthesis that is based n the fllwing assumptins: Design can be sufficiently well represented as a cllectin f prperty-value pairs (i.e., a finite number f decisins). Cnsequently, the synthesis prcess is equivalent t a sequence f assignments f prperty values which ultimately lead t a cmplete design definitin.
Each prperty value assignment (decisin) is made in sme design cntext (i.e., partially defined specificatin and/r design descriptin). It is pssible t cnstruct a sufficiently accurate estimatin f the prbability f each value f each design prperty being used in a given design cntext. Based n the abve assumptins, the gal f identifying learning appraches capable f acquiring synthesis knwledge is transfrmed int tw subgals: Identify a useful prbabilistic cncept which prvide an adequate measure f belief that a prperty value can be used in sme design cntext. Identify an effective methd fr the cnstructin f a prbability estimatin functin that implements the selected prbabilistic cncept. In this reprt we demnstrate that the slutin t these tw subgals (and t the riginal gal) may be achieved by applying cnnectinist learning appraches. Our cnnectinist-based slutin relies n the fllwing: A recent fundamental theretical research result demnstrates that a grup f cnnectinist learning methds is capable f estimating Bayesian a psteriri prbabilities. A representative methd frm the abve grup f cnnectinist learning methds may be successfully used t acquire and stre the prbability estimatin functin, thus, allwing inductive learning t be the methd f cnstructin fr this prbability estimatin functin. This reprt discusses and illustrates the use f a cnnectinist learning apprach t acquire and represent a prbability estimatin functin that can be used t achieve ur gal acquire synthesis knwledge fr its subsequent reuse. 1J MOTIVATIONS FOR USING CONNECTIONIST APPROACH Building a synthesis system in prblem dmains where there is a weak, r nnexistent, dmain thery, and where mathematical ptimizatin methds are nt amenable is a difficult task. Fr these types f prblems, an inductive machine learning apprach may pssibly be used t capture, and cnsequently permit reuse f, the synthesis knwledge embdied in the past design cases. While sme very gd research has been cnducted n hw symblic machine learning techniques can be applied t the synthesis prblem [Reich 91, Lu 92], little research has investigated hw neural netwrks might be used t capture and subsequently use synthesis knwledge. The research reprted here has fcuses n cnnectinist learning methds fr SKAU. The fllwing are the mtivatins fr investigating the cnnectinist learning methds fr SKAU: Cnnectinist systems are adaptive systems capable f capturing cmplex nnlinear relatinships. Bth the theretical results cncerning representatinal capabilities f cnnectinist systems [Cybenk 88, Hrnik 89], and the results f applicatins f cnnectinist mdels t difficult learning prblems in cntrl [Pmerleau 89], signal predictin [Lapedes 88] and pattern recgnitin [Sejnwski 87, Qian 88], illustrate that cnnectinist systems are capable f learning cmplex relatinships frm a sample f input-utput pairs representative f that relatinship. The relatinships between design variables can be very cmplex and it is felt that the pwer f neural netwrks can be emplyed t induce these relatinships frm a representative training sample f previus designs. The knwledge representatin paradigm in cnnectinist systems ffers attractive capabilities. The distributed knwledge representatin fund in cnnectinist systems has been shwn t lead t the kinds f peratins that are reminiscent f peratins f the human brain: cntent addressable memry recall (i.e., recnstructin), nise resistant cmputatin, and gracefully degrading cmputatin [Hintn 86]. Obviusly, a learning system fr SKAU culd beneficially emply these kinds f
peratins, particularly in the cntext f design dmains with weak synthesis knwledge. In additin, distributed knwledge representatin allws autmatic generalizatin [Hintn 86], which is ne f the key requirements fr inductive learning systems fr SKAU. Cnnectinist learning systems have exhibited cnsistently gd perfrmance n tasks similar t SKAU. The applicatin f cnnectinist systems that mtivates ur research is neural pattern recgnitin [Le Cun 89]. Since the dmains f interest t this research are assumed t have weak frmal knwledge f the synthesis prcess, ne apprach t acquire the actual synthesis knwledge is t search fr meaningful patterns f features in past successful designs. Recgnitin f such patterns and their generalizatin t new design scenaris culd assist the synthesis prcess in prducing "gd" designs. The existence f a lng list f successful neural pattern recgnitin applicatins, illustrates the feasibility f a cnnectinist apprach t SKAU [Le Cun 89, Tesaur 90, Pmerleau 89]. Cnnectinist learning systems estimate Bayesian a psteriri prbability. The result that a class f cnnectinist learning systems is capable f estimating Bayesian a psteriri prbability prvides justificatin fr emplying a neural pattern recgnitin system in perfrming SKAU. The fundamental difference between applying a pattern recgnitin system t the task f SKAU and ther similar tasks is the nnunique nature f the mappings used in synthesis. The acquired synthesis knwledge has t preserve all f the relatinships that hld between design prperties fr all f the multiple pssible alternatives. This is fundamentally different frm the requirement psed in ther tasks that search fr nly ne unique slutin t the prblem. By incrprating the cncept f a psteriri prbability estimatin we are able t satisfy the need t represent bth the relatinships and the multiple alternatives fr SKAU. In additin, this result prvides a rigrus theretical backgrund fr analysis f the perfrmance f cnnectinist systems and fr their use in practical applicatins. Cnnectinist systems allw the use f Bayesian learning methd. Sme f the recent research has fcussed its attentin n the analysis f learning in cnnectinist systems by fllwing the Bayesian learning methd [Buntine 91]. While we currently assume that the unifrm cnvergence methd applies (i.e., we assume the existence f a sufficiently large sample size and ignre factrs that appear as prirs in the learning mdel), the Bayesian learning methd prvides imprtant results fr the case f limited sample sizes. Althugh the Bayesian apprach has nt been applied in this research t date, we acknwledge the imprtance f these results fr future wrk. 1.4 ORGANIZATION OF REPORT The fllwing is the rganizatin f this technical reprt: Chapter 2 presents the synthesis prcess cnsidered in this research and discusses ne type f machine learning (i.e., inductive learning) apprach that is capable f SKAU under the cnditins specified in sectin 1.1. We present the characteristics f the synthesis prcess in general and f the synthesis prcess in dmains with weak r nnexistent synthesis knwledge. Frm this discussin we derive basic requirements fr a learning system capable f learning synthesis knwledge. We als describe tw alternative inductive learning appraches which have been emplyed fr SKAU. Chapter 3 develps a prbabilistic apprach t synthesis. The apprach is based n the assumptin that it is pssible t cnstruct a sufficiently accurate estimatin f the prbability f each pssible value f each design prperty being used in a given design cntext. The assumptins abut the synthesis prcess cnsidered in this apprach are presented and a prbabilistic apprach fr synthesis knwledge acquisitin and use (SKAU) is described. We discuss the minimal requirements n the accuracy f the prbability estimatin functin t be useful fr synthesis. The apprach is illustrated n a number f examples taken frm a realistic design dmain. Chapter 4 prvides a detailed accunt f NETSYN - a neural netwrk apprach fr synthesis knwledge acquisitin and use (SKAU). The basic idea f the apprach is t use a cnnectinist learning
system t acquire and represent the prbability estimatin functin intrduced in Chapter 3. Theretical results relevant t the expected perfrmance f this apprach are presented alng with a discussin f hw this learning apprach fits int a mre general machine learning framewrk. Cnnectinist representatins are then discussed. Discussin f the architecture and the training f NETSYN is prvided. Chapter 5 describes the test methdlgy used fr evaluating the perfrmance f NETSYN and its cmparisn t tw symblic inductive learning appraches: ILLS and ECOBWEB. A series f results is presented fr the perfrmance and cmparisn f NETSYN t these symblic learning appraches. Chapter 6 gives directins fr the future wrk cncerning the verall gal f NETSYN: applicatin t a realistic design synthesis task. Chapter 7 summarizes the results presented in this technical reprt.
Chapter 2 Backgrund In this Chapter, we describe in mre detail the synthesis prcess cnsidered in this research and discuss ne a type f machine learning apprach that is capable f SKAU under the cnditins specified in the previus chapter. First we present the characteristics f synthesis prcess in general and f the synthesis prcess in dmains with weak r nnexistent synthesis knwledge. Then, we present basic requirements fr a learning system capable f learning synthesis knwledge in such a dmain. Finally, we describe tw alternative inductive learning appraches which have been emplyed fr SKAU. 2.1 CHARACTERISTICS OF THE SYNTHESIS PROCESS The engineering synthesis prcess is ften viewed as an explratin f a space f pssible slutins fr a psed design prblem. Hwever, the directins fr perfrming this search are rarely given in an explicit frm fr any engineering design dmain. Design prblems are ill-structured and the design prblem frmulatin is itself a dynamic prcess where new design knwledge causes the refrmulatin f the design prblem and, subsequently, refrmulatin f the synthesis search space [Cyne 90]. In certain design dmains, it may be assumed that the dynamics f the design prblem frmulatin are slwer than the actual synthesis space traversal. This bviusly hlds in mre mature design dmains where the knwledge f the design synthesis space (i.e., design variables and the relatinships between them) has been reinfrced thrugh the many successful utcmes f synthesizing designs fr given design specificatins. The synthesis prcess in many design dmains is characterized by generating alternatives which "satisflee" the given design requirements [Simn 81]. Therefre, the utcme f a synthesis prcess in such a design dmain is a cllectin f candidate designs that satisfy initial design requirements. The nnunique nature f the synthesized designs pses a requirement n the designer t cnsider and evaluate mre than ne alternative slutin. Anther characteristic f the synthesis prcess is the utilizatin f varius pieces f infrmatin f different nature. Bth qualitative and quantitative data are cmbined by a designer during the design synthesis prcess. The ways in which such data are cmbined t decide which part f the search space t traverse are ften hard t express r analyze rigrusly using classical mathematical appraches. In the prcess f generating alternatives during the synthesis prcess, the designer will narrw the search space by cnsidering nly selected subintervals f values f design variables. Eventually, the design variables will be assigned specific values t give a mre cmplete design cntext The decisin t select a particular variable value will influence the feasible ranges f ther design variables. The relatinships between design variables are in general cmplex and many-t-many: cnstraining ne design variable will affect the selectin prcess f all ther variables, while a particular design variable assignment may be incnsistent with previusly cnstrained design variables and cause an infeasible design. These relatinships are results f designer's experience and are reinfrced thrughut many successful synthesis prcesses. The designer's ability t capture and t generalize these relatinships frm a finite number
f previus design episdes and, subsequently, t use this knwledge in new design situatins is a crucial capability fr successful synthesis. In the framewric f a frmulated synthesis prblem, where the dimensinality f the search space (i.e., prperties t be specified) is knwn but the synthesis knwledge (i.e., relatinships between design prperties) is weak r nnexistent, the nly surce f synthesis knwledge tends t be the past design experience. In such a design situatin, where an experienced designer is faced with a new design prblem, the first step is t perfrm sme limited (t the extent t which the synthesis thery is available) analysis f the design prblem and then t prceed with the synthesis prcess. The knwledge f first principles f behavir in the cnsidered engineering dmain is f limited utility in cnstructing the design alternatives. T slve the design prblem, the designer relies n his r her past experience t recgnize the similarities f the new situatin t the design prblems he r she has encuntered in the past. The designer recgnizes the imprtant characteristics f the design prblem and fcuses his r her attentin n these imprtant cnsideratins. This subjective, experience-based knwledge f the designer will gvern the synthesis prcess f generating alternative design slutins. The identified nature f the synthesis prcess in dmains with nnexistent r weak dmain thery frms the basis fr deriving the requirements fr systems that are t capable f capturing synthesis knwledge and facilitating its reuse in new design prblems. These requirements are presented next. 2.2 REQUIREMENTS FOR A LEARNING SYSTEM Three requirements are identified as necessary ingredients f any system capable f capturing synthesis knwledge and using this knwledge in nvel design situatins [Reich, 1991]: The ability t capture many-t-many mappings between frm, functin and behavir design prperties. In general, ne must cnsider all design prperties fr which values have been fixed r ranges cnstrained when making decisins abut yet unbund design prperties. Cnsidering nly the design specificatins and disregarding values f design attributes fixed in the curse f synthesis will in general lead t infeasible slutins. The ability t recall r generate multiple design alternatives satisfying the particular cmbinatin f specificatins. As described earlier, the synthesis prcess is cncerned with generating multiple alternatives fr the same design specificatins. The ability t generalize frm the individual design experiences i.e., generating acceptable slutins in design situatins nt seen befre. A learning system is expected t behave well in the new design situatins that are similar t, but nt identical t, previus successful design episdes. 2.3 INDUCTIVE LEARNING APPROACHES FOR SKAU Inductive learning appraches are candidate machine learning appraches fr acquiring synthesis knwledge and allwing the reuse f this knwledge in new design situatins. A brief discussin f tw alternative inductive learning appraches fr synthesis knwledge acquisitin is presented next (fr a mre exhaustive treatment f learning methds fr this task, see [Reich 91]). Inductive learning appraches can be gruped in tw main classes: The first is the class f supervised learning appraches. In supervised learning appraches, learning is based n the availability f a crrect answer fr any input descriptin f a situatin that needs t be classified r t which a value is t be assciated. The learning system has t adjust itself accrding t the representative training cases. The gal f learning is that the system ultimately learns crrect assciatins between input and utput patterns. The secnd is the class f unsupervised learning appraches. In this case, the learning target is nt specified explicitly in terms f crrect results fr a set f training cases. The nly available inf rma-
tin is in the crrelatins f the input data. The system has t discver regularities in the training set and t create categries based n the discvered crrelatins. These categries are the basis fr classificatin f the new input. An example f an unsupervised learning apprach t synthesis knwledge acquisitin is the ECOB WEB learning algrithm used in the Bridger system [Reich 91]. The ECOBWEB learning algrithm is used t frm meaningful clusters ut f design training cases. These clusters can then be used t cmplete the synthesis prcess fr given partial design descriptins. ECOBWEB utilizes a prbabilistic apprach and it uses a heuristic perfrmance measure t cmpare alternative classificatins with respect t their utility in the classificatin f designs. These classes are then used t classify a new, partially cmplete design specificatin. Once the input specificatin is classified, the whle classificatin hierarchy can be used t synthesize design prperty values cnsistent with members f that class. The supervised learning apprach has been criticized as being inapprpriate fr learning synthesis knwledge [Reich91]. Early learning systems used this apprach in a straightfrward manner which prevented them frm learning "cmplete" relatinships between design prperties: nly the mapping frm the set f fixed specificatin prperties t design descriptin prperties was captured. N attentin was paid t the influence that fixing sme design attribute values might have n the feasible ranges f ther design prperties. Hence, many-t-many relatinships between design prperties culd nt be captured and the use f the supervised learning apprach was rightfully judged as inapprpriate fr learning synthesis knwledge. In ur research we have fllwed the supervised learning apprach. Hwever, the fundamental difference between ur apprach and the previus appraches using supervised learning is that ur basis fr classificatin is nt just the subset f design prperties declared t be specificatins. Rather, all design prperties knwn in a given design cntext are used t determine the values f the remaining, unknwn design prperties. The apprach is basically ne f predicting the a psteriri prbabilities f each pssible value f each unknwn prperty fr the given set f knwn prperty values. Using this apprach, it is pssible t accunt fr the many-t-many relatinships within synthesis knwledge. The next sectin gives a detailed descriptin f this apprach.
Chapter 3 A Prbabilistic Apprach T Synthesis In this chapter we develp aprbabilistic apprach t synthesis. The apprach is based n the assumptin that it is pssible t cnstruct a sufficiently accurate estimatin f the prbability f each pssible value f each design prperty being used in a given design cntext. First, assumptins abut the synthesis prcess cnsidered in this research are presented. Then, a prbabilistic apprach fr synthesis knwledge acquisitin and use (SKAU) is described. Next, the minimal requirements n the accuracy f the prbability estimatin functin t be useful fr synthesis are given. Finally, the apprach is illustrated n a number f examples taken frm the dmain f Ml [Gupta 91] - a knwledge based system fr synthesizing single-bard cmputer systems. 3.1 ASSUMPTIONS ABOUT SYNTHESIS In this sectin we list several assumptins abut the kind f synthesis prcesses we address in this research and fr which we attempt t create a learning system. We view a design as a cllectin f prperty-value pairs r, equivalently, cllectin f design decisins. The design synthesis prcess is then equivalent t a sequence f assignments f prperty values which ultimately leads t a cmplete design definitin. Sme prperty values are determined a priri, representing design specificatin values. All design decisins are made in sme design cntext (i.e., a partial design descriptin). T further elabrate n ur assumptins, cnsider a representatin f a simplified design prcess shwn in Fig. 6. On the left side f the figure, an initial design cntext is shwn. T cmplete the design descriptin, the values f three design prperties need t be determined (ne specificatin prperty and tw design descriptin prperties). Each design prperty may take n ne ut f three distinct values (graphically represented by an rectangle, triangle, and an val). In the initial cntext, nly the design specificatin value is determined (dented by a slid shape). The first task in the design prcess is t cnsider all f the design prperties fr which values have nt been determined and t select ne and assign its value. In the mst general case the designer wuld like t cnsider every unreslved design prperty and decide which values are feasible fr the given design cntext. Then, based n sme design strategy, the designer culd select a prperty and its value which he believes will lead t a gd design. In the figure, cmpleting this task gives the intermediate state f design cntext (1) and the new, mre cmplete, design cntexts (2 r 3). The intermediate state (1), represents cnsideratin f unbund design prperties. The different shades f symbls representing values depict that each value may appear in a given design cntext (the darker shades indicate greater level f belief that the value applies fr the given design cntext). Alternative design cntexts (2 and 3) are results f different design decisins. Similarly, ging frm either f tw partial design cntexts (2 r 3), the designer cnsiders the values that the remaining design prperty may assume. Intermediate states f the design cntext (2a) and (3a) represent a degree f belief that the designer may have that each unbund prperty culd be assigned each f its pssible values in design cntext (2) and (3), respectively. The final design cntexts (2b and 3b) are representatins f utcmes f tw alternative decisin sequences. 10
Design attributes edesign sped 01 jaol Initial design cntext PAO1 valued LI1 Final design cntexts Figure 6 Alternative Decisin Sequences fr a Synthesis Prcess A particular pint wrth nting here is that a designer usually has a number f alternative chices fr making synthesis decisins. The designer utilizes his r her subjective knwledge f the synthesis prcess (i.e., knwledge f relatinships amng prperty values) t determine which prperty value will be used in the given cntext (which changes as decisins are made); cnsequently, he r she decides which particular prperty value t determine next. We are specifically interested in acquiring this subjective knwledge that is used t select particular prperty values fr given design cntexts. 32 a psteriri PROBABILITY ESTIMATION FOR SYNTHESIS 3.2.1 General Idea We cast the described synthesis prcess int a prbabilistic framewrk. The prbabilistic cncept we select t emply in this research is Bayesian a psteriri prbability. This cncept has been lng recgnized as useful relative t engineering planning and design, as it incrprates engineering judgment and bservatinal data in a frmal framewrk [Ang 75]. T use the cncept f a psteriri prbability in a traditinal way, ne has t knw a priri prbabilities f classes f phenmena ne deals with as well as likelihd functins (i.e., class cnditinal prbability functins) f the measurements (i.e., dependent variables) that ne can btain fr these phenmena. Measurements may be cntinuus r discrete values and they are represented in the frm f a measurement vectr. The Bayesian a psteriri prbability p(q IX) represents the cnditinal prbability f class Q given the input measurement vectr X. Use f Bayes rule allws it t be expressed as fllws: w _p (X I Q) p(c t ) where X is a measurement vectr p (XI Q) is the likelihd f prducing the measurement X if the class is Q p (Ci) is the a priri prbability f class Q p (X) is the uncnditinal prbability f the measurement In ur synthesis framewrk, the phenmena in which we are interested are the design prperty value assignments fr which we knw in advance the classes (i.e., values) and the measurements are all bund 11
design prperties that make a current design cntext (i.e., the measurement vectr). The a priri prbabilities can in general be estimated fairly well; hwever, the likelihd functins invlve assumptins abut the prbability distributins f values f the phenmena (design prperties) with respect t the measurements (design cntext). Instead f indirectly estimating a psteriri prbabilities f each prperty value assignment by cmputing prbabilities accrding t Bayes rule, we will cnsider an alternative apprach f directly estimating a psteriri prbabilities (hereafter, prbabilities) using a cnnectinist learning mdel. This is a subject f Chapter 4. One way t use the a psteriri prbability cncept is t use it as a prbability estimatin f each value f each unbund design prperty being used in a given design cntext Such prbabilities will be very useful infrmatin during synthesis. Cnsider Fig. 7, which illustrates this idea. Each prperty value is dented by a circle. The prperties are divided int specificatins and frm and behavir prperties. The assigned prperties frm the current design cntext which is perated upn by the prbability estimatin functin. The result f this peratin is a set f estimated prbabilities f each value f each unbund design prperty being used in the given design cntext The shaded circles n the right hand side f the figure represent these prbability estimatins fr each prperty value. Daiker shades represent higher prbability f the value appearing in the given design cntext. Fr a given set f design specificatins, r any partial descriptin f design, by using a prbability estimatin functin we are in a psitin t d the fllwing: Acquire the prbabilities f the values appearing fr each unbund frm and behavir prperty. Reasn abut which values f unbund design prperties are viable alternatives in a given design cntext specificatins s^ frm and behavir prperties s4 dl". values. OO O O OOO OOO OO values " Prbability Estimatin _ values _ OO O O O «O OO CO values I sl ' s2! s3! s4 : di J d2 ; d3 ; d4 specificatins; frm and J behavir ; prperties! Legend: m = attribute value selected = P(dijlSn&Si2&.. &S 42 ) 1.0 0.5 Figure 7 - The Prbability Estimatin Functin Given such a tl, ne can imagine a synthesis prcess being cmpleted by iteratively applying the fllwing steps, starting frm sme design cntext (i.e., design specificatins) and ending with the cmpletin f the design (see Fig. 8): Apply the prbability estimatin functin t acquire the prbability estimatin f each value f each unbund design prperty fr the given design cntext. Cnsider prbability distributins ver the values fr each unbund design prperty and decide t which prperty t assign a value next (this step may be subject t a specific strategy f selecting the next prperty t bind). D 0.0 12
Initial Design Cntext Iteratin 1: Prbab. Estimat. OO Iteratin 2: Prbab. Estimat Iteratin 3: O9O Prbab. Estimat Figure 8 Iterative Design Cmpletin Using the Prbability Estimatin Cnsider the prbabilities fr values f the selected prperty and decide which value t assign t the prperty. Update the design cntext by fixing the prperty value. 3.2.2 Required Accuracy fr the Prbability Estimatin Functin Frm the perspective f the usability f this apprach, the prbability estimatin functin shuld: (1) estimate nn-zer prbabilities fr design prperty values which have appeared in a given design cntext; (2) estimate zer prbabilities fr the prperty values that have nt been in that design cntext; and (3) crrectly rank the mst prbable design prperty value. This minimalist accuracy requirement is necessary as the relatinships between design prperties in a realistic design dmain may be very cmplex, and cnstructin f a prbability estimatin f an arbitrarily high accuracy may be impssible. Althugh the abve requirement is sufficient t render the prpsed apprach useful, it still may be t hard fr practical design prblems. It is likely that fr any prblem f this cmplexity nly an apprximatin f the the prbability estimatin functin will be pssible. We will return t this accuracy requirement again in Chapter 5, where we deal with evaluatin f the prpsed apprach. 3.2.3 Appraches t Prbability Estimatin One may emply different appraches t cnstruct the prbability estimating functin. These appraches may in general be classified as exact prbabilistic methds, apprximatins f exact prbabilistic meth- 13
ds, r heuristic methds. By fllwing an exact apprach, ne needs t fllw axims f prbability thery and prvide a rigrus prf that the cnstructed prbability estimating functin fr the cnsidered prblem has the desired prperties. While this may be pssible fr small prblems, the difficulty f real synthesis prblems prevents the actual usage f this apprach. Methds fr apprximating these prbabilistic methds may be a mre viable apprach t cmplex tasks such as synthesis. The quality f the perfrmance f these apprximating methds depends n the prblem typically, the size f the sample that is used t cnstruct the prbability estimatin functin and the dimensinality f the prblem. Therefre, the usefulness f this apprximating apprach varies with the cnditins that the actual synthesis prblem may impse. Finally, the heuristic methds sacrifice exactness fr feasibility and decreased cmplexity in cnstructing the prbability estimating functin. In additin, these methds are smetimes emplyed t intrduce prblem-dependent biases, which cause the estimatin f the prbabilities t fllw sme assumed prbability distributins peculiar t that prblem. In the next chapter we present a cnnectinist apprach t cnstructing the prbability estimatin functin that actually estimates a psteriri prbability. 3.3 EXAMPLE USAGE: MICON SYNTHESIZER VERSION 1 (Ml) This sectin illustrates the use f the prpsed prbabilistic apprach within the framewrk f a synthesis system fr the design f cmputer systems. First, a descriptin f the synthesis tl in MICON (Ml) and the system-level synthesis task perfrmed by this tl are given (this part is based n [Gupta 91]). 3.3.1 MICON Synthesizer Versin 1 (Ml) Ml (MICON Synthesizer Versin 1) is a part f the MICON system that designs single-bard cmputer systems [Gupta 91]. Specifically, M1 is a knwledge-based synthesis tl that satisfies high level specificatins by perfrming system-level synthesis f cmputer systems frm a set f cmpnents. The bjective f system-level synthesis is t create a cmplete and peratinal cmputer system capable f perfrming general-purpse r special-applicatin cmputing. The cmpnents f design are integrated circuits (i.e., ICs) r applicatin-specific IC libraries. Input specificatins t Ml are based n a functinal descriptin f the elements; n details f cmpnents and their specifics are included in the specificatins. Typically, the result f the system-level synthesis cnsists f a cnfiguratin f CPU (i.e., central prcessing unit), memry system, I/O (input utput) cmpnent, bus interfaces, and ther supprting circuitry. Tw characteristics f this synthesis task are: There exists n mdel r language t clearly define the functin f an artifact being designed. There exists n cmplete and well-defined thery relating structure (i.e., frm) and behavir f the artifact. Being a realistic prblem, the fllwing are sme prperties that make the synthesis prcess in this dmain hard: The design space is large as the parts can be used in a variety f ways t fulfill functinal specificatins. Cmbinatrial explsin is a result f an attempt t thrughly search ver the whle design space. There are interactins between the design sub-prblems. Ml divides the synthesis prblem int a set f sub-prblems and cannt, in general, determine that the design fr a subsystem will yield a satisfactry cmplete design. There are cmplex interactins amng the cmpnents. The relatinships between cmpnents depend n several factrs, including the way in which a cmpnent is prgrammed. The knwledge base is rapidly evlving. New ICs are cnstantly being develped, leading t imprved cmpnents and new design styles. 14
Based n the abve descriptin, this synthesis task belngs t the class f synthesis prblems with well defined design spaces but weak synthesis knwledge. In the fllwing subsectins, we illustrate several ways in which the prpsed prbabilistic apprach fr SKAU culd be used by M1. We cnsider the least cmplex subdmain f M1 CGA cntrller design fr which the synthesis space descriptin is given in the frm f the synthesis hierarchy in Fig. 9. In general, Ml uses an "AND-OR" hierarchy t represent the design space. The synthesis hierarchy fr CGA cntrller design represents a part f this "AND-OR" hierarchy. The state f the Mi's knwledge base fr the CGA cntrller has dictated the shape f, and the size f, this hierarchy. The arch acrss the tp level branches indicates the tp nde being "AND" nde. The lwer level ndes are "OR" ndes (indicated by heavier lines with rdinal numbers in circles in frnt f the alternatives) and "AND" ndes (indicated by lighter lines). The nly decisins that need t be made crrespnd t "OR" ndes. Therefre, nly thse cmpnents crrespnding t "OR" ndes are cnsidered in the fllwing examples. In additin, a subset f perfrmances and specificatins listed in the same figure are included in the illustratins: cst, RAM (Randm Access Memry) size, ROM (Read Only Memry) size, and the speed f the bard. 33.2 Example 1 Cmpleting a Design fr CGA cntrller The initial example illustrates the idea f cmpleting a design fr a given design specificatin (Fig. 10). Fr the given specificatins (cst range, RAM size, ROM size, and desired speed f the bard), the task is t estimate in every step f the design cmpletin prcess the prbability f each value f each unbund design prperty (RAM chip, ROM chip, Address Decder, and number f Address Decder cmpnents) being used in the current design cntext The utcme f the represented design cmpletin prcess can be read frm the final iteratin (4) in Fig. 13: RAM chip = 6264, ROM chip = 27512, Address Decder = OR gate, number f Address Decder cmpnents = 1. Als nte that the synthesized slutin is an utcme f develping nly a single alternative within the synthesis space. 33.3 Example 2 Estimating Behavir fr Partial Designs In the case where a behavir design prperty is included amng the design prperties fr describing design cntext, it is pssible t estimate the behavir f the partially cmpleted designs in the early stages f the synthesis prcess. In Fig. 11, an example f this use f a prbability estimatin functin is illustrated. The indicated input specificatin cnsists f desired RAM size, ROM size, bard speed, and a cnstraint n the desired ROM chip type. By applying the prbability estimatin functin, the prbability f cst being abve $ 150 was estimated t be the highest f all cnsidered cst values, representing the predictin that the cst fr this cnfiguratin will be abve $150. 33.4 Example 3 Recgnizing Incnsistent Specificatins If we estimate the prbabilities f all prperties (including specificatin prperties) based n all ther prperties, then it is pssible t determine whether the given specificatins are cnsistent. Fig. 12 illustrates this by shwing hw the utput f the prbability estimatin functin indicates different values fr prperties fr which values are fixed as a part f specificatin (e.g., the cst was set t be in the range $5(>-$60 n the input, but the estimated prbability indicates that this value fr the cst is nt cnsistent with the rest f the specificatins). T elabrate n this example use, cnsider a subpattern f the input frmed by mitting ne specificatin input This is exactly the input that enters the prbability estimatin functin fr this specificatin. The functin then estimates the prbability fr each value f that specificatin f being a part f cnsidered pattern (i.e., design cntext). Therefre, the estimatin functin is perfectly capable f estimating as nnzer thse prbabilities f design specificatin values which are nt set n input (and vice versa, estimating as zer thse prbabilities f design specificatin values which are set n input) since it des nt cnsider the input values fr this design specificatin in its estimatin prcess. The cause fr this difference 15
^0188_AT_INTF_(HG_SRAM_0 GROM_0 S0188_IO_INTF *0188_PROC_0 80188_ADDR_DEC l # ADDRDEC 3 s I I f n Ci ; CLOCK BUS 3/8DEC 16L8 NOTO FL_FL CONSTANT! 0 27512 I RS_LATCHl j RS_LATCH j (2)2764 27256 2732 27128 27010 L-(T)27020 RES SPDT! CONSTANT!! I t BUS CONSTANT CLOCK 1 BUTTON ~IL DS1232 CONSTANT j OR21 0 [J;CMOS_OSC_2OM 2)CMOS_OSC_16M ScMOS OSC 24M 2 4 DESIGN SPECIFICATIONS AND PERFORMANCES 1) Pwer dissipatin 2) Bard area 3) Cst 4) RAM size 5) ROM size 6) Speed - (L)8O1888-801810 801812
Design Specs: Cst: $80-$90 RAM: 8 KB ROM: 512 KB Speed: 10 MHz «rtcin PmiuHic* RAM chin: ROM chip: Addr. Dec: #Addr.Dec: 6264 7164 62256 TC551 2764 27256 27512 27020 OR 1 L8 & 4 Iteratin 1: * O 0 > < < i <» D O Iteratin 2: t 1 C) O O -JiU IP Iteratin 3: c Lr 1D O O i # O ( Iteratin 4: 4 Figure 10 ler Cmpletin f Design fr CGA cntrl- 17
Cst: RAM: 8KB ROM: 512KB Speed: 10 MHz ROM( ;hip: O O 2764' 27256 27512 2702C O Cst: > $15f 9 O 9 Figure 11 Estimating Behavir fr Partial Design between the input value and the estimated prbability f that value is that the selected design prperties have created a design cntext that is incnsistent with the remaining selected prperty value. This incnsistency wuld nt have been encuntered if the decisin making prcess (i.e., prperty assignment) had been cnducted ne decisin at a time. Pst-$50-$60 RAM- 8KB ROM- 512 KB Speed: 10 MHz ROM chip: O O 2764: 27256 27512 2702( O ROM 32 64 O 9 O 128 O 512 O 0 Figure 12 Recgnizing Incnsistent Specificatins fr CGA Cntrller Frm the examples shwn, this apprach based n prbability estimatin culd supply useful infrmatin during the synthesis prcess that can be used t guide the prcess mre directly t the lcatin in frm and behavir space that satisfy the given requirements. The next chapter describes a cnnectinistbased apprach that acquires and represents this prbability estimatin functin. 18
Chapter 4 NETSYN a Cnnectinist Apprach t SKAU This chapter prvides a detailed accunt f NETSYN - a neural netwrk apprach fr synthesis knwledge acquisitin and use (SKAU). First, the basic idea f the apprach is given by mapping a cnnectinist learning system nt the prbability estimatin functin intrduced in the last sectin. Next, sme theretical results relevant t the expected perfrmance f this apprach are presented alng with a discussin f hw this learning apprach fits int a mre general machine learning framewrk. Cnnectinist representatins are then discussed. Finally, a discussin f the architecture and the training f NETSYN is prvided. 4.1 A CONNECTIONIST a psteriri PROBABILITY ESTIMATION As discussed in subsectin 3.2.3, cnstructin f a prbability estimatin functin is a difficult task fr realistic synthesis prblems. This difficulty is acnsequence f tw prperties f realistic synthesis prblems: The dimensinality f the design space may be very high there may be a large number f interacting design prperties that need t be cnsidered simultaneusly in the synthesis prcess. Relatinships between design prperties may be highly cmplex. Our apprach t cnstructing the prbability estimatin functin is illustrated in Fig. 13. We emply feed-frward neural netwrks (discussed belw in mre detail) as a mechanism by which t acquire and represent the prbability estimatin functin. Tw principal benefits f using a neural netwik in the rle f prbability estimatin functin are: The prbability estimatin functin is acquired thrugh inductive learning. We use existing recrds f previus designs t train the neural netwik t estimate the desired prbabilities. The trained netwrk, under assumptins presented belw, estimates Bayesian a psteriri prbabilities, thus allwing us fr reliance the established theretical backgrund f Bayesian learning appraches. 42 THEORETICAL BASIS FOR NETSYN This sectin presents sme results f fundamental research n which the prpsed cnnectinist apprach t cnstructing the prbability estimatin functin fr SKAU are funded. 4.2.1 Cmputatinal Thery Backgrund In the paper by Richard and Lippmann [Richard 91 ], the fllwing result has been presented, rigrusly prven, and experimentally tested: "Fr an M class prblem, Bayesian prbabilities are estimated when the netwrk has ne utput fr each pattern class, desired utputs are 1 fm (ne utput unity crrespnding t the crrect class, all thers zer), and an apprpriate cst functin is used. " 19