Arbued Graph Machng Based Engneerng Drawngs Rereval Rue Lu, Takayuk Baba, and Dak Masumoo Fusu Research and Developmen Cener Co LTD, Beng, PRChna Informaon Technology Meda Labs Fusu Laboraores LTD, Kawasak, Japan rlu@frdcfusucom Absrac Ths paper presens a mehod for engneerng drawngs rereval by her shape appearances In hs mehod, an engneerng drawng s represened by an arbued graph, where each node corresponds o a meanngful prmve exraced from he orgnal drawng mage Ths represenaon, whch characerzes he prmves as well as her spaal relaonshps by graph nodes arbues and edges arbues respecvely, provdes a global vson of he drawngs Thus, he rereval problem can be formulaed as one of arbued graph machng, whch s realzed by mean feld heory n hs paper The effecveness of hs mehod s verfed by expermens Keywords: Engneerng drawngs rereval, Arbued graph, Mean feld heory Inroducon On-lne manenance of large volume of documens, such as engneerng drawngs, has become a maor research area recenly due o he ever ncreasng rae a whch hese documens are generaed n many applcaon felds [, ] I s very common a hng for desgners and draf people o refer o prevous drawng documens o ge some nspraon or soluons already acheved However, rerevng hese documens generally s a slow and edous work, whch requres an exhausve examnaon of he whole engneerng drawng daabase To faclae such rereval, exual conen such as keywords has been wdely used Whle hs nformaon s helpful n rereval, s, however, a heavy work o generae such descrpons manually, besdes hs, several keywords only are always ncapable of descrbng he rue conen n a drawng Techncal drawngs rereval, or any oher ype of machng problem, can be seen as a correspondence calculaon process Wheher a daabase drawng s rereved or no s hus deermned by he correspondence value beween he query drawng and hs daabase drawng For hs purpose, he echncal drawngs are frs represened by arbued graphs, where graph nodes correspond o meanngful prmves exraced from he orgnal drawng mage, such as lnes and curves, whle he spaal relaonshps beween hese prmves are descrbed by graph edges Nex, he graph nodes n query drawng are regarded as a se of labels o label hose nodes n a daabase drawng so as o deermne he correspondence beween hem Wh hs manner, boh he daabase drawngs whch are smlar o he query one and hose whch nclude a smlar par wh he query one can be obaned To generae arbued graphs from engneerng drawng mages, he orgnal raser mages should be frsly convered o vecor form, say, vecorzaon As a prelmnary S Marna and A Dengel (Eds): DAS 004, LNCS 363, pp 378 388, 004 Sprnger-Verlag Berln Hedelberg 004
Arbued Graph Machng Based Engneerng Drawngs Rereval 379 sep of documen analyss and processng, many vecorzaon echnques have been developed n varous domans [3-5] In hs paper, s assumed ha he afflaed nformaon such as annoaon ex has been removed, and all he curves n he drawng mages have been convered no one-pxel-wdh by some hnnng algorhm Frsly, he drawng mage s decomposed no rough prmves based on he nersecon pons Then, a merge-spl process s used o make hem more meanngful The prmves obaned by above process are reaed as nodes o buld arbued graph Dfferen from Beno s N-Neares neghbor mehod [6], he Delaunay essellaon sraegy s used o generae he srucural descrpon of engneerng drawngs Wh hs manner, he spaal relaonshps beween hese prmves are represened more naurally and meanngfully By above process, he conen and he srucure of engneerng drawngs are represened by means of arbued graphs Ths enables us o nerpre he rereval problem as one of machng wo arbued graphs In recen years, some mehods have been proposed o solve graph machng problem [7-] Based on he compromse beween speed and performance, mean feld heory s adoped n hs paper o solve graph machng problem The remanng of hs paper s arranged as follows: In secon, we descrbe he engneerng drawng decomposon process o dvde he drawng mage no meanngful prmves How o consruc graphs from hese prmves, as well as arbues defnon, s nroduced n secon 3 In secon 4, graph machng algorhm s oulned A las, he expermens as well as some dscusson are gven n secon 5 Prmves Exracon Engneerng drawngs are manly conssed of basc prmves, whch are assembled ogeher by specfc spaal dsrbuon and srucural relaonshp no cognvely meanngful obecs Ths hns us ha a srucural descrpon can be employed o well represen he conen of an engneerng drawng In hs paper, a drawng mage s frsly decomposed no rough prmves, whch are hen refned by a merge-spl process no meanngful form Engneerng Drawng Image Decomposon As menoned n he nroducon, s assumed ha some pre-processng has been appled o orgnal drawng mages o remove he afflaed nformaon such as annoaon ex, make hem bnary and composed of one-pxel-wdh curves only For hs knd of mage, a smple mehod s desgned o decompose no rough prmves: Rough prmves exracon s mplemened by followng sraegy: ) Calculae he number of 8-conneced neghbors for each pxel ) Lnk hese pxels ogeher, whch are 8-conneced and have no more han wo neghbor pxels, o form a rough prmve By hs process, a curve maybe dvded no several fracons, whch should be merged by some measures o form a meanngful one Followng, a merge-spl process wll be used o re-arrange hese rough prmves no meanng form
380 Rue Lu, Takayuk Baba, and Dak Masumoo Merge The obecve of merge s o recover orgnal drawng elemens, such as curves, sragh lnes, from he obaned rough prmves As shown n fgure where hree curves nersec ogeher, 6 rough prmves are obaned from above decomposon process, s hoped ha he orgnal hree curves (lnes) can be recovered by hs mergng process y c c 4 c 6 c 5 c c 3 x Fg Illusraon of mergng operaon I s a naural hng o adop angen drecon and spaal dsance as he bass of mergng operaon If wo rough prmves are collnear and close o each oher a her end pons, here exss possbly for hese wo prmves o be merged ogeher Besdes hs, o ge a reasonable resul, all he rough prmves near he nersecon mus be consdered smulaneously when deermnng he merge operaon Based on above dea, followng algorhm s desgned o deermne mergng operaons ¾ Creron For any par of rough prmves, followng crera mus be sasfed f hey are o be merged: ) Merge operaon can only be performed a he end pons rough prmves ) Exclusvy Mergng s allowed a mos once a an end pon of a rough prmve 3) Collneary and space gap Only he angen drecon and he spaal dfference beween wo rough prmves end pons s small han a hreshold, can hey be merged a hese wo end pons Above crera rule ou mos of rough prmves when consderng possble mergng for one rough prmve However, several canddaes are sll survved, from whch he mos reasonable should be deermned For example, n fgure, segmens c 4 o c 6 are ruled ou from c s canddaes due o collneary creron, n oher words, hey can be merged wh c, however, boh c and c 3 are sll possble o be merged wh c In order for he exclusvy creron o be sasfed, he mos reasonable rough prmve, c n hs example, should be seleced from he canddaes
Arbued Graph Machng Based Engneerng Drawngs Rereval 38 The mergng operaon orderly raverses all rough prmves and processes each of he wo end pons of every rough prmve respecvely Assume c he rough prmve beng processed, he seps of mergng operaon are as follows: () Selec an end pon of c and denoe as a () Exclusvy checkng Check wheher c has been merged a he end pon a, f so go o sep (7), oherwse go on he nex sep (3) Oban he mergng canddaes se of c wh space gap and collneary rules Check oher rough prmves oupued from he mage decomposon par, record he rough prmves and her correspondng end pons ha mee he followng hese rules Denoe c m, m =,, M as he canddae se, d m, m =,, M he angen drecon dfference beween he correspondng end pons of hese rough prmves and end pon a of c m (4) Calculae he mnmum value of d, ˆ m d = mn{ d, m =,, M} (5) Se he olerance value and furher narrow he range of canddae se: m m m c = c d < dˆ m ˆ olerance, ĉ represens he new canddae se { } + m m (6) If he sze of ĉ equals o, say, here s only one elemen n se ĉ, denoed as ĉ Regard ĉ as he curren rough prmve and calculae ĉ s canddae se a s correspondng end pon (recorded n sep (3)) wh seps (3) o (5) If he canddae se of ĉ only conans rough prmve c and he correspondng end pon of c s a, merge hem ogeher a ĉ s correspondng end pon and end pon a of c (7) Denoe he oher end pon of c as b, repea above seps ()-(6) smlar o end pon a (8) Check wheher all rough prmves have been raversed, f no, selec he nex rough prmve as curren one and go o sep (); oherwse go o sep (9) (9) Check wheher any merge operaon occurred durng hs raverse, f no, fnsh he program, oherwse repea seps ()-(8) o perform nex raverse Unlke connuous curve, s nearly mpossble o ge he rue angen drecon value of dscree curve Therefore, some errors wll be resuled f only a smple mnmum drecon dfference sraegy s adoped To cope wh such cases, a olerance value s nroduced n above mehod o preven he loss of rue rough prmve from canddae se 3 Spl In he rough prmves obaned from above process, here may exs sharp bends Here, a spl process s adoped o dvde a prmve wh sharp bends no several smooh ones
38 Rue Lu, Takayuk Baba, and Dak Masumoo Based on he curvaure values, he sgnfcan bendng pons n each prmve can be found, whch are hen used o spl hs prmve no several smooh ones ) Bendng pons deecon Le he prmve pxel and s curvaure be c and k, c s deemed as bendng pons f followng condons are sasfed: (a) k s local maxmum(mnmum) n a k cenered wndow of wdh w (b) k > α * k max, where k max s he maxmum curvaure value (c) k > β, where β s a predefned posve hreshold ) Remove a bendng pon f s no obvous Le p and p be wo consecuve bendng pons, c, c, cavg he curvaures correspondng o p, p and he average curvaure beween hese wo pons respecvely If: c avg > δ *( c + c ) /, where δ s a predefned hreshold Replace p and p wh her average value as he new bendng pon 3 Arbued Graph Consrucon By prmve exracon process, an engneerng drawng has been decomposed no basc prmves, such as curves, sragh lnes, ec To ge a srucural represenaon of he conen of a engneerng drawng, hese prmves as well as her relaonshps mus be well descrbed In hs paper, he arbued graph s adoped o realze hs purpose, where a graph node represens a prmve whle he graph edges ndcae he relaonshp beween hese prmves 3 Graph Consrucon Delaunay essellaon sraegy s a naural way for graph consrucon from engneerng drawng In radonal Delaunay mehod, each prmve (curves, lnes, ec) s represened only by a pon, such as mddle pon, as he npu o graph consrucon However, a long curve (lne) n he orgnal mage may be near o several oher prmves eher a s mddle par or a he end pars due o s long prolongaon, herefore, a connecon should be bul from hs prmve o each of hese nearby prmves If hs long prmve s replaced us by s mddle pon, only he connecons o hese prmves whch are near o hs mddle pon wll be survved, whch leads o connecons loss Besdes hs, some false connecons wll also be generaed due o he loss of prmve exenson nformaon To preven he loss of prmve exenson nformaon, he followng mehod s used n graph consrucon: () Sample a long prmve evenly no mulple pons () Adop hese sampled pons as npu o Delaunay essellaon (3) Graph smplfcaon These graph nodes sampled from he same prmve are merged no one node, wh her correspondng connecons merged ogeher By hs samplng sraegy, a prmve s represened by mulple pons evenly dsrbued along hs curve (lne), herefore, he relaonshp wh oher prmves whch are near o hs prmve eher a he mddle par or a he end pars s reserved
Arbued Graph Machng Based Engneerng Drawngs Rereval 383 3 Arbue Exracon Graph arbues play he key rule n characerzng he conen of a engneerng drawng, where node arbues depc he appearance of he prmves, such as crcular, sragh, or angular, whle edge arbues defne he spaal relaonshp beween hese prmves, such as parallel, nersecan, and so on In hs paper, drecon hsogram s used o descrbe he appearance of each prmve To realze roaon nvarance, Fourer ransform s carred ou wh hs hsogram and he coeffcens of hs ransform are used as node arbue Dreced edge arbues are proposed o hghlgh he relaonshp beween wo sragh lne segmens Le L be he lne under processed, and L s neghbor lne, he relaonshp from L o L s defned by followng componens: ¾Relave angle α, say, he acue angle beween L and L ¾Relave lengh rl : he lengh of L dvded by ha of L ¾Relave poson rp, whch descrbes he poson of he nersecon pon If he nersecon pon of hese wo lnes locaes a L, as shown n fgure and, hs arbue s obaned hrough dvdng he smaller lengh of lne segmen OD and OC by he larger one of hem Oherwse, s calculaed hrough dvdng he lengh of OD by mnus lengh of OC, as shown n fgure 3 ¾Relave dsance rd I s defned as he lengh of he lne segmen connecng he mddle pons of hese wo lnes, dvded by he lengh of L ¾Relave mnmum dsance rmd I s defned as he mnmum dsance beween hese wo lnes, dvded by he lengh of L Fg Defnon of relave poson These arbues, whch are scale and roaon nvaran, can precsely defne he spaal relaonshp beween wo lnes However, our fnal arge s o ge he relave relaon beween wo arbrary curves For hs purpose, he curves are frs approxmaed no sragh lnes, hen, calculae he arbues beween hese wo sragh lnes wh above concep, whch are used as he spaal relaonshp of orgnal curves 4 Graph Machng by Mean Feld Theory Graph machng s n fac a correspondence deermnaon process [7, 8, ], where he nodes n he query graph are seen as a se of labels o label hese nodes n anoher
384 Rue Lu, Takayuk Baba, and Dak Masumoo graph Many algorhms have been proposed n he leraure o solve hs problem [7- ], such as mean feld heory, relaxaon labelng, genec algorhm, ec On he consderaon of speed and performance, mean feld heory s adoped n hs paper for graph machng deermnaon Gven wo graphs g = ( v, e, ), a, b =, A and G = ( V, E, ),, =, I, a a b, where v a and V represen he nodes of hese wo graphs, whle, e a, b and E, represen he dreced edges of hese wo graphs, here, g and G are he absracon graph correspondng o query drawng mage and a daabase drawng mage respecvely Graph machng can be defned as follows: Fnd he mach marx M such ha followng cos funcon s mnmzed: E( M ) = I A I A a= = b= = I M a M b Ds( e A I + a, b, E, ) M a Ds ( va, V ) a= = subec o a, M a = ; M {0, }, where M a means node v a n g s relaed o V n G = a The former par of hs cos funcon represens he cos o mach graph edges, he laer par corresponds o he cos o mach graph nodes ogeher Ds ( ) represens he dsance beween wo edges, whle D s ( ) he dsance beween wo nodes Wh graph nodes and edges arbues descrbed n secon 3, he dsance funcons n above formula are defned as follows: = cos f ea, b or E, s NULL Ds( ea, b, E, ) a a rp rp rl rl rd rd rmd + + ( ) + ( ) + ( rmd e e e ) p / where, { α, rl, rp, rd, rmd},{ α, rl, rp, rd, rmd } are he arbues correspondng o hese wo edges, see secon 3 for deals Above dsance funcon s n fac a weghed average of arbue componens, where he conrbuon of each elemen s normalzed o one, and equal weghs are adoped I should be noed ha he edges are dreced due o he dreced arbues defned n secon 3 cos f va or V s NULL Ds ( va, V ) = Eucldean dsance of he Fourer Coeffcens of hese wo nodes Accordng o mean feld heory, an eraon procedure s obaned o ge he value of M: A I q a = Ds ( v + a, V ) M b Ds( eab, E, ) T b= = M + a = exp( q exp( ) a qa )
Arbued Graph Machng Based Engneerng Drawngs Rereval 385 Fg 3 Illusraon of engneerng drawngs processng (a) Query drawng; (b) Prmves exraced from (a); (c) Consruced graph of (a); (d) A daabase drawng; (e) Prmves exraced from (d); (f) Consruced graph of (d) In machng, he emperaure T s decreased owards 0 as eraon goes The unambguy of M s acheved when T o The convergence s also conrolled d by a quany defned as: Sauraon = M a, A a also called he sauraon of M The eraon s ermnaed f he sauraon value s larger han a predefned value, such as 095 5 Expermens A he begnnng of hs secon, an example s used o demonsrae he whole process descrbed above, see fgure 3 Fgure 3(a) and 3(d) are orgnal query drawng and daabase drawng o be mached; Fgure 3(b) and 3(e) shows he exraced prmves from hese wo drawngs, where a flled recangle s used o denoe a prmve Fgure
386 Rue Lu, Takayuk Baba, and Dak Masumoo 3(c) and 3(f) llusrae he graphs consruced In fgure 3(f) dark flled recangles and hck lnes are used respecvely o denoe he graph nodes and graph edges ha are n correspondence wh hose of he query graph In hs nal expermen, a daabase conanng 00 engneerng drawng mages s se up o evaluae he rereval performance of he proposed mehod These mages, whch correspond o dfferen desgns of washbowls, are manually classfed no hree caegores accordng o her appearances The numbers of mages n each caegory are, and 56 respecvely The mages n eher frs or second caegory are somewha smlar n ha a common par can be found among hem, however, he mages n hrd caegory are dsnc from each oher Fgure 4 shows some represenave mages of each caegory Fg 4 Fg 4 Fg 43 Fg 4 Example drawngs of he daabase (Fg 4 Frs caegory, Fg 4 Second caegory, Fg 43 Thrd caegory) In he expermens, we generae oally 0 query mages o evaluae he rereval performance, wh each 0 of hem beng smlar o he common par of frs and second caegory respecvely, and he average precson and recall values are obaned from hese rals Fgure 5 shows an example of rereval resul, where he former 0 mages are dsplayed In fgure 5, recangles are used o denoe hose pars n he daabase drawngs ha are smlar o query one The average precson and recall value are llusraed n fgure 6 In former rereved mages, whch s he rue number of each caegory, he average precson s abou 56%, and he recall value s more han 80% n he former 50 rereved mages Ths rereval can largely faclae he desgners n searchng a specfc prevous drawng, hus saves he cos n desgnng a new produc All hese expermens are carred ou a a P4-3G compuer, and he average me used for once rereval s 5 seconds, ha s o say, he average me for machng one daabase engneerng drawng wh he query one s abou seconds
Arbued Graph Machng Based Engneerng Drawngs Rereval 387 Fg 5 Fg 5 Fg 5 An example of rereval resul (Fg 5 Query mage, Fg 5 Rereval resul, wh former 0 rereved mages shown) Fg 6 Rereval performance 6 Conclusons In hs paper, we presen a praccal graph machng based mehod for engneerng drawngs rereval By srucural represenaon, he conen as well as he spaal relaonshp of he engneerng drawng are well descrbed Whle he reasonable prmves n an engneerng drawng, say curves, sragh lnes, ellpses and so on, can be exraced by he merge-spl process proposed n hs paper Thus, engneerng drawngs rereval becomes a problem of comparng he srucural represenaons, whch can be mplemened by graph machng In hs paper, s assumed ha all lnes and curves n he obec mage are onepxel wdh n clean background However, as a knd of mehod, he echnque descrbed n hs paper s no lmed o such mages only By some pre-processng, such as de-nose, bar deecon, ec, he rough prmves can be obaned from any knds of engneerng drawngs Afer hs, above mehod can be used o form reasonable prmves, buld srucural descrpons, and machng for rereval
388 Rue Lu, Takayuk Baba, and Dak Masumoo References K Tombre, Analyss of Engneerng Drawngs: Sae of he Ar and Challenges, n K Tombre, AKChhabra, Graphcs Recognon Algorhms and Sysems, 57-64 T Kanungo, R M Haralck, D Dor, Undersandng Engneerng Drawngs: A Survey, Inernaonal Workshop on Graphcs Recognon, pp 9-30, 995 3 R Kasur, S T Bow, W El-Masr, e al, A Sysem for Inerpreaon of Lne Drawngs, IEEE Trans On PAMI, Vol, No 0, pp 978-99, 990 4 Dov Dor, Wenyn Lu, Sparse Pxel Vecorzaon: an algorhm and s Performance Evaluaon, IEEE Trans on PAMI, Vol, No 3, pp 0-5, 999 5 Jqang Song, Feng Su, e al, An Obec-Orened Progressve-Smplfcaon-Based Vecorzaon Sysem for Engneerng Drawngs: Model, Algorhm, and Performance, IEEE Trans on PAMI, Vol 4, No 8, pp 048-060, 00 6 Beno Hue, E R Hancock, Relaonal Obec Recognon from Large Srucural Lbrares, Paern Recognon, Vol 35, pp 895-95, 00 7 W J Chrsmas, J Kler, M Perou, Srucural Machng n Compuaon Vson usng Probablsc Relaxaon, IEEE Trans on P A M I, Vol 7, No 8, pp 749-764, 995 8 Seven Gold, Anand Rangaraan, A Graduaed Assgnmen Algorhm for Graph Machng, IEEE Trans on P A M I, Vol 8, pp 377-388, 996 9 S Z L, H Wang, K L Chan, Energy Mnmzaon and Relaxaon Labelng, Journal of Mahemacal Imagng and Vson, Vol 7, pp 49-6, 997 0 M A Wyk, T S Durran, B J Wyk, A RKHS Inerpolaor-Based Graph Machng Algorhm, IEEE Trans on P A M I, Vol 4, No 7, pp 988-995, 00 A D J Cross, R C Wlson, E R Hancock, Inexac Graph Machng usng Genec Search, Paern Recognon, Vol 30, No 6, pp 953-970, 997