CRAWLING DEEP WEB CONTENT THROUGH QUERY FORMS

Transcription

1 CRAWLING DEEP WEB CONTENT THROUGH QUERY FORMS Jun Lu, Zhaohu Wu, Lu Jang, Qnghua Zheng, Xao Lu MOE KLINNS Lab and SKLMS Lab, X an Jaoong Unversy, X'an 70049, Chna lueen@mal.xu.edu.cn, wzh@su.xu.edu.cn, roadang@26.com, qhzheng@mal.xu.edu.cn, xao_xao@su.xu.edu.cn Keywords: Absrac: Deep Web, Deep Web Surfacng, Mnmum Execuable Paern, Adapve Query Ths paper proposes he concep of Mnmum Execuable Paern (MEP), and hen presens a MEP generaon mehod and a MEP-based Deep Web adapve query mehod. The query mehod exends query nerface from sngle exbox o MEP se, and generaes local-opmal query by choosng a MEP and a eyword vecor of he MEP. Our mehod overcomes he problem of daa slands o a ceran exen whch resuls from defcency of curren mehods. The expermenal resuls on sx real-world Deep Web ses show ha our mehod ouperforms exsng mehods n erms of query capably and applcably. INTRODUCTION There s an enormous amoun of nformaon bured n he Deep Web, and s quany and qualy are far beyond he surface web ha radonal search engnes can reach (Mchael, 200). However, such nformaon canno be obaned hrough sac hml pages bu dynamc pages generaed n response o a query hrough a web form. Due o he enormous volume of Deep Web nformaon and he heerogeney among he query forms, effecve Deep Web crawlng s a complex and dffcul ssue. Deep web crawlng ams o harves daa records as many as possble a an affordable cos (Barbosa, 2004), whose ey problem s how o generae proper queres. Presenly, a seres of researches on Deep Web query has been carred ou, and wo ypes of query mehods, namely pror nowledge-based mehods and non-pror nowledge mehods, have been proposed. The pror nowledge-based query mehods need o consruc he nowledge base beforehand, and generae queres under he gudance of pror nowledge. In (Raghavan, 200) proposed a asspecfc Deep Web crawler and a correspondng query mehod based on Label Value Se (LVS) able; he LVS able as pror nowledge s used for passng values o query forms. (Alvarez, 2007) brough forward a query mehod based on doman defnons whch ncreased he accuracy rae of fllng ou query forms. Such mehods auomae deep crawlng o a grea exen (Barbosa,2005), however, have wo defcences. Frs, hese mehods can only perform well when here s suffcen pror nowledge, whle for he query forms ha have few conrol elemens (such as a sngle ex box), hey may have an unsasfacory performance. Second, each query s submed by fllng ou a whole form, whch reduces he effcency of Deep Web crawlng. The non-pror nowledge mehods are able o overcome he above defcences. These mehods generae new canddae query eywords by analyzng he daa records reurned from he prevous query, and he query process does no rely on pror nowledge. Barbosa e al. frs nroduced he deas, and presened a query selecon mehod whch generaed he nex query usng he mos frequen eywords n he prevous records (Barbosa, 2004). However, queres wh he mos frequen eywords n hand do no ensure ha more new records are reurned from he Deep Web daabase. (Noulas, 2005) proposed a greedy query selecon mehod based on he expeced harves rae. In he mehod, canddae query eywords are generaed from he obaned records, and hen her harves raes are calculaed; he one wh he maxmum expeced harves rae wll be seleced for he nex query. (Wu P, 2006) modeled each web daabase as a dsnc arbue-value graph, and under hs heorecal framewor, he problem of fndng an opmal query selecon was ransferred no fndng a Weghed Mnmum Domnang Se n he

2 correspondng arbued-value graph; accordng o he dea, a greedy ln-based query selecon mehod was proposed o approxmae he opmal soluon. Compared wh he pror nowledge-based mehods, he non-pror nowledge mehods mprove he query capably on Deep Web crawlng. However, hese mehods suffer from he followng hree lmaons: frs, queres are only based on sngle ex box and he canddae eywords are assumed o be sued o he ex box; second, query selecon decson s made solely based on he obaned records, namely myopa esmaon problem (Wu P, 2006); hrd, query selecon lacs suffcen nowledge n he nal perod. The hree problems lm he query capably on Deep Web crawlng, and resul n he phenomenon called daa slands whch means he oal acqured records may consue only a small fracon of he arge daabase. In hs paper, we propose he concep of Mnmum Execuable Paern (MEP) and a MEPbased Deep Web adapve query mehod. The query mehod exends query nerface from sngle exbox o MEP se; performs a query by choosng a MEP and a eyword vecor of he MEP, and generaes he nex query wh he maxmum expeced effcency adapvely hrough he acqured nformaon. Ths mehod no only has he advanages over he nonpror nowledge mehods, bu also has he ably o solve he problem of daa slands by mang he mos of query capably of elemens n MEP se. The expermenal resuls on sx real-world Deep Web ses show ha our mehod ouperforms exsng mehods n erms of query capably and applcably. The res of he paper s organzed as follows: Secon 2 nroduces he concep of MEP. Secon 3 descrbes he algorhm for generang he MEP se of a gven query form. Secon 4, he core of hs paper, sudes he MEP-based adapve query mehod and he convergence of he relaed algorhm. The expermenal resuls are dscussed n Secon 5, and conclusons and fuure wor are offered n he fnal secon. 2 FUNDAMENTAL CONCEPTS Defnon : Query Form. A query form F s a query nerface of Deep Web, whch can be defned as a se of all elemens n. F { e,..., e } n, where e s an elemen of F, such as a checbox, ex box or rado buon. Each elemen e of F has s doman he se of values assocaed wh e. If D whch s D s a fne se, hen e s a fne doman elemen, else e s an nfne doman elemen. Elemens are usually assocaed wh some descrpve ex o help users undersand he semancs of he elemen, namely labels. The expresson label( e ) s used o denoe he label of e ( Raghavan, 200). Defnon 2: Execuable Paern (EP). Gven a query form F { e,..., e } n, { e,..., e } 2 F, m n, { e,..., e } s an execuable paern of F f he Deep Web daabase reurns he correspondng resuls afer he query wh value assgnmens of elemens n { e,..., e } s ssued. Execuable Paern sasfes he followng monooncy properes:. If { e,..., e } s an execuable paern, hen any subse of F ha conans { e,..., e } s also an execuable paern. 2. If { e,..., e } s no an execuable paern, hen any subse of { e,..., e } s no an execuable paern. The resul records reurned based on an execuable paern may also be null. A query based on a non-execuable paern can no perform a successful daabase search, and usually leads o an error repor or swches o anoher page. Defnon 3: Mnmum Execuable Paern (MEP). If { e,..., e } s an execuable paern of query form F { e,..., e } n ( m n), hen {,..., } s a MEP ff any proper subse of e e {,..., } s no an execuable paern. We may e e rewre as MEP( e,..., e ). The eyword vecor v ( v,..., v m ) maes a value assgnmen for MEP( e,..., e ), where v D, =,2,,m. If here s an nfne se D, hen he MEP s called nfne doman MEP, or IMEP for shor; whle f each D s a fne se, hen he MEP s called fne doman MEP (FMEP). All MEP of he query form F consue he MEP se of F whch s denoed as S MEP. Accordng o he monooncy properes of EP, we can draw he followng nference: Inference : An execuable paern { e,..., e } s a MEP ff all s subses of sze m- are no EP.

3 Deep web crawlng ams o rereve daa records from a web daabase hrough erave queres on a gven query form F. In hs process, he MEP se S MEP of he query form F s generaed frsly and hen all subsequen queres are performed based on he MEP se. For hs reason, our sudy focuses on such wo crcal ssues: frs, how o generae he MEP se of a gven query form; second, how o selec a proper MEP and s eywords vecor o harves daa records from web daabase effcenly. 3 MEP SET GENERATION A nave mehod o generae he MEP se s o enumerae all combnaons of elemens n he form F. If he sze of F s n, hen he number of combnaons o be checed oals s abou 2 n. For hs reason, he effcency of enumeraon wll sharply drop when n grows large. Elemens n a form are no ndependen bu always have connecon wh each oher, such as sar cy and desnaon cy n a ce query form. Such elemens always appear smulaneously n he same MEP, and her combnaon s called Condon Paern (CP for shor) (Zhang Z, 2004). By usng CP, he MEP se can be generaed wh greaer granulary han elemen, whch grealy mproves he effcency of MEP se generaon. There have exsed several mehods of generang all CPs of a query form (He B, 2006) (Zhang Z, 2004). For example, Zhang Z. e al. proposed he 2P grammar & bes-effor parser model, by usng whch, a query form can be parsed no a complee parser ree (Zhang Z, 2004), and he CP nodes n hs ree are correspondng wh he CPs of he form. The CP se can be easly generaed hrough fndng all he CP nodes n hs ree. Le he CP se be S CP, and he nal MEP se S MEP s empy. The algorhm MEPGeneraon(S CP, S MEP ) wll generae he MEP se S MEP based on he CP se S CP. In order o faclae he descrpon of he algorhm, we nroduce a funcon defned as: ( A)= A-x x A ( A ). The algorhm s shown n Fg.. MEPGeneraon (S CP, S MEP) Sep : If none of ( Scp ) s EP Add S CP o S MEP; Reurn; Sep 2: Else for each Scp ( Scp ) ha s EP MEPGeneraon ( Scp, S MEP); Sep 3: Reurn S MEP. Fgure : Algorhm for generang he se of MEP. Accordng o he monooncy properes of EP, he algorhm uses dvde-and-conquer o generae S MEP n a recursve way. Consderng ha a sngle CP s also a MEP n mos cases, we can move all CPs ha are EPs from SCP o SMEP before execung he above algorhm, whch can accelerae he generaon of MEP se. 4 QUERY BASED ON MEP Once he MEP se S MEP of a gven form s obaned, he nex as s o selec suable MEPs and he correspondng eywords vecors o perform erave queres on he arge Deep Web daabase. In hs secon, we formulae he problem of MEP-based query, and on he bass of ha, propose a MEPbased adapve query algorhm as well as he esmaon mehods of wo ey parameers. Fnally, we gve he convergence analyss of our query algorhm. 4. Formal Descrpon of MEP-based Query Le q (v, mep ) be he h query on he arge Deep Web se, and q (v, mep ) s mplemened usng he MEP mep and s correspondng eyword vecor v. Here, mep S MEP ncludes m elemens and v=(v, v 2,, v m ) s a m-dmensonal vecor accordngly. q (v, mep ) can be abbrevaed as q. Gven a query q, P (q ) s used o denoe he fracon of daa records reurned form he web daabase hrough he query q. P(q q ) represens he fracon of he common records ha are reurned from q, q 2,... and q. Smlarly, we use P(q q ) o represen he fracon of he unque records ha are reurned from q or q 2,...or q. Addonally, P new (q ) sands for he fracon of he new records ha have no been rereved from prevous queres from q. P new (q ) s compued from Equaon(): P new(q ) =P(q.. q ) - P(q.. q -) () In order o measure he resource consumpon of ssued query, we nroduce cos(q ) o represen cos of ssung he query q. Dependng on he scenaro, he cos can be measured eher n me, newor bandwdh, or he number of neracons wh he se. In hs sudy, we measure cos n erms of consumed me, as follows: cos(q (v,mep ))= q(mep )+ rp(q (v,mep ))+ dp new(q (v,mep )) (2)

4 In he above equaon, query cos consss of hree facors, q (mep ) s fxed par of he query cos, whch ncludes he query ransmng me and query processng me by Deep Web daabase; r s proporonal o he average me of handlng a resul record; whle d s proporonal o he average me of downloadng a new resul record. Wh he above noaons, we can formalze MEP-based query on Deep Web as follows: under he consran n cos( q ) T, fnd a sequence of queres q, q n ha maxmze P new (q q n ). Here, T s he maxmum cos consran. 4.2 Adapve Query Algorhm The sequence of queres q, q n ha maxmze P new (q q n ) s called global-opmal query se. Even f all resuls of canddae queres are clearly nown, fndng he global-opmal query se s an NP-Hard problem. An effcen algorhm o solve hs problem n polynomal me has ye o be found. For hs reason, we presen an adapve query algorhm based on MEP ha ams a fndng a localopmal query se o approxmae he global opmal query se. A query s he local-opmal f has he maxmum value of Effcency. Effcency s defned as follows: Defnon 4: Effcency. Effcency(q ) s used o quanfy new queres reurned from q per un cos: Effcency(q (v,mep))=p new(q (v,mep))/cos(q (v,mep)) (3) By observng equaon (2) and (3), can be seen ha o compue Effcen(q ) s acually o compue P new (q ). Usng chan rules, P new (q ) can be rewren as: all canddae eyword vecors of gven mep. The value depends on he capably of obanng new records of seleced eyword vecor. The esmaon of P new (q(mep )) and P new (q (v mep )) s he ey o fnd he local-opmal query n our adapve query algorhm. The process of esmaon s based on currenly avalable records. In he early sage of Deep Web crawlng, as feedbac records are relavely scarce, he selecon of eyword vecors s lac of bass and nevably leads o he problem of daa slands. To address he problem, we nroduce an LVS able n our algorhm. The algorhm s dvded no wo phases. When he number of queres s less han a ceran hreshold s,.e. n he daa accumulaon phase, he algorhm uses Probablsc Ranng Funcon (Raghavan, 200), an LVS value assgnmen mehod, o selec he mos promsng v and obans daa from he arge Deep Web daabase. Once he number of queres s greaer han or equal o s, he algorhm swches o he predcon phase; n hs phase, analyzes currenly avalable daa and esmaes he mos promsng query. The algorhm fnally oupus he nex local-opmal query. The algorhm flow s shown n Fg.2. In he followng, he mehods for compung P new (q(mep )) and P new (q (v mep ))are dscussed n deals Predcon for P new (q(mep )) In pracce, we use P new (q (mep )) o denoe he predced value of P new (q (mep )) a he h query,and nroduce wo mehods o accomplsh he as. P new(q (v,mep ))=P new(q(mep )) P new(q (v mep )) (4) In equaon (4), he value of P new (q ) s deermned by a on decson of boh P new (q(mep )) and P new (q (v mep )). P new (q(mep )) s also called harves rae of he mep (.e. he capably of obanng new records), whch s ndependen of choce of eyword vecors, bu depends on he paern mep self. For example, assumng ha a Deep Web se abou academc paper has MEP se S MEP = {mep(keywords),mep(absrac)}, s obvous ha he harves rae of Absrac paern s greaer han ha of Keywords, snce eywords are usually ncluded n absrac. P new (q (v mep )) represens he condonal harves rae of v among Fgure 2: Adapve query algorhm based on MEP. ) Connuous predcon: he curren harves rae of a MEP oally depends on he harves rae of

5 he laes ssued query by he MEP, namely: P Pnew ( q ( v, mep )) q - use mep Z P( q ( v, mep )) ( q ( mep )) Pnew ( q ( mep )) q - no use mep Z new where Z s a normalzaon facor. Assume q - new uses mep, hen Z Pnew ( q ( mep )). (5) P ( q ( v, mep )) P( q ( v, mep )) Connuous predcon mehod performs well on FMEP snce here s no sgnfcan varaon n harves raes among dfferen eyword vecors n mos cases, whle may no be effecve for IMEP. 2) Weghed predcon: he curren harves rae of a MEP depends on all s prevous harves raes of ssued query by he MEP, namely: Pnew ( q ( v, mep )) vpnew ( q ( mep )) ( v) P( q ( v, mep )) q - use ( ( )) mep P new q mep Z Pnew ( q ( mep )) q- no use mep Z where v s a wegh o measure he dependence of he curren harves rae on he pas experence. Expermens show ha v ranges from 0.6 o 0.8. Z s a normalzaon facor; assume ha q - use mep, hen, P ( q ( v, mep )) Z P ( q ( mep )) vp ( q ( mep )) ( v)( new new new P( q ( v, mep )) Weghed predcon mehod s he generalzed form of connues predcon mehod, and wors well on boh FMEP and IMEP. Furhermore, he mehod s no sensve o he nal value of each paern n equaon (6). So we adop he weghed predcon mehod n our algorhm o predc P new (q(mep )) Esmaon for P new (q (v mep )) The am of esmang he P new (q (v mep )) s o denfy he mos promsng eyword vecor of he gven mep. Accordng o equaon (), we have (6) whch conan v of mep. The value of P(q (v mep ) (q q - )) can be calculaed hrough currenly avalable daa, whereas P(q (v mep )) needs o be esmaed. The followng par focuses on he calculaon of hese wo values. In order o calculae P(q (v mep ) (q q - )), we nroduce he noon of SampleDF(w),whch means he documen frequency of observed word w n sample croups {d,...,d s } (Iperos, 2002). SampleDF(w) = s b, b = f w n d, 0 oherwse. Unforunaely, SampleDF(w) canno be appled o our wor, for only focuses on sngle eyword and gnores he compably beween v and mep. To denfy he conrbuon of he documen frequency of m-dmensonal eyword vecor on a gven parcular paern, we nroduce cos<vx,mepx>, where vx s he correspondng Boolean vecor of vx n d, and smlarly mepx s he Boolean vecor of mep. Assume ha cosne value of null vecor and any vecor s 0. Here, we defne he documen frequency of v on a gven mep n sample croups {d,...,d s } as SampleDF(v mep), whch s calculaed as follows: s vx mepx SampleDF ( v mep) cos( vx,mepx) g (8) mepx vx In equaon (8), mepx = (mepx,..., mepx m-, mepx m ). Unle IMEP, FMEP eyword vecor can be obaned by parsng he form F. In order o elmnae he nfluence on predcng eyword vecor of IMEP, we assgn zero o mepx of fne elemen; vx = (vx,...vx n), for gven vx and mep, he vx generaon algorhm s shown n fg.3, where label(v ) s he eyword label n d, label(e ) s he label of he h elemen of he mep. The reason of label(v ) = null n Sep 4 s ha eher he label of v s absen or he label can be exraced. To solve hs problem, he algorhm borrows he dea from (Raghavan, 200) o fnd he mos relevan label of he mssng label. In Sep 6, M v (x) s he fuzzy value of x n LVS. Accordng o fg.3, he SampleDF(v mep) of a gven FMEP s 0. s P new(q (v mep )) =P(q (v mep ))-P(q (v mep ) (q q -)) (7) In equaon (7), P(q (v mep )) represens he condonal capacy of obanng daa records of v among all canddae eyword vecors of mep. P(q (v mep ) (q q - )) represens he fracon of prevously downloaded unque records

6 Sep : = 0; Sep 2: ++; If > dmenson of mep hen reurn vx ; Sep 3: If mepx = 0 hen vx = 0; Goo Sep 2; Sep 4: If label(v ) = null n d ;; Goo Sep 6; Sep 5: If label(v ) = label(e ) hen vx = ; Goo Sep 2;Else hen vx = 0; Goo Sep 2; Sep 6: S max=0; //Calculae he mos relevan label n LVS For each (L,V) enry n LVS S = M v(v ); If(S > S max) label(v )= L; Sep 7: Goo Sep 5; Fgure 3: Algorhm for generang he vecor vx. Obvously, when he gven mep s FMEP, all canddae eyword vecors of mep can be obaned by parsng he form F. Assume ha each paern s capable o oban all records from he arge Deep Web daabase whn lmed me of queres, we can smply use average value o esmae P(q (v mep )): P(q (v mep )) = n D where D s he doman of elemen e n mep. Furhermore, suppose he mep consss of p rado buons (or combo boxes) e,...e, e p, and q checboxes e p+,...e p+, e p+q, e has m buons ( D = m ) and e p+ has r boxes ( D p+ = 2 r ). Snce he doman sze of a checbox s an exponenal funcon of he number of s chec buons, s dffcul o cover he full doman whn accepable me. In order o mprove effcency of queres wh checboxes, we replace he doman of a checbox o s subse wh sze of h (2 h r+2). Expermenal resuls ndcae ha he subse formed by empy se, full se and ses of an ndvdual checbox can oban abou 90% of oal records n mos cases. Based on he above consderaons, he value of P(q (v mep )) can be furher opmzed as: P(q (v mep )) = pq p hm p (9) (0) When he gven mep s IMEP and consss of p rado buons and q checboxes, he value of P(q (v mep )) s Compared wh equaon (0), equaon () s more general. When he gven mep s FMEP, f =, whch means ha he query scope s he enre daabase; when he gven mep s IMEP, f s he fracon of records ncludng eywords of nfne doman elemen on he gven mep. Here, we explo Zpf-Mandelbro law o esmae he fracon. Alernave mehod such as Posson Esmaor (Kenneh, 995) may also be exploed. Zpf was he frs o observe ha he word-frequency dsrbuon followed a power law, whch was laer refned by Mandelbro. Mandelbro observed a relaonshp beween he ran r and he frequency f of a word n a ex daabase (Mandelbro, 988): f = ( r ), where, and are parameers. (2) By analyzng dozens of expermenal resuls, we fnd v also follows Zpf-Mandelbro law. If he ran value of SampleDF(v mep) s nown, he value of f can also be esmaed usng equaon (2). Then equaon () can be rewren as follows: ( r ) P(q (v mep )) =. pq p h m p (3) Once SampleDF(v mep) and P(q (v mep )) are calculaed, P new (q (v mep )) can be predced as follows: a canddae query s formulaed as a 4-uple (MEP, Keyword Vecor, SampleDF, AcualDF), where AcualDF ndcaes he acual number of records reurned by he ssued query; all he 4-uples of canddae queres consruc he query canddae pool. Our predcng algorhm manages he query canddae pool based on he acqured daa records of he las query, sors all 4-uples n he pool accordng o sampledf, and hen fs equaon (2) wh he ran and acualdf /S (he sze of he arge daabase)of all uples ha mee sampledf*acualdf 0. Subsequenly, P new (q (v mep )) of all he uples whch mee acualdf =0 can be calculaed accordng o equaon (). The dealed algorhm s shown n Fg.4. P(q (v mep )) = pq f p hm p ()

7 Sep : Creae a Tuple Se(Keyword Vecor, MEP, SampleDF, AcualDF); Sep 2: Fech a new downloaded documen d ; If no more documen hen Goo Sep 7; Sep 3: If d s no a new documen Goo Sep 2; Sep 4: Fnd ou all (Keyword Vecor, MEP) par (v,mep) and s correspondng sampledf n d ; Sep 5: For each (v,mep) If (v,mep) par exss n Tuple Se Then add sampledf o SampleDF of ha uple; Else Then add a new uple (v,mep,sampledf,0) ; Sep 6: Goo Sep 2; Sep 7: Sor all 4-uples n descendng order of SampleDF; Sep 8: For each uple where sampledf * acualdf 0 Smulae n ( r ) usng uple ran and acualdf /S; Sep 9: For each uple where AcualDF = 0 If(mep s a FMEP) hen f = ; Else hen f = ( r ) ; f P new (q (v mep )) = - sampledf / S; pq p h m p Fgure 4: Algorhm for predcng P new (q (v mep )). 4.3 Convergence Analyss When o sop queryng he web daabase s a dffcul ssue, especally when he sze of arge daabase s unnown. To mae a decson on When o sop requres he nowledge of he relaonshp beween fracon of new records and query cos. Assume ha he sze of Deep Web daabase s S, m s he fracon of records reurned by he h query and a s he cumulaed fracon of new records reurned by prevous queres. We have a + = a + m * p, where p s he fracon of new records n m. To smplfy he calculaon, we suppose ha m s fxed and p can be approxmaed as S a / S, where S a s he number of records ha are no rereved.a + = a + m * p can be rewren as a a m( S a / S), from whch equaon a / ( / ) S m S s derved. The comparson beween deal and praccal crawlng fracon curve s carred ou on dozens of Deep Web daabases. Tae he Babe Raccoon (hp://vod.xu.edu.cn) as an example (see Fg.5). On he y-axs, he daabase coverage s ploed, whle he x-axs represens he query number. Fgure 5: Comparson beween deal and praccal crawlng fracon. From boh deal and praccal fracon curve, we can fnd ha when he fracon of he oal records obaned s approachng 90%, he harves rae of he subsequen queres s relavely small and even equals o 0 n a perod of me. We call he phenomenon Crawler Bolenec. Deep web crawlng s a me-consumng as requrng a sgnfcan amoun of newor resources. I would be a wase of boh me and newor bandwdh f he crawler doesn sop crawlng when comes no Crawler Bolenec phase. For example, soppng crawlng a saves approxmaely 50% cos compared o soppng a 2. Assume W s an obaned daa wndow wh he sze of ws, for a query q, f <ws, W q... q,f ws, W q ws... q, cos max s he maxmum avalable resource ha a crawler has, and s a very small posve number, we presen an sraegy for soppng crawlng: afer submng q, f cos ( q ) cosmax W s false, hen sop crawlng a q. The value of ws depends on cos max. The greaer he value of cos max, he bgger ws can reach. When ws s deermned, becomes he crcal facor n he sraegy for soppng crawlng. If s oo small, he soppng me wll be prolonged, whle more daa records may be obaned. To he conrary, he crawler wll sop a an earler me and reduce quany of records rereved. Therefore, he value of depends on he mporance of he resource cos and he crawlng daa.

8 coverage of deep web daabase coverage of deep web daabase 5 EXPERIMENTS To evaluae he performance of our MEP-based adapve query mehod, we performed expermens on 6 real Deep Web ses and compared our mehod wh he represenave mehods of boh non-pror nowledge and pror nowledge-based mehods,.e. adapve query mehod based on sngle IDE (Infne Doman Elemen) (Noulas, 2005) and classcal LVS mehod (Raghavan, 200). 5. Effecveness of MEP-based Adapve Query We es our MEP adapve query mehod on 6 real Deep Web ses, and he dealed expermenal resuls are shown n Table. Resuls show ha our mehod s que effecve for crawlng Deep Web. Table : Web ses and her expermenal resuls. URL(hp) Doman Sze/Harves/query NO. Paper 380/380/43 cc.c.ac.cn Paper 2523/2523/3 Paper 424/424/6 Paper 743,444/730,000/399 vod.xu.edu.cn Move 700/679/3 musc.xu.edu.cn Musc 54,000/46,967/ Performance Comparson We compare he crawlng performance of our MEPbased adapve query (MEP or MEP adapve for shor) mehod wh he adapve query mehod n (Noulas, 2005). The expermenal resul s shown n Fg. 6, where he x-axs represens he query numbers and y-axs plos he daabase coverage. We fnd ha f he query form conans a FMEP, our mehod shows dsnc advanage over he adapve query mehod based on sngle IDE (see Table ). In order o evaluae performance of our mehod on query forms only conanng IMEPs, we conduc oher wo expermens on Babe Raccoon and Blue Lous. Fg. 6(a) shows he expermenal resuls on Babe Raccoon, n whch IDE, IDE2, IDE3 represen he crawlng curve of he mehod n (Noulas, 2005) on elemens move name, acor, drecor respecvely and MEP denoes he curve of our mehod on hree IMEPs. The expermenal resuls on Blue Lous are ploed n Fg. 6(b), n whch IDE, IDE2, IDE3 represen he crawlng curve of he mehod n (Noulas, 2005) on elemen auhor, publsher, le respecvely and MEP denoes he curve of our mehod on hese IMEPs. Fg. 6(a) and 6(b) ndcae ha our mehod s more effcen han he adapve query mehod n (Noulas, 2005) on query forms only conanng IMEPs. The MEP adapve mehod s based on mulpaern, and usually here are several MEPs o be seleced for each query. If each query aes he same MEP, he mehod degeneraes no sngle IDE-based. From hs pon of vew, he adapve query mehod based on sngle nfne elemen n (Noulas, 2005) can be regarded as a specal case of he MEP-based adapve mehod. Sngle paern always brngs localy of he canddae eywords, whle mulpaern can mae full use of each paern s query capacy, and brea hrough he localy of he eywords selecon whch leads o he problem of daa slands MEP IDE IDE2 IDE query number Fgure 6(a): Expermen on Baby Raccoon MEP IDE IDE2 IDE query number Fgure 6(b): Expermen on Paper Open. In order o compare he crawlng performance beween he MEP adapve mehod and he pror nowledge-based mehods, we conduc expermens on he se of Blue Lous. In addon, we fnd ha he eyword wh hgh value of P(q ) n our mehod s useful for he LVS mehod. If we updae such eyword s Mv value wh s correspondng P(q ) n he LVS able and hen perform he LVS mehod (Raghavan, 200), can remarably mprove

9 coverage of deep web daabase crawlng effcency. We call he enhanced LVS mehod. Fg. 7 shows he expermenal resuls MEP Adpve Enhanced LVS Classcal LVS ACKNOWLEDGEMENTS The research was suppored by he Naonal Hgh- Tech R&D Program of Chna under Gran No.2008AA0Z3, he Naonal Scence Foundaon of Chna under Gran Nos , , , he Naonal Key Technologes R&D Program of Chna under Gran Nos. 2006BAKB02,2006BAJ07B query number Fgure 7: Comparson o Classcal LVS. Accordng o Fg. 7, we can gan he effcency order of he hree mehods: he MEP adapve mehod > enhanced LVS mehod > classcal LVS mehod. Snce he enhanced LVS mehod uses he updaed LVS able whch mproves he veracy of he pror nowledge n LVS able, conrbues o beer crawlng effcency. However, because of he nheren defcences of he pror nowledge-based mehods, s sll less effcen han he MEP adapve mehod. 6 CONCLUSIONS Due o he heerogeney among he query forms, effecve Deep Web crawlng s a dffcul ssue. In hs paper, we propose he concep of MEP and a MEP-based Deep Web adapve query mehod. The proposed mehod, le he pror nowledge-based mehods, has he ably o handle a varey of web forms. Moreover, he mehod has an advanage of good crawlng effcency over he non-pror nowledge mehods, and can overcome he problem of daa slands o a ceran exen. Performance comparsons wh he relaed mehod valdae he beer query capably of our mehod. Alhough our mehod can be effecve for mos Deep Web ses, has he followng lmaons: frs, may no perform well for some Deep Web ses ha lm he sze of he resul se; second, canno suppor Boolean logc operaors (e.g. AND, OR, NOT) n queres. We wll focus on hese ssues n our fuure wor. REFERENCES Alvarez M., Raposo J., Pan, A., Cacheda, F., Bellas, F., Carnero, V, (2007). DeepBo: A Focused Crawler for Accessng Hdden Web Conen, In Proceedngs of DEECS2007. San Dego CA, pages Barbosa L. and Frere J. (2005). Searchng for Hdden Web Daabases. In Proceedngs of WEBDB2005, Balmore MD, pages.-6. Barbosa L. and Frere J. (2004). Sphonng Hdden-Web Daa hrough Keyword-Based Inerfaces. In Proceedngs of SBBD2004, Brasla Brazl, pages He B., Chang K. C. C (2006). Auomac Complex Schema Machng across Web Query Inerfaces: A Correlaon Mnng Approach. ACM Transacons on Daabase Sysems, vol. 3, pages.-45. Iperos P., Gravano L. (2002). Dsrbued Search over he Hdden Web: Herarchcal Daabase Samplng and Selecon. In Proceedngs of VLDB2002, Hong Kong Chna, Augus, pages. -2. Kenneh W. Church and Wllam. (995). A. Gale. Posson Mxures. Naural Language Engneerng, vol., pages Mandelbro B. B. (988). The Fracal Geomery of Naure. New Yor: W. H. Freeman and Company. Mchael K. Bergman. (200). The Deep Web: Surfacng Hdden Value. The Journal of Elecronc Publshng from he Unversy of Mchgan, vol. 7, pages 3-2. Noulas A., Zerfos P., Cho J. Downloadng Texual Hdden Web Conen hrough Keyword Queres. In Proceedngs of JCDL2005, Denver CO, June 2005, pages Raghavan S. and Garca-Molna H. (200). Crawlng he Hdden Web. In Proceedngs of VLDB200, Rome Ialy, pages Wu P., Wen J. R., Lu H., Ma W. Y. (2006). Query Selecon Technques for Effcen Crawlng of Srucured Web Source. In Proceedngs of ICDE2006, Alana GA, pages Zhang Z., He B., Chang K. C. C. (2006). Undersandng Web Query Inerfaces: Bes Effor Parsng wh Hdden Synax. In Proceedngs of he ACM SIGMOD2004, Pars France, pages 07-8.