8.1. Review of experiments

Transcription

1 VIII E V A L U A T I N VIII 1 E P E R I M E N T S Review f experiments As nted earlier,, the majr evaluatin tests f manual dexg systems which have been carried ut the last decade prvide the backgrund fr the evaluatin f autmatic dexg0 Thus the Cmparative Systems Labratry (Saracevic 1968^ 1971), (Cleverdn 1966), Inspec (Aitchisn 1970) and ISILT (Keen 1972, 1973) tests all fund simple manual keywrd dexg very cmpetitive with ther languages* (Cllectin details fr all the tests referred t this sectin are summarised Table 1.) In particular, really elabrately cntrlled r structured languages typically perfrmed less well, thugh there is sme evidence that relatively simple cntrlled languages have a slight perfrmance edge. There is als sme evidence that manual thesauri can be imprved by takg accunt f vcabulary distributin prperties: see, fr example, Saltn 1968a. Unfrtunately these evaluatin tests were dne n different cllectins, and there was n cmparisn between cllectins by dividual prjects. The experiments were als cnfed t small cllectins,, Tests f peratinal systems like that cnducted by Lancaster 1968a may well vlve large cllectins, but d nt prvide the cmparative frmatin we need here. Evaluatin tests with autmatic dexg techniques tend t suffer frm the same defects as these manual nes. Many f the Smart experiments, fr example, have been carried ut n the 35x82 ADI cllectin*, Hwever tests with several different cllectins have been cnducted by the Smart prject, and by Sparck Jnes, and the rather variable results btaed serve as useful remders that the relative perfrmance f different devices may vary ver cllectins * Frtunately, the cmmn use f particular cllectins is creasg, which makes it mre ratinal t attempt t relate different prject results t ne anther0 The x cllectin has been widely used, and there are several thers which have been explited by mre than ne prject. It shuld, hwever, be nted that different versins f these cllectins may be vlved which hibits detailed cmparisns. Special difficulties als arise makg cmparisns when different matchg functins are used Strict Blean searches, simple crdatin level matchg, and nrmalisg matchg cefficients all prduce rather different frms f utput. Aga, averagg ver sets f requests may be dne differentlyg In Sectin II, I categrised dexg techniques as simple r cmplex. An example f simple dexg is the use f keywrds taken frm titles, with wrd truncatin allwed searchg. Cmplex dexg is illustrated by the use f a cntrlled language like MeSH* The evaluatin f autmatic dexg must refer t the cmparative perfrmance f simple and cmplex manual dexg, and itself requires cmparisns between simple and cmplex autmatic techniques, and between these and manual nes. 8.1

2 It is a pity that the evidence available fr makg all these cmparisns is s fragmentary. The situatin is cmplicated by the fact that different dexg surces (say title r abstract) may be used^ and by the fact that autmatic dexg systems may have manual cmpnentss an example is Sparck Jnes' use f manual keywrd lists as put t statistical classificatin0 The discussin which fllws is necessarily rather schematic: thus pts like the exhaustivity implicatins f different surces are disregarded, and appraches are defed as manual r autmatic accrdg t their predmant characteristics. After all, at this stage what we are terested is the verall perfrmance f dexg methds, rather than the reasns fr it (this is particularly imprtant relatin t differences between prjects assciated with the use f different matchg techniques)«the fllwg attempts t summarise the useful results which have been btaed. I shall use the symbl A= B t mean that the perfrmance f methds A and B is nt nticeably different, A> B t mean that the perfrmance f methd A is nticeably better than that f B, A^>B t mean that the perfrmance f methd A is materially better than that f B, and t mean that the perfrmance f methd A ranges frm the same as that f B, t nticeably better than that f B. Manual dexg The experiments referred t abve suggest that where cmplex dexg methds refer t fairly straightfrward subject headgs r thesaurus terms, and simple nes t extracted keywrds, r phrases, with truncatins cmplex ^ simple. Autmatic dexg Relevant cmparisns here are thse cncerned with the use f statistical assciatin techniques versus keywrds* (Actual experiments have been based n bth autmatically and manually extracted keywrd lists0) Relevant tests are thse by Lesk 1969, Mker 1972, 1973, Sparck Jnes 1971a, 1973c and Vaswani The result, as appeared Sectin IV, is cmplex ZZ simple0 Allwance shuld be made fr the fact that there is sme variatin 8.2

3 the results with simple techniques. Subsidiary cmparisns are therefre f terest0 These cncern a) titles versus abstracts and b) weightgc a) Cmparisns between simple autmatic dexg frm titles and abstracts have been made by Aitchisn 1970, Barker 1972a, Cleverdn 1966 and Tell 1970, as well as the Smart prject (Saltn 1968a,c). There is sme variatin the results, prbably due t different matchg techniques, Aitchisn, Barker and Tell fund that titles gave better precisin than abstracts, but wrse recall, n Blean matchg. With crdatin levels Aitchisn and Cleverdn fund titles superir t abstracts (except fr the recall ceilg)$ while with cse crrelatin Saltn fund abstracts better than titles at higher recall, r verall. Assumg sme terest recall, the fal result must be abstract ^ title. b) Experiments with keywrd weightg schemes f varius types by Saltn 1972c, 1973b,c and Sparck Jnes 1972, 1973e shw that sme frms f weightg may be useful, and specifically that cllectin frequency based weightg can be helpful. These tests vlved bth autmatically and manually btaed keywrds, s the results are applicable t simple manual dexg as well as simple autmatic dexg. Allwance shuld therefre be made the cmparisns which fllw fr the pssibility that simple dexg perfrmance can be imprved«autmatic versus manual dexg The number f experiments directly cmparg autmatic and manual dexg is small, 10 simple autmatic v. simple manual a) titles vq keywrds* These have been cmpared by Aitchisn 1970, Barker 1972a and Hansen 1973 fr Blean search, Aitchisn and Cleverdn 1966 fr crdatin levels, and Saltn 1968a,c fr crrelatin matchg0 Aitchisn fund titles better n precisin and wrse n recall, Barker fund titles gave slightly better precisin and wrse recall, and Hansen fund the tw the same. Aitchisn fund keywrds superir with crdatin levels while Cleverdn fund them much the same. Saltn*s ne test with the cllectin als shws them much the same. S we cnclude keywrds b) ^ title abstracts v«keywrds. These have als been cmpared by Aitchisn 1970 fr Blean matchg, Aitchisn, Cleverdn 1966 and Sparck Jnes 1973d fr crdatin levels, 8.3

4 and Saltn 1968a,c fr crrelatin matchg. Aitchisn*s Blean matchg shwed the same perfrmance. With crdatin levels all three prjects fund keywrds superir, while Saltn (see als Lesk 1968) als shws a slightly better perfrmance with keywrds his cmparisn fr the cllectin. We therefre have keywrds ^> 2 abstract. simple autmatic v. cmplex manual The cmparisns here mstly cncern titles versus fairly straightfrward subject headgs r thesaurus terms. Cmparisns have been made fr Blean searchg by Aitchisn 1970, Miller 1971a, live 1973, Saracevic 1968, 1971 and Tell 1971? fr crdatin levels by Aitchisn and Cleverdn 1966, and fr crrelatin by Saltn 1968a,c. The results tend t shw keywrds perfrmg better, thugh there is variatin the test fdgsa In crdatin level matchg Aitchisn fund titles ferir, while Cleverdn fund their perfrmance much the same. In the Smart experiments they were ferir. Cleverdn and Saltn als cmpared abstracts and manual dexg, bth cases fdg the frmer ferir. Hwever the recent Smart experiments with weightg (Saltn 1972c, 1973b), directly cmparg imprved simple autmatic dexg fr abstracts with subject headgs, shwed the same perfrmance. Nevertheless the verall cnclusin must be cmplex manual ^ 3 simple autmatic. cmplex autmatic v. simple manual Mst f the prjects vestigatg statistical assciatin techniques cmpared perfrmance with simple keywrds, but sme cases the keywrds were extracted autmatically. Cmparisns with manual keywrds were made fr crdatin levels by Sparck Jnes 1971a, 1973c and fr cse crrelatin by Lesk The results were very variable, s it must be cncluded that cmplex autmatic ZZ simple manual. 40 cmplex autmatic v. cmplex manual Apparently the nly cmparisn between these tw frms f dexg is that made by the Smart prject. Lesk 1969 illustrates the results fr three cllectins. The balance f the evidence is favur f thesauri as ppsed t statistical assciatins, s we have cmplex manual j ^, cmplex autmatic. These remarks are based n the explicit results given fr particular tests the publicatins cncerned. The fact that different tests and cllectins tend t be vlved under the varius headgs may mean that the cnclusins I have drawn are nt necessarily cnsistent. I shall attempt t pull the threads tgether Sectin I. 8,4

5 VIII. 2 The SMART Prject The Smart prject is the largest and lngest term research prject autmatic frmatin retrieval. There are few guestins cnnected with autmatic dexg n which it has nt dne sme wrk ver the last ten years* Individual publicatins have been mentined as apprpriate? but it is useful t lk briefly at the Smart research as a whle, t see what verall cnclusins abut autmatic dexg can be drawn frm it, Details f the wrk are given the prjects Infrmatin Strage^ and Retrieval Reprts (Saltn 1966-) 0> Early research is summarised Saltn 1968a,c c Selected papers are reprted Saltn 1971a<, The vestigatins can be gruped under fur heads. The relevant papers are listed Table 2 D The test cllectins explited are listed Table 3 l Experiments n basic autmatic dexg, 2 0 Experiments n autmatic dex language generatin. 3 4 Experiments n dcument clusterg c Experiments n relevance feedback techniques 5<. Experiments cncerned primarily with methdlgical questins 1<. Early experiments vestigated the simple use f nn-trivial keywrd stems frm titles and abstracts, and cmpared these with manual dexg n the ne hand, and manual and autmatic methds f rganisg the extracted vcabulary n the ther. The results shwed that simple keywrd methds, thugh perfrmg less well than manual thesauri, did nt perfrm cnspicuusly less well* These tests were als cncerned with the effects f withm-dcument frequency weightg, and with different matchg cefficients 0 They shwed that alternatives here culd affect perfrmance, and that n the whle the best results were btaed with weights, and a nrmalisg matchg cefficient like cse crrelatin,, 2 Q Initial experiments with dex language generatin vlved statistical assciatin lists fr expanded dcuments and requests* These were nt particularly successful, n real imprvement ver simple keywrds beg btaedc Recent experiments cntrllg an dexg vcabulary either by deletg nn-discrimatg r cmmn wrds, r by differential weightg usg cllectin frequencies, shw that nticeable perfrmance gas can be btaed, the results beg very cmpetitive with dependent manual dexg usg a cntrlled vcabulary* 3* Clusterg experiments have been based n centrid techniques e The b3ect f clusterg is seen as ecnmic? the results shw perfrmance lsses c This le f wrk has nt been very prductive, perhaps because the clusterg methds were nt really adequate e 4c Tests with relevance feedback have examed the use f frmatin fr relevant and nn-relevant dcuments retrieved an itial search mdifyg requests fr new searches? relevant terms the request may be upgraded, and nn-relevant nes dwngraded. The tests generally shw that nticeable perfrmance imprvements can be btaed by 8.5

6 techniques which d nt vlve the user very much effrt, and which represent autmatic revisin f dexg. The crrect characterisatin f perfrmance this type f experiment is nt readily determed, and care is needed terpretg the rdary recall/precisin graphs given Tests have als been carried ut shwg that permanent changes t dcument descriptins may be useful. 5 The range f experiments dne has raised a number f methdlgical and related questins, and sme experiments have been carried ut t exame pts vlved feedback evaluatin (1970c), relevance judgement variatin (1968d), generality (1972b) and the use f mixed language data bases (1970b). It cannt be claimed that the prblems vlved are all reslved, but mre bvius criticisms may be vercme. It shuld be emphasised that direct cmparisns with manual dexg have been few: fr the x cllectin, and, recently, fr the 29x450 cllectin. The latest results are prmisg fr autmatic dexg, but it wuld be nice t have mre cmparative evidence f this srt. The ma weakness f the Smart experiments has been the small scale f the cllectins maly used, apparent Table 3. A further difficulty is that different tests may have been carried ut with different cllectins, makg detailed crss checkg rather difficult. Further cmplicatin appears where different versins (stem, thesaurus) f particular cllectins are vlved. It is t be hped that sme larger scale experiments will be carried ut with the mre successful techniques, and that fairly rigrus crss cmparisns will be made. A pt which shuld perhaps als be made is that where statistical significance tests may justify the assertin that methd A is better than methd B, many cases the real difference perfrmance is nt large. 8.6

7 Table 1 Evaluatin test cllectins Aitchisn 1970 Barker 1972a Cleverdn 1966 Hansen 1973 Keen 1972,1973 Lancaster 1968a Lesk 1968,1969 Miller 1971a Mker live 1973 Saltn 1968a,c 1972c,1973b 1973c Saracevic 1968,1971 Sparck Jnes 1971a 1972,1973c,e 1973d Tell 1971 Vaswanil970 Table 2 97 x Inspec CAC CBAC/PST CAC I SILT ADI IRE IRE IRE ADI NSA ADI IRE Time Keen (ISILT) Inspec Keen Inspec PST Electrical engeerg Chemistry Chemistry Chemistry Dcumentatin Medice Dcumentatin Cmputg Medice Cmputg Medice Cmputg Medice Dcumenta tin Nuclear science Dcumentatin Cmputg Medice Medice Wrld affairs Trpical diseases Dcumentatin Electrical engeerg Dcumentatin Electrical engeerg Plymer science mixed Smart publicatins; Saltn unless therwise dicated 1«basic 1968a,c, 1970a, Keen 1967, Lesk de^ language 1968a,c, 1969a, 1972a,c, 1973b,c, Lesk 1968,1969 3, dcument clusterg 1968b, 1972d, Rcchi 1966, Dattla 1969, Kerchner 19"M, Murray relevance feedback 1968b, 1969b,c,d, 1972d, Brauen 1969, Ide 1969, Kerchner , methdlgy 1968d, 1970b,c, 1972b 8.7 (8.8 blank)

8

9 CM CM T 1 H 00 CM c^ C 0) T? Si en <t* CM D 0J u ft 00 G c m «* C T5 rh 0) ; H c rd U -d 4 c (N C C C U P W <L) P -P (U H 9 C m m C I i u s CCC^C^Ch Vlk^vVDvDV fhihcslcnrrnr^cc^cric^r-i ^^^rhfhfhfhrhrhi-lr-lr-lrhrhrhihr^rhr-lhrhihihr-lrhrhihih C 03 c ^ 0) (0 S3 C -1 0) 0 3 4J fd -P <u (QQH U s I 8.9