The Single Pass Clustering Method. S. Rieber and V. P. Marathe. In information retrieval, several complex clustering methods exist

Size: px

Start display at page:

Download "The Single Pass Clustering Method. S. Rieber and V. P. Marathe. In information retrieval, several complex clustering methods exist"

Adele Debra Armstrong
5 years ago
Views:

1 XIV- The Single ass Clustering Methd S. Rieber and V.. Marathe Abstract In infrmatin retrieval, several cmplex clustering methds exist which require extensive prcessing time and cmputer memry. A cheap clus- tering methd is develped which requires nly ne pass ver the dcument cllectin t generate clusters. The ne-pass clustering methd is investigated using the ADI cllectin f 82 dcuments and 35 queries which is available n-line in the SMART system. Clusters frmed are nt f unifrm size; ne r tw early clusters are exceptinally large. Variatin f the minimum crrelatin cutff value is an adequate cntrl fr the number f clusters generated. The effect f dcument er n the clustering methd is investigated, and the results are incnclusive. verall, the single-pass clustering methd is surprisingly effective and cmpares favrably with mre cmplicated clustering methds.. Intuctin In infrmatin retrieval, a given Item f infrmatin is ften repre- sented by an N-dimensinal vectr, each dimensin referring t a different prperty f the item. Using matrix algebra, cmparisns, classificatins, and ther prcesses can be perfrmed n the Infrmatin vectr r n a lng series f such vectrs. Classifying the item, that is, placing it in a grup f similar items, can save time later when it becmes necessary t refer t that Item again. Fr example, at the supermarket it is much easier t find the aisle with breakfast cereals, and then t lk fr the crn

2 XIV-2 flakes, than it is t search fr them n every shelf f every aisle. The classificatin prblem has tw basic aspects: ) classificatin already exists, and each new item is placed in that grup t which it is the mst similar; 2) n classificatin is assumed t exist and grups are frmed n the basis f similarities between items. [] Bth types f prcesses have advantages and disadvantages. tin already exists, the prcessing f new items is simple. If a classificawever, the size f the grupings ften becmes unbalanced in time with a resultant lss in efficiency. Merely finding the breakfast fd sectin wntt save much time if half the stre is stcked with cereals. The secnd classificatin prcedure f regruping every time a few new items are added t the set is time-cnsuming. Many cmparisns and decisins must be made t determine the ptimum grups. than ne grup. It may be reasnable t include the same item in mre But nce the time and truble have been taken t classify the items, any specific item is relatively easy t find. In the cntext f autmatic infrmatin retrieval, the items f infrmatin are dcuments and search requests, and the classificatin grups are called clusters. All the vectrs in a cluster are averaged t btain the cluster centrid, a vectr representatin f the entire grup. vectrs are crrelated with the centrids. Query nly thse centrids with suffi- ciently high crrelatins are expanded; that is each dcument in the cluster is crrelated with the query and retrieved in er f higher crrelatin. This type f search is knwn as a cluster search r tw-level serach. T cntrast, a full search wuld crrelate every dcument with the query and retrieve them in er f highest crrelatin. Cluster searching is a pr-

3 XIV-3 ven methd fr reducing search times hpefully withut significant lss f relevant dcuments. Many dcument gruping schemes representing bth slutins are in existence. Fr example, library catalging srts publicatins int already existing classificatins. n the ther hand, much research is being dne t implement the secnd classificatin prcess, that f frming clusters based nly n the similarities between the items. In general these methds require extensive cmputer time and strage space t calculate crrelatins between every pair f dcuments. RcchiTs and Bnner's clustering methds [2], [3] are f this kind, since they require a prcessing time prprtinal 2 t N, where N is the number f dcuments in the cllectin. A mre effi- cient clustering methd develped by Dattla [] des nt cmpute the dcument-dcument crrelatin matrix and reduces the prcessing time t ne f er N Lg N. A different, highly inexpensive methd f clustering has been prpsed; that f examining each dcument nly nce, and frming clusters in the prcess. Such a methd is called the single-pass clustering methd and is the subject f this paper. The single-pass clustering methd is appealing because it saves a significant amunt f time ver ther clustering methds. T illustrate, assume that a clustering methd requires minute f cmputer time fr dcuments. Then t cluster,0,0 dcuments, an N 2 clustering methd will take 70 years, an N Lg N methd 2 days, while the single-pass methd wuld take nly 7 days. The single-pass clustering methd als has a certain aesthetic appeal in that it represents a cmprmise between the tw basic aspects f the classificatin prblem. N initial classificatin exists, but as dcu-

4 XIV-4 ments are prcessed, classificatins are built up. A dcument is cnsidered fr inclusin int all existing clusters befre it is allwed t start a cluster f its wn. Classificatin grups are frmed principally n the basis f the similarities f an item with an existing grup rather than n the basis f similarities between items. wever, nce a dcument is accepted in an existing cluster, the cluster centrid is accingly revised. 2. The rgram A FRTRAN prgram was written t implement the single pass clustering methd. A flw diagram f the prcess is shwn in Fig.. The single-pass clustering methd assigns the first dcument vectr scanned as a cluster centrid. The next and succeeding dcuments are crrelated with existing cluster centrids. If a minimum cutff crrelatin is equaled r surpassed, the dcument is included in every such cluster and the cluster centrid is revised Cverlappingl, r the dcument is included nly in that cluster with the highest crrelatin, assuming the minimum cutff is again equaled r surpassed Cdisjint). If the minimum crrelatin is nt reached, a new cluster is frmed with the dcument as centrid. Tw crrelatin methds csine and Tanimt can be used with the prgram. Csine crrelatin nrmalizes all vectrs t length.0 in N-dimemsinal space, N being the number f cncept-dimensins in every dcument r query vectr* The csine f the angle between the tw vectrs is cmputed and used as a similarity measure. N Z.a 0 I d i d 2 Csrne S, = 2 = i=l N *",* 9 i=l N i=l 4 0

5 XIV-5 / TRL CARDS T INITIALIZE TYE F CRRELATIN CSINE, TANIMT, DISTANCE TYE F CLUSTERING: DISJINT, VERLAING MIN. CUTFF CRRELATIN, DC. RDER, ETC. READ ST DC READ NEXT DC > _ CRRELATE DC. WIT EXISTING CLUSTER CENTRIDS D ANY CRRELATINS EXCEED MIN. CUTFF CRRELATIN Y N 4 YES DISJINT/VERLAING DISJINT VERLAING INCLUDE DC. IN ALL CLUSTERS WIT CRR. ABVE MIN. CUTFF. UDATE CLUSTER CENTRIDS INCLUDE DC. IN CLUSTER WIT IGEST CRR. UDATE CLUSTER CENTRID DC. BECMES A CENTRID F A NEW CLUSTER -< < - N" AVE ALL DCS. BEEN RCESSED? Y YES RINT INFRMATIN FR CLUSTERS FRMED Y ABVE CLUSTERS - STRED IN SUITABLE FRMAT ARE FED INT SMART SEARC AND AVERAGE RGRAM T GET VARIUS MEASURES LIKE RECISIN-RECALL ETC. Q Single ass Clustering Methds Fig. Y

6 XIV-6 Tanimt T s crrelatin als perfrms vectr multiplicatin t btain a similarity measure. The denminatr is a nrmalizing term t prevent dense dcument vectrs frm being assigned disprprtinately high crrelatins. Tanimt S = " 2 d n d d.5 + d.d - d,.d l" " ~2"~2 ~"~2 N.. ~2 S d d 2 N.. N. _ N I (dv + I Cdi 2 - J i=l = = The prgram cntrl parameters are as fllws:. Type f clustering verlapping r disjint. 2. Crrelatin Methd csine r Tanimt. 3. Minimum Cncept Weight fr Centrid T prevent the dilutin f the cluster centrid, cncept weights f less than this value are set t zer. All runs were made at Cncept bund the highest cncept number f the dcument cllectin. 5. Mult Cncept weights f the nrmalized centrid vectr in the single-pass prgram are less than.0. T prevent mst f these cncepts frm disappearing when cnverting t integer (fixed pint) values fr the SMART system, all centrid weights are multiplied by MULT. All runs were made with MULT equal t., 6. Cutff The value f the minimum crrelatin cutff, belw which a dcument will nt be included in a cluster. Six cntrl cas cause the abve quantities t be intuced int the prgram, and als determine whether the utput will be printed r punched r

7 XIV-7 bth. Fur f the cas are name cas and place the prper SMART cntrl cas at the beginning f the utput t facilitate SMART evaluatin. 3. Investigatin and Results Using the 82 dcument, 35 query American Dcumentatin Institute cllectin filed n-line in the SMART S (Crnell Data Set), the fllwing investigatins f the single-pass clustering methd were undertaken: ) crrelatin cmparisn: csine crrelatin, frwa er cmparisn f clusters frmed by the tw different crrelatin, methds ver a range f crrelatin cutff values; 2) Disjint-verlapping cmparisn: csine crrelatin, frwa er cmparisn f disjint and verlapping clusters ver a range f crrelatin cutff values; 3). Variatin f dcument er: csine crrelatin, fixed crre- latin cutff cmparisn f bth verlapping and disjint clusters frmed with three different initial dcument ers; 4) SMART evaluatin: SMART evaluatin f the 6 sets f clusters frmed in part 3 t cmpare verlapping and disjint methds as well as the three dcument ers; al full search f the dcument cllectin and evaluatin fr cmparisn t single-pass runs; bl SMART evaluatin f clusters frmed by Dattla's methd, fr cmparisn; c) SMART evaluatin f Dattla!s clusters frmed frm initial single pass clusters. A) Crrelatin Cmparisn Fr csine crrelatins, crrelatin cutff values ranging frm.0 t.30 were investigated. The number f clusters frmed varied frm 5 at

8 XIV-8.0 t 3 at.30 with 4 clusters frmed at a crrelatin cutff f.20. Tanimt crrelatin cutff was varied frm.02 t.30. The number f clusters frmed varied frm 7 at.02 t 8 at.30 with 5 clusters frmed at a cutff f.04. All clustering was disjint using frwa dcument er. Appendix cntains tablesand graphs f the results f all runs perfrmed fr this phase f the investigatin. Fr every crrelatin methd, the relatinship between the number f clusters generated and the crrelatin cutff values is smth and nt unmanageably steep. Suitable variatin f the crrelatin cutff value can therefre be used t rughly cntrl the number f clusters that will be frmed fr a given dcument er. A striking fact abut csine and Tanimt clusters is the large variatin between sizes f clusters frmed, as shwn in Fig. 2. The largest clusters are usually amng the first 2 r 3 clusters, n dubt because they are frmed early and have a chance t inspect nearly the entire dcument cllectin. Tanimt clusters seem t be slightly mre evenly distributed than csine clusters. There is n apparent relatinship between dcuments included in a given csine cluster and dcuments included in a given Tanimt cluster. Bl Disjint-verlapping Cmparisn Fr csine crrelatin, frwa dcument er, the crrelatin cutff was varied frm.0 t.30, and bth verlapping and disjint clusters were frmed. were frmed. At a cutff f.20, 4 disjint and 7 verlapping clusters The average dcument is nly included in ne cluster fr the disjint methd but is included in 4.8 clusters fr the verlapping methd. Apparently the dilutin f the verlapping clusters resulted in the frma-

9 XIV-9 Csine.20 Crrelatin Cutff Tanimt.04 Crrelatin Cutff Number f Clusters Number f Dcuments in each Cluster Number f Clusters Number f dcuments in each Cluster 2 20 r ,6, r 7 3 8,9,0, r 2 3 r 4 5,6, r 7 6 r r 4 5 r 2 4 Ttal Clusters 5.9 Average Dcuments per Cluster 5 Ttal Clusters 5.5 Average Dcuments per Cluster Cluster Size Variatins Fig. 2

10 xiv-i tin f three mre clusters than the disjint methd. Any relatinship be- tween the dcument cmpsitin f a given verlap cluster and that f a gi/en disjint cluster is submerged by the large amunt f verlap. The results f the SMART evaluatin runs made n disjint and verlapping clusters will be discussed in that sectin. Tables and graphs fr all runs appear in Appendix. C) Variatin f Dcument er Three different initial dcument ers were used fr this investigatin frwa, reverse, and middle. The frwa dcument er lists the dcuments sequentially, while the reverse er has them backwas. Mid- dle er intuces the dcuments in an inside-ut er; fr example, dcuments, 2, 3, 4, 5, 6 in middle er wuld be 4, 3, 5, 2, 6,. Using a fixed csine crrelatin cutff f.20, bth disjint and verlapping clusters were frmed with the three different initial dcument ers. Fig. 3 shws the number f clusters frmed fr each situatin. Frm this data, it can be tentatively cncluded that a 0% r 20% variatin in the number f clusters frmed frm ne initial dcument er t anther will nt be unusual. D) SMART Evaluatin recisin and recall data were averaged ver all disjint and all verlapping runs t btain a measure f effectiveness reasnably independent f dcument er. recisin-recall graphs f the fllwing runs are pltted in Appendix 2.. Full search 2. Dattlafs cluster search

11 FL CFrwa verlap) - 7 clusters ML (Middle verlap) - RL (Reverse verlap) - FDJ (Frwa Disjint)- MDJ (Middle Disjint) - RDJ (Reverse Disjint)- Average Cluster Size Square Rt. f Std. DeV. 4 clusters 9 clusters 4 clusters 4 clusters 9 clusters 6.2 clusters f N 2 \ V2 =( X i ~ X AV j =2.07 clusters. V --i / Variatin in Cluster Size fr Different Dcument ers Fig. 3

12 XIV-2 3. Single-pass disjint cluster search 4. Single-pass verlapping cluster search The full search curve is naturally mre effective than ther curves. The single-pass disjint cluster search is the next best search. DattlaTs cluster search appears higher at bth ends f the graph than the singlepass verlapping cluster search, but lses sme grund in the middle t high-recall regin. Fig. 4 shws fr each run, the index f rank recall plus lg precisin, anther methd f system evaluatin, and reveals nearly the same results. Using ne set f disjint and ne set f verlapping clusters as initial clusters, DattlaTs clustering methd was evaluated. recisin- recall graphs f these results are fund in graph 2 f Appendix 2. The disjint initial clusters seem t give a higher precisin at lw recall values but a lwer precisin at high recall values than the verlapping initial clusters. Fr cmparisn DattlaTs previus run with unknwn initial clusters is als pltted and falls in the middle. In Appendix 2 graphs 3 and 4, the precisin-recall curves f the disjint and verlapping clusters used as initial clusters fr DattlaTs methd are pltted. Als Dattlafs results using these clusters as initial clusters are pltted t determine if any significant imprvement in clustering has ccurred. Fr the disjint clusters n imprvement ccurrs. Dattlafs clustering slightly reduces the effectiveness f the initial clusters. Fr the verlapping clusters, Dattlafs methd increases precisin at the lw recall end but reduces it at the high recall end f the graph, a significant imprvement. It is felt that the apparently high recall and precisin results b-

13 XIV-3 Search Type Rank Recall plus Lg recisin Full Average Disjint Cluster Average verlapping Cluster Dattla T s Cluster A arameter fr Cmparing Varius Methds Fig. 4

14 XIV-4 tained by the single-pass methd relative t DattlaTs methd is in part due t the expansin f ne r tw very large clusters, prviding a search f ver half the dcument cllectin. Fur mre runs were made t reduce the number f clusters expanded s that a mre legitimate cmparisn between the tw methds culd be made. Althugh a reductin in the number f clusters expanded ccurrs, n appreciable effect is bserved n the perfrmance measures, prbably due t the cntinued presence f thse extremely large clusters. Fr each run, the average dcuments checked per query are tabulated in Fig. 5. Mst single-pass runs checked far mre dcuments than DattlaTs runs, revealing that a single-pass cluster search is nt as efficient as a Dattla cluster search in terms f search time. Recall ceilings, hwever, naturally favr the system which lks at the mst dcuments. Glbal cmparisns f all evaluatin runs are tabulated in Table 2 f Appendix 2. Nrmalized precisin, nrmalized recall, rank recall, lg precisin, rank recall plus lg precisin, and recall ceiling are included. 4. Cnclusins ver the 82 dcument 35 query ADI cllectin, the single-pass clus- tering methd as described here is effective. Because the dcument cllec- tin is s small, these prmising results cannt be cnclusive. Mre ex- tensive evaluatin runs with larger dcument cllectins shuld be cnducted befre allwing the single-pass clustering methd a permanent place in the infrmatin retrieval wrld. The wide gap in the number f clusters frmed fr different input ers cntributes t a grwing suspicin that the single-pass methd may

15 XIV-5 be very er-dependent. The three ers used were nt truly randm. The middle er, fr example, is a cmpsite f bth the frwa and reverse ers as shwn in Fig. 6. Further investigatins f the single-pass clus- tering methd shuld include evaluatin with randm initial dcument ers t determine mre precisely the standa variatin f the number f clusters frmed at a given crrelatin cutff. Disjint clusters are mre effective than verlapping clusters. Re- ducing the figure f 4.3, the number f verlapping clusters t which the average dcument belngs, wuld certainly imprve th verlapping feature. It is recmmended that the verlapping algrithm be revised s that a dcument is included in the cluster with which it crrelates the highest (abve cutff) and nly in ne r tw ther clusters if the crrelatin is within the value f a small parameter, epsiln, f its highest crrelatin. The number f clusters which will be frmed can be effectively cntrlled by the establishment f a range f values fr the crrelatin cutff. erhaps the biggest pitfall f the single-pass methd is the frmatin f ne r tw excessively large clusters. It is these large clusters which cause t many dcuments t be checked and reduces the efficiency f the cluster search. T prevent large clusters frm frming, it Is recmmended that the single-pass clustering prgram be extended s that, as cluster size increases, the csine crrelatin cutff value slides up thrugh a sequence f values. As the number f items in a cluster increases, it becmes mre difficult fr a dcument t assimilate int that cluster. Such a prgram feature wuld reduce the large variatin in the sizes f the clusters. Finally the pssibility f mre than ne pass thrugh the dcument cllectin t determine the ptimum csine crrelatin cutff shuld be cn-

16 XIV-6 Run Descriptin Dattla (ML initial clusters) Dattla (MDJ initial clusters) RDJ (Reverse disjint) FDJ (Frwa disjint) MDJ (Middle disjint) ML (Middle verlapping) Average Dcuments Checked per Query Recall Ceiling Cmparisn f Average Dcuments fr Varius Methds Fig. 5 Frwa er Middle er Reverse er 3 2 Relatin in Three Dcument ers Taken An Illustratin Fig. 6

17 XIV-7 sidered. Fr large dcument cllectins a small representative subset culd be reprcessed until an ptimum crrelatin cutff is determined.

18 References [] Dattla, R. T.,A Fast Algrithm fr Autmatic Classificatin, Reprt ISR-4 t the Natinal Science Fundatin, Sectin V, Department f Cmputer Science, Crnell University, 968. {2] Bnner, R. E., "n Sme Clustering Techniques", IBM Jurnal f Research and Develpment, Vl. 8, N., January 964. [3] Rcchi, J. J., Jr., Dcument Retrieval Systems ptimizatin and Evaluatin, Reprt ISR-0 t the Natinal Science Fundatin, arva Cmputatin Labratry, Cambridge, Massachusetts, March 966.

19 XIV-9 Appendix Cluster Size Infrmatin

20 XIV-20 Cutff Number f Clusters Average Size f Clusters Maximum Size f Clusters Minimum Size f Clusters > >50.0 j Cluster Size bservatins fr Frwa Dcument er, Tanimt Crrelatin and Disjint Clusters Table

21 XIV-2 40 Dcument er: Frwa Crrelatin: Tanimt Cluster Type: Disjint 30 Number f Clusters *- 20 -w /Maximum Cluster Size 0 \ faverage Cluster Size 0 _L Cutff X Recall-recisin Graph Crrespnding t Table Graph

22 XIV-22 Cutff Number f Clusters Average Cluster Size Maximum Minimum Cluster Size bservatins fr Frwa Dcument er, Csine Crrelatin and Disjint Clusters Table 2

23 Dcument er: Frwa Crrelatin: Csine Cluster Type: Disjint Maximum Cluster Size Number f Clusters Cutff Recall-recisin Graph Crrespnding t Table 2 Graph 2

24 XV-24 Number f Clusters A Dcument Belngs T Average Maximum Minimum r-\ zf CM L [> C J- it CM cu TJ 4! c Q) 6 :3 0 Q u 4-J C 2 r b Cluster Size Number f Clusters Average Maximum Minimum r L [> I> L CM C CM zf L C zf zf C L r- [>- L r r r C 'Tj fc IU S & 0 U-. r 0 4 (/l c: 4 ' m > 0) LT) CI) N - C ci) +J (/I : a, rn ai r fc (D > T c nj c 0 +J ffl r-4 & CJ C C a C Cutff

25 XIV-25 Number f Clusters Disjint (frm Graph 2) CL Maximum Cluster Size AAverage Cluster Size A X ± Cutff Recall-recisin Graph Crrespnding t Table 3 Graph 3

26 XIV-26 ' ' - ' - - Number f Clusters A Dcument Belngs T Cluster Size Minimum ] Maximum Average Minimum Maximum Average Number f Clusters r r CM 5.9 -Zt" r r CM r C J- L 23.2 i r r CM C cn r C r L 23.5 en r! en CM en L r r 2.8 ' J- C 0 M m ) fc ',-. '/ C () a & IJ C C 4- J > & (D C.n cu M & a) 4 ' C 2 r C D CM ( i J 2 r ) r 4 J r () <M & a a :i r: x Dcument er Frwa Reverse Cluster Type Disjint verlapping Disjint verlapping Disjint 4-> a Middle verlapping

27 XIV-27 Appendix 2 Clustering Evaluatin

28 -28 Recall Full Search Dattla's Clusters Dattla s Clusters Using Initial Clus ter As ML MDJ 2 Sing le ass Clu sters Average* verlapping Average* Disjint ML = Single pass clusters with Csine Crrelatin, Cutff 0.2, Dcument er Middle, Cluster type verlapping. 2. MDJ = Same as abve, except Cluster type Disjint. * These Average recisins are taken ver three Dcument ers Recall-recisin Tables Table

29 XIV-29 Y A Full Search A Dattla's Cluster Search Single ass Disjint Cluster Search Single ass verlapping Cluster Search ( and are averages ver three dcument ers) 3 4 = 3h X \ K' QD- A-\ TI--D lh I I I I I I Recall Recall-recisin Graph Crrespnding t Table Graph

30 XIV-30 8 r~" attla T s Cluster Search Using Initial Clusters as Single ass Clusters (Disjint) Csine, 0.2 Cutff, Middle Dcument er Nt Using Single ass Clusters Using Initial Cluster as Single ass Clusters (verlap.) Csine, 0.2 Cutff, Recall * X Recall-recisin Graph Crrespnding t Table Graph 2

31 XIV-3.8 Y.7 A Single ass Clusters (Disjint) Middle Dcument er, Csine Crrelatin, Cutff Crrelatin 0.2 Dattla's Clusters (Disjint) Using Clusters in A as Initial Clusters A A Recall-recisin Graph Crrespnding t Table Graph 3

32 Y.8 A Single ass Clusters (verlapping) Dcument er Middle, Csine Crrelatin, Cutff 0.20 Dattla's Clusters (verlapping) Using Clusters in A as initial Clusters c (/) Recall Recall-recisin Graph Crrespnding t Table Graph 4

33 XIV-33 J w C L [> [> en ij cn C J- C C> a) N" I>. i 'U r. J i /"~N J S I **C J w k c -t c - I C cn en _r ) zf f c C xj CI) i: G JM I w 2 I> / -\ ~ J CM W 0) c. L C r r. t ) L - b > M, M", () <M -. I> "-"* b»- en d- Q L -f r >-> ' =t L t L. t «L x: n. -. / S -D Q L r S c>. IC) r L --J- L C. ' S *r L *"0 Q ;* 3 C) < - r- C C CJ re- > C if) w 0) T / -\ J" CM I r y <»"D Q S C L L C [> c > L r T) L c0 L C en L r 3 l> c cn N- d cu ci a p C - s. r C s J- 0>. t L C [> [ 4J [ Q C L L -t r C zr r-l lm C L C ) c :> p. < > / "> J ~ C / / c ' ' g3 C S / ~ ~ N r S N r r r M* C - C - e u CM a CD CM' b J r* C M M* ci) ' C 0 Q <l (/) -f- (0 -r C b U J n s~~< b b C -r < - > < -> 0 r, 'r ' r cd > I n p,. i n td, (D C (D - ' T I 0 - r ) ;: in 'i J - u U C r t p : cl). ' J n >; 0 J r-l :: C K () p d 'T j - ' d (/) (D, ' ] ; : C r-l 0 M i s_> p. r,m n :< j () r (0' (!). -n. (d (D C p r f C) N L ) C C ) : C 0) J.. p p (h 0!, r I C i' X C c 4 J (=u..ci) (0, In M hi) cn c > L L I> C v_> IX) [> [ J 0 C [.,(/') i j g Z C p. r - <> a i > p» «p I, L -, U C r i f ) p U U U U G J (0, \ ;* >.. C M r c. C b f j c d «C - LD Q t p ' ~ C L I <i) U -d L. M* r>. C ) C U r

A New Evaluation Measure. J. Joiner and L. Werner. The problems of evaluation and the needed criteria of evaluation

A New Evaluation Measure. J. Joiner and L. Werner. The problems of evaluation and the needed criteria of evaluation III-l III. A New Evaluatin Measure J. Jiner and L. Werner Abstract The prblems f evaluatin and the needed criteria f evaluatin measures in the SMART system f infrmatin retrieval are reviewed and discussed.