Let the set of original documents (to be searched) be D = {D 1, D 2, D 3 },

Transcription

1 EXAMLE Let the set of originl documents (to be serched) be D = {D 1, D 2, D 3 }, where D 1 = Byes' rinciple: The principle tht, in estimting prmeter, one should initilly ssume tht ech possible vlue hs equl probbility ( uniform prior distribution). D 2 = Byesin Decision Theory: A mthemticl theory of decision mking which presumes utility nd probbility functions, nd ccording to which the ct to be chosen is the Byes ct, i.e. the one with highest Subjective Expected Utility. If one hd unlimited time nd clculting power with which to mke every decision, this procedure would be the best wy to mke ny decision. D 3 = Byesin Epistemology: A philosophicl theory which holds tht the epistemic sttus of proposition (i.e. how well proven or well estblished it is) is best mesured by probbility nd tht the proper wy to revise this probbility is given by Byesin conditionlistion or similr procedures. A Byesin epistemologist would use probbility to define, nd explore the reltionship between, concepts such s epistemic sttus, support or explntory power. TASK: process nd prepre D for retrievl. SOLUTION: pply IR technology. Steps 1. Identify lexicl units. Write computer progrm to recognise words (word = ny sequence of chrcters preceded nd followed by spce, dot, comm ). For exmple: probbility, Byesin, epistemology. 1

2 2. Stoplisting. Crete list of rnked words (ower Lw). Exclude frequent nd rre words (use threshold vlues). 3. Stemming. Apply stemming to remining words. The thus obtined words re clled index terms (becuse they re used to identify documents), nd form set T. Originl text: Byes' rinciple: The principle tht, in estimting prmeter, one should initilly ssume tht ech possible vlue hs equl probbility ( uniform prior distribution). function processline($line,$outputfile) { $elements=split("[^a-z-z0-9]",$line); forech ($elements s $e) if (trim($e)!='') fwrite($outputfile,strtolower(trim($e))."\r\n"); } Output: byes ssume principle tht the ech principle possible tht vlue in hs estimting equl probbility prmeter one uniform should prior initilly distribution 2

3 Stoplist: bord bout bove ccordingly cross ctully dd dded fter fterwrds gin ginst go ll llows lmost lone long longside... 3

4 The orter Stemming Algorithm In Linguistics, UmorphemeU is the smllest meningful unit in given lnguge. A UsuffixU is morpheme tht is ttched to t the bck of bse morpheme to form word. Ex.: connect + ion=connection The orter stemming lgorithm (or orter stemmer ) is process for removing the suffixes from words in English. Ex.: connection connect Complex suffixes re removed step by step. Thus: GENERALIZATIONS is stripped to GENERALIZATION (Step 1), then to GENERALIZE (Step 2), then to GENERAL (Step 3), nd then to GENER (Step 4). 4

5 The rules for removing suffix re given in the form: (condition) S1 -> S2 This mens tht if word ends with the suffix S1, nd the stem before S1 stisfies the given condition, S1 is replced by S2. Ex #1.: (*S or *T) ION ->. Here S1 is `ION' nd S2 is null. This would mp ADOTION to ADOT becuse the word ends with letter S or T. Rules: SSES -> SS cresses -> cress IES -> I ponies -> poni ties -> ti SS -> SS cress -> cress S -> cts -> ct ATIONAL->ATE reltionl -> relte TIONAL ->TION conditionl -> condition rtionl -> rtionl IZER ->IZE digitizer -> digitize ATION ->ATE prediction -> predicte 5

6 Stemming Hungrin The stemmer is bsed on spell checker which hs rules like: Rule: Z -> Z Z -> ZÁL E D Z -> ENEK Exmple: ráz -> rázz ráz -> rázzál edz -> edzenek, pedz -> pedzenek [ÓUÚ] -> VAL mnó -> mnóvl, hmu -> hmuvl, bú -> búvl [^AE] -> NAK okos -> okosnk. -> KÉNT kuty -> kutyként, okos -> okosként A -> -A,ÁNAK mcsk -> mcskánk [^SD] ZIK -> -IK,Z brátkozik -> brátkozz 6

7 7 Text before nd fter stemmig: byes principle the principle tht in estimting prmeter one should initilly ssume tht ech possible vlue hs equl probbility uniform prior distribution by principl the principl tht in estim prmet on should initi ssum tht ech possibl vlu h equl probbl uniform prior distribut

8 Stoplist before nd bord bout bove ccordingly cross ctully dd dded fter fterwrds gin ginst go ll llows lmost lone long longside... fter stemming: bord bout bov ccordingli cross ctul d dd fter fterwrd gin ginst go ll llow lmost lon long longside... 8

9 Originl text: Byes' rinciple: The principle tht, in estimting prmeter, one should initilly ssume tht ech possible vlue hs equl probbility ( uniform prior distribution). After stemming nd removing stopwords: by principl principl estim prmet initi ssum equl uniform prior distribut Wht if the source is n HTML document? évi felvételi jelentkezési dtok, Gzdságtudományi kr 9

10 DB1B = DB2B = DB3B = = = in One method to resolve this: $string = str_replce ( rry('á','&cute;'), 'á', $string ); $string = str_replce ( rry('á','´'), 'Á', $string ); $string = str_replce ( rry('û','û','ű','ı'), 'ű', $string ); Let the set T of index terms be (not stemmed here): T = {tb1b, tb2b, tb3b} = {tb1b = Byes, tb2b probbility, tb3b epistemology}. 4. Build term-document mtrix. Conceive the documents s sets of terms together with their frequencies): { (Byes, 1); (probbility, 1); (epistemology, 0) } { (Byes, 2); (probbility, 1); (epistemology, 0) } { (Byes, 3); (probbility, 3); (epistemology, 3) } Term-document mtrix: TD TDB3 3B = (wbijb), where wbijb denotes the weight of term tbib document DBjB 10

11 Frequency: TD = Binry: TD = Uncompressed Mtrix Compressed formt A= Row Coloumn Vlue Length Normlistion Tke the frequency TD =

12 Represent documents in the spce of terms: - use system of coordinte xes to represent documents - ech term t i corresponds to coordinte xis - every document D j will be vector Dj hving coordintes D j = (f 1j, f 2j, f 3j ) epistemology D3 probbility D1 D2 Byes We cn see tht document vectors hve different lengths. Thus, we my define the notion of length of document. Document length is quntity which is proportionl to the number of terms the document contins. Hence, longer documents will dominte shorter ones. Unfir! Why should wordy writer (who sys everything five times in five different wys) be preferred to writer who hs worked hrd to find the most pproprite words to express something? 12

13 = = = = = One wy to overcome this problem is normlistion. There re severl normlistion methods. Mximum normlistion: weights on 0-1 scle reltive to itself TD = Length normlistion: weights on 0-1 scle every document vector will hve unit length For exmple: vector DB1B (1 1 0) length of vector DB1B (1 + + = ) weights of new vector DB1B (1/ 2 1/ 2 0/ 2) length of new vector DB1B 2 ((1/ 2) (1/ 2) + = 0 ) (1/2 + 1/2 + 0) New term-document mtrix (with length-normlised weights): TD = 1/ 1/ / 1/ / 3/ 3/