Lecture 1: Topics in Information Retrieval. Johan Bollen Old Dominion University Department of Computer Science

Transcription

1 Lecture 1: Topics in Information Retrieval. Johan Bollen Old Dominion University Department of Computer Science jbollen January 16, 2003 Page 1

2 Structure 1. Class approach and schedule 2. Class methods and software: (a) LaTeX (b) Gnuplot (c) UNIX (d) Perl 3. Introduction to IR: (a) problem statement (b) definitions (c) history January 16, 2003 Page 2

3 Class Approach 1. LaTeX (a) Submission of reports, etc in PDF only (b) Use provided LaTeX templates to produce PDF (c) Word, Excel, etc: NO 2. Gnuplot (a) GPL licensed graph tool (b) Command-line driven (c) Available for most platforms 3. UNIX (a) Don t re-invent the wheel! (b) Use UNIX tools (c) sort, uniq, awk, grep, sed 4. Perl (a) For projects etc all programming languages are OK (b) Class examples: Perl (c) Strong recommendation to use Perl for your programming January 16, 2003 Page 3

4 Class Schedule 1. JAN 16: Class approach and schedule 2. JAN 23: IR models: taxonomy, formal characterization 3. JAN 30: Probabilistic models, Alternative Set Theoretic Models 4. FEB 6: Alternative Algabraic Models, Alternative Probabilistic Models. 5. FEB 13: Text Operations 6. FEB 20: Indexing and Searching. 7. FEB 27: Query Operations 8. MAR 6: Retrieval Evaluation 9. MAR 20: Human Computer Interaction 10. MAR 27: Digital Libraries 11. APR 3: Web IR, Citation analysis and link analysis 12. APR 10: Latent Semantic Analysis and Multi-Dimensional Scaling 13. APR 17: Recommender Systems 14. APR 24: Adaptive Hypertext and Hypermedia Subject to change January 16, 2003 Page 4

5 2 exams and 1 project 1. Exams: (a) Dates: i. Midterm: take home, scheduled Feb 27th (class 9), submission deadline: March 6th (class 10) ii. Final: Take home, May 8th - Exam (b) Submission: 2. Projects: i. LaTeX templates ii. PDF format only (a) I ll provide list of suggestions (b) Free to choose programming languages, software and tools (c) Proposal defenses: class 8 (d) Project progress report: class 12 (e) Final Submission and presentation: final exam 3. Grade distribution (a) Projects: 30% (b) Midterm: 30% (c) Final: 40% January 16, 2003 Page 5

6 Exams 1. Take home questions 3. choose points per question (a) 0=bad (b) 1=fair (c) 2=good (d) 3=excellent 5. Questions will cover: (a) Theory (b) Practice (c) Technical matters (d) Readings January 16, 2003 Page 6

7 Readings 1. Readings: (a) Seminal articles in IR (b) Additional Material (c) 2 readings per lecture 2. Important: (a) Connect to existing academic practices (b) Scholarly publications (c) IR is still very much alive 3. Presentations: (a) Two random students picked each start of class (b) 10 minute (precisely!) presentations (c) 1 Presentation = 1 exam question 4. Grading: (a) 0-3 scale (b) Subjective but we all know a bad presentation when we see one (c) You may not agree but your peers will! January 16, 2003 Page 7

8 Some words on Academic Honesty Everything you submit to me is supposed to be 1) your, 2) original work. 1. Your work: not that of others (a) Collaboration with other students: only when explicitly required (b) Copying other people s work, texts, software, articles, ideas and attributing it to yourself is fraud (c) Whether something is openly avalaible on the WWW or anywhere else is irrelevant 2. Original work and references: (a) Science is all about using other people s ideas as the foundation of your own (b) Your work can be based on other people s ideas as long as you identify the source (c) When you base your argument, idea, tenets, hypothesis on other people s work: i. Use a citation: clearly identify the source ii. Citations may not comprise more than 10% of your text (d) A citation or quotation does not constitute a valid response, answer or submission: I need to see your work January 16, 2003 Page 8

9 Last words Personally, I ABHOR academic dishonesty 1. The academic systems functions on trust: people building on what others have produced and identifying their sources 2. Open communication of ideas etc only works when people are honest enough to give due credit and work on their own ideas 3. People who steal words, texts, ideas are parasites within that fragile system and will worsen the situation of us all 4. In game theoretical terms, I believe in a tit-for-tat strategy: you cheat once, I cancel our cooperation. 5. The objective of this class is not to teach you how to copy and slavishly replay other people s ideas but to generate your own: be pretentious! Show some hubris! Be an iconoclast! Think for yourself! January 16, 2003 Page 9

10 Class software and tools: GnuPlot Gnuplot is free, powerful and generates great graphs! 1. Open Source product 2. Available for many platforms, including Windows 3. Command-line driven 4. Reads data files in plain text 5. Exports png, eps, and LaTeX! 6. Can be scripted 7. Generates great looking, no-frills graphs 8. Low overhead Did I mention it is free? January 16, 2003 Page 10

11 A Gnuplot Session (1): January 16, 2003 Page 11

12 A Gnuplot Session (2): January 16, 2003 Page 12

13 A Gnuplot Session (3): January 16, 2003 Page 13

14 Class software and tools: LaTeX LaTeX: submission of exams and project reports. 1. Why not use Word, WordPerfect, or your favorit word processor here (a) Proprietary formats that put burden on user to layout rather than write and structure content (b) Closed approach: you want your documents to be available anywhere on any system/os (c) Scientific publishing: LaTeX does math, references, etc much better and more transparantly than any other WYSIWYG processor (d) I prefer standardized submission format and layout for your project reports and exams (e) LaTeX is scientific standard among mathematicians, physicists, and computer scientists: essential for your career to learn principles 2. Principles (a) Markup language like HTML, SGML, etc (b) Markup specifies document structure and semantics, not layout (c) Writer takes care of content and structure, not how things look (d) LaTeX source code is compiled to good looking document by LaTeX compiler January 16, 2003 Page 14

15 Basics of LaTeX 1. LaTeX source code: text file generated in any editor you like (I prefer EMACS) 2. File consists of two parts: (a) Preamble i. Definition of document style: article, book, etc, ii. Definition of other general (b) Body document features (title, author, etc) i. Contains actual document body, interspersed with structural markup ii. Markup declares environments for text: indicate features and structure January 16, 2003 Page 15

16 LaTeX Compilation LaTeX latex pdflatex dvi PDF dvips PS 1. Compilation: (a) from LaTeX to postscript: latex compiler to dvi, dvips to postscript, ps2pdf postscript to pdf (b) from LaTeX to PDF: pdflatex 2. View final result (a) GhostView (b) Acrobat reader January 16, 2003 Page 16

17 Latex Example: \ documentclass { a r t i c l e } \ t i t l e { F i n a l Exam : CS695 T opics i n IR} \ author {You} \ begin { document } \ m a k e t i t l e \ s e c t i o n { I n t r o d u c t i o n } \ s u b s e c t i o n { Q u e s t i o n 1 } Etenim s p e c i e s e s t p u l c h r i s c o r p o r i b u s \ s u b s u b s e c t i o n { Q u e s t i o n 2 } Quid ego m i s e r i n t e amavi, o furtum meum Some math : $f ( x ) = x i ˆ2 $ and : $f ( x ) = \ sum { i =0}ˆ{10} w { i, j }$ \ s e c t i o n { C o n c l u s i o n } Invoco t e, deus meus, m i s e r i c o r d i a mea, q u i f e c i s t i me \end{ document } January 16, 2003 Page 16

18 Looks like: January 16, 2003 Page 17

19 Latex lists \ documentclass { a r t i c l e } \ begin { document } \ begin { enumerate } \ item F i r s t item \ begin { i t e m i z e } \ item B u l l e t 1 \ item B u l l e t 2 \end{ i t e m i z e } \ item Second item \end{ enumerate } \end{ document } January 16, 2003 Page 17

20 Looks like: January 16, 2003 Page 18

21 Latex lists \ documentclass { a r t i c l e } \ begin { document } Math e n v i r o n m e n t can be s t a r t e d with \ $ \\ and t e r m i n a t e d with \ $.\\ Exponents and i n d i c e s : $f ( x ) = x i ˆ2 $, or $f ( x ) = x { i, j }ˆ{12} $\\ Sigma : $f ( x ) = sum { i =0}ˆ{10} w { i, j }$\\ F r a c t i o n : $\ f r a c {\sum { i =0}ˆ{10} w { i, j }} {\ sum j w j }$\\ Roots and i n t e g r a l s :\\ $\ s q r t [ n ]{\ f r a c {1}{ sum { i =0}ˆ{10} w { i, j }}}$,\\ and $\ i n t a ˆ b f i ( x ) $\\ S e t t h e o r y :\\ $\ f o r a l l \ vec {x } \ in \ m a t h c a l {N} : \ e x i s t s \ vec {y } \ in A \ cup B$ \end{ document } January 16, 2003 Page 18

22 Looks like: January 16, 2003 Page 19

23 Latex lists \ documentclass [ p d f t e x ]{ a r t i c l e } \ usepackage [ d v i p s ]{ g r a p h i c x } \ begin { document } PNG, JPG and PDF images can be i n s e r t as f o l l o w s :\\ \ begin { f i g u r e } \ begin { c e n t e r } \ i n c l u d e g r a p h i c s [ s c a l e = 0. 4 ] { pngs / t e x i m a g e } \end{ c e n t e r } \ c a p t i o n {\ l a b e l { example } The f u n c t i o n $\ cos (\ f r a c {1}{\ alpha }) $} \end{ f i g u r e } The l a b e l i n c a p t i o n can be used t o r e f e r t o t h e image. For example, s e e f i g u r e \ r e f { example }. \end{ document } January 16, 2003 Page 19

24 Looks like: January 16, 2003 Page 20

25 Latex lists \ documentclass { a r t i c l e } \ begin { document } C i t a t i o n s a r e v ery easy. No more manually m a i n t a i n i n g a b i b l i o g r a p h y!\\ You keep a b i b t e x f i l e ( e. g. exam 1. b i b ) which s t o r e s your b i b l i o g r a p h i c r e c o r d s. In t h i s b i b t e x f i l e you a s s i g n each r e c o r d a key t h a t i s unique and makes s e n s e. Then c i t i n g a r e c o r d i s as easy as s i m p l y s t a t i n g : \ c i t e {key 1 }, i n your LaTeX code! Adding a n o t h e r c i t a t i o n? Simply use t h e r i g h t key : \ c i t e {key 2}.\\ To add a b i b l i o g r a p p h y s t o your document simply s t a t e : \ b i b l i o g r a p h y s t y l e { p l a i n } \ b i b l i o g r a p h y {exam1} \end{ document } January 16, 2003 Page 20

26 Looks like: January 16, 2003 Page 21

27 LaTeX Resources on the web http: // Best book: A Guide to LaTeX, by Helmut Kopka and Patrick W. Daly January 16, 2003 Page 22

28 Exams and projects I will provide templates for all project reports and exams You need to simply fill in your answers and text, and compile these documents to PDF files following the provided instructions This will provide you 1. An introduction into LaTeX authoring 2. A convenient means to produce professionally looking reports and take-home exams 3. An open standard for document exchange across multiple platforms January 16, 2003 Page 23

29 Perl programming and UNIX tools 1. Information Retrieval often deals with texts 2. Choose language designed to deal with text 3. Perl! (a) Convenient text parsing (b) Convenient access to high-level data objects e.g, associative arrays (c) Excellent database interface support (d) Easy to implement crawlers and HTTP clients (e) Perfect for UNIX scripting January 16, 2003 Page 24

30 Perl variable types #! / u s r / b i n / p e r l jbollen@cs.odu.edu # s c a l a r s $a= A l i s t of words d e l i m i t e d by s p a c e s ; $n = 0 ; # a l i s l i s t = ( item1, item2, item3 ) ; # f o r loop on l i s t i t e m s foreach $item l i s t ){ p r i n t $item. \ n ; } # s p l i t $a a c c o r d i n g t o l i s t 1 = s p l i t / \ s + /, $a ; foreach $item l i s t 1 ){ p r i n t $item. \ n ; } January 16, 2003 Page 24

31 Perl RE and associative arrays #! / u s r / b i n / p e r l $a= A l i s t of words d e l i m i t e d by s p a c e s l i s t 1 = s p l i t / \ s + /, $a ; # an a s s o c i a t i v e a r r a y ( marked %) %f r e q t a b l e = { } ; foreach $item l i s t 1 ){ # $item s p e c i f i e s key i n t o f r e q t a b l e # t h e n add 1 t o $ f r e q t a b l e { $item } $ f r e q t a b l e { $item }=1; } foreach $key ( keys % f r e q t a b l e ){ p r i n t $key. \ t. $ f r e q t a b l e { $key }. \ n ; } January 16, 2003 Page 24

32 Perl RE and associative arrays #! / u s r / b i n / p e r l # r e a d from STDIN while (<>){ # r e s u l t i s s t o r e d i n $ $ l i n e = $ ; chomp ( $ l i n e ) ; # s p l i t l i n e a t s p a c e l i s t = s p l i t / \ s + /, $ l i n e ; } foreach $item l i s t ){ $ f r e q t a b l e { $item }++; } foreach $key ( keys % f r e q t a b l e ){ p r i n t $ f r e q t a b l e { $key }. \ t. $key. \ n ; } January 16, 2003 Page 24

33 Perl: Reading and writing from and to files #! / u s r / b i n / p e r l p r i n t f i l e n a m e?\ n ; $ f i l e = <STDIN>; chomp ( $ f i l e ) ; open ( IN, < $ f i l e ) = < IN>; c l o s e ( IN ) ; $n =1; foreach $ l i n e ){ p r i n t $n. ). $ l i n e. \ n ; $n ++; } open ( OUT, > r e s u l t s ) ; f o r ( $ i = 0 ; $ i <10; $ i ++){ p r i n t OUT $ i. \ n ; } c l o s e ( OUT ) ; January 16, 2003 Page 24

34 UNIX tools 1. sort: alphabetically sorts a file 2. head: first n lines 3. tail: last n lines 5. wc: count number of lines and words in text 6. Pipe any of these tools: powerful combinations! 4. uniq: remove duplicates January 16, 2003 Page 25

35 Information Retrieval: problem statement 1. Rapid Growth in amount of information published (a) Moore s law (b) Papers published: (c) (d) Exponential growth (e) Doubling rate: years, 4-7% per annum for last 300 years (f) Scientists: exponential growth as well (g) Singularity approaching? (Vernor Vinge) Hyper-exponential growth 2. Finding a needle in a haystack (a) Stress inevitably needs to shift from mere storage to accomodating user information needs (b) Digitalization of scholarly communication is another important factor January 16, 2003 Page 26

36 Information Retrieval: origins 1. Library Science (a) Collection management (b) Indexes and organization (c) Rapid growth of volume and costs (d) Automation of indexes to automation of retrieval to recognition of user information needs 2. Information systems (a) Realization that computers can outperform humans in tedium of text indexing and analysis (b) Rapid growth of scholarly publication and need for supported analysis and synthesis January 16, 2003 Page 27

37 Information Retrieval: origins 1. First systems since 1950s (a) First applications of initial computer systems (b) Much of science developed in 1960 (c) Relatively slow growth in applications and data set since 1960s-1980s (d) Highly specific applications in following years 2. Then: Internet and World Wide Web (a) IR: a solution waiting for a problem (b) How about billion pages? (c) Hyperlinked, in a graphical markup language (HTML)? (d) Millions of users, hundreds of languages? (e) Mind the singularity! (f) Infinite user information needs? 3. Ideas, theories, systems surpassed by parallel developments 4. The future? January 16, 2003 Page 28

38 Relational models vs. Information Retrieval 1. DBs: Prevalent and best known approach (a) Oracle, MySQL, PostGreSQL, etc (b) DB tables (c) SQL queries (d) this is data retrieval 2. Information Retrieval? No. (a) no means to respond intelligently to user information needs (b) queries limited to Boolean queries (c) logical view of documents is very limited (d) approach is small and not very powerful subset of IR domain January 16, 2003 Page 29

39 Information Retrieval 1. Information Retrieval is about (a) representation (b) storage (c) organization (d) access 2. Mission: finding relevant information based on user information needs 3. Information vs. data retrieval (a) Data retrieval: i. Known location and nature ii. User query matches data objects exactly iii. Question of storage and retrieval (b) Information Retrieval: i. User information need can be specific to highly abstract/general ii. System to respond to such need: Find About problem iii. Relevancy of response is issue January 16, 2003 Page 30

40 Information Retrieval: common text substrate Framework 1. collection of text documents 2. User formulates text query (a) set of keyterms, words (b) indicates user information need 3. a subset of set of documents, or collections, is retrieved 4. user can accept or browse returbed results Basic concepts 1. Keyterm indexing 2. User task 3. Logical view of documents January 16, 2003 Page 31

41 Text representation January 16, 2003 Page 32

42 Index or keyterms 1. Means to represent semantics of document 2. Aboutness : what is the specific document about? (a) Define semantic layer over collection (b) Idea is that meaning of document can be represented by specific set of keyterms 3. Ties to library classification 4. Interpretation in IR is more general and natural 5. Issues: (a) Extraction: only works for text documents, unless human intervention (b) Languages: the world is not uni-lingual (c) Synonymy, homonymy (d) Jargon and user communities (e) Orthogonality January 16, 2003 Page 33

43 User task 1. User: formulates query which expresses information need 2. System: responds with answer set 3. User tasks: narrowness of interests (a) retrieval: specific information need (b) browsing: vague information need 4. Pull vs. push January 16, 2003 Page 34

44 Logical view 1. Description of content and semantics of document 2. Structure 3. Derivation of index terms: (a) removal of stopwords (b) stemming (c) identification of nouns 4. Pull vs. push January 16, 2003 Page 35

45 Retrieval Process 1. DB manager: (a) Text database (b) Logical view of document (c) Index 2. Retrieval process (a) user information need (b) query (c) query operations (d) retrieval January 16, 2003 Page 36

46 Information Retrieval is not a single application 1. Defined in functional terms (a) IR is defined as a process (b) different processes can lead to similar results (c) platonic ideals of relevancy (d) there does not exist a perfect system (e) IR toolbox 2. Many different models (a) Assumptions: text based, keyterms, (partial) matching, etc (b) Mechanisms: vector space, set-theory, probabilistic (c) Goals: push-pull, retrieval vs. recommendation, browsing 3. Let 10 6 flowers bloom! 4. Necessary to evaluate and compare results January 16, 2003 Page 37

47 An attempt to pigeonhole 1. Three basic classes re: retrieval (a) Boolean (Set theory) (b) Vector (c) Probabilistic 2. Expansion (a) Set theoretical: fuzzy sets, extended boolean) (b) Algebraic: Gen. Vector, LSI, NN (c) Probabilistic: Inference, belief netw. 3. Browsing: (a) Flat (b) Structure Guided (c) Hypertext, WWW January 16, 2003 Page 38

48 UserTask Retrieval Classic Model Boolean Fuzzy Ext Boolean Vect Gen. Vec LSI NN Prob Inf Netw Struct Models Non-overlap Lists Belief Netw Prox Nodes Browsing Flat Guided HT January 16, 2003 Page 38

49 Ad hoc, retrieval, filtering, browsing 1. Ad hoc: (a) Static collection (b) User formulates queries (c) User pull 2. Filtering (a) Collection changes (b) User doesn t? (c) Filter pushed information (d) User modeling (personal) 3. Browsing (a) Interactive environment of user push & pull (b) Hypertext: network structure + interface (c) User modeling and filtering: adaptive hypertext and hypermedia January 16, 2003 Page 39

50 Readings 1. Vannevar Bush (1945) As we may think. Atlantic Monthly, July. 2. Belkin (1992) Information Filtering and Information Retrieval. Communications of the ACM. 35(12). Download PDFs/URLs from web site: spring03_ir/readings. Presentations: 1. point-by-point overview minutes! 3. start with main message or problem statement 4. end by discussing how problem was solved 5. Be nice to your audience! January 16, 2003 Page 40