Text and multimedia languages and properties

Transcription

1 Text and multimedia languages and properties (Modern IR, Ch. 6) I VP R 1 Introduction Metadata Text Markup Languages Multimedia Trends and Research Issues Bibliographical Discussion I VP R 2

2 Introduction Document: single unit of info Text Other media, too Aspect of document I VP R 3 Aspect of document Syntax Semantics (Also pragmatics) Presentation style Application dependent Print vs. WWW vs. handheld etc. Metadata I VP R 4

3 Syntax Implicit / explicit / declarative I VP R 5 Formatting style Separate from content Manipulate independently I VP R 6

4 Metadata Data about data Descriptive How created Dublin Core Metadata Element Set 15 fields to describe doc Semantic About content è ontologies I VP R 7 MARC Machine Readable Cataloging Record Most used for library records Fields for bibliographic info USMARC: US specific I VP R 8

5 BiBTeX Originally, TeX Now, general cataloging I VP R 9 RDF Resource Descriptive Framework Web metadata standard XML-based è machine interoperability Nodes (URI) + attribute=value pairs I VP R 10

6 Text Formats ASCII, EBCDIC, RTF, PDF, MIME, etc. Information Theory Modeling Natural Language Similarity Models I VP R 11 Information Theory Entropy Fewer symbols è less meaning E = σ i= 1 p i log 2 p i Amount of info measure Symbols of alphabet Coded in binary E.g, σ = 2 è E = 1 if both symbols appear same # times E = 0 if only one symbol appears I VP R 12

7 Modeling Natural Language Text: Symbols from finite alphabet Belong to words Letters Separate words Vowels more frequent e : highest frequency (English) I VP R 13 Binomial model Each symbol è given probability But, dependence on previous symbols E.g., English è no cf è prob of symbol depends on previous Finite-context Markovian model I VP R 14

8 Finite-context Markovian model Consider 1, 2, more letters to generate next k-order model: consider previous k for next Binomial = 0-order More complex I VP R 15 More complex Finite-state machines è regular languages Grammars è context-free languages è context-sensitive languages Natual But correct grammar, hard I VP R 16

9 Word distribution models Zipf s law f(i s most frequent word) = 1/i Θ * f (most frequent word) è text: n words, vocabulary V words è f(i s most frequent word) = n / i Θ H v (Θ) H v (Θ) = harmonic number of order Θ of V V Hv( θ) = θ I VP R j= 1 j 17 1 Word frequency distribution Words arranged in decreasing f order Θ = 1 è H v (Θ) = O(log n) Θ = better fit for real data è H v (Θ) = O(1) Mandelbrot distribution: k/(c + i) Θ c = add l parameter k = such that all f s add to n F words I VP R 18

10 Word distribution skewed Few hundred words = ~50% of text è can ignore if too frequent e.g., stopwords Doesn t carry meaning Most frequent è can drop è reduce index space overhead I VP R 19 E.g., TREC-2 collection Most frequent: the, of, and, a, to, in I VP R 20

11 Word distribution in collection Simple model: same in all doc s But not true Better: negative binomial distribution Fraction of doc s containing word k times: F( k) = α + k 1 k p (1 + p) k α k Parameters: depend on word and doc collection I VP R 21 e.g., Brown Corpus, said è p = 9.24, α = 0.42 Other models from Poisson distribution I VP R 22

12 Vocabulary size # distinct words in doc / collection Heap s Law V Finite but high è better model than O(1) Also growth due to typos V = Kn β = O ( n β ) Text dependent 10 K < β < 1 # words in text Text size I VP R 23 Similarity Models Syntactic similarity between strings / doc s Distance function E.g., Hamming distance # positions w/ diff char s Equal è = 0 I VP R 24

13 Distance function properties Symmetric d(a, b) = d(b, a) Triangle equality d(a, c) d(a, b) + d(b, c) I VP R 25 Edit (Levenshtein) distance Min # char s, insert, delete, sub needed a à b e.g., color à colour = 1 survey à surgery = 2 Considered superior to other models I VP R 26

14 LCS longest common subsequence Only char deletion allowed LCS what s left after all noncommon char s deleted E.g., survey + surgery è surey Doc s è longest common line Unix diff command Time consuming I VP R 27 Also visual tools E.g., Dotplot Rectangular map Coordinates = file lines Each coo entry = gray pixel Depends on edit distance between associated lines I VP R 28

15 Markup Languages SGML DTD Schema HTML XML I VP R 29 Multimedia Formats Textual Images Graphics and Virtual Reality CGM: computer graphics metafile VRML HyTime: Hypermedia/Time-based Structuring Language Multimedia document markup standard SGML architecture Fig. 6.6, p. 161: taxonomy of Web languages I VP R 30