Concepts & Categorization Measurement of Similarity Geometric approach Featural approach both are vector representations
Vector-representation for words Words represented as vectors of feature values Similar words have similar vectors 9 9 radio 5 6 pet 5 7 6 dog 6 cat 9 9 radio 5 6 pet 5 7 6 dog 6 cat How to get vector representations Multidimensional scaling on similarity ratings Tversky s (977) contrast model Latent Semantic Analysis (Landauer & Dumais, 997) Topics Model (e.g., Griffiths & Steyvers, 00)
Multidimensional Scaling (MDS) Approach Suppose we have N stimuli Measure the (dis)similarity between every pair of stimuli (N x (N-) / pairs). Represent each stimulus as a point in a multidimensional space. Similarity is measured by geometric distance, e.g., Minkowski distance metric: d ij n = x k= ik x r r jk Multidimensional Scaling Represent observed similarities by a multidimensional space close neighbors should have high similarity Multidimensional Scaling: iterative procedure to place points in a (low) dimensional space to model observed similarities
Data: Matrix of (dis)similarity MDS procedure: move points in space to best model observed similarity relations
Example: D solution for bold faces D solution for fruit words 5
Critical Assumptions of Geometric Approach Psychological distance should obey three axioms Minimality d( a, b) d( a, a) = d( b, b) = 0 Symmetry d ( a, b) = d( b, a) Triangle inequality d( a, b) + d( b, c) d( a, c) For conceptual relations, violations of distance axioms often found Similarities can often be asymmetric North-Korea is more similar to China than vice versa Pomegranate is more similar to Apple than vice versa Violations of triangle inequality: Lemon Orange Apricot 6
Triangle Inequality and similarity constraints on words with multiple meanings SOCCER AB AC FIELD BC MAGNETIC Euclidian distance: AC AB + BC Nearest neighbor problem (Tversky & Hutchinson (96) In similarity data, Fruit is nearest neighbor in out of 0 items In D solution, Fruit can be nearest neighbor of at most 5 items High-dimensional solutions might solve this but these are less appealing 7
Feature Contrast Model (Tversky, 977) Represent stimuli with sets of discrete features Similarity is an increasing function of common features decreasing function of distinct features Common features Features unique to I Features unique to J Sim( I, J ) = af ( I J ) bf ( I J ) cf ( J I) a,b, and c are weighting parameters Contrast model predicts asymmetries Weighting parameter b > c pomegranate is more similar to apple than vice versa because pomegranate has fewer distinctive features
Contrast model predicts violations of triangle inequality Weighting parameter a > b > c (common feature should be weighted more) Additive Tree solution 9
Latent Semantic Analysis (LSA) Landauer & Dumais (997) Assumptions ) words similar in meaning occur in similar verbal contexts (e.g., magazine articles, book chapters, newspaper articles) ) we can count number of times words occur in documents and construct a large word x document matrix ) this co-occurrence matrix contains a wealth of latent semantic information that can be extracted by statistical techniques ) words can be represented as points in a multidimensional space Latent Semantic Analysis (Landauer & Dumais, 97) (high dimensional space) TERMS DOCUMENTS D FIELD 5 MEADOW BASEBALL 0 MAJOR 5 MAJOR FOOTBALL FIELD CORN BASEBALL GRASS MEADOW Information in matrix is compressed; relationships between words through other words are used. 0
Problem: LSA has to obey triangle inequality SOCCER AB AC FIELD BC MAGNETIC Euclidian distance: AC AB + BC The Topics Model (Griffith & Steyvers, 00 & 00) A probabilistic version of LSA: no spatial constraints. Each document (i.e. context) is a mixture of topics. Each topic is a distribution over words Each word chosen from a single topic: P T ( w ) = P( w z = j) P( z = j) i j= i i i word probability in topic j probability of topic j in document
A toy example w i P( z = ) P( z = ) TOPIC MIXTURE P( w z ) HEART 0. LOVE 0. SOUL 0. TEARS 0. MYSTERY 0. JOY 0. Words can occur in multiple topics P( w z ) SCIENTIFIC 0. KNOWLEDGE 0. WORK 0. RESEARCH 0. MATHEMATICS 0. MYSTERY 0. MIXTURE COMPONENTS All probability to topic w i P( z = ) = P( z = ) = 0 TOPIC MIXTURE P( w z ) HEART 0. LOVE 0. SOUL 0. TEARS 0. MYSTERY 0. JOY 0. P( w z ) SCIENTIFIC 0. KNOWLEDGE 0. WORK 0. RESEARCH 0. MATHEMATICS 0. MYSTERY 0. MIXTURE COMPONENTS Document: HEART, LOVE, JOY, SOUL, HEART,.
All probability to topic w i P( z = ) = 0 P( z = ) = TOPIC MIXTURE P( w z ) HEART 0. LOVE 0. SOUL 0. TEARS 0. MYSTERY 0. JOY 0. P( w z ) SCIENTIFIC 0. KNOWLEDGE 0. WORK 0. RESEARCH 0. MATHEMATICS 0. MYSTERY 0. MIXTURE COMPONENTS Document: SCIENTIFIC, KNOWLEDGE, SCIENTIFIC, RESEARCH,. Mixing topic and w i P( z = ) = 0.5 P( z = ) = 0.5 TOPIC MIXTURE P( w z ) HEART 0. LOVE 0. SOUL 0. TEARS 0. MYSTERY 0. JOY 0. P( w z ) SCIENTIFIC 0. KNOWLEDGE 0. WORK 0. RESEARCH 0. MATHEMATICS 0. MYSTERY 0. MIXTURE COMPONENTS Document: LOVE, SCIENTIFIC, HEART, SOUL, KNOWLEDGE, RESEARCH,.
Application to corpus data TASA corpus: text from first grade to college representative sample of text 6,000+ word types (stop words removed) 7,000+ documents 6,000,000+ word tokens A selection from 500 topics THEORY SCIENTISTS EXPERIMENT OBSERVATIONS SCIENTIFIC EXPERIMENTS HYPOTHESIS EXPLAIN SCIENTIST OBSERVED EXPLANATION BASED OBSERVATION IDEA EVIDENCE THEORIES BELIEVED DISCOVERED OBSERVE FACTS SPACE EARTH MOON PLANET ROCKET MARS ORBIT ASTRONAUTS FIRST SPACECRAFT JUPITER SATELLITE SATELLITES ATMOSPHERE SPACESHIP SURFACE SCIENTISTS ASTRONAUT SATURN MILES ART PAINT ARTIST PAINTING PAINTED ARTISTS MUSEUM WORK PAINTINGS STYLE PICTURES WORKS OWN SCULPTURE PAINTER ARTS BEAUTIFUL DESIGNS PORTRAIT PAINTERS STUDENTS TEACHER STUDENT TEACHERS TEACHING CLASS CLASSROOM SCHOOL LEARNING PUPILS CONTENT INSTRUCTION TAUGHT GROUP GRADE SHOULD GRADES CLASSES PUPIL GIVEN BRAIN NERVE SENSE SENSES ARE NERVOUS NERVES BODY SMELL TASTE TOUCH MESSAGES IMPULSES CORD ORGANS SPINAL FIBERS SENSORY PAIN IS CURRENT ELECTRICITY ELECTRIC CIRCUIT IS ELECTRICAL VOLTAGE FLOW BATTERY WIRE WIRES SWITCH CONNECTED ELECTRONS RESISTANCE POWER CONDUCTORS CIRCUITS TUBE NEGATIVE
Polysemy: words with multiple meanings represented in different topics FIELD MAGNETIC MAGNET WIRE NEEDLE CURRENT COIL POLES IRON COMPASS LINES CORE ELECTRIC DIRECTION FORCE MAGNETS BE MAGNETISM POLE INDUCED SCIENCE STUDY SCIENTISTS SCIENTIFIC KNOWLEDGE WORK RESEARCH CHEMISTRY TECHNOLOGY MANY MATHEMATICS BIOLOGY FIELD PHYSICS LABORATORY STUDIES WORLD SCIENTIST STUDYING SCIENCES BALL GAME TEAM FOOTBALL BASEBALL PLAYERS PLAY FIELD PLAYER BASKETBALL COACH PLAYED PLAYING HIT TENNIS TEAMS GAMES SPORTS BAT TERRY JOB WORK JOBS CAREER EXPERIENCE EMPLOYMENT OPPORTUNITIES WORKING TRAINING SKILLS CAREERS POSITIONS FIND POSITION FIELD OCCUPATIONS REQUIRE OPPORTUNITY EARN ABLE No Problem of Triangle Inequality TOPIC TOPIC SOCCER FIELD MAGNETIC Topic structure easily explains violations of triangle inequality 5
How to get vector representations Multidimensional scaling on similarity ratings Tversky s (977) contrast model Latent Semantic Analysis (Landauer & Dumais, 997) Topics Model (e.g., Griffiths & Steyvers, 00) 6