Problem sets. Late policy (5% off per day, but the weekend counts as only one day). E.g., Friday: -5% Monday: -15% Tuesday: -20% Thursday: -30%

Transcription

1 Problem sets Late policy (5% off per day, but the weekend counts as only one day). E.g., Friday: -5% Monday: -5% Tuesday: -0% Thursday: -30%

2 Outline Final thoughts on hierarchical Bayesian models and MCMC Bayesian classification Bayesian concept learning

3 Gibbs sampling MCMC methods Factorize hypotheses h = <h, h,, h n > Cycle through variables h, h,, h n Draw h i (t+) from P(h i h -i, evidence) Metropolis-Hastings Propose changes to hypothesis from some distribution Q(h (t+) h (t) ) Accept proposals with probability A(h (t+) h (t) ) = min{, } P(h (t+) evidence) Q(h (t) h (t +) ) P(h (t) evidence) Q(h (t+) h (t) )

4 Why MCMC is important Simple Can be used with just about any kind of probabilistic model, including complex hierarchical structures Always works pretty well, if you re willing to wait a long time (cf. Back-propagation for neural networks.)

5 A model for cognitive development? Some features of cognitive development: Small, random, dumb, local steps Takes a long time Can get stuck in plateaus or stages Two steps forward, one step back Over time, intuitive theories get consistently better (more veridical, more powerful, broader scope). Everyone reaches basically the same state (though some take longer than others).

6 Topic models of semantic structure: e.g., Latent Dirichlet Allocation (Blei, Ng, Jordan) Each document in a corpus is associated with a distribution θ over topics. Each topic t is associated with a distribution φ(t) over words. Image removed due to copyright considerations. Please see: Blei, David, Andrew Ng, and Michael Jordan. "Latent Dirichlet Allocation." Journal of Machine Learning Research 3 (Jan 003):

7 Choose mixture weights for each document, generate bag of words θ = {P(z = ), P(z = )} {0, } {0.5, 0.75} {0.5, 0.5} {0.75, 0.5} {, 0} MATHEMATICS KNOWLEDGE RESEARCH WORK MATHEMATICS RESEARCH WORK SCIENTIFIC MATHEMATICS WORK SCIENTIFIC KNOWLEDGE MATHEMATICS SCIENTIFIC HEART LOVE TEARS KNOWLEDGE HEART MATHEMATICS HEART RESEARCH LOVE MATHEMATICS WORK TEARS SOUL KNOWLEDGE HEART WORK JOY SOUL TEARS MATHEMATICS TEARS LOVE LOVE LOVE SOUL TEARS LOVE JOY SOUL LOVE TEARS SOUL SOUL TEARS JOY

8

9 Gibbs sampling i w i d i z i MATHEMATICS KNOWLEDGE RESEARCH WORK MATHEMATICS RESEARCH WORK SCIENTIFIC MATHEMATICS WORK SCIENTIFIC KNOWLEDGE... JOY... 5 iteration...

10 Gibbs sampling iteration i w i d i z i z i MATHEMATICS KNOWLEDGE RESEARCH WORK MATHEMATICS RESEARCH WORK SCIENTIFIC MATHEMATICS WORK SCIENTIFIC KNOWLEDGE... JOY ?

11

12

13

14

15

16

17

18 A selection of topics (TASA) FIELD MAGNETIC MAGNET WIRE NEEDLE CURRENT COIL POLES IRON COMPASS LINES CORE ELECTRIC DIRECTION FORCE MAGNETS BE MAGNETISM POLE INDUCED SCIENCE STUDY SCIENTISTS SCIENTIFIC KNOWLEDGE WORK RESEARCH CHEMISTRY TECHNOLOGY MANY MATHEMATICS BIOLOGY FIELD PHYSICS LABORATORY STUDIES WORLD SCIENTIST STUDYING SCIENCES BALL GAME TEAM FOOTBALL BASEBALL PLAYERS PLAY FIELD PLAYER BASKETBALL COACH PLAYED PLAYING HIT TENNIS TEAMS GAMES SPORTS BAT TERRY JOB WORK JOBS CAREER EXPERIENCE EMPLOYMENT OPPORTUNITIES WORKING TRAINING SKILLS CAREERS POSITIONS FIND POSITION FIELD OCCUPATIONS REQUIRE OPPORTUNITY EARN ABLE STORY STORIES TELL CHARACTER CHARACTERS AUTHOR READ TOLD SETTING TALES PLOT TELLING SHORT FICTION ACTION TRUE EVENTS TELLS TALE NOVEL MIND WORLD DREAM DREAMS THOUGHT IMAGINATION MOMENT THOUGHTS OWN REAL LIFE IMAGINE SENSE CONSCIOUSNESS STRANGE FEELING WHOLE BEING MIGHT HOPE WATER FISH SEA SWIM SWIMMING POOL LIKE SHELL SHARK TANK SHELLS SHARKS DIVING DOLPHINS SWAM LONG SEAL DIVE DOLPHIN UNDERWATER DISEASE BACTERIA DISEASES GERMS FEVER CAUSE CAUSED SPREAD VIRUSES INFECTION VIRUS MICROORGANISMS PERSON INFECTIOUS COMMON CAUSING SMALLPOX BODY INFECTIONS CERTAIN

19 A selection of topics (TASA) FIELD MAGNETIC MAGNET WIRE NEEDLE CURRENT COIL POLES IRON COMPASS LINES CORE ELECTRIC DIRECTION FORCE MAGNETS BE MAGNETISM POLE INDUCED SCIENCE STUDY SCIENTISTS SCIENTIFIC KNOWLEDGE WORK RESEARCH CHEMISTRY TECHNOLOGY MANY MATHEMATICS BIOLOGY FIELD PHYSICS LABORATORY STUDIES WORLD SCIENTIST STUDYING SCIENCES BALL GAME TEAM FOOTBALL BASEBALL PLAYERS PLAY FIELD PLAYER BASKETBALL COACH PLAYED PLAYING HIT TENNIS TEAMS GAMES SPORTS BAT TERRY JOB WORK JOBS CAREER EXPERIENCE EMPLOYMENT OPPORTUNITIES WORKING TRAINING SKILLS CAREERS POSITIONS FIND POSITION FIELD OCCUPATIONS REQUIRE OPPORTUNITY EARN ABLE STORY STORIES TELL CHARACTER CHARACTERS AUTHOR READ TOLD SETTING TALES PLOT TELLING SHORT FICTION ACTION TRUE EVENTS TELLS TALE NOVEL MIND WORLD DREAM DREAMS THOUGHT IMAGINATION MOMENT THOUGHTS OWN REAL LIFE IMAGINE SENSE CONSCIOUSNESS STRANGE FEELING WHOLE BEING MIGHT HOPE WATER FISH SEA SWIM SWIMMING POOL LIKE SHELL SHARK TANK SHELLS SHARKS DIVING DOLPHINS SWAM LONG SEAL DIVE DOLPHIN UNDERWATER DISEASE BACTERIA DISEASES GERMS FEVER CAUSE CAUSED SPREAD VIRUSES INFECTION VIRUS MICROORGANISMS PERSON INFECTIOUS COMMON CAUSING SMALLPOX BODY INFECTIONS CERTAIN

20 The 4 shape 7 of 4 a 3 female 5 mating 5 preference 5 is 3 the 4 relationship 7 between 4 a 3 male 5 trait 5 and 37 the 4 probability 7 of 4 acceptance as 43 a 3 mating 5 partner 0, The 4 shape 7 of 4 preferences 5 is 3 important 49 in 5 many 39 models 6 of 4 sexual 5 selection 46, mate 5 recognition 5, communication 9, and 37 speciation 46, yet 50 it 4 has 8 rarely 9 been 33 measured 7 precisely 9, Here I 9 examine 34 preference 7 shape 7 for 5 male 5 calling 5 song 5 in a 3 bushcricket *3 (katydid *48 ). Preferences 5 change 46 dramatically 9 between races 46 of 4 a 3 species 5, from strongly 9 directional to 3 broadly 9 stabilizing 45 (but 50 with a 3 net 49 directional 46 effect 46 ), Preference 5 shape 46 generally 9 matches 0 the 4 distribution 6 of 4 the 4 male 5 trait 5, This 4 is 3 compatible 9 with a 3 coevolutionary 46 model 0 of 4 signal 9 -preference 5 evolution 46, although 50 it 4 does 33 not 37 rule 0 out 7 an 3 alternative model 0, sensory 5 exploitation 50. Preference 46 shapes 40 are 8 shown 35 to 3 be 44 genetic in 5 origin 7. (graylevel = membership in topic 5) Ritchie, Michael G. "The Shape of Female Mating Preferences." PNAS 93 (996): Copyright 996. Courtesy of the National Academy of Sciences, U.S.A. Used with permission.

21 The 4 shape 7 of 4 a 3 female 5 mating 5 preference 5 is 3 the 4 relationship 7 between 4 a 3 male 5 trait 5 and 37 the 4 probability 7 of 4 acceptance as 43 a 3 mating 5 partner 0, The 4 shape 7 of 4 preferences 5 is 3 important 49 in 5 many 39 models 6 of 4 sexual 5 selection 46, mate 5 recognition 5, communication 9, and 37 speciation 46, yet 50 it 4 has 8 rarely 9 been 33 measured 7 precisely 9, Here I 9 examine 34 preference 7 shape 7 for 5 male 5 calling 5 song 5 in a 3 bushcricket *3 (katydid *48 ). Preferences 5 change 46 dramatically 9 between races 46 of 4 a 3 species 5, from strongly 9 directional to 3 broadly 9 stabilizing 45 (but 50 with a 3 net 49 directional 46 effect 46 ), Preference 5 shape 46 generally 9 matches 0 the 4 distribution 6 of 4 the 4 male 5 trait 5, This 4 is 3 compatible 9 with a 3 coevolutionary 46 model 0 of 4 signal 9 -preference 5 evolution 46, although 50 it 4 does 33 not 37 rule 0 out 7 an 3 alternative model 0, sensory 5 exploitation 50. Preference 46 shapes 40 are 8 shown 35 to 3 be 44 genetic in 5 origin 7. (graylevel = membership in topic 5, 46) Ritchie, Michael G. "The Shape of Female Mating Preferences." PNAS 93 (996): Copyright 996. Courtesy of the National Academy of Sciences, U.S.A. Used with permission.

22 The 4 shape 7 of 4 a 3 female 5 mating 5 preference 5 is 3 the 4 relationship 7 between 4 a 3 male 5 trait 5 and 37 the 4 probability 7 of 4 acceptance as 43 a 3 mating 5 partner 0, The 4 shape 7 of 4 preferences 5 is 3 important 49 in 5 many 39 models 6 of 4 sexual 5 selection 46, mate 5 recognition 5, communication 9, and 37 speciation 46, yet 50 it 4 has 8 rarely 9 been 33 measured 7 precisely 9, Here I 9 examine 34 preference 7 shape 7 for 5 male 5 calling 5 song 5 in a 3 bushcricket *3 (katydid *48 ). Preferences 5 change 46 dramatically 9 between races 46 of 4 a 3 species 5, from strongly 9 directional to 3 broadly 9 stabilizing 45 (but 50 with a 3 net 49 directional 46 effect 46 ), Preference 5 shape 46 generally 9 matches 0 the 4 distribution 6 of 4 the 4 male 5 trait 5, This 4 is 3 compatible 9 with a 3 coevolutionary 46 model 0 of 4 signal 9 -preference 5 evolution 46, although 50 it 4 does 33 not 37 rule 0 out 7 an 3 alternative model 0, sensory 5 exploitation 50. Preference 46 shapes 40 are 8 shown 35 to 3 be 44 genetic in 5 origin 7. (graylevel = membership in topic 5, 46, 5) Ritchie, Michael G. "The Shape of Female Mating Preferences." PNAS 93 (996): Copyright 996. Courtesy of the National Academy of Sciences, U.S.A. Used with permission.

23 Joint models of syntax and semantics (Griffiths, Steyvers, Blei & Tenenbaum, NIPS 004) Embed topics model inside an nth order Hidden Markov Model: Image removed due to copyright considerations. Please see: Griffiths, T. L., M. Steyvers, D. M. Blei, and J. B. Tenenbaum. "Integrating Topics and Syntax." Advances in Neural Information Processing Systems 7 (005).

24 Image removed due to copyright considerations. Please see: Griffiths, T. L., M. Steyvers, D. M. Blei, and J. B. Tenenbaum. "Integrating Topics and Syntax." Advances in Neural Information Processing Systems 7 (005). Semantic classes FOOD FOODS BODY NUTRIENTS DIET FAT SUGAR ENERGY MILK EATING FRUITS VEGETABLES WEIGHT FATS NEEDS CARBOHYDRATES VITAMINS CALORIES PROTEIN MINERALS MAP NORTH EARTH SOUTH POLE MAPS EQUATOR WEST LINES EAST AUSTRALIA GLOBE POLES HEMISPHERE LATITUDE PLACES LAND WORLD COMPASS CONTINENTS DOCTOR PATIENT HEALTH HOSPITAL MEDICAL CARE PATIENTS NURSE DOCTORS MEDICINE NURSING TREATMENT NURSES PHYSICIAN HOSPITALS DR SICK ASSISTANT EMERGENCY PRACTICE BOOK BOOKS READING INFORMATION LIBRARY REPORT PAGE TITLE SUBJECT PAGES GUIDE WORDS MATERIAL ARTICLE ARTICLES WORD FACTS AUTHOR REFERENCE NOTE GOLD IRON SILVER COPPER METAL METALS STEEL CLAY LEAD ADAM ORE ALUMINUM MINERAL MINE STONE MINERALS POT MINING MINERS TIN BEHAVIOR SELF INDIVIDUAL PERSONALITY RESPONSE SOCIAL EMOTIONAL LEARNING FEELINGS PSYCHOLOGISTS INDIVIDUALS PSYCHOLOGICAL EXPERIENCES ENVIRONMENT HUMAN RESPONSES BEHAVIORS ATTITUDES PSYCHOLOGY PERSON CELLS CELL ORGANISMS ALGAE BACTERIA MICROSCOPE MEMBRANE ORGANISM FOOD LIVING FUNGI MOLD MATERIALS NUCLEUS CELLED STRUCTURES MATERIAL STRUCTURE GREEN MOLDS PLANTS PLANT LEAVES SEEDS SOIL ROOTS FLOWERS WATER FOOD GREEN SEED STEMS FLOWER STEM LEAF ANIMALS ROOT POLLEN GROWING GROW

25 Image removed due to copyright considerations. Please see: Griffiths, T. L., M. Steyvers, D. M. Blei, and J. B. Tenenbaum. "Integrating Topics and Syntax." Advances in Neural Information Processing Systems 7 (005). Syntactic classes SAID ASKED THOUGHT TOLD SAYS MEANS CALLED CRIED SHOWS ANSWERED TELLS REPLIED SHOUTED EXPLAINED LAUGHED MEANT WROTE SHOWED BELIEVED WHISPERED THE HIS THEIR YOUR HER ITS MY OUR THIS THESE A AN THAT NEW THOSE EACH MR ANY MRS ALL MORE SUCH LESS MUCH KNOWN JUST BETTER RATHER GREATER HIGHER LARGER LONGER FASTER EXACTLY SMALLER SOMETHING BIGGER FEWER LOWER ALMOST ON AT INTO FROM WITH THROUGH OVER AROUND AGAINST ACROSS UPON TOWARD UNDER ALONG NEAR BEHIND OFF ABOVE DOWN BEFORE GOOD SMALL NEW IMPORTANT GREAT LITTLE LARGE * BIG LONG HIGH DIFFERENT SPECIAL OLD STRONG YOUNG COMMON WHITE SINGLE CERTAIN ONE SOME MANY TWO EACH ALL MOST ANY THREE THIS EVERY SEVERAL FOUR FIVE BOTH TEN SIX MUCH TWENTY EIGHT HE YOU THEY I SHE WE IT PEOPLE EVERYONE OTHERS SCIENTISTS SOMEONE WHO NOBODY ONE SOMETHING ANYONE EVERYBODY SOME THEN BE MAKE GET HAVE GO TAKE DO FIND USE SEE HELP KEEP GIVE LOOK COME WORK MOVE LIVE EAT BECOME

26 Corpus-specific factorization (NIPS) Image removed due to copyright considerations. Please see: Griffiths, T. L., M. Steyvers, D. M. Blei, and J. B. Tenenbaum. "Integrating Topics and Syntax." Advances in Neural Information Processing Systems 7 (005).

27 Syntactic classes in PNAS 5 IN FOR ON BETWEEN DURING AMONG FROM UNDER WITHIN THROUGHOUT THROUGH TOWARD INTO AT INVOLVING AFTER ACROSS AGAINST WHEN ALONG 8 ARE WERE WAS IS WHEN REMAIN REMAINS REMAINED PREVIOUSLY BECOME BECAME BEING BUT GIVE MERE APPEARED APPEAR ALLOWED NORMALLY EACH 4 THE THIS ITS THEIR AN EACH ONE ANY INCREASED EXOGENOUS OUR RECOMBINANT ENDOGENOUS TOTAL PURIFIED TILE FULL CHRONIC ANOTHER EXCESS 5 SUGGEST INDICATE SUGGESTING SUGGESTS SHOWED REVEALED SHOW DEMONSTRATE INDICATING PROVIDE SUPPORT INDICATES PROVIDES INDICATED DEMONSTRATED SHOWS SO REVEAL DEMONSTRATES SUGGESTED 6 LEVELS NUMBER LEVEL RATE TIME CONCENTRATIONS VARIETY RANGE CONCENTRATION DOSE FAMILY SET FREQUENCY SERIES AMOUNTS RATES CLASS VALUES AMOUNT SITES 30 RESULTS ANALYSIS DATA STUDIES STUDY FINDINGS EXPERIMENTS OBSERVATIONS HYPOTHESIS ANALYSES ASSAYS POSSIBILITY MICROSCOPY PAPER WORK EVIDENCE FINDING MUTAGENESIS OBSERVATION MEASUREMENTS REMAINED 33 BEEN MAY CAN COULD WELL DID DOES DO MIGHT SHOULD WILL WOULD MUST CANNOT THEY ALSO BECOME MAG LIKELY

28 Semantic highlighting Darker words are more likely to have been generated from the topic-based semantics module:

29 Outline Final thoughts on hierarchical Bayesian models and MCMC Bayesian classification Bayesian concept learning

30 Concepts and categories A category is a set of objects that are treated equivalently for some purpose. A concept is a mental representation of the category. Functions for concepts: Categorization/classification Prediction Inductive generalization Explanation Reference in communication and thought

31 Classical view of concepts (950 s-960 s): Concepts are rules or symbolic representations for classifying. Examples Psychology: Bruner et al. "Striped and Three Borders": Conjunctive Concept Figure by MIT OCW.

32 Classical view of concepts (950 s-960 s): Concepts are rules or symbolic representations Examples AI: Winston s arch learner Image removed due to copyright considerations. Please see: Winston, P. H., ed. The Psychology of Computer Vision. New York, NY: McGaw-Hill, 975. ISBN:

33 Statistical view of concepts (960 s-970 s) Examples Machine learning/statistics: Iris classification Images removed due to copyright considerations.

34 Standard version (960 s-970 s): Concepts are statistical representations for classifying. Examples Psychology: Posner and Keele Image removed due to copyright considerations. Please see: Posner, M. I., and S. W. Keele. "On the Genesis of Abstract Ideas." Journal of Experimental Psychology 77 (968):

35 Different levels of random distortion: Images removed due to copyright considerations.

36 Statistical pattern recognition Two-class classification problem: Images removed due to copyright considerations. The task: Given an object generated from class or class, infer the generating class.

37 Formalizing two-class classification: Images removed due to copyright considerations. The task: Observe x generated from c or c, compute: ) ( ) ( ) ( ) ( ) ( ) ( ) ( c p c x p c p c x p c p c x p x c p + = Different approaches vary in how they represent p(x c j ).

38 Parametric approach Assume a simple canonical form for p(x c j ). E.g., Gaussian distributions: Images removed due to copyright considerations.

39 Parametric approach Assume a simple canonical form for p(x c j ). The simplest Gaussians have all dimensions independent, variances equal for all classes: Classification based on distance to means. Covariance ellipse determines the distance metric.

40 Parametric approach p Assume a simple canonical form for p(x c j ). The simplest Gaussians have all dimensions independent, variances equal for all classes: Bayes net representation: C x x ( x c j ) = p( x c j ) p( x c j ) p ( x ij ) i ( x c ) e i µ i j naïve Bayes /(σ )

41 Parametric approach Other possible forms: All dimensions independent with variances equal across dimensions and classes: C x x naïve Bayes p ( x c j ) = p( x c j ) p( x p( x i c j ) e ( x i j µ ij ) /(σ ) c )

42 Parametric approach Other possible forms: All dimensions independent with equal variances, but variances differ across classes: C x x naïve Bayes p ( x c j ) = p( x c j ) p( x p j ( x ij ) j ( x c ) e i µ i j /(σ ) c )

43 Parametric approach Other possible forms: All dimensions independent, variances differ across dimensions and across classes: C x x naïve Bayes p ( x c j ) = p( x c j ) p( x p j ( x ij ) ij ( x c ) e i µ i j /(σ ) c )

44 Parametric approach Other possible forms: Arbitrary covariance matrices for each class. C x = {x, x } Board formula

45 Parametric approach p Assume a simple canonical form for p(x c j ). The simplest Gaussians have all dimensions independent, variances equal for all classes: Bayes net representation: C x x ( x c j ) = p( x c j ) p( x c j ) p ( x ij ) i ( x c ) e i µ i j naïve Bayes /(σ )

46 Learning Hypothesis space of possible Gaussians: Images removed due to copyright considerations. Find parameters that maximize likelihood of examples. µ r = mean of examples of class j. σij = standard deviation along dimension i, for examples in each class.

47 Relevance to human concept learning Natural categories often have Gaussian (or other simple parametric forms) in perceptual feature spaces. Prototype effects in categorization (Rosch) Posner & Keele studies of prototype abstraction in concept learning.

48 Posner and Keele: design Image removed due to copyright considerations. Please see: Posner, M. I., and S. W. Keele. "On the Genesis of Abstract Ideas." Journal of Experimental Psychology 77 (968):

49 Posner and Keele: results Image removed due to copyright considerations. Please see: Posner, M. I., and S. W. Keele. "On the Genesis of Abstract Ideas." Journal of Experimental Psychology 77 (968): Unseen prototype ( Schema ) classified as well as memorized variants, and much better than new random variants ( 5 ).

50 Parametric approach Other possible forms: All dimensions independent with variances equal across dimensions and classes: C x x naïve Bayes p ( x c j ) = p( x c j ) p( x j c ) p( x i c j ) e ( x i µ ij ) /(σ ) Equivalent to prototype model: r Prototype of class j: µ j = { µ j, µ Variability of categories: σ j }

51 Limitations Of this empirical paradigm? Of this computational approach?

52 Limitations Is categorization just discrimination among mutually exclusive classes? Overlapping concepts? Hierarchies? None of the above? Can we learn a single new concept? How do we learn concepts from just a few positive examples? Learning with high certainty from little data. Schema abstraction from one imperfect example. Are most categories Gaussian, or any simple parametric shape? What about superordinate categories? What about learning rule-based categories?

53 Limitations Is prototypicality = degree of membership? Armstrong et al.: No, for classical rule-based categories Not for complex real-world categories either: Christmas eve, Hollywood actress, Californian, Professor For natural kinds, huge variability in prototypicality independent of membership. Richer concepts? Meaningful stimuli, background knowledge, theories? Role of causal reasoning? Essentialism? Difference between perceptual and cognitive concepts?