Outline Theory-sed Byesin frmework for property indution Cusl struture indution Constrint-sed (ottom-up) lerning Theory-sed Byesin lerning
The origins of usl knowledge Question: how do people relily ome to true eliefs out the usl struture of their world? Answer must speify: Prior usl knowledge Cusl inferene proedure
Desriptive: Multiple gols Prior knowledge must e psyhologilly relisti. Inferene proedure must generte the sme eliefs tht people do, given the sme input. Explntory: Prior knowledge must e pproximtely orret. Inferene proedure (onstrined y prior knowledge) must e relile.
Anlogy with vision (Perl, Cheng, Gopnik et l.) Externl world struture Vision (inverse grphis) Grphis Oserved imges
The fundmentl prolem Hidden usl struture: A B Cusl indution Oserved dt: C E D Cusl struture uses oservtions Cse A B C D E 1 0 1 1 1 1 2 1 0 1 0 1 3 0 0 0 1 0 4 0 1 1 0 1....
Under-onstrined prolems In oth visul pereption nd usl indution, mny world strutures ould hve produed the sme dt. Imge removed due to opyright onsidertions. Plese see: Freemn, WT. "The Generi Viewpoint Assumption in Frmework for Visul Pereption." Nture 368 (7 April 1994): 542-545. Imge Possile world strutures
Under-onstrined prolems In oth visul pereption nd usl indution, mny world strutures ould hve produed the sme dt. A B A B P ( A, B ) z P ( A ) P ( B ) X X A B A B A B Correltion Possile world strutures
Questions in visul pereption How is the externl world represented? 3-D models 2-D views Intermedite: 2 1/2-D sketh, lyers, intrinsi imges, et. Wht kind of knowledge does the mind hve out the world? truture of ojets Physis of surfes ttistis of senes How does inferene work? Bottom-up, modulr, ontext-free Top-down, flexile, ontext-sensitive
Questions in usl indution How is the externl world represented? Assoitions Cusl strutures Intermedite: Cusl strength prmeters Wht kind of knowledge does the mind hve out the world? Constrints on usl struture (e.g., usl order) Fithfulness (oserved independene reltions re rel) Cusl mehnisms How does inferene work? Bottom-up: onstrint-sed (dt mining) pproh Top-down: theory-sed Byesin pproh
ome voulry Cusl struture peifies nothing out usl mehnisms or Wht uses wht. prmeteriztions. A B A B C D vs. C D E E
ome voulry Cusl struture Wht uses wht. Cusl mehnism How uses influene effets. C X D C D E E
ome voulry Cusl struture Wht uses wht. Cusl mehnism How uses influene effets. C X D C D E E
ome voulry Cusl struture Wht uses wht. Cusl mehnism How uses influene effets. C X D C D E E E f ( C, D )
ome voulry Cusl struture Wht uses wht. Cusl mehnism How uses influene effets. C X D C D E E E f ( C, D, İ) İ ~ Gussin(µ, ı)
ome voulry Cusl struture Wht uses wht. Cusl mehnism How uses influene effets. Knowledge out usl strutures nd mehnisms n e represented t different sles of detil. Astrt ( light ) mehnism knowledge will e prtiulrly importnt: e.g., - deterministi, qusi-deterministi, semi-deterministi or stohsti? - strong or wek? - genertive or preventive influene? - independent of or intertive with other uses?
ome voulry Cusl struture Wht uses wht. Cusl mehnism How uses influene effets. Prmeteriztion Form of P(effet uses), e.g. noisy-or Cusl strengths (prmeters) Reltive ontriutions of different uses given prtiulr mehnism or prmeteriztion.
Approhes to struture lerning Constrint-sed lerning (Perl, Glymour, Gopnik): Assume struture is unknown, no knowledge of prmeteriztion or prmeters Byesin lerning (Hekermn, Friedmn/Koller): Assume struture is unknown, ritrry prmeteriztion. Theory-sed Byesin inferene (T & G): Assume struture is prtilly unknown, prmeteriztion is known ut prmeters my not e. Prior knowledge out struture nd prmeteriztion depends on domin theories (derived from ontology nd mehnisms).
Approhes to struture lerning Constrint-sed lerning (Perl, Glymour, Gopnik): Assume struture is unknown, no knowledge of prmeteriztion or prmeters Byesin lerning (Hekermn, Friedmn/Koller): Assume struture is unknown, ritrry prmeteriztion. Theory-sed Byesin inferene (T & G): Assume struture is prtilly unknown, prmeteriztion is known ut prmeters my not e. Prior knowledge out struture nd prmeteriztion depends on domin theories (derived from ontology nd mehnisms).
Cusl inferene in siene tndrd question: is X diret use of? tndrd empiril methodologies in mny domins: Psyhology Mediine Epidemiology Eonomis Biology Constrint-sed inferene ttempts to formlize this methodology.
Constrint-sed lerning Cusl grph: A B C D Fithfulness ssumption E Proility distriution: P ( A, B, C, D, E ) P ( V prents [ V ]) V { A, B, C, D, E } Cusl Mrkov ssumption P ( A, B, C, D, E ) P ( A ) P ( B ) P ( C A, B ) P ( D B ) P ( E C,D )
Definition of use Under the usl Mrkov priniple, A is diret use of B implies tht when ll other potentilly relevnt vriles re held onstnt, the proility of B depends upon the presene or sene of A. Under the fithfulness ssumption, (in)dependene nd onditionl (in)dependene reltions in the oserved dt imply onstrints on the hidden usl struture (see piture).
Exmple Wht is the usl struture relting smoking (), yellow teeth (), nd lung ner ()? Epidemiologil Dt: Ptient moking? ellow teeth? ung Cner? 1 yes yes yes 2 yes yes no 3 yes no yes 4 no no no 5 yes yes yes 6 yes no no 7 yes no yes 8 no no no....
Full Common Effet Common Cuse Chin One link Empty
Inferene proess A hypothesis:
Inferene proess A hypothesis: Wht evidene would support this hypothesis? Would tht evidene e onsistent with ny other hypothesis?
Exmple Wht is the usl struture relting smoking (), yellow teeth (), nd lung ner ()? Expeted simple orreltions: smoking, yellow teeth: yes smoking, lung ner: yes yellow teeth, lung ner: yes Expeted prtil (onditionl) orreltions: smoking, yellow teeth lung ner: yes smoking, lung ner yellow teeth: yes yellow teeth, lung ner smoking: no
Exmple Wht is the usl struture relting smoking (), yellow teeth (), nd lung ner ()? Expeted simple orreltions: smoking, yellow teeth: yes smoking, lung ner: yes yellow teeth, lung ner: yes Under fithfulness, two vriles tht re orrelted must shre ommon nestor. In this exmple, eh pir of nodes must shre ommon nestor.
Common Effet Chin Common Cuse n Ch Full One link Empty
Glol semntis Joint proility distriution ftorizes into produt of lol onditionl proilities: n P ( V 1,, V n ) P ( V i prents [V i ]) i 1 Burglry Erthquke Alrm JohnClls MryClls P ( B, E, A, J, M ) P ( B ) P ( E ) P ( A B, E ) P ( J A ) P (M A )
ol semntis Glol ftoriztion is equivlent to set of onstrints on pirwise reltionships etween vriles. Mrkov property : Eh node is onditionlly independent of its non-desendnts given its prents. U 1 U m Z 1j X Z nj 1 n Imge y MIT OCW.
ol semntis Glol ftoriztion is equivlent to set of onstrints on pirwise reltionships etween vriles. Eh node is onditionlly independent of ll others given its Mrkov lnket : prents, hildren, hildren s prents. U 1 U m Z 1j X Z nj 1 n Imge y MIT OCW.
Exmple Wht is the usl struture relting smoking, yellow teeth, nd lung ner? Expeted prtil (onditionl) orreltions: smoking, yellow teeth lung ner: yes smoking, lung ner yellow teeth: yes yellow teeth, lung ner smoking: no Under fithfulness: If two vriles nd re onditionlly independent given, then nd must not e in eh other s Mrkov lnket, nd must e in the Mrkov lnket of oth.
Common Effet Chin Common Cuse n Ch Full One link Empty
Empty Full Common Effet Chin Common Cuse Cn we distinguish etween the remining strutures? One link
The limits of onstrint-sed inferene Mrkov equivlene lss: A set of usl grphs tht nnot e distinguished sed on (in)dependene reltions. With two vriles, there re three possile usl grphs nd two equivlene lsses:
The limits of onstrint-sed inferene Mrkov equivlene lss: A set of usl grphs tht nnot e distinguished sed on (in)dependene reltions. With two vriles, there re three possile usl grphs nd two equivlene lsses: A nd B not independent. A nd B independent.
Full Common Effet Common Cuse Chin One link Empty
Full Common Effet One link Chin n Ch Common Cuse Empty
Additionl soures of onstrint Prior knowledge out usl struture Temporl order Domin-speifi onstrints Interventions Exogenously lmp one or more vriles to some known vlue, nd oserve other vriles over series of ses.
Interventions Exmple: Fore smple of sujets to smoke. Idel interventions lok ll other diret uses of the mnipulted vrile:
Interventions Exmple: Fore smple of sujets to smoke, nd nother smple to not smoke. Idel interventions lok ll other diret uses of the mnipulted vrile: I I I
Interventions Exmple: Fore smple of sujets to smoke, nd nother smple to not smoke. Non-idel interventions simply dd n extr use tht is under the lerner s ontrol: I I I
Advntges of the onstrint- Dedutive Domin-generl sed pproh No essentil role for domin knowledge: Knowledge of possile usl strutures not needed. Knowledge of possile usl mehnisms not used.
Disdvntges of the onstrint- Dedutive Domin-generl sed pproh No essentil role for domin knowledge: Knowledge of possile usl strutures not needed. Knowledge of possile usl mehnisms not used. Requires lrge smple sizes to mke relile inferenes.
Exmple Wht is the usl struture relting smoking, yellow teeth, nd lung ner? Epidemiologil Dt: Ptient moking? ellow teeth? ung Cner? 1 yes yes yes 2 yes yes no 3 yes no yes 4 no no no 5 yes yes yes 6 yes no no 7 yes no yes 8 no no no....
Computing (in)dependene tndrd methods sed on F 2 test: V=0 V=1 U=0 U=1 d 2 Ȥ 2 ( d )( u d u ) ( )( d )( )( d ) signifintly > 0: not independent not signifintly > 0: independent
Computing (in)dependene Are smoking nd yellow teeth independent? =0 =1 =0 2 0 =1 3 3 F2 = 1.6, p = 0.21
Computing (in)dependene Are smoking nd lung ner independent? =0 =1 =0 2 0 =1 2 4 F2 = 2.67, p = 0.10
Computing (in)dependene Are lung ner nd yellow teeth onditionlly independent given smoking? =1 =0 =1 =0 =0 =1 =0 1 2 =0 2 0 =1 1 2 =1 0 0 F2 = 0, p = 1.0 F 2 = undefined
Disdvntges of the onstrint- Dedutive Domin-generl sed pproh No essentil role for domin knowledge: Knowledge of possile usl strutures not needed. Knowledge of possile usl mehnisms not used. Requires lrge smple sizes to mke relile inferenes.
The Bliket detetor Imge removed due to opyright onsidertions. Plese see: Gopnik, A., nd D. M. oel. "Deteting Blikets: How oung Children use Informtion out Novel Cusl Powers in Ctegoriztion nd Indution." Child Development 71 (2000): 1205-1222.
Imge removed due to opyright onsidertions. Plese see: Gopnik, A., nd D. M. oel. "Deteting Blikets: How oung Children use Informtion out Novel Cusl Powers in Ctegoriztion nd Indution." Child Development 71 (2000): 1205-1222.
The Bliket detetor Cn we explin these inferenes using onstrint-sed lerning? Wht other explntions n we ome up with?