Prof. Dr. Ralf Möller Dr. Özgür L. Özçep Universität zu Lübeck Institut für Informationssysteme. Tanya Braun (Exercises)

Transcription

1 Prof. Dr. Ralf Möller Dr. Özgür L. Özçep Universität zu Lübeck Institut für Informationssysteme Tanya Braun (Exercises)

2 Slides taken from the presentation (subset only) Learning Statistical Models From Relational Data Lise Getoor University of Maryland, College Park Includes work done by: Nir Friedman, Hebrew U. Daphne Koller, Stanford Avi Pfeffer, Harvard Ben Taskar, Stanford

3 Outline Motivation and Background PRMs w/ Attribute Uncertainty PRMs w/ Link Uncertainty PRMs w/ Class Hierarchies

4 Discovering Patterns in Structured Data Strain Contact Patient Treatment

5 Learning Statistical Models Traditional approaches work well with flat representations fixed length attribute-value vectors assume independent (IID) sample Problems: introduces statistical skew loses relational structure incapable of detecting link-based patterns must fix attributes in advance Patient flatten Contact

6 Probabilistic Relational Models Combine advantages of relational logic & Bayesian networks: natural domain modeling: objects, properties, relations; generalization over a variety of situations; compact, natural probability models. Integrate uncertainty with relational model: properties of domain entities can depend on properties of related entities; uncertainty over relational structure of domain.

7 Relational Schema Author Good Writer Smart Review Mood Length Author of Quality Accepted Has Review Describes the types of objects and relations in the database

8 Probabilistic Relational Model Author Smart Review Mood Good Writer Length Quality Accepted

9 Probabilistic Relational Model Author Smart Review Mood Good Writer Length P.Accepted.Quality,.Review.Mood Quality Accepted

10 Probabilistic Relational Model Author Smart Review Mood Good Writer Length Q, M f, f f, t t, f t, t P(A Q, M) Quality Accepted

11 Relational Skeleton Author A1 Author A2 P1 Author: A1 Review: R1 P2 Author: A1 Review: R2 P3 Author: A2 Review: R2 Primary Keys Review R1 Review R2 Review R2 Foreign Keys Fixed relational skeleton σ: l set of objects in each class l relations between them

12 PRM w/ Attribute Uncertainty Author A1 Smart Good Writer P1 Author: A1 Review: R1 Quality Accepted Review R1 Mood Length Author A2 Smart Good Writer P2 Author: A1 Review: R2 Quality Accepted Review R2 Mood Length P3 Author: A2 Review: R2 Quality Accepted Review R3 Mood Length PRM defines distribution over instantiations of attributes

13 P2.Accepted P2.Quality r2.mood P3.Accepted P3.Quality ,,,,, t t f t t f f f P(A Q, M) Q M Bad Low ,,,,, t t f t t f f f P(A Q, M) Q M r3.mood A Portion of the BN

14 A Portion of the BN P2.Quality Low r2.mood Bad Q, M f, f P(A Q, M) P2.Accepted f, t P3.Quality High r3.mood Bad t, f t, t P3.Accepted

15 P2.Accepted P2.Quality r2.mood P3.Accepted P3.Quality ,,,,, t t f t t f f f P(A Q, M) Q M Pissy Low ,,,,, t t f t t f f f P(A Q, M) Q M A Portion of the BN

16 A Portion of the BN P2.Quality Low r2.mood Pissy Q, M f, f P(A Q, M) P2.Accepted f, t t, f P3.Quality High t, t P3.Accepted

17 PRM: Aggregate Dependencies Quality Accepted Review Mood Length Review R1 Mood P1 Review R2 Length Mood Quality Accepted Review R3 Length Mood Length

18 PRM: Aggregate Dependencies Quality Accepted Review Mood Length P1 Quality Q, M f, f f, t Accepted t, f t, t P(A Q, M) mode Review R1 Mood Review R2 Length Mood Review R3 Length Mood sum, min, max, avg, max-occurrence, count Length

19 PRM with AU Semantics Author Review Author A1 Author A2 P1 P2 P3 Review R1 Review R2 Review R3 PRM + relational skeleton σ = probability distribution over completions I: P( I σ, S, Θ) = P( x. A parents S, x σ x. A Objects Attributes σ ( x. A))

20 Learning PRMs w/ AU Database Strain Patient Contact Strain Patient Contact Relational Schema Parameter estimation Structure selection

21 Parameter Estimation in PRMs Assume known dependency structure S Goal: estimate PRM parameters θ entries in local probability models, θ is good if it is likely to generate the observed data, instance I. l( θ : I, S) = log P( I S, θ) θ x. A parents ( x. A) MLE Principle: Choose θ * so as to maximize l

22 Learning PRMs w/ AU Database Author Review Author Review Relational Schema Parameter estimation Structure selection

23 Review Mood Length ML Parameter Estimation θ * = N Quality Accepted P. Q, R. M, P. A N P. Q, R. M Q, M f, f f, t t, f t, t P(A Q, M)???????? where N P. Q, R. M, P. A is the number of accepted, low quality papers whose reviewer was in a poor mood

24 Review Mood Length ML Parameter Estimation θ * = N Quality Accepted P. Q, R. M, P. A N Query for counts: P. Q, R. M Q, M f, f f, t t, f t, t P(A Q, M)???????? Count σ P.Q = q R.M = m P.A = m π P. Quality R. Mood P. Accepted Review table table

25 Idea: Structure Selection define scoring function do local search over legal structures Key Components: legal models scoring models searching model space

26 Idea: Structure Selection define scoring function do local search over legal structures Key Components:» legal models scoring models searching model space

27 Legal Models PRM defines a coherent probability model over a skeleton σ if the dependencies between object attributes are acyclic (prop. BN). Researcher Prof. Gump Reputation high sum author-of P1 Accepted yes P2 Accepted yes How do we guarantee that a PRM is acyclic for every skeleton?

28 Attribute Stratification PRM dependency structure S.Accecpted Researcher.Reputation dependency graph if Researcher.Reputation depends directly on.accepted Attribute stratification: dependency graph acyclic acyclic for any σ Algorithm more flexible; allows certain cycles along guaranteed acyclic relations

29 Blood Type (Father) Person Blood Type (Mother) Person P-chromosome M-chromosome P-chromosome M-chromosome P-chromosome Person M-chromosome Blood Type Contaminated Result Blood Test

30 Idea: Structure Selection define scoring function do local search over legal structures Key Components: legal models» scoring models searching model space

31 Scoring Models Bayesian approach: Score ( S : I) = log P( S I) marginal likelihood prior %"$"#! log[ P( I S) P( S)] Standard approach to scoring models; used in Bayesian network learning

32 Idea: Structure Selection define scoring function do local search over legal structures Key Components: legal models scoring models» searching model space

33 Searching Model Space Phase 0: consider only dependencies within a class Author Review Author Review Potential Parents( R. A) = R. B R. B descriptive attributes ( R ) Author Review

34 Phased Structure Search Phase 1: consider dependencies from neighboring classes, via schema relations Author Review Author Review Potential Parents( R. A) = S. C S. C descriptive attributes ( R S ) Author Review

35 Phased Structure Search Phase 2: consider dependencies from further classes, via relation chains Author Review Author Review Potential Parents( R. A) = T. D T. D descriptive attributes ( R S T ) Author Review

36 Issue PRM w/ AU applicable only in domains where we have full knowledge of the relational structure Next we introduce PRMs which allow uncertainty over relational structure

37 Kinds of structural uncertainty How many objects does an object relate to? how many Authors does 1 have? Which object is an object related to? does 1 cite 2 or 3? Which class does an object belong to? is 1 a JournalArticle or a Conference? Does an object actually exist? Are two objects identical?

38 Structural Uncertainty Motivation: PRM with AU only well-defined when the skeleton structure is known May be uncertain about relational structure itself Construct probabilistic models of relational structure that capture structural uncertainty Mechanisms: Reference uncertainty Existence uncertainty Number uncertainty Type uncertainty Identity uncertainty

39 PRMs w/ Link Uncertainty Advantages: Applicable in cases where we do not have full knowledge of relational structure Incorporating uncertainty over relational structure into probabilistic model can improve predictive accuracy Two approaches: Reference uncertainty Existence uncertainty Different probabilistic models; varying amount of background knowledge required for each

40 Citation Relational Schema Author Institution Research Area Wrote Word1 Word2 WordN Citing Cites Count Cited Word1 Word2 WordN

41 Attribute Uncertainty Author Research Area Institution P( Institution Research Area) Wrote P(.Author.Research Area P( WordN ) Word1... WordN

42 Reference Uncertainty Bibliography ? ` ? ? Scientific Document Collection

43 PRM w/ Reference Uncertainty Words Cites Cited Citing Words Dependency model for foreign keys Naïve Approach: multinomial over primary key noncompact limits ability to generalize Use attribute partition instead

44 Reference Uncertainty Example P5 P4 P3 M2 AI AI P1 AI AI Theory P5 AI P3 AI C1. = AI P4 P2 Theory Theory P1 Theory C2. = Theory Cites Citing Cited C1 C

45 Reference Uncertainty Example P5 P4 P3 M2 P1 AI AI AI AI Theory P5 AI P3 AI C1. = AI P4 P2 Theory Theory P1 Theory C2. = Theory Words Cites Citing Cited C1 C2 Theory AI C1 C

46 Introduce Selector RVs Cites1.Selector Cites1.Cited P2. P3. P1. P4. Cites2.Selector Cites2.Cited P5. P6. Introduce Selector RV, whose domain is {C1,C2} The distribution over Cited depends on all of the topics, and the selector

47 PRMs w/ RU Semantics Words Cites Cited Citing Words P2 P5 P4 Theory AI P3 P1 Theory AI??? Reg Reg Cites P2 P5 P4 Theory AI P3 P1 Theory AI??? PRM RU entity skeleton σ PRM-RU + entity skeleton σ probability distribution over full instantiations I

48 Learning PRMs w/ RU Idea: just like in PRMs w/ AU define scoring function do greedy local structure search Issues: expanded search space construct partitions new operators

49 Learning Idea: define scoring function do phased local search over legal structures Key Components: legal models model new dependencies scoring models PRMs w/ RU unchanged searching model space new operators

50 Legal Models Review Mood Important Accepted Cites Citing Cited Important Accepted

51 Legal Models Cites1.Selector Cites1.Cited P2.Important R1.Mood P3.Important P1.Accepted P4.Important When a node s parent is defined using an uncertain relation, the reference RV must be a parent of the node as well.

52 Structure Search Words Cites Citing Cited Words Author Institution Cited

53 Structure Search: New Operators Words Cites Citing Cited Words Author Institution Cited

54 Structure Search: New Operators Words Cites Citing Cited Words Author Institution Cited Institution = MIT = AI

55 PRMs w/ RU Summary Define semantics for uncertainty over foreign-key values Search now includes operators Refine and Abstract for constructing foreign-key dependency model Provides one simple mechanism for link uncertainty

56 Existence Uncertainty??? Document Collection Document Collection

57 PRM w/ Exists Uncertainty Words Cites Exists Words Dependency model for existence of relationship

58 Exists Uncertainty Example Words Cites Exists Words Citer. Cited. False True Theory Theory Theory AI AI Theory AI AI

59 Introduce Exists RVs Author #1 Area Inst Author #2 Area Inst #1 #2 #3 Word1 Word1 WordN WordN WordN Word1 Exists Exists Exists Exists Exists Exists #1-#3 #1-#2 #2-#1 #3-#1 #2-#3 #3-#2

60 Introduce Exists RVs Author #1 Area Inst Author #2 Area Inst #1 #2 #3 Word1 Word1 WordN WordN... WordN Word1 Exists #1-#3 Exists #1-#2 Exists Exists Exists Exists #2-#1 #3-#1 #2-#3 #3-#2

61 PRMs w/ EU Semantics Words Cites Exists Words P2 P5 P4 Theory AI P3 P1 Theory AI?????? P2 P5 P4 Theory AI P3 P1 Theory AI??? PRM EU object skeleton σ PRM-EU + object skeleton σ probability distribution over full instantiations I

62 Learning PRMs w/ EU Idea: just like in PRMs w/ AU define scoring function do greedy local structure search Issues: efficiency Computation of sufficient statistics for exists attribute Do not explicitly consider relations that do not exist

63 Structure Selection PRMs w/ EU Idea: define scoring function do phased local search over legal structures Key Components: legal models model new dependencies scoring models unchanged searching model space unchanged

64 Results

65 PRMs w/ Class Hierarchies Allows us to: Refine a heterogeneous class into more coherent subclasses Refine probabilistic model along class hierarchy Can specialize/inherit CPDs Construct new dependencies (that were originally cyclic) Provides bridge from class-based model to instance-based model

66 Learning PRM-CHs Vote Database: Instance I TVProgram Person Vote TVProgram Relational Schema Person Class hierarchy provided Learn class hierarchy

67 Guaranteeing Acyclicity w/ Subclasses Quality Accepted Journal Quality Accepted Conf- Quality Accepted.Accepted Journal.Accepted Conf-.Accepted.Class

68 Learning PRM-CH Scenario 1: Class hierarchy is provided New Operators l Specialize/Inherit Accepted Accepted Journal Accepted Conference Accepted Workshop

69 Learning Class Hierarchy Issue: partially observable data set Construct decision tree for class defined over attributes observed in training set New operator l Split on class attribute l Related class attribute class1 journal high.venue conference.author.fame class2 medium workshop low class3 class4 class5 class6

70 PRM-CH Summary PRMs with class hierarchies are a natural extension of PRMs: Specialization/Inheritance of CPDs Allows new dependency structures Provide bridge from class-based to instancebased models Learning techniques proposed Need efficient heuristics Empirical validation on real-world domains

71 Conclusions PRMs can represent distribution over attributes from multiple tables PRMs can capture link uncertainty PRMs allow inferences about individuals while taking into account relational structure (they do not make inappropriate independence assumptions)

72 Selected Publications Learning Probabilistic Models of Link Structure, L. Getoor, N. Friedman, D. Koller and B. Taskar, JMLR Probabilistic Models of Text and Link Structure for Hypertext Classification, L. Getoor, E. Segal, B. Taskar and D. Koller, IJCAI WS Text Learning: Beyond Classification, Selectivity Estimation using Probabilistic Models, L. Getoor, B. Taskar and D. Koller, SIGMOD-01. Learning Probabilistic Relational Models, L. Getoor, N. Friedman, D. Koller, and A. Pfeffer, chapter in Relation Data Mining, eds. S. Dzeroski and N. Lavrac, see also N. Friedman, L. Getoor, D. Koller, and A. Pfeffer, IJCAI-99. Learning Probabilistic Models of Relational Structure, L. Getoor, N. Friedman, D. Koller, and B. Taskar, ICML-01. From Instances to Classes in Probabilistic Relational Models, L. Getoor, D. Koller and N. Friedman, ICML Workshop on Attribute-Value and Relational Learning: Crossing the Boundaries, Notes from AAAI Workshop on Learning Statistical Models from Relational Data, eds. L.Getoor and D. Jensen, Notes from IJCAI Workshop on Learning Statistical Models from Relational Data, eds. L.Getoor and D. Jensen, See