Conditional Independence and Factorization
|
|
- Amie Peters
- 6 years ago
- Views:
Transcription
1 Conditional Independence and Factorization Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea 1/37
2 Graphical Models: Efficient Representation Consider a set of (discrete) random variables {X 1,...,X n } where X i takes on each of its r different values x i. A direct representation of p(x 1,...,x n ) requires an n-dimensional table (table of r n values representing the probabilities of each of the r n possible values of X 1,...,X n ). Graphical models provide an efficient means of representing joint probability distributions over many random variables in situations where each random variable is conditionally independent of all but a handful of the n random variables. A graphical model can be thought of as a probabilistic database, a machine that can answer queries regarding the values of sets of random variables. We build up the database in pieces, using probability theory to ensure that the pieces have a consistent overall interpretation. Probability theory also justifies the inferential machinery that allows the pieces to be put together on the fly to answer queries. 2/37
3 The chain rule gives p(x 1,x 2,x 3,x 4 ) = p(x 4 x 3,x 2,x 1 )p(x 3 x 2,x 1 )p(x 2 x 1 )p(x 1 ). X 1 X 2 X 3 X 4 X 1 X 2 X 3 X 4 GM (chain rule) GM (Markov chain) 3/37
4 Outline: Directed Graphical Models Directed graphs and joint probabilities Conditional independence and d-separation Three canonical graphs Bayes ball algorithm Characterization of directed graphical models 4/37
5 Notations Given a set of random variables, {X 1,...,X n }, let x i represent the realization of random variable X i. The probability mass function p(x 1,...,x n ) is defined as P(X 1 = x 1,...,X n = x n ). Use X to stand for {X 1,...,X n } and x to stand for {x 1,...,x n }. X A is shorthand for {X 1,X 2 } if A = {1,2}. 5/37
6 Directed Graphs A directed graph is a pair, G(V,E), where V is a set of nodes (vertices) and E is a set of oriented edges. Assume that G is acyclic. Nodes Edges Associated with random variables. a One-to-one mapping from nodes to random variables. Parent node, πi, is a set of parents of node i. Edges represent conditional dependence. 6/37
7 Conditional Independence X A and X B are independent, X A X B, if p(x A,x B ) = p(x A )p(x B ). X A and X C are conditionally independent given X B, X A X C X B, if p(x A,x C x B ) = p(x A x B )p(x C x B ) or p(x A x B,x C ) = p(x A x B ), for all x B such that p(x B ) > 0. 7/37
8 An Example of DAG X 4 X 2 X 1 X 6 X 3 X 5 p(x 1,x 2,x 3,x 4,x 5,x 6 ) = p(x 1 )p(x 2 x 1 )p(x 3 x 1 )p(x 4 x 2 )p(x 5 x 3 )p(x 6 x 2,x 5 ) 8/37
9 Factorization of Joint Probability Distributions Use the locality defined by the parent-child relationship to construct economical representation of joint probability distributions. Associate a function f i(x i,x πi ) to each node i V (Properties of conditional probability distributions: Nonnegativity and sum-to-one). Given a set of functions {f i(x i,x πi ) : i V} for V = 1,2,...,n, we define a joint probability distribution as follows: p(x 1,x 2,...,x n) n f i(x i,x πi ). Given that f i(x i,x πi ) are conditional probabilities, we write in terms of p(x i x πi ): p(x 1,x 2,...,x n) i=1 n p(x i x πi ). i=1 9/37
10 An Example of DAG: Revisited X 4 X 2 X 1 X 6 X 3 X 5 p(x 1,x 2,x 3,x 4,x 5,x 6 ) = p(x 1 )p(x 2 x 1 )p(x 3 x 1 )p(x 4 x 2 )p(x 5 x 3 )p(x 6 x 2,x 5 ) 10/37
11 Economical Representation? Consider n discrete random variables {x 1,x 2,...,x n } with each variable x i ranging over r values. Naive approach PGM Need an n-dimensional table of size r n Need an (mi+1)-dimensional table of size r m i+1 where m i is the number of parents of node X i 11/37
12 Conditional Independence in DAG Two different factorization of a probability distribution function The chain rule leads to DAG leads to p(x 1,x 2,...,x n ) = n p(x i x 1,...,x i 1 ). i=1 p(x 1,x 2,...,x n ) = n p(x i x πi ). Comparing these expressions, we might interpret that missing variables in the local conditional probability functions corresponds to missing edges (conditional independence) in the underlying graph. i=1 12/37
13 Basic Conditional Independence Statements An ordering I of the nodes in a graph G is said to be topological if the nodes in π i appear before i in I for every node i V. (I = {1,2,3,4,5,6} is a topological ordering in our example) Let ν i denote the set of all nodes that appear earlier than i in I, excluding π i. (for example, ν 5 = {1,2,4}) Given a topological ordering I, the set of basic conditional independence statements: for i V. {X i X νi X πi }, These statement can be verified by algebraic calculations Example: X 4 {X 1,X 3} X 2 p(x 4 x 1,x 2,x 3) = p(x1,x2,x3,x4) p(x 1,x 2,x 3) = p(x1)p(x2 x1)p(x3 x1)p(x4 x2) p(x 1)p(x 2 x 1)p(x 3 x 1) = p(x 4 x 2) 13/37
14 Markov Chain X Y Z X Y Z X Z Y p(z x,y) = p(x,y,z) p(x, y) = p(x)p(y x)p(z y) p(x)p(y x) There are no other conditional independencies = p(z y) Asserted conditional independencies always hold for the family of distributions associated with a given graph. Non-asserted conditional independencies sometimes fail to hold but sometimes do hold. 14/37
15 Hidden Cause Y Y X Z X Z X Z Y p(x,z y) = p(y)p(x y)p(z y) p(y) = p(x y)p(z y) We do not necessarily assume that X and Z are dependent. 15/37
16 Explaining-Away X Z X Z Y Y X Z p(y x,z) = p(x)p(z)p(y x,z) p(x, z) p(x,z) = p(x)p(z) No X Z Y 16/37
17 Bayes Ball Algorithm We wish to decide whether a given conditional independence statement, X A X B X C, is true for a directed graph G. A reachability algorithm A d-separation test Shade the nodes in X C. Place balls at each node in X A, let them bounce around according to some rules, and then ask if any of the balls reach any of the nodes in X B. 17/37
18 Bayes Ball Algorithm - Rules 1,2 X Y Z X Y Z Y Y X Z X Z 18/37
19 Bayes Ball Algorithm - Rules 3,4,5 X Z X Z Y Y X Y X Y X Y X Y 19/37
20 Example 1 X 4 X 2 X 1 X 6 X 3 X 5 X 4 {X 1,X 3} X 2 is true. 20/37
21 Example 2 X 4 X 2 X 1 X 6 X 3 X 5 X 1 X 6 {X 2,X 3} is true. 21/37
22 Example 3 X 4 X 2 X 1 X 6 X 3 X 5 X 2 X 3 {X 1,X 6} is not true. 22/37
23 Characterization of Directed Graphical Models A graphical model is associated with a family of probability distributions and this family can be characterized in two equivalent ways. D 1= { p(x) = n i=1 p(xi xπ i )}. D 2= {p(x) subject to X i X νi X πi }. (A family of probability distributions associated with G, that includes all p(x v) that satisfy every conditional independence statement associated with G) D 1 = D 2 (Details will be discussed later) This provides a strong and important link between graph theory and probability theory. 23/37
24 Markov Blanket Markov blanket: V is a Markov blanket for X iff X Y V for all Y / V. In a DAG, Markov blanket for node i is composed of its parents, its children, and its children s parents. Markov blanket identifies all the variables that shield off the node from the rest of the network. This means that the Markov blanket of a node is the only knowledge that is needed to predict the behaviour of that node. (coined by J. Pearl) 24/37
25 A 25/37
26 Outline: Undirected Graphical Models Markov networks Clique potentials Characterization of undirected graphical models 26/37
27 Markov Networks (Undirected Graphical Models) X 1 X 4 Examples of Markov networks: X 2 X 3 X 5 Boltzmann machines Markov random fields Semantics: Every node is conditionally independent from its non-neighbors, given its neighbors. Markov blanket: V is a Markov blanket for X iff X Y V for all Y / V. Markov boundary: Minimal Markov blanket 27/37
28 Boltzmann Machines Markov network over a vector of binary variables, x i {0,1}, where some variables may be hidden (xi H ) and some may be observed (visible, xi V ). Learning for Boltzmann machines involves adjusting weights W such that the generative model p(x W) = 1 { } 1 Z exp 2 x Wx is well-matched to a set of examples {x(t)} (t = 1,...,N). The learning is done via EM! 28/37
29 Conditional Independence For undirected graphs, the conditional independence semantics is the more intuitive. Every path from a node in X A to a node in X C includes at least one node in X B X A X C X B. X B X A X A X C X B X C 29/37
30 Comparative Semantics W X Y {W,Z} W Z {X,Y} X Y not X Y Z X Y X Y Z X Y Z not X Y Z X Y X Z X Y X Y Z Z 30/37
31 Clique Potentials Clique: A fully connected subgraph (usually maximal) Denote by x Ci the set of variables in clique C i. X 1 X 2 X 3 X 4 X 5 For each clique C i, we assign a nonnegative function (potential function), ψ Ci (x Ci ) which measures compatibility (agreement, constraint, or energy). where Z = x p(x) = 1 Z ψ Ci (x Ci ), C i C C i C ψ C i (x Ci ) is the normalization. 31/37
32 Example 0 x 1 1 x X 1 0 x 2 1 X 2 x X 4 0 x 5 1 x x 2 x 3 X 6 x X 3 x x X 5 32/37
33 Potential Functions: Parameterization The potential functions must be nonnegative. ψ C (x C ) = exp{ H C (x C )}. Therefore, the joint probability for undirected models can be derived as follows : p(x) = 1 Z exp H Ci (x Ci ). C i C The sum in the expression is generally referred to as the energy : H(x) C i CH Ci (x Ci ) Finally, the joint probability can be represented as a Boltzman distribution : p(x) = 1 Z exp{ H(x)}. 33/37
34 Potential Functions, again! The factorization of the joint probability distribution in a Markov network is given by p(x) = 1 Z ψ Ci (x Ci ). C i C If all potential functions are strictly positive, then we have p(x) = exp logψ Ci (x Ci ) logz }{{}}{{} C i C. θ i T i (X) A(θ) In some cases, we have { } p(x) = exp θ it i(x) A(θ), i which is known as exponential family. In statistical physics, log Z is referred to as free energy. 34/37
35 Markov Random Fields Let X = {X 1,...,X n} be a family of random variables defined on the set S in which each random variable X i takes a value in L. Definition The family X is called a random field. Definition X is said to be Markov random field (MRF) with respect to a neighborhood system N if and only if the following two conditions are satisfied: 1. Positivity: P(x) > 0 x X. 2. Markovianity: P(x i x S\i ) = P(x i x Ni ). 35/37
36 Gibbs Random Fields Definition A set of random variables X is said to be a Gibbs random field (GRF) if and only iff its configurations obey the Gibbs distribution that has the form p(x) = 1 { Z exp 1 } T E(x), where E(x) = ψ C (x). We often consider cliques of size up to 2, E(x) ψ 1(x i)+ ψ 2(x i,x j). i S i S j N i Theorem (Hammersley-Clifford) X is an MRF on S with respect to N if and only iff X is a GRF on S with respect to N. 36/37
37 Characterization of Undirected Graphical Models U 1 : A family of probability distributions by ranging over all possible choices of positive potential functions on the maximal cliques of the graph. p(x) = 1 } { H(x) Z exp. U 2 : A family of probability distributions via the conditional independence assertions associated (X A X B X C ) with G. U 1 U 2 by Hammersley-Clifford theorem (proof, p85) 37/37
Lecture 4 October 18th
Directed and undirected graphical models Fall 2017 Lecture 4 October 18th Lecturer: Guillaume Obozinski Scribe: In this lecture, we will assume that all random variables are discrete, to keep notations
More informationReview: Directed Models (Bayes Nets)
X Review: Directed Models (Bayes Nets) Lecture 3: Undirected Graphical Models Sam Roweis January 2, 24 Semantics: x y z if z d-separates x and y d-separation: z d-separates x from y if along every undirected
More information3 : Representation of Undirected GM
10-708: Probabilistic Graphical Models 10-708, Spring 2016 3 : Representation of Undirected GM Lecturer: Eric P. Xing Scribes: Longqi Cai, Man-Chia Chang 1 MRF vs BN There are two types of graphical models:
More information4.1 Notation and probability review
Directed and undirected graphical models Fall 2015 Lecture 4 October 21st Lecturer: Simon Lacoste-Julien Scribe: Jaime Roquero, JieYing Wu 4.1 Notation and probability review 4.1.1 Notations Let us recall
More informationUndirected Graphical Models: Markov Random Fields
Undirected Graphical Models: Markov Random Fields 40-956 Advanced Topics in AI: Probabilistic Graphical Models Sharif University of Technology Soleymani Spring 2015 Markov Random Field Structure: undirected
More informationChris Bishop s PRML Ch. 8: Graphical Models
Chris Bishop s PRML Ch. 8: Graphical Models January 24, 2008 Introduction Visualize the structure of a probabilistic model Design and motivate new models Insights into the model s properties, in particular
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
More informationPart I. C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS
Part I C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Probabilistic Graphical Models Graphical representation of a probabilistic model Each variable corresponds to a
More informationUndirected Graphical Models
Undirected Graphical Models 1 Conditional Independence Graphs Let G = (V, E) be an undirected graph with vertex set V and edge set E, and let A, B, and C be subsets of vertices. We say that C separates
More informationUndirected Graphical Models
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Properties Properties 3 Generative vs. Conditional
More informationExpectation Maximization
Expectation Maximization Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr 1 /
More informationCSC 412 (Lecture 4): Undirected Graphical Models
CSC 412 (Lecture 4): Undirected Graphical Models Raquel Urtasun University of Toronto Feb 2, 2016 R Urtasun (UofT) CSC 412 Feb 2, 2016 1 / 37 Today Undirected Graphical Models: Semantics of the graph:
More informationDirected and Undirected Graphical Models
Directed and Undirected Davide Bacciu Dipartimento di Informatica Università di Pisa bacciu@di.unipi.it Machine Learning: Neural Networks and Advanced Models (AA2) Last Lecture Refresher Lecture Plan Directed
More information1 Undirected Graphical Models. 2 Markov Random Fields (MRFs)
Machine Learning (ML, F16) Lecture#07 (Thursday Nov. 3rd) Lecturer: Byron Boots Undirected Graphical Models 1 Undirected Graphical Models In the previous lecture, we discussed directed graphical models.
More informationRapid Introduction to Machine Learning/ Deep Learning
Rapid Introduction to Machine Learning/ Deep Learning Hyeong In Choi Seoul National University 1/24 Lecture 5b Markov random field (MRF) November 13, 2015 2/24 Table of contents 1 1. Objectives of Lecture
More informationStatistical Approaches to Learning and Discovery
Statistical Approaches to Learning and Discovery Graphical Models Zoubin Ghahramani & Teddy Seidenfeld zoubin@cs.cmu.edu & teddy@stat.cmu.edu CALD / CS / Statistics / Philosophy Carnegie Mellon University
More information1. what conditional independencies are implied by the graph. 2. whether these independecies correspond to the probability distribution
NETWORK ANALYSIS Lourens Waldorp PROBABILITY AND GRAPHS The objective is to obtain a correspondence between the intuitive pictures (graphs) of variables of interest and the probability distributions of
More informationCOS402- Artificial Intelligence Fall Lecture 10: Bayesian Networks & Exact Inference
COS402- Artificial Intelligence Fall 2015 Lecture 10: Bayesian Networks & Exact Inference Outline Logical inference and probabilistic inference Independence and conditional independence Bayes Nets Semantics
More information3 Undirected Graphical Models
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 3 Undirected Graphical Models In this lecture, we discuss undirected
More information2 : Directed GMs: Bayesian Networks
10-708: Probabilistic Graphical Models 10-708, Spring 2017 2 : Directed GMs: Bayesian Networks Lecturer: Eric P. Xing Scribes: Jayanth Koushik, Hiroaki Hayashi, Christian Perez Topic: Directed GMs 1 Types
More informationRepresentation of undirected GM. Kayhan Batmanghelich
Representation of undirected GM Kayhan Batmanghelich Review Review: Directed Graphical Model Represent distribution of the form ny p(x 1,,X n = p(x i (X i i=1 Factorizes in terms of local conditional probabilities
More informationRecall from last time: Conditional probabilities. Lecture 2: Belief (Bayesian) networks. Bayes ball. Example (continued) Example: Inference problem
Recall from last time: Conditional probabilities Our probabilistic models will compute and manipulate conditional probabilities. Given two random variables X, Y, we denote by Lecture 2: Belief (Bayesian)
More informationUndirected graphical models
Undirected graphical models Semantics of probabilistic models over undirected graphs Parameters of undirected models Example applications COMP-652 and ECSE-608, February 16, 2017 1 Undirected graphical
More informationA graph contains a set of nodes (vertices) connected by links (edges or arcs)
BOLTZMANN MACHINES Generative Models Graphical Models A graph contains a set of nodes (vertices) connected by links (edges or arcs) In a probabilistic graphical model, each node represents a random variable,
More informationCS 2750: Machine Learning. Bayesian Networks. Prof. Adriana Kovashka University of Pittsburgh March 14, 2016
CS 2750: Machine Learning Bayesian Networks Prof. Adriana Kovashka University of Pittsburgh March 14, 2016 Plan for today and next week Today and next time: Bayesian networks (Bishop Sec. 8.1) Conditional
More informationAn Introduction to Bayesian Machine Learning
1 An Introduction to Bayesian Machine Learning José Miguel Hernández-Lobato Department of Engineering, Cambridge University April 8, 2013 2 What is Machine Learning? The design of computational systems
More informationMachine Learning Lecture 14
Many slides adapted from B. Schiele, S. Roth, Z. Gharahmani Machine Learning Lecture 14 Undirected Graphical Models & Inference 23.06.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ leibe@vision.rwth-aachen.de
More informationGibbs Fields & Markov Random Fields
Statistical Techniques in Robotics (16-831, F10) Lecture#7 (Tuesday September 21) Gibbs Fields & Markov Random Fields Lecturer: Drew Bagnell Scribe: Bradford Neuman 1 1 Gibbs Fields Like a Bayes Net, a
More informationGraphical Models and Kernel Methods
Graphical Models and Kernel Methods Jerry Zhu Department of Computer Sciences University of Wisconsin Madison, USA MLSS June 17, 2014 1 / 123 Outline Graphical Models Probabilistic Inference Directed vs.
More informationUC Berkeley Department of Electrical Engineering and Computer Science Department of Statistics. EECS 281A / STAT 241A Statistical Learning Theory
UC Berkeley Department of Electrical Engineering and Computer Science Department of Statistics EECS 281A / STAT 241A Statistical Learning Theory Solutions to Problem Set 1 Fall 2011 Issued: Thurs, September
More informationLecture 8: Bayesian Networks
Lecture 8: Bayesian Networks Bayesian Networks Inference in Bayesian Networks COMP-652 and ECSE 608, Lecture 8 - January 31, 2017 1 Bayes nets P(E) E=1 E=0 0.005 0.995 E B P(B) B=1 B=0 0.01 0.99 E=0 E=1
More informationBased on slides by Richard Zemel
CSC 412/2506 Winter 2018 Probabilistic Learning and Reasoning Lecture 3: Directed Graphical Models and Latent Variables Based on slides by Richard Zemel Learning outcomes What aspects of a model can we
More informationMarkov Random Fields
Markov Random Fields Umamahesh Srinivas ipal Group Meeting February 25, 2011 Outline 1 Basic graph-theoretic concepts 2 Markov chain 3 Markov random field (MRF) 4 Gauss-Markov random field (GMRF), and
More informationProbabilistic Graphical Networks: Definitions and Basic Results
This document gives a cursory overview of Probabilistic Graphical Networks. The material has been gleaned from different sources. I make no claim to original authorship of this material. Bayesian Graphical
More informationCS Lecture 4. Markov Random Fields
CS 6347 Lecture 4 Markov Random Fields Recap Announcements First homework is available on elearning Reminder: Office hours Tuesday from 10am-11am Last Time Bayesian networks Today Markov random fields
More informationBayes Nets: Independence
Bayes Nets: Independence [These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188 materials are available at http://ai.berkeley.edu.] Bayes Nets A Bayes
More informationMarkov Chains and MCMC
Markov Chains and MCMC Markov chains Let S = {1, 2,..., N} be a finite set consisting of N states. A Markov chain Y 0, Y 1, Y 2,... is a sequence of random variables, with Y t S for all points in time
More information6.867 Machine learning, lecture 23 (Jaakkola)
Lecture topics: Markov Random Fields Probabilistic inference Markov Random Fields We will briefly go over undirected graphical models or Markov Random Fields (MRFs) as they will be needed in the context
More informationProbabilistic Graphical Models (I)
Probabilistic Graphical Models (I) Hongxin Zhang zhx@cad.zju.edu.cn State Key Lab of CAD&CG, ZJU 2015-03-31 Probabilistic Graphical Models Modeling many real-world problems => a large number of random
More informationIntroduction to Probabilistic Graphical Models
Introduction to Probabilistic Graphical Models Franz Pernkopf, Robert Peharz, Sebastian Tschiatschek Graz University of Technology, Laboratory of Signal Processing and Speech Communication Inffeldgasse
More informationDirected Graphical Models or Bayesian Networks
Directed Graphical Models or Bayesian Networks Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Bayesian Networks One of the most exciting recent advancements in statistical AI Compact
More information2 : Directed GMs: Bayesian Networks
10-708: Probabilistic Graphical Models, Spring 2015 2 : Directed GMs: Bayesian Networks Lecturer: Eric P. Xing Scribes: Yi Cheng, Cong Lu 1 Notation Here the notations used in this course are defined:
More informationMarkov Random Fields
Markov Random Fields 1. Markov property The Markov property of a stochastic sequence {X n } n 0 implies that for all n 1, X n is independent of (X k : k / {n 1, n, n + 1}), given (X n 1, X n+1 ). Another
More informationStatistical Learning
Statistical Learning Lecture 5: Bayesian Networks and Graphical Models Mário A. T. Figueiredo Instituto Superior Técnico & Instituto de Telecomunicações University of Lisbon, Portugal May 2018 Mário A.
More informationChapter 16. Structured Probabilistic Models for Deep Learning
Peng et al.: Deep Learning and Practice 1 Chapter 16 Structured Probabilistic Models for Deep Learning Peng et al.: Deep Learning and Practice 2 Structured Probabilistic Models way of using graphs to describe
More informationMachine Learning 4771
Machine Learning 4771 Instructor: Tony Jebara Topic 16 Undirected Graphs Undirected Separation Inferring Marginals & Conditionals Moralization Junction Trees Triangulation Undirected Graphs Separation
More informationMassachusetts Institute of Technology Department of Electrical Engineering and Computer Science Algorithms For Inference Fall 2014
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Problem Set 3 Issued: Thursday, September 25, 2014 Due: Thursday,
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning Undirected Graphical Models Mark Schmidt University of British Columbia Winter 2016 Admin Assignment 3: 2 late days to hand it in today, Thursday is final day. Assignment 4:
More informationIndependencies. Undirected Graphical Models 2: Independencies. Independencies (Markov networks) Independencies (Bayesian Networks)
(Bayesian Networks) Undirected Graphical Models 2: Use d-separation to read off independencies in a Bayesian network Takes a bit of effort! 1 2 (Markov networks) Use separation to determine independencies
More informationA brief introduction to Conditional Random Fields
A brief introduction to Conditional Random Fields Mark Johnson Macquarie University April, 2005, updated October 2010 1 Talk outline Graphical models Maximum likelihood and maximum conditional likelihood
More informationMachine Learning Summer School
Machine Learning Summer School Lecture 1: Introduction to Graphical Models Zoubin Ghahramani zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ epartment of ngineering University of ambridge, UK
More informationArtificial Intelligence Bayes Nets: Independence
Artificial Intelligence Bayes Nets: Independence Instructors: David Suter and Qince Li Course Delivered @ Harbin Institute of Technology [Many slides adapted from those created by Dan Klein and Pieter
More informationRecall from last time. Lecture 3: Conditional independence and graph structure. Example: A Bayesian (belief) network.
ecall from last time Lecture 3: onditional independence and graph structure onditional independencies implied by a belief network Independence maps (I-maps) Factorization theorem The Bayes ball algorithm
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Variational Inference II: Mean Field Method and Variational Principle Junming Yin Lecture 15, March 7, 2012 X 1 X 1 X 1 X 1 X 2 X 3 X 2 X 2 X 3
More informationProbabilistic Graphical Models
2016 Robert Nowak Probabilistic Graphical Models 1 Introduction We have focused mainly on linear models for signals, in particular the subspace model x = Uθ, where U is a n k matrix and θ R k is a vector
More informationInformation Theory Primer:
Information Theory Primer: Entropy, KL Divergence, Mutual Information, Jensen s inequality Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
More informationBayes Decision Theory
Bayes Decision Theory Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr 1 / 16
More informationAlternative Parameterizations of Markov Networks. Sargur Srihari
Alternative Parameterizations of Markov Networks Sargur srihari@cedar.buffalo.edu 1 Topics Three types of parameterization 1. Gibbs Parameterization 2. Factor Graphs 3. Log-linear Models Features (Ising,
More informationBayesian Networks. Alan Ri2er
Bayesian Networks Alan Ri2er Problem: Non- IID Data Most real- world data is not IID (like coin flips) MulBple correlated variables Examples: Pixels in an image Words in a document Genes in a microarray
More informationProbabilistic Graphical Models and Bayesian Networks. Artificial Intelligence Bert Huang Virginia Tech
Probabilistic Graphical Models and Bayesian Networks Artificial Intelligence Bert Huang Virginia Tech Concept Map for Segment Probabilistic Graphical Models Probabilistic Time Series Models Particle Filters
More informationECE521 Tutorial 11. Topic Review. ECE521 Winter Credits to Alireza Makhzani, Alex Schwing, Rich Zemel and TAs for slides. ECE521 Tutorial 11 / 4
ECE52 Tutorial Topic Review ECE52 Winter 206 Credits to Alireza Makhzani, Alex Schwing, Rich Zemel and TAs for slides ECE52 Tutorial ECE52 Winter 206 Credits to Alireza / 4 Outline K-means, PCA 2 Bayesian
More informationUniversity of Washington Department of Electrical Engineering EE512 Spring, 2006 Graphical Models
University of Washington Department of Electrical Engineering EE512 Spring, 2006 Graphical Models Jeff A. Bilmes Lecture 1 Slides March 28 th, 2006 Lec 1: March 28th, 2006 EE512
More informationRandom Field Models for Applications in Computer Vision
Random Field Models for Applications in Computer Vision Nazre Batool Post-doctorate Fellow, Team AYIN, INRIA Sophia Antipolis Outline Graphical Models Generative vs. Discriminative Classifiers Markov Random
More informationAlternative Parameterizations of Markov Networks. Sargur Srihari
Alternative Parameterizations of Markov Networks Sargur srihari@cedar.buffalo.edu 1 Topics Three types of parameterization 1. Gibbs Parameterization 2. Factor Graphs 3. Log-linear Models with Energy functions
More informationDirected and Undirected Graphical Models
Directed and Undirected Graphical Models Adrian Weller MLSALT4 Lecture Feb 26, 2016 With thanks to David Sontag (NYU) and Tony Jebara (Columbia) for use of many slides and illustrations For more information,
More informationProbabilistic Machine Learning
Probabilistic Machine Learning Bayesian Nets, MCMC, and more Marek Petrik 4/18/2017 Based on: P. Murphy, K. (2012). Machine Learning: A Probabilistic Perspective. Chapter 10. Conditional Independence Independent
More informationProbabilistic Reasoning. (Mostly using Bayesian Networks)
Probabilistic Reasoning (Mostly using Bayesian Networks) Introduction: Why probabilistic reasoning? The world is not deterministic. (Usually because information is limited.) Ways of coping with uncertainty
More informationNonparameteric Regression:
Nonparameteric Regression: Nadaraya-Watson Kernel Regression & Gaussian Process Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
More informationCS446: Machine Learning Fall Final Exam. December 6 th, 2016
CS446: Machine Learning Fall 2016 Final Exam December 6 th, 2016 This is a closed book exam. Everything you need in order to solve the problems is supplied in the body of this exam. This exam booklet contains
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationAn Introduction to Exponential-Family Random Graph Models
An Introduction to Exponential-Family Random Graph Models Luo Lu Feb.8, 2011 1 / 11 Types of complications in social network Single relationship data A single relationship observed on a set of nodes at
More informationCS 5522: Artificial Intelligence II
CS 5522: Artificial Intelligence II Bayes Nets: Independence Instructor: Alan Ritter Ohio State University [These slides were adapted from CS188 Intro to AI at UC Berkeley. All materials available at http://ai.berkeley.edu.]
More informationECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale Learning
ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale Learning Topics Markov Random Fields: Representation Conditional Random Fields Log-Linear Models Readings: KF
More informationMarkov Networks. l Like Bayes Nets. l Graph model that describes joint probability distribution using tables (AKA potentials)
Markov Networks l Like Bayes Nets l Graph model that describes joint probability distribution using tables (AKA potentials) l Nodes are random variables l Labels are outcomes over the variables Markov
More informationGibbs Field & Markov Random Field
Statistical Techniques in Robotics (16-831, F12) Lecture#07 (Wednesday September 19) Gibbs Field & Markov Random Field Lecturer: Drew Bagnell Scribe:Minghao Ruan 1 1 SLAM (continued from last lecture)
More informationProf. Dr. Lars Schmidt-Thieme, L. B. Marinho, K. Buza Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany, Course
Course on Bayesian Networks, winter term 2007 0/31 Bayesian Networks Bayesian Networks I. Bayesian Networks / 1. Probabilistic Independence and Separation in Graphs Prof. Dr. Lars Schmidt-Thieme, L. B.
More information1 : Introduction. 1 Course Overview. 2 Notation. 3 Representing Multivariate Distributions : Probabilistic Graphical Models , Spring 2014
10-708: Probabilistic Graphical Models 10-708, Spring 2014 1 : Introduction Lecturer: Eric P. Xing Scribes: Daniel Silva and Calvin McCarter 1 Course Overview In this lecture we introduce the concept of
More informationMarkov Networks. l Like Bayes Nets. l Graphical model that describes joint probability distribution using tables (AKA potentials)
Markov Networks l Like Bayes Nets l Graphical model that describes joint probability distribution using tables (AKA potentials) l Nodes are random variables l Labels are outcomes over the variables Markov
More informationCausal Bayesian networks. Peter Antal
Causal Bayesian networks Peter Antal antal@mit.bme.hu A.I. 4/8/2015 1 Can we represent exactly (in)dependencies by a BN? From a causal model? Suff.&nec.? Can we interpret edges as causal relations with
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Lecture 4 Learning Bayesian Networks CS/CNS/EE 155 Andreas Krause Announcements Another TA: Hongchao Zhou Please fill out the questionnaire about recitations Homework 1 out.
More informationIntroduction to Graphical Models
Introduction to Graphical Models The 15 th Winter School of Statistical Physics POSCO International Center & POSTECH, Pohang 2018. 1. 9 (Tue.) Yung-Kyun Noh GENERALIZATION FOR PREDICTION 2 Probabilistic
More informationDirected Probabilistic Graphical Models CMSC 678 UMBC
Directed Probabilistic Graphical Models CMSC 678 UMBC Announcement 1: Assignment 3 Due Wednesday April 11 th, 11:59 AM Any questions? Announcement 2: Progress Report on Project Due Monday April 16 th,
More informationMarkov properties for undirected graphs
Graphical Models, Lecture 2, Michaelmas Term 2011 October 12, 2011 Formal definition Fundamental properties Random variables X and Y are conditionally independent given the random variable Z if L(X Y,
More informationEE512 Graphical Models Fall 2009
EE512 Graphical Models Fall 2009 Prof. Jeff Bilmes University of Washington, Seattle Department of Electrical Engineering Fall Quarter, 2009 http://ssli.ee.washington.edu/~bilmes/ee512fa09 Lecture 3 -
More informationUsing Graphs to Describe Model Structure. Sargur N. Srihari
Using Graphs to Describe Model Structure Sargur N. srihari@cedar.buffalo.edu 1 Topics in Structured PGMs for Deep Learning 0. Overview 1. Challenge of Unstructured Modeling 2. Using graphs to describe
More informationGraphical Models and Independence Models
Graphical Models and Independence Models Yunshu Liu ASPITRG Research Group 2014-03-04 References: [1]. Steffen Lauritzen, Graphical Models, Oxford University Press, 1996 [2]. Christopher M. Bishop, Pattern
More informationRepresentation. Stefano Ermon, Aditya Grover. Stanford University. Lecture 2
Representation Stefano Ermon, Aditya Grover Stanford University Lecture 2 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 2 1 / 32 Learning a generative model We are given a training
More informationGraphical Models for Collaborative Filtering
Graphical Models for Collaborative Filtering Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Sequence modeling HMM, Kalman Filter, etc.: Similarity: the same graphical model topology,
More informationMachine Learning Lecture Notes
Machine Learning Lecture Notes Predrag Radivojac January 3, 25 Random Variables Until now we operated on relatively simple sample spaces and produced measure functions over sets of outcomes. In many situations,
More informationDirected Graphical Models
CS 2750: Machine Learning Directed Graphical Models Prof. Adriana Kovashka University of Pittsburgh March 28, 2017 Graphical Models If no assumption of independence is made, must estimate an exponential
More informationVariational Inference (11/04/13)
STA561: Probabilistic machine learning Variational Inference (11/04/13) Lecturer: Barbara Engelhardt Scribes: Matt Dickenson, Alireza Samany, Tracy Schifeling 1 Introduction In this lecture we will further
More informationGraphical Models - Part II
Graphical Models - Part II Bishop PRML Ch. 8 Alireza Ghane Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Graphical Models Alireza Ghane / Greg Mori 1 Outline Probabilistic
More informationGraphical models and causality: Directed acyclic graphs (DAGs) and conditional (in)dependence
Graphical models and causality: Directed acyclic graphs (DAGs) and conditional (in)dependence General overview Introduction Directed acyclic graphs (DAGs) and conditional independence DAGs and causal effects
More informationCausal Bayesian networks. Peter Antal
Causal Bayesian networks Peter Antal antal@mit.bme.hu A.I. 11/25/2015 1 Can we represent exactly (in)dependencies by a BN? From a causal model? Suff.&nec.? Can we interpret edges as causal relations with
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2016
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2016 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationIntelligent Systems:
Intelligent Systems: Undirected Graphical models (Factor Graphs) (2 lectures) Carsten Rother 15/01/2015 Intelligent Systems: Probabilistic Inference in DGM and UGM Roadmap for next two lectures Definition
More informationCausality in Econometrics (3)
Graphical Causal Models References Causality in Econometrics (3) Alessio Moneta Max Planck Institute of Economics Jena moneta@econ.mpg.de 26 April 2011 GSBC Lecture Friedrich-Schiller-Universität Jena
More informationLecture 1: Probabilistic Graphical Models. Sam Roweis. Monday July 24, 2006 Machine Learning Summer School, Taiwan
Lecture 1: Probabilistic Graphical Models Sam Roweis Monday July 24, 2006 Machine Learning Summer School, Taiwan Building Intelligent Computers We want intelligent, adaptive, robust behaviour in computers.
More informationGraphical Models - Part I
Graphical Models - Part I Oliver Schulte - CMPT 726 Bishop PRML Ch. 8, some slides from Russell and Norvig AIMA2e Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Outline Probabilistic
More informationProbabilistic Graphical Models Lecture Notes Fall 2009
Probabilistic Graphical Models Lecture Notes Fall 2009 October 28, 2009 Byoung-Tak Zhang School of omputer Science and Engineering & ognitive Science, Brain Science, and Bioinformatics Seoul National University
More information