Why (Bayesian) inference makes (Differential) privacy easy
|
|
- Blake Bennett
- 5 years ago
- Views:
Transcription
1 Why (Bayesian) inference makes (Differential) privacy easy Christos Dimitrakakis 1 Blaine Nelson 2,3 Aikaterini Mitrokotsa 1 Benjamin Rubinstein 4 Zuhe Zhang 4 1 Chalmers university of Technology 2 University of Potsdam 3 Google 4 University of Melbourne 10/3/2015
2 Overview Example (Health insurance) We collect data x about treatments and patients. We disclose conclusions about treatment effectiveness. We want to hide individual patient information. Encryption does not help The general problem We wish to estimate something from a dataset x S. We wish to communicate what we learn to a third party. How much can they learn about x?
3 Bayesian inference and differential privacy Bayesian estimation What are its robustness and privacy properties? How important is the selection of the prior? Limiting the communication channel How should we communicate information about our posterior? How much can an adversary learn from our posterior?
4 Setting Dramatis personae x data. B a (Bayesian) statistician. ξ the statistician s prior belief. θ a parameter A an adversary. He knows ξ, should not learn x.
5 Setting Dramatis personae x data. B a (Bayesian) statistician. ξ the statistician s prior belief. θ a parameter A an adversary. He knows ξ, should not learn x. The game 1. B selects a model family (F) and a prior (ξ). 2. B observes data x and calculates the posterior ξ(θ x). 3. A queries B. 4. B responds with a function of the posterior ξ(θ x). 5. Goto 3.
6 Two related problem viewpoints Bayesian inference view Prior Data Posterior Observer Mechanism design view Mechanism Data Query Observer
7 Two related problem viewpoints Bayesian inference view Prior Data Posterior Observer B has a prior belief about how well a drug works. B obtains some data about the drug. B s conclusion from the data is a posterior belief. B s communicates its conclusion to an observer which: 1. Must be accurate. 2. Must not reveal a lot about the data.
8 Two related problem viewpoints Mechanism design view Mechanism Data Query Observer We want to design a mechanism. The observer can ask queries. The mechanism responds with a response that: 1. Is useful. 2. Does not reveal a lot about the data.
9 Two related problem viewpoints Bayesian inference view Prior Data Posterior Observer Mechanism design view Mechanism Data Query Observer We ll discuss a specific mechanism that provides privacy. The amount of privacy depends on the prior.
10 Privacy properties of Bayesian inference Bayesian inference Robustness of the posterior distribution Posterior sampling query model
11 Bayesian inference Estimating a coin s bias A fair coin comes heads 50% of the time. We want to test an unknown coin, which we think may not be completely fair. 4 prior Figure: Prior belief ξ about the coin bias θ.
12 Bayesian inference 4 prior Figure: Prior belief ξ about the coin bias θ. For a sequence of throws x t {0, 1}, P θ (x) θ x t (1 θ) 1 x t t = θ #Heads (1 θ) #Tails
13 Bayesian inference 10 8 prior likelihood Figure: Prior belief ξ about the coin bias θ and likelihood of θ for the data. Say we throw the coin 100 times and obtain 70 heads. Then we plot the likelihood P θ (x) of different models.
14 Bayesian inference 10 8 prior likelihood posterior Figure: Prior belief ξ(θ) about the coin bias θ, likelihood of θ for the data, and posterior belief ξ(θ x) From these, we calculate a posterior distribution over the correct models. This represents our conclusion given our prior and the data.
15 Bayesian inference Setting Dataset space S. Distribution family F { P θ θ Θ }. Each P θ is a distribution on S. We wish to identify which θ generated the observed data x. Prior distribution ξ on Θ (i.e. initial belief) Posterior given data x S (i.e. conclusion) ξ(θ x) = P θ(x)ξ(θ) φ(x) φ(x) P θ (x)ξ(θ). θ Θ (posterior) (marginal) Standard calculation that can be done exactly or approximately.
16 What we want to show If we assume the family F is well-behaved or that the prior ξ is focused on the nice parts of F
17 What we want to show If we assume the family F is well-behaved or that the prior ξ is focused on the nice parts of F Inference is robust. Our knowledge is private. There are also well-known F satisfying our assumptions.
18 What we want to show If we assume the family F is well-behaved or that the prior ξ is focused on the nice parts of F Inference is robust. Our knowledge is private. There are also well-known F satisfying our assumptions. First, we must generalise differential privacy...
19 Differential privacy of conditional distribution ξ( x) Definition ((ɛ, δ)-differential privacy) ξ( x) is (ɛ, δ)-differentially private if, x S = X n, B Θ ξ(b x) e ɛ ξ(b y) + δ, for all y in the hamming-1 neighbourhood of x.
20 Differential privacy of conditional distribution ξ( x) Definition ((ɛ, δ)-differential privacy) ξ( x) is (ɛ, δ)-differentially private if, x S = X n, B Θ ξ(b x) e ɛ ξ(b y) + δ, for all y in the hamming-1 neighbourhood of x. Definition ((ɛ, δ)-differential privacy under ρ.) ξ( x) is (ɛ, δ)-differentially private under a pseudo-metric ρ : S S R + if, B Θ and x S, ξ(b x) e ɛρ(x,y) ξ(b y) + δρ(x, y), y S If two datasets x, y are close, then the distributions ξ( x) and ξ( y) are also close.
21 Sufficient conditions Assumption (F is Lipschitz) For a given ρ on S, L > 0 s.t. θ Θ: ln P θ(x) P θ (y) Lρ(x, y), x, y S, (1) x y theta
22 Sufficient conditions Assumption (F is Lipschitz) For a given ρ on S, L > 0 s.t. θ Θ: ln P θ(x) P θ (y) Lρ(x, y), x, y S, (1) 1e+0 1e-10 x y 1e-20 1e-30 1e-40 1e-50 1e-60 1e theta
23 Stochastic Lipschitz condition Assumption (The prior is concentrated on nice parts of F) Let the set of L-Lipschitz parameters be Θ L. Then c > 0 s.t. ξ(θ L ) 1 exp( cl), L (2) 1e+0 1e-10 x y 1e-20 1e-30 1e-40 1e-50 1e-60 1e theta
24 Stochastic Lipschitz condition Assumption (The prior is concentrated on nice parts of F) Let the set of L-Lipschitz parameters be Θ L. Then c > 0 s.t. ξ(θ L ) 1 exp( cl), L (2) 1e+50 1e+00 x y prior 1e-50 1e-100 1e-150 1e theta
25 Robustness of the posterior distribution Definition (KL divergence) D (P Q) ln dp dp. (3) dq Theorem
26 Robustness of the posterior distribution Definition (KL divergence) D (P Q) ln dp dp. (3) dq Theorem (i) Under Assumption 1, D (ξ( x) ξ( y)) 2Lρ(x, y) (4)
27 Robustness of the posterior distribution Definition (KL divergence) D (P Q) ln dp dp. (3) dq Theorem (i) Under Assumption 1, D (ξ( x) ξ( y)) 2Lρ(x, y) (4) (ii) Under Assumption 2, D (ξ( x) ξ( y)) κc ξ c ρ(x, y) (5)
28 Differential privacy of the posterior Theorem 1. Under Assumption 1, B S Θ : ξ(b x) e 2Lρ(x,y) ξ(b y) (6) i.e. the posterior is (2L, 0)-DP under ρ.
29 Differential privacy of the posterior Theorem 1. Under Assumption 1, B S Θ : ξ(b x) e 2Lρ(x,y) ξ(b y) (6) i.e. the posterior is (2L, 0)-DP under ρ. 2. Under Assumption 2, for all x, y S, B S Θ : κcξ ξ(b x) ξ(b y) ρ(x, y) 2c ( ) κcξ i.e. the posterior is 0, 2c -DP under ρ.
30 Posterior sampling query model We select a prior ξ. We observe data x. We calculate a posterior ξ( x). An adversary has sampling-based access to the posterior.
31 Posterior sampling query model We select a prior ξ. We observe data x. We calculate a posterior ξ( x). An adversary has sampling-based access to the posterior. Interactive access model At time t, the adversary observes a sample from the posterior: θ t ξ(θ x),
32 Posterior sampling query model We select a prior ξ. We observe data x. We calculate a posterior ξ( x). An adversary has sampling-based access to the posterior. Interactive access model At time t, the adversary observes a sample from the posterior: θ t ξ(θ x), More generally, A can select a question q : Θ R, where R is a response space: r t = q(θ t )
33 Posterior sampling query model We select a prior ξ. We observe data x. We calculate a posterior ξ( x). An adversary has sampling-based access to the posterior. Interactive access model At time t, the adversary observes a sample from the posterior: θ t ξ(θ x), More generally, A can select a question q : Θ R, where R is a response space: r t = q(θ t ) Non-interactive access model We simply release n samples from the posterior θ t ξ(θ x), t { 1,..., n }
34 Other differentially private mechanisms Laplace mechanism Add noise to responses to queries. r = q(x) + ω, ω Laplace(λ) Exponential mechanism Define a utility function u(x, r) p(r) e ɛu(x,r) µ(r). Subsampling Perform inference on a random sample of the data. Random projections / compressed sensing Perform a random linear projection.
35 Exponential mechanism Define a utility function u(x, r) p(r) e ɛu(x,r) µ(r). Respond with r with probability p(r). Connection to posterior mechanism Responses are parameters θ. Take u(θ, x) = log P θ (x). Take µ(θ) = ξ(θ). Then p(θ) = ξ(θ x). Rather than tuning ɛ, we can tune The prior ξ. The number of samples n.
36 Reduction of privacy to a testing problem How many samples n should we release? Different datasets x, y give different ξ( x), ξ( y). How many samples are needed to differentiate them?
37 Reduction of privacy to a testing problem How many samples n should we release? Different datasets x, y give different ξ( x), ξ( y). How many samples are needed to differentiate them? Theorem Under Assumption 1, the adversary can distinguish between data x, y with probability 1 δ if: Under Assumption 2, this becomes: ρ(x, y) 3 4Ln ln 1 δ. (7) ρ(x, y) 3c 2κC ξ n ln 1 δ. (8)
38 Conclusion Bayesian inference is inherently robust and private. Privacy is achieved by posterior sampling [Dimitrakakis et al] It suffices to select the right prior. But how? By construction for certain cases [Zhang et al] If we know constants L, c, C ξ, we can tune n. But similar problems exist more generally for DP mechanisms [Dwork] References C. Dimitrakakis, B. Nelson, A. Mitrokotsa, B. Rubinstein, Robust and Private Bayesian Inference, ALT Z. Zhang, B. Rubinstein, C. Dimitrakakis, Hide-And-Peek: Differential Privacy in Probabilistic Graphical Models, under review. C. Dwork, Differential privacy, ICALP 2006.
39 When do these assumptions hold? Example (Exponential families) Family F, with sufficient statistic T. { } p θ (x) = h(x) exp ηθ T (x) A(η θ) For a given θ Θ, we want to test if: ln(h(x)/h(y)) + ηθ (T (x) T (y)) Lρ(x, y), x, y X.
40 Example (Exponential distribution) Exponential prior with parameter c > 0, satisfies Assumption 2. Example (Discrete Bayesian networks) ln P θ(x) P θ (y) ln 1 ρ(x, y), ɛ where: ρ(x, y) is the number of edges and nodes influenced by the differences in x, y and ɛ the smallest P θ -mass placed to any event.
41 The Le Cam method for lower bounds Idea: Use S for the parameter space of an estimator.
42 The Le Cam method for lower bounds Idea: Use S for the parameter space of an estimator. The family of posterior measures Ξ { ξ( x) x S }, (9)
43 The Le Cam method for lower bounds Idea: Use S for the parameter space of an estimator. The family of posterior measures Ξ { ξ( x) x S }, (9) Lemma (Le Cam s method) Let ψ(θ) be an estimator of x. Let S 1, S 2 be 2δ-separated and say x S i ξ( x) Ξ i Ξ. Then: sup x S E ξ (ρ(ψ, x) x) δ sup ξ i co(ξ i ) ξ 1 ξ 2. (10)
44 The Le Cam method for lower bounds Idea: Use S for the parameter space of an estimator. The family of posterior measures Ξ { ξ( x) x S }, (9) Lemma (Le Cam s method) Let ψ(θ) be an estimator of x. Let S 1, S 2 be 2δ-separated and say x S i ξ( x) Ξ i Ξ. Then: sup x S E ξ (ρ(ψ, x) x) δ sup ξ i co(ξ i ) ξ 1 ξ 2. (10) Expected distance between the real and guessed data: E ξ (ρ(ψ, x) x) = ρ(ψ(θ), x) dξ(θ x), Θ
Differential Privacy for Bayesian Inference through Posterior Sampling
Journal of Machine Learning Research 8 (07) -39 Submitted 5/5; Revised /7; Published 3/7 Differential Privacy for Bayesian Inference through Posterior Sampling Christos Dimitrakakis christos.dimitrakakis@gmail.com
More informationRobust and Private Bayesian Inference
Robust and Private Bayesian Inference Christos Dimitrakakis 1, Blaine Nelson 2, Aikaterini Mitrokotsa 1, and Benjamin I. P. Rubinstein 3 1 Chalmers University of Technology, Sweden 2 University of Potsdam,
More informationAnalysis of Networks: Privacy in Bayesian Networks and Problems in Lattice Models
Analysis of Networks: Privacy in Bayesian Networks and Problems in Lattice Models A thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy The University of Melbourne,
More informationOn the Differential Privacy of Bayesian Inference
On the Differential Privacy of Bayesian Inference Zuhe Zhang, Benjamin Rubinstein, Christos Dimitrakakis To cite this version: Zuhe Zhang, Benjamin Rubinstein, Christos Dimitrakakis. On the Differential
More informationBayesian reinforcement learning
Bayesian reinforcement learning Markov decision processes and approximate Bayesian computation Christos Dimitrakakis Chalmers April 16, 2015 Christos Dimitrakakis (Chalmers) Bayesian reinforcement learning
More informationComplexity of stochastic branch and bound methods for belief tree search in Bayesian reinforcement learning
Complexity of stochastic branch and bound methods for belief tree search in Bayesian reinforcement learning Christos Dimitrakakis Informatics Institute, University of Amsterdam, Amsterdam, The Netherlands
More information1 Differential Privacy and Statistical Query Learning
10-806 Foundations of Machine Learning and Data Science Lecturer: Maria-Florina Balcan Lecture 5: December 07, 015 1 Differential Privacy and Statistical Query Learning 1.1 Differential Privacy Suppose
More informationClassical and Bayesian inference
Classical and Bayesian inference AMS 132 Claudia Wehrhahn (UCSC) Classical and Bayesian inference January 8 1 / 11 The Prior Distribution Definition Suppose that one has a statistical model with parameter
More informationBayesian Dropout. Tue Herlau, Morten Morup and Mikkel N. Schmidt. Feb 20, Discussed by: Yizhe Zhang
Bayesian Dropout Tue Herlau, Morten Morup and Mikkel N. Schmidt Discussed by: Yizhe Zhang Feb 20, 2016 Outline 1 Introduction 2 Model 3 Inference 4 Experiments Dropout Training stage: A unit is present
More informationPMR Learning as Inference
Outline PMR Learning as Inference Probabilistic Modelling and Reasoning Amos Storkey Modelling 2 The Exponential Family 3 Bayesian Sets School of Informatics, University of Edinburgh Amos Storkey PMR Learning
More informationCS 361: Probability & Statistics
March 14, 2018 CS 361: Probability & Statistics Inference The prior From Bayes rule, we know that we can express our function of interest as Likelihood Prior Posterior The right hand side contains the
More informationBayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
More informationPart 1: Expectation Propagation
Chalmers Machine Learning Summer School Approximate message passing and biomedicine Part 1: Expectation Propagation Tom Heskes Machine Learning Group, Institute for Computing and Information Sciences Radboud
More informationCalibrating Noise to Sensitivity in Private Data Analysis
Calibrating Noise to Sensitivity in Private Data Analysis Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith Mamadou H. Diallo 1 Overview Issues: preserving privacy in statistical databases Assumption:
More informationSome Asymptotic Bayesian Inference (background to Chapter 2 of Tanner s book)
Some Asymptotic Bayesian Inference (background to Chapter 2 of Tanner s book) Principal Topics The approach to certainty with increasing evidence. The approach to consensus for several agents, with increasing
More informationDifferential Privacy without Sensitivity
Differential Privacy without Sensitivity Kentaro Minami The University of Tokyo kentaro minami@mist.i.u-tokyo.ac.jp Hiromi Arai The University of Tokyo arai@dl.itc.u-tokyo.ac.jp Issei Sato The University
More informationProbabilistic and Bayesian Machine Learning
Probabilistic and Bayesian Machine Learning Lecture 1: Introduction to Probabilistic Modelling Yee Whye Teh ywteh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit University College London Why a
More informationPrivacy-Preserving Data Mining
CS 380S Privacy-Preserving Data Mining Vitaly Shmatikov slide 1 Reading Assignment Evfimievski, Gehrke, Srikant. Limiting Privacy Breaches in Privacy-Preserving Data Mining (PODS 2003). Blum, Dwork, McSherry,
More informationOutline Lecture 2 2(32)
Outline Lecture (3), Lecture Linear Regression and Classification it is our firm belief that an understanding of linear models is essential for understanding nonlinear ones Thomas Schön Division of Automatic
More informationTutorial on Probabilistic Programming with PyMC3
185.A83 Machine Learning for Health Informatics 2017S, VU, 2.0 h, 3.0 ECTS Tutorial 02-04.04.2017 Tutorial on Probabilistic Programming with PyMC3 florian.endel@tuwien.ac.at http://hci-kdd.org/machine-learning-for-health-informatics-course
More informationan introduction to bayesian inference
with an application to network analysis http://jakehofman.com january 13, 2010 motivation would like models that: provide predictive and explanatory power are complex enough to describe observed phenomena
More informationIntroduction: MLE, MAP, Bayesian reasoning (28/8/13)
STA561: Probabilistic machine learning Introduction: MLE, MAP, Bayesian reasoning (28/8/13) Lecturer: Barbara Engelhardt Scribes: K. Ulrich, J. Subramanian, N. Raval, J. O Hollaren 1 Classifiers In this
More informationBayesian Machine Learning
Bayesian Machine Learning Andrew Gordon Wilson ORIE 6741 Lecture 4 Occam s Razor, Model Construction, and Directed Graphical Models https://people.orie.cornell.edu/andrew/orie6741 Cornell University September
More informationBayesian Models in Machine Learning
Bayesian Models in Machine Learning Lukáš Burget Escuela de Ciencias Informáticas 2017 Buenos Aires, July 24-29 2017 Frequentist vs. Bayesian Frequentist point of view: Probability is the frequency of
More informationBayesian Inference by Density Ratio Estimation
Bayesian Inference by Density Ratio Estimation Michael Gutmann https://sites.google.com/site/michaelgutmann Institute for Adaptive and Neural Computation School of Informatics, University of Edinburgh
More informationConfident Bayesian Sequence Prediction. Tor Lattimore
Confident Bayesian Sequence Prediction Tor Lattimore Sequence Prediction Can you guess the next number? 1, 2, 3, 4, 5,... 3, 1, 4, 1, 5,... 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1,
More informationIntroduction to Probabilistic Machine Learning
Introduction to Probabilistic Machine Learning Piyush Rai Dept. of CSE, IIT Kanpur (Mini-course 1) Nov 03, 2015 Piyush Rai (IIT Kanpur) Introduction to Probabilistic Machine Learning 1 Machine Learning
More informationSemiparametric posterior limits
Statistics Department, Seoul National University, Korea, 2012 Semiparametric posterior limits for regular and some irregular problems Bas Kleijn, KdV Institute, University of Amsterdam Based on collaborations
More informationComputational Perception. Bayesian Inference
Computational Perception 15-485/785 January 24, 2008 Bayesian Inference The process of probabilistic inference 1. define model of problem 2. derive posterior distributions and estimators 3. estimate parameters
More informationNotes on pseudo-marginal methods, variational Bayes and ABC
Notes on pseudo-marginal methods, variational Bayes and ABC Christian Andersson Naesseth October 3, 2016 The Pseudo-Marginal Framework Assume we are interested in sampling from the posterior distribution
More informationLecture 5: GPs and Streaming regression
Lecture 5: GPs and Streaming regression Gaussian Processes Information gain Confidence intervals COMP-652 and ECSE-608, Lecture 5 - September 19, 2017 1 Recall: Non-parametric regression Input space X
More informationIntroduction to Bayesian inference
Introduction to Bayesian inference Thomas Alexander Brouwer University of Cambridge tab43@cam.ac.uk 17 November 2015 Probabilistic models Describe how data was generated using probability distributions
More informationCS 361: Probability & Statistics
October 17, 2017 CS 361: Probability & Statistics Inference Maximum likelihood: drawbacks A couple of things might trip up max likelihood estimation: 1) Finding the maximum of some functions can be quite
More informationPrivacy in Statistical Databases
Privacy in Statistical Databases Individuals x 1 x 2 x n Server/agency ) answers. A queries Users Government, researchers, businesses or) Malicious adversary What information can be released? Two conflicting
More information1 Bayesian Linear Regression (BLR)
Statistical Techniques in Robotics (STR, S15) Lecture#10 (Wednesday, February 11) Lecturer: Byron Boots Gaussian Properties, Bayesian Linear Regression 1 Bayesian Linear Regression (BLR) In linear regression,
More informationBayesian inference. Fredrik Ronquist and Peter Beerli. October 3, 2007
Bayesian inference Fredrik Ronquist and Peter Beerli October 3, 2007 1 Introduction The last few decades has seen a growing interest in Bayesian inference, an alternative approach to statistical inference.
More informationOverview of Course. Nevin L. Zhang (HKUST) Bayesian Networks Fall / 58
Overview of Course So far, we have studied The concept of Bayesian network Independence and Separation in Bayesian networks Inference in Bayesian networks The rest of the course: Data analysis using Bayesian
More informationRobustness and duality of maximum entropy and exponential family distributions
Chapter 7 Robustness and duality of maximum entropy and exponential family distributions In this lecture, we continue our study of exponential families, but now we investigate their properties in somewhat
More informationBayesian Inference and MCMC
Bayesian Inference and MCMC Aryan Arbabi Partly based on MCMC slides from CSC412 Fall 2018 1 / 18 Bayesian Inference - Motivation Consider we have a data set D = {x 1,..., x n }. E.g each x i can be the
More informationCOMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 2. Statistical Schools Adapted from slides by Ben Rubinstein Statistical Schools of Thought Remainder of lecture is to provide
More informationVariational Inference via Stochastic Backpropagation
Variational Inference via Stochastic Backpropagation Kai Fan February 27, 2016 Preliminaries Stochastic Backpropagation Variational Auto-Encoding Related Work Summary Outline Preliminaries Stochastic Backpropagation
More informationBayesian Inference. Introduction
Bayesian Inference Introduction The frequentist approach to inference holds that probabilities are intrinsicially tied (unsurprisingly) to frequencies. This interpretation is actually quite natural. What,
More informationECE521 W17 Tutorial 6. Min Bai and Yuhuai (Tony) Wu
ECE521 W17 Tutorial 6 Min Bai and Yuhuai (Tony) Wu Agenda knn and PCA Bayesian Inference k-means Technique for clustering Unsupervised pattern and grouping discovery Class prediction Outlier detection
More informationDifferential Privacy
CS 380S Differential Privacy Vitaly Shmatikov most slides from Adam Smith (Penn State) slide 1 Reading Assignment Dwork. Differential Privacy (invited talk at ICALP 2006). slide 2 Basic Setting DB= x 1
More informationCSC321 Lecture 18: Learning Probabilistic Models
CSC321 Lecture 18: Learning Probabilistic Models Roger Grosse Roger Grosse CSC321 Lecture 18: Learning Probabilistic Models 1 / 25 Overview So far in this course: mainly supervised learning Language modeling
More informationLecture 20: Introduction to Differential Privacy
6.885: Advanced Topics in Data Processing Fall 2013 Stephen Tu Lecture 20: Introduction to Differential Privacy 1 Overview This lecture aims to provide a very broad introduction to the topic of differential
More informationIntelligent Systems:
Intelligent Systems: Undirected Graphical models (Factor Graphs) (2 lectures) Carsten Rother 15/01/2015 Intelligent Systems: Probabilistic Inference in DGM and UGM Roadmap for next two lectures Definition
More informationCS 188: Artificial Intelligence Spring Today
CS 188: Artificial Intelligence Spring 2006 Lecture 9: Naïve Bayes 2/14/2006 Dan Klein UC Berkeley Many slides from either Stuart Russell or Andrew Moore Bayes rule Today Expectations and utilities Naïve
More informationThe Bayesian approach to inverse problems
The Bayesian approach to inverse problems Youssef Marzouk Department of Aeronautics and Astronautics Center for Computational Engineering Massachusetts Institute of Technology ymarz@mit.edu, http://uqgroup.mit.edu
More informationLecture 11- Differential Privacy
6.889 New Developments in Cryptography May 3, 2011 Lecture 11- Differential Privacy Lecturer: Salil Vadhan Scribes: Alan Deckelbaum and Emily Shen 1 Introduction In class today (and the next two lectures)
More informationLEARNING WITH BAYESIAN NETWORKS
LEARNING WITH BAYESIAN NETWORKS Author: David Heckerman Presented by: Dilan Kiley Adapted from slides by: Yan Zhang - 2006, Jeremy Gould 2013, Chip Galusha -2014 Jeremy Gould 2013Chip Galus May 6th, 2016
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning MCMC and Non-Parametric Bayes Mark Schmidt University of British Columbia Winter 2016 Admin I went through project proposals: Some of you got a message on Piazza. No news is
More informationPosterior Regularization
Posterior Regularization 1 Introduction One of the key challenges in probabilistic structured learning, is the intractability of the posterior distribution, for fast inference. There are numerous methods
More informationLocally Private Bayesian Inference for Count Models
Locally Private Bayesian Inference for Count Models Aaron Schein UMass Amherst Zhiwei Steven Wu Microsoft Research Mingyuan Zhou UT Austin Hanna Wallach Microsoft Research 1 Introduction Data from social
More informationMachine Learning. Bayes Basics. Marc Toussaint U Stuttgart. Bayes, probabilities, Bayes theorem & examples
Machine Learning Bayes Basics Bayes, probabilities, Bayes theorem & examples Marc Toussaint U Stuttgart So far: Basic regression & classification methods: Features + Loss + Regularization & CV All kinds
More informationLecture 35: December The fundamental statistical distances
36-705: Intermediate Statistics Fall 207 Lecturer: Siva Balakrishnan Lecture 35: December 4 Today we will discuss distances and metrics between distributions that are useful in statistics. I will be lose
More informationTerminology. Experiment = Prior = Posterior =
Review: probability RVs, events, sample space! Measures, distributions disjoint union property (law of total probability book calls this sum rule ) Sample v. population Law of large numbers Marginals,
More informationSample complexity bounds for differentially private learning
Sample complexity bounds for differentially private learning Kamalika Chaudhuri University of California, San Diego Daniel Hsu Microsoft Research Outline 1. Learning and privacy model 2. Our results: sample
More informationWhat are the Findings?
What are the Findings? James B. Rawlings Department of Chemical and Biological Engineering University of Wisconsin Madison Madison, Wisconsin April 2010 Rawlings (Wisconsin) Stating the findings 1 / 33
More informationLecture : Probabilistic Machine Learning
Lecture : Probabilistic Machine Learning Riashat Islam Reasoning and Learning Lab McGill University September 11, 2018 ML : Many Methods with Many Links Modelling Views of Machine Learning Machine Learning
More informationPHASES OF STATISTICAL ANALYSIS 1. Initial Data Manipulation Assembling data Checks of data quality - graphical and numeric
PHASES OF STATISTICAL ANALYSIS 1. Initial Data Manipulation Assembling data Checks of data quality - graphical and numeric 2. Preliminary Analysis: Clarify Directions for Analysis Identifying Data Structure:
More informationIntroduction to Bayesian Statistics
Bayesian Parameter Estimation Introduction to Bayesian Statistics Harvey Thornburg Center for Computer Research in Music and Acoustics (CCRMA) Department of Music, Stanford University Stanford, California
More informationBayesian Inference: Posterior Intervals
Bayesian Inference: Posterior Intervals Simple values like the posterior mean E[θ X] and posterior variance var[θ X] can be useful in learning about θ. Quantiles of π(θ X) (especially the posterior median)
More informationProbabilistic Graphical Models
Parameter Estimation December 14, 2015 Overview 1 Motivation 2 3 4 What did we have so far? 1 Representations: how do we model the problem? (directed/undirected). 2 Inference: given a model and partially
More informationExpectation Maximization
Expectation Maximization Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr 1 /
More informationBayesian Analysis (Optional)
Bayesian Analysis (Optional) 1 2 Big Picture There are two ways to conduct statistical inference 1. Classical method (frequentist), which postulates (a) Probability refers to limiting relative frequencies
More informationHow Well Does Privacy Compose?
How Well Does Privacy Compose? Thomas Steinke Almaden IPAM/UCLA, Los Angeles CA, 10 Jan. 2018 This Talk Composition!! What is composition? Why is it important? Composition & high-dimensional (e.g. genetic)
More informationBayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
More informationCPSC 340: Machine Learning and Data Mining
CPSC 340: Machine Learning and Data Mining MLE and MAP Original version of these slides by Mark Schmidt, with modifications by Mike Gelbart. 1 Admin Assignment 4: Due tonight. Assignment 5: Will be released
More informationCS 229: Lecture 7 Notes
CS 9: Lecture 7 Notes Scribe: Hirsh Jain Lecturer: Angela Fan Overview Overview of today s lecture: Hypothesis Testing Total Variation Distance Pinsker s Inequality Application of Pinsker s Inequality
More informationLecture 16 Deep Neural Generative Models
Lecture 16 Deep Neural Generative Models CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago May 22, 2017 Approach so far: We have considered simple models and then constructed
More informationConsensus and Distributed Inference Rates Using Network Divergence
DIMACS August 2017 1 / 26 Consensus and Distributed Inference Rates Using Network Divergence Anand D. Department of Electrical and Computer Engineering, The State University of New Jersey August 23, 2017
More informationNaive Bayes classification
Naive Bayes classification Christos Dimitrakakis December 4, 2015 1 Introduction One of the most important methods in machine learning and statistics is that of Bayesian inference. This is the most fundamental
More informationNonparameteric Regression:
Nonparameteric Regression: Nadaraya-Watson Kernel Regression & Gaussian Process Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
More informationPart 3 Robust Bayesian statistics & applications in reliability networks
Tuesday 9:00-12:30 Part 3 Robust Bayesian statistics & applications in reliability networks by Gero Walter 69 Robust Bayesian statistics & applications in reliability networks Outline Robust Bayesian Analysis
More informationGWAS IV: Bayesian linear (variance component) models
GWAS IV: Bayesian linear (variance component) models Dr. Oliver Stegle Christoh Lippert Prof. Dr. Karsten Borgwardt Max-Planck-Institutes Tübingen, Germany Tübingen Summer 2011 Oliver Stegle GWAS IV: Bayesian
More informationQuiz 2 Date: Monday, November 21, 2016
10-704 Information Processing and Learning Fall 2016 Quiz 2 Date: Monday, November 21, 2016 Name: Andrew ID: Department: Guidelines: 1. PLEASE DO NOT TURN THIS PAGE UNTIL INSTRUCTED. 2. Write your name,
More informationLecture 3: More on regularization. Bayesian vs maximum likelihood learning
Lecture 3: More on regularization. Bayesian vs maximum likelihood learning L2 and L1 regularization for linear estimators A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting
More informationInference as Optimization
Inference as Optimization Sargur Srihari srihari@cedar.buffalo.edu 1 Topics in Inference as Optimization Overview Exact Inference revisited The Energy Functional Optimizing the Energy Functional 2 Exact
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning Empirical Bayes, Hierarchical Bayes Mark Schmidt University of British Columbia Winter 2017 Admin Assignment 5: Due April 10. Project description on Piazza. Final details coming
More informationCourse: ESO-209 Home Work: 1 Instructor: Debasis Kundu
Home Work: 1 1. Describe the sample space when a coin is tossed (a) once, (b) three times, (c) n times, (d) an infinite number of times. 2. A coin is tossed until for the first time the same result appear
More informationAnswering Many Queries with Differential Privacy
6.889 New Developments in Cryptography May 6, 2011 Answering Many Queries with Differential Privacy Instructors: Shafi Goldwasser, Yael Kalai, Leo Reyzin, Boaz Barak, and Salil Vadhan Lecturer: Jonathan
More informationLecture 6: Model Checking and Selection
Lecture 6: Model Checking and Selection Melih Kandemir melih.kandemir@iwr.uni-heidelberg.de May 27, 2014 Model selection We often have multiple modeling choices that are equally sensible: M 1,, M T. Which
More informationModeling Data Correlations in Private Data Mining with Markov Model and Markov Networks. Yang Cao Emory University
Modeling Data Correlations in Private Data Mining with Markov Model and Markov Networks Yang Cao Emory University 207..5 Outline Data Mining with Differential Privacy (DP) Scenario: Spatiotemporal Data
More informationCS839: Probabilistic Graphical Models. Lecture 7: Learning Fully Observed BNs. Theo Rekatsinas
CS839: Probabilistic Graphical Models Lecture 7: Learning Fully Observed BNs Theo Rekatsinas 1 Exponential family: a basic building block For a numeric random variable X p(x ) =h(x)exp T T (x) A( ) = 1
More informationBayesian RL Seminar. Chris Mansley September 9, 2008
Bayesian RL Seminar Chris Mansley September 9, 2008 Bayes Basic Probability One of the basic principles of probability theory, the chain rule, will allow us to derive most of the background material in
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Uncertainty & Probabilities & Bandits Daniel Hennes 16.11.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Uncertainty Probability
More informationCS Lecture 19. Exponential Families & Expectation Propagation
CS 6347 Lecture 19 Exponential Families & Expectation Propagation Discrete State Spaces We have been focusing on the case of MRFs over discrete state spaces Probability distributions over discrete spaces
More informationProbabilistic Graphical Networks: Definitions and Basic Results
This document gives a cursory overview of Probabilistic Graphical Networks. The material has been gleaned from different sources. I make no claim to original authorship of this material. Bayesian Graphical
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationGWAS V: Gaussian processes
GWAS V: Gaussian processes Dr. Oliver Stegle Christoh Lippert Prof. Dr. Karsten Borgwardt Max-Planck-Institutes Tübingen, Germany Tübingen Summer 2011 Oliver Stegle GWAS V: Gaussian processes Summer 2011
More informationInconsistency of Bayesian inference when the model is wrong, and how to repair it
Inconsistency of Bayesian inference when the model is wrong, and how to repair it Peter Grünwald Thijs van Ommen Centrum Wiskunde & Informatica, Amsterdam Universiteit Leiden June 3, 2015 Outline 1 Introduction
More informationDecision Trees. Tirgul 5
Decision Trees Tirgul 5 Using Decision Trees It could be difficult to decide which pet is right for you. We ll find a nice algorithm to help us decide what to choose without having to think about it. 2
More informationBiology as Information Dynamics
Biology as Information Dynamics John Baez Biological Complexity: Can It Be Quantified? Beyond Center February 2, 2017 IT S ALL RELATIVE EVEN INFORMATION! When you learn something, how much information
More informationBayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
More information1 Hoeffding s Inequality
Proailistic Method: Hoeffding s Inequality and Differential Privacy Lecturer: Huert Chan Date: 27 May 22 Hoeffding s Inequality. Approximate Counting y Random Sampling Suppose there is a ag containing
More informationIntegrated Non-Factorized Variational Inference
Integrated Non-Factorized Variational Inference Shaobo Han, Xuejun Liao and Lawrence Carin Duke University February 27, 2014 S. Han et al. Integrated Non-Factorized Variational Inference February 27, 2014
More informationDS-GA 1002 Lecture notes 11 Fall Bayesian statistics
DS-GA 100 Lecture notes 11 Fall 016 Bayesian statistics In the frequentist paradigm we model the data as realizations from a distribution that depends on deterministic parameters. In contrast, in Bayesian
More informationGeneral Bayesian Inference I
General Bayesian Inference I Outline: Basic concepts, One-parameter models, Noninformative priors. Reading: Chapters 10 and 11 in Kay-I. (Occasional) Simplified Notation. When there is no potential for
More informationOutline lecture 2 2(30)
Outline lecture 2 2(3), Lecture 2 Linear Regression it is our firm belief that an understanding of linear models is essential for understanding nonlinear ones Thomas Schön Division of Automatic Control
More information