Axiomatic Attribution of Neural Networks
|
|
|
- Reginald Thomas
- 5 years ago
- Views:
Transcription
1 Axiomatic Attribution of Neural Networks Mukund Sundararajan* 1, Ankur Taly* 1, Qiqi Yan* 1 1 Google Inc. Presenter: Arshdeep Sekhon Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 1 / 25
2 Motivation 1 Use attribution concept: assigning blame/credit to features 2 Understand the input output behaviour of a network. 3 Interpretability of black box neural networks. Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 2 / 25
![Attribution of Neural Networks Attribution Given a function F : R n [0, 1], and an input x R n, Attribution of x relative to baseline x is A F (x, x ) = (a 1, a 2,, a n ) R n](/docs-images/86/93999152/images/3-0.jpg "where a i is the contribution of x i to prediction at x i.e. F (x) Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 3 / 25")
3 Attribution of Neural Networks Attribution Given a function F : R n [0, 1], and an input x R n, Attribution of x relative to baseline x is A F (x, x ) = (a 1, a 2,, a n ) R n where a i is the contribution of x i to prediction at x i.e. F (x) Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 3 / 25
4 Challenges for Attribution methods Hard to evaluate: Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 4 / 25
5 Challenges for Attribution methods Hard to evaluate: If an attribution method assigns an incorrect attribution: Is the model not doing well (unseen data)? Or is the attribution method not good? Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 4 / 25
6 Challenges for Attribution methods Hard to evaluate: If an attribution method assigns an incorrect attribution: Is the model not doing well (unseen data)? Or is the attribution method not good? Proposed Approach: Introduce two axioms/desirable characteristics that every attribution method should satisfy previous methods do not satisfy atleast one of these two axioms Introduce a new method that satisfies these two axioms Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 4 / 25
7 Related Work DeepLift Layer Wise Relevance Propagation Deconvolutional Networks Guided Backpropagation Gradients: Saliency maps, Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 5 / 25
8 Need for a baseline/ choosing a baseline 1 Intuituve: When we assign blame to a certain cause we implicitly consider the absence of the cause for comparison 2 a natural baseline exists in the input space where the prediction is neutral. 3 Example: in object recognition networks, it is the black image. (does not indicate anything) 4 Example: all zero embedding vector for text based tasks indicates nothing. Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 6 / 25
9 Axiom 1: Sensitivity Sensitivity(a) if for every input and baseline that differ in one feature but have different predictions then the differing feature should be given a non-zero attribution Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 7 / 25
10 Gradients violate Axiom 1 Consider ReLU Network, f (x) = 1 ReLU(1 x). baseline x = 0 : F (x) = 0 input x = 2: F (x) = 1 Gradient = 0 at x=2 : function flattens at x = 2 Violate Sensitivity(a): If input is different, the attribution should be non zero. Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 8 / 25
11 DeConvNets and Guided Back Propagation violate Axiom 1 Value of function for fixed x 1 > 1 decreases linearly as x 2 increases from 0 to x 1 1 Guided Back Propagation/ DeConvNet Rule: zero attribution of x 2 because the back-propagated signal received at the node is negative hence not back propagated back Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 9 / 25
12 Axiom 2: Implementation Invariance Functionally Equivalent Networks Two networks are functionally equivalent if their outputs are equal for all inputs, despite having very different implementations. Implementation Invariance Attribution methods should satisfy Implementation Invariance, i.e., the attributions are always identical for two functionally equivalent networks. Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 10 / 25
13 Gradients and Implementation Invariance Gradients by default are implementation invariant f h h g (1) 1 If h is some implementation detail of the system, gradient f g can either be computed directly or through the chain rule. 2 Chain Rule fails for discrete gradients f (x 1 ) f (x 0 ) g(x 1 ) g(x 0 ) f (x 1) f (x 0 ) h(x 1 ) h(x 0 ) h(x 1 ) h(x 0 ) g(x 1 ) g(x 0 ) (2) Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 11 / 25
14 Why implementation invariance? 1 If an attribution method fails to satisfy Implementation Invariance, the attributions are potentially sensitive to unimportant aspects of the models. 2 For instance, if the network architecture has more degrees of freedom than needed to represent a function then there may be two sets of values for the network parameters that lead to the same function. The training procedure can converge at either set of values depending on the initializtion or for other reasons, but the underlying network function would remain the same. It is undesirable that attributions differ for such reasons. Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 12 / 25
15 Proposed Method: Integrated Gradients 1 Combine sensitivity with Implementation Invariance of true gradients 2 a function F : R n [0, 1] a deep network 3 input: x R n 4 baseline: x R n Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 13 / 25
16 Integrated Gradients 1 Consider straightline path from x to x 2 calculate gradients at each point of these paths 3 Integrated gradients are obtained by cumulating these gradients. 4 1 IntegratedGrads i (x) ::= (x i x F (x + α (x x )) i ) dα (3) 0 x i Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 14 / 25
17 Axiom 3: Completeness The attributions add up to the difference between the output of F at x and x Completeness If F : R n R is differentiable almost everywhere: n IntegratedGrads i (x) = F (x) F (x ) (4) i=1 If the method satisfies completeness, it clearly satisfies Axiom 1 of sensitivity desirable if the networks score is used in a numeric sense Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 15 / 25
18 Generalization: Path Gradients Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 16 / 25
19 Path Gradients let γ = (γ 1,, γ n ) : [0, 1] R n be a smooth function specifying a path in R n fron x to x γ(0) = x and γ(1) = x path integrated gradients: integrate along the path γ(α) for α [0, 1] Path Integrated Gradients PathIntegratedGrads γ i (x) ::= 1 α=0 F (γ(α)) γ i (α) γ i (α) α dα Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 17 / 25
20 Axioms satisfied by Path Gradients Sensitivity(b): If a function does not depend on the value of a certain variable, attribution is zero for that variable Linearity Linearly compose a function f 3 = a f 1 + b f 2 Attributions should also be the weighted sum of the attributions for f 1, f 2 Intuitively, preerve linearity within network Path methods satisfy Implementation Invariance, Sensitivity(b), Linearity, and Completeness [add citation] Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 18 / 25
21 Integrated Gradients are symmetry preserving Symmetry preserving inputs Two input variables are symmetry preserving if swapping them does not change the output F (x, y) = F (y, x) x, y (5) Symmetry preserving attribution methods For all inputs that have identical values for symmetric variables and baselines that have identical values for symmetric variables, the symmetric variables receive identical attributions. Proof: 1 non straightline path γ : [0, 1] R n 2 Without loss of generality, there exists t 0 [0, 1] such that for two dimensions i, j, γ i (t 0 ) > γ j (t 0 ). 3 (t 1, t 2 ) is the maximum real open interval containing t 0 s.t. γ i (t) > γ j (t) t (t 1, t 2 ) Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 19 / 25
22 Proof: Integrated Gradients are Symmetry Preserving 1 Define f : x [0, 1] n R 2 0 if min(x i, x j ) a 3 (b a) 2 if max(x i, x j ) b 4 otherwise (x i a)(x j a) 5 compute the attributions of f at x = (1,..., 1) n in with baseline x 0 = (0,..., 0) n. 6 For t / [t1, t2] the function is a constant, zero attribution to all 7 For t [t1, t2] the integrand of attribution of f is γ j (t) a to x i and γ i (t) a to x j 8 latter is always strictly larger by choice of the interval. 9 contradiction: x j gets a larger attribution than x i : contradiction Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 20 / 25
23 Computing Integrated Gradients Riemman Sum approximation of an integral: S = m f (xi ) (x i ) (6) i=1 IntegratedGrads approx i (x) = (x i x i ) m k=1 ( F (x + k m (x x )) x i ) 1 m (7) Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 21 / 25
24 Experiment 1: Object Recognition Network using the GoogleNet architecture and trained over the ImageNet object recognition The gradients are computed for the output of the highest-scoring class with respect to pixel of the input image dataset. Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 22 / 25
25 Experiment 2: Question Classification What type of question: For example, yes/no or a date? triggers: what / when Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 23 / 25
26 Experiment 3: Neural Machine Translation Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 24 / 25
27 Conclusions 1 integrated gradients approach that attributes the prediction of a deep network to its inputs 2 Easy to implement 3 clarifies desirable features of an attribution method using an axiomatic framework 4 does not address the interactions between the input features or logic of network Mukund Sundararajan*, Ankur Taly*, Qiqi Yan*Axiomatic Attribution of Neural Networks Presenter: Arshdeep Sekhon 25 / 25
Axiomatic Attribution for Deep Networks
Mukund Sundararajan * 1 Ankur Taly * 1 Qiqi Yan * 1 Abstract We study the problem of attributing the prediction of a deep network to its input features, a problem previously studied by several other works.
arxiv: v1 [cs.lg] 26 Jul 2018
![arxiv: v1 [cs.lg] 26 Jul 2018](/thumbs/88/116101853.jpg "arxiv: v1 [cs.lg] 26 Jul 2018")
Computationally Efficient Measures of Internal Neuron Importance Avanti Shrikumar* Stanford University avanti@stanford.edu Jocelin Su* Evergreen Valley High School jocelin.su@gmail.com Anshul Kundaje Stanford
Introduction to Machine Learning Spring 2018 Note Neural Networks
CS 189 Introduction to Machine Learning Spring 2018 Note 14 1 Neural Networks Neural networks are a class of compositional function approximators. They come in a variety of shapes and sizes. In this class,
Ch.6 Deep Feedforward Networks (2/3)
Ch.6 Deep Feedforward Networks (2/3) 16. 10. 17. (Mon.) System Software Lab., Dept. of Mechanical & Information Eng. Woonggy Kim 1 Contents 6.3. Hidden Units 6.3.1. Rectified Linear Units and Their Generalizations
Neural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann
Neural Networks with Applications to Vision and Language Feedforward Networks Marco Kuhlmann Feedforward networks Linear separability x 2 x 2 0 1 0 1 0 0 x 1 1 0 x 1 linearly separable not linearly separable
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS LAST TIME Intro to cudnn Deep neural nets using cublas and cudnn TODAY Building a better model for image classification Overfitting
Artificial Neural Networks D B M G. Data Base and Data Mining Group of Politecnico di Torino. Elena Baralis. Politecnico di Torino
Artificial Neural Networks Data Base and Data Mining Group of Politecnico di Torino Elena Baralis Politecnico di Torino Artificial Neural Networks Inspired to the structure of the human brain Neurons as
Understanding How ConvNets See
Understanding How ConvNets See Slides from Andrej Karpathy Springerberg et al, Striving for Simplicity: The All Convolutional Net (ICLR 2015 workshops) CSC321: Intro to Machine Learning and Neural Networks,
Learning the Number of Neurons in Deep Networks
Learning the Number of Jose M. Alvarez 1 Mathieu Salzmanno 2 1 Data61 @ CSIRO,Canberra, ACT 2601, Australia 2 CVLab, EPFL,CH-1015 Lausanne, Switzerland NIPS,2016 Presenter: Arshdeep Sekhon NIPS,2016 Presenter:
Tasks ADAS. Self Driving. Non-machine Learning. Traditional MLP. Machine-Learning based method. Supervised CNN. Methods. Deep-Learning based
UNDERSTANDING CNN ADAS Tasks Self Driving Localizati on Perception Planning/ Control Driver state Vehicle Diagnosis Smart factory Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning
How to do backpropagation in a brain
How to do backpropagation in a brain Geoffrey Hinton Canadian Institute for Advanced Research & University of Toronto & Google Inc. Prelude I will start with three slides explaining a popular type of deep
Convolutional Neural Network Architecture
Convolutional Neural Network Architecture Zhisheng Zhong Feburary 2nd, 2018 Zhisheng Zhong Convolutional Neural Network Architecture Feburary 2nd, 2018 1 / 55 Outline 1 Introduction of Convolution Motivation
Convolutional Neural Networks
Convolutional Neural Networks Books» http://www.deeplearningbook.org/ Books http://neuralnetworksanddeeplearning.com/.org/ reviews» http://www.deeplearningbook.org/contents/linear_algebra.html» http://www.deeplearningbook.org/contents/prob.html»
Artificial Neural Networks. Introduction to Computational Neuroscience Tambet Matiisen
Artificial Neural Networks Introduction to Computational Neuroscience Tambet Matiisen 2.04.2018 Artificial neural network NB! Inspired by biology, not based on biology! Applications Automatic speech recognition
Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Motivation Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses
Spatial Transformer Networks
BIL722 - Deep Learning for Computer Vision Spatial Transformer Networks Max Jaderberg Andrew Zisserman Karen Simonyan Koray Kavukcuoglu Contents Introduction to Spatial Transformers Related Works Spatial
Feedforward Neural Networks
Feedforward Neural Networks Michael Collins 1 Introduction In the previous notes, we introduced an important class of models, log-linear models. In this note, we describe feedforward neural networks, which
Neural Networks. David Rosenberg. July 26, New York University. David Rosenberg (New York University) DS-GA 1003 July 26, / 35
Neural Networks David Rosenberg New York University July 26, 2017 David Rosenberg (New York University) DS-GA 1003 July 26, 2017 1 / 35 Neural Networks Overview Objectives What are neural networks? How
Machine Learning for Computer Vision 8. Neural Networks and Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group
Machine Learning for Computer Vision 8. Neural Networks and Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group INTRODUCTION Nonlinear Coordinate Transformation http://cs.stanford.edu/people/karpathy/convnetjs/
Cheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
Jakub Hajic Artificial Intelligence Seminar I
Jakub Hajic Artificial Intelligence Seminar I. 11. 11. 2014 Outline Key concepts Deep Belief Networks Convolutional Neural Networks A couple of questions Convolution Perceptron Feedforward Neural Network
Machine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
Artificial Neural Networks
Introduction ANN in Action Final Observations Application: Poverty Detection Artificial Neural Networks Alvaro J. Riascos Villegas University of los Andes and Quantil July 6 2018 Artificial Neural Networks
Deep Learning: Self-Taught Learning and Deep vs. Shallow Architectures. Lecture 04
Deep Learning: Self-Taught Learning and Deep vs. Shallow Architectures Lecture 04 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Self-Taught Learning 1. Learn
Deep Feedforward Networks
Deep Feedforward Networks Liu Yang March 30, 2017 Liu Yang Short title March 30, 2017 1 / 24 Overview 1 Background A general introduction Example 2 Gradient based learning Cost functions Output Units 3
Machine Learning Basics
Security and Fairness of Deep Learning Machine Learning Basics Anupam Datta CMU Spring 2019 Image Classification Image Classification Image classification pipeline Input: A training set of N images, each
(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
What Do Neural Networks Do? MLP Lecture 3 Multi-layer networks 1
What Do Neural Networks Do? MLP Lecture 3 Multi-layer networks 1 Multi-layer networks Steve Renals Machine Learning Practical MLP Lecture 3 7 October 2015 MLP Lecture 3 Multi-layer networks 2 What Do Single
From perceptrons to word embeddings. Simon Šuster University of Groningen
From perceptrons to word embeddings Simon Šuster University of Groningen Outline A basic computational unit Weighting some input to produce an output: classification Perceptron Classify tweets Written
Neural Networks 2. 2 Receptive fields and dealing with image inputs
CS 446 Machine Learning Fall 2016 Oct 04, 2016 Neural Networks 2 Professor: Dan Roth Scribe: C. Cheng, C. Cervantes Overview Convolutional Neural Networks Recurrent Neural Networks 1 Introduction There
Need for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
CSE 190 Fall 2015 Midterm DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO START!!!!
CSE 190 Fall 2015 Midterm DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO START!!!! November 18, 2015 THE EXAM IS CLOSED BOOK. Once the exam has started, SORRY, NO TALKING!!! No, you can t even say see ya
Lecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets)
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets) Sanjeev Arora Elad Hazan Recap: Structure of a deep
Understanding CNNs using visualisation and transformation analysis
Understanding CNNs using visualisation and transformation analysis Andrea Vedaldi Medical Imaging Summer School August 2016 Image representations 2 encoder Φ representation An encoder maps the data into
CS 4700: Foundations of Artificial Intelligence
CS 4700: Foundations of Artificial Intelligence Prof. Bart Selman selman@cs.cornell.edu Machine Learning: Neural Networks R&N 18.7 Intro & perceptron learning 1 2 Neuron: How the brain works # neurons
Machine Learning Lecture 10
Machine Learning Lecture 10 Neural Networks 26.11.2018 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Today s Topic Deep Learning 2 Course Outline Fundamentals Bayes
A practical theory for designing very deep convolutional neural networks. Xudong Cao.
A practical theory for designing very deep convolutional neural networks Xudong Cao notcxd@gmail.com Abstract Going deep is essential for deep learning. However it is not easy, there are many ways of going
<Special Topics in VLSI> Learning for Deep Neural Networks (Back-propagation)
Learning for Deep Neural Networks (Back-propagation) Outline Summary of Previous Standford Lecture Universal Approximation Theorem Inference vs Training Gradient Descent Back-Propagation
Neural Network Back-propagation HYUNG IL KOO
Neural Network Back-propagation HYUNG IL KOO Hidden Layer Representations Backpropagation has an ability to discover useful intermediate representations at the hidden unit layers inside the networks which
Introduction Neural Networks - Architecture Network Training Small Example - ZIP Codes Summary. Neural Networks - I. Henrik I Christensen
Neural Networks - I Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I Christensen (RIM@GT) Neural Networks 1 /
Multilayer Neural Networks. (sometimes called Multilayer Perceptrons or MLPs)
Multilayer Neural Networks (sometimes called Multilayer Perceptrons or MLPs) Linear separability Hyperplane In 2D: w x + w 2 x 2 + w 0 = 0 Feature x 2 = w w 2 x w 0 w 2 Feature 2 A perceptron can separate
Multilayer Neural Networks. (sometimes called Multilayer Perceptrons or MLPs)
Multilayer Neural Networks (sometimes called Multilayer Perceptrons or MLPs) Linear separability Hyperplane In 2D: w 1 x 1 + w 2 x 2 + w 0 = 0 Feature 1 x 2 = w 1 w 2 x 1 w 0 w 2 Feature 2 A perceptron
Introduction to Machine Learning
Introduction to Machine Learning Neural Networks Varun Chandola x x 5 Input Outline Contents February 2, 207 Extending Perceptrons 2 Multi Layered Perceptrons 2 2. Generalizing to Multiple Labels.................
Explaining Machine Learning Decisions
Explaining Machine Learning Decisions Grégoire Montavon, TU Berlin Joint work with: Wojciech Samek, Klaus-Robert Müller, Sebastian Lapuschkin, Alexander Binder 18/09/2018 Intl. Workshop ML & AI, Telecom
BACKPROPAGATION. Neural network training optimization problem. Deriving backpropagation
BACKPROPAGATION Neural network training optimization problem min J(w) w The application of gradient descent to this problem is called backpropagation. Backpropagation is gradient descent applied to J(w)
Machine Learning and Data Mining. Multi-layer Perceptrons & Neural Networks: Basics. Prof. Alexander Ihler
+ Machine Learning and Data Mining Multi-layer Perceptrons & Neural Networks: Basics Prof. Alexander Ihler Linear Classifiers (Perceptrons) Linear Classifiers a linear classifier is a mapping which partitions
Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers
Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers Alexander Binder 1, Grégoire Montavon 2, Sebastian Lapuschkin 3, Klaus-Robert Müller 2,4, and Wojciech Samek 3 1 ISTD
Machine Learning for Signal Processing Neural Networks Continue. Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec 2016
Machine Learning for Signal Processing Neural Networks Continue Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec 2016 1 So what are neural networks?? Voice signal N.Net Transcription Image N.Net Text
Introduction to Natural Computation. Lecture 9. Multilayer Perceptrons and Backpropagation. Peter Lewis
Introduction to Natural Computation Lecture 9 Multilayer Perceptrons and Backpropagation Peter Lewis 1 / 25 Overview of the Lecture Why multilayer perceptrons? Some applications of multilayer perceptrons.
5.6 Nonparametric Logistic Regression
5.6 onparametric Logistic Regression Dmitri Dranishnikov University of Florida Statistical Learning onparametric Logistic Regression onparametric? Doesnt mean that there are no parameters. Just means that
Introduction to (Convolutional) Neural Networks
Introduction to (Convolutional) Neural Networks Philipp Grohs Summer School DL and Vis, Sept 2018 Syllabus 1 Motivation and Definition 2 Universal Approximation 3 Backpropagation 4 Stochastic Gradient
Lecture 4: Perceptrons and Multilayer Perceptrons
Lecture 4: Perceptrons and Multilayer Perceptrons Cognitive Systems II - Machine Learning SS 2005 Part I: Basic Approaches of Concept Learning Perceptrons, Artificial Neuronal Networks Lecture 4: Perceptrons
Introduction to Deep Learning
Introduction to Deep Learning A. G. Schwing & S. Fidler University of Toronto, 2015 A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding 2015 1 / 39 Outline 1 Universality of Neural Networks
arxiv: v1 [cs.lg] 30 May 2018
![arxiv: v1 [cs.lg] 30 May 2018](/thumbs/88/114888560.jpg "arxiv: v1 [cs.lg] 30 May 2018")
How Important Is a Neuron? Kedar Dhamdhere Google AI kedar@google.com Mukund Sundararajan Google AI mukunds@google.com arxiv:1805.12233v1 [cs.lg] 30 May 2018 Qiqi Yan Google AI qiqiyan@google.com Abstract
CS168: The Modern Algorithmic Toolbox Lecture #6: Regularization
CS168: The Modern Algorithmic Toolbox Lecture #6: Regularization Tim Roughgarden & Gregory Valiant April 18, 2018 1 The Context and Intuition behind Regularization Given a dataset, and some class of models
Deep Feedforward Networks
Deep Feedforward Networks Yongjin Park 1 Goal of Feedforward Networks Deep Feedforward Networks are also called as Feedforward neural networks or Multilayer Perceptrons Their Goal: approximate some function
Convolution and Pooling as an Infinitely Strong Prior
Convolution and Pooling as an Infinitely Strong Prior Sargur Srihari srihari@buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 Topics in Convolutional
A Randomized Approach for Crowdsourcing in the Presence of Multiple Views
A Randomized Approach for Crowdsourcing in the Presence of Multiple Views Presenter: Yao Zhou joint work with: Jingrui He - 1 - Roadmap Motivation Proposed framework: M2VW Experimental results Conclusion
Neural Networks biological neuron artificial neuron 1
Neural Networks biological neuron artificial neuron 1 A two-layer neural network Output layer (activation represents classification) Weighted connections Hidden layer ( internal representation ) Input
Sequence Modeling with Neural Networks
Sequence Modeling with Neural Networks Harini Suresh y 0 y 1 y 2 s 0 s 1 s 2... x 0 x 1 x 2 hat is a sequence? This morning I took the dog for a walk. sentence medical signals speech waveform Successes
Lecture 7 Convolutional Neural Networks
Lecture 7 Convolutional Neural Networks CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 17, 2017 We saw before: ŷ x 1 x 2 x 3 x 4 A series of matrix multiplications:
Advanced Machine Learning
Advanced Machine Learning Lecture 4: Deep Learning Essentials Pierre Geurts, Gilles Louppe, Louis Wehenkel 1 / 52 Outline Goal: explain and motivate the basic constructs of neural networks. From linear
Binary Classification / Perceptron
Binary Classification / Perceptron Nicholas Ruozzi University of Texas at Dallas Slides adapted from David Sontag and Vibhav Gogate Supervised Learning Input: x 1, y 1,, (x n, y n ) x i is the i th data
Introduction to Machine Learning (67577)
Introduction to Machine Learning (67577) Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Deep Learning Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks
Deep Feedforward Networks. Sargur N. Srihari
Deep Feedforward Networks Sargur N. srihari@cedar.buffalo.edu 1 Topics Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation and Other Differentiation
Neural networks and support vector machines
Neural netorks and support vector machines Perceptron Input x 1 Weights 1 x 2 x 3... x D 2 3 D Output: sgn( x + b) Can incorporate bias as component of the eight vector by alays including a feature ith
Reading Group on Deep Learning Session 1
Reading Group on Deep Learning Session 1 Stephane Lathuiliere & Pablo Mesejo 2 June 2016 1/31 Contents Introduction to Artificial Neural Networks to understand, and to be able to efficiently use, the popular
Revealed Preference 2011
Revealed Preference 2011 Motivation: 1. up until now we have started with preference and then described behaviour 2. revealed preference works backwards - start with behaviour and describe preferences
Deep Learning (CNNs)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Deep Learning (CNNs) Deep Learning Readings: Murphy 28 Bishop - - HTF - - Mitchell
Machine Learning Lecture 12
Machine Learning Lecture 12 Neural Networks 30.11.2017 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Course Outline Fundamentals Bayes Decision Theory Probability
Lecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
Neural Networks. Nicholas Ruozzi University of Texas at Dallas
Neural Networks Nicholas Ruozzi University of Texas at Dallas Handwritten Digit Recognition Given a collection of handwritten digits and their corresponding labels, we d like to be able to correctly classify
Backpropagation Introduction to Machine Learning. Matt Gormley Lecture 12 Feb 23, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Backpropagation Matt Gormley Lecture 12 Feb 23, 2018 1 Neural Networks Outline
Neural networks COMS 4771
Neural networks COMS 4771 1. Logistic regression Logistic regression Suppose X = R d and Y = {0, 1}. A logistic regression model is a statistical model where the conditional probability function has a
Feed-forward Network Functions
Feed-forward Network Functions Sargur Srihari Topics 1. Extension of linear models 2. Feed-forward Network Functions 3. Weight-space symmetries 2 Recap of Linear Models Linear Models for Regression, Classification
Final Examination CS 540-2: Introduction to Artificial Intelligence
Final Examination CS 540-2: Introduction to Artificial Intelligence May 7, 2017 LAST NAME: SOLUTIONS FIRST NAME: Problem Score Max Score 1 14 2 10 3 6 4 10 5 11 6 9 7 8 9 10 8 12 12 8 Total 100 1 of 11
SGD and Deep Learning
SGD and Deep Learning Subgradients Lets make the gradient cheating more formal. Recall that the gradient is the slope of the tangent. f(w 1 )+rf(w 1 ) (w w 1 ) Non differentiable case? w 1 Subgradients
Need for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
Lecture 12. Neural Networks Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 12 Neural Networks 10.12.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ leibe@vision.rwth-aachen.de This Lecture: Advanced Machine Learning Regression
Fast classification using sparsely active spiking networks. Hesham Mostafa Institute of neural computation, UCSD
Fast classification using sparsely active spiking networks Hesham Mostafa Institute of neural computation, UCSD Artificial networks vs. spiking networks backpropagation output layer Multi-layer networks
Deep Feedforward Networks. Seung-Hoon Na Chonbuk National University
Deep Feedforward Networks Seung-Hoon Na Chonbuk National University Neural Network: Types Feedforward neural networks (FNN) = Deep feedforward networks = multilayer perceptrons (MLP) No feedback connections
RegML 2018 Class 8 Deep learning
RegML 2018 Class 8 Deep learning Lorenzo Rosasco UNIGE-MIT-IIT June 18, 2018 Supervised vs unsupervised learning? So far we have been thinking of learning schemes made in two steps f(x) = w, Φ(x) F, x
Natural Language Processing with Deep Learning CS224N/Ling284
Natural Language Processing with Deep Learning CS224N/Ling284 Lecture 4: Word Window Classification and Neural Networks Richard Socher Organization Main midterm: Feb 13 Alternative midterm: Friday Feb
Bayesian Networks (Part I)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Bayesian Networks (Part I) Graphical Model Readings: Murphy 10 10.2.1 Bishop 8.1,
Supervised Learning. George Konidaris
Supervised Learning George Konidaris gdk@cs.brown.edu Fall 2017 Machine Learning Subfield of AI concerned with learning from data. Broadly, using: Experience To Improve Performance On Some Task (Tom Mitchell,
Neural Networks. Intro to AI Bert Huang Virginia Tech
Neural Networks Intro to AI Bert Huang Virginia Tech Outline Biological inspiration for artificial neural networks Linear vs. nonlinear functions Learning with neural networks: back propagation https://en.wikipedia.org/wiki/neuron#/media/file:chemical_synapse_schema_cropped.jpg
CSCI 315: Artificial Intelligence through Deep Learning
CSCI 35: Artificial Intelligence through Deep Learning W&L Fall Term 27 Prof. Levy Convolutional Networks http://wernerstudio.typepad.com/.a/6ad83549adb53ef53629ccf97c-5wi Convolution: Convolution is
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Neural networks Daniel Hennes 21.01.2018 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Logistic regression Neural networks Perceptron
text classification 3: neural networks
text classification 3: neural networks CS 585, Fall 2018 Introduction to Natural Language Processing http://people.cs.umass.edu/~miyyer/cs585/ Mohit Iyyer College of Information and Computer Sciences University
Deep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, Spis treści
Deep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, 2017 Spis treści Website Acknowledgments Notation xiii xv xix 1 Introduction 1 1.1 Who Should Read This Book?
CSC321 Lecture 16: ResNets and Attention
CSC321 Lecture 16: ResNets and Attention Roger Grosse Roger Grosse CSC321 Lecture 16: ResNets and Attention 1 / 24 Overview Two topics for today: Topic 1: Deep Residual Networks (ResNets) This is the state-of-the
CSE446: Neural Networks Spring Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer
CSE446: Neural Networks Spring 2017 Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer Human Neurons Switching time ~ 0.001 second Number of neurons 10 10 Connections per neuron 10 4-5 Scene
Deep Learning for NLP
Deep Learning for NLP Instructor: Wei Xu Ohio State University CSE 5525 Many slides from Greg Durrett Outline Motivation for neural networks Feedforward neural networks Applying feedforward neural networks
Deep Feedforward Networks
Deep Feedforward Networks Liu Yang March 30, 2017 Liu Yang Short title March 30, 2017 1 / 24 Overview 1 Background A general introduction Example 2 Gradient based learning Cost functions Output Units 3
Lecture 16: Introduction to Neural Networks
Lecture 16: Introduction to Neural Networs Instructor: Aditya Bhasara Scribe: Philippe David CS 5966/6966: Theory of Machine Learning March 20 th, 2017 Abstract In this lecture, we consider Bacpropagation,
Notes on Adversarial Examples
Notes on Adversarial Examples David Meyer dmm@{1-4-5.net,uoregon.edu,...} March 14, 2017 1 Introduction The surprising discovery of adversarial examples by Szegedy et al. [6] has led to new ways of thinking
Learning Deep Architectures for AI. Part II - Vijay Chakilam
Learning Deep Architectures for AI - Yoshua Bengio Part II - Vijay Chakilam Limitations of Perceptron x1 W, b 0,1 1,1 y x2 weight plane output =1 output =0 There is no value for W and b such that the model
EE-559 Deep learning Recurrent Neural Networks
EE-559 Deep learning 11.1. Recurrent Neural Networks François Fleuret https://fleuret.org/ee559/ Sun Feb 24 20:33:31 UTC 2019 Inference from sequences François Fleuret EE-559 Deep learning / 11.1. Recurrent
Deep Feedforward Networks. Han Shao, Hou Pong Chan, and Hongyi Zhang
Deep Feedforward Networks Han Shao, Hou Pong Chan, and Hongyi Zhang Deep Feedforward Networks Goal: approximate some function f e.g., a classifier, maps input to a class y = f (x) x y Defines a mapping
Logistic Regression & Neural Networks
Logistic Regression & Neural Networks CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Logistic Regression Perceptron & Probabilities What if we want a probability