A Study of Complex Deep Learning Networks on High Performance, Neuromorphic, and Quantum Computers

Similar documents
LOCKHEED MARTIN SITE UPDATE 11 APRIL 2018 MUNICH, GERMANY Kristen Pudenz Senior Quantum Applications Engineer

UNSUPERVISED LEARNING

Implementation of a Restricted Boltzmann Machine in a Spiking Neural Network

Neuromorphic computing with Memristive devices. NCM group

How to do backpropagation in a brain

Quantum Computing & Scientific Applications at ORNL

Jakub Hajic Artificial Intelligence Seminar I

Deep Learning Srihari. Deep Belief Nets. Sargur N. Srihari

(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann

Introduction to (Convolutional) Neural Networks

Machine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6

Knowledge Extraction from DBNs for Images

Building a Multi-FPGA Virtualized Restricted Boltzmann Machine Architecture Using Embedded MPI

Deep Learning Architecture for Univariate Time Series Forecasting

Convolutional Neural Networks. Srikumar Ramalingam

Addressing Challenges in Neuromorphic Computing with Memristive Synapses

What Do Neural Networks Do? MLP Lecture 3 Multi-layer networks 1

LOCKHEED MARTIN SITE UPDATE

How can ideas from quantum computing improve or speed up neuromorphic models of computation?

Convolutional Neural Network Architecture

Restricted Boltzmann Machines

Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box

Generalizing Convolutional Neural Networks to Graphstructured. Ben Walker Department of Mathematics, UNC-Chapel Hill 5/4/2018

De Novo molecular design with Deep Reinforcement Learning

<Special Topics in VLSI> Learning for Deep Neural Networks (Back-propagation)

Modularity Matters: Learning Invariant Relational Reasoning Tasks

Opportunities and challenges in near-term quantum computers

Artificial Neural Networks. Introduction to Computational Neuroscience Tambet Matiisen

Spatial Transformer Networks

TYPES OF MODEL COMPRESSION. Soham Saha, MS by Research in CSE, CVIT, IIIT Hyderabad

Fast classification using sparsely active spiking networks. Hesham Mostafa Institute of neural computation, UCSD

STDP Learning of Image Patches with Convolutional Spiking Neural Networks

Speaker Representation and Verification Part II. by Vasileios Vasilakakis

LEADING THE EVOLUTION OF COMPUTE MARK KACHMAREK HPC STRATEGIC PLANNING MANAGER APRIL 17, 2018

Neural Network Representation of Tensor Network and Chiral States

Opportunities and challenges in quantum-enhanced machine learning in near-term quantum computers

Artificial Neural Networks D B M G. Data Base and Data Mining Group of Politecnico di Torino. Elena Baralis. Politecnico di Torino

Topic 3: Neural Networks

GPGPU ACCELERATED SIMULATION AND PARAMETER TUNING FOR NEUROMORPHIC APPLICATIONS

Comparison of Modern Stochastic Optimization Algorithms

Bayesian Networks (Part I)

Grundlagen der Künstlichen Intelligenz

Evolving Keras Architectures for Sensor Data Analysis

CSCI 315: Artificial Intelligence through Deep Learning

Deep Learning (CNNs)

A Video from Google DeepMind.

Spatial Transformation

The Origin of Deep Learning. Lili Mou Jan, 2015

IBM Systems for Cognitive Solutions

Classification with Perceptrons. Reading:

Regression Adjustment with Artificial Neural Networks

Convolutional Neural Networks

REAL-TIME COMPUTING WITHOUT STABLE

Machine Learning. Neural Networks. (slides from Domingos, Pardo, others)

Neural Networks, Computation Graphs. CMSC 470 Marine Carpuat

Probabilistic Models in Theoretical Neuroscience

RegML 2018 Class 8 Deep learning

Deep Learning for NLP

ECE521 Lectures 9 Fully Connected Neural Networks

A 68 Parallel Row Access Neuromorphic Core with 22K Multi-Level Synapses Based on Logic- Compatible Embedded Flash Memory Technology

Staying on the path Fred Rothganger

Artificial Neural Networks

Mitosis Detection in Breast Cancer Histology Images with Multi Column Deep Neural Networks

Introduction to Deep Learning

CSE446: Neural Networks Spring Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer

Multivariate Methods in Statistical Data Analysis

Learning to Disentangle Factors of Variation with Manifold Learning

A brain-inspired neuromorphic architecture for robust neural computation

Deep Neural Networks (1) Hidden layers; Back-propagation

Apprentissage, réseaux de neurones et modèles graphiques (RCP209) Neural Networks and Deep Learning

Learning Deep Architectures for AI. Part II - Vijay Chakilam

Neural networks. Chapter 19, Sections 1 5 1

Supervised Learning. George Konidaris

Neuroevolution for sound event detection in real life audio: A pilot study

NEUROMORPHIC COMPUTING WITH MAGNETO-METALLIC NEURONS & SYNAPSES: PROSPECTS AND PERSPECTIVES

STA 414/2104: Lecture 8

Deep Learning with Coherent Nanophotonic Circuits

SOLVING SPARSE REPRESENTATIONS FOR OBJECT CLASSIFICATION USING QUANTUM D-WAVE 2X MACHINE

CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning

Representational Power of Restricted Boltzmann Machines and Deep Belief Networks. Nicolas Le Roux and Yoshua Bengio Presented by Colin Graber

PREDICTION OF HETERODIMERIC PROTEIN COMPLEXES FROM PROTEIN-PROTEIN INTERACTION NETWORKS USING DEEP LEARNING

CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS

Neural Networks. Mark van Rossum. January 15, School of Informatics, University of Edinburgh 1 / 28

Magnetic tunnel junction beyond memory from logic to neuromorphic computing WANJUN PARK DEPT. OF ELECTRONIC ENGINEERING, HANYANG UNIVERSITY

Dimensionality Reduction and Principle Components Analysis

PV021: Neural networks. Tomáš Brázdil

COMPLEX INPUT CONVOLUTIONAL NEURAL NETWORKS FOR WIDE ANGLE SAR ATR

Neutron inverse kinetics via Gaussian Processes

CS:4420 Artificial Intelligence

Synaptic Devices and Neuron Circuits for Neuron-Inspired NanoElectronics

Feed-forward Network Functions

Deep learning quantum matter

Neural networks. Chapter 20, Section 5 1

Few-shot learning with KRR

Neural Networks. Chapter 18, Section 7. TB Artificial Intelligence. Slides from AIMA 1/ 21

Neural Networks. Bishop PRML Ch. 5. Alireza Ghane. Feed-forward Networks Network Training Error Backpropagation Applications

Neural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann

COSC 594 Final Presentation Membrane Systems/ P Systems

Fundamentals of Computational Neuroscience 2e

Neural Networks. David Rosenberg. July 26, New York University. David Rosenberg (New York University) DS-GA 1003 July 26, / 35

Transcription:

A Study of Complex Deep Learning Networks on High Performance, Neuromorphic, and Quantum Computers Thomas E. Potok, Ph.D. Computational Data Analytics Group Oak Ridge National Laboratory ORNL is managed by UT-Battelle 1 for Computational the US Department Data Analytics of Energy

Motivation and Goals Scientific data is increasingly large and complex, making new discovery difficult Can deep learning and novel architectures provide a way of aiding scientific discovery? Each pixel represents a 512x512 matrix of values from Spallation Neutron Source at ORNL 2 Computational Data Analytics

Deep Learning Performance What could you do with HPC, Quantum Neuromorphic? 3 Computational Data Analytics

Motivations for novel architectures Deep Learning network topologies are layered due to computability Does a complex topology offer better results? Deep Learning Hyper-parameters are hard to tune Can a CNN be quickly tuned for a new scientific datasets? Deep Learning Trained models are hard to deploy Can models be deployed on or near scientific instruments? 4 Computational Data Analytics

Scientific Data Scientific Instrument Hypothesis Complex Topology Hardware Implementation Auto Tuned Hyper Parameters 5 Computational Data Analytics

Scientific Data Scientific Instrument Methods Complex Topology Hardware Implementation Auto Tuned Hyper Parameters 6 Computational Data Analytics

Rational Quantum HPC Neuromorphic Hints at speed up of certain hard problems Computational parallelization Native neural networks, very low power 7 Computational Data Analytics

Experimental Goals Quantum Test feasibility of complex topologies (intra-layer connections) USC Lockheed Martin Quantum Computation Center D-Wave HPC Test evolutionary algorithms to auto-tune hyperparameters ORNL s Titan Neuromorphic Test the ability to represent neural network models in low power hardware UTK Memristive Nida System 8 Computational Data Analytics

How do we compare the architectures? Boltzmann Machine Convolutional Neural Network Spiking Neural Network Quantum HPC Neuromorphic The MNIST database contains 60,000 training images and 10,000 testing images Very small images size, 28x28 http://yann.lecun.com/exdb/mnist/ 9 Computational Data Analytics

Common Ground Quantum Physics Computer Science Electrical Engineering Neuroscience Entanglement Link Synapse Qubit Node Neuron Chimera Network 10 Computational Data Analytics

Limited Boltzmann Machine Network 10 Output Nodes (one for each digit) Fully-Connected (every node in output is connected to every node in hidden). One Layer Fully-Connected (every node in visible is connected to every node in hidden). 784 Visible Nodes (28x28 pixels for image) 11 Computational Data Analytics

Quantum Results USC/ISI D-Wave Complex topology learns and provides better results than no intra-layer connection Limited Boltzmann Machine learning from training examples Limited Boltzmann Machine more accurate than restricted BM 12 Computational Data Analytics

HPC Results 500 nodes on Titan Demonstrates an effective way of auto tuning a CNN 13 Computational Data Analytics

Neuromorphic ORNL/UT NIDA network on Memristor 20X more energy-efficient than their CMOS counterparts 14 Computational Data Analytics

Conclusion Complex topologies with intra-layer connects have better classification performance than without connections HPC can be used to auto-tune CNN topologies Neuromorphic hardware has the potential to implement deep learning network in very low-power hardware A first step towards richer DL on novel architectures 15 Computational Data Analytics

Next Step Evaluate the strengths and weaknesses of each approach Quantum: Explore more complex networks One Layer HPC: Auto tune on Limited Boltzmann machines model Neuromorphic: Implement of Limited Boltzmann on FPGA version of neuromorphic hardware 16 Computational Data Analytics

Team Deep Learning/HPC Robert Patton (ORNL) Steven Young (ORNL) Thomas Potok (PI/ORNL) Quantum Computing Federico Spedalieri (USC/ISI) Ke-Thia Yao(USC/ISI) Bob Lucas (USC/ISI) Jeremy Liu (USC) Neuromorphic Computing Garrett Rose (UT) Katie Schuman (ORNL) Gangotree Chakma (UT) Program Manager Robinson Pino (DOE ASCR) 17 Computational Data Analytics

Questions? 18 Computational Data Analytics

2024 © sciencedocbox.com Privacy Policy | Terms of Service | Feedback