Motivation. Lecture 23. The Stochastic Neuron ( ) Properties of Logistic Sigmoid. Logistic Sigmoid With Varying T. Logistic Sigmoid T = 0.
|
|
- Martha Higgins
- 6 years ago
- Views:
Transcription
1 Motivation Lecture 23 Idea: with low probability, go against the local field move up the energy surface make the wrong microdecision Potential value for optimization: escape from local optima Potential value for associative memory: escape from spurious states because they have higher energy than imprinted states /2/07 /2/07 2 The Stochastic Neuron Deterministic neuron : s " i = sgn h i Pr { s " i = +} = # h i Pr { s " i = $} =$ # h i Stochastic neuron : Pr { s " i = +} = #( h i ) Pr { s " i = $} =$# h i Logistic sigmoid : " h = + exp #2 h T σ(h) h Properties of As h +, σ(h) As h, σ(h) 0 σ(0) = /2 = " h + e #2h T /2/07 3 /2/07 4 With Varying T T = 0.5 T varying from 0.05 to (/T = β = 0,, 2,, 20) Slope at origin = / 2T /2/07 5 /2/07 6
2 T = 0.0 T = 0. /2/07 7 /2/07 8 T = T = 0 /2/07 9 /2/07 0 T = 00 Pseudo-Temperature Temperature = measure of thermal energy (heat) Thermal energy = vibrational energy of molecules A source of random motion Pseudo-temperature = a measure of nondirected (random) change Logistic sigmoid gives same equilibrium probabilities as Boltzmann-Gibbs distribution /2/07 /2/07 2 2
3 Transition Probability Recall, change in energy "E = #"s k h k = 2s k h k Pr { s " k = ±s k = m} = #( ±h k ) = #( $s k h k ) Pr{ s k " #s k } = + exp 2s k h k T = + exp $E T /2/07 3 Stability Are stochastic Hopfield nets stable? Thermal noise prevents absolute stability But with symmetric weights: average values s i become time - invariant /2/07 4 Does Thermal Noise Improve Memory Performance? Experiments by Bar-Yam (pp ): n = 00 p = 8 Random initial state To allow convergence, after 20 cycles set T = 0 How often does it converge to an imprinted pattern? /2/07 5 Probability of Random State Converging on Imprinted State (n=00, p=8) T = / β /2/07 (fig. from Bar-Yam) 6 Probability of Random State Converging on Imprinted State (n=00, p=8) Analysis of Stochastic Hopfield Network Complete analysis by Daniel J. Amit & colleagues in mid-80s See D. J. Amit, Modeling Brain Function: The World of Attractor Neural Networks, Cambridge Univ. Press, 989. The analysis is beyond the scope of this course /2/07 (fig. from Bar-Yam) 7 /2/07 8 3
4 Phase Diagram (D) all states melt Conceptual Diagrams of Energy Landscape (C) spin-glass states (A) imprinted = minima (B) imprinted, but s.g. = min. /2/07 (fig. from Domany & al. 99) 9 /2/07 (fig. from Hertz & al. Intr. Theory Neur. Comp.) 20 Phase Diagram Detail Simulated Annealing (Kirkpatrick, Gelatt & Vecchi, 983) /2/07 (fig. from Domany & al. 99) 2 /2/07 22 Dilemma In the early stages of search, we want a high temperature, so that we will explore the space and find the basins of the global minimum In the later stages we want a low temperature, so that we will relax into the global minimum and not wander away from it Solution: decrease the temperature gradually during search /2/07 23 Quenching vs. Annealing Quenching: rapid cooling of a hot material may result in defects & brittleness local order but global disorder locally low-energy, globally frustrated Annealing: slow cooling (or alternate heating & cooling) reaches equilibrium at each temperature allows global order to emerge achieves global low-energy state /2/
5 Multiple Domains Moving Domain Boundaries global incoherence local coherence /2/07 25 /2/07 26 Effect of Moderate Temperature Effect of High Temperature /2/07 (fig. from Anderson Intr. Neur. Comp.) 27 /2/07 (fig. from Anderson Intr. Neur. Comp.) 28 Effect of Low Temperature Annealing Schedule Controlled decrease of temperature Should be sufficiently slow to allow equilibrium to be reached at each temperature With sufficiently slow annealing, the global minimum will be found with probability Design of schedules is a topic of research /2/07 (fig. from Anderson Intr. Neur. Comp.) 29 /2/
6 Typical Practical Annealing Schedule Initial temperature T 0 sufficiently high so all transitions allowed Exponential cooling: T k+ = αt k typical 0.8 < α < 0.99 at least 0 accepted transitions at each temp. Final temperature: three successive temperatures without required number of accepted transitions Summary Non-directed change (random motion) permits escape from local optima and spurious states Pseudo-temperature can be controlled to adjust relative degree of exploration and exploitation /2/07 3 /2/07 32 Additional Bibliography. Anderson, J.A. An Introduction to Neural Networks, MIT, Arbib, M. (ed.) Handbook of Brain Theory & Neural Networks, MIT, Hertz, J., Krogh, A., & Palmer, R. G. Introduction to the Theory of Neural Computation, Addison-Wesley, 99. V. Genetics & Evolution /2/07 33 /2/07 34 The Second Law of Thermodynamics The Second Law and Open Systems closed system entropy H H open system energy concentration H waste /2/07 35 /2/
7 Nonequilibrium Thermodynamics Classical thermodynamics limited to systems in equilibrium Extended by thermodynamics of transport processes i.e. accounting for entropy changes when matter/energy transported into or out of an open system Flow of matter/energy can maintain a dissipative system far from equilibrium for long periods Hence, nonequilibrium thermodynamics An Energy Flow Can Create Structure /2/07 37 /2/07 (photo from Camazine & al. Self-Org. Bio. Sys.) 38 Bénard Convection Cells Persistent Nonequilibrium Systems If flow creates system so structured to maintain flow then positive feedback causes nonequilibrium system to persist indefinitely but not forever (2 nd law) Systems we tend to see are those most successful at maintaining nonequil. state Applies to species as well as organisms /2/07 (photo from Camazine & al. Self-Org. Bio. Sys.) 39 /2/07 40 Order Through Fluctuations Stabilization at Successively Higher Levels of Organization Fluctuations (esp. when system forced out of ordinary operating range) test boundaries & nonlinear effects May lead to stabilization of new structures /2/07 4 fig. < Hart & Gregor, Inf. Sys. Found. /2/07 42 fig. < Hart & Gregor, Inf. Sys. Found. 7
8 Order for Free Relatively simple sets of rules or equations can generate rich structures & behaviors Small changes can lead to qualitatively different structures & behaviors A diverse resource for selection A basis for later fine tuning (microevolution) See Kaufmann (At Home in the Universe, etc.) and Wolfram (A New Kind of Science) Evolution in Broad Sense Evolution in the broadest terms: blind variation selective retention Has been applied to nonbiological evolution evolutionary epistemology creativity memes /2/07 43 /2/07 44 Evolution atoms & molecules replicating molecules living things prebiotic evolution biotic evolution /2/07 45 /2/07 (from NECSI) 46 8
( ) T. Reading. Lecture 22. Definition of Covariance. Imprinting Multiple Patterns. Characteristics of Hopfield Memory
Part 3: Autonomous Agents /8/07 Reading Lecture 22 Flake, ch. 20 ( Genetics and Evolution ) /8/07 /8/07 2 Imprinting Multiple Patterns Let x, x 2,, x p be patterns to be imprinted Define the sum-of-outer-products
More informationVI: Genetics & Evolution 11/12/03 1
VI: Genetics & Evolution 11/12/03 1 Genotype vs. Phenotype Genotype = the genetic makeup of an individual organism Phenotype = the observed characteristic of the organism Through interaction with environment,
More informationA. The Hopfield Network. III. Recurrent Neural Networks. Typical Artificial Neuron. Typical Artificial Neuron. Hopfield Network.
III. Recurrent Neural Networks A. The Hopfield Network 2/9/15 1 2/9/15 2 Typical Artificial Neuron Typical Artificial Neuron connection weights linear combination activation function inputs output net
More informationA. The Hopfield Network. III. Recurrent Neural Networks. Typical Artificial Neuron. Typical Artificial Neuron. Hopfield Network.
Part 3A: Hopfield Network III. Recurrent Neural Networks A. The Hopfield Network 1 2 Typical Artificial Neuron Typical Artificial Neuron connection weights linear combination activation function inputs
More informationHertz, Krogh, Palmer: Introduction to the Theory of Neural Computation. Addison-Wesley Publishing Company (1991). (v ji (1 x i ) + (1 v ji )x i )
Symmetric Networks Hertz, Krogh, Palmer: Introduction to the Theory of Neural Computation. Addison-Wesley Publishing Company (1991). How can we model an associative memory? Let M = {v 1,..., v m } be a
More informationNeural Nets and Symbolic Reasoning Hopfield Networks
Neural Nets and Symbolic Reasoning Hopfield Networks Outline The idea of pattern completion The fast dynamics of Hopfield networks Learning with Hopfield networks Emerging properties of Hopfield networks
More information7 Rate-Based Recurrent Networks of Threshold Neurons: Basis for Associative Memory
Physics 178/278 - David Kleinfeld - Fall 2005; Revised for Winter 2017 7 Rate-Based Recurrent etworks of Threshold eurons: Basis for Associative Memory 7.1 A recurrent network with threshold elements The
More informationDynamical Systems and Deep Learning: Overview. Abbas Edalat
Dynamical Systems and Deep Learning: Overview Abbas Edalat Dynamical Systems The notion of a dynamical system includes the following: A phase or state space, which may be continuous, e.g. the real line,
More informationMicrocanonical Mean Field Annealing: A New Algorithm for Increasing the Convergence Speed of Mean Field Annealing.
Microcanonical Mean Field Annealing: A New Algorithm for Increasing the Convergence Speed of Mean Field Annealing. Hyuk Jae Lee and Ahmed Louri Department of Electrical and Computer Engineering University
More information7 Recurrent Networks of Threshold (Binary) Neurons: Basis for Associative Memory
Physics 178/278 - David Kleinfeld - Winter 2019 7 Recurrent etworks of Threshold (Binary) eurons: Basis for Associative Memory 7.1 The network The basic challenge in associative networks, also referred
More informationCOMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines COMP9444 17s2 Boltzmann Machines 1 Outline Content Addressable Memory Hopfield Network Generative Models Boltzmann Machine Restricted Boltzmann
More informationHopfield Network for Associative Memory
CSE 5526: Introduction to Neural Networks Hopfield Network for Associative Memory 1 The next few units cover unsupervised models Goal: learn the distribution of a set of observations Some observations
More information( ) ( ) ( ) ( ) Simulated Annealing. Introduction. Pseudotemperature, Free Energy and Entropy. A Short Detour into Statistical Mechanics.
Aims Reference Keywords Plan Simulated Annealing to obtain a mathematical framework for stochastic machines to study simulated annealing Parts of chapter of Haykin, S., Neural Networks: A Comprehensive
More informationSIMU L TED ATED ANNEA L NG ING
SIMULATED ANNEALING Fundamental Concept Motivation by an analogy to the statistical mechanics of annealing in solids. => to coerce a solid (i.e., in a poor, unordered state) into a low energy thermodynamic
More informationMarkov Chains and MCMC
Markov Chains and MCMC Markov chains Let S = {1, 2,..., N} be a finite set consisting of N states. A Markov chain Y 0, Y 1, Y 2,... is a sequence of random variables, with Y t S for all points in time
More informationNeural Networks for Machine Learning. Lecture 11a Hopfield Nets
Neural Networks for Machine Learning Lecture 11a Hopfield Nets Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed Hopfield Nets A Hopfield net is composed of binary threshold
More informationOptimization Methods via Simulation
Optimization Methods via Simulation Optimization problems are very important in science, engineering, industry,. Examples: Traveling salesman problem Circuit-board design Car-Parrinello ab initio MD Protein
More informationStochastic Networks Variations of the Hopfield model
4 Stochastic Networks 4. Variations of the Hopfield model In the previous chapter we showed that Hopfield networks can be used to provide solutions to combinatorial problems that can be expressed as the
More informationEnergy-Decreasing Dynamics in Mean-Field Spin Models
arxiv:cond-mat/0210545 v1 24 Oct 2002 Energy-Decreasing Dynamics in Mean-Field Spin Models L. Bussolari, P. Contucci, M. Degli Esposti, C. Giardinà Dipartimento di Matematica dell Università di Bologna,
More information5. Simulated Annealing 5.1 Basic Concepts. Fall 2010 Instructor: Dr. Masoud Yaghini
5. Simulated Annealing 5.1 Basic Concepts Fall 2010 Instructor: Dr. Masoud Yaghini Outline Introduction Real Annealing and Simulated Annealing Metropolis Algorithm Template of SA A Simple Example References
More informationHopfield Networks. (Excerpt from a Basic Course at IK 2008) Herbert Jaeger. Jacobs University Bremen
Hopfield Networks (Excerpt from a Basic Course at IK 2008) Herbert Jaeger Jacobs University Bremen Building a model of associative memory should be simple enough... Our brain is a neural network Individual
More informationBranislav K. Nikolić
Interdisciplinary Topics in Complex Systems: Cellular Automata, Self-Organized Criticality, Neural Networks and Spin Glasses Branislav K. Nikolić Department of Physics and Astronomy, University of Delaware,
More informationMarkov Chain Monte Carlo. Simulated Annealing.
Aula 10. Simulated Annealing. 0 Markov Chain Monte Carlo. Simulated Annealing. Anatoli Iambartsev IME-USP Aula 10. Simulated Annealing. 1 [RC] Stochastic search. General iterative formula for optimizing
More informationLearning and Memory in Neural Networks
Learning and Memory in Neural Networks Guy Billings, Neuroinformatics Doctoral Training Centre, The School of Informatics, The University of Edinburgh, UK. Neural networks consist of computational units
More informationA.I.: Beyond Classical Search
A.I.: Beyond Classical Search Random Sampling Trivial Algorithms Generate a state randomly Random Walk Randomly pick a neighbor of the current state Both algorithms asymptotically complete. Overview Previously
More informationimmune response, spin glasses and, 174, 228 Ising model, definition of, 79-80
Index Amit, D., 135-136 Anderson localization, 49-50, 245n9 Anderson, Philip, 5, 49, 79, 127-129, 155, 169-170, 223, 225-228, 248n2, 251n2 Anfinsen, C.B., 163 annealing, 119, 245n7 simulated, see simulated
More informationStochastic Model for Adaptation Using Basin Hopping Dynamics
Stochastic Model for Adaptation Using Basin Hopping Dynamics Peter Davis NTT Communication Science Laboratories 2-4 Hikaridai, Keihanna Science City, Kyoto, Japan 619-0237 davis@cslab.kecl.ntt.co.jp Abstract
More informationLecture 4: Simulated Annealing. An Introduction to Meta-Heuristics, Produced by Qiangfu Zhao (Since 2012), All rights reserved
Lecture 4: Simulated Annealing Lec04/1 Neighborhood based search Step 1: s=s0; Step 2: Stop if terminating condition satisfied; Step 3: Generate N(s); Step 4: s =FindBetterSolution(N(s)); Step 5: s=s ;
More information3.012 PS 6 THERMODYANMICS SOLUTIONS Issued: Fall 2005 Due:
3.012 PS 6 THERMODYANMICS SOLUTIONS 3.012 Issued: 11.28.05 Fall 2005 Due: THERMODYNAMICS 1. Building a binary phase diagram. Given below are data for a binary system of two materials A and B. The two components
More informationNonequilibrium dynamics in Coulomb glasses near the metal-insulator transition
Nonequilibrium dynamics in Coulomb glasses near the metal-insulator transition Dragana Popović National High Magnetic Field Laboratory Florida State University, Tallahassee, FL, USA Boulder 2009 Summer
More information5. Simulated Annealing 5.2 Advanced Concepts. Fall 2010 Instructor: Dr. Masoud Yaghini
5. Simulated Annealing 5.2 Advanced Concepts Fall 2010 Instructor: Dr. Masoud Yaghini Outline Acceptance Function Initial Temperature Equilibrium State Cooling Schedule Stopping Condition Handling Constraints
More informationPhysics of disordered materials. Gunnar A. Niklasson Solid State Physics Department of Engineering Sciences Uppsala University
Physics of disordered materials Gunnar A. Niklasson Solid State Physics Department of Engineering Sciences Uppsala University Course plan Familiarity with the basic description of disordered structures
More informationMotivation, Basic Concepts, Basic Methods, Travelling Salesperson Problem (TSP), Algorithms
Motivation, Basic Concepts, Basic Methods, Travelling Salesperson Problem (TSP), Algorithms 1 What is Combinatorial Optimization? Combinatorial Optimization deals with problems where we have to search
More informationGood vibrations: the issue of optimizing dynamical reservoirs
Good vibrations: the issue of optimizing dynamical reservoirs Workshop on ESNs / LSMs, NIPS 2006 Herbert Jaeger International University Bremen (Jacobs University Bremen, as of Spring 2007) The basic idea:
More information7.1 Basis for Boltzmann machine. 7. Boltzmann machines
7. Boltzmann machines this section we will become acquainted with classical Boltzmann machines which can be seen obsolete being rarely applied in neurocomputing. It is interesting, after all, because is
More informationCSC321 Lecture 8: Optimization
CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationAI Programming CS F-20 Neural Networks
AI Programming CS662-2008F-20 Neural Networks David Galles Department of Computer Science University of San Francisco 20-0: Symbolic AI Most of this class has been focused on Symbolic AI Focus or symbols
More informationSpin Glas Dynamics and Stochastic Optimization Schemes. Karl Heinz Hoffmann TU Chemnitz
Spin Glas Dynamics and Stochastic Optimization Schemes Karl Heinz Hoffmann TU Chemnitz 1 Spin Glasses spin glass e.g. AuFe AuMn CuMn nobel metal (no spin) transition metal (spin) 0.1-10 at% ferromagnetic
More informationEntropy. Governing Rules Series. Instructor s Guide. Table of Contents
Entropy Governing Rules Series Instructor s Guide Table of duction.... 2 When to Use this Video.... 2 Learning Objectives.... 2 Motivation.... 2 Student Experience... 2 Key Information.... 2 Video Highlights....
More informationENTROPY
ENTROPY 6.2.8 6.2.11 ENTHALPY VS. ENTROPY ENTROPY (S) the disorder of a system - solid liquid gas = entropy - gas liquid solid = entropy - mixing substances always = entropy SPONTANEOUS VS. NONSPONTANEOUS
More informationLast update: October 26, Neural networks. CMSC 421: Section Dana Nau
Last update: October 26, 207 Neural networks CMSC 42: Section 8.7 Dana Nau Outline Applications of neural networks Brains Neural network units Perceptrons Multilayer perceptrons 2 Example Applications
More informationAssociative Memories (I) Hopfield Networks
Associative Memories (I) Davide Bacciu Dipartimento di Informatica Università di Pisa bacciu@di.unipi.it Applied Brain Science - Computational Neuroscience (CNS) A Pun Associative Memories Introduction
More informationOnsager theory: overview
Onsager theory: overview Pearu Peterson December 18, 2006 1 Introduction Our aim is to study matter that consists of large number of molecules. A complete mechanical description of such a system is practically
More informationEffects of Interactive Function Forms in a Self-Organized Critical Model Based on Neural Networks
Commun. Theor. Phys. (Beijing, China) 40 (2003) pp. 607 613 c International Academic Publishers Vol. 40, No. 5, November 15, 2003 Effects of Interactive Function Forms in a Self-Organized Critical Model
More informationEmergence of complex structure through co-evolution:
Emergence of complex structure through co-evolution: The Tangled Nature model of evolutionary ecology Institute for Mathematical Sciences & Department of Mathematics Collaborators: Paul Anderson, Kim Christensen,
More informationarxiv:cond-mat/ v1 [cond-mat.dis-nn] 18 Nov 1996
arxiv:cond-mat/9611130v1 [cond-mat.dis-nn] 18 Nov 1996 Learning by dilution in a Neural Network B López and W Kinzel Institut für Theoretische Physik, Universität Würzburg, Am Hubland, D-97074 Würzburg
More informationIntroduction to Fluctuation Theorems
Hyunggyu Park Introduction to Fluctuation Theorems 1. Nonequilibrium processes 2. Brief History of Fluctuation theorems 3. Jarzynski equality & Crooks FT 4. Experiments 5. Probability theory viewpoint
More informationRelaxation in Glass: Review of Thermodynamics. Lecture 11: Thermodynamics in the Glass Transition Region
Relaxation in Glass: Review of hermodynamics Lecture 11: hermodynamics in the Glass ransition Region hermodynamic Functions 1 st Derivatives emperature Dependence of the Entropy swmartin@iastate.edu MI:
More informationSystems Biology: A Personal View IX. Landscapes. Sitabhra Sinha IMSc Chennai
Systems Biology: A Personal View IX. Landscapes Sitabhra Sinha IMSc Chennai Fitness Landscapes Sewall Wright pioneered the description of how genotype or phenotypic fitness are related in terms of a fitness
More informationT sg. α c (0)= T=1/β. α c (T ) α=p/n
Taejon, Korea, vol. 2 of 2, pp. 779{784, Nov. 2. Capacity Analysis of Bidirectional Associative Memory Toshiyuki Tanaka y, Shinsuke Kakiya y, and Yoshiyuki Kabashima z ygraduate School of Engineering,
More informationLogic Learning in Hopfield Networks
Logic Learning in Hopfield Networks Saratha Sathasivam (Corresponding author) School of Mathematical Sciences, University of Science Malaysia, Penang, Malaysia E-mail: saratha@cs.usm.my Wan Ahmad Tajuddin
More informationChemical thermodynamics the area of chemistry that deals with energy relationships
Chemistry: The Central Science Chapter 19: Chemical Thermodynamics Chemical thermodynamics the area of chemistry that deals with energy relationships 19.1: Spontaneous Processes First law of thermodynamics
More informationLecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feed-forward neural networks Backpropagation Tricks for training neural networks COMP-652, Lecture
More informationEnergy Landscapes of Deep Networks
Energy Landscapes of Deep Networks Pratik Chaudhari February 29, 2016 References Auffinger, A., Arous, G. B., & Černý, J. (2013). Random matrices and complexity of spin glasses. arxiv:1003.1129.# Choromanska,
More informationEnergy Barriers and Rates - Transition State Theory for Physicists
Energy Barriers and Rates - Transition State Theory for Physicists Daniel C. Elton October 12, 2013 Useful relations 1 cal = 4.184 J 1 kcal mole 1 = 0.0434 ev per particle 1 kj mole 1 = 0.0104 ev per particle
More informationBoltzmann Machine and Hyperbolic Activation Function in Higher Order Network
Modern Applied Science; Vol. 8, No. 3; 2014 ISSN 1913-1844 E-ISSN 1913-1852 Published by Canadian Center of Science and Education Boltzmann Machine and Hyperbolic Activation Function in Higher Order Network
More informationCollective Effects. Equilibrium and Nonequilibrium Physics
Collective Effects in Equilibrium and Nonequilibrium Physics: Lecture 3, 3 March 2006 Collective Effects in Equilibrium and Nonequilibrium Physics Website: http://cncs.bnu.edu.cn/mccross/course/ Caltech
More informationSYSTEMS SCIENCE AND CYBERNETICS Vol. III Simulated Annealing: From Statistical Thermodynamics to Combinatory Problems Solving - D.
SIMULATED ANNEALING: FROM STATISTICAL THERMODYNAMICS TO COMBINATORY PROBLEMS SOLVING D. Thiel ENITIAA, Nantes, France Keywords: Combinatory Problem, Optimizing Problem, Global Search Method, Statistical
More informationSlightly off-equilibrium dynamics
Slightly off-equilibrium dynamics Giorgio Parisi Many progresses have recently done in understanding system who are slightly off-equilibrium because their approach to equilibrium is quite slow. In this
More informationSelf-Organization in Nonequilibrium Systems
Self-Organization in Nonequilibrium Systems From Dissipative Structures to Order through Fluctuations G. Nicolis Universite Libre de Bruxelles Belgium I. Prigogine Universite Libre de Bruxelles Belgium
More informationCSC321 Lecture 7: Optimization
CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationPart II. Combinatorial Models. 5 A Sketch of Basic Statistical Physics. 5.1 Thermodynamics
Part II Combinatorial Models 5 A Sketch of Basic Statistical Physics Statistical physics = Thermodynamics (macroscopic) + Statistical mechanics (microscopic) 5.1 Thermodynamics A thermodynamic system is
More informationThrough Kaleidoscope Eyes Spin Glasses Experimental Results and Theoretical Concepts
Through Kaleidoscope Eyes Spin Glasses Experimental Results and Theoretical Concepts Benjamin Hsu December 11, 2007 Abstract A spin glass describes a system of spins on a lattice (or a crystal) where the
More informationHOPFIELD neural networks (HNNs) are a class of nonlinear
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 52, NO. 4, APRIL 2005 213 Stochastic Noise Process Enhancement of Hopfield Neural Networks Vladimir Pavlović, Member, IEEE, Dan Schonfeld,
More informationMineral Stability and Phase Diagrams Introduction
1 of 10 10/10/2002 2:50 PM Prof. Stephen A. Nelson Geology 211 Tulane University Mineralogy and Phase Diagrams Introduction This document last updated on 10-Oct-2002 As we discussed previously, there are
More informationStatistical Thermodynamics
Statistical Thermodynamics Basic Theory and Equations: A White Paper Alianna J. Maren, Ph.D. Themasis December, 2013 THM TR2013-001(ajm) 1 Goal of This Paper Identify and present basic equations from statistical
More informationHill climbing: Simulated annealing and Tabu search
Hill climbing: Simulated annealing and Tabu search Heuristic algorithms Giovanni Righini University of Milan Department of Computer Science (Crema) Hill climbing Instead of repeating local search, it is
More informationNeural Networks. Textbook. Other Textbooks and Books. Course Info. (click on
636-600 Neural Networks Textbook Instructor: Yoonsuck Choe Contact info: HRBB 322B, 45-5466, choe@tamu.edu Web page: http://faculty.cs.tamu.edu/choe Simon Haykin, Neural Networks: A Comprehensive Foundation,
More informationSingle Solution-based Metaheuristics
Parallel Cooperative Optimization Research Group Single Solution-based Metaheuristics E-G. Talbi Laboratoire d Informatique Fondamentale de Lille Single solution-based metaheuristics Improvement of a solution.
More informationCasting Polymer Nets To Optimize Molecular Codes
Casting Polymer Nets To Optimize Molecular Codes The physical language of molecules Mathematical Biology Forum (PRL 2007, PNAS 2008, Phys Bio 2008, J Theo Bio 2007, E J Lin Alg 2007) Biological information
More informationSimulated Annealing. 2.1 Introduction
Simulated Annealing 2 This chapter is dedicated to simulated annealing (SA) metaheuristic for optimization. SA is a probabilistic single-solution-based search method inspired by the annealing process in
More informationIs there a de Almeida-Thouless line in finite-dimensional spin glasses? (and why you should care)
Is there a de Almeida-Thouless line in finite-dimensional spin glasses? (and why you should care) Peter Young Talk at MPIPKS, September 12, 2013 Available on the web at http://physics.ucsc.edu/~peter/talks/mpipks.pdf
More information12. LOCAL SEARCH. gradient descent Metropolis algorithm Hopfield neural networks maximum cut Nash equilibria
12. LOCAL SEARCH gradient descent Metropolis algorithm Hopfield neural networks maximum cut Nash equilibria Lecture slides by Kevin Wayne Copyright 2005 Pearson-Addison Wesley h ttp://www.cs.princeton.edu/~wayne/kleinberg-tardos
More informationPROBLEM SOLVING AND SEARCH IN ARTIFICIAL INTELLIGENCE
Artificial Intelligence, Computational Logic PROBLEM SOLVING AND SEARCH IN ARTIFICIAL INTELLIGENCE Lecture 4 Metaheuristic Algorithms Sarah Gaggl Dresden, 5th May 2017 Agenda 1 Introduction 2 Constraint
More informationAnalysis of Neural Networks with Chaotic Dynamics
Chaos, Solitonr & Fructals Vol. 3, No. 2, pp. 133-139, 1993 Printed in Great Britain @60-0779/93$6.00 + 40 0 1993 Pergamon Press Ltd Analysis of Neural Networks with Chaotic Dynamics FRANCOIS CHAPEAU-BLONDEAU
More informationInfluence of Criticality on 1/f α Spectral Characteristics of Cortical Neuron Populations
Influence of Criticality on 1/f α Spectral Characteristics of Cortical Neuron Populations Robert Kozma rkozma@memphis.edu Computational Neurodynamics Laboratory, Department of Computer Science 373 Dunn
More informationSecond Law of Thermodynamics
Second Law of Thermodynamics First Law: the total energy of the universe is a constant Second Law: The entropy of the universe increases in a spontaneous process, and remains unchanged in a process at
More informationDeriving Thermodynamics from Linear Dissipativity Theory
Deriving Thermodynamics from Linear Dissipativity Theory Jean-Charles Delvenne Université catholique de Louvain Belgium Henrik Sandberg KTH Sweden IEEE CDC 2015, Osaka, Japan «Every mathematician knows
More informationCase study: molecular dynamics of solvent diffusion in polymers
Course MP3 Lecture 11 29/11/2006 Case study: molecular dynamics of solvent diffusion in polymers A real-life research example to illustrate the use of molecular dynamics Dr James Elliott 11.1 Research
More informationStorage Capacity of Letter Recognition in Hopfield Networks
Storage Capacity of Letter Recognition in Hopfield Networks Gang Wei (gwei@cs.dal.ca) Zheyuan Yu (zyu@cs.dal.ca) Faculty of Computer Science, Dalhousie University, Halifax, N.S., Canada B3H 1W5 Abstract:
More informationAdvanced computational methods X Selected Topics: SGD
Advanced computational methods X071521-Selected Topics: SGD. In this lecture, we look at the stochastic gradient descent (SGD) method 1 An illustrating example The MNIST is a simple dataset of variety
More information6 Markov Chain Monte Carlo (MCMC)
6 Markov Chain Monte Carlo (MCMC) The underlying idea in MCMC is to replace the iid samples of basic MC methods, with dependent samples from an ergodic Markov chain, whose limiting (stationary) distribution
More informationChapter 11. Stochastic Methods Rooted in Statistical Mechanics
Chapter 11. Stochastic Methods Rooted in Statistical Mechanics Neural Networks and Learning Machines (Haykin) Lecture Notes on Self-learning Neural Algorithms Byoung-Tak Zhang School of Computer Science
More informationChapter 19. Chemical Thermodynamics
Chapter 19. Chemical Thermodynamics 19.1 Spontaneous Processes Chemical thermodynamics is concerned with energy relationships in chemical reactions. We consider enthalpy and we also consider entropy in
More informationMarkov Chain Monte Carlo Lecture 4
The local-trap problem refers to that in simulations of a complex system whose energy landscape is rugged, the sampler gets trapped in a local energy minimum indefinitely, rendering the simulation ineffective.
More informationIN THIS PAPER, we consider a class of continuous-time recurrent
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 51, NO. 4, APRIL 2004 161 Global Output Convergence of a Class of Continuous-Time Recurrent Neural Networks With Time-Varying Thresholds
More informationChapter 11 Spontaneous Change and Equilibrium
Chapter 11 Spontaneous Change and Equilibrium 11-1 Enthalpy and Spontaneous Change 11-2 Entropy 11-3 Absolute Entropies and Chemical Reactions 11-4 The Second Law of Thermodynamics 11-5 The Gibbs Function
More informationOPTIMIZATION BY SIMULATED ANNEALING: A NECESSARY AND SUFFICIENT CONDITION FOR CONVERGENCE. Bruce Hajek* University of Illinois at Champaign-Urbana
OPTIMIZATION BY SIMULATED ANNEALING: A NECESSARY AND SUFFICIENT CONDITION FOR CONVERGENCE Bruce Hajek* University of Illinois at Champaign-Urbana A Monte Carlo optimization technique called "simulated
More informationSAMPLE ANSWERS TO HW SET 3B
SAMPLE ANSWERS TO HW SET 3B First- Please accept my most sincere apologies for taking so long to get these homework sets back to you. I have no excuses that are acceptable. Like last time, I have copied
More informationZebo Peng Embedded Systems Laboratory IDA, Linköping University
TDTS 01 Lecture 8 Optimization Heuristics for Synthesis Zebo Peng Embedded Systems Laboratory IDA, Linköping University Lecture 8 Optimization problems Heuristic techniques Simulated annealing Genetic
More informationTerminal attractor optical associative memory with adaptive control parameter
1 June 1998 Ž. Optics Communications 151 1998 353 365 Full length article Terminal attractor optical associative memory with adaptive control parameter Xin Lin a,), Junji Ohtsubo b, Masahiko Mori a a Electrotechnical
More informationLandscape Approach to Glass Transition and Relaxation. Lecture # 4, April 1 (last of four lectures)
Landscape Approach to Glass Transition and Relaxation Lecture # 4, April 1 (last of four lectures) Relaxation in the glassy state Instructor: Prabhat Gupta The Ohio State University (gupta.3@osu.edu) Review
More informationNeural networks. Chapter 19, Sections 1 5 1
Neural networks Chapter 19, Sections 1 5 Chapter 19, Sections 1 5 1 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 19, Sections 1 5 2 Brains 10
More informationVertex Routing Models and Polyhomeostatic Optimization. Claudius Gros. Institute for Theoretical Physics Goethe University Frankfurt, Germany
Vertex Routing Models and Polyhomeostatic Optimization Claudius Gros Institute for Theoretical Physics Goethe University Frankfurt, Germany 1 topics in complex system theory Vertex Routing Models modelling
More information4. References. words, the pseudo Laplacian is given by max {! 2 x,! 2 y}.
words, the pseudo Laplacian is given by ma {! 2,! 2 y}. 4. References [Ar72] Arbib, M.A., The Metaphorical Brain, John Wiley, New York, 1972. [BB82] Ballard, D. H. and Brown, C. M., Computer Vision, Prentice-Hall,
More informationFundamentals of Metaheuristics
Fundamentals of Metaheuristics Part I - Basic concepts and Single-State Methods A seminar for Neural Networks Simone Scardapane Academic year 2012-2013 ABOUT THIS SEMINAR The seminar is divided in three
More informationWeb Course Physical Properties of Glass. Range Behavior
Web Course Physical Properties of Glass Glass Transformation- Range Behavior Richard K. Brow Missouri University of Science & Technology Department of Materials Science & Engineering Glass Transformation-1
More informationLecture 2: Learning from Evaluative Feedback. or Bandit Problems
Lecture 2: Learning from Evaluative Feedback or Bandit Problems 1 Edward L. Thorndike (1874-1949) Puzzle Box 2 Learning by Trial-and-Error Law of Effect: Of several responses to the same situation, those
More informationArtificial Neural Network Simulation of Battery Performance
Artificial work Simulation of Battery Performance C.C. O Gorman, D. Ingersoll, R.G. Jungst and T.L. Paez Sandia National Laboratories PO Box 58 Albuquerque, NM 8785 Abstract Although they appear deceptively
More informationChapter 19 Chemical Thermodynamics
Chapter 19 Chemical Thermodynamics Kinetics How fast a rxn. proceeds Equilibrium How far a rxn proceeds towards completion Thermodynamics Study of energy relationships & changes which occur during chemical
More information