Manhattan Rule Training for Memristive Crossbar Circuit Pattern Classifiers

Transcription

1 Manhaan Rule Training for Memrisive Crossbar Circui Paern Classifiers Elham Zamanidoos, Farnood M. Baya, Dmiri Srukov Elecrical and Compuer Engineering Deparmen Universiy Of California Sana Barbara Sana Barbara, CA, USA {elham, farnoodmb, Irina Kaaeva Advanced Research Division Denso Corporaion Komenoki-cho, Nisshin, Japan Absrac We invesigaed bach and sochasic Manhaan Rule algorihms for raining mulilayer percepron classifiers implemened wih memrisive crossbar circuis. In Manhaan Rule raining, he weighs are updaed only using sign informaion of classical backpropagaion algorihm. The main advanage of Manhaan Rule is is simpliciy, which leads o more compac hardware implemenaion and faser raining ime. Addiionally, in case of sochasic raining, Manhaan Rule allows performing all weigh updaes in parallel, which furher speeds up he raining procedure. The radeoff for simpliciy is slighly worse classificaion performance. For example, simulaion resuls showed ha classificaion fideliy on Proben1 benchmark for memrisor-based implemenaion rained wih bach Manhaan Rule were comparable o ha of classical backpropagaion algorihm, and abou 2.8 percen worse han he bes repored resuls. Keywords Crossbar memory, Memrisor, Arificial neural nework, Training algorihm, Paern classificaion. I. INTRODUCTION Arificial neural nework is a biologically inspired compuing paradigm suiable for variey of applicaions. To approach energy efficiency of biological neural neworks in informaion processing, a specialized hardware mus be developed [1] [5]. Crossbar-based hybrid circuis [6] [1], and in paricular hose of CrossNes variey [1], [11], are idenified as one of he mos promising soluions, because such circuis can provide high inegraion densiy of arificial synapses and high conneciviy beween arificial neurons, which are wo major challenges for efficien arificial neural nework hardware implemenaion. The disadvanages of crossbar circuis are cerain resricions on how he signals can be applied o he arificial synapses, which, in urn, impose limiaions on raining algorihms. The main conribuion of his paper is he developmen of a crossbar circui compaible raining approach for mulilayer percepron (MLP) neworks. The performance of he proposed raining approach was simulaed using a sandard benchmark suie for specific memrisive crossbar nework, which has recenly been uilized in successful experimenal demonsraion of a small-scale paern classifier [1, 5]. While he considered MLPs are small and hardly pracical, heir key feaures, e.g. nework opology and raining algorihms, are similar o hose of more pracical neworks such as deep learning convoluional neural neworks [12]. To he bes of our knowledge, he presened work is novel and Manhaan Rule raining has never been invesigaed before in he conex of crossbar circui hardware. The majoriy of previous work is devoed o unsupervised spiking neworks [4], [7], [8]. The mos relevan sudies are, perhaps, experimenal demonsraion of a single layer percepron [1, 5], and simulaions of small-scale [6], [9], and larger-scale paern classifiers [13]. In he nex secion, we provide brief background informaion on he considered neural nework, is crossbar circui implemenaion, and memrisor models used for simulaion sudies in his paper. II. BACKGROUND A. Muli-Layer Percepron In is simples form, a feedforward neural nework can be represened by a direced acyclic graph (Fig. 1a) in which neurons and synapses are nodes and edges of a graph, respecively. Each neuron applies a cerain ransfer funcion o he sum of is inpus and hen passes informaion forward o he nex layer of neurons. A synapse muliplies is weigh W wih he oupu of a pre-synapic neuron and passes he resuling produc o he inpu of he pos-synapic neuron. Mahemaically, he operaion wihin a single layer of he nework can be formulaed as f i = anh(βu i ), wih u i = Ʃ j W ij X j, (1) where u i and f i are he inpu and oupu of he i-h pos-synapic neuron, respecively, X j is he oupu of j-h pre-synapic neuron, and W ij is he synapic weigh beween j-h presynapic and i-h pos-synapic neurons. Each neuron is (a) synapses X W pre-synapic neurons f possynapic neurons (b) (c) W = Fig. 1. Feedforward arificial neural nework: (a) Absraced graph represenaion of one layer wih hree inpu and wo oupu neurons, (b) is crossbar circui mapping, and (c, d) memrisor-based crossbar implemenaion. u G G - (d) memrisor I I - This work is suppored by AFOSR under MURI gran FA and Denso Corporaion, Japan. Also we acknowledge suppor from he Cener for Scienific Compuing from he CNSI, MRL: an NSF MRSEC (DMR ) and NSF CNS

2 assumed o have a anh acivaion funcion wih slope β. For he firs layer of he nework, X 1 values correspond o he applied inpu paern. Feedforward neural neworks, and in paricular mulilayered percepron which are based on such neworks, allow performing paern classificaion ask, i.e. mapping of inpu paerns o cerain classes. The classificaion is considered successful if he specific oupu neuron, corresponding o he applied paern, produces he larges value. Such operaion is achieved by properly seing weighs in he nework, which in he mos general case, canno be calculaed analyically bu raher is found via some opimizaion procedure, e.g. using he backpropagaion raining algorihm in MLP neworks [14]. Backpropagaion raining can be implemened in bach or sochasic mode. For sochasic (someimes called online) raining, weighs are adjused immediaely afer applicaion of a single paern from a raining se. In he firs sep of his algorihm, a randomly chosen paern n from a raining se is applied o he nework and he oupu is calculaed according o (1). In he second sep, he synapic weighs are adjused according o ΔW ij (n) = - αδ i (n)x j (n), (2) where α is learning rae and δ i is he local (backpropagaed) error of he corresponding pos-synapic neuron. δ i is calculaed firs for oupu neurons, for which i is equal o he produc of he derivaive of neuron oupu wih respec o is inpu and he difference beween he acual oupu f and he desired value of he oupu f (g), i.e. δ i (n) =[ f i (g) (n) - f i (n)], (3) The error is hen propagaed backward (i.e. from he oupu o he inpu layer) using he recurrence relaion δ j pre (n) = Ʃ i δ i pos (n)w ij (n), (4) (Addiional superscrips are added o disinguish beween preand pos-synapic variables when describing operaion wihin he nework layer.) The applicaion of all paerns from a raining se consiues one epoch of raining wih muliple ypically required for successful raining. In he simples version of he bach backpropagaion algorihm, he synapic weighs are adjused by ΔW ij = Ʃ n ΔW ij (n), (5) only a he end of each epoch, i.e. afer all raining paerns are applied o he nework. Reaching perfec mapping during raining is no guaraneed. In addiion, classificaion performance is ypically measured on a separae se of es paerns, which are no used in he raining process. Therefore classifier performance is characerized by he misclassificaion rae (), i.e. he percenage of inpu paerns, which are classified incorrecly. The oher imporan meric is raining ime which characerizes how quickly he raining algorihm converges. B. MLP Implemenaion wih Crossbar Circuis The MLP srucure maps naurally o he crossbar array circui (Fig. 1b). In paricular, X and f are physically implemened wih volages read, while neuron s inpu u wih curren I. Synapses are implemened wih crosspoin devices whose conducance G is proporional o he synapic weigh W. Because weigh values can be negaive, while physical conducance is sricly posiive, one soluion is o represen each synapic weigh wih a pair of crosspoin devices (Fig. 1c), which are denoed as G and G -, i.e. W ij G ij -G ij -, (6) In such configuraion, neuron receives wo currens one from he crossbar line wih weighs G and anoher from he line wih weighs G -, so ha he negaive weighs are implemened due o he subracion of hese wo currens inside he neuron (Fig. 1d). Wih G max and G min being he maximum and he minimum conducances of he crosspoin devices, respecively, he effecive weigh ranges from -G max G min o G max - G min. Assuming virually-grounded inpus of he pos-synapic neuron, inpu curren I is equal o he produc G. The curren difference is hen convered o volage via an operaional amplifier wih feedback resisor R and hen applied o a sauraing operaional amplifier o approximae he hyperbolic angen acivaion funcion [1, 5, 11], i.e. implemening (1) on he physical level: i pos = read anh[r(i i - I i - )], I i = I i - I i - = Σ j (G ij - G ij - ) j pre, (7a) (7b) In general, a crossbar classifier can be rained ex-siu or insiu. In he firs mehod, he neural nework is firs implemened in sofware and he proper weighs are calculaed by simulaing he raining process. The calculaed weighs are hen impored ino hardware, which is somewha similar o he wrie operaion for convenional memory. The main difference, however, is ha impored values are analog (or muli-bi), which dramaically increases complexiy and ime of he wrie operaion. Alernaively, for in-siu approach, raining is implemened direcly in he hardware. In his case, weighs are physically adjused in hardware during raining as described by (2) or (5). Boh ex-siu and in-siu raining approaches have recenly been demonsraed experimenally for memrisive crossbar circuis [1, 5]. The advanage of he ex-siu mehod is ha any (i.e. sae-of-he-ar) raining algorihm ha resuls in he bes classificaion performance can be implemened in sofware wihou incurring much overhead in hardware. In-siu raining, however, auomaically adjuss o any hardware variaions, which are unavoidable in analog circuis. Noe ha obaining and supplying deailed informaion of circui s defecs and variaions o he sofware-implemened nework for ex-siu

3 raining is hardly pracical for large-scale sysems. Because of his issue, he paricular focus of his paper is on in-siu raining. C. Memrisive Devices In general, differen ypes of wo-erminal resisive swiching devices [15], can be inegraed ino crossbar array circuis o implemen a paern classifier. In his paper, a paricular ype of crosspoin devices - P/TiO 2-x /P memrisors - for which an accurae device model is available [16] has been invesigaed. A ypical swiching I- for such devices is shown in Fig. 2. The device conducance can be gradually decreased (rese) by applying posiive volages above rese = 1.3 and gradually increased (se) wih negaive volages below se = -.9. The rae of conducance change for boh swiching ransiions increases wih he applied volage (Fig. 3). The conducance remains unchanged when small volages, i.e. read =.5 for he considered devices, are applied o he device. Therefore, we assumed ha relaively large (exceeding se or rese hreshold) volages were applied o adjus synapic weighs during he raining process. Alernaively, smaller (read) volages, which do no modify synapic conducances, were assumed o be used for calculaion of nework oupu during raining and/or operaion of he classifier. III. IN-SITU MANHATTAN RULE TRAINING For he in-siu raining o be pracical, is area and ime overhead should be minimized. Sraighforward implemenaion of he backpropagaion algorihm in memrisive hardware does no seem o be pracical, because each weigh mus be modified by unique amoun according o (2) or (5). Such analog adjusmen of weighs is possible (e.g. using feedback wrie algorihm [17]) and could be reasonable for small circuis [1], bu would cerainly be oo slow for he desired large-scale circuis, especially aking ino accoun ha large number of is ypically required o perform raining [12]. Forunaely, here are some useful variaions of he backpropagaion algorihm, which allow much more efficien implemenaion of raining in he considered memrisive crossbar neworks. Here, we consider one such example - Manhaan Rule raining [18] - which is a coarse-grain variaion of backpropagaion algorihm. In Manhaan Rule only sign informaion of weigh adjusmen is uilized so ha weigh updaes for (2) and (5) become and ΔW ij M (n) = sgn[δw ij (n)], (8) ΔW ij M = sgn[δw ij ], (9) The main appeal of such a raining algorihm is ha all weighs are updaed by he same amoun, which simplifies he weigh updae operaion and creaes an opporuniy for efficien implemenaion of in-siu raining in hardware. Fig. 4 shows one insance of such implemenaion which we considered in his paper. In paricular, le us consider a small porion of he crossbar consising of 4 2 effecive weighs, or equivalenly 4 4 Curren (A) x 1-3 Experimen Simulaion se SET S read rese RESET olage () Fig. 2. Simulaed and experimenally measured I- swiching characerisics for P/TiO 2-x/P memrisor for an applied volage sweep shown in he inse [16]. ΔG (S) x G (S) x 1-3 Fig. 3. Simulaed swiching dynamics for rese and se ransiions for he considered memrisors, in paricular, showing absolue change in conducance as a funcion of iniial conducance, for several values of wrie volages (incremened in.1 seps). The conducances are measured a.5 (i.e. a read bias). differenial weighs (Fig. 4c). According o (3), for sochasic raining, he sign of he weigh updae depends on peripheral values of local error δ (associaed wih horizonal crossbar lines on Fig. 4c) and inpu X (associaed wih verical lines). There are four possible combinaions of signs of δ and X and, herefore, adjusmens of all weighs can be performed in four seps wih each sep corresponding o a paricular combinaion of signs. For example, Fig. 4c shows weigh updae for a specific case δ 1 <, δ 2 > and X 1 >, X 2 <, X 3 <, X 4 >. (Noe ha wih considered differenial weigh implemenaion, boh posiive and negaive synapses are adjused during he weigh updae, wih he laer always updaed in he opposie direcion.) Because all updaes have he same magniude, all he weighs sharing he same sign of δ and X in each sep could be updaed simulaneously. To implemen his parallel updae, each crossbar line receives a specific volage pulse sequence shown on Fig. 4a. In any paricular sep of such sequence, A se = rese =

4 only one specific se of memrisors, which are locaed a he crossbar lines wih he same signs of δ and X, receive large enough volage bias of a proper polariy o modify heir conducances (Fig. 4b). The remaining memrisors, which are no supposed o be modified during he same sep, receive volages below corresponding swiching hresholds, which is ensured by using rese rese < 2 rese, and se se > 2 se, (1) The hardware implemenaion of Manhaan Rule raining is quie sraighforward and involves applicaion of pulse sequences s 1 and s 2 o he verical crossbar lines wih X < and X >, respecively, and pulse sequences and o he horizonal crossbar lines wih δ < and δ >. In bach Manhaan Rule raining, he weigh updaes are no longer correlaed wih peripheral error and inpu values (Fig. 4d). In his case, memrisors can be updaed in parallel for wo crossbar lines (which form a differenial pair) using he scheme proposed for sochasic raining. Muliple pairs of crossbar lines, however, are updaed sequenially in his case (Figs. 4e and 4f). I. SIMULATION RESULTS The considered raining approach was evaluaed on hree daases - Cancer1, Diabees1 and Thyroid1 from Proben1 benchmark [19]. Each daase is implemened wih a wo-layer differenial-weigh MLP nework wih 4 hidden neurons. There are 9, 8 and 21 inpu neurons, and 2, 2, and 3 oupu neurons for Cancer1, Diabees1, and Thyroid1 daases, respecively. The oal number of paerns in he raining se were 35, 384 and 36 for Cancer1, Diabaes1 and, Thyroid1 daases, respecively. Several cases of weigh updaes were considered. In all simulaions, conducances were iniialized randomly beween G min =.1 ms and G max = 1 ms, and clipped a G min and G max during raining. Also, R = 2.27 kω and arge oupu values were (g) = ±.29, which correspond o he recommended sigmoid funcion from [14] for he considered read and range of conducances. The benchmark inpus were scaled o [- read, read ] range. All compuaions were performed using 32-bi floaing poin precision arihmeic. In he firs ( ideal ) case, weighs were updaed according o (8) and (9) wihou using he device model. Table I shows he bes classificaion performance achieved wihin 4 and he corresponding number of required o achieve bes performance, wih boh values averaged over 15 runs. For comparison, his able also shows simulaion resuls for he convenional backpropagaion algorihm and some of TABLE I. CLASSIFICATION PERFORMANCE FOR IDEAL NETWORK Daase Bach Backpropagaion Avg. Avg # of Bach Manhaan Rule Avg. Avg # of [19] Bes repored Cancer Diabees Thyroid (a) rese /2 (b) se /2 rese se (c) sep δ 1 < δ 1- > δ 2 > s 1 s 2 s 3 s 4 s 1 - s 1 - s 2 - s 2 - s 1 s 2 δ 2- < X 1 > X2 < X 3< X4 > Fig. 4. Manhaan Rule raining implemenaion: (a) 4-sep pulse sequences which are applied o he crossbar lines and (b) corresponding volage biases across memrisor (wih respec o he boom erminal) as a resul of an applicaion of pulse sequence. (c) A specific example of desired weigh updae for sochasic raining in a 4 4 memrisive crossbar circui and is corresponding implemenaion. (d) A specific example of desired weigh updae for bach raining, and (e, f) is corresponding implemenaions. On panel (d), red and green backgrounds correspond o negaive and posiive updaes, respecively. he bes repored resuls. In he remaining sudies, weigh updaes were performed using he memrisor device model described in Sec. IIC. The bes classificaion performance resuls were chosen wihin 15 and 3 of raining for bach and sochasic algorihms, respecively. Fig. 5 shows simulaion resuls for bach raining using various pairs of se and rese volages saisfying (1). The performance resuls are summarized in Table II. The Manhaan Rule raining was also simulaed for a more realisic case wih added defecs and variaions o he memrisive crossbar nework. Fig. 6 shows simulaion resuls wih a fracion of randomly chosen memrisors suck in eiher high conducive sae G max (suck-on-close) or low conducive sae G min (suck-on-open). In paricular, defecive memrisors are assumed o be equally spli beween suck-on-close and suck-on-open, so ha, e.g., he defec fracion of.2 (d) (e) s 1 s 2 s 2 s 1 (f)

5 Cancer1 Diabees1 Thyroid rese () rese () se () se () rese () se () Fig. 5. Misclassificaion rae for bach raining as a funcion of se and rese. The resuls are averaged over 5 runs. Daase Avg. TABLE II. CLASSIFICATION PERFORMANCE FOR MEMRISTIE CROSSBAR CIRCUITS (5 RUNS) Bach Manhaan Rule Opimal se / rese updaes Avg. Sochasic Manhaan Rule Opimal se / rese updaes Cancer / / Diabees / / Thyroid / / corresponds o 1% of suck-on-open and 1% of suck-onclose devices. Fig. 7 shows simulaion resuls for memrisive crossbar circuis wih device-o-device swiching hreshold variaions. Such variaions were simulaed by assuming ha se and rese of each device were normally disribued wih mean values corresponding o he opimal ones repored in Table II.. DISCUSSION AND SUMMARY Simulaion resuls summarized in Table I and II show ha classificaion fideliy for bach Manhaan Rule raining is comparable o ha of convenional backpropagaion raining and somewha worse as compared o he bes repored performance. Moreover, classificaion fideliy remained he same (or even slighly improved) when performing simulaions wih realisic device models. The sligh improvemen in performance could be explained by more opimal raining condiions, i.e. he opimal choice of rese and se volages, which effecively defines he learning rae for he simulaed insiu raining. Sochasic Manhaan Rule raining requires fewer o converge, hough is classificaion performance was significanly worse as compared o bach raining. A similar oucome is quie ypical for classical backpropagaion raining [14]. The addiional coarsening of he weigh updae for Manhaan Rule algorihm seems o be he reason behind furher increase in performance gap beween he wo modes of raining. As Figs. 6 and 7 show, he considered nework is quie robus o he variaions in device swiching dynamics and suck-on defecs. The main effec of device-o-device variaions is on convergence speed. For example, he number of raining o reach he classificaion fideliy of he variaion-free nework increased by a leas 1%, 4% and, 32% for Cancer1, Diabaes1 and, Thyroid1, respecively, while he classificaion performance degraded raher gracefully wih added variaions. Because in sochasic raining weighs are updaed for each applied paern, i is useful o esimae raining ime in erms of elemenary weigh updaes, raher han. Assuming ha applicaion of he four-sep pulse sequence is one elemenary updae, he raining ime for he sochasic algorihm is a produc of he number of paerns in he raining se and he oal number of. Here we neglec oher compuaions during raining such as described by (3) and (4) and assume ha he weighs can be updaed in parallel in differen MLP layers (even hough error is back-propagaed sequenially). For bach raining, he weighs are updaed only once per epoch, however, because of he sequenial updae, he raining ime for a paricular crossbar layer is he produc of he number of pos-synapic neurons and he oal number of. Table II provides raining ime expressed in elemenary updaes. Clearly, bach raining is he fases when aking ino accoun implemenaion deails. I is unclear hough if his will hold for more pracical circuis wih much larger crossbar arrays. Since in his paper we only focused on he weigh updae implemenaion, le us briefly discuss area overhead of oher operaions during raining. I should be noed firs ha he mos compuaionally expensive operaion for error backpropagaion is vecor δ by marix W compuaion (4). Such operaion can easily be performed wihou much addiional overhead using he same crossbar array hardware bu wih reverse direcion of compuaion. Oher operaions, e.g. derivaive calculaions in (3) and (4), are performed a he

6 periphery of he array, and hence heir relaive conribuion o he oal area is expeced o shrink as he crossbar arrays ge larger (which will happen for more pracical applicaions). The mos challenging operaion in bach raining is calculaion and soring of emporal weigh incremens which mus be performed for each weigh of he array. If he nework does no have o be rerained frequenly, one soluion would be o implemen his operaion off-chip. Invesigaion of beer soluions, which e.g. would combine he small overhead of sochasic Manhaan Rule raining and he high classificaion performance of bach raining is our immediae goal. In summary, we proposed a raining approach based on Manhaan Rule algorihm for mulilayer percepron neworks implemened wih memrisive crossbar circuis. The classificaion performance of he proposed approach was evaluaed on Proben1 benchmark for bach and sochasic modes of raining and compared wih sae-of-he-ar resuls. We found ha bach raining resuls in beer classificaion performance and poenially faser uning ime among he wo, hough a he price of significanly higher implemenaion overhead. X1-2 Cancer X1-1 Diabees Defec Fracion X1-1 Thyroid Fig. 6. Classificaion performance for bach raining wih suck-on-open and suck-on-close devices. For all panels, righ verical axis shows he percenage of increase in he number of cases ha did no converge o an accepable soluion, namely when remained above.1,.4 and.3 for Cancer1, Diabees1 and, Thyroid1, respecively, wihin 15 raining. The resuls are averaged over 15 runs. Cancer1 x1-2 Diabees1 x1-2 Thyroid Sandard Deviaion () Fig. 7. Classificaion performance as a funcion of sandard deviaion in se and rese swiching hreshold volages. The resuls are averaged over 15 runs. ACKNOWLEDGMENT The auhors would like o acknowledge helpful discussions wih F. Alibar, O. Bichler, C. Gamra, K. K. Likharev, G. Snider, and D. Querlioz Increase in non-converged Cases% REFERENCES [1] F. Alibar, E. Zamanidoos, and D. B. Srukov, Paern classificaion by memrisive crossbar circuis using ex-siu and in-siu raining, Naure Communicaions, vol. 4, Jun [2] C. Farabe, B. Marini, B. Corda, P. Akselrod, E. Culurciello, and Y. LeCun, NeuFlow: A runime reconfigurable daaflow processor for vision, in 211 IEEE Compuer Sociey Conference on Compuer ision and Paern Recogniion Workshops (CPRW), 211, pp [3] J. Lu, S. Young, I. Arel, and J. Holleman, A 1TOPS/W analog deep machine-learning engine wih floaing-gae sorage in.13 um CMOS, in Solid-Sae Circuis Conference Diges of Technical Papers (ISSCC), 214 IEEE Inernaional, 214, pp [4] P. Merolla, J. Arhur, F. Akopyan, N. Imam, R. Manohar, and D. S. Modha, A digial neurosynapic core using embedded crossbar memory wih 45pJ per spike in 45nm, in 211 IEEE Cusom Inegraed Circuis Conference (CICC), 211, pp [5] M. Prezioso, F. Merrikh-Baya, B. Hoskins, G. Adam, K. K., Likharev, and D. B. Srukov, Training and operaion of inegraed neuromorphic nework based on meal-oxide memrisors, available online a hp://arxiv.org/abs/ [6] D. Chabi, D. Querlioz, W. Zhao, and J.-O. Klein, Robus learning approach for neuro-inspired nanoscale crossbar archiecure, J. Emerg. Technol. Compu. Sys., vol. 1, no. 1, pp. 5:1 5:2, Jan [7] Y. Kim, Y. Zhang, and P. Li, A digial neuromorphic LSI archiecure wih memrisor crossbar synapic array for machine learning, in SOC Conference (SOCC), 212 IEEE Inernaional, 212, pp [8] D. Querlioz, O. Bichler, and C. Gamra, Simulaion of a memrisorbased spiking neural nework immune o device variaions, in The 211 Inernaional Join Conference on Neural Neworks (IJCNN), 211, pp [9] C. Yakopcic and T. M. Taha, Energy efficien percepron paern recogniion using segmened memrisor crossbar arrays, in The 213 Inernaional Join Conference on Neural Neworks (IJCNN), 213, pp [1] K. K. Likharev, Hybrid CMOS/nanoelecronic circuis: Opporuniies and challenges, Journal of Nanoelecronics and Opoelecronics, vol. 3, no. 3, pp , Dec. 28. [11] K. K. Likharev, CrossNes: Neuromorphic hybrid CMOS/nanoelecronic neworks, Science of Advanced Maerials, vol. 3, no. 3, pp , Jun [12] A. Krizhevsky, I. Suskever, and G.E. Hinon, ImageNe classificaion wih deep convoluional neural neworks, in NIPS 12, 212, pp [13] J. H. Lee and K. K. Likharev, Defec-oleran nanoelecronic paern classifiers, In. J. Circ. Theor. Appl., vol. 35, no. 3, pp , May 27. [14] Y. A. LeCun, L. Boou, G. B. Orr, and K.-R. Müller, Efficien BackProp, in Neural Neworks: Tricks of he Trade, G. Monavon, G. B. Orr, and K.-R. Müller, Eds. Springer Berlin Heidelberg, 212, pp [15] D. B. Srukov and H. Kohlsed, Resisive swiching phenomena in hin films: Maerials, devices, and applicaions, MRS Bullein, vol. 37, no. 2, pp , Feb [16] F. Merrikh-Baya, B. Hoskins, and D. B. Srukov, Phenomenological modeling of memrisive devices, Appl. Phys. A, vol. 118, no. 3, pp , Jan [17] F. Alibar, L. Gao, B. D. Hoskins, and D. B. Srukov, High precision uning of sae for memrisive devices by adapable variaion-oleran algorihm, Nanoechnology, vol. 23, no. 7, p. 7521, Feb [18] W. Schiffmann, M. Joos, and R. Werner, Opimizaion of he backpropagaion algorihm for raining mulilayer perceprons, Technical Repor, Universiy of Koblenz, [19] L. Prechel, PROBEN1-A se of benchmarks and benchmarking rules for neural nework raining algorihms, Technical Repor 21/94, Fakula fur Informaik, Universia Karlsruhe, p