COMPLEX-VALUED ASSOCIATIVE MEMORIES WITH PROJECTION AND ITERATIVE LEARNING RULES

Transcription

1 JAISCR, 2018, Vol 8, o 3, /jaiscr COMPLEX-VALUED ASSOCIATIVE MEMORIES WITH PROJECTIO AD ITERATIVE LEARIG RULES Teijiro Isokawa 1, Hiroki Yamamoto 1, Haruhiko ishimura 2, Takayuki Yumoto 1, aotake Kamiura 1, obuyuki Matsui 1 1 Graduate School of Engineering, University of Hyogo, 2167 Shosha, Himeji, Hyogo, Jaan 2 Graduate School of Alied Informatics, University of Hyogo, Minatojima-Minami-cho, Chuo-ku, Kobe, Hyogo, Jaan Submitted: 16th October 2017; acceted: 14th October 2017 Abstract In this aer, we investigate the stability of atterns embedded as the associative memory distributed on the comlex-valued Hofield neural network, in which the neuron states are encoded by the hase values on a unit circle of comlex lane As learning schemes for embedding atterns onto the network, rojection rule and iterative learning rule are formally exanded to the comlex-valued case The retrieval of atterns embedded by iterative learning rule is demonstrated and the stability for embedded atterns is quantitatively investigated Keywords: comlex-valued neural networks, associative memory, rojection 1 Introduction Comlex-valued neural networks CVs are neural networks of which neuronal arameters, such as inut, outut, and connection weights, are encoded by comlex values Various tyes of CVs have been extensively investigated [1, 2, 3] One of the advantages in CVs is that they can treat two-dimensional signal as a single entity, eg amlitude and hase of signals and twodimensional coordinates Hence CVs can be naturally alied to engineering roblems, such as land-mine detection based on radar wave [4] and equalization for communication channels [5] Comlex-valued multistate neural network is a tye of CVs where the state of a neuron is reresented by a distinct oint on a unit circle, ie a hase value, in a comlex lane [6, 7, 8, 9, 10] This network can be used as an associative memory because some discrete values such as ixel values in an image can be maed to the hase values and udates for neuron s state can be easily conducted Several tyes of multistate networks have been roosed and analyzed for both theoretical and exerimental aroaches It is imortant to develo learning schemes for embedding atterns onto associative memory There are several schemes for multistate networks [10, 11, 12, 13] The Hebbian rule is a basic and straightforward scheme for storing atterns, but it kees a major limitation for the roerties of atterns to be stored; atterns must be orthogonal to each other For ractical uses of associative memory, orthogonality is hardly satisfied among all atterns Projection rule is an imroved scheme from the Hebbian rule by rojecting non-orthogonal atterns to orthogonal ones [12, 14, 15] This learning scheme requires to calculate a seudo-inverse of matrix and its comutational cost becomes rather

2 238 Teijiro Isokawa, Hiroki Yamamoto, Haruhiko ishimura, Takayuki Yumoto, aotake Kamiura, obuyuki Matsui higher according to the size of each attern and the number of stored atterns Thus an iterative learning rule for comlexvalued and quaternionic multistate networks has been roosed and analyzed in [10] It is an imlementation of Projection rule and was firstly roosed for the real-valued Hofield networks in [16] By this rule, the connection weights in the network are gradually modified by iterative resentations of stored atterns, instead of the calculation of inverse matrix The modification of connection weights works so as to enlarge the basin of attractor for the resented attern in the energy landscae sanned by the connection weights, enabling to the resented attern being more deely embedded to the network Thus this rule could have a flexibility concerning the basins of attractors by controlling the resentation frequencies of stored atterns For the comlex-valued multistate networks, there is only a theoretical analysis in [10] roving that embedding atterns can be successfully conducted by this scheme, but none of exerimental results by using real or artificially generated atterns have been found This aer exlores the stabilities of atterns embedded on the multistate neural networks by the iterative learning rule through comarisons with Projection rule a short version of this aer was resented in [17] Embedding and retrieving atterns are demonstrated by using gray-scaled images, and the stabilities of embedded atterns with these rules are evaluated by using the randomly generated atterns with several resolution of a neuron state This aer is organized as follows Section 2 describes the fundamentals of comlex-valued multistate network and its learning rules Exerimental results for the stability of embedded atterns are shown in Section 3 This aer concludes in Section 4 2 Preliminaries 21 Comlex-valued Multistate eural etwork We first describe the comlex-valued multistate neuron model used in this aer that is also used in [7, 9, 10] In this model, the state and threshold of a neuron and the connection weights between the neurons are reresented by comlex values The outut of a neuron is also a comlex value, but it is restricted to one of the distinct oints on the unit circle in a comlex lane Therefore, the outut of this model can only be reresented by a hase value Figure 1 shows an examle of outut oints in a comlex lane where the number of hase quantizing delimiter K is 6 z 3 Figure 1 An examle of outut oints in the comlex-valued multistate neuron model We consider a Hofield neural network with comlex-valued multistate neurons The action otential h t of a neuron at a discrete time t is given by h t= q w q u q t, 1 where u q t is the state of the neuron q at a time t, and w q is the connection weight from neuron q to neuron The state of neuron at t + 1 is determined by u t + 1=csignh t z 1/2, 2 where z 1/2 = e iφ 0/2 is a fixed threshold value, and φ 0 = 2π/K defines a quantized unit The function csign is an activation function of a multistate neuron defined as csignu= z2 Im z 1 z 4 z 5 z 0 0 argu < φ 0 z 1 φ 0 argu < 2φ 0 z 0 z K 1 K 1φ 0 argu < Kφ 0 Re 3

3 COMPLEX-VALUED ASSOCIATIVE MEMORIES 239 The state of a neuron has K quantized levels, called K-stage hase quantizer The rocedure for udating neuron s state is illustrated in Figure 2 where K = 6 is adoted In this rocedure, 1 the action otential h t is firstly calculated, 2 this action otential is rotated by z 1/2, and 3 the udated state of a neuron is determined by csign function Im Figure 2 Procedures for calculation of neuron s state The stability of the network using this model was roved by showing that the energy function of the network monotonically decreased under the condition φ < φ 0 /2, where φ is a hase difference between the state at time t +1 and the action otential at time t for the neuron undergoing its udate The energy function E is given by E = 1 2 w q u u q, 4 q where the connection matrix W = {w q } is a Hermitian matrix w q = w q with the condition of w rr 0 1 r A roof for this stability is shown in Section A of Aendix 22 Condition for Embedding Patterns z 1 2 h t z 1/2 1 h t 3 u t + 1 z 0 Re In order to introduce the learning schemes into the multistate network, the stability condition for the neurons states is imortant Let ξ µ = {ξ µ } = 1,,;µ = 1,,n be a vector of the µ- th memory attern, where n denotes the number of atterns to be embedded ξ µ takes one of K integer values ξ µ {0,,K 1} Therefore, storing a attern to the network, each ξ µ is maed onto ε µ, which is a oint on the unit circle in a comlex lane by ε µ = z ξµ = e iξµ φ 0 5 ε µ reresents a stable network configuration if u t + 1=u t=ε µ 6 is satisfied for every neuron = 1,, This condition leads to the hase relation argε µ argh t z 1/2 < argε µ +φ 0 7 According to Eq2, this relation can be exressed as argh t argε µ < φ Also, in order to obtain greater stability of the desired memory atterns, a threshold arameter κ is introduced so that the left-hand side of Eq8 should be smaller than κ, that is, argh t argε µ < κ < φ For an aroriate κ, the trainable {w q } in h should satisfy this condition 23 Projection Rule A straightforward way to embed atterns onto the associative memory networks is the use of Hebbian rule The Hebbian rule is defined as w q = 1 n ε µ ε µ q, 10 µ=1 where n is the number of atterns to be embedded However, there is a major limitation in the Hebbian rule, ie, it works only when the atterns ε µ = {ε µ 1,,εµ } satisfy the condition ε µ ε ν = 0 11 for all combination of µ and ν where 1 µ,ν n and µ =ν This means that the atterns to be embedded in the network must be orthogonal to each other, though the atterns rovided in most cases are non-orthogonal Projection rule [14, 15, 12] is a learning scheme that can embed non-orthogonal atterns in a network The key idea of the Projection rule is that non-orthogonal atterns are first rojected onto orthogonal ones, and then the Hebbian rule is alied to the rojected atterns [10] Projection is conducted by introducing the matrix {Q µν }, defined as: Q µν = 1 ε µ ε ν 12

4 240 Teijiro Isokawa, Hiroki Yamamoto, Haruhiko ishimura, Takayuki Yumoto, aotake Kamiura, obuyuki Matsui The weight matrix of the network, w, is calculated by w q = 1 µ,ν ε µ Q 1 µν εν q, 13 where Q 1 is the seudo inverse matrix of Q Patterns embedded by this scheme become stable oints in the network, as in the case of the Hebbian rule This can be checked by calculating the action otential of a neuron h by alying an embedded attern ε σ as the inut to the network: h = w q ε σ q q=1 = 1 ε µ Q 1 µν ε ε σ q µ,ν q = ε Q 1 µν Q νσ = µ,ν ε Q 1 Q µσ µ = ε δ µ,σ µ = ε σ, 14 where δ µ,σ denotes the Kronecker delta function 24 Iterative Learning Rule Iterative learning rule for a comlex-valued multistate neural network has been roosed and theoretically analyzed in [10] This learning rule is a comlex-valued extension of the iterative learning for real-valued one in [16] The connection weight w q is udated by using the desired memory attern as w new q = w old q + δw q, 15 δw q = 1 εµ ε µ q, 16 where ε µ is the comlex conjugate of ε µ, ie, ε µ = e iξµ φ 0 After the udate, the action otential of neuron becomes h new = q=1 = q = h old w new q u q w old q + δw q u q + q 1 εµ ε µ q u q 17 Consider the effect of udating the connection weight on the memory attern state ε µ This can be confirmed by setting u q = ε µ q in Eq17 as h new = h old + q 1 εµ ε µ q ε µ q = h old + ε µ, 18 since q ε µ q ε µ q = holds These vectors in a comlex lane are shown in Figure 3 The relation of these vectors indicates that the direction of h new is turning towards ε µ by the iterative alications of Eq15 Hence, the memory attern will be sufficiently stable when the condition described in Eq9 is satisfied The achievement of stability by the iterative alications of Eq15 is exlicitly shown in Section B of Aendix Since the modification of the connection weights by Eq18 is conducted for a given memory attern, this rule could have a flexibility for making the basin of attractor for a memory attern by controlling the resentation frequencies of the memory atterns h old θ new θ old Figure 3 Behavior of vectors in a comlex lane due to the alication of iterative learning 3 Exerimental Results h new ξ µ ϕ 0 31 Image Retrieval from oisy Inut We first investigate the ability for retrieving the stored atterns from the noisy inuts, in order to ensure that associative memory with the iterative learning rule actually works Figure 4 shows three stored atterns for this exeriment, which are used in [18] The size of these images are adjusted to ε µ

5 COMPLEX-VALUED ASSOCIATIVE MEMORIES = 8100 ixels and each ixel value is reresented by 5 bits 32 levels The number of neurons in the associative memory network is the same as the size of the images, ie, = 8100 The resolution factor for the neuron state is also set to the same as the resolution of the ixel value, K = 32 In learning stage, each of the inut images are used for modifying the connection weights, in order of the images in Figs 4a, 4b, and 4c The number of iterations is set to 10 After learning, the test inut images are used to the initial configuration of the network, and then the udates of the neuron states are conducted until the configuration of the network converges, ie until the states of all neurons in the network do not change by using the Eqs1 and 2 For the test inut images to the associative memory, one of these stored atterns with corrution is adoted The corrution is conducted so that, for each of ixels in the target image, the ixel value is relaced to random value with a certain robability r n Examles of test inut images with various r n s are shown in Figure 5 Higher value of r n results in an image with larger region being corruted a r n = 01 b r n = 03 c r n = 06 Figure 6: Outut images from the network when the Figure 6 Outut images from the network when the images in Figure 5 are used for inut Figure 6 shows the outut configurations of the network for the inuts of Figures 4a, 4b, and 4c, resectively This result shows that the network can retrieve the correct image for the inut image with lower r n, thus iterative rule could certainly embed each of the stored atterns with a certain attractors around it By using other inut images that are originated from the images in Figures 4b and 4c, similar results can be obtained as shown in Figure 7 a Inut image r n = 04 b Outut from the image in Fig 7a a b c Figure 4: Three stored images for the image retrieval Figure 4 Three stored images for the image retrieval exeriment c Inut image r n = 05 d Outut from the image in Fig 7c Figure 7 Other examle of inut and outut images a r n = 01 b r n = 03 c r n = 06 Figure 5 Inut images with various noise robabilities r n 32 Stability Analysis for Embedded Patterns We next exlore the stabilities of atterns embedded by two learning rules This is conducted by evaluating retrieval erformances for the network from one of stored atterns as its initial state In this exeriment, the embedded atterns are com-

6 242 Teijiro Isokawa, Hiroki Yamamoto, Haruhiko ishimura, Takayuki Yumoto, aotake Kamiura, obuyuki Matsui osed of randomly generated values The size of the attern the number of neurons in the network is set to 100, and the resolution factor K is set to K = 4,6,8,16,32,64 The number of the stored atterns, denoted by M, varies such that M = 1,,30 The stability of the atterns are checked by the following rocedure First, for given K and M, the stored atterns are reared and are embedded to the network In the iterative learning, resenting the stored atterns to the network with 10 iterations is conducted for modifying the connection weights of the network After the learning, each of the stored atterns is set to the network as its initial configuration, then the udates of the network are conducted for all the neurons in the network If the configuration of the network does not change, the inut attern one of the stored atterns can be stable; otherwise the inut attern cannot be embedded An embedding rocess with given arameters is regarded as successful if all the stored atterns are stable Figure 8 shows the M deendencies of the retrieval success rates with several Ks, in which 100 different sets of atterns are used for each M For the fixed size of the network, it becomes difficult to embed the larger number of atterns, as in the case of real-valued Hofield networks Also, the number of atterns to be embedded correctly deends on the resolution K for the neuron state A neuron with larger value of K should contain much more information for the embedded atterns, and it is equivalent to larger value of M in the network Thus, increasing M and/or K in the network with small number of neurons will lead to degradation of embedded atterns in the network For comarison of erformance, the M deendence of the retrieval success rates obtained by rojection rule is shown in Figure 9 Similar tendencies to the results obtained by iterative learning rule are shown though overall caacities of atterns in the network are much higher Retrieval Success Rate [%] K=4 K=6 K=8 K=16 K=32 K= M Figure 8 M deendencies of the retrieval success rates by iterative learning Retrieval Success Rate [%] K=4 K=6 K=8 K=16 K=32 K= M Figure 9 M deendencies of the retrieval success rates by Projection rule The comutational costs for the rojection rule and iterative learning rule are discussed Most of the comutation on the rojection rule is devoted to the creation of the weight matrix Eq13, thus the number of neurons becomes a major arameter determining the comutational time Figure 10a shows comutational time by the rojection rule with resect to the number of neurons, in which evaluations are conducted on a PC Core i GHz, Memory 8GB The curves in this figure have similar tendencies and are aroximated as a 2 where a is a arameter deending on the number of memory atterns M In the iterative learning, comutational cost is affected by the numbers of iterations, neurons, and atterns The comutational time, with various Ms and 10 iterations, with resect to the number of neurons is shown in Figure 10b Comutational time can also be arox-

7 COMPLEX-VALUED ASSOCIATIVE MEMORIES 243 imated as a 2, though overall comutational costs for the iterative learning are relatively low Comutational time msec Comutational time msec M=5 M=10 M=15 M=20 M=25 M= umber of neurons a Projection rule M=5 M=10 M=15 M=20 M=25 M= umber of neurons b Iterative learning rule Figure 10 Comutational time for a rojection rule and b iterative learning rule with resect to the number of embedded atterns 33 Stability of Embedded Patterns by Iterative Learning Rule In the revious Section, we show some comarisons of iterative learning rule and rojection rule from the viewoint of retrieval erformances by setting one of stored atterns as an initial state of the network Performance differences arise due to the stability in the embedding rocesses by these learning schemes In this Section, we investigate the stabilities on the atterns embedded by the iterative learning in more detail, through an analysis on the evolutionary rocess of weight connections in learning We have already discussed the stability of the memory atterns in Section 22 by using the hase relation Eqs7 and 8 Thus, it is useful to adot the hase difference between the action otential of a neuron and a memory state for the corresonding neuron, in order to evaluate the stability of an embedded attern A hase difference d µ for the - th neuron in the µ-th attern at the t-th iteration in learning is defined as d µ t= argh t argε µ 19 Thus the basic stability condition is d µ < φ 0 /2 The average of d µ t for all neurons is denoted as d µ t We introduce a rocedure for embedding the memory atterns by iterative learning In the first iteration in embedding atterns, each of the memory atterns is reared and the connection weights are udated by this attern according to w new q = w old q + α δw q, 20 where α is a real-valued arameter When α = 10, this udating scheme corresonds to the original udating scheme Eq15 Then, d µ is calculated for each of the memory atterns and for each of the neurons At each subsequent iteration in learning, only the atterns with the condition of d µ φ 0 /2 for any are reared and used for udating the connection weights The following conditions are used for the exeriments The numbers of neurons and memory atterns are = 100 and M = 15, resectively, and the resolution factor K is set to 4 thus φ 0 /2 = Each element of the memory atterns is randomly generated Iterative learning with α = 01 in Eq20 and rojection rule are used for embedding the memory atterns, and the erformances of these schemes are evaluated As in the exeriment in the revious Section, all atterns are embedded to the network and then each of the atterns is set to the initial state of the network If the configuration of the network do not change from the initial configuration, this attern is stably embedded Retrieval is regarded as successful if all the atterns are stably embedded First we show the evolutions of the averaged hase differences by iterative learning Table 1 shows an examle of averaged hases for the first six iterations of learning In this table there are the averaged hase differences for only four atterns µ = 2,4,5,14 that are involved in the iterations of learning At the first iteration of learning there are two neurons that do not satisfy the condition d µ < φ 0 /2 for the 14-th attern Thus in

8 244 Teijiro Isokawa, Hiroki Yamamoto, Haruhiko ishimura, Takayuki Yumoto, aotake Kamiura, obuyuki Matsui the second iteration, only the 14-th attern is used for udating the connection weights As a result, the averaged hase difference for this attern actually decreased but the hase differences for the other atterns slightly increased At the third iteration, the second attern is involved in learning and then the hase difference for this attern decreased, by the reason that some hase differences with resect to the second attern exceed to the threshold φ 0 /2 Similar rocess is conducted at the each subsequent iteration in learning The averaged differences gradually decreases according to the iterations, though sometimes they slightly increase As described above, strengthening the embed for a articular attern often results in weakening the other atterns, though embedding all atterns is still ossible Figure 11 shows an examle of changes of retrieval success rate with resect to the iteration in learning The success rate increases according to the iterations of learning and sometimes fluctuates by strengthening and weakening the stored atterns by learning At the 66-th iteration, all the atterns could be retrieved thus embedded The averaged hase differences for the network with 66-th iteration in learning are shown in Figure 12 The results for the network in which the atterns are embedded by rojection rule are also shown in this figure The hase differences by rojection rule are much smaller than those by iterative learning rule, thus the atterns are better embedded by rojection rule This result would reflect the ossible number of embedded atterns, shown in Figs 8 and 9 The arameters for embedding atterns, such as the number of memory atterns M and the resolution factor K, imact the rocess of embedding, ie, the evolution of the retrieval success rate Figure 13 shows the retrieval success rates with resect to the learning iterations in the case of M = 20 and K = 4 and in the case of M = 15 and K = 6 Increase of K or M makes difficult to embed atterns in the iterative learning rule, as suggested in Figure 8 Retrieval Success Rate [%] Learning Iteration Figure 11 Evolution of retrieval success rate by iterative learning rule = 100, M = 15, K = 4 Averaged hase difference 03 Iterative Learning Rule Projection Rule Memory attern number Figure 12 Averaged hase differences for 15 atterns by iterative learning rule with 66-th iteration and rojection rule Retrieval Success Rate [%] K=4, M=20 K=6, M= Learning Iteration Figure 13 Evolutions of retrieval success rates by iterative learning rule = 100, M = 20, K = 4 and = 100,M = 15,K = 6

9 COMPLEX-VALUED ASSOCIATIVE MEMORIES 245 Table 1 An examle of evolutions for the averaged hase differences according to the iterations by iterative learning rule At the first iteration of learning, all the atterns are involved in learning At the second, third, fourth, and fifth iteration of learning, the 14-th, the second, the fourth, and the fifth atterns, resectively, are involved in learning µ d µ 1 d µ 2 d µ 3 d µ 4 d µ 5 d µ Conclusion In this aer, the stabilities of embedded atterns are investigated in the comlex-valued associative memory The associative memory is based on comlex-valued multistate Hofield neural network, and iterative learning rule and rojection rule are adoted for embedding atterns The exerimental results show that the atterns can be certainly embedded in the network with their attractors, and the caability for embedding deends on the number of atterns and the resolution for reresenting neuron states The stability of embedded atterns have also been exlored through measuring the hase differences on resenting the memory atterns to the trained networks Though the comutational cost for iterative learning rule is much lower than that of rojection rule, the caability of iterative learning rule is not high as rojection rule, as indicated in the exerimental results One of our future researches directs to develoing imrovement schemes for the learning caability by iterative learning rule This could be realized by incororating a more sohisticated control of udating the connection weights in the network Analyzing other tyes of learning schemes, such as the scheme in [11], will be necessary The analysis of this learning scheme for higher dimensional associative memory is also a challenging roblem, such as for quaternionic multistate Hofield neural networks [19, 20, 21] in which two or three kinds of hase values are available for reresenting neuron s state Acknowledgment This study was financially suorted by the Jaan Society for the Promotion of Science Grantin-Aids for Young Scientists B and Scientific Research C 16K00248 References [1] A Hirose, editor, Comlex-Valued eural etworks: Theories and Alication, volume 5 of Innovative Intelligence, World Scientific Publishing, Singaore, 2003 [2] T itta, editor, Comlex-Valued eural etworks: Utilizing High-Dimensional Parameters, Information Science Reference, Hershey, ew York, 2009 [3] A Hirose, editor, Comlex-Valued eural etworks: Advances and Alications, The IEEE Press Series on Comutational Intelligence, Wiley- IEEE Press, 2013 [4] Y akano and A Hirose, Imrovement of Plastic Landmine Visualization Performance by Use of Ring-CSOM and Frequency-Domain Local Correlation, IEICE Transactions, 92-C1, , 2009 [5] Rajoo Pandey, Comlex-Valued eural etworks for Blind Equalization of Time-Varying Channels, International Journal of Signal Processing, 11, 1 8, 2004 [6] A J oest, Associative Memory in Sarse eural etworks, Eurohysics Letters, 66, , 1988 [7] Aizenberg and I Aizenberg, C Based on Multi-Valued euron as a Model of Associative Memory for Gray-Scale Images, Proceedings of the 2nd IEEE International Worksho on Cellular eural etworks and their Alications CA- 92, 36 41, 1992 [8] I Aizenberg, Aizenberg, and J Vandewalle, Multi-Valued and Universal Binary eurons Theory, Learning and Alications, Kluwer Academic Publishers, Boston/Dordrecht/London, 2000

10 246 Teijiro Isokawa, Hiroki Yamamoto, Haruhiko ishimura, Takayuki Yumoto, aotake Kamiura, obuyuki Matsui [9] S Jankowski, A Lozowski, and J M Zurada, Comlex-Valued Multistate eural Associative Memory, IEEE Transactions on eural etworks, 76, , 1996 [10] T Isokawa, H ishimura, and Matsui, An Iterative Learning Scheme for Multistate Comlex- Valued and Quaternionic Hofield eural etworks, Proceedings of International Joint Conference on eural etworks IJC2009, , 2009 [11] M K Müezzinoğlu, C Güzeliş, and J M Zurada, A ew Design Method for the Comlex-Valued Multistate Hofield Associative Memory, IEEE Transactions on eural etworks, 144, , 2003 [12] D-L Lee, Imrovements of comlex-valued Hofield associative memory by using generalized rojection rules, IEEE Transaction on eural etworks, 175, , 2006 [13] M Kobayashi, Pseudo-relaxation learning algorithm for comlex-valued associative memory, International Journal of eural Systems, 182, , 2008 [14] T Kohonen, Self-Organization and Associative Memory, Sringer, Berlin, Heidelberg, 1984 [15] L Personnaz, I Guyon, and G Dreyfus, Collective Comutational Proerties of eural etworks: ew Learning Mechanisms, Physical Review A, 34, , 1986 [16] S Diederich and M Oer, Learning of Correlated Patterns in Sin-Glass etworks by Local Learning Rules, Physical Review Letters, 58, , 1987 [17] H Yamamoto, T Isokawa, H ishimura, Kamiura, and Matsui, Pattern Stability on Comlex-Valued Associative Memory by Local Iterative Learning Scheme, Proceedings of 6th International Conference on Soft Comuting and Intelligent Systems & 13th International Symosium on Advanced Intelligent Systems SCIS-ISIS 2012, 39 42, 2012 [18] F Flueret and D Geman, Stationary Features and Cat Detection, Journal of Machine Learning Research, 9, , 2008 [19] T Isokawa, H ishimura, A Saitoh, Kamiura, and Matsui, On the Scheme of Quaternionic Multistate Hofield eural etwork, Proceedings of Joint 4th International Conference on Soft Comuting and Intelligent Systems and 9th International Symosium on advanced Intelligent Systems SCIS & ISIS 2008, , 2008 [20] T Isokawa, H ishimura, and Matsui, Commutative Quaternion and Multistate Hofield eural etworks, Proceedings of IEEE World Congress on Comutational Intelligence WCCI2010, , 2010 [21] T Minemoto, T Isokawa, H ishimura, and Matsui, Quaternionic multistate Hofield neural network with extended rojection rule, Artificial Life and Robotics, 211, , 2016 A Proof for the Stability of the etwork This Section describes a roof for the stability of the network, ie, the network state with an initial configuration always converges to one of the local minima This can be shown by monotonically decrease of the energy defined in Eq4 under the following conditions: W = {w q } being a Hermitian matrix w q = w q, selfconnections of the network take real and non-negative values w rr 0, and a hase difference in neuron state s udate is not large such as φ < φ 0 The first stage of the roof is to exlicitly write the contribution of a neuron to the whole of energy in the network with neurons Let E r t be a contribution of neuron r at the time t to the energy This can be written as E r t = 1 w r u 2 t u r t =1 1 w rq u q t u r t 2 = w rru r tu r t = 1 w 2 ru t u r t =1 1 w r u q t u r t 2 = w rr = 1 2 h r tu r t 1 2 h rtu r t+ 1 2 w rr = Reu r th r t w rr 21 Let us assume that this neuron r udates its state The contribution E r t + 1 is then described as E r t + 1= Reu r t + 1h r t w rr 22

11 COMPLEX-VALUED ASSOCIATIVE MEMORIES 247 The action otential of the neuron r, h r t + 1, can be reresented by its variables at the time t and time t +1 h r t + 1 = = w rq u q t + 1 q=1 w rq u q t + 1 w rr u r t q=1 +w rr u r t + 1 = h t w rr u r t u r t + 1 Using this variable, Eq22 can be rewritten as E r t + 1 = Reu r t + 1h r t w rr u r t u r t w rr = Reu r t + 1h r t 23 +w rr Reu r t + 1u r t u r t w rr = Reu r t + 1h r t +w rr Reu r t + 1u r t w rr 24 The difference of the contribution between the time t and t + 1, E, is calculated from Eqs21 and 22 E = E r t + 1 E r t where = {Reu r t + 1h r t Reu r th r t} +w rr {Reu r t + 1u r t 1} = X 1 X 2 +w rr X 3, 25 X 1 = Reu r t + 1h r t, X 2 = Reu r th r t, X 3 = Reu r t + 1u r t 1 Proving the monotonically decrease of E corresonds to showing E 0 with resect to the change of network state Thus it is necessary to investigate each of X 1, X 2, and X 3 For this urose the following two relations are required One is the relation of the oututs between the time t and t + 1: u r t + 1=u r te iaφ 0, 26 where a is an integer The other relation refers to the action otential h r t and the outut u r t: h r t = h r t e i φ u r t + 1 = h r t e iaφ0+ φ u r t 27 Investigation the term X 3 is firstly conducted the term u r t + 1u r t is decomosed by using the abovementioned relations, as u r t + 1u r t = u r te iaφ 0 ur t = u r te iaφ 0 u r t = e iaφ 0 Thus X 3 = cos aφ 0 1 is obtained This term does not take a ositive value due to 1 cos aφ 0 1 Consequently w rr X 3 0 is satisfied under the condition of w rr being a non-negative value and Similarly the terms X 1 and X 2 are investigated as X 1 = Re u r te iaφ 0 h r t e iaφ0+ φ u r t = h r t Ree i φ = h r t cos φ, X 2 = Re u r t h r t e iaφ0+ φ u r t = h r t cosaφ 0 + φ, resectively Thus under the condition φ < φ 0, X 1 X 2 = h r t cos φ cosaφ 0 + φ 0, 28 is obtained Finally E 0 is obtained from these relations B Proof for the stability of local iterative learning scheme This Section describe a roof concerning the stability of iterative learning rule, ie, this rule can actually embed each of the stored atterns as its stable configuration in the network Let θ new h new and h old 18 can be exressed as and θ old be the angles of in a comlex lane, resectively Equation h new e iθnew = h old e iθold + ε µ e iξµ φ 0 Multilication this equation with e iξµ φ 0 gives h new e iθnew ξ µ φ 0 = h old e iθold ξ µ φ 0 + ε µ This can be written as two equations for real and imaginary comonents: h new cos θ new ξ µ φ 0 and h new = h old cos sin θ new θ old ξ µ φ 0 + ε µ 29 ξ µ φ 0 = h old sin θ old ξ µ φ 0 30

12 248 Teijiro Isokawa, Hiroki Yamamoto, Haruhiko ishimura, Takayuki Yumoto, aotake Kamiura, obuyuki Matsui To reresent the value of the udate frequency, h L+1 and h L for h new and h old, resectively, are substituted where L is the number of udates in iterative learning Then, Eq29 can be exressed as h L+1 cosθ L+1 ξ µ φ 0 = h L cosθ L ξ µ φ 0 + ε µ = h L 1 cosθ L 1 ξ µ φ 0 +2 ε µ = h 0 cosθ 0 ξ µ φ 0 +L + 1 ε µ Hence, the following relation is obtained: cosθ L+1 ξ µ φ 0 = h0 cosθ 0 ξ µ φ 0 +L + 1 ε µ h L+1 31 In a similar way, using Eq30, the following relation is obtained: h L+1 sinθ L+1 ξ µ φ 0 = h 0 sinθ 0 ξ µ φ 0 32 From these equations, = cosθ L+1 ξ µ φ 0 h 0 Θ +L + 1 ε µ h L + 1 h 0 ε µ Θ +L ε µ 2 33 where Θ = cosθ 0 ξ µ φ 0 According to the iterative udate in the connection weights, Eq33 becomes cosθ L+1 ξ µ φ 0 L + 1 εµ L ε µ = 1, 34 2 for large values of L This shows that the direction of aroaches that of ε µ, that is, h L+1 argh L+1 ξ µ φ 0 < argh 0 ξ µ φ 0 35 for large value of L It can be seen that the direction of h would move towards that of ε µ by even one udate From the Eqs29 and 30, the following relations hold: cosθ new ξ µ φ 0 = hnew 2 h old 2 + ε µ 2 2 h new ε µ and cosθ old ξ µ φ 0 = hnew 2 h old 2 ε µ 2 2 h old ε µ We see that the difference between the above two equations is always greater than 0, ie, cosθ new ξ µ φ 0 cosθ old ξ µ φ 0 h new + h old { h new h old } 2 µ + ε 2 = 2 h new h old ε µ > 0 36 Consequently, we obtain argh new ξ µ φ 0 < argh old ξ µ φ 0 37 This indicates that even one udate can make the direction of h move towards that of ε µ Thus the desired memory atterns can be embedded by the roosed learning scheme exressed in Eq15 In the case of all the atterns being orthogonal to one another, this rule becomes equivalent to the Hebbian learning rule The connection weights are defined by the Hebbian rule in Eq10 The action otential at u = ε µ becomes h = = w q u q q=1 q=1 In this case, the relation always holds 1 n ε ν ε ν q ε µ q ν=1 = ε ν 1 ν ε ν q ε µ q q = ε ν δ ν,µ ν = ε µ argh ξ µ φ 0 = 0 < κ 38

13 COMPLEX-VALUED ASSOCIATIVE MEMORIES 249 Teijiro Isokawa received his BE degree Electronic Engineering, ME degree Electronic Engineering, and DE degree Doctor of Engineering in 1996, 1999, and 2004, resectively, from Himeji Institute of Technology, Jaan He is currently an associate rofessor in the Graduate School of Engineering, University of Hyogo, Jaan His research interests include nanocomuting, molecular robotics, hyercomlex-valued neural networks, and cognitive models Hiroki Yamamoto received his BE degree and ME degree in 2012 and 2014, resectively, from University of Hyogo, Jaan His research interests include associative memories based on comlex and quaternionic neural networks Haruhiko ishimura graduated from the Deartment of Physics, Shizuoka University in 1980, and comleted the doctoral rogram at Kobe University and received the PhD degree in 1985 He is currently a rofessor in the Graduate School of Alied Informatics, University of Hyogo His research field is intelligent systems science by several architectures such as neural networks and comlex systems He is also resently engaged in research on biomedical, healthcare, and high confidence sciences He is a member of the IEEE, IEICE, IPSJ, ISCIE, JS, and others, and was awarded ISCIE aer rize in 2001 and JSKE aer rize in 2010 Takayuki Yumoto received the BE, Master of Informatics, and PhD degrees in Informatics from Kyoto University, in 2002, 2004, and 2007, resectively He is an assistant rofessor at University of Hyogo since 2007 His research interests include Web search and mining He is a member of the ACM, the IEEE Comuter Society, the Information Processing Society of Jaan IPSJ, the Institute of Electronics, Information and Communication Engineers IEICE, and the Database Society of Jaan DBSJ aotake Kamiura received the BE degree Electronic Engineering in 1990, the ME degree Electronic Engineering in 1992 and the DE degree Doctor of Engineering in 1995 from Himeji Institute of Technology, Jaan He is currently a rofessor in the Deartment of Electronics and Comuter Science, Graduate School of Engineering, University of Hyogo His research interests include the alication of soft comuting to medical engineering He is members of the Institute of Electronics, Information and Communication Engineers, the the Jaanese Society of Medical Imaging Technology and the IEEE obuyuki Matsui received his BS degree in hysics from the Faculty of Science, Kyoto University, Jaan, in 1975, and ME and Dr Eng degrees in nuclear engineering from Kyoto University in 1977 and 1980, resectively He is currently a rofessor emeritus at University of Hyogo Dr Matsui is a member of IS, IEICE, ISCIE, SICE and the Physical Society of Jaan