A Solution to the 4-bit Parity Problem with a Single Quaternary Neuron

Transcription

1 Neura Information Processing - Letters and Reviews Vo. 5, No. 2, November 2004 LETTER A Soution to the 4-bit Parity Probem with a Singe Quaternary Neuron Tohru Nitta Nationa Institute of Advanced Industria Science and Technoogy (AIST), AIST Tsukuba Centra 2, Umezono, Tsukuba-shi, Ibaraki, Japan E-mai: tohru-nitta@aist.go.jp (Submitted on August 20, 2004) Abstract - This etter wi carify the fundamenta properties of a quaternary neuron whose weights, threshod vaues, input and output signas are a quaternions, which is an extension of a usua rea-vaued neuron to quaternions. The main resuts of this etter are summarized as foows. A quaternary neuron has an orthogona decision boundary. The 4-bit parity probem which cannot be soved with a singe usua rea-vaued neuron, can be soved with a singe quaternary neuron with the orthogona decision boundary, resuting in the highest generaization abiity. Keywords - Quaternion, decision boundary, parity probem 1. Introduction Severa neura network modes with compex-vaued (i.e., two-dimensiona) or three-dimensiona parameters have been proposed [1, 2, 3, 4, 5, 6] which can dea with compex-vaued signas or three-dimensiona vectors naturay and demonstrated to have the inherent properties such as the abiities to earn 2D or 3D affine transformations [2, 7, 5, 8, 9, 10]. Particuary, the Compex-BP [2, 5] and the 3DV-BP [3] have been successfuy appied to computer vision [11]. We can find some other appications of the compex-vaued neura networks to various fieds such as optica processing and image processing in the iterature [12, 13, 14]. Quaternary neura networks were proposed by Arena and Nitta independenty in the mid-1990s [15, 16]. The quaternary neura network is an extension of the cassica rea-vaued neura network to quaternions, whose weights, threshod vaues, input and output signas are a quaternions where a quaternion is a four-dimensiona number and was invented by W. R. Hamiton in 1843 [17]. It is expected that the quaternary neura network can be effectivey used in the fieds such as robotics and computer vision in which quaternions have been found usefu. Actuay, it was shown in [18] that the quaternary neura network can sove severa probems such as the interpoation of the eectric fied generated by two charges ocated in a 3D space, the cassification probem of three species of the Iris fower, the chaotic time series prediction, and the attitude contro of a rigid body in a 3D space with fewer neurons and connections than the cassica rea-vaued neura network. Isokawa et a. successfuy appied a quaternary neura network which cacuated a rotation to a coor image compression probem [19]. This etter wi carify the fundamenta properties of a quaternary neuron. The main resuts are summarized as foows. The decision boundary of the quaternary neuron consists of four hypersurfaces which intersect orthogonay each other, and divides a decision region into 2 4 (= 16) equa sections. The 4-bit parity probem which cannot be soved with a singe rea-vaued neuron, can be soved by a singe quaternary neuron with the orthogona decision boundary, resuting in the highest generaization abiity, which reveas a potent computationa power of the quaternary neuron. 33

2 4-bit Parity Probem with a Singe Quarternary Neuron T. Nitta 2. The Quaternary Neuron There appear to be severa approaches for extending the standard neuron to higher dimensions. One approach is to extend the number fied, i.e., from rea numbers x (1 dimension), to compex numbers z = x + yi (2 dimensions), to quaternions q = a + bi + cj + dk (4 dimensions), to octaves (8 dimensions), to sedenions (16 dimensions),. Another approach is to extend the dimensionaity of the weights and threshod vaues from 1 dimension to n dimensions using n-dimensiona rea vaued vectors. Moreover, the atter approach has two varieties : (a) weights are n-dimensiona matrices [3], (b) weights are n-dimensiona vectors [4]. In this etter we dea with the quaternary neuron, which is an extension of the rea-vaued neuron to 4 dimensions in the former approach. A quaternary neuron is defined as foows [16]. The input signas, weights, threshods and output signas are a quaternions. The activity A n (anaogous to the rea activity in the standard neuron) of neuron n is defined to be: A n = m W nm S m + T n, (1) where S m is the quaternary input signa coming from the output of neuron m, W nm is the quaternary weight connecting the neurons m and n, and T n is the quaternary threshod vaue of the neuron n. To obtain the quaternary output signa, convert the activity vaue A n into its four parts as foows: A n = x 1 + x 2 i + x 3 j + x 4 k = x, (2) where i 2 = j 2 = k 2 = 1, ij = ji = k, jk = kj = i and ki = ik = j. The output signa 1 Q (x) is defined to be 1 Q (x) = 1 R (x 1 ) + 1 R (x 2 )i + 1 R (x 3 )j + 1 R (x 4 )k, (3) where 1 R is a rea-vaued step function defined on R, that is, 1 R (u) = 1 (if u 0), 1 R (u) = 0 (otherwise) for any u R ( R denotes the set of rea numbers). The mutipication W nm S m in Eq. (1) shoud be carefuy treated, because the equation W nm S m = S m W nm does not hod (the non-commutative property of quaternions on mutipication), which produces two kinds of quaternary neurons: one is caed norma quaternary neuron which cacuates A n = m W nms m + T n, the other is caed inverse quaternary neuron which cacuates A n = m S mw nm + T n. 3. Orthogonaity of Decision Boundary in the Quaternary Neuron We can find that the decision boundary of a quaternary neuron consists of four hyperpanes which intersect orthogonay each other, and divides a decision region into 2 4 (= 16) equa sections as that of a compex-vaued neuron case [8]. We show this property beow ony in the case of a norma quaternary neuron. An inverse quaternary neuron case can be easiy shown in a simiar manner. The net input U to a norma quaternary neuron with M inputs can be rewritten as: U = = = M w x + θ =1 M =1 (w (1) + w (2) i + w (3) j + w (4) k) (x (1) + x (2) i + x (3) j + x (4) k) +(θ (1) + θ (2) i + θ (3) j + θ (4) k) {[ t w (1) t w (2) t w (3) t w (4) ] t[ t x (1) t x (2) t x (3) t x (4) ] + θ (1)} { + [ t w (2) t w (1) t w (4) t w (3) ] t[ t x (1) t x (2) t x (3) t x (4) ] + θ (2)} i { + [ t w (3) t w (4) t w (1) t w (2) ] t[ t x (1) t x (2) t x (3) t x (4) ] + θ (3)} j { + [ t w (4) t w (3) t w (2) t w (1) ] t[ t x (1) t x (2) t x (3) t x (4) ] + θ (4)} k, (4) 34

3 Neura Information Processing - Letters and Reviews Vo. 5, No. 2, November 2004 where x (s) = t [x (s) 1 x (s) M ] and w(s) = t [w (s) 1 w (s) M ] (s = 1, 2, 3, 4). Thus, the decision boundary of the norma quaternary neuron with M inputs consists of the four equations obtained by etting each term of Eq.(4) be equa to zero. For exampe, Q(x 1,, x M ) = [ t w (1) t w (2) t w (3) t w (4) ] t[ t x (1) t x (2) t x (3) t x (4) ] + θ (1) = 0 (5) is the decision boundary for the rea part of an output of the norma quaternary neuron with M inputs. That is, input signas t [x 1 x M ] H M are cassified into two decision regions { t [x 1 x M ] H M Q(x 1,, x M ) 0} and { t [x 1 x M ] H M Q(x 1,, x M ) < 0} by the hyperpane given by Eq.(5) (H denotes the set of quaternions). We can find that the inner product of the two norma vectors of any two distinct decision boundaries is zero. For exampe, the inner product of the norma vectors of the decision boundaries for the j-part and k-part is cacuated as foows: [ t w (3) t w (4) t w (1) t w (2) ] t[ t w (4) t w (3) t w (2) t w (1) ] = 0. (6) Thus, the decision boundary of a norma quaternary neuron consists of four hyperpanes which intersect orthogonay each other. 4. Soving the 4-bit Parity Probem by a Singe Quaternary Neuron We wi find a soution to the 4-bit parity probem, using a singe norma quaternary neuron with the orthogona decision boundary with the highest generaization abiity. Minsky and Papert carified the imitations of a singe rea-vaued neuron: in a arge number of interesting cases, a singe rea-vaued neuron is incapabe of soving the probems [20]. The most difficut probem among them is the parity probem, in which the output required is 1 if the input pattern contains an odd number of 1s and 0 otherwise. Rumehart, Hinton and Wiiams showed that the 3-ayered rea-vaued neura network (i.e., with one hidden ayer) can sove the N-bit parity probem (N = 2,, 8) [21]. As described above, the N-bit parity probem cannot be soved with a singe rea-vaued neuron (N 2). Then, it wi be proved that the 4-bit parity probem can be soved by a singe norma quaternary neuron with the orthogona decision boundary (i.e., N = 4). Rumehart, Hinton and Wiiams showed that increasing the number of ayers made the computationa power of neura networks high. We wi show that extending the dimensionaity of neura networks to 4 dimensions originates the simiar effect on neura networks. In this connection, the excusive-or (XOR) probem and the detection of symmetry probem which cannot be soved with a singe rea-vaued neuron [20], can be soved with a singe compex-vaued neuron with the orthogona decision boundaries [9, 10]. The input-output mapping in the 4-bit parity probem is shown in Tabe 1(a). In order to sove the 4-bit parity probem with a norma quaternary neuron, the input-output mapping is encoded as shown in Tabe 1(b) where the outputs 0, j + k, i + k, i + j, 1 + k, 1 + j, 1 + i and 1 + i + j + k are interpreted to be 0, and k, j, i, 1, i + j + k, 1 + j + k, 1 + i + k and 1 + i + j are interpreted to be 1 of the origina 4-bit parity probem (Tabe 1(a)), respectivey. We use a singe norma quaternary neuron with ony one input and a weight w = w 1 + w 2 i + w 3 j + w 4 k H (we assume that it has no threshod parameters). The decision boundary of the norma quaternary neuron described above consists of the foowing four hyperpanes which intersect orthogonay each other: [w 1 w 2 w 3 w 4 ] t[x 1 x 2 x 3 x 4 ] = 0, (7) [w 2 w 1 w 4 w 3 ] t[x 1 x 2 x 3 x 4 ] = 0, (8) [w 3 w 4 w 1 w 2 ] t[x 1 x 2 x 3 x 4 ] = 0, (9) [w 4 w 3 w 2 w 1 ] t[x 1 x 2 x 3 x 4 ] = 0 (10) for any input signa x = x 1 + x 2 i + x 3 j + x 4 k H. Letting w 1 = 1 and w 2 = w 3 = w 4 = 0 (i.e., the weight w = 1), we can find that the norma quaternary neuron impements the input-output mapping shown in Tabe 1(b), the decision boundary of which consists of the four orthogona hyperpanes x s = 0 (1 s 4) (11) 35

4 4-bit Parity Probem with a Singe Quarternary Neuron T. Nitta and divides the input space (the decision region) into 2 4 equa sections, and has the highest generaization abiity for the 4-bit parity probem. There exist some neura network modes that can sove the N-bit parity probem. The comparison between our resut and the previous works for the 4-bit parity probem is shown in Tabe 2. The number of neurons, the number of parameters, and the number of ayers of the norma quaternary neuron are the east. In addition, as described above the generaization abiity is the highest. Thus, we concude that the norma quaternary neuron is the best totay. It shoud be emphasized here that the number of neurons needed for the norma quaternary neuron is ony one (i.e., a singe neuron). Tabe 1(a). The 4-bit Parity Probem. Input Output x 1 x 2 x 3 x 4 y Tabe 1(b). An Encoded 4-bit Parity Probem. Input Output x 1 x 2 x 3 x 4 y k j i j + k i + k i + j k j i i + j + k j + k i + k i + j i + j + k 36

5 Neura Information Processing - Letters and Reviews Vo. 5, No. 2, November 2004 Tabe 2. The Comparison between Our Resut and the Previous Works for the 4-bit Parity Probem. The number of ayers incudes an input ayer; it is 3 if the network has one hidden ayer. Direct ink means that there are at east one direct ink between the input ayer and the output ayer in the neura network with at east one hidden ayer. Note that the number of parameters in Aizenberg s work in the tabe is the estimated one by the author because Aizenberg et a. soved ony the 3, 8 and 9-bit parity probems with a singe compex-vaued neuron. The number The number The number Direct Activation of neurons of parameters of ayers ink function Ours No Step function Setiono 8 or more 16 or more 3 No Sigmoida [22] function Stork and No Consideraby Aen [23] compicated Minor [24] 7 or more 9 3 Yes Sigmoida function Lavretsky Yes Sigmoida [25] function Liu et a. 7 or more 14 or more 3 Yes Step [26] function Aizenberg et a No Somewhat [27] specia 5. Concusions We have found that a singe quaternary neuron has the orthogona decision boundary and can sove the 4-bit parity probem with the highest generaization abiity, which suggests that making the dimensionaity of neura networks high (from one to four dimensions with the agebraic structure in this etter) is a new directionaity for enhancing the abiity of neura networks, and that it is worth researching the neura networks with high dimensiona parameters. We wi appy the quaternary neuron to fieds suitabe for the orthogona decision boundary in a future. References [1] I. N. Aizenberg, N. N. Aizenberg & J. Vandewae, Muti-Vaued and Universa Binary Neurons, Kuwer Academic Pubishers, [2] T. Nitta & T. Furuya, A compex back-propagation earning, Transactions of Information Processing Society of Japan, Vo.32, No.10, pp , 1991 (in Japanese). [3] T. Nitta & H. D. Garis, A 3D vector version of the back-propagation agorithm, in: Proc. IEEE/INNS Internationa Joint Conference on Neura Networks, Vo.2, pp , [4] T. Nitta, An extension of the back-propagation agorithm to three dimensions by vector product, in: Proc. IEEE Internationa Conference on Toos with Artificia Inteigence, pp , [5] T. Nitta, An extension of the back-propagation agorithm to compex numbers, Neura Networks, Vo.10, pp , [6] B. Widrow, J. McCoo & M. Ba, The compex LMS agorithm, in: Proc. the IEEE, Vo.63, pp , [7] T. Nitta, Abiity of the 3D vector version of the back-propagation to earn 3D motion, in: Proc. INNS Word Congress on Neura Networks, Vo.3, pp ,

6 4-bit Parity Probem with a Singe Quarternary Neuron T. Nitta [8] T. Nitta, An anaysis on fundamenta structure of compex-vaued neuron, Neura Processing Letters, Vo.12, pp , [9] T. Nitta, Soving the XOR probem and the detection of symmetry using a singe compex-vaued neuron, Neura Networks, Vo.16, pp , [10] T. Nitta, Orthogonaity of decision boundaries in compex-vaued neura networks, Neura Computation, Vo.16, No.1, pp.73-97, [11] A. Watanabe, N. Yazawa, A. Miyauchi & M. Miyauchi, A method to interpret 3D motions using neura networks, IEICE Transactions on Fundamentas of Eectronics, Communications and Computer Sciences, Vo.E77-A(8), pp , [12] A. Hirose & S. Kawata, Recent progress in coherent ightwave neura systems, in: Proc. Internationa Conference on Neura Information Processing, Vo.3, pp , [13] Y. Kuroe, N. Hashimoto & T. Mori, On energy function for compex-vaued neura networks and its appications, in: Proc. Internationa Conference on Neura Information Processing, Vo.3, pp , [14] H. Aoki, A compex-vaued neuron to transform gray eve images to phase information, in: Proc. Internationa Conference on Neura Information Processing, Vo.3, pp , [15] P. Arena, L. Fortuna, L. Occhipinti & M. G. Xibiia, Neura networks for quaternion vaued function approximation, in: Proc. IEEE Int. Symp. on Circuit and Systems, Vo.6, pp , [16] T. Nitta, A quaternary version of the back-propagation agorithm, in: Proc. IEEE Internationa Conference on Neura Networks, Vo.5, pp , [17] H. -D. Ebbinghaus, et a. (Eds.), Zahen, Springer-Verag, Berin Heidederg, 1988 (in German). [18] P. Arena, L. Fortuna, G. Muscato & M. G. Xibiia, Neura Networks in Mutidimensiona Domains, Lecture Notes in Contro and Information Sciences, 234, Springer, [19] T. Isokawa, T. Kusakabe, N. Matsui & F. Peter, Quaternion neura network and its appication, in V. Paade, R.J. Howett and L.C. Jain (Eds.), LNAI 2774 (KES2003), Springer-Verag, Berin, pp , [20] M. I. Minsky & S. A. Papert, Perceptrons, MIT Press, Cambridge, [21] D. E. Rumehart, G. E. Hinton & R. J. Wiiams, Parae Distributed Processing, Vo.1, MIT Press, [22] R. Setiono, On the soution of the parity probem by a singe hidden ayer feedforward neura network, Neurocomputing, Vo.16, pp , [23] D. G. Stork & J. D. Aen, How to sove the N-bit parity probem with two hidden units, Neura Networks, Vo.5, pp , [24] J. M. Minor, Parity with two ayer feedforward nets, Neura Networks, Vo.6, pp , [25] E. Lavretsky, On the exact soution of the parity-n probem using ordered neura networks, Neura Networks, Vo.13, pp , [26] D. Liu, M.E. Hohi & S.H. Smith, N-bit parity neura networks: new soutions based on inear programming, Neurocomputing, Vo.48, pp , [27] N. N. Aizenberg, I. N. Aizenberg & G. A. Krivosheev, CNN based on universa binary neurons: earning agorithm with error-correction and appication to impusive-noise fitering on gray-scae images, in: Proc. IEEE Int. Workshop on Ceuar Neura Networks and their Appications, Sevie, Spain, pp ,

7 Neura Information Processing - Letters and Reviews Vo. 5, No. 2, November 2004 Tohru Nitta received his Ph.D. in information science from University of Tsukuba in 1995, and is currenty a Senior Research Scientist in Nationa Institute of Advanced Industria Science and Technoogy, Japan and is aso an Associate Professor at Department of Mathematics, Graduate Schoo of Science, Osaka University, Japan. His research interests incude theoretica issues in computationa modes such as artificia neura networks, especiay compex-vaued neura networks. (Home page: 39