N-bit Parity Neural Networks with minimum number of threshold neurons

Transcription

1 Open Eng. 2016; 6: Research Article Open Access Marat Z. Arslanov*, Zhazira E. Amirgalieva, and Chingiz A. Kenshimov N-bit Parity Neural Netorks ith minimum number of threshold neurons DOI /eng Received May 17, 2016; accepted June 24, 2016 Abstract: In this paper ordered neural netorks for the N- bit parity function containing log 2 (N + 1 threshold elements are constructed. The minimality of this netork is proved. The connection beteen minimum perceptrons of Gamb for the N-bit parity function and one combinatorial problems is established. Keyords: neural netorks, XOR problem, perceptrons of Gamb 1 Introduction In this article, developing [1], the ell knon XOR/parity problem is considered. The N-bit parity function is a mapping defined on 2 N distinct Boolean vectors that indicates hether the sum of the N components of a Boolean vector is odd or even. In [1-11] many solutions of this problem are suggested by various neural netorks. Some of these solutions apply complicated activation functions for using neurons. We think, that a choice of different activation functions for solving this problem is non constructive, because there eists the netork ith only one output neuron ith the non monotone activation function f ( = 0 if < cos(π if 0 2 N/ if > 2 N/2 + 1, here N is the number of incoming bits to the output neuron. The compleity of realizing a Boolean function by some neural netork is estimated by the number of neurons. As the threshold element [3] has the most simple (1 structure, the compleity of realizing a Boolean function can be estimated by the number of the threshold elements necessary for its realization. The threshold element ith n incoming bits 1, 2,... n, n eights 1, 2,..., n, and the threshold T is defined as follos [3]: its output is equal to 1 if n i i T and 0 otherise. By using the function θ( = 1 if 0 0 if < 0 the output of the threshold element can be ritten as θ( n i i T. The XOR problem as solved by threshold neural netorks in [4, 5, 8]. In [8] the XOR problem is solved by perceptrons of Gamb, i.e. to-layer neural netorks ith one output threshold neuron in the second layer, and m threshold neurons in the first layer (see Fig. 1. The thresholds of neurons are in their top part and an input vector = ( 1, 2..., N is presented by one circle. 1 i m T i s i T m s m v 1 v i v m T 0 s 0 *Corresponding Author: Marat Z. Arslanov: Institute of information and computation technologies, Almaty , Pushkin str.125, Kazakhstan, mzarslanov@hotmail.com Zhazira E. Amirgalieva: Institute of information and computation technologies, Almaty , Pushkin str.125, Kazakhstan Chingiz A. Kenshimov: Institute of information and computation technologies, Almaty , Pushkin str.125, Kazakhstan Figure 1: Perceptron of Gamb 2016 M.Z. Arslanov et al., published by De Gruyter Open. This ork is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License. N s i = θ i,j j T i, i = 1, 2,..., m (2

2 310 Marat Z. Arslanov, Zhazira E. Amirgalieva, and Chingiz A. Kenshimov ( m s 0 = θ v i s i T 0. (3 In [8] the perceptron of Gamb having N neurons in an intermediate layer for the N-bit parity function is presented. Moreover, in [8] net estimations are presented log 2 N m(n N (4 for the minimum number m of neurons in the first layer of the perceptron of Gamb for the N-bit parity function. There the problem about eacter estimations of m(n is posed. It is possible to sho, that for N = 2, 3, 4 m(n = N. Nevertheless, the problem about the minimum perceptrons of Gamb for the N-bit parity function remains open. In [4, 5] neural netorks hich generalize perceptrons of Gamb are considered. In these netorks an input vector can be directed into an output neuron (Fig. 2 1 i m 0 T i s i T m s m v 1 v i v m T 0 s 0 neurons of this netork is not proved. Thus for the architecture (5-(6 of neural netorks there is an open problem about the minimum number of neurons in an intermediate layer for the N-bit parity function. We consider solving the XOR problem by the ordered neural netork [6]. In the ordered neural netork neurons are numbered, and any neuron n 1 (including input units can be connected to any other neuron n 2 (including output units, as long as n 1 < n 2. Ordered netorks generalize feedforard neural netorks. In Fig. 3 the structure of the ordered neural netork is presented. v 1,l v 1,m v 1,m 1 v 2,m v 2,m 1 v l,m v l,m 1 v 2,l v 1,2 T 2... T l... T m 1 T m b 1 b 2 b l b m 1 b m 2 m 1 1 l m Figure 3: Ordered neural netork Analytical formulas are b 1 = θ 1,i i, (7 Figure 2: Generalized perceptron of Gamb N s i = θ i,j j T i, i = 1, 2,..., m (5 N m s 0 = θ 0,j j + v i s i T 0. (6 In [4, 5] such a neural netork for the N-bit parity function is constructed. It contains N/2 neurons in an interior layer, that tice less upper estimation (4 for the perceptron of Gamb. Hoever, the minimality on number of N l 1 b l = θ l,i i + v j,l b j T l, l = 2, 3,..., m, (8 here m is the number of neurons in the netork and the last neuron is an output of the netork. In [6] solving the XOR problem is offered for the N-bit input by the ordered neural netork ith N threshold neurons. In Section 2 the ordered neural netork ith log 2 (N + 1 threshold neurons is offered, that is less, than in [4 6]. It is also shon that there are no ordered neural netorks for the N-bit parity function ith smaller number of threshold neurons. In Section 3 some remarks about the minimality problem for the perceptron of Gamb and its connection ith one combinatorial problem are given.

3 N-bit Parity Neural Netorks ith minimum number of threshold neurons Ordered neural netorks for solving the XOR problem The construction of the ordered neural netork for solving the XOR problem is founded on representing the sum of all inputs in the binary notation. It is easy to see, that the value of the N-bit parity function is equal to the latest digit. The structure of such a neural netork for N = 15, containing 4 threshold elements, is in Fig. 4, here the neuron s threshold is in the top part of the neuron and for convenience all 15 bits are presented by the Boolean vector = ( 1,..., 15. It is easy to see, that b 1 b 2...b k is the binary notation of N i. Let c 1 c 2...c k be the binary notation of the number y = N i. Obviously c 1 = 1 if and only if y 2 k 1, i.e. b 1 = c 1. Thus, y 2 k 1 c 1 is less than 2 k 1 and it is possible to apply an induction to the binary notation for y 2 k 1 c 1. There is a natural question about the eistence of ordered neural netorks for the N-bit parity function ith smaller number of threshold neurons. The further part of this paper is devoted to the negative anser to this question. The idea of the decision is based on the geometrical fact about the structure of a boolean cube B N placed in R N. It turns out, that for every hyperplane in R N one of N/2 facets of B N lays on one side of a hyperplane s -4 2 s s 4 XOR Definition 1. The facet of the dimension n of a boolean cube B N = = ( 1, 2,..., N i 0, 1}, i = 1, 2,..., N} or n-facet is the set of all boolean vectors ith N n fied components. Lemma 1. For arbitrary threshold function b( of N boolean variables = ( 1, 2..., N there eists the N/2 facet of a boolean cube B N on hich this function is constant. Figure 4: Ordered neural netork for 15-bit parity function b 1 = θ i 7.5, b 2 = θ b 3 = θ b 4 = θ i 8b 1 3.5, i 8b 1 4b 2 1.5, i 8b 1 4b 2 2b (9 It is easy to see, that b 1 b 2 b 3 is the binary notation of 7 i. The simple generalization of this eample gives the folloing neural netork, hich calculates the parity function of N input bits = ( 1, 2,..., N, and contains k = log 2 (N + 1 threshold elements: b 1 = θ i 2 k , (10 N l 1 b l = θ i 2 k j b j 2 k l + 0.5, l = 2, 3,..., k. (11 Proof. Let the threshold function 1, if n b( = i i T, 0, if n i i < T (12 be given. If some eights i are negative, then by the standard substitution i = 1 i for all such i and i = i for others e receive the threshold function c( 1, if n = i i T + i J i 0, if n i i < T + (13 i J i, here J = i i < 0}, such that b( = c(. So, ithout loss of generality e can consider, that all i 0, i = 1, 2,..., N. Let N/2 i T. (14 It is easy to see, that on the folloing facet of the dimension N N/2 N/2 F 1 = B N 1 = 2 =... = N/2 = 1} (15 the function b( is equal to 1. If, on the other hand, N/2 i < T, (16

4 312 Marat Z. Arslanov, Zhazira E. Amirgalieva, and Chingiz A. Kenshimov then on the N/2 -facet F 2 = B N N/2 +1 = N/2 +2 =... = N = 0} (17 the function b( is equal to 0. The lemma is proved. Note that it is impossible to increase the dimension of the facet on hich the threshold function is constant. An eample is the threshold function b( = θ( N i N/ It is easy to sho, that on any facet of the dimension greater than N/2 this function is not constant. No e can prove the minimality of the ordered neural netork (10-(11 on number of threshold neurons. Theorem 1. Let N = 2 m. Then the N-bit parity function and its negation cannot be presented by an ordered neural netork ith m threshold neurons. Proof. We prove the theorem by induction. For m = 1 the theorem asserts the ell knon fact of threshold logic [3, 8] that the function of eclusive-or 1 2 and its negation are not threshold functions. Let the theorem be valid for m = k. On the other hand, let for m = k + 1 and N = 2 k+1 the N-bit parity function be presented by the ordered neural netork ith k + 1 threshold neurons, in the analytical form b 1 = θ 1,i i, (18 N l 1 b l = θ l,i i + v j,l b j T l, l = 2, 3,..., k + 1. (19 The contradiction turns out as follos. Consider the first neuron of the given ordered neural netork. Its output b 1 depends only on N input bits, i.e. it is the boolean threshold function b 1 ( of the boolean vector = ( 1, 2..., N. According to Lemma 1 there eists the facet of the dimension N/2 = 2 k on hich the given function is constant. By the permutation of variables one can achieve that this facet is defined by fiing the variables 2 k +1, 2 k +2,..., 2k+1 or this facet is defined by equations F = B N 2 k +1 = α 1, 2 k +2 = α 2,..., 2 k+1 = α 2 k }. (20 If to fi variables 2 k +1, 2 k +2,..., 2 in such a ay, k+1 the given ordered neural netork can be considered as the realization of the boolean function of N/2 variables 1, 2,..., 2 k. Clearly, an output of this netork is the boolean function 2 k i α, here α = 2 k α i. Thus if α = 0 e have ordered neural netork for the parity function of 1, 2..., 2 k, or if α = 1 for its negation. It is easy to see, that e can reorganize this netork by omitting the first neuron as it is constant and decreasing the thresholds T 2, T 3,..., T k+1 of other neurons by multiplying this constant by appropriate eights. So, e receive the netork ith k threshold neurons hich calculates, contrary to the assumption of the induction, 2 k -bit parity function or its negation by the formulas: b 2 = θ 2,i i (T 2 v 1,2 b 1, (21 2 k l 1 b l =θ l,i i + v j,l b j (T l v 1,l b 1, (22 j=2 l = 3, 4,..., k + 1, in hich the first neuron has an inde 2 etc. This contradiction proves the theorem for the N-bit parity function. Analogously the theorem is proved for the negation of the N-bit parity function. From the proved theorem the minimality on number of threshold neurons of the designed ordered neural netork (10-(11 for the N-bit parity function follos. Indeed, if for some N there is an ordered neural netork for the N-bit parity function ith smaller than k = log 2 (N + 1 number of threshold neurons, then, as N 2 k 1, from here follos an eistence of the ordered neural netork ith k 1 threshold neurons for the 2 k 1 -bit parity function. This contradicts ith Theorem 1. 3 Some notes about minimum perceptrons of Gamb Let s consider the problem of the perceptron of Gamb N s i = θ i,j j T i, i = 1, 2,..., m (23 ( m s 0 = θ v i s i T 0 (24 for the N-bit parity function ith minimum number of intermediate threshold neurons, hich e denote by m(n. The upper estimation in (4 is obtained by a construction of the concrete perceptron of Gamb. An eample is the fol-

5 N-bit Parity Neural Netorks ith minimum number of threshold neurons 313 loing perceptron: N s i = θ ( 1 i+1 j (i 0.5( 1 i+1, i = 1, 2,..., N (25 s 0 = θ s i N/ (26 The loer estimation in (4 is accompanied by ords it s not hard to sho, that log 2 N m(n. Hoever, the loer estimation easily follos from Theorem 1. The bibliographic searching has shon, that other estimations for m(n are absent. We shall connect this problem ith the more common combinatorial problem. Let s consider the problem of the dissection of all edges of a Boolean cube B N located in R N by the minimum number of hyperplanes, hich do not pass through vertices of B N. Denote this minimum number of hyperplanes by m 1 (N. Net lemma is valid. Lemma 2. The net inequality is true: m(n m 1 (N. Proof. Let (23-(24 be the minimum perceptron of Gamb for the N-bit parity function. The thresholds in it can be taken such, that hyperplanes N i,j j = T i, i = 1, 2,..., m (27 do not pass through vertices of B N. We sho, that a set of hyperplanes (27 dissects all edges of B N. If it is not so, there is an edge in B N ((α 1, α 2,..., α i, α i+1,..., α N, (α 1, α 2,..., 1 α i, α i+1,..., α N hich is not dissected by all hyperplanes from (27. Then both boundary points of this edge are on one side from each of hyperplanes (27. This means, that all neurons of the perceptron (23-(24 are constant on this edge. From (24 it follos that an output neuron s 0 is constant on this edge. But the parity function is not constant on any edge. This inconsistency proves the lemma. It is easy to see, that for N = 2, 3 one have m 1 (N = N, as it is impossible by one line dissect all 4 edges of a guadrate and by to planes dissect all 12 edges of a 3-dimensional cube. From here follos, that m(n = N ith N = 2, 3. It is possible to sho, that m(4 = 4. In connection ith above the folloing problems are natural. 1. Whether alays m(n = N? 2. Are there N, for hich m(n > m 1 (N? Similar is the problem of the minimality of neural netorks ith N/2 intermediate threshold neurons constructed in [4, 5]. Hoever, as it is mentioned in [8], these problems have not simple solution. 4 Conclusions We have offered a simple ordered neural netork for the N- bit parity function, hich contains log 2 (N + 1 threshold elements. For a comparison, the ordered neural netork in [6] contains N threshold gates, and in [4, 5] N/2 + 1 threshold gates. The problems of proving the minimality (on number of threshold neurons for various architecture of neural netorks are still difficult. We have proved the minimality of the offered construction for the N-bit parity function in the class of ordered neural netorks. Some considerations of analogous problem for perceptron of Gamb are undertaken and open problems are formulated. References [1] Arslanov M.Z., Ashigaliev D.U., Ismail E.E., N-bit parity ordered netorks, Neurocomputing, 2002, 48, [2] Bron D.A., N-bit parity netorks, Neural Netorks, 1993, 6, [3] Dertouzos M.L., Threshold Logic, The M.I.T. Press, Cambridge, Massachusets, [4] Hohil M.E., Liu D., Smith S.H., Solving the N-bit Parity Problem Using Neural Netorks, Neural Netorks, 1999, 12, [5] Hohil M.E., Liu D., Smith S.H., N-bit parity netorks: a simple solution, Proceedings of the 32nd Annual Conference on Information Sciences and Systems, Princeton University, Princeton, NJ, 1998, [6] Lavretsky E., On the Eact Solution of the Parity - N Problem using ordered Neural Netorks, Neural Netorks, 2000, 13, [7] Minor J.M., Parity ith to layer feedforard nets, Neural Netorks, 1993, 6, [8] Minsky M., Papert P., Perseptrons, The M.I.T. Press, Cambridge, Massachusets, [9] Setiono R., On the Solution of the Parity Problem by a Single Hidden layer Feedforard Neural Netork, Neurocomputing, 1997, 16, [10] Stork D.G., N-Bit Parity Netorks: A Reply to Bron and Korn, Neural Netorks, 1993,6, 609. [11] Stork D.G., Allen J.D., Ho to solve the N-bit parity problem ith to hidden units, Neural Netork992, 5,