Sigmoid functions for the smooth approximation to x

Transcription

1 Sigmoid functions for the smooth approimation to Yogesh J. Bagul,, Christophe Chesneau,, Department of Mathematics, K. K. M. College Manwath, Dist : Parbhani(M.S.) , India. yjbagul@gmail.com LMNO, University of Caen Normandie, France christophe.chesneau@unicaen.fr Abstract. We present smooth approimations to using sigmoid functions. In particular erf(/) is proved to be better smooth approimation for than tanh(/) and + with respect to accuracy. To accomplish our goal we also provide sharp hyperbolic bounds for error function. Introduction An S - shaped function which usually monotonically increases on R (the set of all real numbers) and has finite limits as ± is known as sigmoid function. Sigmoid functions have many applications including the one in artificial neural networks. Rigorously, a sigmoid function is bounded and differentiable real function that is defined for all real input values and has a non-negative derivative at each point 6]. It has bell shaped first derivative. A sigmoid function is constrained by two parallel and horizontal asymptotes. Some eamples of sigmoid functions include logistic function, i.e. /( + e ), tanh(), tan, Gudermannian function, i.e. gd(), error function, i.e. erf(), ( + ) / etc. Some of them are described below. The Gudermannian function is defined as follows: Alternatively, gd() = 0 cosh(t) dt. gd() = sin (tanh()) = tan (sinh()). Corresponding author 00 Mathematics Subject Classification: 6A99, 6E99, 33E99, 4A30. Keywords: Sigmoid functions; error function; smooth approimation; Hyperbolic tangent. 09 by the author(s). Distributed under a Creative Commons CC BY license.

2 The error function or Gaussian error function is defined as follows: erf() = e t dt. π The Gudermannian and error functions are special functions and they have many applications in mathematics and applied sciences. All the above mentioned sigmoid functions are differentiable and their limits as ± are listed below: lim + e ] 0 =, lim + + e lim tanh() =, lim tanh() = + lim tan () = π, lim + tan () = π lim gd() = π, lim gd() = π + lim erf() =, lim erf() = + lim + =, lim + + =. ] = Due to these properties it is easy to see that the functions tanh(/), /( + e / ) / ], (/π) tan (/), (/π) gd(/), erf(/) and ( + ) / as 0 can be used as smooth approimations for. In 3], + is proved to be computationally efficient smooth approimation of, since it involves less number of algebraic operations. In spite of being this, as far as accuracy is concerned some of the above mentioned functions are better transcendental approimations to. In ] tanh(/) was proposed by first author and it is recently proved ] that this approimation is better than + in terms of accuracy by Yogesh J. Bagul and Bhavna K. Khairnar. One of the users of Mathematics Stack Echange 7] suggested erf(/) as a smooth approimation to. However that user did not give the logical proof or did not compare this approimation with eisting ones. In fact, it is better than + or + in terms of accuracy; but it is not proved in 7]. To prove this fact is the main goal of this paper. We shall prove this thing by showing erf(/) to be better than tanh(/). We avoid logical proofs for other approimations presented above, since they are

3 not as good as tanh(/) or erf(/) for accuracy which can be seen in the figures given at the end of this article. The rest of the paper is organized in the following manner. Section presents the main results, with proofs. Two tight approimations are then compared numerically and graphically in Section 3. A conclusion is given in Section 4. Main Results with Proofs We need the following lemmas to prove our main result. Lemma. (l Hôpital s Rule of Monotonicity 4]): Let f, g : c, d] R be two continuous functions which are differentiable on (c, d) and g 0 in (c, d). If f ()/g () is increasing (or decreasing) on (c, d), then the functions (f() f(c))/(g() g(c)) and (f() f(d))/(g() g(d)) are also increasing (or decreasing) on (c, d). If f ()/g () is strictly monotone, then the monotonicity in the conclusion is also strict. Lemma. For R, the following inequality holds: Proof: Suppose that By differentiation we get e e. h() = e. h () = e ( ). This implies = 0, ± are the critical points for h(). Again differentiation gives h () = e ( ) 4 e ( ) Hence, h (0) =, h ( ) = 4 e, h () = 4 e. By second derivative test, h() has minima at = 0 and maima at = ±. Therefore 0 is the minimum value and /e is the maimum value of h(), ending the proof of Lemma. Lemma 3. For R {0}, one has with α = /(e π) erf() + α 3 >, (.)

4 Proof: We consider two cases depending on the sign of as follows: Case(): For > 0, let us consider the function f() = erf() + α which on differentiation gives f () = e α π = e ]. π e By Lemma, f () 0 and hence f() is decreasing on (0, + ). So, for any > 0, f() > f(+ ), i.e. erf() + α >. Case(): For < 0 let us consider the function g() = erf() + α/ +. As in Case(), g () 0 and is decreasing in (, 0). Hence, for any < 0, g() < g( + ). So we get erf() + α <, which completes the proof of Lemma 3. Theorem. Let > 0 and α = /(e π) For R, the approimation F () = erf(/) to satisfies and Proof: We have F () = F () = e + erf π e + π F () F () < α. (.) ( ) = e π + F (). For = 0 the inequality (.) is obvious. For 0, it follows from Lemma 3 that ( ) ( F () = erf = erf ) ( ) ] = erf < α = α. 4

5 The proof of Theorem is completed. In the following theorem we give sharp bounds for error function erf() implying that the present approimation to is better than tanh(/). Theorem. For > 0, it is true that Proof: Consider the function tanh() < erf() < π tanh(). (.3) G() = erf() tanh() = G () G (), where G () = erf() and G () = tanh() with G (0) = G (0) = 0. On differentiating we get G () G () = e cosh () = λ(), π π where λ() = e cosh (), derivative of which is given by λ () = e cosh() sinh() cosh()]. Since sinh()/ < cosh() (see, for instance, 5]), we have λ () < 0 and hence λ() is decreasing in (0, + ). By Lemma, G() is also decreasing in (0, + ). So, for > 0, G(0 + ) > G() > G(+ ). It is easy to evaluate G(0 + ) = / π by l Hospital s rule and G(+ ) =. This ends the proof of Theorem. 3 Comparison between two approimations By virtue of Theorem, for all R and > 0, we get the following chain of inequalities: ( ) ( ) tanh < erf < < +. (3.) Again in ], it is proved that tanh(/) is better than + or + with respect to accuracy. Consequently, erf(/) is better than + 5

6 or + in the same regard. Numerical and graphical studies support the theory. In Table, we compare numerically some of these approimations by investigating global L error which is given by e(f) = + f()] d, where f() denotes an approimation to. With this criterion, a lower e(f) value indicates a better approimation. Table indicates that erf(/) is the best of the considered approimation (for = 0. and = 0.0, but other values can be considered for, with the same conclusion). Table : Global L errors e(f) for the functions f(). f() + e / ] π gd ( = 0. ) tanh ( ) erf ( ) e(f) f() + e / ] π gd ( = 0.0 ) tanh ( ) erf ( ) e(f) By considering the setting of Table, Figures and also support our theoretical findings. 6

7 = erf( ) tanh( ) ( π)gd( ) ( ( + ep( ) ) Figure : Graphs of the functions in Table with = 0. for ( 0., 0.). = erf( ) tanh( ) ( π)gd( ) ( ( + ep( ) ) Figure : Graphs of the functions in Table with = 0.0 for ( 0.04, 0.04). 7

8 4 Conclusion Sigmoid functions can be used for smooth approimation of. In particular erf(/) is proved to be better smooth approimation for with respect to accuracy. References ] Y. J. Bagul, A smooth transcendental approimation to, International Journal of Mathematical Sciences and Engineering Applications(IJMSEA), Volume, Number (August 07), pp. 3-7, 07. ] Y. J. Bagul and Bhavna K. Khairnar, A note on smooth transcendental approimation to, Preprints 09, doi: /preprints v. Online]. URL: 3] C. Ramirez, R. Sanchez, V. Kreinovich and M. Argaéz, + is the most computationally efficient smooth approimation to : a Proof, Journal of Uncertain Systems, Volume 8, Number 3, pp. 05-0, 04. 4] G. D. Anderson, M. K. Vamanamurthy and M. Vuorinen, Conformal Invarients, Inequalities and Quasiconformal maps, John Wiley and Sons, New York, ] R. Klén, M. Visuri and M. Vuorinen, On Jordan Type Inequalities for Hyperbolic Functions, J. Inequal. and Appl. Volume (00), 4 pages, Art. ID 36548, 00. doi: 0.55/00/ Online]. URL: 6] J. Han and C. Moraga, The influence of the sigmoid function parameters on the speed of backpropagation learning, In: Mira José, Sandoval Francisco(Eds), From Natural to Artificial Neural computation, IWANN 995, Lecture Note in Computer Science, Volume 930, pp. 95-0, Springer, Berlin Heidelberg. 7] Ravi ( Smooth approimation of absolute value inequalities, URL (version: ): 8