On a constructive proof of Kolmogorov s superposition theorem

Transcription

1 On a constructive proof of Kolmogorov s superposition theorem Jürgen Braun, Michael Griebel Abstract Kolmogorov showed in [4] that any multivariate continuous function can be represented as a superposition of one dimensional functions, ie ( n n ) f(x,,x n ) = Φ q ψ q,p (x p ) q=0 The proof of this fact, however, was not constructive and it was not clear how to choose the inner and outer functions Φ q and ψ q,p respectively Sprecher gave in [7,8] a constructive proof of Kolmogorov s superposition theorem in form of a convergent algorithm which defines the inner functions explicitly via one inner function ψ by ψ p,q := λ p ψ(x p + qa) with appropriate values λ p, a R Basic features of this function as monotonicity and continuity were supposed to be true, but were not explicitly proved and turned out to be not valid Köppen suggested in [6] a corrected definition of the inner function ψ and claimed, without proof, its continuity and monotonicity In this paper we now show that these properties indeed hold for Köppen s ψ and present a correct constructive proof of Kolmogorov s superposition theorem for continuous inner functions ψ similar to Sprecher s approach Keywords: Kolmogorov s superposition theorem, superposition of functions, representation of functions AMS-Classification: 6B40 p= Introduction The description of multivariate continuous functions as a superposition of a number of continuous functions [3,4] is closely related to Hilbert s thirteenth problem [0] from his Paris lecture in 900 In 957 the Russian mathematician Kolmogorov showed the remarkable fact that any continuous function f of many variables can be represented as a composition of addition and some functions of one variable [4] The original version of this theorem can be expressed as follows: Theorem Let f : I n := [0,] n R be an arbitrary multivariate continuous function Then it has the representation n n f(x,,x n ) = ψ q,p (x p ), () q=0 with continuous one dimensional inner and outer functions Φ q and ψ q,p All these functions Φ q, ψ q,p are defined on the real line The inner functions ψ q,p are independent of the function f Kolmogorov s student Arnold also made contributions [ 3] in this context that appeared at nearly the same time Several improvements of Kolmogorov s original version were published Φ q p=

2 in the following years Lorentz showed that the outer functions Φ q can be chosen to be the same [9,0] while Sprecher proved that the inner functions ψ q,p can be replaced by λ p ψ q with appropriate constants λ p [5,6] A proof of Lorentz s version with one outer function that is based on the Baire category theorem was given by Hedberg [9] and Kahane [3] A further improvement was made by Friedman [5], who showed that the inner functions can be chosen to be Lipschitz continuous A geometric interpretation of the theorem is that the n+ inner sums n p= ψ q,p map the unit cube I n homeomorphically onto a compact set Γ R n+ Ostrand [3] and Tikhomirov [5] extended Kolmogorov s theorem to arbitrary n dimensional metric compact sets The fact that any compact set K R n can be homeomorphically embedded into R n+ was already known from the Menger Nöbeling theorem [] More recently, Kolmogorov s superposition theorem found attention in neural network computation by Hecht Nielsen s interpretation as a feed-forward network with an input layer, one hidden layer and an output layer [7, 8, 5] However, the inner functions in all these versions of Kolmogorov s theorem are highly non-smooth Also, the outer functions depend on the specific function f and hence are not representable in a parameterized form Moreover, all one dimensional functions are the limits or sums of some infinite series of functions, which cannot be computed practically Therefore Girosi and Poggio [6] criticized that such an approach is not applicable in neurocomputing The original proof of Kolmogorov s theorem is not constructive, ie one can show the existence of a representation () but it cannot be used in an algorithm for numerical calculations Kurkova [7,8] partly eliminated these difficulties by substituting the exact representation in () with an approximation of the function f She replaced the one variable functions with finite linear combinations of affine transformations of a single arbitrary sigmoidal function ψ Her direct approach also enabled an estimation of the number of hidden units (neurons) as a function of the desired accuracy and the modulus of continuity of f being approximated In [] a constructive algorithm is proposed that approximates a function f to any desired accuracy with one single design, which means that no additional neurons have to be added There, also a short overview over the history of Kolmogorov s superposition theorem in neural network computing is given Other approximative, but constructive approaches to function approximation by generalizations of Kolmogorov s superposition theorem can be found in [4,,] Recently, Sprecher derived in [7, 8] a numerical algorithm for the implementation of both, internal and external univariate functions, which promises to constructively prove Kolmogorov s superposition theorem In these articles, the inner function ψ is explicitly defined as an extension of a function which is defined on a dense subset of the real line Throughout his proof, Sprecher relies on continuity and monotonicity of the resulting ψ It can however be shown that his ψ does not possess these important properties This was already observed by Köppen in [6] where a modified inner function ψ was suggested Köppen claims, but does not prove the continuity of his ψ and merely comments on the termination of the recursion which defines his corrected function ψ In this article we close these gaps First, since the recursion is defined on a dense subset of R, it is necessary to show the existence of an expansion of Köppen s ψ to the real line We give this existence proof Moreover it is also a priori not clear that Köppen s ψ possesses continuity and monotonicity, which are necessary to proof the convergence of Sprecher s algorithm and therefore Kolmogorov s superposition theorem We provide these properties Altogether, we thus derive a complete constructive proof of Kolmogorov s superposition theorem along the lines of Sprecher based on Köppen s ψ The remainder of this article is organized as follows: As starting point, we specify Sprecher s version of Kolmogorov s superposition theorem in section Then, in section 3 we briefly

3 repeat the definitions of the original inner function ψ and the constructive algorithm that was developed by Sprecher in [7,8] The convergence of this algorithm would prove Kolmogorov s superposition theorem First, we observe that Sprecher s ψ is neither continuous nor monotone increasing on the whole interval [0,] We then show that Köppen s ψ indeed exists, ie it is well defined and has the necessary continuity and monotonicity properties Endowed with this knowledge, we then follow Sprecher s lead and prove the convergence of the algorithm, where the original inner function is replaced by the corrected one This finally gives a constructive proof of Kolmogorov s superposition theorem Definitions and algorithm A version of Kolmogorov s superposition theorem Many different variants of Kolmogorov s superposition theorem () were developed since the first publication of this remarkable result in 957 Some improvements can be found eg in [0,5] In [5] it was shown that the inner functions ψ q,p can be chosen to be Lipschitz continuous with exponent one Another variant with only one outer function and n + inner functions was derived in [0] A version of Kolmogorov s superposition theorem recently developed by Sprecher in [5] reads as follows: Theorem Let n, m n and γ m + be given integers and let x = (x,,x n ) and x q = (x + qa,,x n + qa), where a = [γ(γ )] Then, for any arbitrary continuous function f : R n R, there exist m + continuous functions Φ q : R R, q = 0, m, such that f(x) = m Φ q ξ(x q ), with ξ(x q ) = q=0 n α p ψ(x p + qa), () p= α =, α p = r= γ (p )β(r) for p > and β(r) = (n r )/(n ) This version of Kolmogorov s superposition theorem involves m one dimensional outer functions Φ q and one single inner function ψ The definition of ψ will be discussed in detail in the following For a fixed basis γ > we define for any k N the set of terminating rational numbers { } k D k = D k (γ) := d k Q : d k = i r γ r,i r {0,,γ } () r= Then the set D := k N D k (3) is dense in [0,] In [8] Sprecher formulated an algorithm, whose convergence proves the above theorem constructively In this algorithm, the inner function ψ was defined point-wise on the set D Further investigations on this function were made in [7] However, to make this proof work, two fundamental properties of ψ namely continuity and monotonicity are needed Unfortunately, the inner function ψ in [7,8] is neither continuous nor monotone In the following, we repeat the definition of ψ here and show that it indeed does not define a continuous and monotone increasing function 3

4 k = 5 06 k = Figure : Graph of the function ψ from (4) on the interval [0,] (left) and a zoom into a smaller interval (right), computed for the values of the set D 5, γ = 0 and n = One can clearly see the non monotonicity and discontinuity near the value x = 059 (right) Let i := 0 and for r let i r := { 0 when i r = 0,,γ, when i r = γ Furthermore, we define [i ] := 0 and, for r, { 0 when i r = 0,,γ 3, [i r ] := when i r = γ,γ, and m r := i r The function ψ is then defined on D k by ĩ r := i r (γ ) i r, ( r ( [i s ] [i r ]) ) s= ψ(d k ) := k ĩ r mr γ β(r mr) (4) r= The graph of the function ψ is depicted in figure for k = 5, ie it was calculated with the definition (4) on the set of rational numbers D k The function ψ from (4) has an extension to [0,], which also will be denoted by ψ if the meaning is clear from the contents The following calculation shows directly that this function is not continuous in contrast to the claim in [7] With the choice γ = 0 and n = one gets with the definition (4) the function values ψ(058999) = and ψ(059) = 055 (5) 4

5 This counter example shows that the function ψ is not monotone increasing We furthermore can see from the additive structure of ψ in (4) that ψ(058999) < ψ(x) for all x ( ,059 ) (6) This shows that the function ψ is also not continuous Remark Discontinuities of ψ arise for all values x = 0i 9, i = 0,,9 Among other things, the convergence proof in [7,8] is based on continuity and monotonicity of ψ As the inner function defined by Sprecher does not provide these properties the convergence proof also becomes invalid unless the definition of ψ is properly modified To this end, Köppen suggested in [6] a corrected version of the inner function and stated its continuity This definition of ψ is also restricted to the dense set of terminating rational numbers D Köppen defines recursively d k for k =, ψ k (d k ) = ψ k (d k i k γ k) + i k for k > and i γ β(k) k < γ, ( ) (7) ψ k (d k ) + ψ γ k k (d k + ) for k > and i γ k k = γ and claimed that this recursion terminates He assumed that there exists an extension from the dense set D to the real line as in Sprecher s construction and that this extended ψ is monotone increasing and continuous but did not give a proof for it In the following, we provide such a proof We first consider the existence of an extension and begin with the remark that every real number x R has a representation x = r= i r γ r = lim k For such a value x, we define the inner function and show the existence of this limit k r= ψ(x) := lim k ψ k(d k ) = lim k ψ k i r γ r = lim k d k ( k r= ) i r γ r For the following calculations it is advantageous to have an explicit representation of (7) as a sum To this end, we need some further definitions The values of ψ k j at the points d k j and d k j + γ k j are denoted as ψ k j := ψ k j (d k j ) and ψ + k j := ψ k j(d k j + ) γk j Then, the recursion (7) takes for k j > the form (8) and ψ k j = { ik j + ψ γ β(k j) k j for i k j < γ, γ + γ β(k j) ψ k j + ψ+ k j for i k j = γ i k j + + ψ ψ + γ β(k j) γ β(k j) k j for i k j < γ, k j = i k j + γ β(k j) ψ k j + ψ+ k j for i k j = γ, ψ + k j for i k j = γ (9) (0) 5

6 With the definition of the values 0 for i k j+ < γ, s j := for i k j+ = γ, for i k j+ = γ and s j := { 0 for i k j+ < γ, for i k j+ = γ, () the representations (9) and (0) can be brought into the more compact form and ψ k j = ( s j+ )ψ k j + s j+ ψ + k j + ( s j+) i k j γ β(k j) + s γ j+, () γβ(k j) ψ + k j = ( s j+ )ψ k j + s j+ ψ + k j + ( s j+) [ ik j γ β(k j) + ( s ] j+) γ β(k j), (3) respectively Now, for a compact representation of the continuation of this recursion, we define the values α 0 :=, α + 0 := 0, α := s, α + := s, and α j+ = α j ( s j+ ) + α + j ( s j+), α + j+ = α j s j+ + α + j s j+, (4) for j =,,k By induction we can directly deduce from (4) and () the useful properties α j + α + j = and α j,α + j > 0 (5) With these definitions, the ξ th step of the recursion can be written as the sum ψ k = ξ [ α j ( s j+ ) i k j γ β(k j) + s γ ] j+ γ β(k j) j=0 [ ( + α + ik j j ( s j+ ) γ β(k j) + ( s )] j+) γ β(k j) + α ξ ψ k ξ + α + ξ ψ+ k ξ (6) Choosing ξ = k we finally obtain a point-wise representation of the function ψ k as the direct sum ψ k (d k ) = k [ α j ( s j+ ) i k j γ β(k j) + s γ ] j+ γ β(k j) j=0 [ ( + α + ik j j ( s j+ ) γ β(k j) + ( s )] j+) γ β(k j) i + α k γ + i + α+ k γ (7) Now we have to show the existence of the limit (8) To this end, we consider the behavior of the function values ψ k and ψ + k as k tends to infinity: Lemma 3 For growing values of k one has ψ + k = ψ k + O( k ) 6

7 Proof With (3), (), the fact that γ β(j) = γ β(j ) γ nj and γ n >, we have ψ + k ψ k ψ+ k ψ k + γ = and the assertion is proved ( ) k ψ + ψ + ( ) k ψ + ψ + ( ) k [ γ γ β(k) ] (γ )γn + γ n ( ) k (γ ) ( ) k (γ ) k j γ β(j) j= j=0 ( ) j γ n If we now apply this result to arbitrary values k and k, we can show the following lemma: Lemma 4 The sequence ψ k is a Cauchy sequence Proof For k, k N and without loss of generality k > k, we set ξ := k k in (6) Then, we obtain by (5) and with lemma 3 the following estimate: ψ k ψ k αk k ψ k + α + k k ψ + k ψ k + (γ ) αk k ψ k + α + k k ψ + k ψ k + (γ ) = ) α k k ψ k + α + k k (ψ k + O( k ) k j=k + k j=k + γ β(j) ( ) j γ n ψ k + γn (γ ) γ n ( ( ) k ( ) ) k γ n γ n O( k ) + γn (γ ) γ n ( ( ) k ( ) ) k γ n γ n The right hand side tends to 0 when k,k The real numbers R are complete and we therefore can infer the existence of a function value for all x [0,] Thus the function ψ from (8) is well defined It remains to show that this ψ is continuous and monotone increasing This will be the topic of the following subsections The continuity of ψ We now show the continuity of the inner function ψ To this end we first recall some properties of the representations of real numbers Let x := r= i r γ r and x 0 := r= i 0,r γ r 7

8 be the representation of the values x and x 0 in the basis γ, respectively Let x 0 (0,) be given and i 0,r δ(k 0 ) := min γ r, γ k i 0,r 0 γ r r=k 0 + For any x ( x 0 δ(k 0 ),x 0 + δ(k 0 ) ) it follows that r=k 0 + i r = i 0,r for r =,,k 0 (8) Special attention has to be paid to the values x 0 = 0 and x 0 = In both cases, we can choose δ(k 0 ) = γ k 0 Then (8) holds for all x [ 0,δ(k 0 ) ) if x 0 = 0 and all x ( δ(k 0 ), ] if x 0 = The three different cases are depicted in figure Altogether we thus can find for any given arbitrary x 0 [0,] a δ neighborhood U := ( x 0 δ(k 0 ),x 0 + δ(k 0 ) ) [0,] in which (8) holds To show the continuity of the inner function ψ in x 0, we now choose this neighborhood and see from (7) for x,x 0 U: ψ(x) ψ(x 0 ) = lim ψ(d k) ψ(d 0,k ) k k k 0 = lim k j=0 + α + j k k 0 j=0 k k 0 lim k j=0 + lim k α j [ ( s j+ ) i k j [ ( ik j ( s j+ ) α 0,j [ ( s 0,j+ ) i 0,k j + α + 0,j α j k k 0 j=0 4γ n (γ ) lim k γ n γ β(k j) + s γ ] j+ γ β(k j) γ β(k j) + ( s )] j+) γ β(k j) γ β(k j) + s γ ] 0,j+ γ β(k j) [ ( i0,k j ( s 0,j+ ) γ β(k j) + ( s 0,j+) [ ( s j+ ) i k j γ β(k j) + s γ ] j+ γ β(k j) [ ( ik j ( s j+ ) γ β(k j) + ( s j+) )] γ β(k j) + )] α+ j γ β(k j) [ α 0,j ( s 0,j+ ) i 0,k j γ β(k j) + s γ ] 0,j+ γ β(k j) [ ( + i0,k j α+ 0,j ( s 0,j+ ) γ β(k j) + ( s )] 0,j+) γ β(k j) ( ) k ( ) k0 ( ) γ n γ n = 4γn (γ ) k0 γ n γ n (9) Note that the estimation of the last two sums was derived similar to that of the proof of lemma 4 In conclusion we can find for any given ε > 0 a k 0 N and thus a δ(k 0 ) > 0 such that ψ(x) ψ(x 0 ) < ε whenever x,x 0 U = ( x 0 δ(k 0 ),x 0 + δ(k 0 ) ) [0,] This is just the definition of continuity of ψ in x 0 (0,) Since the interval U is only open to the right if 8

9 δ(k 0 ) i 0,r γ r r=k 0+ γ k 0 r=k 0+ i 0,r γ r δ(k 0) 0 i r γ r r=k 0+ γ k 0 k 0 r= x 0 i 0,r k0 γ r r= i 0,r γ r + γ k 0 γ k 0 k 0 r= γ γ + r i 0 γ r r=k 0+ = r= γ γ r Figure : The figure shows the interval [0,] For any two values x and x that both lie in one of the depicted small intervals it holds that i,r = i,r for r =,,k 0 The three intervals represent the possible cases that occur in the proof of theorem 5 x 0 = 0 and open to the left if x 0 =, the inequality (9) also shows for these two cases continuity from the right and from the left, respectively We hence have proved the following theorem: Theorem 5 The inner function ψ from (8) is continuous on [0,] 3 The monotonicity of ψ A further crucial property of the function ψ is its monotonicity We show this first on the dense subset D R of terminating rational numbers Lemma 6 For every k N, there holds Proof by induction k = : k k + : ψ + k ψ k + γ β(k) ψ + ψ = ψ (d + γ ) ψ (d ) = d + γ d = γ = γ β() ψ + k+ ψ k+ = (s 0 s 0 )(ψ + k ψ ( k) + ( s0 γ β(k+) s 0 )i k+ + ( s 0 )( s 0 ) s 0 (γ ) ) for i γ β(k+) k+ < γ (s 0 = s 0 = 0), = (ψ+ k ψ k) i k+ for i γ k+ k+ = γ (s 0 =, s 0 = 0), (ψ+ k ψ k) γ for i γ β(k+) k+ = γ (s 0 =, s 0 = ) For the first case i k+ < γ, the assertion is trivial For the other two cases, we have (ψ+ k ψ k) i k+ γ k+ (ψ+ k ψ k) γ γ β(k+) ( γ β(k) γ ) γ β(k+) γ β(k+) Here, the validity of the last estimate can be obtained from ( γ β(k) γ ) ( ) γ β(k+) γ β(k+) γ nk γ β(k+) γ γ β(k+) γ β(k+) γ nk γ + γ nk γ n k 9

10 We have thus shown that ψ is strictly monotone increasing on a dense subset of [0,] Since the function is continuous, this holds for the whole interval [0, ] This proves the following theorem: Theorem 7 The function ψ from (8) is monotone increasing on [0,] In summary, we have demonstrated that the inner function ψ defined by Sprecher (cf [7,8]) is neither continuous nor monotone increasing, whereas the definition (8) of ψ by Köppen from [6] possesses these properties 3 The algorithm of Sprecher We will now demonstrate that Sprecher s constructive algorithm from [8] with Köppen s definition of the inner function ψ from [6] is indeed convergent We start with a review of Sprecher s algorithm First, some definitions are needed Definition 3 Let σ : R R be an arbitrary continuous function with σ(x) 0 when x 0, and σ(x) when x For q {0,,m} and k N given, define d q k,p := d k,p + q and set d q k = (dq k,,,dq k,n ) Then for each number ξ(dq k ) := n p= α p ψ(d q k,p ) we set ( ) n β(r) b k := γ and r=k+ p= α p ( θ(d q ;y q ) := σ γ β(k+)( ) ( y q ξ(d q k )) + σ γ β(k+)( y q ξ(d q k ) (γ )b k) ) (3) We are now in the position to present the algorithm of Sprecher which implements the representation of an arbitrary multivariate function f as superposition of single variable functions Let denote the usual maximum norm of functions and let f : I n R be a given continuous function with known uniform maximum norm f Furthermore, let η and ε be fixed real numbers such that 0 < m n+ m+ ε + n m+ k r= γ r η < which implies ε < n m n+ Algorithm 3 Starting with f 0 f, for r =,,3,, iterate the following steps: I Given the function f r (x), determine an integer such that for any two points x,x R n x x γ kr it holds that f r (x) f r (x ) ε f r This determines rational coordinate points d q = (d q k, r,,dq k ) r,n II For q = 0,,,m: II Compute the values ψ(d q ) II Compute the linear combinations ξ(d q ) = n p= α p ψ(d q,p ) II 3 Compute the functions θ(d q ;y q ) III III Compute for q =,,m the functions Φ r q(y q ) = f r (d kr )θ(d q m + ;y q ) (3) d q kr 0

11 III Substitute for q =,,m the transfer functions ξ(x q ) and compute the functions III 3 Compute the function Φ r q ξ(x q ) := m + f r (d kr )θ(d q ;ξ(x q )) d q kr f r (x) := f(x) m q=0 j= r Φ j q ξ(x q) (33) This completes the r th iteration loop and gives the r th approximation to f Now replace r by r + and go to step I The convergence of the series {f r } for r to the limit lim r f r =: g 0 is equivalent to the validity of theorem The following convergence proof essentially follows [7,8] It differs however in the arguments that refer to the inner function ψ which is now given by (8), ie we always refer to Köppen s definition (8), if we use the inner function ψ The main argument for convergence is the validity of the following theorem: Theorem 33 For the approximations f r, r = 0,,, defined in step III 3 of Algorithm 3 there holds the estimate m f r = f r (x) Φ r q ξ(x q ) η f r q=0 To proof this theorem, some preliminary work is necessary To this end, note that a key to the numerical implementation of Algorithm 3 is the minimum distance of images of rational grid points d k under the mapping ξ We omit the superscript of d q k here for convenience, since d q k Dn k and the result holds for all d k Dk n This distance can be bounded from below The estimate is given in the following lemma Lemma 34 For each integer k N, set µ k := n [ α p ψ(dk,p ) ψ(d k,p )], (34) p= where d k,p,d k,p D k Then when min µ k γ nβ(k) (35) Dk n n p= d k,p d k,p 0 (36) Proof Since for each k the set D k is finite, a unique minimum exists For each k N, let d k,p,d k,p D k and A k,p := ψ(d k,p ) ψ(d k,p ) for p =,,n Since ψ is monotone increasing, we know that A k,p 0 for all admissible values of p Now from lemma 6 it follows directly that min A k,p = γ β(k), (37) D k

12 where each minimum is taken over the decimals for which d k,p d k,p 0 The upper bound min µ k α n γ β(k) (38) Dk n can be gained from the definition of the µ k and the fact that = α > α > > α n as follows: Since µ k n p= α p A k,p we can see from (37) and (38) that a minimum of µ k can only occur if A k,t 0 for some T {,,n} Let us now denote the k th remainder of α p by such that and consider the expression We claim the following: ε k,p := α p ε k,p = A k, + r=k+ γ (p )β(r) (39) k γ (p )β(r) (30) r= T (α p ε k,p )A k,p (3) p= If A k,t 0 then A k, + T (α p ε k,p )A k,p 0, ie the term (α T ε k,t )A k,t cannot be annihilated by the preceding terms in the sum To show this, observe that p= α T ε k,t = γ (T ) + γ (T )β() + + γ (T )β(k) Also note that, for the choice k = and i,t = γ as well as i,t = 0 in (7), the largest possible term in the expansion of A k,t in powers of γ is γ γ Therefore, (α T ε k,t ) A k,t contains at least one term τ such that (T 0 < τ γ )β(k)γ γ But according to (37) and (30) the smallest possible term of (α p ε k,p ) A k,p for p < T is γ (T )β(k) γ β(k) (T )β(k) = γ so that the assertion holds and (3) indeed does not vanish If i k,t i k,t =, we have without loss of generality in the representation (7) the values i k i k s s α 0 α + 0 s s α 0 α + 0 γ γ γ 3 γ γ 4 γ

13 and we can directly infer that the expansion of (3) in powers of γ contains the term γ (T )β(k) γ β(k) = γ Tβ(k) (3) We now show that this is the smallest term in the sum (3) To this end, we use the representation (7) for A k,p and factor out γ β(k j) for each j Since α j and α + j be become smaller than j, we can bound each term in the sum (3) from below by γ β(k j) j The further estimation γ β(k j) j > γ β(k) shows that (3) is indeed the smallest term in the sum and hence cannot be annihilated by other terms in (3) Therefore, T A k, + (α p ε k,p )A k,p γ Tβ(k) p= But this implies that also T A k, + α p A k,p γ Tβ(k) p= since all possible terms in the expansion of T p= ε k,pa k,p in powers of γ are too small to annihilate γ Tβ(k) Thus, the lemma is proven The linear combinations ξ(d q k ) of the inner functions serve for each q = 0,,m as a mapping from the hypercube I n to R Therefore, further knowledge on the structure of this mapping is necessary To this end, we need the following lemma: Lemma 35 For each integer k N, let δ k := γ (γ )γ k (33) Then for all d k D k and ε k, as given in (39) we have ψ(d k + δ k ) = ψ(d k ) + (γ )ε k, Proof The proof relies mainly on the continuity of ψ and some direct calculations If we express δ k as an infinite sum we have { k 0 d k + δ k = lim d k + k 0 r= } γ γ k+r =: lim k 0 d k 0 Since ψ is continuous we get ( ) ψ lim d k 0 k 0 = lim k 0 ψ(d k 0 ) and since i k+r = γ for r =, k 0, it follows directly that s r = 0 for j = 0,,k 0 k Therefore α + j = 0 and α j = for j = 0,,k 0 k With the representation (6) and the choice ξ = k 0 k, the assertion follows As a direct consequence of this lemma, we have the following corollary, in which the onedimensional case is treated 3

14 Corollary 36 For each integer k N and d k D k, the pairwise disjoint intervals are mapped by ψ into the pairwise disjoint image intervals E k (d k ) := [d k,d k + δ k ] (34) H k (d k ) := [ψ(d k ),ψ(d k ) + (γ )ε k, ] (35) Proof From their definition it follows directly that the intervals E k (d k ) are pairwise disjoint The corollary then follows from lemma 34 and lemma 35 We now generalize this result to the multidimensional case Lemma 37 For each fixed integer k N and d k Dk n, the pairwise disjoint cubes S k (d k ) := n E k (d k,p ) (36) p= in I n are mapped by n p= α p ψ(d k,p ) into the pairwise disjoint intervals n n n T k (d k ) := α p ψ(d k,p ), α p ψ(d k,p ) + (γ )ε k, (37) p= p= p= Proof This lemma is a consequence of the previous results and can be found in detail in [7] We now consider Algorithm 3 again We need one more ingredient: Lemma 38 For each value of q and r, there holds the following estimate: α p Φ r q(y q ) m + f r Proof The support of each function θ(d q k ;y q) is the open interval ( ) U q k (dq k ) := ξ(d q k ) γ β(k+), ξ(d q k ) + (γ )b k + γ β(k+) Then, by lemma 37 the following holds: If ξ(d q k ) q ξ(d k ) then Uq k (dq k ) Uq q k (d k ) = Now we derive from (3): f r (d q m + )θ(d q ;y q ) = m + max f r (d kr ) d kr d q kr The lemma then follows from the definition of the maximum norm, see also [8], lemma We are now ready to prove theorem 33, compare also [8] 4

15 d k x d k ˆdk Ek 0( ˆd 0 k ) Ek ( ˆd k ) Ek ( ˆd k ) Ek 3( ˆd 3 k ) Ek 4( ˆd 4 k ) ˆd k, d k, Sk 0(d0 k ) Sk (d k ) Sk (d k ) Sk 3(d3 k ) Sk 4(d4 k ) d k, ˆdk, Figure 3: Let k be a fixed integer, m = 4, γ = 0 and d k,i := d k,i γ k, ˆdk,i := d k,i + γ k, i {,} The left figure depicts the intervals E q k (dq k ) for q =,,m The subscript i indicating the coordinate direction is omitted for this one dimensional case The point x is contained in the intervals Ek 0( d 0 k ), E k ( d k ), E3 k (d3 k ), E4 k (d4 k ) (shaded) and in the gap G k ( d k ) (dark shaded) The figure on the right shows the cubes S q k (dq k ) for n =, q =,,m and different values d k Dk n For q {,3}, the marked point is not contained in any of the cubes from the set { S q k (dq k ) : d k Dk} n Proof of theorem 33 For simplicity, we include the value d k = in the definition of the rational numbers D k Consider now for each integer q the family of closed intervals E q k (dq k ) := [ d q k q a, dq k q a + δ ] k (38) With δ k = (γ )(γ ) γ k we can see that [ E q k (dq k ) = d k q γ γ k, d k q γ γ k + γ ] γ γ k and that these intervals are separated by gaps G q k (dq k ) := ( d q k q a + δ k, d q k q a + γ k) of width (γ ) γ k, compare figure 3 With the intervals E q k we obtain for each k and q = 0,,m the closed (Cartesian product) cubes S q k (dq k ) := Eq k (dq k, ) Eq k (dq k,n ), whose images under ξ(x q ) = n p= α p ψ(x p + qa) are the disjoint closed intervals T q k (dq k ) = [ ξ(d q k ), ξ(dq k ) + (γ )b ] k, as derived in lemma 37 For the two dimensional case, the cubes S q k (dq k ) are depicted in figure 3 Now let k be fixed The mapping ξ(x q ) associates to each cube S q k (dq k ) from the coordinate space a unique image T q k (dq k ) on the real line For fixed q the images of any two cubes from the set { S q k (dq k ) : d k Dk} n have empty intersections This allows a local approximation of the target function f(x) on these images T q k (dq k ) for x Sq k (dq k ) However, as the outer functions Φ r q have to be continuous, these images have to be separated by gaps in which f(x) cannot be approximated Thus an error is introduced that cannot be made arbitrarily small This 5

16 deficiency is eliminated by the affine translations of the cubes S q k (dq k ) through the variation of the q s To explain this in more detail, let x [0, ] be an arbitrary point With (38) we see that the gaps G q k (dq k ) which separate the intervals do not intersect for variable q Therefore, there exists only one value q such that x G q k (dq k ) This implicates that for the remaining m values of q there holds x E q k (dq k ) for some d k See figure 3 (left) for an illustration of this fact If we now consider an arbitrary point x [0,] n, we see that there exist at least m n + different values q j, j =,,m n + for which x S q j k (dq j k ) for some d k, see figure 3 (right) Note that the points d k can differ for different values q j From (38) we see that d k S q j k (dq j k ) Now we consider step I of Algorithm 3 To this end, remember that η and ε are fixed numbers such that 0 < m n+ m+ ε + n m+ η < Let be the integer given in step I with the associated assumption that f r (x) f r (x ) ε f r when x p x p γ kr for p =,,n Let x [0,] n be an arbitrary point and let q j, j =,,m n +, denote the values of q such that x S q j (d q j ) For the point d kr S q j (d q j ) we have f r (x) f r (d kr ) ε f r (39) and for x it holds that ξ(x qj ) T q j (d q j ) The support U q j (d q j ) of the function θ(d q j ;y qj ) contains the interval T q j (d q j ) Furthermore, from definition (3) we see that θ is constant on that interval With (33) we then get Φ r q j ξ(x qj ) = = m + d q j kr m + f r (d kr ) Together with (39) this shows m + f r (x) Φ r q j ξ(x qj ) f r (d kr )θ(d q j ;ξ(x qj )) (30) ε m + f r (3) for all q j, j =,,m n + Note that this estimate does not hold for the remaining values of q for which x is not contained in the cube S q (d q j ) Let us now denote these values by q j, j =,,n We can apply lemma 38 and with the special choice of the values ε and η we obtain the estimate m f r (x) = f r (x) Φ r q ξ(x q) q=0 m m n+ = m + f n r (x) Φ r q j ξ(x qj ) Φ r q j ξ(x qj ) q=0 j= j= (3) m n+ n m + f r (x) + m + f r (x) Φ r q j ξ(x qj ) j= + n m + f r [ m n + ε + n ] f r η f r m + m + This completes the proof of theorem 33 We now state a fact that follows directly from the previous results 6

17 Corollary 39 For j =,, 3, there hold the following estimates: and f r = Φ r q (y q) f(x) m q=0 j= m + ηr f (33) r Φ j q ξ(x q ) ηr f (34) Proof Remember that f 0 f The first estimate follows from lemma 38 and a recursive application of theorem 33 The second estimate can be derived from the definition (33) of f r and again a recursive application of theorem 33 We finally are in the position to prove theorem Proof of theorem From corollary 39 and the fact that η < it follows that, for all q =,,m, we have r Φ j r q(y q ) Φ j q (y q ) r m + f η j < m + f η j < (35) j= j= The functions Φ j q(y q ) are continuous and therefore each series r j= Φj q(y q ) converges absolutely to a continuous function Φ q (y q ) as r Since η < we see from the second estimate in corollary 39 that f r 0 for r This proves Sprecher s version of Kolmogorov s superposition theorem with Köppen s inner function ψ j=0 j=0 4 Conclusion and outlook In this paper we filled mathematical gaps in the articles of Köppen [6] and Sprecher [7, 8] on Kolmogorov s superposition theorem We first showed that Sprecher s original inner function ψ is not continuous and monotone increasing Thus the convergence proof of the algorithm from [8] that implements () constructively is incomplete We therefore considered a corrected version of ψ as suggested in [6] We showed that this function is well defined, continuous and monotone increasing Then, we carried the approach for a constructive proof of Kolmogorov s superposition theorem from [7, 8] over to the new continuous and monotone ψ and showed convergence Altogether we gave a mathematically correct, constructive proof of Kolmogorov s superposition theorem The present result is, to our knowledge, the first correct constructive proof of () and thus of () It however still involves (with r ) an in general infinite number of iterations Thus, any finite numerical application of algorithm 3 can only give an approximation of a n dimensional function up to an arbitrary accuracy ǫ > 0 (compare corollary 39) While the number of iterations in algorithm 3 to achieve this desired accuracy is independent of the function f and its smoothness, the number which is determined in step I can become very large for oscillating functions This reflects the dependency of the costs of algorithm 3 on the smoothness of the function f: In step II the functions θ(d q,y q ) are computed for all rational values d q which can be interpreted as a construction of basis functions on a regular grid in the unit cube [0,] n Since the number of grid-points in a regular grid increases exponentially 7

18 with the dimensionality n, the overall costs of the algorithm increase at least with the same rate for n This makes algorithm 3 highly inefficient in higher dimensions To overcome this problem and thus to benefit numerically from the constructive nature of the proof further approximations to the outer functions in () have to be made This will be discussed in a forthcoming paper References [] V Arnold On functions of three variables Dokl Akad Nauk SSSR, 4:679 68, 957 English translation: American Math Soc Transl (), 8, 5-54, 963 [] V Arnold On the representation of functions of several variables by superpositions of functions of fewer variables Mat Prosveshchenie, 3:4 6, 958 [3] V Arnold On the representation of continuous functions of three variables by superpositions of continuous functions of two variables Mat Sb, 48:3 74, 959 English translation: American Math Soc Transl (), 8, 6-47, 963 [4] R J P de Figueiredo Implications and applications of Kolmogorov s superposition theorem IEEE Transactions on Automatic Control, AC-5(6), 980 [5] B Fridman An improvement on the smoothness of the functions in Kolmogorov s theorem on superpositions Dokl Akad Nauk SSSR, 77:09 0, 967 English translation: Soviet Math Dokl (8), , 967 [6] F Girosi and T Poggio Representation properties of networks: Kolmogorov s theorem is irrelevant Neural Comp, : , 989 [7] R Hecht-Nielsen Counter propagation networks Proceedings of the International Conference on Neural Networks II, pages 9 3, 987 [8] R Hecht-Nielsen Kolmogorov s mapping neural network existence theorem Proceedings of the International Conference on Neural Networks III, pages 4, 987 [9] T Hedberg The Kolmogorov superposition theorem, Appendix II to H S Shapiro, Topics in Approximation Theory Lecture Notes in Math, 87:67 75, 97 [0] D Hilbert Mathematical problems Bull Amer Math Soc, 8:46 46, 90 [] W Hurewicz and H Wallman Dimension Theory Princeton University Press, Princeton, NJ, 948 [] B Igelnik and N Parikh Kolmogorov s spline network IEEE transactions on Neural Networks, 4:75 733, 003 [3] S Khavinson Best approximation by linear superpositions Translations of Mathematical Monographs, 59, 997 AMS [4] A N Kolmogorov On the representation of continuous functions of many variables by superpositions of continuous functions of one variable and addition Doklay Akademii Nauk USSR, 4(5): , 957 8

19 [5] A N Kolmogorov and V M Tikhomirov ǫ entropy and ǫ capacity of sets in function spaces Uspekhi Mat Nauk, 3():3 86, 959 English translation: American Math Soc Transl 7, (), , 96 [6] M Köppen On the training of a Kolmogorov network ICANN 00, Lecture Notes In Computer Science, 45: , 00 [7] V Kurkova Kolmogorov s theorem is relevant Neural Computation, 3:67 6, 99 [8] V Kurkova Kolmogorov s theorem and multilayer neural networks Neural Networks, 5:50 506, 99 [9] G Lorentz Approximation of functions Holt, Rinehart & Winston, 966 [0] G Lorentz, M Golitschek, and Y Makovoz Constructive Approximation 996 [] M Nakamura, R Mines, and V Kreinovich Guaranteed intervals for Kolmogorov s theorem (and their possible relation to neural networks) Interval Comput, 3:83 99, 993 [] M Nees Approximative versions of Kolmogorov s superposition theorem, proved constructively Journal of Computational and Applied Mathematics, 54:39 50, 994 [3] P A Ostrand Dimension of metric spaces and Hilbert s problem 3 Bull Amer Math Soc, 7:69 6, 965 [4] T Rassias and J Simsa Finite sum decompositions in mathematical analysis Pure and applied mathematics, 995 [5] D A Sprecher On the structure of continuous functions of several variables Transactions Amer Math Soc, 5(3): , 965 [6] D A Sprecher An improvement in the superposition theorem of Kolmogorov Journal of Mathematical Analysis and Applications, 38:08 3, 97 [7] D A Sprecher A numerical implementation of Kolmogorov s superpositions Neural Networks, 9(5):765 77, 996 [8] D A Sprecher A numerical implementation of Kolmogorov s superpositions II Neural Networks, 0(3): , 997 9