Algorithms for PAC Learning of Functions with Smoothness Properties

Transcription

1 Algorithms for PAC Learning of Functions with Smoothness Properties Nageswara S. V. Rao and Vladimir A. Protopopescu Center for Engineering Systems Advanced Research P. 0. Box 2008 Oak Ridge National Laboratory Oak Ridge, Tennessee { raons,pro t op op gov The submitted manuscript haa been authored by a contractor of the U.S. Government under contr.ct No. DE AC05-(llOR Accordingly, the US. Government retains a nonexclurive, royalty-free license to puhlirh or reproduce the published form of this contribution, or allow othcrr to do so, for US. Government purpoaes. Paper submitted to Fourth International Symposium on Artificial Intelligence and Mathematics, January 3-5, 1996, Fort Lauderdale, Florida. tresearch sponsored by the Engineering Research Program of the Office of Basic Energy Sciences, of the U.S. Department of Energy, under Contract No. DE-AC05-840R21400 with Martin Marietta Energy Systems, Inc. 1

2 Algorithms for PAC Learning of Functions with Smoothness Properties Nageswara S. V. Rao and Vladimir A. Protopopescu Center for Engineering Systems Advanced Research Oak Ridge National Laboratory Oak Ridge, Tennessee {raons, Summary - We present three computationally efficient algorithms for Probably and Approtimately Correct (PAC) learning of an unknown function f : [0, lid t-+ [0,1], based on finite samples. The function f is chosen from the family 3 n C([O, lid) or T n C"([O, l]'), where F has either bounded modulus of smoothness or bounded capacity or both. Three function estimators based on: (i) local averaging, (ii) nearest neighbor rule, and (iii) Nadaraya-Watson estimator, all computed using the Haar system, are analyzed. With no preprocessing of the sample, estimated function value at a given point can be computed in O ( n )time. With preprocessing, the first and third estimators can be computed in O((1og T Z ) ~time ) using a range-tree precomputed in O(dn(1og n ) d )time. Introduction The problem of learning functions in the PAC learning framework of (Valiant 1984) continues to generate significant interest and activity (Alon et al. 1993; Bartlett, Long, & Williamson 1994; Ben-David et al. 1992; Kimber & Long 1992; Apsitis, Freivalds, & Smith 1995). Initial efforts have been concentrated on indicator functions and functions on simpler domains (Natarajan 1991) with increasing attention being paid to real functions (Auer et al. 1993; Simon 1994). Recent results establish that a function that achieves small empirical error on an independently and identically distributed sample yields a PAC a p proximation, under the finiteness of a combinatorial parameter such as the fat-shattering index (Bartlett, Long, & Williamson 1994; Anthony & Bartlett 1994). The difficulties of computing such PAC estimators for functions are well-known in that the problem is NPcomplete even for simple indicator function classes (Blumer e i al. 1989; Pitt & Valiant 1988). We show that mild smoothness properties enable us to identify such classes and to actually construct computationally eficient function estimators that guarantee PAC type results. Our algorithms are applicable to more general cases than in (Kimber & Long 1992) and also yieid lower computational complexity than those in (Auer et af. 1993). Regression estimation problem, which subsumes the function estimation problem, is well-known in the statistics literature (Rao 1983; Brieman et al. 1984). Typical statistical consistency results of regression estimates are asymptotic and do not yield finite sample results required by the PAC paradigm with the exception of some results that utilize Borel-Cantelli Lemma (Nadaraya 1970). On the other hand, a key attractive feature of some of these methods is their low computationally complexity. In general, complexity measures such as the finite capacity (Vapnik 1982) or the fat-shattering index (Anthony & Bartlett 1994; Bartlett, Long, & Williamson 1994) and the smoothness properties of functions do not imply each other. We show that the two types of conditions can be combined to yield low complexity under mild additional restrictions that preserve the strength of PAC method without over-constraining the problem. We study three function estimators based on: (i) local averaging, (ii) nearest neighbor rule, and (iii) Nadaraya-Watson estimator, all computed using the Haar system. For these estimators, we provide PAC results on function error based on a randomly chosen point, expected error, and supremum of the error, respectively. The estimated function value at a given point can be computed in O(n) time, for all three cases, and with preprocessing, the first and third can be computed in O((logn)d) time using a range-tree precomputed in O(dn(1og n ) d )time. Our results differ from those based on capacity (or related combinatorial parameters) in two directions: (a) under mild smoothness conditions on the function and/or the density, we can obtain stronger guarantees for the error between the function and the estimator; and, (b) our estimators can be computed in linear time, unlike the general PAC solutions that usually require solving NP-hard problems. A brief comparison with feedforward sigmoidal neural networks indicates that our nearest neighbor estimate approximates a neural network based estimator within the PAC paradigm. The proofs of the theorems presented in this paper will appear elsewhere (Rao & Protopopescu 1995).

3 Preliminaries Let Q = [O,lId, and let C(Q) and P ( Q ) denote the classes of continuous and essentially bounded functions defined on Q,respectively. For f E L"O(Q),we have II f llm= as e s s s ~ ~ { l f ( z :) 2 l E &I. The modulus of smoothness off E L"O(Q)is defined Then an estimator for a density p E C"O(Q) based on n-sample is given by (Ciesielski 1988) which can also be written in the form fim.n(z)= n ( J ) h j ( z ) with n(j) = il{j : Xj E J)I and J Qm u w ( f r; ) = SUP Ihloo<r (ess SUP Q(h) If(%+ h) - f(z)i) + where Q ( h ) = {z E Q : z h E Q } and lhla = max(lh11,..., / / a d ( ). We note that for continuous functions, f E C(Q), the modulus of smoothness coincides with the ordinary modulus of continuity defined as The following identity is established in (Vapnik 1982): ifnsk H { A, } ( n ) = 2" < 1. 5 g if n > A. H5re A: is called the Vapnik-Chervonenkzs (VC) dimension of the family of sets A,. For a set of functions, the capacity (Vapnik 1989) is defined as the largest number h of pairs (zi,yj) that can be subdivided in all possible ways into two classes by means of rules of the form {O[(y- f(z))' ] ) ( j, p ) where f E 3,j? E W, and Q ( z ) is the Heaviside stepfunction defined as { e(%) = 1 ifz>o 0 fz<0. Thus the capacity of a family of functions 3 is the VC dimension of the set of indicator functions {Q[(y- fw2+ P l 1 ( f, P ) E F X S 1. For rn = 0, 1,..., let Q, denote a famiiy of diadic cubes such that [0, lid= U J, JnJ' = 0 for J # J', JEQm and the d-dimensional volume of J, denoted by IJI, is 2-dm. Let l~(z)denote the indicator function of J E Qm: lj(t) = 1 if z E J, and ~ J ( z=) 0 otherwise. For given rn, we define P, : Coo(&) Cm(Q) by c.$ for z E J and J E Q, (Ciesielski 1988). Consider a kernel given by P,(z, y) = 2dm 1J ( z )~~( y )for JEQm &YEQ. h ~ ( z=) mlj(z)lemma 1 (Ciesielski 1988) Let 0 < a 5 1 and f E C ( Q ) or f E P ( Q ) be given. Then the condition w ( f;r ) = O(ra) as r --+ O+ implies 11 f - P, f Iloo= as rn O(1/2am) PAC Estimators for Continuous Functions We consider three types of function estimators. The first is a local estimator based on "averaging" the function values within each cell of suitably chosen Qm. The second one is based on the nearest neighbor rule applied to each cell of &., When the functions have bounded moduli of smoothness, these estimators provide distribution-free results. The third estimator, called Nadaraya-Watson, applies to a more particular case where the density exists and also satisfies some smoothness properties. Not surprisingly, the Nadaraya-Watson estimator based on the Haar system provides better guarantees. Local Averaging Based on the n-sample, the first estimator of the function f is given by. which evaluates to n C XjEJ w, for z E J. Estimators of this general structure are called regressograms (Rao 1983) (this estimator, however, is not identical to the traditional regressogram). The next theorem provides a finite sample result for this estimate. Theorem 1 Consider a family of continuous functions 3 C C([O, lid), with range [0,1] and capacity h such that for every f E F,we have um(f;r ) = O(r") as r 0, f o r 0 < a 5 1. Suppose that the size of the sample, n, is larger than --+ 'Some additional measurability conditions are required F,which are assumed to be satisfied throughout the paper (Pollard 1984). on

4 where m = $log(4c/r6) is larger than a suitable constant mo. Then for any X chosen according to the distribution Px and any f E 3, we have P [ M X ) - fm,n(x)i > 1 < 6. Local Nearest Neighbor Rule Based on the n-sample, the second estimator for the function f is given by fm,n(x) = 1J(x)n/J(Z) JEBm where for 2 E J, N j ( x ) yields f ( X i ) such that Xi E J is closest to x in sup norm. As shown later, this estimator has a higher computational complexity than the other two when preprocessing is not allowed and also provides a weaker performance guarantee. The trade-off is that it only requires a bounded modulus of smoothness and does not require bounded capacity. Theorem 2 Consider a family of continuous functions F C C([O, lid), with range [0,1] such that for, every f E F, we have woo(f;r) = O(ro) as r -+ 0, for 0 < CY 5 1. Suppose that the size of the sample, n, is larger than kzd3e22m+2 k2d3e23m+2 In2 2 &2 +2 ( ) where k = C/2um and m = Llog(4C/c6) is larger than a suitable mo. For X distri&uted according to Px and any f E 3, we have P [Elf ( X ) - fm,n(x)i > < 6. 1 Nadaraya-Watson Estimator We now present the third estimator that provides a better guarantee under additional conditions on the densities (simiiar in some sense to (Sakai, Takimoto, & Maruoka 1995)). Based on the n-sample, the estimator is defined by n where 0 < p < d/2(a + d), rn = r-1 f E T, we have P If(2) - fm,n(z)i> and X = c Computing the Estimates Computation of estimators fm,n(z) or fm,n (z) at given t involves obtaining the local sum of f ( X i ) ' s that are contained in J containing x. The range-tree (Preparata & Shamos 1985) can be constructed to store the cells J that contain at least one Xi; with each such cell we store the number of the Xi's that are contained in J and the s u m of the corresponding f(xi)'s. This computation can be achieved by known methods (Preparata & Shamos 1985), and the values corresponding to J that contains z can be retrieved or f,,,n(x) can in O((logn)d) time, and then fm,n(z) be computed in additional constant time. This same structure can be used to store the training sample o_f each J; once J containing x has been identified, fm,n(z) can be computed in linear time. Theorem 4 The estimators fm,n(z),j,,+(z), and Z;n,n(+) at given x E [O,1jd can be computed in O(n) time. Based on a preprocessing in O(n(1o n d-l time, resulting in a structure of size O(n(1og n)pi-) ), the estt' motor im,n(x) or fm.,,(z)for given z can be computed in O((1og n ) d ) time. LOO-Functions Recall that C([O, lid) c t " ( [ O, lid) and the latter allows for discontinuities in the functions. Only minor changes to the proofs of results in last section are needed to establish analogous results for t w ( [ O, lid). Approximation to Neural Networks We consider a feedforward network with a single hidden layer of m hidden nodes and a single output node. The output of the j t h hidden node is a(b?z t j ), where x E [ O, l l d, bj E?Xd, t j E 8,bTx is the scalar product, and the monotonically increasing function u :?X w [O,11 is called an activation function. The output of the network corresponding to input x is + for z E J. Estimators of this general structure are called Nadaraya-Watson kernel estimators.(rae 1983). Here we use the kernels generated by the Haar functions (Engel 1994). Theorem 3 Consider a family of functions F C_ C([O, 1Id) with range [0,1]and capacity h < 00 such that wm(f;r) = O(rP) as r + 0, for 0 < CY 5 1. We assume that: (i) there ezists a family of densities P E C([O, ljd); (ii) for each p E P,w,(p; r) = O(ra) as r + 0, for 0 < Q 5 1; and (iii) there ezists p > 0 such that f o r each p E P,p(x) > p. Suppose that the sample size, n, is larger than m where a = ( a l,0 2,..., a m ) E?Xm and w is the weight vector of the network consisting of a, bl, b2,...,b, and t f, t 2,...,tm. We consider neural networks with bounded weights such that w E [-W,+W]"(d+l) for some fixed positive W < 00. We consider u ( z ) = 1/(1 e-y'), for 7, z E R. Let Fw = {fw : w E [-Wl+W]m(d+l)}. Then the nearest neighbor estimator fm,n can approximate any f E 3w to any desired accuracy in terms of the expected error. +

5 Corollary 1 For any f E F w,we have 4 p [ W X )- L, n ( W I > < 6 corresponding to the sample sire given in Theorem 2 with the Lipschitr constant k given by where u = mwlaji and b = miqxlbijl. a Tfi '$1 Note that boundedness of capacity of neural networks of this type can be deduced from the results of (Macintyre & Sontag 1993), which can be used t o obtain sample bounds along the lines of Theorem 1. The associated computation problem, however, is NPcomplete (Roychowdhury, Siu, & Orlitsky 1994). Thus it is of practical interest to approximate neural networks by the nearest neighbor rule that guarantees a PAC result is linear-time computable. Acknowledgements The authors acknowledge the guidance of Prof. Leo Brieman, Prof. Anselm Blumer and Dr.V. R. R. Uppuluri. This research is sponsored by the Engineering Research Program of the Office of Basic Energy Sciences, of the U.S. Department of Energy, under Contract No. DEAC05-840R21400 with Lockheed Martin Energy Systems, Inc. References Alon, N.; Ben-David, S.; Cesa-Bianchi, N.; and Hausler, D Scale-sensitive dimensions, uniform convergence, and learnability. In Proc. of 1993 IEEE Symp. on Foundations of Computer Science. Anthony, M., and Bartlett, P Function learning from interpolation. NeuroCOLT Technical Report Series NC-TR-94013, Royal Holloway, University of London. Apsitis, K.; F'reivalds, R.; and Smith, C. H On the inductive inference of real valued functions. In Proc. of 8th Ann. ACM Conf. on Computational Learning Theory. Auer, P.; Long, P. M.; Mass, W.; and Woeginger, G. J On the complexity of function learning. In Proc. of 6ih Ann. ACM Conf. on Computational Learning Theory, Bartlett, P. L.; Long, P. M.; and Williamson, R. C Fat-shattering and the learnability of realvalues fucntions. In Proc. of 7th Ann. ACM Conf. on Computational Learning Theory. Ben-David, S.; Cesa-Bianchi, N.; Haussler, D.; and Long, P Characterizations of learnability for classes (0,...,n)-valued functions. In Proc. of 5th Ann. ACM Conf. on Computational Learning Theory. Blumer, A.; Ehrenfeucht, A.; Haussler, D.; and Warmuth, M Learnability and the VapnikChervonenkis dimension. Journal of the Association of Computing Machinery 36(4): Brieman, L.; Friedman, J. H.; Olshen, R. A.; and Stone, C. J Classijcation and Regression Trees. Belmont, C A: Wadswor t h. Ciesielski, Z Haar system and nonparametric density estimation in several variables. Probability and Mathematical Statistics 9:l-11. Engel, J A simple wavelet approach to nonparametric regression from recursive partitioning schemes. Journal of Multivariate Analysis 49~ Kimber, D., and Long, P. M The learning complexity of smooth functions of a single variable. In Proc. of 1992 Workshop on Computational Learning, Macintyre, A., and Sontag, E. D Finiteness results for sigmoidal neural networks. In Proc. 25th Annual ACM Symp. on Theory of Computing Nadaraya, E. A Remarks on non-parametric estimates for density functions and regression curves. Theory of Probability and Applications 15: Natarajan, B. K Machine Learning: A Theoretical Approach. San Mateo, California: Morgan Kaufmann Pub. Inc. Pitt, L., and Valiant, L. G Computational limitations on learning from examples. Journal of the Association f o r Computing Machinery 35(4): Pollard, D Convergence of Stochastic Processes. New York: Springer-Verlag. Preparata, F. P., and Shamos, M. I Computational Geometry: An Introduction. Springer-Verlag. Rao, N. S. V., and Protopopescu, V. A Pac learning of functions with smoothness properties: Implications for feedforward sigmoidal networks. manuscript, to be published. Rao, B. L. S. P Nonparametric Functional Estimation. New York: Academic Press. Roychowdhury, V.; Siu, K.; and Orlitsky, A., eds Theoretical Advances in Neural Computation and Learning. Kluwer Academic Pub. Sakai, Y.; Takimoto, E.; and Maruoka, A Proper learning algorithm for functions of k terms under smooth distributions. In Proc. of 8th Ann. ACM Con$ on Computational Learning Theory. Simon, H. U Bounds on the number of examples needed for learning functions. In Shawe-Taylor, J., and Anthony, M., eds., Computational Learning Theory: EUROCOLT'93. Oxford University Press. Valiant, L. G A theory of the learnable. Communications of the ACM 27(11): Vapnik, V. N Estimation of Dependences Based on Empirical Data. New York: Springer Verlag. Vapnik, V. N Inductive principles of the search for empirical dependences. In Proceedings of Second Ann. Workshop on Computational Learning Theory, 3-21.

6 DISCLAIMER This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercia1 product, process, or service by trade name, trademark, manufac: turer, or otherwise does not necessarily constitute or imply its endorsement, m m- mendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.