No Free Lunch Theorem. Simple Proof and Interpretation of No Free Lunch Theorem

Transcription

1 No ree Lunch Theorem No ree Lunch Treorem NL Simple Proof and Interpretation of No ree Lunch Theorem Kohsuke Yanai Hitoshi Iba Dept. of rontier Infomatics, Graduate School of rontier Sciences, The University of Tokyo. In this paper, a simple proof of No ree Lunch Theorem (NL) is presented. NL asserts that no search algorithm exists which can solve every problem efficiently. This theorem has serious implications for the study of search algorithms and optimization. The proof given in this paper is simpler and more intuitive than the original one. In the proof, we divide the fitness function space into a set of subsets, and calculate the sum of probabilities in each subset. The division of the function space helps us to understand the theorem more clearly and highlights the relationship between algorithms and problems. 1. No ree Lunch Theorem NL NL [ 03] [ 03] Wolpert Macready 1995 [Wolpert 95] NL Evolutionary Strategy Memetic Algorithm 2 NL (1) (2) [ 01] 2 GECCO Genetic and Evolutionary Computation Conference NL [Kauffman 00] 2 NL NL Wolpert

2 GA Wolpert 2 GA [Wolpert 97] Wolpert GA TSP NL NL 1 NL x 1 x 1 x 1 y 1 x i+1, 1 GA 2 [Wolpert 95] Evolutionary Strategy NL 3 NL 4 Memetic Algorithm 5 2. No ree Lunch Theorem 2 1 NL Y [Wolpert = {f : X Y } f 95] f x X 4 2 y Y 1 a (x 1,x 2,,x i ) X i (y 1,y 2,,y i ) Y i x i+1 X X Y X Y TSP Y X Y Y X = Y X Genetic Algorithm GA NL X Y Travelling Salesman Problem 1 TSP 2 GA 2 NL 2 TSP

3 The first candidate solution is uniquely determined by algorithm. a x1 f a x2 f a... a xk f x = dk : The trace of candidate solutions y1 y2 yk y = dk : The trace of cost values 1 a x i+1 a a f d k D k S(a, f) 1 NL X Y Dk X f 1, f 2 2 3bit (x 1,x 2,,x k ) Dk X f 1 f x1 y1 x2 y2 dk x dk y xk yk f i j [i j x i x j ] (1) a A X (d k ) d k f 2 f (d k ) i [y i = f(x i )] = Y X (2) D Y k = Y k (3) (d k ) = Y X k (4) 2 (d k ) d k f (d k ) k d k f (d k ) i [y i = f(x i )] d k 2 2 A k k < X X k X k Y X k f d k a f k S k (a,f) D k 1 a d y k DY k X a d y k k DX k Y a = {f : X Y } x 1 y 1 Dk X k x 1 y 1 a x 2 Dk X Xk x i Dk Y a d y k k k R k (a,d y k ) D k Dk Y = Y k R k D k k R k (a,d y k ) D k = Dk X DY k (R k (a,d y k )) A f (R k (a,d y k )) (d k ) d k D k f R k (a,d y k ) R k (a,d y k ) = S k(a,f) (5)

4 a x1 x2 a a a... xk y1 y2 yk y dk is given. 3 R k (a, d y k ) y i x i a d y k d k D k R k (a, d y k ) S k R k B 3 k Dk x k D y k A P(D y 3 = (3,7,4) A = a) (6) a (3,7,4) Wolpert NL Wolpert NL NL 1 NL a 1,a 2 A d y k DY k [Sharpe 00] f a 1 a 2 d y k [ P(D y k = dy k A = a 1, = f) f f = V P(D y k = dy k A = a 2, = f)] f 1 ϕ : V a A (7) a ϕ k [ ϕ(f) = ϕ(f)] f t D Y f (R k k (a,t)) (9) f 4 Y t Dk Y f m d y m = (y 1,y 2,,y m ) Dm Y max y k (8) k m NL

5 t 1 t 2 (t 1,t 2 Dk Y ) (R k (a,t 1 )) (R k (a,t 2 )) φ t2 t1 t3 t4 P=1 f [f (R k (a,t 1 )) f (R k (a,t 2 ))] (10) t5 t7 f S k R k R k (a,t 1 ) = S k (a,f) = R k (a,t 2 ) (11) 5 t6 t 4 = d y k t 4 t 1 t 2 B 4 (R k (a,t 1 )) (R k (a,t 2 )) = φ (12) f (R k (a 1,t))P(D y k = dy k A = a 1, = f) 0 d y t Dk Y f (R k k(a,t)) = t (R k (a 1,t)) = Y X k d y k = t (R k (a,t)) = (21) (R k (a,t)) (13) t Dk Y t Dk Y = Y X k P(D y (14) k = dy k A = a 1, = f) = Y X k (22) t D Y k = D Y k Y X k (15) = Y k Y X k (16) = Y X = (17) f P(D y k = dy k A = a 1, = f) f a 1 NL f 4. NL t D NL a A k Y (R(a,t)) a 1 d y k DY k (R k(a,d y k )) f P(D y k = dy k A = a 1, = f) support) (R k (a,d y k )) = (R k (a,t)) P(D y k = dy k A = a 1, = f) f t Dk Y f (R k (a 1,t)) (18) k (R k (a,t)) a f (R k (a 1,t)) a 1 t S k R k t f (R k (a 1,t)) d y k DY k P(D y k = t A = a 1, = f) = 1 (19) P(D y k t A = a 1, = f) = 0 (20) (R k (a,t)) = Y X k

6 t (2) Y X k seed GA t t NL X Y t 1 X 4 2 NL NL X Y P( = f) = 1 (24) X Y NL NL = {f : X Y } a 1 a 2 d y = Y X k [P(Dy k = dy k A = a 1) = P(D y k = dy k A = a 2)] (25) Y f g x X [f(x) = g(x)] (23) Y / Y X Y / / X f g X f g 4 4 Y NL NL 2 2 (1) D y k

7 a1 a2 NL [ 01][ 98] NL GA [Jones 95] [Kauffman 00] One order relation is determined for the trace of cost values The function space which is not included in the problem. NL NL NL D y k 4 t Dk Y t 6 2 NL A. X Y k k < X 1 X Y = {f : X Y } (A.1) 2 Dk X = {(x 1, x 2,, x k ) X k i j [i j x i x j ] } (A.2) 4 5 Dk Y = Y k (A.3) X σ f σf σf(x) = f(σ 1 (x)) D k = Dk X DY k (A.4) [Schumacher 01] 3 x k NL : DX k X Xk+1 (x 1, x 2,, x k ) x k x = (x 1, x 2,, x k, x) (A.5) NL y k : DY k Y DY k+1 (y [Igel 03] 1, y 2,, y k ) y k y = (y 1, y 2,, y k, y) (A.6)

8 k : D k (X Y ) X k+1 Dk+1 Y (d x k, dy k ) k(x, y) = (d x k x k x, dy k y ky) (A.7) a = (a 0, a 1,, a m 1 ) (B.24) t = (η 1, η 2,, η k ) (B.25) 4 S k (a, f) = ((x 1, x 2,, x k ), (y 1, y 2,, y k )) R k (a, t) = ((ξ 1, ξ 2,, ξ k ), t) (B.26) (B.27) A k = {a k : D k X d k i [a k (d k ) x i ] } A = X A 1 A 2 A m 1 (A.8) (A.9) d k = ((x 1, x 2,, x k ), (y 1, y 2,, y k )) A i d i D i X 2 a k A k d k = (d x k, dy k ) D k a k d k [d x k a k(d k ) D X k+1 ] d x k = (x 1, x 2,, x k ) d x k a k(d k ) = (x 1, x 2,, x k, a k (d k )) 5 (A.10) (A.11) f (R k (a, t)) i [f(ξ i ) = η i ] (ξ 1, η 1 ) = (ξ 1, f(ξ 1 )) = (a 0, f(a 0 )) = (x 1, y 1 ) (B.28) ξ i+1 = a(d i ) = x i+1 η i+1 = f(ξ i+1 ) = f(x i+1 ) = y i+1 d x k DX k i j [i j x i x j ] a t f (R k (a, t)) [P(D y k A k i [a k (d k ) x i ] d x k a k(d k ) Dk+1 X 3 (B.29) 4 a A t Dk Y = t A = a, = f) = 1] (B.30) (B.31) S y k (a, f) = Ry k (a, t) = t (d k ) = {f i [f(x i ) = y i ] } (A.12) f (R k (a, t)) D y k = t d k = ((x 1, x 2,, x k ), (y 1, y 2,, y k )) 6 a = (a 0, a 1,, a m 1 ) A f S k : A D k S 1 (a, f) = (a 0, f(a 0 )) (A.13) [Igel 03] Igel, C. and Toussaint, M.: On classes of functions for which No ree Lunch results hold, Information x = a i (S i (a, f)) (A.14) Processing Letters (2003) S i+1 (a, f) = S i (a, f) (x, f(x)) (A.15) [Jones 95] Jones, T. and orrest, S.: itness Distance Correlation as a Measure of Problem Difficulty for GAs, in Proceedings of 6th International Conference on Genetic Algorithms, pp (1995) 2 welldefined S k (a, f) = (d x k, dy k ) [Kauffman 00] Kauffman, S. A.: Investigations, Oxford Sk x (a, f) = dx k (A.16) University Press (2000) [Schumacher 01] Schumacher, C., Vose, M. D., and S y k (a, f) = dy k (A.17) Whitley, L. D.: The No ree Lunch and Problem Description Length, in Proceedings of the Genetic 7 a = (a 0, a 1,, a m 1 ) A t = (y 1, y 2,, y k ) and Evolutionary Computation Conference (GECCO- 2001), pp , Morgan Kaufmann (2001) D k [Sharpe 00] Sharpe, O. J.: Towards a Rational Methodology c 1 = (a 0, y 1 ) (A.18) for Using Evolutionary Search Algorithms, PhD thesis, University of Sussex (2000) c i+1 = c i (a i (c i ), y i+1 ) (A.19) [Wolpert 95] Wolpert, D. H. and Macready, W. G.: No ree Lunch Theorems for Search, Technical Report R k : A D y k D k SI-TR , Santa e, NM (1995) R k (a, t) = c k (A.20) [Wolpert 97] Wolpert, D. H. and Macready, W. G.: No ree Lunch Theorems for Optimization, IEEE Transactions on Evolutionary Computation, Vol. 1, No. 1, 2 well-defined R k (a, t) = (d x pp (1997) k, t) [ 01], Rk x (a, t) = dx k (A.21),, Vol. 16, No. 3, pp (2001) R y k (a, t) = t (A.22) [ 03], R y k t Dy k GA,, Vol. 18, No. 5, pp (2003) [ 98], Colin, R. R., B., Vol. 39, No. 7, pp (1998) 3 a A t Dk Y [ 03],,,, a t f (R k (a, t)) [S k (a, f) = R k (a, t)] (B.23) Vol. 18, No. 5, pp (2003)