The Application of Markov Chain in the Realization of AI of Draughts

Transcription

1 The Application of Markov Chain in the Realization of AI of Draughts Junchao CHEN Master 1 Model Statistic 31/3/2017

2 Abstract Artificial intelligence has made great success in the field of chess. This paper is devoted to the application of Markov decision process model in the realization of artificial intelligence of draughts. In order to improve the intelligent effects and calculation speed, I used some knowledge about reinforcement Learning. In theory, the size of the checker board that I studied is p q, and the empty lines in the middle is r. Where p,q,r are even number. In the code, the value of p,q,r are decide by the player. In order to reduce the computational complexity, we let p=q=6,r=2generally. In this paper, I used the Markov Decision Process to establish the model of the game. However, we can not expression the transition matrix since there are too many states and actions. Back Propagation Neural Network was used to evaluate the state values. And temporal d- ifference learning method was used to optimize the neural network. At last, I choose to use the α β search method to decide the actions. In chapter 5, I will evaluate the model with different parameter. Keywords: draughts; Markov Decision Process; Back propagation Neural Network; Temporal difference learning; Minimax search method; artificial intelligence

3 Contents Abstract I 1 Introduction Preface Structure of the paper Introduction of draughts Rules Computer complexity Fundamentals knowledges Fundamentals of the discrete-time Markov chains Basic model of Discrete Markov Decision Process The simple decision model in which the actions only rely on the state they are in The construction of optimal Markov policy under the total reward of time-bounded Back propagation neural network Temporal difference learning method α β minimax search method minimax search method α β search method Summary of this chapter Establish MDP model for draughts State space Action space

4 CONTENTS Transition matrix, discount factor, reward function and state value Summary of this chapter Algorithm of Realizing AI of the draughts Algorithm of BPNN Algorithm of TD learning Algorithm of α β minimax method Summary of this chapter Analysis of the effect of AI The speed of computer computing Compare search depth Compare size of the checker-board The level of intelligent computer Compare search depth Compare the numbers of learning Compare parameters γ Compare parameters λ Advantages and disadvantages Summary of this chapter Summary and Looking Ahead Conclusion Future work Acknowledgements 46 Bibliography 48 Appendix A: Other important pseudo codes 49 Appendix B: The checker board of draughts 51 2

5 Chapter 1 Introduction 1.1 Preface In the field of artificial intelligence research, the formalization method based on agent provides a unified framework for the modeling, design and implementation of intelligent system. Nowadays, artificial intelligence has achieved great success, especially in the field of chess. On 11 May 1997, Deep Blue became the first computer chess-playing system to beat a reigning world chess champion, Garry Kasparov. In March 2016, AlphaGo beat Li Shishi, became the first computer chess program win the professional Go chess without reduce pieces. Draughts also has been solved by AI. Draughts is a two-person board game. Its history is longer than the chess, the ancestor for the Middle East checkers and 8 8 draughts are official event of the first World Mind Sports Games. From 1952 to 1962, Arthur Samuel, IBM s employee, wrote the first checkers program. July 2007, Canadian computer scientists officially announced that the draughts have been solved[1]. The decision theory, which is represented by the Markov decision process, provides an important basis in terms of theory and algorithm for the optimal solution of artificial intelligence. MDP model is very useful tool in the application of solving checkers. Completely solving large MDPs model exactly is usually intractable due to the curse of dimensionality - that is, the size of the state space increases exponentially with the number of states[2]. We need the help of some other knowledge. 3

6 1.2. STRUCTURE OF THE PAPER 1 Artificial neural networks are usually optimized by a mathematical method based on mathematical statistical methods. By means of statistical standard mathematical methods we can obtain a large number of local structural spaces that can be expressed by functions. We can use neural networks to approximate any function, proved by Cybenko theorem. BP(Back Propagation) neural network, also known as multi-layer feedforward neural network, is a more mature method in artificial neural network, using non-feedback multi-layer forward network topology, BP algorithm is also called back propagation algorithm. In my paper, we use the BP neural network to approximate the state value function. Reinforcement learning is a field of machine learning, temporal difference learning is the core idea of reinforcement learning. We need to optimize the neural network by temporal difference learning. Minimax search method is one of the most useful method to search the best action with finite depth. α β search method can reduce lots of unnecessary nodes so that we can improve the compute speed. 1.2 Structure of the paper First, I will give some theoretical knowledge in chapter 2, including the discrete-time Markov process and discrete Markov decision process theory, neural network, temporal difference learning and minimax search method. All of them will be adapted in the model to solve the problem. In chapter 3, I will give a introduction about the how to establish the MDP model for realizing AI of draughts. A Markov decision process is a 5-tuple (states, actions, transfer matrix, reward, discount factor). The difficulty is how to calculate the transfer matrix when faced with the actual model and how to decide the return function. Is there any other method to solve it? In chapter 4, I will introduce some important algorithms of Realizing the draughts AI. And in chapter 5, I will evaluate this model and the degree of artificial intelligence. Chapter 6 is the summary of this paper. 4

7 1.3. INTRODUCTION OF DRAUGHTS Introduction of draughts Draughts is one of the world s oldest and most popular puzzle game. Draughts developed on the basis of checkers are popular in many countries. The International Checkers Association has more than 50 Member States. Many mathematicians and computer experts study the procedures to break checkers. Until July 2007, Canadian computer scientists Jonathan Schaeffer officially announced that the British checkers have been studied, the program called Chinook, they said the program can find the best way to play the chess, if both sides in accordance with the best way to play chess, Then the chess game will end with the Bureau. But he just solved the 8 8 variant of draughts[1]. In my paper, I will try to realize the artificial intelligence of draughts by using Markov Decision Process and Monte Carlo method. Draughts game is developed by Codeblocks on Windows 10 platform. In this game, two computers or one player with one computer can play the game. In the design process, we use the object-oriented programming method. Because many functions need to access some data, these data are often designed as global variables Rules Draughts is played by two opponents, on opposite sides of the checker board. One player has the black pieces; the other has the white pieces. Players alternate turns. All pieces are only walking in the black lattice. Move and Jump: The men can only move one grid in the diagonal line and it can not go backwards. When Black and white two pieces tightly connected to a slash, such as it s the turn of a player, the other side of the opponent s piece is just an empty chess bit, then you can skip the opponent s piece, eat and remove from the board. Before the start of the game, on both sides of the checker board are men, in the game process, when a piece of men come or jump to the other side of the bottom line to stop, you can change to kings, but you can only enjoy the right of the king from the next step. 5

8 1.3. INTRODUCTION OF DRAUGHTS 1 The kings can move in a long distance in the diagonal line, and the king can go backwards. When Black and white pieces in the same diagonal line, such as the turn of a player, there is an empty chess bit in the other side of the opponent s piece, then you can skip the opponent s piece, eat and remove from the board. The kings can jump several grids! The rules of eating: When there is a jump or jump continuously opportunities, regardless of whether they are beneficial to you, you have to jump, especially the king. After a piece eat the opponent s pieces, if it can continue to eat a new piece of enemy, it must eat, until you can not eat so far. If there are two routes or two pieces can eat opponent s pieces, then whether or not for y- our own benefit, you must eat as more pieces as you can. Game over: 1. If all the pieces are eaten by the enemy, you lost. 2. If all the remaining pieces on the board, was blocked by the enemy, no piece can move, you lost. 3. The game to the last without any possible victory over the both player, the result is draw Computer complexity Draughts is played on an p q board. It is PSPACE hard ( In computational complexity theory, PSPACE is the set of all decision problems that can be solved by a Turing machine using a polynomial amount of space.) to determine whether a specified player has a winning strategy. And if a polynomial bound is placed on the number of moves that are allowed in between jumps (which is a reasonable generalization of the drawing rule in standard Checkers), then the problem is in PSPACE, thus it is PSPACE complete[3]. However, without this bound, Checkers is EXPTIME complete[4]( In computational complexity theory, the complexity class EXPTIME is the set of all decision problems that have exponential runtime.). 6

9 Chapter 2 Fundamentals knowledges In this chapter, I will introduce some fundamentals knowledges for solving the problem. If we want to realize the artificial intelligence (AI) of the draughts through the Markov decision process, we must first have a good understand of the Markov chain. And we need the help of neural network and some knowledge of Reinforcement learning. We only consider the discrete-time Markov process since we do not use the continuous-time Markov process in solving the problem of implementing artificial intelligence of draughts. The Markov process (MP) which defined on the discrete state space is also called Markov chain (MC). The size of checker board of traditional draughts is or 8 8, the number of empty lines in the middle is 2. In my paper, the size of checker board of draughts is p q, the number of empty lines in the middle is r, where p,q,r are chosen by players, but they are all should be even numbers and r< p Fundamentals of the discrete-time Markov chains A Markov chain is a stochastic process with the Markov property. The term Markov chain refers to the sequence of random variables such a process moves through, with the Markov property defining serial dependence only between adjacent periods (as in a chain ). It can thus be used for describing systems that follow a chain of linked events, what happens next depends only on the current state of the system[5]. Definition (Markov chains)[6] A Markov chain on a finite or countably infinite state space S is a family of S-valued random variables {X n : n 0} with the property that, for all 7

10 2.1. FUNDAMENTALS OF THE DISCRETE-TIME MARKOV CHAINS 2 n 0 and (i 0,...,i n, j) S n+2, P(X n+1 = j X 0 = i 0,...,X n = i n )=(P) in j (2.1.1) Where P is a matrix all of whose entries are non-negative and each of whose rows sum to 1. Equivalently P(X n+1 = j X 0,...,X n )=(P) Xn j (2.1.2) Remark. It should be clear that (2.1.2) is a mathematically precise expression of the idea that, when a Markov chain jumps, the distribution where it lands depends only on where it was at the time and not on where it was in the past. Definition [7] A transition matrix describes a Markov chain X t over a finite state space S with cardinality n. If the probability of moving from i to j in one step is P( j i)=p i, j, the transition matrix P is given by using P i, j as the i th row and j th column element, e.g., P 1,1... P 1, j... P 1,n P= P i,1... P i, j... P i,n P n,1... P n, j... P n,n Since the total of transition probability from a state i to all other states must be 1, so that n P i, j = 1 j=1 Property (Markov Property)[6] The Markov property refers to the memoryless property of a stochastic process. In mathematical, if m,n 1 and F : S n+1 R is either bounded and non negative, then E[F(X m,x m+1,...,x m+n ) X 0 = i 0,...,X m = i m ]=E[F(X 0,X 1,...,X n ) X 0 = i m ] (2.1.3) A stochastic process has the Markov property if the conditional probability distribution of future states of the process (conditional on both past and present values) depends only on the present state; that is, given the present state, the future state does not depend on the past state. 8

11 2.2. BASIC MODEL OF DISCRETE MARKOV DECISION PROCESS Basic model of Discrete Markov Decision Process Before Markov Decision Process, we must know Markov reward process. In probability theory, a Markov reward process is a Markov process which extends a Markov chain by adding a reward rate to each state. An additional variable records the reward accumulated up to the current time. Features of interest in the model include expected reward at a given time and expected time to accumulate a given reward. Definition (Markov reward process) A Markov Reward Process(MRP) is a tuple S, P, R, γ. S is a finite set of states. P is a state transition probability matrix, P ij = P[S t+1 = j S t = i]. R is a reward function, R s = E[R t+1 S t = s]. γ is a discount factor, γ [0,1]. Definition (Return) The return G t is the total discounted reward from time step t. G t = R t+1 + γr t = i=0 γ i R i+t+1 (2.2.1) A Markov decision process (MDP) is a Markov reward process with decisions. It is an environment in which all states are Markov. Definition (Markov decision process) Markov Decision Process(MDP) is a tuple S, A, P, R, γ. S is a finite set of states. A is a finite set of actions. A(i) is the action space of state i S. Then, A= i S A(i). P is a state transition probability matrix, P a ij = P[S t+1 = j S t = i,a t = a]. R is a reward function, R a s = E[R t+1 S t = s,a t = a]. γ is a discount factor, γ [0,1] The simple decision model in which the actions only rely on the state they are in Definition (MDP policies) A policy π is a distribution over actions given states at the moment t, π(a s)=p[a t = a S t = s] (2.2.2) 9

12 2.2. BASIC MODEL OF DISCRETE MARKOV DECISION PROCESS 2 A policy fully defines the behaviour of an agent. MDP policies only depend on the current state (not the history). Policies are stationary (time-independent), for any t > 0, we have A t π t ( S t ). On the other hand consider, π( s) is a mapping function from the state space S to the action space A(s). Given a MDP M = S,A,P,R,γ and policies π( s) for each s S. The state and action sequence S 1,a 1,S 2,a 2,...,S N,a N, is a finite MDP. Where a i A(S i ),i=1,2,...,n. Let Ps,s π = π(a s)p a s,s (2.2.3) a A(s) Rs π = π(a s)r a s (2.2.4) a A(s) Definition (State Value Function) The state value function v π (s) of an MDP is the expected return starting from state s, and then following the policy π v π (s)=e[g t S t = s] (2.2.5) Definition (Action Value Function) The action state value function q π (s,a) of an MDP is the expected return starting from state s, taking action a A(s), and then following the policy π Theorem (Bellman Expectation Equation) q π (s,a)=e[g t S t = s,a t = a] (2.2.6) v π (s)= π(a s)q π (s,a)= π(a s)(r a s + γ P a s,s v π(s )) (2.2.7) a A(s) a A(s) s S q π (s,a)=r a s + γ s S P a s,s v π(s )=R a s + γ s S P a s,s a A(s ) The Bellman equation can be expressed concisely using matrices, π(a s )q π (s,a ) (2.2.8) v π = R π + γp π v π (2.2.9) with direct solution v π =(I γp π ) 1 R π (2.2.10) 10

13 2.2. BASIC MODEL OF DISCRETE MARKOV DECISION PROCESS The construction of optimal Markov policy under the total reward of time-bounded Definition The optimal state value function v (s) is the maximum value function over all policies v (s)= max π A(s) v π(s) (2.2.11) The optimal action value function q (s,a) is the maximum action value function over all policies q (s,a)= max π A(s) q π(s,a) (2.2.12) The optimal value function specifies the best possible performance in the MDP. An MDP is solved when we know the maximise value of q (s,a). If we know q (s,a), we immediately have the optimal policy. Theorem The optimal Markov policy under the total reward of time-bounded exists in all the Markov policy but not necessarily unique. In mathematical, for any Markov decision process, There exists an optimal policy π that is better than or equal to all other policies, v π (s) v π (s), π, s. All optimal policies achieve the optimal value function, v π (s)=v (s). All optimal policies achieve the optimal action-value function, q π (s,a)=q (s,a) By Theorem 2.2.1, we can get v (s)= max a A(s) Ra s + γ P a ss v (s ) (2.2.13) s S q (s,a)=r a s + γ P a ss max q (s,a ) (2.2.14) s S a A(s ) In practical problems, sometimes even the best Markov policy exists, but the calculation is too large or computation time is too long, it becomes virtually impossible. And, as defined below ε Optimal Markov strategies more practicable than optimal Markov policy. Definition We said a ε A(s) is a ε Optimal Markov strategies, if q(s,a ε )>q (s,a) ε (2.2.15) 11

14 2.3. BACK PROPAGATION NEURAL NETWORK 2 Remark. When the computational complexity of the best Markov policy is non-polynomial, the ε best Markov policy can often reduce the computational complexity to the polynomial, and it can be seen that the cost of this ε can bring great benefits. Remark. It is not difficult to find q (s,a) by computer algorithm, The problem is that when the action set A is large, the amount of computation will be very large and even difficult to complete in the allowed time, so instead of using the ε best Markov policy. 2.3 Back propagation neural network As we know, for a p q checker board, there are 5 pq 2 states, it is a large number for computer when p,q 6. It impossible to use a tabular representation for the values of the different states. So it is necessary to use a approximation function that is chosen by artificial neural network. Now I will introduce some knowledge about neural network that we need in this paper. The neural network that I used is named Back propagation neural network (BPNN). It is a common method of training artificial neural networks and used in conjunction with an optimization method such as gradient descent. A neural network consists of three parts, input layer, hidden layer and output layer. The input layer and output layer are only 1 layer, but the hidden layer can have many layers, I only introduce the model with 1 layer since I just use 1 hidden layer in this paper. The neural network has three layers in total. Input layer, hidden layer and output layer are made up of neurons. Suppose there are N 1,N 2,N 3 neurons for input layer, hidden layer, output layer, respectively. Three-layer back propagation neural network structure shown in Figure 2.1. X =(x 1,x 2,...,x N1 ) is the value of input layer, Y =(y 1,y 2,...,y N2 ) is the value of hidden layer, Z = (z 1,z 2,...,z N3 ) is the value of output layer. w i j,i = 1,2,...,N 2, j = 1,2,...,N 1 are the weights between input layer and hidden layer. v i j,i = 1,2,...,N 3, j = 1,2,...,N 2 are the weights between hidden layer and output layer. T = (T 1,T 2,...,T N3 ) is the expect value. In other words, (X,T) is the a sample. 12

15 2.3. BACK PROPAGATION NEURAL NETWORK 2 Figure 2.1: Three layers BPNN Remark. (1) When the number of hidden layer neurons is too small, not very good learning network, the number of training iterations will be more, training accuracy is not high. (2) When the number of hidden layer neurons is too large, the more powerful the network function, the higher the accuracy, the number of training iterations is also large, may appear over fitting phenomenon. Now we give the relation of X and Y, Y and Z [8], y i = f( z i = f( N 1 j=1 N 1 j=1 x j w i j ), i=1,2,...,n 2 (2.3.1) y j v i j ), i=1,2,...,n 3 (2.3.2) Where f() is a transfer function. The transfer function adopted by the BPNN is a non linear transformation function which is named sigmod function. It is characterized by the function 13

16 2.3. BACK PROPAGATION NEURAL NETWORK 2 Figure 2.2: Unipolar S type function curve Figure 2.3: Bipolar S type function curve itself and its derivatives are continuous, and thus very convenient in dealing with. There are two sigmod functions [8]. Unipolar S type function curve shown in figure 2.2, Bipolar S type function curve shown in figure 2.3, f 1 (x) = 1 1+e x (2.3.3) f 2 (x) = ex e x e x + e x (2.3.4) You can find that the value field of f 1 (x) is (0,1), the value field of f 2 (x) is ( 1,1). And it is easily to know that f 1 (x)= f 1(x)(1 f 1 (x)) and f 2 (x)=(1 f 2(x)) 2. Here I only talk about bipolar S type function, Because I choose to use the bipolar S type function in the model, I will give the reason later. Definition [8] When the value of output layer is not equal to the expect value, there is 14

17 2.3. BACK PROPAGATION NEURAL NETWORK 2 a output error E, defined as follows E = 1 (T Z)2 2 = 1 2 = 1 2 = 1 2 N 3 k=1 N 3 k=1 N 3 k=1 (T k z k ) 2 (T k f 2 ( (T k f 2 ( N 2 j=1 N 2 j=1 v k j y j )) 2 N 1 v k j f 2 ( i=1 w ji x i ))) 2 (2.3.5) From formula 2.3.5, we can know that the error E is a function of w i j,v i j. In order to reduce the error E, we should adjust the weights by let the weights is inversely proportional to error gradient, that is, w i j = α E = α(1 y 2 w i)x j i j N 3 k=1 (T k z k )(1 z 2 k )v ik i=1,2,...,n 2 j=1,2,...,n 1 v jk = α E v jk = α(t k z k )(1 z 2 k )y j (2.3.6) j=1,2,...,n 3 k=1,2,...,n 2 Where α is learning rate. It is belong to (0,1). Learning rates are generally chosen to be , and large learning rates can lead to system instability, but small learning rates lead to convergence too slow and require longer training times. For more complex networks, different learning rates may be required at different locations on the error surface. In order to reduce the number of training times and time to find the learning rate, a more appropriate approach is to use a changing adaptive learning rate to set the network different learning rates at different stages. After that, we can get the new weights, w i j = w i j + w i j, i=1,2,...,n 2 j = 1,2,...,N 1 v jk = v jk + v jk, j=1,2,...,n 3 k=1,2,...,n 2 (2.3.7) The theory has proved that a network with deviations and at least one S-type hidden layer plus a linear output layer can approximate any rational function, increasing the number of layers can further reduce the error and improve the accuracy, but also the network is complicated. The improvement of network training accuracy can be also achieved by using a hidden layer and increasing the number of neurons, which is much simpler in structuring than adding network layers. 15

18 2.4. TEMPORAL DIFFERENCE LEARNING METHOD Temporal difference learning method Temporal difference(td) learning is declared to be a reinforcement learning method. It is one of the most used approaches for policy evaluation. For deriving optimal control, policies have to be evaluated. This task requires value function approximation. At this point TD methods find application. TD methods are called bootstrapping methods, as they do not learn by the difference to the final outcome but the difference between each update step. Instead of a single update, TD methods calculate T 1 updates for a episode of T time steps[9]. TD learning is particularly well suited to game playing because instead of forming pairs between the actual outcome and each state encountered in the game, instead it updates it s prediction at each time step to the prediction at the next time step[10]. TD learning aims to achieve a approximation function as V θ (S) close to the state value function V(S) as possible by minimizing the mean squared error MES which defined as MSE= 1 n n k=1 (V θ (S i ) V(S i )) 2 (2.4.1) As V S is unknown, it is estimated by applying Equation on the current approximation V(S t ), V(S t ) E(R t + γv θ (S t+1 )) (2.4.2) The application of the Bellman Equation is the core idea of Temporal Difference Learning and allows to calculate the error denoted by Equation However analytic computation of the minimum of the error is not possible for systems with huge state spaces. Instead, a local minimum is searched numerically by Stochastic Gradient Descent. The method calculates new search weights θ by following a approximation of the gradient of the error function[9]. V θ (S i )= V θ (S i ) α MSE = V θ (S i ) α(v θ (S i ) V θ (S i )) V θt (S t ) (2.4.3) Where α is a learning rate factor which is used to adjust the step size of the SGD method and prevent overshooting, it has been introduced before. The value function approximation applied on equation 2.4.3, calculating the gradient. 16

19 2.5. α β MINIMAX SEARCH METHOD 2 To reduce the number of computations needed, the gradient is approximated by φ(s t ). The update function of the TD learning method can be displayed as: V θt+1 (S t )= V θt (S t )+αδ t e t δ t = R t+1 + γv θt (S t+1 ) V θt (S t ) (2.4.4) e t = φ(s t ) Where δ t denotes the temporal difference and e t denotes the approximation of the gradient V θt (S t ). e t allows the method to carry rewards backward over the sampled trajectory without the need to store the trajectory itself. The reach of this effect is depending on the factor λ [0,1]. The factor λ determines the degree to which extend the changes are propagated. the value of λ is very important on real tasks that the algorithm is named TD(λ)[9]. [11]One kind of attenuation way is { γλet 1 (S)+1 S=S t e t (S)= γλe t 1 (S) S = S t When λ = 1, it is the same with Monte Carlo learning, while a λ = 0 results in a one-step lookahead. 2.5 α β minimax search method minimax search method Minimax search method generally used in game search, such as: Go, backgammon, chess and so on. There are three possible outcomes: victory, failure and draw. Violence search, if you want to search through violence, the final result of the search, the depth of the search tree is too large, the machine can not meet, generally provide a search depth, Depth-first search within this depth range. We assume that two people involved in the competition are Black and White, Suppose we are black. We always want to choose the action that strives for the greatest benefit each step, it is named Max process. But we don t know the level of white, we suppose he always strives for the greatest benefit each step, too. It means he strives to let me get the smallest benefit 17

20 2.5. α β MINIMAX SEARCH METHOD 2 each step, it is named Min process. So the game is actually Max process and Min process alternately the process until game over or reached the search depth. In the search tree, the node that represents black is the max node, indicating that the node of white is a min node. Example Figure 2.4 is a Minimax search tree for black player. The root node is a max node, the depth of this search tree is 3. The value of each leaf node(depth=3) was calculated Figure 2.4: Minimax tree for black by the state value function, obtaining the values shown. When depth=2, the nodes are min nodes, it is the min process, it equal to the smallest value of its child nodes. For example, the first node equal to max(1,-1), that is 1. When depth=1, the nodes are max nodes. For example, the second node equal to min(5,-1), that is -1. The root node is a max node, equal to max(1,-1)=1. So the action should be the first node of the second layer if we are in the state of the root node. We will get a state that its value is at least 2. However, for the white player, he will try to find a state with the smallest value. The Minimax search tree for white player as figure 2.5. He will get a state that its value at most 3. 18

21 2.5. α β MINIMAX SEARCH METHOD 2 Figure 2.5: Minimax tree for white α β search method For general minimax search method, even if only a few choices each step, the number of search positions will grow exponentially as the search depth. It is difficult for a common computer to accomplish. But based on general minimax search method, α β search method can reduce many unnecessary nodes, it can save a lot of time to calculate. In the figure 2.5 of example 2.5.1, firstly, we get the value of the first node of the second layer is 3. And later, we get the value of the fourth node of the third layer is 5, since the second layer is Max process, so the value of the second node of the second layer is not small than 5. Then, the value of root node is equal to min(3,5). It is not necessary to calculate the value of the fifth node of the third layer. When the algorithm is implemented, two values are introduced for each node: α and β, respectively, which represent the lower and upper limits of the node s estimate, which constitute an interval, it means the state value belong to (α,β). In the root node, α and β defined by ourselves, we can let it be and +, respectively. The α and β values of other nodes are inherited from their parent nodes in the depth-first search process. After we get the value of the fourth node of the third layer, if it is lager than 3, we don t need to search the fifth node. In fact, 3 is the upper limit of the root nodes after updating, β = 3, 5 is the lower limit of the second node of the second layer, it is α. when α β, we don t need to the other child 19

22 2.6. SUMMARY OF THIS CHAPTER 2 nodes of this node. α and β are updated in the depth-first search process. For Max process, if the return value of child node lager than α, we let α be this value. For Min process, if the return value of child node little than β, let β equal to this value. Remark. We can see that α β search method can reduce many unnecessary nodes, but it depend on the order of the nodes appear. 2.6 Summary of this chapter This chapter introduced the knowledge that the paper need in detail. A Markov chain requires a no memory nature: The probability distribution of the next state can only be determined by the current state. MDP provides the basic theoretical model of action decision in uncertain environments. Bellman Expectation Equation writes the value of a decision problem at a certain point in time in terms of the pay off from some initial choices and the value of the remaining decision problem that results from those initial choices. BPNN is a common method of training an artificial networks since it can be trained to approximate any function. This chapter give the detailed principle how the three layer neural network run with the bipolar S type function curve. TD(λ) learning is one of the most important reinforcement method that combined with Monte Carlo learning and dynamic programming methods. Instead of learning 1 times each game, we can learn each step of the game with TD(λ) learning. This chapter introduced how the expected state value updated by TD(λ) learning method in theory. Minimax search method choose the best action for the player by traversing all the possibilities with a finite depth. But the depth is limited by the time. α β search method is an effective way to save time of searching. This chapter explained how the two method searched. 20

23 Chapter 3 Establish MDP model for draughts In chapter 1, I have given a introduction about draughts. Through the theoretical knowledge of chapter 2, we can establish a Markov decision process model for draughts. As we know, a MDP model must include the state space S, the action space A, the transition matrix P and the reward function f. And we need a discount factor γ [0,1], since immediate rewards are more important than delayed rewards. Now, Let s begin to decide the structure of the model in this chapter. The pieces on the checker board is a state of the model, move or jump a piece is an action of the model, the transition matrix is decide by the probability of the starting point and end point of the selected pieces, but there are too many states to write a transition matrix. However, the discount factor decide decide by comparing the intelligent effect of the computer. There is only a reward when the game over. 3.1 State space In the checker board, there are p q grids, and each grid is black kings, black men, white kings, or empty. Let the checker board be a matrix with the size of p q, and let 1 if it is a black kings piece in the position(i, j) 0.5 if it is a black men piece in the position(i, j) S i, j = 0 if it is no piece in the position(i, j) 0.5 if it is a white men piece in the position(i, j) 1 if it is a white kings piece in the position(i, j) 21

24 3.2. ACTION SPACE 3 As we know, when i + j is even number, there is always no piece in the position(i, j) of the checker board, so we have S i, j = 0 if i+ j is divisible by 2. And the initial state matrix S 0 = The set of all possible scenarios on the checker board, that is, all possible matrices S form a state space S. When one of the player move or jump a piece, the state matrix change from S i to S i+1. We know that S i+1 is just decide on S i and the piece that the player moved, it means that there is no relation between S i+1 and S 0, S 1,..., S i 1. So{S n : n 0} is a discrete time Markov process, and the state space S= {Sk : k=0,1,2,...,n, and S n is the matrix of the last scenarios of the checker-board} p q And this is a finite Markov chain since the game will end in a limited time. (3.1.1) 3.2 Action space Now, the state of the checker board is S, it s your turn to move a piece. You must choose the route of the pieces that can eat the largest number of opponent s pieces. Since Different kinds of pieces have different ways to move, the actions for each piece are different, and the positions of the pieces also influence the actions. Let s discuss the actions of each situation. 22

25 3.2. ACTION SPACE 3 Suppose your pieces are white, and you move or jump the piece in the position (i 1, j 1 ) to the position (i 2, j 2 ). Suppose your choice is allowable, we know that i 1 i 2 = j 1 j 2 since the piece can only move on the diagonal line. If the piece is a men, i 1 i 2 = j 1 j 2 =1 or 2. There are 2 actions when the piece moved since it can t go back forward, and there are 4 actions when the piece jumped. So there are 6 actions(figure 3.1) for this piece in total. Figure 3.1: The actions of men If the piece is a king, j 1 j 2 1. There are 4 actions when the piece moved since it can t go back forward, and there are 4 directions when the piece jumped, but maybe there are several actions each direction since a king can jump more than 2 grids. So there are 6 directions(figure 3.2) for this piece in total. However the number of actions decide by the position of the piece. We also need to notice that the actions is not the same for two players. By the analyse before, we have the actions are decided by the function of player(pl), the kind of piece(pi), the start position(i, j) (po), the direction(dir) and the distance that the piece move or jump(dis) a(pl, po, pi, dir, dis) (3.2.1) where { 1 if the piece of the player is white pl = 1 if the piece of the player is black (3.2.2) 23

26 3.2. ACTION SPACE 3 Figure 3.2: The actions of kings So we get the action space pi= { k if the piece is a king m if the piece is a man 1 if the piece moved to southeast 2 if the piece moved to southwest 3 if the piece moved to northwest 4 if the piece moved to northeast dir= 5 if the piece jumped to southeast 6 if the piece jumped to southwest 7 if the piece jumped to northwest 8 if the piece jumped to northeast = 1 if dir=1,2,3,4 dis = 2 if pi=m and dir=5,6,7,8 min{ p i, q j } if pi=k and dir=5,6,7,8 (3.2.3) (3.2.4) (3.2.5) A={a(pl, po, pi,dir,dis) : pl, po, pi, dir, dis are defined before} (3.2.6) For each fixed state i S,that is, for each step of game, the player and the pieces in each grids are fixed. So the actions are just decided by po,dir,dis, therefore, the action space for state i is A(i) ={a(po, dir, dis) : po,dir, dis are defined before} (3.2.7) 24

27 3.3. TRANSITION MATRIX, DISCOUNT FACTOR, REWARD FUNCTION AND STATE VALUE 3 And the action space is n A= A(i) (3.2.8) i=0 3.3 Transition matrix, discount factor, reward function and state value Transition matrix Since the amount of states is too large to write the transpose matrix explicitly, The amount of storage required to create a lookup table is too large, and we don t know the probability in practice, we couldn t solve the problem by traditional method. However, we can solve the problem by neural network. This will be explained in Chapter 4. Discount factor We can t know the exact value of γ now, but we can let it equal to 0.6, 0.7, 0.8, 0.9 respectively by comparing the effect of AI in practice. But how to compare? We could training the same times for each γ, and let all of them play with the random computer 50 times, compare the odds of winning. Reward function We set the reward function to be 1 if game over and white palyer win R(S,a)= 0 if game isn t over or the game draw 1 if game over and black palyer win (3.3.1) State value In the beginning, all the states value are 0. Farther, The states that correspond to final boards are considered to have a value of 0, the values of the rest of the states are given by the output of the function approximation which obtained by BPNN[12]. 25

28 3.4. SUMMARY OF THIS CHAPTER Summary of this chapter In this chapter, through the actual rules of draughts, we construct a Markov decision process model that includes state space, action space, transfer matrix, reward function, and discount factor. The checker board is too large to enumerate each state. A state function effectively represents each state of the board, thus constructing a state space. The rules of draughts are relatively complex, so the action space is also very large. However this chapter researched the determining factors of the actions successfully. We can not accurately express the transfer matrix of the model. The discount factor should be given by comparison experiments. The reward function just give to the last state of each game. 26

29 Chapter 4 Algorithm of Realizing AI of the draughts After we established the model in Chapter 3, now we need to write the code the the game. I don t introduce how to write the code about the rules of the game, though it is not easy, because I the most important point is to achieve artificial intelligence. Therefore, I just introduce the three key algorithm of the code, they are BPNN, TD learning and α β search method. 4.1 Algorithm of BPNN Though the introduction of draughts in the chapter 1, we know that we will win the game when the opponent has no piece or all his piece can not move. And in the game a king is much better than a men. And the state value is also depend the position of the pieces. So I give a vector with the length of pq+2 to store the features of the checker board. Let it be X[pq+2]=(S 11,S 12,...,S 1q,S 21,...,S 2q,...S pq,move(white), move(black)) (4.1.1) where move() is a function to calculate how many pieces can move for one player. So the number of the nodes of the input layer of the neural network is pq+2. The number of the nodes of the hidden layer is decide by ourselves, we usually let it be half of the nodes of the input layer, that is pq+2, it is a integer since p and q are even numbers. There 2 is only one node of the output layer. We expect the output be 1 when white piece win the game, 1 when white piece lost and 0 when it is draw. 27

30 4.1. ALGORITHM OF BPNN 4 We choose the bipolar S type function to be the network transfer function, because we hope the value field of the output be( 1,1). And the output error is defined as The learn rate is changed with the times of learning, With the increase in the number of learning, learning rate is getting smaller and smaller. learnrate = 1 1+learntimes (4.1.2) For the first time to train the neural network, We use the pseudo-random number to generate two vectors, the sizes are (pq+2) pq+2 and pq+2. Each element of the two vector 2 2 belongs to ( 0.05,0.05). The two vector are the initial value of the weights w and v. Where w is the weights between input layer and hidden layer, v is the weights between hidden layer and output layer. They are updating by the formula and at each time learning. The learning process specific steps are as follows: (1) Give random values to the weights w and v. (2) For the state S, extract the features to vector X[pq+2], and calculate the expected value of the state S. (3) Calculate the value of the nodes of hidden layer by the network transfer function and the weights w with the formula (4) Calculate the value of output layer by the network transfer function and the weights v with the formula (5) Calculate the error by the expected value and the value of out layer. (6) Updating the weights with formula and (7) Choose another state, and go back to the second step. (8) Learning until the error small than ε, learning finished! The pseudo code of BPNN is as algorithm 1 28

31 4.1. ALGORITHM OF BPNN 4 Algorithm 1 Algorithm of Back propagation Neural Network input: transfer function f(x), feature function g(s), states and the corresponding expected value output: weights 1: Initialize the weights vector w and v. 2: repeat 3: Input a state S; 4: X[pq+2]=g(S); This is the vector of input layer 5: for each i [1,N 2 ] do Compute the values of the nodes of hidden layer 6: y i = N 1 j=1 x jw i j ; 7: y i = f(y i ); 8: end for 9: z= N 2 j=1 y jv i j ; 10: z= f(z i ); The value of output layer 11: E= 1 2 (T z)2 ; Compute the error 1 12: learnrate = 1+learntimes ; 13: for each j [1, pq : v j = α(t z)(1 z 2 )y j ; 15: v j = v j + v j ; 16: end for Updating learn rate ] do Updating weights v 17: for each i [1, pq+2] do Updating weights w 18: for each j [1, pq+2 ] do 2 19: w i j = α(1 y 2 i )x j(t z)(1 z 2 )v i ; 20: w i j = w i j + w i j ; 21: end for 22: end for 23: until (E < ε) 29

32 4.2. ALGORITHM OF TD LEARNING Algorithm of TD learning In the algorithm of TD learning, we also choose the learn rate to be formula And the discount factor has been introduced in the subsection 3.3. After the game finish, there is a Markov decision chain {(S i,a i ) i = 0,1,2,...,n}, where a 1 = a 2 =...=a n 1 = 0 and a n decide by Now, we can compute the expected state value of every state by the formula Suppose ˆv i is the expected state value of state S i. We can train the neural network by updating the weights. But how to update the weights? Let the output value of transfer function of each state equal to the corresponding expected state value, that is: f(s n )= ˆv n. f(s n 1 )= ˆv n 1 (4.2.1) f(s 0 )= ˆv 0 Remark. The equation set must learn from state S n to state S 0, because we can only calculate the expected state value from state S n to state S 0. The specific steps are as follows: (1) Select the states one by one in descending order. (2) For the state S i, compute the learn rate. (3) Compute the approximation of the gradient e t. (4) Compute f 2 (S i+1 ) and f 2 (S i ). (5) For the state S i, compute ˆv i by formula (6) Updating the weights by algorithm 1. The pseudo code of TD learning is as algorithm 2 30

33 4.3. ALGORITHM OF α β MINIMAX METHOD 4 Algorithm 2 Algorithm of TD learning input: states S i,i=0,1,..,n, the result of the game 1: i n; 2: e 1; 3: Training state S n by BPNN; 4: i i 1; 5: while i 0 do 6: learnrate = 7: e γλe; 8: ˆv i+1 f 2 (S i+1 ); 9: ˆv i f 2 (S i ); 1 1+learntimes ; 10: ˆv i ˆv i + learnrate (R t+1 + γ ˆv i+1 ˆv i ) e; 11: Training state S i by BPNN; 12: end while 4.3 Algorithm of α β minimax method Different from other game models, one player can jump many steps continuously, so it is not always Max process and Min process alternately the process. But don t worry, since for each player, he must choose the action that he can eat the most opponent s pieces all the time, therefore the nodes of every layer of the minimax search tree are belong the same player. Though sometimes the steps that the players can jump continuously lager than the search depth, we still can search the search depth and find the best action, after we execute this action, we search again... When we jump to the last step, we search again, we still searched the states after the opponent moved. Example In the figure 4.1, if the search depth is 4, We will get the value of root is 6. But if the depth is 2, we compute the value of the nodes of the third layer and choose the best action states in the first. Then, we arrived the second layer of the search layer, we compute the value of the nodes of the fourth layer and choose the best action states. At last, we arrived the third layer of the search layer, we compute the value of the nodes of the fifth layer and choose the best action states. 31

34 4.3. ALGORITHM OF α β MINIMAX METHOD 4 Figure 4.1: Minimax search tree Remark. In our model, the value of every state is belong to( 1,1), this is just an example to explain the problem. Since Minimax search method is just simplified by α β search method. I only write the pseudo code of α β search method, please see algorithm 3 and 4. 32

35 4.3. ALGORITHM OF α β MINIMAX METHOD 4 Algorithm 3 Algorithm of α β search method input: state S, α, β, depth depth, player player. output: the coordinate(end i,end j ) of best action 1: Search depth depthlimit; 2: function ALPHABETA( player, state, depth, α, β) 3: if depth=0or gameover then 4: return Evaluate(board); 5: else if player = white then 6: for each position(i, j) can move or jump do 7: ALPHABETAIJ(white, state, depth, α, β, i, j); 8: end for 9: else 10: for each position(i, j) can move or jump do 11: ALPHABETAIJ(black, state, depth, α, β, i, j); 12: end for 13: end if 14: end function 15: function ALPHABETAIJ(player, state, depth, α, β, i, j) 16: float score;; 17: if depth=0or gameover then 18: return Evaluate(board); 19: else if player = white then 20: for each position(x,y) that peice (i, j) can move or jump do 21: Piece move or jump from (i, j) to(x,y); Get a new state S 22: depth depth 1; 23: if the piece jumped and can jump continue then 24: score ALPHABETAIJ(white,S,depth,α,β,x,y); 25: else 26: score ALPHABETA(black,S,depth,α,β); 27: end if 28: if score > α then 29: α score; 30: if depth = depthlimit 1 then 31: (end i,end j ) (x,y); 33

36 4.3. ALGORITHM OF α β MINIMAX METHOD 4 Algorithm 4 Algorithm of α β search method 32: end if 33: end if 34: depth depth+1; 35: Piece move or jump from (x,y) to(i, j); Go back to the original state 36: end for 37: return α; 38: else The player is black pieces 39: for each position(x,y) that peice (i, j) can move or jump do 40: Piece move or jump from (i, j) to(x,y); Get a new state S 41: depth depth 1; 42: if the piece jumped and can jump continue then 43: score ALPHABETAIJ(black,S,depth,α,β,x,y); 44: else 45: score ALPHABETA(white,S,depth,α,β); 46: end if 47: if score < β then 48: β score; 49: if depth = depthlimit 1 then 50: (end i,end j ) (x,y); 51: end if 52: end if 53: depth depth+1; 54: Piece move or jump from (x,y) to(i, j); Go back to the original state 55: end for 56: return β; 57: end if 58: end function 34

37 4.4. SUMMARY OF THIS CHAPTER Summary of this chapter This chapter described the process and algorithm of BPNN, TD(λ) learning and Minimax search method combined with the actual situation of draughts in detail. We use a vector with the length of pq+2 to store the features of the checker board, and then calculate the value of output by transfer function. TD learning method is used to evaluate the expected state value of the game. And then updating the weights of the neural network by back Propagation the neural network. Finally, The weight converges to the optimal solution of evaluate the state value of draughts. Therefore, we can use the transfer function to calculate the state value of each state. At last, α β search method is used to choose the best actions for the AI player. 35