Solved Problems Solved Problems P Solve the three simle classification roblems shown in Figure P by drawing a decision boundary Find weight and bias values that result in single-neuron ercetrons with the chosen decision boundaries (a) (b) (c) Figure P Simle Classification Problems First we draw a line between each set of dark and light data oints (a) (b) (c) The next ste is to find the weights and biases The weight vectors must be orthogonal to the decision boundaries and ointing in the direction of oints to be classified as (the dark oints) The weight vectors can have any length we like w w (a) (b) (c) w Here is one set of choices for the weight vectors: (a) w T (b) w T (c) w T -
Percetron Learning Rule Now we find the bias values for each ercetron by icking a oint on the decision boundary and satisfying Eq (5) b b w T w T This gives us the following three biases: (a) b (b) b (c) b 6 We can now check our solution against the original oints Here we test the first network on the inut vector T a hardlim( w T b) hardlim hardlim( 6)» ans We can use MATLAB to automate the testing rocess and to try new oints Here the first network is used to classify a oint that was not in the original roblem w[- ]; b ; a hardlim(w*[;]b) a P Convert the classification roblem defined below into an equivalent roblem definition consisting of inequalities constraining weight and bias values t t 3 t 3 t Each target t i indicates whether or not the net inut in resonse to i must be less than or greater than or equal to For examle since is we t -
Solved Problems know that the net inut corresonding to to Thus we get the following inequality: must be greater than or equal Alying the same rocedure to the inut/target airs for { t } { 3 t 3 } and { t } results in the following set of inequalities W b b w w w b w w w w b () i b ( ii) b < ( iii) b < ( iv) Solving a set of inequalities is more difficult than solving a set of equalities One added comlexity is that there are often an infinite number of solutions (just as there are often an infinite number of linear decision boundaries that can solve a linearly searable classification roblem) However because of the simlicity of this roblem we can solve it by grahing the solution saces defined by the inequalities Note that w only aears in inequalities (ii) and (iv) and w only aears in inequalities (i) and (iii) We can lot each air of inequalities with two grahs ii w w iv b iii i b Any weight and bias values that fall in both dark gray regions will solve the classification roblem Here is one such solution: W 3 b 3-3
Percetron Learning Rule P3 We have a classification roblem with four classes of inut vector The four classes are class : class : 3 class 3: 5 6 class : 7 8 Design a ercetron network to solve this roblem To solve a roblem with four classes of inut vector we will need a ercetron with at least two neurons since an S-neuron ercetron can categorize S classes The two-neuron ercetron is shown in Figure P Inut Hard Limit Layer x W x b x n Figure P Two-Neuron Percetron LetÕs begin by dislaying the inut vectors as in Figure P3 The light circles indicate class vectors the light squares indicate class vectors the dark circles indicate class 3 vectors and the dark squares indicate class vectors A two-neuron ercetron creates two decision boundaries Therefore to divide the inut sace into the four categories we need to have one decision boundary divide the four classes into two sets of two The remaining boundary must then isolate each class Two such boundaries are illustrated in Figure P We now know that our atterns are linearly searable x a hardlim (W b) a x -
Solved Problems 3 Figure P3 Inut Vectors for Problem P3 3 Figure P Tentative Decision Boundaries for Problem P3 The weight vectors should be orthogonal to the decision boundaries and should oint toward the regions where the neuron oututs are The next ste is to decide which side of each boundary should roduce a One choice is illustrated in Figure P5 where the shaded areas reresent oututs of The darkest shading indicates that both neuron oututs are Note that this solution corresonds to target values of class : t t class : t 3 t class 3: t 5 t 6 class : t 7 t 8 We can now select the weight vectors: -5
Percetron Learning Rule w 3 and w Note that the lengths of the weight vectors is not imortant only their directions They must be orthogonal to the decision boundaries Now we can calculate the bias by icking a oint on a boundary and satisfying Eq (5): b w T 3 b w T 3 In matrix form we have Figure P5 Decision Regions for Problem P3 W wt 3 and b wt which comletes our design P Solve the following classification roblem with the ercetron rule Aly each inut vector in order for as many reetitions as it takes to ensure that the roblem is solved Draw a grah of the roblem only after you have found a solution -6
Solved Problems t t 3 t 3 t Use the initial weights and bias: W( ) b( ) We start by calculating the ercetronõs outut a using the initial weights and bias for the first inut vector a hardlim( W( ) b( ) ) hardlim hardlim( ) The outut a does not equal the target value t so we use the ercetron rule to find new weights and biases based on the error e t a T W( ) W( ) e ( ) b( ) b( ) e ( ) We now aly the second inut vector bias using the udated weights and a hardlim( W( ) b( ) ) hardlim hardlim( ) This time the outut a is equal to the target t Alication of the ercetron rule will not result in any changes W( ) W( ) b( ) b( ) We now aly the third inut vector -7
Percetron Learning Rule a hardlim( W( ) 3 b( ) ) hardlim hardlim( ) The outut in resonse to inut vector 3 is equal to the target t 3 so there will be no changes W( 3) W( ) b( 3) b( ) We now move on to the last inut vector a hardlim( W( 3) b( 3) ) hardlim hardlim( ) This time the outut a does not equal the aroriate target t The ercetron rule will result in a new set of values for W and b e t a T W( ) W( 3) e ( ) 3 b( ) b( 3) e We now must check the first vector again This time the outut a is equal to the associated target t a hardlim( W( ) b( ) ) hardlim 3 hardlim( 8) Therefore there are no changes W( 5) W( ) b( 5) b( ) The second resentation of of weight and bias values results in an error and therefore a new set -8
Solved Problems a hardlim( W( 5) b( 5) ) hardlim 3 hardlim( ) Here are those new values: e t a T W( 6) W( 5) e 3 ( ) 3 b( 6) b( 5) e Cycling through each inut vector once more results in no errors a hardlim( W( 6) 3 b( 6) ) hardlim 3 t 3 a hardlim( W( 6) b( 6) ) hardlim 3 t a hardlim( W( 6) b( 6) ) hardlim 3 t a hardlim( W( 6) b( 6) ) hardlim 3 t Therefore the algorithm has converged The final solution is: W 3 b Now we can grah the training data and the decision boundary of the solution The decision boundary is given by n W b w w b 3 To find the intercet of the decision boundary set : b --------- ----- -- if 3 3 w To find the intercet set : b --------- ----- -- if w -9
Percetron Learning Rule The resulting decision boundary is illustrated in Figure P6 W Figure P6 Decision Boundary for Problem P Note that the decision boundary falls across one of the training vectors This is accetable given the roblem definition since the hard limit function returns when given an inut of and the target for the vector in question is indeed P5 Consider again the four-class decision roblem that we introduced in Problem P3 Train a ercetron network to solve this roblem using the ercetron learning rule If we use the same target vectors that we introduced in Problem P3 the training set will be: t t 3 t 3 t LetÕs begin the algorithm with the following initial weights and biases: The first iteration is 5 t 5 7 t 7 W( ) 8 t 8 b( ) 6 t 6 a hardlim ( W( ) b( ) ) hardlim ( ) -3
Solved Problems e t a T W( ) W( ) e b( ) b( ) e The second iteration is a hardlim ( W( ) b( ) ) hardlim ( ) e t a T W( ) W( ) e b( ) b( ) e The third iteration is a hardlim ( W( ) 3 b( ) ) hardlim ( ) e t 3 a T W( 3) W( ) e 3-3
Percetron Learning Rule b( 3) b( ) e Iterations four through eight roduce no changes in the weights W( 8) W( 7) W( 6) W( 5) W( ) W( 3) b( 8) b( 7) b( 6) b( 5) b( ) b( 3) The ninth iteration roduces a hardlim ( W( 8) b( 8) ) hardlim ( ) e t a T W( 9) W( 8) e b( 9) b( 8) e At this oint the algorithm has converged since all inut atterns will be correctly classified The final decision boundaries are dislayed in Figure P7 Comare this result with the network we designed in Problem P3 3 Figure P7 Final Decision Boundaries for Problem P5-3
Percetron Learning Rule Exercises E Consider the classification roblem defined below: t t 3 t 3 5 t 5 t i Draw a diagram of the single-neuron ercetron you would use to solve this roblem How many inuts are required? ii Draw a grah of the data oints labeled according to their targets Is this roblem solvable with the network you defined in art (i)? Why or why not? E Consider the classification roblem defined below t t 3 t 3 t i Design a single-neuron ercetron to solve this roblem Design the network grahically by choosing weight vectors that are orthogonal to the decision boundaries» ans ii Test your solution with all four inut vectors iii Classify the following inut vectors with your solution You can either erform the calculations manually or with MATLAB 5 6 7 8 iv Which of the vectors in art (iii) will always be classified the same way regardless of the solution values for W and b? Which may vary deending on the solution? Why? E3 Solve the classification roblem in Exercise E by solving inequalities (as in Problem P) and reeat arts (ii) and (iii) with the new solution (The solution is more difficult than Problem P since you canõt isolate the weights and biases in a airwise manner) -36
Exercises E Solve the classification roblem in Exercise E by alying the ercetron rule to the following initial arameters and reeat arts (ii) and (iii) with the new solution W( ) b( ) E5 Prove mathematically (not grahically) that the following roblem is unsolvable for a two-inut/single-neuron ercetron t t 3 t 3 t (Hint: start by rewriting the inut/target requirements as inequalities that constrain the weight and bias values) a hardlims (n) n W b E6 The symmetric hard limit function is sometimes used in ercetron networks instead of the hard limit function Target values are then taken from the set [- ] instead of [ ] i Write a simle exression that mas numbers in the ordered set [ ] into the ordered set [- ] Write the exression that erforms the inverse maing ii Consider two single-neuron ercetrons with the same weight and bias values The first network uses the hard limit function ([ ] values) and the second network uses the symmetric hard limit function If the two networks are given the same inut and udated with the ercetron learning rule will their weights continue to have the same value? iii If the changes to the weights of the two neurons are different how do they differ? Why? iv Given initial weight and bias values for a standard hard limit ercetron create a method for initializing a symmetric hard limit ercetron so that the two neurons will always resond identically when trained on identical data» ans E7 The vectors in the ordered set defined below were obtained by measuring the weight and ear lengths of toy rabbits and bears in the Fuzzy Wuzzy Animal Factory The target values indicate whether the resective inut vector was taken from a rabbit () or a bear () The first element of the inut vector is the weight of the toy and the second element is the ear length t t 3 t 3 t 5 5-37
Percetron Learning Rule 3 5 t 5 3 6 t 6 7 t 7 8 t 8 i Use MATLAB to initialize and train a network to solve this ÒracticalÓ roblem ii Use MATLAB to test the resulting weight and bias values against the inut vectors iii Alter the inut vectors to ensure that the decision boundary of any solution will not intersect one of the original inut vectors (ie to ensure only robust solutions are found) Then retrain the network E8 Consider again the four-category classification roblem described in Problems P3 and P5 Suose that we change the inut vector to 3 3» ans i Is the roblem still linearly searable? Demonstrate your answer grahically ii Use MATLAB and to initialize and train a network to solve this roblem Exlain your results iii If 3 is changed to 3 5 is the roblem linearly searable? iv With the 3 from (iii) use MATLAB to initialize and train a network to solve this roblem Exlain your results E9 One variation of the ercetron learning rule is W new W old αe T b new b old αe where α is called the learning rate Prove convergence of this algorithm Does the roof require a limit on the learning rate? Exlain -38