1、Neural Network Theory and ApplicationsHomework Assignment 2oxstarSJTUJanuary 19,2012Suppose the output of each neuron in a multilayer quadratic perceptron(MLQP)networkisxkj=fNk1Xi=1(ukjix2k1,i+vkjixk1,i)+bkjfor k=2,3,.,M and j=1,2,.,Nkwhere both ukjiand vkjiare the weights connecting the ith unit in
2、 the layer k 1 to thejth unit in the layer k,bkjis the bias of the jth unit in the layer k,Nkis the number ofunits in the k(1 k M),and f(.)is the sigmoidal activation function.1.Please design the corresponding back propagation algorithm.Ans.From the network,we haveinput=x1(1)xk=f(ukx2k1+vkxk1+bk),k=
3、2,3,.,M(2)output=xM(3)The steepest descent algorithm for the approximate mean square error isukji(m+1)=ukji(m)Fukjivkji(m+1)=vkji(m)Fvkjibkj(m+1)=bkj(m)FbkjThe net input to layer k is an explicit function of the weights and bias in that layernkj=Nk1Xi=1(ukjix2k1,i+vkjixk1,i)+bkjThereforenkjukji=x2k1
4、,i,nkjvkji=xk1,i,nkjbkj=1LetskjFnkj,skFnk1Using the chain rule of calculus,we haveuk(m+1)=uk(m)sk(x2k1)T(4)vk(m+1)=vk(m)sk(xk1)T(5)bk(m+1)=bk(m)sk(6)In order to obtain the recurrence relation,we should calculatenk+1,jnki=?PNki=1(uk+1,jix2ki+vk+1,jixki)+bk+1,j?nki=uk+1,jix2kinki+vk+1,jixkinki=uk+1,ji
5、f2(nki)nki+vk+1,jif(nki)nki=(2uk+1,jixki+vk+1,ji)f(nki)=(2uk+1,jixki+vk+1,ji)(1 xki)xkink+1nk=(2uk+1xk+vk+1)F(nk)We can now write out the recurrence relation for the sensitivity by using the chain rulein matrix form:sk=Fnk=?nk+1nk?TFnk+1(7)=F(nk)(2uk+1xk+vk+1)Tsk+1(8)The starting point,sM,for the re
6、currence relation has been derived:sM=2F(nM)(t xM)(9)2.Write a program to realize it(3 layers).Ans.cf.src/mlqp.m3.Run your program for pattern classification on the two-spiral data set,which is thesame as homework one.You can choose 10 hidden units in this problem.Ans.We can first discuss about the
7、process of convergence.When learning rate is setto 0.1 and initial values are randomly chosen.We choose MSE(Mean Squared Error)topresent the learning quality of our NN.However,the final results are classifications,so wealso present Error Rate(Figure 1).In Figure 2,we present the process of convergen
8、ce.From Figure 1b we can notice thatwhen iteration is between 200 and 250,Error Rate changed a lot.So we snapshot moredecision boundaries in this domain.Trying 3 different learning ratesWe choose three different learning rates,which are 0.1,0.5 and 1.Here we fix initialvalues to the same random valu
9、es.Results are shown in Figure 3.Trying 3 different initial valuesThree different kinds of initial values are chosen,which are random values,all 0.01s andall 0s.Here we fix learning rate to 0.1.Results are shown in Figure 4.2010020030040050060070080000.10.20.30.40.50.60.7IterationMSE TrainTest(a)Mea
10、n Squared Error010020030040050060070080000.10.20.30.40.50.60.70.80.91IterationError Rate TrainTest(b)Error RateFigure 1:Results of Ex.2.3(learning rate:0.1,initial values:random)4321012343210123 ClassificationClass 1Class 2(a)Iteration=1004321012343210123 ClassificationClass 1Class 2(b)Iteration=200
11、4321012343210123 ClassificationClass 1Class 2(c)Iteration=2144321012343210123 ClassificationClass 1Class 2(d)Iteration=2204321012343210123 ClassificationClass 1Class 2(e)Iteration=2504321012343210123 ClassificationClass 1Class 2(f)Iteration=500Figure 2:Process of Convergence305010015020025000.10.20.
12、30.40.50.60.7IterationMSE TrainTest(a)MSE(learning rate:0.5)0501001500.20.250.30.350.40.450.50.550.60.650.7IterationMSE TrainTest(b)MSE(learning rate:1)05010015020025000.10.20.30.40.50.60.70.80.91IterationError Rate TrainTest(c)ER(learning rate:0.5)05010015000.10.20.30.40.50.60.70.80.91IterationErro
13、r Rate TrainTest(d)ER(learning rate:1)Figure 3:Results of Ex.2.3(initial values:random)401002003004005006007008000.40.420.440.460.480.50.52IterationMSE TrainTest(a)MSE(initial values:0s)01002003004005006007008000.40.420.440.460.480.50.52IterationMSE TrainTest(b)MSE(initial values:0.01s)0100200300400
14、5006007008000.20.30.40.50.60.70.80.91IterationError Rate TrainTest(c)ER(initial values:0s)01002003004005006007008000.20.30.40.50.60.70.80.91IterationError Rate TrainTest(d)ER(initial values:0.01s)Figure 4:Results of Ex.2.3(learning rate:0.1)5Discussing the algorithm convergenceA.Different Learning R
15、atesComparing Figure 3 with Figure 1,we can obtain that the more learning rates are,thefaster the algorithm converges,but the more rough the MSE and ER curves are.Whenlearning rate is 0.5,correct decision boundary can still be learned.But when learning rateis as large as 1,the NN can hardly find the
16、 final correct decision boundary and the algorithmconverged to a local minimum points(ER does not reach 0).B.Different Initial ValuesComparing Figure 4 with Figure 1,we can find that setting initial values as 0s or 0.01sdoes not affect the smoothness of MSE and ER curves,but they both converged to a
17、 localminimum points.In order to understand local minimum points intuitively,we draw thedecision boundary for initial values are 0s and 0.01s(Figure 5).4321012343210123 ClassificationClass 1Class 2(a)initial values:0s4321012343210123 ClassificationClass 1Class 2(b)initial values:0.01sFigure 5:Final
18、Decision Boundaries(local optimization)Obviously,it is difficult for the algorithm to jump out of the local solution(compare itwith Figure 2f).4.Run you program for a yeast data set from UCI benchmark,which is collected fromyeast protein locations.It is a 4-class problem so that more complicated tha
19、n two-spiralproblem.Ans.This NN can not solve a 4-class problem,so we use a two phase method.Firstly,we map class 1 and class 2 to new class 0,map class 3 and class 4 to new class 1.Secondly,we map class 1 and class 3 to new class 0,map class 2 and class 4 to new class 1.Finally,we combine results o
20、f these two classification procedure(Table 1).Table 1:Two Phase MethodStep 1Step 2Combination001012103114In this exercise,we also use MSE and ER to evaluate the learning quality.But 8-dimension data is difficult to be plotted on the canvas,so decision boundaries are notprovided as we did in the last
21、 exercise.6Before trying different learning rates and initial values,we also provide our baseline(Figure 6).Note that it includes ER results of the two steps as well as the combination.However,the MSE results of combination can not be calculated because the final classifica-tion is based on the forw
22、ard two 2-class classifications results but the values calculated bysigmoid function are put away.050010001500200025000.10.150.20.250.30.350.40.450.5IterationMSE TrainTest(a)Mean Squared Error(Step 1)050010001500200025000.050.10.150.20.250.30.350.40.450.50.55IterationMSE TrainTest(b)Mean Squared Err
23、or(Step 2)050010001500200025000.10.150.20.250.30.350.40.45IterationError Rate TrainTest(c)Error Rate(Step 1)050010001500200025000.20.250.30.350.40.450.50.55IterationError Rate TrainTest(d)Error Rate(Step 2)050010001500200025000.350.40.450.50.550.60.650.7IterationError Rate TrainTest(e)Error Rate(Com
24、bination)Figure 6:Results of Ex.2.4(learning rate:0.1,initial values:random)Trying 3 different learning ratesWe choose three different learning rates,which are 0.1,0.5 and 1.Here we fix initialvalues to the same random values.Results are shown in Figure 7.MSE1 is short for MSE(Step 1),ERcom stands f
25、or ER(Combination)and LR represents learning rate.Trying 3 different initial valuesThree different kinds of initial values are chosen,which are random values,all 0.01sand all 0s.Here we fix learning rate to 0.1.Results are shown in Figure 8.Note that IVrepresents initial value.Discussing the algorit
26、hm convergenceA.OverfittingFrom Figure 6,we can see the situation of overfitting.For example,while the error rateof training data has the downward trend,the error rate of testing data has a upward trendwhen it meet its minimum.As a matter of fact,this task is so hard that we can hardly leadthe error
27、 rate of testing data,even training data,to a quite low level.B.Different Learning RatesComparing Figure 7 with Figure 6,we also obtain that when learning rates is increasing,both MSE and ER curves are becoming more rough.Also,high learning rates will lead thealgorithm to be unstable and hardly conv
28、erged.C.Different Initial ValuesComparing Figure 8 with Figure 6,we can still find that setting initial values as 0s or7050010001500200025000.10.150.20.250.30.350.40.450.50.550.6IterationMSE TrainTest(a)MSE1(LR:0.5)050010001500200025000.10.150.20.250.30.350.40.450.50.550.6IterationMSE TrainTest(b)MS
29、E2(LR:0.5)050010001500200025000.20.250.30.350.40.450.50.55IterationMSE TrainTest(c)MSE1(LR:1)050010001500200025000.050.10.150.20.250.30.350.40.450.50.55IterationMSE TrainTest(d)MSE2(LR:1)050010001500200025000.050.10.150.20.250.30.350.4IterationError Rate TrainTest(e)ER1(LR:0.5)050010001500200025000.
30、20.250.30.350.40.450.50.55IterationError Rate TrainTest(f)ER2(LR:0.5)050010001500200025000.050.10.150.20.250.30.350.4IterationError Rate TrainTest(g)ER1(LR:1)050010001500200025000.20.250.30.350.40.450.50.55IterationError Rate TrainTest(h)ER2(LR:1)050010001500200025000.250.30.350.40.450.50.550.60.650
31、.7IterationError Rate TrainTest(i)ERcom(LR:0.5)050010001500200025000.250.30.350.40.450.50.550.60.650.7IterationError Rate TrainTest(j)ERcom(LR:1)Figure 7:Results of Ex.2.4(initial values:random)050010001500200025000.10.150.20.250.30.350.40.450.5IterationMSE TrainTest(a)MSE1(IV:0s)0500100015002000250
32、00.050.10.150.20.250.30.350.40.450.50.55IterationMSE TrainTest(b)MSE2(IV:0s)050010001500200025000.10.150.20.250.30.350.40.450.5IterationMSE TrainTest(c)MSE1(IV:0.01s)050010001500200025000.050.10.150.20.250.30.350.40.450.50.55IterationMSE TrainTest(d)MSE2(IV:0.01s)050010001500200025000.20.250.30.350.
33、4IterationError Rate TrainTest(e)ER1(IV:0s)050010001500200025000.250.30.350.40.450.50.550.6IterationError Rate TrainTest(f)ER2(IV:0s)050010001500200025000.20.250.30.350.4IterationError Rate TrainTest(g)ER1(IV:0.01s)050010001500200025000.250.30.350.40.450.50.550.6IterationError Rate TrainTest(h)ER2(IV:0.01s)050010001500200025000.350.40.450.50.550.60.650.7IterationError Rate TrainTest(i)ERcom(IV:0s)
