Neural Networks For Control - Oklahoma State University-Stillwater

11113w 1, 1 10 , w 2, 1 10 , b 1 – 10 , b 2 10 ,22The network response for these parameters is shown2in Figure 10, which plots the network output a asthe input p is varied over the range [ – 2, 2 ] .Notice that the response consists of two steps, one foreach of the log-sigmoid neurons in the first layer. Byadjusting the network parameters we can change theshape and location of each step, as we will see in thefollowing discussion.The centers of the steps occur where the net input toa neuron in the first layer is zero: 1b1 0 1100-1-2b1– 10p – --------- – --------- 1 ,110w 1, 1-10(a)1w 1, 1-1-22332w 1, 2221100-1-211w 1, 1 p2222w 1, 1 1 , w 1, 2 1 , b 0 .1n131b2-101-1b-12012(d)(c)(7)12-1-220(b)Figure 11 Effect of Parameter Changes on NetworkResponse1111n 2 w 2, 1 p b 2 0 b210p – --------- – ------ – 1 .110w 2, 1(8)The steepness of each step can be adjusted by changing the network weights.32a24. Training Multilayer Networks10-1-2-1012pFigure 10 Nominal Response of Network of Figure 9Figure 11 illustrates the effects of parameter changes on the network response. The nominal response isrepeated from Figure 10. The other curves correspond to the network response when one parameterat a time is varied over the following ranges:2From this example we can see how flexible the multilayer network is. It would appear that we could usesuch networks to approximate almost any function,if we had a sufficient number of neurons in the hidden layer. In fact, it has been shown that two-layernetworks, with sigmoid transfer functions in the hidden layer and linear transfer functions in the outputlayer, can approximate virtually any function of interest to any degree of accuracy, provided sufficientlymany hidden units are available (see [HoSt89]).212– 1 w 1, 1 1 , – 1 w 1, 2 1 , 0 b 2 20 , – 1 b 1 .(9)Figure 11 (a) shows how the network biases in thefirst (hidden) layer can be used to locate the positionof the steps. Figure 11 (b) illustrates how the weightsdetermine the slope of the steps. The bias in the second (output) layer shifts the entire network responseup or down, as can be seen in Figure 11 (d).Now that we know multilayer networks are universal approximators, the next step is to determine aprocedure for selecting the network parameters(weights and biases) which will best approximate agiven function. The procedure for selecting the parameters for a given problem is called training thenetwork. In this section we will outline a trainingprocedure called backpropagation, which is based ongradient descent. (More efficient algorithms thangradient descent are often used in neural networktraining. The reader is referred to [HBD96] for discussions of these other algorithms.)As we discussed earlier, for multilayer networks theoutput of one layer becomes the input to the following layer (see Figure 8). The equations that describethis operation aream 1m 1m 1 m f(Wa bm 0, 1, , M – 1 ,m 1) for(10)where M is the number of layers in the network. Theneurons in the first layer receive external inputs:0a p,(11)

which provides the starting point for Eq. (10). Theoutputs of the neurons in the last layer are considered the network outputs:Ma a .(12)4.1. Performance IndexThe backpropagation algorithm for multilayer networks is a gradient descent optimization procedurein which we minimize a mean square error performance index. The algorithm is provided with a set ofexamples of proper network behavior:{p 1, t 1} , { p 2, t 2} , , {p Q, tQ} ,2 2.Qmnieq q 1 ( tq – aq )T( tq – aq ) .(15)q 1TF̂ ( x ) ( t ( k ) – a ( k ) ) ( t ( k ) – a ( k ) ) e ( k )e ( k ) , (16)where the expectation of the squared error has beenreplaced by the squared error at iteration k . F̂mm- ,w i, j ( k 1 ) w i, j ( k ) – α ----------m w i, j 1) mbi ( k ) F̂– α --------m- , b imm–1m(21) n im – 1 n----------- a j , --------im- 1 .m w i, j b i(22) Thereforem F̂ms i --------m- , n iFor a single-layer linear network these partial derivatives in Eq. (17) and Eq. (18) are conveniently computed, since the error can be written as an explicit F̂m m–1----------- si a j ,m w i, j(24) F̂m--------m- s i . b i(25)We can now express the approximate steepest descent algorithm asmmm m–1w i, j ( k 1 ) w i, j ( k ) – αs i a jwhere α is the learning rate.4.2. Chain Rule(23)(the sensitivity of F̂ to changes in the ith element ofthe net input at layer m ), then Eq. (19) and Eq. (20)can be simplified to(17)(18)mIf we now defineThe steepest descent algorithm for the approximatemean square error ismbi ( k wi, j a j bi .Using a stochastic approximation, we will replacethe sum squared error by the error on the latest target:Tm–1j 1QT(20)The second term in each of these equations can beeasily computed, since the net input to layer m is anexplicit function of the weights and bias in that layer:(14)where x is a vector containing all of network weightsand biases. If the network has multiple outputs thisgeneralizes to eqmS ( t q – aq )(19) F̂ n i F̂--------m- --------m- --------m- . n i b i b iq 1q 1F (x) m n i F̂ F̂-,----------- --------m- ----------mm n i w i, j w i, jQQ eqBecause the error is an indirect function of theweights in the hidden layers, we will use the chainrule of calculus to calculate the derivatives in Eq.(17) and Eq. (18):(13)where p q is an input to the network, and t q is thecorresponding target output. As each input is applied to the network, the network output is comparedto the target. The algorithm should adjust the network parameters in order to minimize the sumsquared error:F (x) linear function of the network weights. For the multilayer network the error is not an explicit functionof the weights in the hidden layers, therefore thesederivatives are not computed so easily.mm,(26)mb i ( k 1 ) b i ( k ) – αs i .(27)In matrix form this becomes:mmmW ( k 1 ) W ( k ) – αs ( am–1 T),(28)

(29)4.5. Generalization (Interpolation & Extrapolation)where the individual elements of s are given by Eq.(23).We now know that multilayer networks are universal approximators, but we have not discussed how toselect the number of neurons and the number of layers necessary to achieve an accurate approximationin a given problem. We have also not discussed howthe training data set should be selected. The trick isto use enough neurons to capture the complexity ofthe underlying function without having the networkoverfit the training data, in which case it will notgeneralize to new situations. We also need to havesufficient training data to adequately represent theunderlying function.mmb ( k 1 ) b ( k ) – αsm,m4.3. Backpropagating the SensitivitiesmIt now remains for us to compute the sensitivities s ,which requires another application of the chain rule.It is this process that gives us the term backpropagation, because it describes a recurrence relationshipin which the sensitivity at layer m is computed fromthe sensitivity at layer m 1 :ssmmMm Ḟ (n ) ( WMM – 2Ḟ (n ) ( t – a ) ,m 1 T m 1) s(30), m M – 1, , 2, 1 (31)whereTo illustrate the problems we can have in networktraining, consider the following general example. Assume that the training data is generated by the following equation:tq g ( pq ) eq ,mf mmḞ ( n ) m( n1 )00 0m( n2 ) 0mf 0(32) 0.(33)where p q is the system input, g( ) is the underlyingfunction we wish to approximate, e q is measurementnoise, and t q is the system output (network target).25m m f ( n S m )2015(See [HDB96], Chapter 11 for a derivation of this result.)a)105In some ways it is unfortunate that the algorithm weusually refer to as backpropagation, given by Eq. (28)and Eq. (29), is in fact simply a steepest descent algorithm. There are many other optimization algorithms that can use the backpropagation procedure,in which derivatives are processed from the last layer of the network to the first (as given in Eq. (31)).For example, conjugate gradient and quasi-Newtonalgorithms ([Shan90], [Scal85], [Char92]) are generally more efficient than steepest descent algorithms,and yet they can use the same backpropagation procedure to compute the necessary derivatives. TheLevenberg-Marquardt algorithm is very efficient fortraining small to medium-size networks, and it usesa backpropagation procedure that is very similar tothe one given by Eq. (31) (see [HaMe94]).We should emphasize that all of the algorithms thatwe will describe in this chapter use the backpropagation procedure, in which derivatives are processedfrom the last layer of the network to the first. Forthis reason they could all be called “backpropagation” algorithms. The differences between the algorithms occur in the way in which the resultingderivatives are used to update the 015b)1050t4.4. Variations of Backpropagation-5-10-15-20-25-30-3Figure 12 Example of Overfitting a) and Good Fit b)

Figure 12 shows an example of the underlying function g( ) (thick line), training data target values t q(large circles), network responses for the training inputs a q (small circles with imbedded crosses), and total trained network response (thin line).In the example shown in Figure 12 a), a large network was trained to minimize squared error (Eq.(14)) over the 15 points in the training set. We cansee that the network response exactly matches thetarget values for each training point. However, thetotal network response has failed to capture the underlying function. There are two major problems.First, the network has overfit on the training data.The network response is too complex, because thenetwork has too many independent parameters (61)and they have not been constrained in any way. Thesecond problem is that there is no training data forvalues of p greater than 0. Neural networks (and allother data-based approximation techniques) cannotbe expected to extrapolate accurately. If the networkreceives an input which is outside of the range covered in the training data, then the network responsewill always be suspect.While there is little we can do to improve the network performance outside the range of the trainingdata, we can improve its ability to interpolate between data points. Improved generalization can beobtained through a variety of techniques. In onemethod, called early stopping, we place a portion ofthe training data into a validation data set. The performance of the network on the validation set is monitored during training. During the early stages oftraining the validation error will come down. Whenoverfitting begins, the validation error will begin toincrease, and at this point the training is stopped.tice that the network response no longer exactlymatches the training data points, but the overall network response more closely matches the underlyingfunction over the range of the training data.Even with Bayesian regularization, the network response is not accurate outside the range of the training data. As we mentioned earlier, we cannot expectthe network to extrapolate accurately. If we want thenetwork to respond accurately throughout the range[-3, 3], then we need to provide training datathroughout this range. This can be more problematicin multi-input cases, as shown in Figure 13. On thetop graph we have the underlying function. On thebottom graph we have the neural network approximation. The training inputs were provided over theentire range of each input, but only for cases wherethe first input was greater than the second input. Wecan see that the network approximation is good forcases within the training set, but is poor for all caseswhere the second input is larger than the first input.Peaks86420-2-4-632312010-1-1-2Another technique to improve network generalization is called regularization. With this method theperformance index is modified to include a termwhich penalizes network complexity. The most common penalty term is the sum of squares of the network weights: eq-3y-3xNNPeaks864QF (x) -2Teq ρ k 2( w i, j )(34)q 120-2This performance index forces the weights to besmall, which produces a smoother network response.The trick with this method is to choose the correctregularization parameter ρ . If the value is too large,then the network response will be too smooth andwill not accurately approximate the underlying function. If the value is too small, then the network willoverfit. There are a number of methods for selectingthe optimal ρ . One of the most successful is Bayesian regularization ([MacK92] and [FoHa97]). Figure 12 b) shows the network response when thenetwork is trained with Bayesian regularization. No--4-632312010-1-1-2y-2-3-3xFigure 13 Two-Input Example of Poor Network ExtrapolationA complete discussion of generalization and overfitting is beyond the scope of this tutorial. The interest-

ed reader is referred to [HDB96], [Hayk99],[MacK92] or [FoHa97].In the next section we will describe how multilayernetworks can be used in neurocontrol applications.5. Control System ApplicationsNeural networks have been applied very successfullyin the identification and control of dynamic systems.The universal approximation capabilities of the multilayer perceptron have made it a popular choice formodeling nonlinear systems and for implementinggeneral-purpose nonlinear controllers. In the remainder of this tutorial we will introduce some of themore popular neural network architectures for system identification and control.5.1. Fixed Stabilizing ControllersFixed stabilizing controllers (see Figure 14) havebeen proposed in [Kawa90], [KrCa90], and [Mill87].This scheme has been applied to the control of robotarm trajectory, where a proportional controller withgain was used as the stabilizing feedback controller.From Figure 14 we can see that the total input thatenters the plant is the sum of the feedback controlsignal and the feedforward control signal, which iscalculated from the inverse dynamics model (neuralnetwork). That model uses the desired trajectory asthe input and the feedback control as an error signal.As the NN training advances, that input will converge to zero. The neural network controller willlearn to take over from the feedback controller.The advantage of this architecture is that we canstart with a stable system, even though the neuralnetwork has not been adequately trained. A similar(although more complex) control architecture, inwhich stabilizing controllers are used in parallelwith neural network controllers, is described in[SaSl92].NNInverse mandInput StabilizingController- PlantPlantOutputFeedbackControlFigure 14 Stabilizing Controller5.2. Adaptive Inverse ControlFigure 15 shows a structure for the Model ReferenceAdaptive Inverse Control proposed in [WiWa96]. Theadaptive algorithm receives the error between theplant output and the reference model output. Thecontroller parameters are updated to minimize thattracking error. The basic model reference adaptivecontrol approach can be affected by sensor noise andplant disturbances. An alternative which allows cancellation of the noise and disturbances includes aneural network plant model in parallel with theplant. That model will be trained to receive the sameinputs as the plant and to produce the same output.The difference between the outputs will be interpreted as the effect of the noise and disturbances at theplant output. That signal will enter an inverse plantmodel to generate a filtered noise and disturbancesignal that is subtracted from the plant input. Theidea is to cancel the disturbance and the noisepresent in the plant.5.3. Nonlinear Internal Model ControlNonlinear Internal Model Control (NIMC), shown inFigure 16, consists of a neural network controller, aneural network plant model, and a robustness filterwith a single tuning parameter [NaHe92]. The neural network controller is generally trained to represent the inverse of the plant, if the inverse exists.The error between the output of the neural networkplant model and the measurement of plant output isused as the feedback input to the robustness filter,which then feeds into the neural network controller.The NN plant model and the NN controller (if it is aninverse plant model) can be trained off-line, usingdata collected from plant operations. The robustnessfilter is a first order filter whose time constant is selected to ensure closed loop stability.

Plant DisturbanceCommandInputNNControllerSensor Noise PlantOutputPlant -AdaptationAlgorithm -NNPlant ModelNoise &Disturbanceat Plant OutputNNInverse PlantModel Tracking Error-ReferenceModelFigure 15 Adaptive Inverse Control SystemCommandInput tPlantNNPlant ModelPredictedPlantOutput -Figure 16 Nonlinear Internal Model Control5.4. Model Predictive ControlModel Predictive Control (MPC), shown in Figure 18,optimizes the plant response over a specified timehorizon [HuSb92]. This architecture requires a neural network plant model, a neural network controller, a performance function to evaluate systemresponses, and an optimization procedure to selectthe best control input.The optimization procedure can be computationallyexpensive. It requires a multi-step ahead calculation, in which the neural network model is used topredict the plant response. The neural network controller learns to produce the input selected by the optimization process. When training is complete, theoptimization step can be completely replaced by theneural network controller.5.5. Model Reference Control or NeuralAdaptive ControlAs with other techniques, the Model Reference Adaptive Control (MRAC) configuration [NaPa90] usestwo neural networks: a controller network and amodel network. (See Figure 17.) The model networkcan be trained off-line using historical plant measurements. The controller is adaptively trained toforce

neural networks. Figure 1 Neural Network as Function Approximator In the next section we will present the multilayer perceptron neural network, and will demonstrate how it can be used as a function approximator. 2. Multilayer Perceptron Architecture 2.1 Neuron Model The multilayer perceptron neural network is built up of simple components.