1 Appendix 14.1 he opimal conrol problem and is soluion using he maximum principle NOE: Many occurrences of f, x, u, and in his file (in equaions or as whole words in ex) are purposefully in bold in order o refer o vecors. Opimal conrol heory, using he Maximum Principle, is a echnique for solving consrained dynamic opimisaion problems. In his appendix we aim o explain wha is mean by a consrained dynamic opimisaion problem; show one echnique - he maximum principle - by which such a problem can be solved. We will no give any proofs of he condiions used in he Maximum Principle. Our emphasis is on explaining he echnique and showing how o use i. For he reader who wishes o go hrough hese proofs, some recommendaions for furher reading are given a he end of Chaper 14. Afer you have finished reading his appendix, i will be useful o go hrough Appendices 14.2 and 14.3. Appendix 14.2 shows how he maximum principle is used o derive he opimal soluion o he simple non-renewable resource depleion problem discussed in Par 1 of his chaper. Appendix 14.3 considers he opimal allocaion of a renewable or non-renewable resource in he case where exracion of he resource involves coss, he model discussed in Par 2 of his chaper. Le us begin by laying ou he various elemens of a consrained dynamic opimisaion problem. In doing his, you will find i useful o refer o ables 14.2 and 14.3, where we have summarised he key elemens of he opimal conrol problem and is soluion.
2 able 14.2 he opimal conrol problem and is soluion Objecive funcion (See Noes 1 & 5) Sysem erminal sae max J( u) L( x, u, ) d 0 F( x( x f( x, u,) x( 0 ) x 0 ), x( ) x x ) free erminal poin fixed free fixed free ( ) Hamilonian (See Noe 2) H C H( x, u,, ) = L( x, u,) λf ( x, u,) Equaions of moion Max H (See Noe 3) ransversaliy condiion H x f( x, u, ) λ x H 0 u x( ) x λ ( ) 0 if = or F( ) = 0 oherwise λ ( ) F ( ) (See Noe 4) H C ( F / ) 0 H C ( F / ) 0
3 Noes o able 14.2 Noe 1 he erm F(x( ), ) may no be presen in he objecive funcion, and canno be if =. Noe 2 H C denoes he curren value Hamilonian (see he noes o able 14.3) Noe 3 he Max H condiion given in he able is for he special case of an inerior soluion (u is an inerior poin). A more general saemen of his condiion (he maximum principle ) is: u() maximises H over u() U, for 0. Noe 4 he erm F/ does no ener he ransversaliy condiion in he final line of he able if F(.) = 0. Noe 5 Noaion used in he able (see below): x() he vecor of sae variables u() he vecor of conrol variables J(u) he objecive funcion o be maximised, which may be augmened by a final funcion F( ) () he vecor of co-sae variables (or shadow prices) f(x,u,) he sae equaion funcions, describing he relevan physical-economic sysem 0 H F(x( ), ) he iniial poin in ime he erminal poin in ime he Hamilonian funcion a 'final funcion' (he role of which is explained below)
4 able 14.3 he opimal conrol problem wih a discouning facor and is soluion Objecive Funcion (See Noe 1) Sysem maxj( u ) L( x, u,)e 0 d F( x( x f( x, u,) x( 0 ) x 0 ), )e erminal sae x( ) x x ) free erminal poin fixed free fixed free ( Presen-value Hamilonian (See Noe 2) Curren-value Hamilonian (See Noe 2) Equaions of moion Max H C (See Noe 3) ransversaliy condiion H P ρ H( x, u,, λ) = L( x, u, )e λf( x, u,) ρ H H e L( x, u,) μf( x, u,), ( μ λe ) C P x f ( x, u, ) μ μ H C x H C 0 u x( ) x μ ( ) 0 if = or if F( ) = 0 ρ oherwise μ ( ) F ( ) (See Noe 4) H C ( F / ) 0 H C ( F / ) 0
5 Noes o able 14.3 Noe 1 he erm F(x ( ), ) e may no be presen in he objecive funcion, and canno be if =. Noe 2 wo versions of he Hamilonian funcion are given. he firs (called here H P ) is known as he 'presen value' Hamilonian, as he presence of he erm e - in he objecive funcion (which convers magniudes ino presen value erms) carries over ino he Hamilonian, H P. he second (called here H C ) is known as he 'curren-value' Hamilonian (see Chiang [1992], page 210). Premuliplying H P by e removes he discouning facor from he expression, and generaes he curren-value Hamilonian, H C. he propery ha H C is expressed in currenvalue erms can be seen by noing ha he L(x, u, ) funcion is no muliplied by he discouning erm e -. For mahemaical convenience, i is usually beer o work wih he Hamilonian in curren-value erms. Noe 3 he Max H C condiion given in he able is for he special case of an inerior soluion (u is an inerior se). A more general saemen of his condiion (he maximum principle ) is: u() maximises H C over u() U, for 0. Noe 4 he erm F/ does no ener he ransversaliy condiion in he final line of he able if F(.) = 0. Noe 5 Noaion is as defined in able 14.2. he erm denoes he uiliy social discoun rae.
6 1. he funcion o be maximised is known as he objecive funcion, denoed J(u). his akes he form of an inegral over a ime period from an iniial ime 0 o he erminal ime. wo poins should be borne in mind abou he erminal poin in ime, : In some opimisaion problems is fixed; in ohers i is free (and so becomes an endogenous variable, he value of which is solved for as par of he opimisaion exercise). In some opimisaion problems he erminal poin is a finie quaniy (i is a finie number of ime periods laer han he iniial ime); in ohers, he erminal poin is infinie ( = ). When a problem has an infinie erminal poin in ime, should be regarded as free. 2. he objecive funcion will, in general, conain as is argumens hree ypes of variable: x(), a vecor of n sae variables a ime ; u(), a vecor of m conrol (or insrumen) variables a ime ;, ime iself. Alhough he objecive funcion may conain each of hese hree ypes of variables, i is no necessary ha all be presen in he objecive funcion (as you will see from he examples worked hrough in Appendices 14.2 and 14.3). 3. he objecive funcion may (bu will ofen no) be augmened wih he addiion of a 'final funcion' (also known as a salvage funcion ), denoed by he funcion F( ) in ables 14.2 and 14.3. I will appear whenever he value of he objecive funcion is dependen on some paricular funcion F( ) of he levels of he sae variables x a he erminal ime, and possibly on he erminal ime iself. (he applicaions in Chaper 14 did no involve he use of a final funcion.) 4. he soluion o a dynamic opimal conrol problem will conain, among oher hings, he values of he sae and conrol variables a each poin in ime over he period from 0 o. I is his ha makes he exercise a dynamic opimisaion exercise.
7 5. Underlying he opimal conrol problem will be some economic, biological or physical sysem (which we shall call simply 'he economic sysem'), describing he iniial values of a se of sae variables of ineres, and how hey evolve over ime. he evoluion of he sae variables over ime will, in general, be described by a se of differenial equaions (known as sae equaions) of he form: x f( x, u, ) where x = dx/d is he ime derivaive of x (he rae of change of x wih respec o ime). Noe ha as x is a vecor of n sae variables, here will in general be n sae equaions. Any soluion o he opimal conrol problem mus saisfy hese sae equaions. his is one reason why we use he phrase consrained dynamic opimisaion problems. 6. A second way in which consrains may ener he problem is hrough he erminal condiions of he problem. here are wo aspecs here: one concerns he value of he sae variables a he erminal poin in ime, he oher concerns he erminal poin in ime iself. Firs, in some problems he values ha he sae variables ake a he erminal poin in ime are fixed; in ohers hese values are free (and so are endogenously deermined in he opimisaion exercise). Secondly, eiher he paricular problem ha we are dealing wih will fix he erminal poin in ime, or ha poin will be free (and so, again, be deermined endogenously in he opimisaion exercise). 7. he opimisaion exercise mus saisfy a so-called ransversaliy condiion. he paricular ransversaliy condiion ha mus be saisfied in any paricular problem will depend upon which of he four possibiliies oulined in (6) applies. (Four possibiliies exis because for each of he wo possibiliies for he erminal values of he sae variables here are wo possibiliies for he erminal poin in ime.) I follows from his ha when we read ables 14.2 and 14.3, hen (ignoring he column of labels) here are four columns referring o hese four possibiliies. Where cells are merged and so cover more han one column, he condiion shown refers o all he possibiliies i covers. We shall come back o he ransversaliy condiion in a momen. 8. he conrol variables (or insrumens) are variables whose value can be chosen by he decision maker in order o seer he evoluion of he sae variables over ime in a desired manner.
8 9. In addiion o he hree kinds of variables we have discussed so far ime, sae and conrol variables a fourh ype of variable eners opimal conrol problems. his is he vecor of cosae variables (or in he case where he objecive funcion conains a discoun facor). Cosae variables are similar o he Lagrange muliplier variables one finds in saic consrained opimisaion exercises. Bu in he presen conex, where we are dealing wih a dynamic opimisaion problem over some sequence of ime periods, he value aken by each co-sae variable will in general vary over ime, and so i is appropriae o denoe () as he vecor of co-sae variables a ime. 10. he analogy of co-sae variables wih Lagrange mulipliers carries over o heir economic inerpreaion: he co-sae variables can be inerpreed as shadow prices, which denoe he decision maker s marginal valuaion of he sae variable a each poin in ime (along he opimal ime pah of he sae and conrol variables). 11. Finally, le us reurn o he ransversaliy condiion. Looking a he final rows in ables 14.2 and 14.3 you will see four possible configuraions of ransversaliy condiion. All relae o somehing ha mus be saisfied by he soluion a he erminal poin in ime. Where he erminal value of he sae variables is fixed, his will always be refleced in he ransversaliy condiion. On he oher hand, where he erminal value of he sae variables is free, he ransversaliy condiion will usually 9 require ha he shadow price of he sae variables be zero. Inuiively, his means ha if we do no pu any consrains on how large he socks of he sae variables mus be a he erminal poin in ime, hen hey mus have a zero value a ha ime. For if hey had any posiive value, i would have been opimal o deplee hem furher prior o he end of he planning horizon. Noe also ha whenever he erminal poin in ime is free (wheher or no he sae variables are fixed a he erminal poin), an addiional par of he ransversaliy condiion requires ha he Hamilonian have a zero value a he endogenously 9 Sricly speaking, he shadow price will be zero where he ime horizon is of infinie lengh or if here is no final funcion in he objecive funcion. However, where he objecive funcion conains a final funcion, he shadow price mus equal he firs derivaive of ha final funcion wih respec o he sae variable. his is shown in ables 14.2 and 14.3.
9 deermined erminal poin in ime. 10 If i did no, hen he erminal poin could no have been an opimal one! he general case referred o in ables 14.2 and 14.3, and special cases In he descripion we have given above of he opimal conrol problem, we have been considering a general case. For example, we allow here o be n sae variables and m conrol variables. In some special cases, m and n may each be one, so here is only one sae and one conrol variable. Also, we have wrien he sae equaion for he economic sysem of ineres as being a funcion of hree ypes of variables: ime, sae and conrol. In many paricular problems, no all hree ypes of variables will be presen as argumens in he sae equaion. For example, in many problems, ime does no ener explicily in he sae equaion. A similar commen applies o he objecive funcion: while in general i is a funcion of hree ypes of variables, no all hree will ener in some problems. Finally, ofen he objecive funcion will no be augmened by he presence of a final funcion. Limiaions o he opimal conrol echnique oulined in his appendix he saemen of he opimal conrol problem and is soluion given in his appendix is no as general as i migh be. For example, he erminal condiion migh require ha a conrol variable mus be greaer han some paricular quaniy (bu is oherwise unconsrained). Or he problem migh require ha one or more sae variables (such as a resource sock) be non-negaive in he erminal sae. Anoher case of obvious ineres o economiss is he requiremen ha a conrol variable (such as a resource depleion rae) be non-negaive over he planning horizon. Deails of generalisaions of he opimal conrol model o cover hese (and many oher) cases can be found in Chiang (1992) or, more rigorously, Kamien and Schwarz (1991). 10 Or, as shown in able 14.3, he addiional par of he ransversaliy condiion requires ha he Hamilonian plus he derivaive of F wih respec o have a zero value a he endogenously deermined erminal poin in ime.
10 he presence of a discoun facor in he objecive funcion For some dynamic opimisaion problems, he objecive funcion o be maximised, J(u), will be an inegral over ime of some funcion of ime, sae variables and conrol variables. ha is: J( u) 0 L( x, u, )d However, in many dynamic opimisaion problems ha are of ineres o economiss, he objecive funcion will be a discouned (or presen-value) inegral of he form: J( u) 0 L( x, u, )e ρ d For example, equaion 14.8 in he ex of his chaper is of his form. here, L is acually a uiliy funcion U( ) (which is a funcion of only one conrol variable, C). Indeed, hroughou his book, he objecive funcions wih which we deal are almos always discouned or presen-value inegral funcions. he soluion of he opimal conrol problem he naure of he soluion o he opimal conrol problem will differ depending on wheher or no he objecive funcion conains a discouning facor. able 14.2 saes formally he opimal conrol problem and is soluion using general noaion, for he case where he objecive funcion does no include a discoun facor. able 14.3 presens he same informaion for he case where he objecive funcion is a discouned (or presen-value) inegral. Some (brief) explanaion and discussion of how he condiions lised in ables 14.2 and 14.3 may be used o obain he required soluion is provided below hose ables. However, we srongly urge you also o read Appendices 14.2 and 14.3, so ha you can ge a feel for how he general resuls we have described here can be used in pracice (and how we have used hem in his chaper). Inerpreing he wo ables
11 I will help o focus on one case: we will look a an opimal conrol problem wih a discouning facor, an infinie ime horizon (so ha is deemed o be free), and no resricion being placed on he values of he sae variable in he erminal ime period (so ha x( ) is free). he relevan saemen of he opimal conrol problem and is soluion is, herefore, ha given in he final column o he righ in able 14.3. We can express he problem as max J(u) 0 L( x, u, )e ρ d subjec o x f( x, u,) and x( 0 ) x 0, x(0) given, x ) free. o obain he soluion we firs consruc he curren-value Hamilonian: H L( x, u,) μf( x, u,) C ( he curren-value Hamilonian consiss of wo componens: he firs L(x,u,) is he funcion which, afer being muliplied by he discouning facor and hen being inegraed over he relevan ime horizon, eners he objecive funcion. Noe carefully by examining able 14.3 ha in he Hamilonian he L funcion iself eners, no is inegral. Furhermore, alhough he discouning facor eners he objecive funcion, i does no ener in he curren-value Hamilonian. he second componen ha eners he Hamilonian is he righ-hand side of he sae variable equaions of moion, f(x, u, ), afer having been premuliplied by he co-sae variable vecor in curren-value form. Remember ha in he general case here are n sae variables, and so n cosae variables, one for each sae equaion. In order for his muliplicaion o be conformable, i is acually he ranspose of he co-sae vecor ha premuliplies he vecor of funcions from he sae equaions. Our nex ask is o find he values of he conrol variables u which maximise he curren-value Hamilonian a each poin in ime; i is his which gives his approach is name of he maximum principle. If he Hamilonian funcion H C is non-linear and differeniable in he conrol variables u,
12 hen he problem will have an inerior soluion, which can be found easily. his is done by differeniaing H C wih respec o u and seing he derivaives equal o zero. Hence in his case one of he necessary condiions for he soluion will be HC 0 (a se of m equaions, one for each of he m conrol variables). u More generally, here may be a corner soluion. Obaining his soluion may be a difficul ask in some circumsances, as i involves searching for he values of u() which maximises H C () (a all poins in ime) in some oher way. Bringing ogeher all he necessary condiions for he complee soluion of he opimisaion problem we have:
13 he maximum principle condiions (assuming an inerior soluion and no final funcion presen): HC 0 (a se of m equaions, one for each of he m conrol variables). u hose given in he row labelled Equaions of moion in able 14.3, ha is x f( x, u,) (a se of n equaions) H μ μ C (a se of n equaions) x he iniial condiion x( 0 ) x 0 he ransversaliy condiion H ( ) 0 C Solving hese necessary condiions simulaneously, we can obain he opimal ime pah for each of he m conrol variables over he (infinie) ime horizon. Corresponding o his ime pah of he conrol variables are he opimised ime pahs of he n sae variables and heir associaed currenvalue shadow prices (values of he co-sae variables) along he opimal pah. I should be clear ha obaining his complee soluion could be a dauning ask in problems wih many conrol and sae variables. However, where he number of variables is small and he relevan funcions are easy o handle, he soluion can ofen be obained quie simply. We demonsrae his asserion in he following wo appendices. One final poin warrans menion. ables 14.2 and 14.3 give necessary bu no sufficien condiions for a maximum. In principle, o confirm ha our soluion is indeed a maximum, second-order condiions should be checked as well. However, in mos problems of ineres o economiss (and in all problems invesigaed in his book), assumpions are made abou he shapes of funcions which guaranee ha second-order condiions for a maximum will be saisfied, hereby obviaing he need for checking second-order condiions.
14 Le us ry o provide some inuiive conen o he foregoing by considering a problem where here is jus one sae variable, x, and one conrol variable, u, where does no ener eiher he objecive funcion or he equaion describing he sysem, no final funcion is presen, and where 0 = 0 and we have an infinie erminal poin ( = ). hen he problem is o maximise 1 0 L(x,u subjec o )e -ρ d x f(x,u ) and x(0) xo for which he curren-value Hamilonian is H C L(x,u ) μ f(x,u ) L(x,u ) μ x In he original problem, we are looking o maximise he inegral of he discouned value of L(x, u ). he firs erm in he Hamilonian is jus L(x, u ), he insananeous value of ha we seek he maximum of. Recalling ha co-sae variables are like Lagrangian mulipliers and ha hose are shadow prices (see Appendix 4.1), he second erm in he Hamilonian is he increase in he sae variable, some sock, valued by he appropriae shadow price. So, H C can be regarded as he value of ineres a plus he increase in he value of he sock a. In ha case, he maximum principle condiion H C / u = 0 makes a good deal of sense. I says, a every poin in ime, se he conrol variable so ha i maximises H C, which is value plus an increase in value. I is inuiive ha such maximisaion a every poin in ime is required for maximisaion of he inegral. he equaion of moion condiion x f(x,u ) ensures ha he opimal pah is one ha is feasible for he sysem. Aside from ransversaliy, he remaining condiion is H C / x which governs how he shadow, or impued, price of he sae variable mus evolve over ime. his condiion can be given some inuiive conen by considering a model which is, mahemaically, furher specialised, and which has some economic conen. Consider he simples possible opimal growh model in which he only argumen in he producion funcion is capial. hen, he opimal pahs for consumpion and capial accumulaion are given by maximising 1 As x and u are now single variables, no vecors, we now drop he bold (vecor) noaion
15 0 U(C )e subjec o -ρ d K Q(K ) C giving he curren-value Hamilonian H C U(C ) μ (Q(K ) C ) U(C ) μ K Here he Hamilonian is curren uiliy plus he increase in he capial sock valued using he shadow price of capial. In Appendices 19.1 and 19.2 we shall explore his kind of Hamilonian in relaion o he quesion of he proper measuremen of naional income. he maximum principle condiion here is H C / C = U / C - = 0 which gives he shadow price of capial as equal o he marginal uiliy of consumpion. Given ha a marginal addiion o he capial sock is a he cos of a marginal reducion in consumpion, his makes sense. Here he condiion governing he behaviour of he shadow price over ime is μ ρμ H C / K ρμ μ Q / K where Q / K is he marginal produc of capial. his condiion can be wrien wih he proporionae rae of change of he shadow price on he lef-hand side, as μ /μ ρ ( Q / K ) where he righ-hand side is he difference beween he uiliy discoun rae and he marginal produc of capial adjused for he marginal uiliy of consumpion. he firs erm on he righ-hand side reflecs impaience for fuure consumpion and he second erm he pay-off o delayed consumpion. According o his expression for he proporional rae of change of he shadow price of capial: a) is increasing when 'impaience' is greaer han 'pay-off'; b) is consan when 'impaience' is equal o 'pay-off'; c) is decreasing when 'impaience' is less han 'pay-off'. his makes sense, given ha: a) when 'impaience' is greaer han 'pay-off', he economy will be running down K;
16 b) when 'impaience' and 'pay-off' are equal, K will be consan; c) when 'impaience' is less han 'pay-off', he economy will be accumulaing K. hese remarks should be compared wih he resuls in able 14.1 where i will be seen ha he calculaed shadow price of capial decreases over ime, while he shadow price of oil, which is becoming scarcer, increases over ime.