Journal of Statistical and Econoetric Methods, vol.4, no., 05, 45-60 ISSN: 4-084 (print), 4-076 (online) Scienpress Ltd, 05 Applications of Maiu Entropy Principle in Forulatin Solution of Constrained Non-Linear Prorain Proble O Parash, Muesh and Radhey S. Sinh Abstract The aiu entropy principle finds its applications in a variety of disciplines related with Operations Research. The use of the principle can be etended to obtain the solution of a atheatical prorain proble. In the present counication, an alorith has been developed for solvin constrained non-linear prorain proble throuh the equivalent surroate proble. In the solution process of so fored surroate proble, the surroate ultipliers need to be updated and therefore, aiu entropy forulation has been used for the update of the ultipliers. For the ipleentation and well understandin of the alorith, an eaple has been discussed which confirs that entropy based updates for the surroate ultipliers lead to a fast and accurate solution to the proble. Departent of Matheatics, Guru Nana Dev University, Aritsar-4005, India. E-ail: oparash777@yahoo.co.in Departent of Matheatics, Guru Nana Dev University, Aritsar-4005, India. E-ail: saranal.uesh@yahoo.co.in Departent of Statistics and Actuarial Science, University of Waterloo, Ontario NL G, Canada. E-ail: rssinh@uwaterloo.ca Article Info: Received : February, 05. Revised : March 6, 05. Published online : June 5, 05.
46 Maiu entropy principle in non-linear prorain proble Matheatics Subect Classification: 94A5; 94A7 Keywords: Entropy; Maiu entropy principle; Non-linear prorain; Saddle point; Surroate proble Introduction Inforation theory provides a constructive role for settin up probability distributions when partial inforation is available and leads to a type of statistical inference which is called the aiu-entropy estiate. This estiate is least biased as it is aially noncoittal with reard to issin inforation. The principle of aiu entropy introduced by Jaynes [] provides a unique solution for the posterior probability distribution based on the intuition that the inforation ain consistent with assuptions and evidence should be inial. In real life situations as well as in several disciplinary pursuits, we are very frequently concerned with constrained optiization. For such pursuits, Kapur [] eplained the principle of aiu entropy with the view that at any stae, we ay have infinity of probability distributions consistent with the iven constraints, such as, n n p, p a, r,,...,, + < n, p 0,..., p 0 (.) i i ri r n i i Out of these distributions, one has aiu uncertainty uncertainty H of any other distribution is less than H a H a and the and if we desire to reduce uncertainty, then can be done only with the help of soe additional inforation. Thus, the use of any distribution other than the aiu uncertainty distribution iplies the use of soe inforation in addition to that iven by equation (.). Now, accordin to Jaynes [] aiu entropy principle which is nothin but the principle of scientific obectivity and it iplies that we should use all the inforation iven to us and avoid usin any inforation not iven to us.
O Parash, Muesh and Radhey S. Sinh 47 For the deeper study of aiu entropy principle to a variety of disciplines, Kapur [] provided the applications of Shannon s entropy [], iven by H (P) n p lo p (.) i i i But there eist a variety of inforation theoretic entropies-paraetric as well non-paraetric investiated by various researchers. To cite a few, we have α n n α H( P) lo pi pi α i i, α, α > 0 (.) which is Renyi s entropy [4]. Followin this developent, Havrada-Charvat [5] introduced a non-additive entropy, iven by H α n α pi i ( P), α, α > 0 α (.4) Parash and Muesh [6, 7] investiated and introduced new eneralized paraetric easures of entropy for the real paraeters, iven by the followin epressions n pi H ( P) ( p α α i ), α 0, α > 0 (.5) α H ( P) α i ( α ) n α [ α( p i ) + lo p i ] α i, α, α > 0 (.6) Furtherore, Parash and Muesh [8] provided the applications of these eneralized easures to the fields of Statistics for the developent of interrelationships between inforation easures and Chi-square variate. Moreover, Parash and Muesh [7] investiated the study of aiu entropy principle to the field of queuein theory. Fan, Raaseera and Tsao [9] reared that entropy optiization is a useful cobination of classical entropy theory with atheatical optiization. The resultin entropy optiization odels have proved their usefulness with
48 Maiu entropy principle in non-linear prorain proble successful applications in a variety of atheatical areas. Greenber and Piersalla [0] presented an approach for solvin arbitrary atheatical proras. The authors observed that the surroate atheatical prora is a lesser constrained proble that, in soe cases, ay be solved with dynaic prorain. A siilar study of the surroate atheatical prora has been ade by Glover []. Erlander [] provided a treatent of entropy constrained linear proras fro odellin as well as coputational aspects. Soe other contributors, due to Teplean and Xinsi [], Das, Mazuder and Kaal [4] and Yin-Xin [5], concerned the developent of any inforation-theoretic aloriths for solvin constrained linear and non-linear prorain probles based upon the principle of iniu cross-entropy and surroate atheatical prorain. In the present counication, a solution of eneral constrained non-linear prorain proble has been provided by forulatin it into an equivalent surroate proble which reached the solution point iteratively by usin the aiu entropy forulation. Further a nuerical eaple is iven for solvin a siple conve non-linear prorain proble which ensures the advantae of usin aiu entropy technique for solvin the proble. General Non-linear Prorain Proble Many probles in enineerin sciences involve the optiization of non-linear functions f and/or and alost invariably active constraints. Consider such a proble of a eneralized constrained non-linear prorain proble iven by Proble-I: Miniize f ( ) (.)
O Parash, Muesh and Radhey S. Sinh 49 subect to the constraints ( ) 0,,, (.) where denotes the vector of real valued variable i ( i,, n) constraint functions (,, ) and the represent the constraint vector. There are widespread applications of the Proble-I and plenty of techniques for the solution of this proble. The nuerical solutions of this type of probles are of reat iportance for enineerin desin, synthesis and analysis. Thus, Proble-I has an equivalent surroate for epressed in the followin. Proble-II: Miniize f ( ) subect to a sinle constraint ( ) 0 (.) where (,, ) are non-neative surroate ultipliers tered as, surroate ultipliers. Without any loss of enerality, the surroate ultipliers ay be assued to be noralized to unity, that is, (.4) The solution process is set in a probabilistic contet by considerin the Proble-I with estiatin the probability of assinin the event that each constraint is active at the optiu. Denotin these probabilities by (,, ), it is obvious that at least one of the constraints ust be active, so that the condition (.4) ust hold. We reach the solution of the Proble-I indirectly throuh a sequence of solutions of the Proble-II. Thus, the probles I and II are equivalent at the solution point. The Larane s function of the Proble-II is iven by
50 Maiu entropy principle in non-linear prorain proble Ω f ( ) + ( ) λ (.5) where λ is the Larane ultiplier associated with the constraint (.) in the Proble-II. An essential condition to be satisfied is that (,, λ ) saddle point of the Larane s function Ω of the Proble-II, that is, (,, λ ) Ω(,, λ ) Ω(,, λ) should be a Ω (.6) The left hand inequality of (.6) ives the iniization over variables for specified and λ whereas the riht hand inequality iplies the aiization over and λ for specified. In fact, and λ are related throuh the stationarity of the Larane s function Ω with respect to as shown in the followin epression Ω f λ, i,, n (.7) ( ) + ( ) i i i The saddle point condition (.6), which is written in ters of Larane s function Ω, ay however be satisfied by iterative eans usin Proble-II itself, with alternative iterations in -space and -space. The alorith is as follows: Choose the initial set of the surroate ultipliers 0 and solve the Proble-II to obtain the values of 0 (by iniization, correspondin to the left hand inequality in (.6)). The surroate ultipliers are then updated to (by aiization for fied 0, correspondin to the left hand inequality in (.6)) and the Proble-II is solved aain to ive. The process is repeated until the 0 0 sequence (, ), (, ),,(, ) and hence of the Proble-I, at (, ), converes to the solution of the Proble-II. It is evident that the updated surroate ultipliers + ust satisfy the norality condition and the surroate constraint of the Proble-II. Thus + (.8)
O Parash, Muesh and Radhey S. Sinh 5 and + + ( ) 0 (.9) + For the condition (.9) to be satisfied we require the values for ( ),,, which are not nown yet. Therefore, we assue that the best + estiates for ( ) are the values ( ) which is odified to the followin equation to be used in the equation (.9), ( ) ε + (.0) where ε is the unnown error introduced due to the replaceent of the + values ( ) by the values ( ),,,. Furtherore, we would epect ε to approach zero as the sequence of the iterations, +, approaches to. Therefore, we reached the followin assinent proble Assin,,, subect to the conditions + + + and ( ) ε with + ε 0 Since the surroatin has no eanin for a sinle constraint, that is,, therefore the cases for are of interest. For the special case, the equations (.8) and (.0) ive a unique assinent + ε [ ( )] [ ( ) ( )] (.) and [ ( ) ] [ ( ) ( )] + ε (.) which depends upon the unnown error ter ε. For the eneral case >, the equations (.8) and (.0) are insufficient to yield a unique assinent +. In the absence of any eplicit criterion of -aiization (necessary to satisfy the
5 Maiu entropy principle in non-linear prorain proble saddle point condition (.6)) which would lead to a unique assinent, an artificial criterion ust be iposed. However, on the basis of the inforation available, there is no loical ustification for a criterion which favours one specific assinent rather than another. The artificial criterion should correspond to a aiization process in -space with the conditions (.8) and (.0) satisfied alonside, otherwise the criterion should introduce iniize bias into the + assinent of. Hence, it is loical to aiize the entropy of the ultipliers + by usin the aiu entropy forulation. Thus, the unsolvable assinent proble is replaced by the followin solvable proble in the contet of Havrada-Charvat s entropy [5]. Proble-III: α + α Maiize ( ) ( ) H subect to the constraints (.) α + + and ( ) ε where + 0,,, The Larane s function of the Proble-III is iven by α + α + + ( ) + υ + η ( ) ε (.4) where υ and η are the Larane s ultipliers associated with the constraints. Differentiatin equation (.4) with respect to et α + α ( ) [ υ + η ( )] α Now, tain in particularα, the equation (.5) becoes + and settin equal to zero, we (.5)
O Parash, Muesh and Radhey S. Sinh 5 + + 4 [ υ η ( )] (.6) In view of the equations (.8), (.0) and (.6), we eliinate the Larane s ultipliers υ and η to et the updated surroate ultipliers (See Appendi A), iven by ( ) ( ) ( ) ε + ( ) ( ),,, (.7) The equation (.7) is proposed to be a relationship for updatin the surroate ultipliers in the iterative alorith described above. The error ter ε is unnown but should approach zero fro positive side as iterations approach the optiu. Thus ε ay be considered as a control paraeter whose value should be chosen at each iteration with the desired behaviour. In the special case of Proble-I with two constraints ( ), equations (.) and (.) represent a unique updatin forula for and quite independently of the aiu entropy forulation. It ay be noted that for, the entropy based update forula (.7) yields the sae results as the update in the equations (.) and (.). This is not surprisin, but it does perit the update forula (.7) to be viewed as a least biased etension of the unique result represented by the set of the equations (.) and (.) for in the ore eneral real of >. This concludes the derivation of an entropy based update forula for the surroate ultipliers. Nuerical Eaple We consider the followin eaple in which an analytical solution for the -phase iniization is available for any set of values of. Consequently, the
54 Maiu entropy principle in non-linear prorain proble eaple focuses attention on the -phase iterations which are the centre of the present ethod. The eaple is as follows: Miniize ( ) f (.) subect to the constraints ( ) + + 0 (.) ( ) + + 0 (.) ( ) + + 0 (.4),, > 0 The surroate for of the iven proble is Miniize ( ) f subect to ( + + ) + ( + + ) + ( + + ) 0 which rearranes to ( + + ) + ( + + ) + ( + + ) 0 (.5) The correspondin Larane s function is iven by Ω i [ ( + + ) + ( + + ) + ( + + ) ] + λ (.6) Ω Now 0, i,, ives the followin set of equations ( + ) λ + ( + ) λ + (.7) (.8) ( + ) λ + (.9) In view of the equations (.5), (.7), (.8) and (.9), we et the followin values of, and in ters of, and.
O Parash, Muesh and Radhey S. Sinh 55 (.0) ( + + ) ( + + ) (.) (.) ( + + ) Table : Iterations with the updated. i ε 0 -- 0.5 0.0 0.6667 0.0-0.8 0.8 0.09 0.4494 0.769 0.4084 0.558 0.85 0.4894 0.05-0.496 0.0840 0.0008 0.464 0.7540 0.4085 0.56 0.848 0.4875 0.0047-0.440 0.08 0.00004 0.46 0.75 0.40846 0.57 0.846 0.4875 0.0045-0.44 0.085 Now, we choose the initial set of the surroate ultipliers, and which would be used in the equations (.0), (.) and (.) to obtain the values of, and. Jaynes aiu entropy forulation shows that in the absence of any etra inforation about the proble, the least-biased assuption one can choose is that all the constraints should be equally weihted. Thus we have taen for the initial iteration. The surroate ultipliers are then updated by usin the relationship (.7) for the repetition of the process till we reach the desired solution of the proble as shown in the Table.
56 Maiu entropy principle in non-linear prorain proble Table ives the iterative result and is as eact as the four decial places accuracy and therefore the iniu value of ( ) f is found to be55. 6858. 4 Conclusion The present wor eplores the use of inforation theory in the atheatical prorain proble. An alorith has been developed for solvin constrained non-linear prorain proble where the oriinal proble is forulated into equivalent surroate proble and surroate ultipliers are updated iteratively with the use of aiu entropy forulation. The illustrated eaple confirs that entropy based updates for the surroate ultipliers lead to a fast and accurate solution of the Proble-I. The eaple which have discussed above is relatively siple and conve proble in which the converence rate is dependent upon the sequence of the decreasin values specified forε. Other sequences than that used above have been tested and confired for converence. Thus, the entropy aiization appears to be quantitatively sound. With siilar aruents, one can provide the applications of entropy easures either by usin the standard easures of entropy or by developin new easures of entropy for the solutions of atheatical prorain probles. Moreover, in the literature there eist a variety of ethods for obtainin the solutions of linear and non-linear prorain probles; we have provided the solution of non-linear prorain proble only whereas the solutions for the linear prorain probles can siilarly be provided. ACKNOWLEDGEMENTS. The authors are thanful to the University Grants Coission, New Delhi, for providin financial assistance for the preparation of the anuscript.
O Parash, Muesh and Radhey S. Sinh 57 References [] E.T. Jaynes, Inforation theory and statistical echanics, The Physical Review, 06, (957), 60-60. [] J.N. Kapur, Maiu Entropy Models in Science and Enineerin, Wiley Eastern Liited, New Delhi, 989. [] C.E. Shannon, A atheatical theory of counication, Bell Syste Technical Journal, 7, (948), 79-4, 6-659. [4] A. Renyi, On easures of entropy and inforation, Proceedins 4th Bereley Syposiu on Matheatical Statistics and Probability,, (96), 547-56. [5] J.H. Havrada and F. Charvat, Quantification ethods of classification process: Concept of structural α-entropy, Kybernetia,, (967), 0-5. [6] O. Parash and Muesh, New eneralized paraetric easure of entropy and cross-entropy, Aerican Journal of Matheatics and Sciences,, (0), 9-96. [7] O. Parash and Muesh, Contribution of aiu entropy principle in the field of queuein theory, Counications in Statistics--Theory and Methods, (04). (Accepted) [8] O. Parash and Muesh, Relation between inforation easures and chi-square distribution, International Journal of Pure and Applied Matheatics, 84, (0), 57-54. [9] S-C. Fan, J.R. Raaseera and H-S.J. Tsao, Entropy Optiization and Matheatical Prorain, Kluwer Acadeic Publishers, USA, 997. [0] H.J. Greenber and W.P. Piersalla, Surroate atheatical prorain, Operations Research, 8, (970), 94-99. [] F. Glover, Surroate constraints, Operations Research, 6, (986), 74-749. [] S. Erlander, Entropy in linear proras, Matheatical Prorain,, (979), 7-50.
58 Maiu entropy principle in non-linear prorain proble [] A.B. Teplean and L.A. Xinsi, Maiu-entropy approach to constrained non-linear prorain, Enineerin Optiization, (), (987), 9-05. [4] N.C. Das, S.K. Mazuder and D.E. Kaal, Constrained non-linear prorain: A iniu cross-entropy alorith, Enineerin Optiization, (4), (999), 479-487. [5] L. Yin-Xin, Maiu entropy ethod for linear prorain with bounded variables, Journal of Northwest University, 5(5), (005), 507-50.
O Parash, Muesh and Radhey S. Sinh 59 Appendi A. Calculation for +. Usin equation (.6) in equations (.8) and (.0), we et ( ) 4 + η υ (A) ( ) ( ) ε η υ 4 + (A) Solvin equations (A) and (A), we et ( ) ( ) ( ) 4 ε η (A) and ( ) ( ) ( ) ( ) 4 ε υ (A4) Substitutin the vales of υ and η in equation (.6), we et ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) + + ε ε ( ) ( ) ( ) ( ) ( ) + ε
60 Maiu entropy principle in non-linear prorain proble Therefore, we et ( ) ( ) ( ) ( ) ( ) + ε