Entropy Methods For Univariate Distributions in Decision Analysis

Transcription

1 Etropy Methods For Uvarate Dstrbutos Decso Aalyss Al E. Abbas Ctato: AIP Coferece Proceedgs 659, 339 (23); do: 1.163/ Vew ole: Vew Table of Cotets: Publshed by the AIP Publshg Artcles you may be terested O Bayesa Iferece, Maxmum Etropy ad Support Vector Maches Methods AIP Cof. Proc. 872, 43 (26); 1.163/ O The Relatoshp betwee Bayesa ad Maxmum Etropy Iferece AIP Cof. Proc. 735, 445 (24); 1.163/ Maxmum Probablty ad Maxmum Etropy methods: Bayesa terpretato AIP Cof. Proc. 77, 49 (24); 1.163/ A Etropy Approach for Utlty Assgmet Decso Aalyss AIP Cof. Proc. 659, 328 (23); 1.163/ Usg Bayesa Aalyss ad Maxmum Etropy To Develop No parametrc Probablty Dstrbutos for the Mea ad Varace AIP Cof. Proc. 659, 53 (23); 1.163/ Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

2 Etropy Methods For Uvarate Dstrbutos Decso Aalyss Al E. Abbas Departmet of Maagemet Scece ad Egeerg, Staford Uversty, Staford, Ca, 9435 Abstract. Oe of the most mportat steps decso aalyss practce s the elctato of the decso-maker's belef about a ucertaty of terest the form of a represetatve probablty dstrbuto. However, the probablty elctato process s a task that volves may cogtve ad motvatoal bases. Alteratvely, the decso-maker may provde other formato about the dstrbuto of terest, such as ts momets, ad the maxmum etropy method ca be used to obta a full dstrbuto subject to the gve momet costrats. I practce however, decso makers caot readly provde momets for the dstrbuto, ad are much more comfortable provdg formato about the fractles of the dstrbuto of terest or bouds o ts cumulatve probabltes. I ths paper we preset a graphcal method to determe the maxmum etropy dstrbuto betwee upper ad lower probablty bouds ad provde a terpretato for the shape of the maxmum etropy dstrbuto subject to fractle costrats, (FMED). We also dscuss the problems wth the FMED that t s dscotuous ad flat over each fractle terval. We preset a heurstc approxmato to a dstrbuto f addto to ts fractles, we also kow t s cotuous ad work through full examples to llustrate the approach. INTRODUCTION I may Decso Aalyss applcatos, a decso-maker s terested a ucerta quatty. Whe data sets are avalable she may provde momets of the dstrbuto but may stuatos that arse practce, momets are ot avalable ad the decso maker provdes percetles, (fractles), or other formato represetg her belef about the shape of the dstrbuto. The maxmum etropy formulato for momets ad (or) fractle costrats s gve as: max f( x) l( f( x)) dx f ( x) g g a subject to h ( x) f ( x) dx.,1,..., ; a f( x). (1) Where, [a,g] s the support of the maxmum etropy dstrbuto, h ( x) s ether a dcator fucto over a terval for fractle costrats,or x rased to a certa Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

3 power, for momet costrats, ad s are a gve sequece of fractles or momets. The soluto to ths problem has the form: k f ( x) e xp( h( x)) (2) Where s the Lagrage multpler for each fractle or momet costrat. The maxmum etropy soluto subject to fractle costrats produces a dscotuous star case desty fucto satsfyg the costrats each terval. We wll refer to ths dstrbuto as the Fractle Maxmum Etropy Dstrbuto (FMED). If the desty fucto has a uboud support ay drecto, the uboud FMED has a expoetal tal that drecto. A example of a FMED dstrbuto gve fve commo fractles used practce, (1%, 25%, 5%, 75%, ad 99%), ad a bouded support s show Fgure Taut Strg gve fractle costrats a b c d e f g Fractle Maxmum Etropy Dstrbuto.25 ( dc).25 ( ed) ( f e) ( cb).1 ( ba).1 ( gf) a b c d e f g FIGURE 1. (a) Fractle Maxmum Etropy dstrbuto obtaed usg the.1,.25,.5,.75,.99 fractles. (b) Probablty desty fucto correspodg to the gve fractle costrats. I the ext secto we wll preset a soluto to a more geeral formulato, where bouds o the cumulatve dstrbuto are avalable rather tha precse fractles. We wll refer to ths dstrbuto as the Taut Strg dstrbuto. Note that the FMED s a specal case of the taut strg dstrbuto whe upper ad lower bouds cocde. We wll also preset a heurstc approxmato to a dstrbuto f addto to ts fractles, we also kow t s cotuous. We wll refer to ths dstrbuto as the Md Pot Maxmum Etropy Dstrbuto, MMED, ad compare t to other approaches. THE MAXIMUM ENTROPY DISTRIBUTION BETWEEN UPPER AND LOWER PROBABILITY BOUNDS I may decso aalyss applcatos, a decso-maker s terested a probablty dstrbuto for some radom quatty,. The decso aalyst may help Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

4 elct her probablty dstrbuto, however, the elctato of a full probablty dstrbuto s a task that volves may cogtve ad motvatoal bases. [1], [2]. Alteratvely, the decso-maker may provde other formato about the fractles of the dstrbuto or bouds o these fractles. The decso maker may also cosult a expert who provdes aother cumulatve dstrbuto for. Faced wth two probablty dstrbutos, or upper ad lower bouds o the cumulatve dstrbuto, we would lke to fd a ubased dstrbuto that les betwee these bouds. A smlar problem that arses s whe the aalyst has kowledge of some utlty values or bouds o some utlty values, obtaed durg a utlty assessmet, ad would lke to fd a ubased utlty curve betwee the upper ad lower utlty bouds. [3]. We wll preset a exact graphcal method to solve these problems based o the maxmum etropy prcple. [4]. We wll refer to the soluto as the Taut Strg dstrbuto. Wthout sgfcat loss of geeralty, we wll focus o the maxmum etropy dstrbuto betwee upper ad lower bouds of two probablty dstrbutos. Now let us move to the formulato of our problem: gve two cumulatve probablty dstrbutos, R ad Q for the same quatty, what s the maxmum etropy dstrbuto, P, whch les betwee ther upper ad lower bouds? The dstrbutos, P ad Q, may be cotuous or dscrete. We wll start the aalyss wth the case where there s stochastc domace betwee the two dstrbutos, ths mples the two dstrbutos do ot cross, ad the geeralze to the case where stochastc domace does ot exst. The problem ca be formulated dscrete form as follows: such that 1 maxmze p log( p ) 1 rj pj qj, 1,... Costrats (II) j1 j1 j1 p 1, Costrat (I) p, 1,... Costrats (III) j (3) Where, rj, qj adp j are the dscrete probabltes of outcome j for dstrbutos R, P, ad Q respectvely. For shorthad we wll use R r j j1, ad Q q. j1 j We wll preset the soluto to ths problem dscrete form but the soluto to the cotuous form follows by aalogy. We start the aalyss by solvg the formulato usg the frst costrat aloe ad ay strctly addtve cocave fucto, hp ( ), where hp ( ) s ay strctly cocave term of 1 p (ot ecessarly the etropy expresso). Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

5 subject to maxmze hp ( ) 1 1 p 1 (4) Usg the method of Lagrage multplers, (5) 1 1 L h( p ) ( p 1) L p ' ' h( p) h( p) 1,..., (6) From the strct cocavty of hp ( ), equalty of the dervatves mples that p p, j 1,...,. (7) Now f we take the dervatve wth respect to the Lagrage multpler ad use the results of Equato 7, we get, j L p 1 p p, 1,..., (8) Ths s the famlar uform soluto obtaed whe maxmzg the etropy of a dstrbuto but ote that ths soluto s varat for ay strctly addtve cocave fucto, hp ( ), ad s ot uque to the etropy expresso. I other words we ca 1 maxmze ay cocave term, or mmze ay covex term, ad obta the same result. Now we cosder the secod costrat. The soluto to the formulato wth the secod costrat aloe appears may etwork flow ad supply cha problems where the objectve fucto to be mmzed s a covex cost fucto. The soluto, f oe exsts, s uque ad ca be determed usg the Karush-Kuh-Tucker optmalty codtos. The soluto s also varat to the maxmzato of ay addtve cocave term or the mmzato of ay addtve covex term. [5]. Now we move o to the thrd costrat: The o-egatvty of the probablty values mples we select a o-decreasg probablty dstrbuto out of all the possble solutos satsfyg costrats (I) ad (II). Usg the prevous results, the soluto to our problem s thus equvalet to the maxmzato of ay addtve cocave fucto, or the mmzato of ay addtve Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

6 covex fucto subject to the gve costrats. Wth o loss of geeralty, let us 2 2 choose to mmze a addtve covex fucto, p, where s the 1 dscretzg terval for the values of the varable. Ths ew objectve fucto s actually the dstace (path) the plae of the cumulatve probablty dstrbuto startg from pot A to pot B, as show Fgure 2. So fact our problem s reduced to the problem of fdg the shortest path the plae, (ew objectve fucto), that les betwee the two dstrbutos,(codto 2), starts at a probablty zero ad eds wth a probablty 1, (requred by codto 1ad automatcally satsfed by codto 2), ad s o-decreasg, (requred by codto 3 ad s satsfed by beg the shortest path as wll follow). The problem of choosg a dstrbuto that maxmzes the etropy subject to lower ad upper costrats ow has a geometrc terpretato. It s the shortest path from the frst to the last pot amog the paths that le betwee the lower ad upper bouds. To fd ths path, mage ps the plae at the pots (, R) ad (, Q) for, where s the value of the varable at whch the assessmet took place, R r1r2... r, ad Q q 1... q. Now thread a strg betwee the ps at (, R) ad (, Q ) for 1, ad pull the strg taut. The taut strg traces out the shortest path ad s also the maxmum etropy soluto. The taut strg dstrbuto s o-decreasg, satsfyg codto 3, because both upper ad lower bouds are o-decreasg. Ths s show Fgure 2. Note that the taut strg does ot have to be lear but takes the shape of the shortest path, whch could be oe of the bouds themselves depedg o the geometry of the upper ad lower bouds. 1 Taut Strg soluto betwee two dstrbutos wth Stochastc Domace B A p Theta Dst Q Dst R Taut Strg FIGURE 2. The taut strg dstrbuto betwee two dstrbutos wth stochastc domace. Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

7 Now we move to the case where there s o stochastc domace betwee the two dstrbutos. I ths case we are terested the maxmum etropy dstrbuto betwee ther upper boud, max( R, Q ), ad lower boud, m( R, Q ). Oce aga, we ca solve for ths dstrbuto usg a taut strg betwee the upper ad lower bouds. Ths s justfed by the followg argumet: If there s o stochastc domace, the the dstrbutos wll cross at least oce. Ths crossg pot comprses both upper ad lower bouds at that value of so the taut strg must pass through t. Now f we start at the frst crossg, the o-decreasg value of the cumulatve probablty dstrbutos guaratees that further crossgs wll have a hgher cumulatve probablty. A taut strg through them thus satsfes costrats (III). The fact that the dstrbutos ed at uty satsfes costrat (I) ad sce the taut strg les betwee the upper ad lower bouds, costrats (II) are satsfed. Ths s show Fgure 3. Taut Strg soluto betwee two dstrbutos wth o Stochastc Domace Dst Q Dst R Taut Strg Theta FIGURE 3. The taut strg dstrbuto betwee the upper ad lower bouds of two dstrbutos where there s o stochastc domace. We summarze the prevous results the followg theorem: Theorem 1: The Taut Strg Dstrbuto "The dstrbuto that maxmzes the etropy ad les betwee the upper ad lower bouds of two cumulatve dstrbutos ca be determed graphcally by a taut-strg dstrbuto." Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

8 A specal case of the taut strg dstrbuto s the maxmum etropy dstrbuto gve fractle costrats. We ca thk of fractles costrats as upper ad lower bouds that cocde ad so the taut strg must pass through them. Ths explas the pece wse lear shape of the FMED Fgure 1 (a). Note however, that the probablty desty fucto correspodg to ths soluto s dscotuous. Fgure 1 (b). I the ext secto we wll explore some methods that buld o the taut strg soluto gve fractle costrats ad corporate kowledge of cotuty of the correspodg probablty desty fucto. MID-POINT MAXIMUM ENTROPY DISTRIBUTION (MMED) I ths secto we show a heurstc approxmato to a dstrbuto f addto to ts fractles ad support, we also kow t s cotuous. We wll make use of a smple geometrc property of the FMED to obta ths approxmato: Property 1: Equal Area Costrat If the probablty desty fucto, from whch the fractles were assessed, s cotuous ad mootocally creasg or decreasg o ay fte terval, the ths probablty desty fucto crosses the FMED at least oce each fractle terval. Ths property s a result of the equal area costrat for both desty fuctos o each fractle terval. Further, f the dstrbuto s umodal, the t crosses the FMED exactly oce every fractle terval except for the terval cotag the mode, whch ca cross twce. Fgure 4 shows a comparso of the Beta(4,6) dstrbuto ad the FMED costructed from ts fractles. Beta(4,6) vs. FMED FMED Beta(4,6) FIGURE 4. Beta (4,6) dstrbuto ad FMED. A tersecto of the two dstrbutos occurs at each fractle terval. Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

9 Costructg The Md-pot Maxmum Etropy Dstrbuto Property (1) says there wll be at least oe crossg, however, we are ucerta about ts precse locato. For smplcty, let us assume t occurs at the md-terval for each fractle of the FMED. The md-pot of each terval s where the actual tersecto s probablstcally equally lkely to be above or below. Now, we coect these md pots ad ormalze the resultg dstrbuto. We wll call ths dstrbuto the Md- Pot Maxmum Etropy dstrbuto. Fgure 5 (a) shows the costructo of the MMED from the uboud FMED gve fractles of a Beta (4,6). The md-tervals for each er fractle are coected. The resultg MMED s show Fgure 5 (b) ad compared to the orgal Beta (4,6). Usg the uboud FMED s a good explotato to kowledge of zero values of the desty fucto at the ed pots of the support. I Fgure 6 (a) we show the bouded 1 FMED vs. a trucated expoetal, f ( x).19e, ad Fgure 6 (b), we compare t to the correspodg MMED. x.3 MMED costructo from FMED Uboud.3 Scaled Beta(4,6) vs. MMED V FMED Uboud Explaato FIGURE 5. (a). Costructo of the MMED by coectg the mdpots of er fractle tervals of the uboud FMED. (b) Comparso of MMED ad Beta (4,6) Trucated Expoetal vs. FMED Trucated Expoetal vs. MMED FIGURE 6. (a). Trucated expoetal vs. bouded FMED from ts fractles. (b) Trucated expoetal vs. MMED costructed by coectg md-terval pots of the FMED. Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

10 Comparso Wth Other Approaches Several other approaches for estmatg desty fuctos usg complete formato have bee proposed lterature. Oe method uses the theory of sples to coect the fractles o the cumulatve dstrbuto curve. Whle ths method yelds good results, t s more complex ad there s some arbtraress the order of the splcg polyomal. I addto, costrats may be eeded to prevet the polyomal from gog egatve. Other methods suggest usg a skew logstc dstrbuto whose parameters are calculated from three fractles. [6]. Fgure 7 shows the cumulatve dstrbuto for a Beta(4,6), the MMED, ad a skew logstc dstrbuto costructed from fractles of the Beta (4,6). The skew logstc dstrbuto method yelds very good results ad oly requres three fractles to determe the parameters of the equvalet dstrbuto. The problem, (as metoed Ldley s paper), s that there s a lmt o the amout of skewess that ca be modeled by a skew logstc dstrbuto ad t ca oly be appled to umodal dstrbutos. CDF for Beta, MMED ad Skew Logstc Dstrbuto Skew Logstc Beta MMED Beta(4,1) MMED Skew Logstc Ft FIGURE 7. Skew Logstc Dstrbuto vs. MMED for a Beta (4,6) I Fgure 8, we show a comparatve example for the closeess of a Beta (4,6) to ts estmated dstrbutos usg dfferet methods. We use the total varato as a measure of closeess. The total varato betwee two dstrbutos s the sum of absolute dfferece probabltes. Total Varato p q (9) 1 Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

11 The total varato also has a terestg terpretato. It ca be show that f the total varato betwee two dstrbutos, P ad Q, s, the the maxmum dfferece betwee ay two dscrete probabltes, p ad q, caot exceed. 2 From the results of Fgure 8, we ca see that the MMED outweghs the other approaches o the total varato measure. Its ma drawback, however, s that ts desty fucto, despte cotuous, s pecewse lear. O the other had, t s very smple to costruct ad gves good approxmatos for fereces about a cotuous, or dscrete, dstrbuto wth fractle costrats. Total Varato vs. Estmato Method Oe- Mo met FMED Skew Logs tc Tw o- Momets Three- Momets Four- Momets MMED Total Varato FIGURE 8. Comparso of total varato for dfferet estmato methods CONCLUSIONS We have show a applcato of the maxmum etropy prcple to the assgmet of uvarate dstrbutos ad preseted a graphcal method to determe the maxmum etropy dstrbuto betwee upper ad lower probablty bouds. We also preseted a heurstc approxmato to a cotuous dstrbuto gve fractle costrats. We compared ths approxmato to several other approaches to llustrate ts smplcty ad closeess to the orgal dstrbuto. REFERENCES 1. Kahema, D., ad Tversky, A. Subjectve probablty: A judgmet of represetatveess. Cogtve Psychology. 3, (1972). Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22

12 2. Spetzler, C. Probablty ecodg decso aalyss Readgs o the Prcples ad Applcatos of Decso Aalyss, Strategc Decsos Group, Melo Park, CA.(1972) 3. Abbas, A. A Etropy Approach for Utlty Assgmet Decso Aalyss Proceedgs of the 22 d Iteratoal Workshop o Bayesa Iferece ad Maxmum Etropy Methods Scece ad Egeerg, edted by Wllams, C. (22). 4. Jayes, E.T. Iformato theory ad statstcal mechacs. Physcal Revew, 16, 62-63, (1957). 5. Veott, A. Least d-majorzed Network Flows wth Ivetory ad Statstcal Applcatos Maagemet Scece, 17, , (1971). 6. Ldley, D.V. Usg expert advce o a skew judgmetal dstrbuto Operatos Research, 35, , (1987) 7. Shao, C. "The Mathematcal Theory of Commucato," Bell System Techcal Joural July ad October (1948). 8. Cover, T. ad Thomas, J. Elemets of Iformato Theory. Joh Wley ad Sos Ic. New York. N.Y. (1991). 9. Howard, R.A. Rsk Preferece, Readgs o the prcples ad applcatos of decso aalyss. Strategc Decsos Group. Melo Park, CA, (1983) Ths artcle s copyrghted as dcated the artcle. Reuse of AIP cotet s subject to the terms at: Dowloaded to IP: O: Mo, 29 Ju :17:22