The Metalog Distributions

Size: px

Start display at page:

Download "The Metalog Distributions"

Alan Walker
5 years ago
Views:

1 Keelin Reeds Partners 770 Menlo Ave., Ste 230 Menlo Park, CA phone The Metalog Distributions A Soon-To-Be-Published Paper in Decision Analysis, and Supporting Website Presented at the INFORMS Annual Meeting Nashville, Tennessee By Tom Keelin November 13, 2016 Page 0

2 3-term equation: What s this? M 3 (y) 3-term metalog distribution varying skewness parameter a 3 varying kurtosis parameter a 4 add a 4 th term M 4 (y) 4-term metalog distribution generalizes to any number of terms M n (y) Page 1

3 Metalog constants a i are determined linearly from CDF data. y (x 1,y 1 ) Cumulative Probability (x 3,y 3 ) (x 2,y 2 ) (x m,y m ) x invertibility guaranteed except in pathological cases case 1 * case 2 works either way Feasibility of : PDF (probability density function) is everywhere positive. Easy to check by plotting * Excel array formula for column vector : Page 2

4 Metalog moments are closed-form polynomials of the a i s. For example, for the 4-term metalog More generally, the k th central moment of the n-term metalog is simply a k th -order polynomial of the a i s. Page 3

5 How about simple and flexible semi-bounded or bounded distributions? Name Interpretation CDF (quantile function) Condition metalog (unbounded) semi-bounded metalog bounded metalog generalized logistic distribution log x is metalog distributed logit( ) = ln ( ) is metalog distributed = + ln ln 1 + = + = = + 1+ = = 0< <1 0< <1 =0 (given lower bound ) 0< <1 =0 =1 (given lower and upper bounds and ) Page 4

6 Types of Continuous Probability Distributions Basis of Legitimacy Criteria Examples Type I Derived from an underlying probability model Distribution reflects the model normal exponential Type II Matches specific types of empirical data Distribution matches data generalized logit-normal (Mead, 1965) skewed generalized t distribution (Theodossiou,1994) (dozens of others) Type III Matches most any set of empirical (or assessed) data Flexibility Simplicity Ease of use Pearson distributions (1895,1901,1916) Johnson distributions (1949,1982) Quantile parameterized distributions (Keelin and Powley, 2011) Metalog distributions (this research) Page 5

7 What did Pearson do specifically? Normal Distribution y' x 12 solutions to Pearson s differential equation normal pdf: f(x) = differential equation: f (x) f(x) = - - Pearson s modification: f (x) f(x) = The E = mc 2 of classical statistics Page 6

8 Strenghts and Shortcomings of the Pearson System 1 uniform 2 normal 3 logistic 4 Gumbel t 7 df: exponential 9 β 2 (kurtosis) Flexibility: Can match any combination of skewness and kurtosis triangular Pearson unbounded (Pearson 4, t) F Pearson semi-bounded Pearson bounded (beta) Shortcomings: Limited to 2 shape parameters Given a point (β 1, β 2 ), Pearson and system offers zero choice of boundedness zero ability to match 5 th or higher-order moments A dozen functional forms, some of which are duplicative, with incomplete guidance for which to use. Parameter estimation can require non-linear optimization (with situation-specific manual intervention). β 1 (skewness^2) Page 7

9 Engineering a new probability distribution strategy table Base Distribution Form Selected for Modification Modification Method Distribution Selection Parameter Estimation normal (Edgeworth 1896, 1907; probability density function (PDF) parameter addition ability to match moments method of moments Pearson 1895,1901,1916; (Edgeworth 1896, 1907; (Mead, 1965; (Pearson 1895,1901,1916; (Pearson 1895,1901,1916) Charlier 1928; Johnson Pearson 1895,1901,1916; Theodossiou,1994) Johnson 1949; Tadikamalla 1949) Charlier 1928; ) and Johnson 1982) logistic (Tadikamalla and Johnson 1982; Balakrishnan, 1992) cumulative distribution function (CDF) parameter substitution (Pearson 1895,1901,1916) match natural bounds maximum likelihood (Fisher 1932) (Burr, 1942) student t (McDonald and Newey, 1988; Theodossiou,1994) quantile function (inverse CDF) (Keelin and Powley, 2011) transformation (Johnson 1949; Tadikamalla and Johnson 1982; Hadlock probabilityweighted- and L- moments and Bickel, tbd) (Greenwood, et. al. 1979; Hosking, 1990) characteristic function series expansion (Edgeworth 1896, 1907; quantile parameterized (Ord 1972) Charlier 1928) (Keelin and Powley, Hadlock and Bickel, tbd.) Page 8

10 Logistic Distribution logistic cdf logistic pdf y y' x x cdf: y = pdf: y = quantile function: x = pdf: y = where: Page 9

11 The Metalog: A Generalized Logistic Distribution quantile function: where: x = series expansion thus: x = pdf: y [ ] -1 where: = number of series terms in use. s are constants. metalog: short for meta-logistic Page 10

12 Flexibility Comparison: Metalog vs. Pearson Distributions 1 uniform normal Flexibility: Metalog flexibility expands with number of terms triangular logistic Gumbel Example: Unbounded Metalog Shape Flexibility Metalog Relative Strengths: Unlimited shape parameters For many areas of (β 1, β 2 ), the metalog offers choice of boundedness exponential ability to match 5 th and higher-order moments 10 β 2 (kurtosis) Pearson unbounded β 1 Pearson semi-bounded (skewness^2) Pearson bounded 3 functional forms (one each for unbounded, semi-bounded, and bounded) Linear quantile parameterization Metalog Relative Weaknesses: Certain very extreme distributions (e.g. Cauchy with infinite moments) have no good metalog representation. Page 11

13 Metalogs can effectively represent a wide range of previously-named distributions. 5 terms 5 terms metalog Source: extreme value ( =100, =20, = 0.5) metalog Source: beta ( =0.8, =0.9, =10, =50) 5 terms 5 terms pdf: y = metalog Source: exponential (λ=0.5) metalog Source: triangular ( =20, =10, =50) Page 12

14 Metalog representations are increasingly accurate with increased numbers of terms. Probability Density terms 5-terms 0.02 y' x 10-term metalog 5-term metalog Source source Source: extreme value ( =100, =20, = 0.5) Page 13

Application 1: Fish Biology By enabling the data to

fish records 2010-2014. Babine River, British Columbia.

2 salt fish-biology research questions: fish weights

15 Application 1: Fish Biology By enabling the data to speak for itself, metalogs can transform data into knowledge. Steelhead Trout Weight (lbs) 3,474 catch-and-release fish records Babine River, British Columbia. 1 salt vs. 2 salt fish-biology research questions: fish weights (relative and absolute)? relative population sizes? Steelhead Life Cycle river 1 salt 2 salt Silver Hilton Steelhead Lodge 0 age (years) 10 ocean Page 14

Steelhead Trout Weight (lbs) 3,474 catch-and-release fish

16 Application 1: Fish Biology By enabling the data to speak for itself, metalogs can transform data into knowledge. Steelhead Trout Weight (lbs) 3,474 catch-and-release fish records Babine River, British Columbia. 2-salt? 10-terms 1-salt? Page 15

17 Application 1: Fish Biology Metalog family molds itself to the data -- potentially telling a more nuanced story than previous Type III families. Metalog Panel for Fish Biology Data (n = 2-16 terms) n=2 n=3 n=4 n=5 n=6 n=7 n=8 n=9 n=10 n=11 b_ c_ n=12 n=13 n=14 n=15 n=16 Page 16

Application 2: Hydrology Metalogs enable

Williamson River (below the Sprague River),

18 Application 2: Hydrology Metalogs enable examination of whether the shape of the data is consistent with a given Type I model. Maximum Annual River Gauge Height (ft) Williamson River (below the Sprague River), near Chiloquin, Oregon. USGS data terms Page 17

19 Application 3: Decision Analysis In decision analysis, metalogs can convert simply expert assessments into real-time representations and feedback. Case 1 Case 2 Expert assessments with any number of data parameters (including inconsistent ones) Symmetric-percentile-triplet (SPT) parameters Cumulative Probability y x Probability Density Asset 1 CDF CDF 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0 5,000 10,000 15,000 20,000 25,000 30,000 Data Metalog (n=3) metalogs pass through all 3 points exactly simplified expressions for constants a i and feasibility, e.g. y' x enables real-time feedback as each point is added Asset 1 PDF PDF ,000 10,000 15,000 20,000 25,000 30,000 Metalog (n=3) Given =( α,., α)then = q. = ln α α α α Page 18

20 Application 3: Bidding decision analysis A bidding decision that would have been made wrongly if 3- branch discrete probabilities had been relied upon. SPT assessments for 259 assets Simulated portfolio value (sum of 259 asset values) discrete simulation vs. continuous (metalog) simulation Asset 1 CDF CDF 100% 90% 80% 70% 60% 50% 40% 30% % % 0% ,000 10,000 15,000 20, ,000 30,000 Data Metalog (n=3) Asset 1 PDF PDF ,000 10,000 15,000 20,000 25,000 30, , , , , , , , ,000 Portfolio Value ($ '000s) Discrete Simulation Metalog (n=5) Continuous Simulation Metalog (n=5) Metalog (n=3) 3-branch discretizations cut off the tails which would have led to a wrong bidding-decision in this actual case. Page 19

21 Summary Metalogs provide simple, flexible, easy-to-use continuous probability distributions to represent CDF data. Allow frequency data to "speak for itself" with highly-flexible continuous representations. Select among unbounded, semi-bounded, or bounded distributions Skip time-consuming parameter estimation Facilitate Monte Carlo Simulation by convenient Sampling from input distributions Representing simulation outputs as smooth, continuous distributions Use simple, closed-form equations -- easily-programmable-in-excel -- for quantile function and PDF. For Excel workbooks, publications, and supporting information, go to Page 20

Selected References Balakrishnan, Handbook of the Logistic Distribution, Marcel Dekker, 1992 Burr, Irving W. "Cumulative frequency functions." The Annals of mathematical statistics 13.

22 Selected References Balakrishnan, Handbook of the Logistic Distribution, Marcel Dekker, 1992 Burr, Irving W. "Cumulative frequency functions." The Annals of mathematical statistics 13.2 (1942): Greenwood, J. Arthur, et al. "Probability weighted moments: definition and relation to parameters of several distributions expressable in inverse form. "Water Resources Research 15.5 (1979): Johnson, Norman L. "Systems of frequency curves generated by methods of translation." Biometrika 36.1/2 (1949): Johnson, Kolz, and Balakrishnan, Continuous Univariate Distributions, 2nd Edition, Wiley, New York, Karvanen, Juha. "Estimation of quantile mixtures via L-moments and trimmed L-moments." Computational Statistics & Data Analysis 51.2 (2006): Keelin, Thomas W., and Bradford W. Powley. "Quantile-parameterized distributions." Decision Analysis 8.3 (2011): Mead, R. "A generalised logit-normal distribution." Biometrics 21.3 (1965): Ord, Families of Frequency Distributions, Griffin, London 1970 Tadikamalla, Pandu R., and Norman L. Johnson. "Systems of frequency curves generated by transformations of logistic variables." Biometrika 69.2 (1982): Theodossiou, Panayiotis. "Financial data and the skewed generalized t distribution." Management Science part-1 (1998): Page 21

LQ-Moments for Statistical Analysis of Extreme Events

Journal of Modern Applied Statistical Methods Volume 6 Issue Article 5--007 LQ-Moments for Statistical Analysis of Extreme Events Ani Shabri Universiti Teknologi Malaysia Abdul Aziz Jemain Universiti Kebangsaan