Model-Based synthesis of indicators Statistical Composite Indicators to convey consistent policy messages

Size: px

Start display at page:

Download "Model-Based synthesis of indicators Statistical Composite Indicators to convey consistent policy messages"

Ernest Dean
5 years ago
Views:

Model-Based synthesis of indicators Statistical Composite Indicators to convey consistent policy messages Carlo Cavicchia, Maurizio Vichi Department of Statistical

1 Model-Based synthesis of indicators Statistical Composite Indicators to convey consistent policy messages Carlo Cavicchia, Maurizio Vichi Department of Statistical Sciences Sapienza University of Rome em: Workshop:THE IMPACTS AND METHODOLOGY OF INDICATORS AND SCOREBOARDS -3 March 018

2 Outline of the Presentation FROM DATA to KNOWLEDGE by Dimensionality Reduction: a model-based Composite Indicator (CI) is the result of the dimensional reduction of the observed multivariate data. We will start from Properties on which a Composite Indicator should be based We will discuss HCI : Hierarchical Composite Indicator Model Application study: Human Development Index; Multidimensional Poverty Index SDGs for Europe with data by EUROSTAT

3 From DATA to KNOWLEDGE via GENERAL COMPOSITE INDICATOR Data Dimensionality Reduction KNOWLEDGE for DECISION MAKING SPECIFIC COMPOSITE INDICATORS INFORMATION Heat-map DATA 3

4 Moving from DATA to KNOWLEDGE by a Dimensionality Reduction 4

5 Let us start from the Observed Data (Manifest Indicators) on the multidimensional concept formed by DATA 5 Observed set of indicators

6 Specific Composite Indicators (1/) for detecting the specific concepts describing the phenomenon First Level Synthesis First Order Specific Composite Indicators COMPOSITE INDICATOR DATA 6 Each SCI synthesises a set of indicators Each SCI synthesises a set of indicators

7 Specific Composite Indicators (/) for detecting the main concepts describing the phenomenon Second Level Synthesis Second Order Specific Composite Indicators COMPOSITE INDICATOR DATA 7 Each SCI synthesises a set of indicators Each SCI synthesises a set of indicators or factors

8 General Composite Indicator (for decision making) Third Level Synthesis General Composite Indicator GENERAL COMPOSITE INDICATOR Second Order Specific Composite Indicators COMPOSITE INDICATOR DATA Each SCI synthesises a set of Indicators Each SCI synthesises a set of indicators or SCIs 8 Hierarchical CI model to move from DATA to KNWOLEDGE

9 PROPERTIES of CI FIRST PART Model-Based & Statistically estimated (non-normative) Confirmatory, Exploratory or Mixed Reflective and/or Formative Model assessment SECOND PART Scale-invariance Non-Compensability & Non-Negativity Reliability Unidimensionality, forming a General CI 9

10 PROPERTIES of CI FIRST PART Model-Based & Statistically estimated (non-normative) Confirmatory, Exploratory or Mixed Reflective and/or Formative Model assessment SECOND PART Scale-invariance Non-Compensability & Non-Negativity Reliability Unidimensionality, forming a General CI 10

11 Statistical Model: Hierarchical CI Model-based CI & its statistical estimation (i.e., non-normative): Data = Hierarchical CI model + error Manifest Indicators Measurement error + residual Advantages Statistical estimation (LS, MLE, ) Validation: Goodness of Fit (to confirm the model) Inference on the weights, GoF, Which typology of constructive approach: - Confirmatory a Scientific Theory (ST) is assumed and has to be confirmed by the observed indicators; - Exploratory no clear ST is known, thus, regularities are searched in the data; - Mixed Confirmatory & Exploratory part of the ST is known, but it is not completely known Which typology of relations between indicators: - Reflective - Formative 11

12 Confirmatory, Exploratory, Mixed-Confirmatory/Exploratory Confirmatory model: if a theory on the model of the CI is available, i.e., all relationships between manifest variables and latent variables are and a priori known; Exploratory model: all relationships between manifest variables and latent variables are not a priori known; Mixed-confirmatory/exploratory : some relationships are known according to a theory and some are unknown and must be achieved by exploratory analysis. A GCI B GCI C?? SCI1 SCI SCI1 SCI SCI1 GCI SCI????????? e 1 e e 3 e 4 e 5 e 6 e 1 e e 3 e 4 e 5 e 6 e 1 e e 3 e 4 e 5 e 6 1

Relations between Composite Indicators (GCI & SCIs) and Manifest Indicators A) Reflective B) Formative GCI GCI A B SCI1 SCI SCI1 SCI 1 3 4 5 6 1 3 4 5 6 e 1 e e 3 e 4 e 5 e 6 The General Composite

13 Relations between Composite Indicators (GCI & SCIs) and Manifest Indicators A) Reflective B) Formative GCI GCI A B SCI1 SCI SCI1 SCI e 1 e e 3 e 4 e 5 e 6 The General Composite Indicator is a determinant (causes) the Specific Composite Indicators & these last are determinant (causes) of the Manifest Indicators, i.e., The GCI reconstructs the SCIs that reconstruct the MI e 1 e e 3 e 4 e 5 e 6 Independent Manifest Indicators are determinant (cause, explain) of independent Specific Composite indicators that are determinant of the General Composite Indicator) 13

14 Hierarchical Composite Indicator (HCI) a model to identify the latent Hierarchical Composite Indicator and the set of specific Composite Indicators that best reconstruct the observed data SOME METODOLOGICAL CONSIDERATIONS GCI SCI SCI 3 1 SCI V = b 1 b 4 b 5 b 6 b 7 b b 3 c e 1 e e 3 e 8 e e 9 e 10 4 e 5 e 6 e 7 g c c 3 c b 8 b 9 b 10 Data MODEL ERROR X=gc' V' B+E Parameters of the model c weights for SCIs V relations between MIs & SCIs Weights for variables B = diag(b) b 1, b,, b 10 14

15 Special cases of HDFA(1/) g=arithmetic mean of MIs if :c 1 =c = =c Q =1; b 1 =b = b J =1 (equal weights) g M = X(1 H V ) + = X1 J + = 1 J (x 1 + x +...+xj), GCI SCI SCI 3 1 SCI e 1 e e 3 e 4 e 5 e 6 e 7 e 8 e 9 e 10 g c c 1 =1 c =1 c3 =1 b 1 =1 b =1 b 3 =1 b 4 =1 b 5 =1 b 6 =1 b 7 =1 b 8 =1 b9 =1 b 10 =1 Data MODEL ERROR X=gc' V' B+E Parameters of the model c weights for SCIs V relations between MIs & SCIs V = Weights for variables B = diag(b) b 1, b,, b 10 14

16 Special cases of HCI (/) g = weighted arithmetic mean of MIs (i.e., different weights) g WM =X( B V c)( c V B1 J ) -1 Data MODEL ERROR c 1 GCI SCI SCI 3 1 SCI e 1 e e 3 e 4 e 5 e 6 e 7 e 8 e 9 e 10 g c c c 3 b 1 b b 3 b 4 b 5 b 6 b 7 b 8 b 9 b 10 X=gc' V' B+E Parameters of the model c weights for SCIs V relations between MIs & SCIs V = Weights for MIs B = diag(b) b 1, b,, b 10 14

17 PROPERTIES of CI FIRST PART Model-Based & Statistically estimated (non-normative) Confirmatory, Exploratory or Mixed Reflective and/or Formative Model assessment SECOND PART Scale-invariance Non-Compensability & Non-Negativity Reliability Unidimensionality, forming a General CI 17

18 MODEL ASSESSMENT The goodness of fit of the CI model: R GCI = 1 SS res = 1 tr(x X) tr( B V SS tot tr(x X) c g g( c V B)) R SCI = 1 SS res = 1 tr(x X) tr( B V Y Y V B) SS tot tr(x X) R SCIh = 1 SS res Yh SS toth = 1 tr(x h X h ) tr( B h v h y h y h v h B h ) tr(x h X h ) The Information criteria AIC -log Ol(, ) + d BIC -log Ol(, ) + d log n 19

0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 X s X m X L Error: R GCI R SCI1 Small 0.974 0.988 0.988 0.989 Medium 0.

19 Example 1 : Assessment of the Model-Based CI X=gc' V' B+E Case of ARITHMETIC MEAN g GCI 1 1 c 1 g N(0,1) SCI 1 SCI SCI 3 X s = X m = X L = B e 1 e e 3 e e 8 e 9 e 10 4 e 5 e 6 e 7 if c=1 3 and B =I 10, V = X s X m X L Error: R GCI R SCI1 Small Medium Large R SCI R SCI3 Arithmetic mean is a good GCI only when the MIs are very similar 0

20 Example : Assessment of the Model-Based CI Case of ARITHMETIC MEAN GCI SCI 1 SCI SCI 3 X= e 1 e e 3 e e 8 e 9 e 10 4 e 5 e 6 e 7 X divided in three blocks R GCI = SS mod SS tot = R SCI1 R SCI R SCI In a situation like this is better to stop at an intermediate level of synthesis (i.e., SCIs level) because a GCI built as the arithmetic mean of MIs is not a good representation of the phenomenon to describe 1

21 Example : Assessment of the Model-Based CI Case of ARITHMETIC MEAN GCI SCI 1 SCI SCI 3 X= e 1 e e 3 e e 8 e 9 e 10 4 e 5 e 6 e 7 R GCI = SS mod SS tot = R SCI1 R SCI R SCI In a situation like this is better to stop at an intermediate level of synthesis (i.e., SCIs level) because a GCI built as the arithmetic mean of MIs is not a good representation of the phenomenon to describe

Example : Assessment of the Model-Based CI Case of ARITHMETIC MEAN Final correct description SCI 1 SCI SCI 3 X= 1 3 4 5 6 8 7 9 10 e 1 e e 3 e e 8 e 9 e 10 4 e 5 e 6 e 7 R GCI = SS mod SS tot = 0.

22 Example : Assessment of the Model-Based CI Case of ARITHMETIC MEAN Final correct description SCI 1 SCI SCI 3 X= e 1 e e 3 e e 8 e 9 e 10 4 e 5 e 6 e 7 R GCI = SS mod SS tot = R SCI1 R SCI R SCI In a situation like this is better to stop at an intermediate level of synthesis (i.e., SCIs level) because a GCI built as the arithmetic mean of MIs is not a good representation of the phenomenon to describe

23 PROPERTIES of CI FIRST PART Model-Based & Statistically estimated (non-normative) Confirmatory, Exploratory or Mixed Reflective and/or Formative Model assessment SECOND PART Scale-invariance Non-Compensability & Non-Negativity Reliability Unidimensionality, forming a General CI 3

24 Scale-invariance Data are normalized in order to allow the comparison and the combination of the MIs into the SCIs and GCI. Standardization Z = JXdiag(dg(Σ X )) 1/ with J=I n (1/n) 1 n 1 n Min-max normalization Z = X 1 n minx./(1 n maxx-1 n minx) Normalized dispersion Z = JXdiag(μ X ) -1 with J=I n -(1/n) 1 n 1' n A scale-invariant CI is a latent Indicator that is not sensitive to linear transformations such as normalization methods. 4

25 PROPERTIES of CI FIRST PART Model-Based & Statistically estimated (non-normative) Confirmatory, Exploratory or Mixed Reflective and/or Formative Model assessment SECOND PART Scale-invariance Non-Compensability & Non-Negativity Reliability Unidimensionality, forming a general CI 5

26 Non-Compensability & Non-Negativity. The CI satisfies the non-compensability property if its relationships with latent and/or MIs are all positives. Thus, the effect of the SCIs and/or MIs do not compensate each other. Ranking of the not compensated model: GCI SCI1 SCI e 1 e e 3 e 4 e 5 e 6 So non-negativity and non-compensability are strictly connected. 6

27 Non-Compensability & Non-Negativity. The CI satisfies the non-compensability property if its relationships with latent and/or MIs are all positives. Thus, the effect of the SCIs and/or MIs do not compensate each other. Ranking of the not compensated model: GCI Ranking of the compensated model: SCI1 SCI e 1 e e 3 e 4 e 5 e 6 So non-negativity and non-compensability are strictly connected. 7

28 PROPERTIES of CI FIRST PART Model-Based & Statistically estimated (non-normative) Confirmatory, Exploratory or Mixed Reflective and/or Formative Model assessment SECOND PART Scale-invariance Non-Compensability & Non-Negativity Reliability Unidimensionality, forming a general CI 8

29 Reliability, Unidimensionality & General Factor Reliability of a CI is the global consistency of MIs based on the correlations between different MIs on the same CI. It is frequently called internal consistency and it is usually measured with Cronbach's alpha (Cronbach, 1951) Unidimensionality evaluates to which extend a single latent indicator, generally a SCI, has been measured with a set of MIs. Unidimensionality is more realistic for SCIs, while Revelle and Zinbarg, (009) hypothesize that there is a general factor, i.e., a GCI that can be tested by nested confirmatory SCIs. A measure of unidimensionality for each SCI might be the variance of the second component of the the set of MIs explained by the related SCI. Example: GCI Factor 1 Factor Unidimensionality Reliability SCI 1 SCI SCI Factor 1 Factor Factor 3 Unidimensionality Reliability e 1 e e 3 e 4 e 5 e 6 e 7 e 8 e 9 e 10 8

30 30 APPLICATIONS

Human Development Index - HDI The HDI is the geometric mean of the previous three normalized indices; and we can measure the goodness of fit of the HDI by considering that the logarithm of the

31 Human Development Index - HDI The HDI is the geometric mean of the previous three normalized indices; and we can measure the goodness of fit of the HDI by considering that the logarithm of the geometric mean is equal to the arithmetic mean of the logarithm of MIs. Each dimension is represented by a specific index(normalized with a own method): Life Expectancy Index (LEI), Education Index (EI) and Income Index (II). Let us consider: B = V = I 3 R HDI = SS mod = SS tot c = 1 3 = tr B V c log g HDI log g HDI c V B tr( log X log X ) Based on the above informations: Life Expectancy Index (LEI) = Actual LE 0/(85-0) Income Index (II) = {ln(gni pc)- ln(100)}/{ln(75,000) ln(100)} Education Index (EI) = MYSI+EYSI / Mean Years of Schooling Index (MYSI) = MYS-0 / 15-0 Expected Years of Schooling Index (EYSI) = EYS-0 / 18-0 Now, HDI is the geometric mean of previous three indices i.e. HDI= where log X is a matrix where each column is the logarithmic transformation of the respectively column of X.

Multidimensional Poverty Index- MPI The global Multidimensional Poverty Index (MPI) is an international measure of acute poverty covering over 100 developing countries developed by OPHI and the

32 Multidimensional Poverty Index- MPI The global Multidimensional Poverty Index (MPI) is an international measure of acute poverty covering over 100 developing countries developed by OPHI and the United Nations Development Programme. The index uses the same three dimensions as the Human Development Index: health, education, and standard of living. These are measured using ten indicators divided in three dimensions. Let us consider: B = diag( ) V = c = ( ) R MPI = SS mod SS tot = tr(( c V B B V c) 1 B V c g MPI g MPI ( c V B)( c V B B V c) 1 ) tr(x X) =

33 Application to Sustainable Development Goals 30

SDGs Europe: 100 Indicators, 17 Goals Goal1: 1- People at risk of poverty or social exclusion 01.11 - People at risk of poverty after social transfers 01.1 3- Severely materially deprived people 01.

1 6- Share of total population living in a dwelling with a leaking roof, damp walls, floors or foundation, or rot in window frames or floor 01. Goal3: 13- Life expectancy at birth 03.

36 18- Self-reported unmet need for medical examination and care 03.41 Goal: 7- Obesity rate 0.11 8- Agricultural factor income per annual work unit (AWU) 0.

34 SDGs Europe: 100 Indicators, 17 Goals Goal1: 1- People at risk of poverty or social exclusion People at risk of poverty after social transfers Severely materially deprived people People living in households with very low work intensity Housing cost overburden rate Share of total population living in a dwelling with a leaking roof, damp walls, floors or foundation, or rot in window frames or floor 01. Goal3: 13- Life expectancy at birth Self-perceived health Death rate due to chronic diseases Suicide death rate Smoking prevalence Self-reported unmet need for medical examination and care Goal: 7- Obesity rate Agricultural factor income per annual work unit (AWU) Government support to agricultural research and development Area under organic farming Ammonia emissions from agriculture Gross nutrient balance on agricultural land 0.54 Goal4: 19- Early childhood education and care Early leavers from education and training Tertiary educational attainment Employment rate of recent graduates Adult participation in learning Underachievement in reading, maths and science Goal5: 5- Gender pay gap Gender employment gap Proportion of seats held by women in national parliaments and local government Proportion of women in senior management positions Physical and sexual violence by a partner or a non-partner Inactivity rates due to caring responsibilities Goal7: 37- Percentage of people affected by fuel poverty (inability to keep home adequately warm) Share of renewable energy in gross final energy consumption Primary energy consumption; final energy consumption by sector Final energy consumption in households per capita Energy dependence Energy productivity Goal6: 31- Share of total population having neither a bath, nor a shower, nor indoor flushing toilet in their household Population connected to urban wastewater treatment with at least secondary treatment Biochemical oxygen demand in rivers Nitrate in groundwater Phosphate in rivers Water exploitation index (WEI) Goal8: 43- Real GDP per capita - growth rate Young people neither in employment nor in education and training Total employment rate Long-term unemployment rate Involuntary temporary employment Fatal accidents at work by sex (NACE Rev., A, C-N) - Unstandardised incidence rate

35 35 Goal9: 49- Gross domestic expenditure on R&D Employment in high- and medium-high technology manufacturing sectors and knowledgeintensive service sectors Total R&D personnel Patent applications to the European Patent Office (EPO) Share of collective transport modes in total passenger land transport Share of rail and inland waterways activity in total freight transport Goal11: 61- Overcrowding rate by degree of urbanisation Distribution of population by level of difficulty in accessing public transport People killed in road accidents Urban population exposure to air pollution by particulate matter Proportion of population living in households considering that they suffer from noise Recycling rate of municipal waste 11.5 Goal13: 73- Greenhouse gas emissions (indexed totals and per capita) Greenhouse gas emissions intensity of energy consumption Global (and European) near surface average temperature Economic losses caused by climate extremes (consider climatological, hydrological, meteorological) Contribution to the 100bn international commitment on climate related expending (public finance) Share of EU population covered by the new Covenant of Mayors for Climate and Energy (integrating mitigation, adaptation, and access to clean and affordable energy) Goal15: 84- Forest area as a proportion of total land area Artificial land cover per capita Change in artificial land cover per year Common bird index Sufficiency of terrestrial sites designated under the EU habitats directive Estimated soil erosion by water Goal17: 96- Official development assistance as share of gross national income EU financing for developing countries EU Imports from developing countries General government gross debt Shares of environmental and labour taxes in total tax revenues Goal10: 55- GDP per capita in PPS Real adjusted gross disposable income of households per capita in PPS Relative median at-risk-of-poverty gap Gini coefficient of equivalised disposable income Income growth of the bottom 40 per cent of the population and the total population Number of first time asylum applications (total and accepted) per capita Goal1: 67- Generation of waste excluding major mineral wastes Recycling and landfill rate of waste excluding major mineral wastes Consumption of toxic chemicals Resource productivity Average CO emissions per km from new passenger cars Volume of freight transport relative to GDP 1.54 Goal14: 79- Bathing water quality Sufficiency of marine sites designated under the EU habitats directive Ocean acidification (CLIM 043) Catches in major fishing areas Assessed fish stocks exceeding fishing mortality at maximum sustainable yield (Fmsy) Goal16: 90- Death due to homicide, assault, by sex (tps00146) 91- Share of population which reported occurrence of crime, violence or vandalism in their area General government total expenditure on law courts Corruption Perception Index Perceived independece of the justice system Level of citizens' confidence in EU institutions 16.6

ASSESSMENT of HCI model: 17 goals BIC= 47.

MIs are not statistically significant for the model (correlation 0) (They are

36 ASSESSMENT of HCI model: 17 goals BIC= Polarity: 38 MIs need to change polarity 100 Manifest Indicators 6 for each goal 33 MIs are not statistically significant for the model (correlation 0) (They are STATISTICS, but not INDICATORS) Reliability: 8 goals are not reliable (low Cronbach's alpha) Unidimensionality: only the goal 14 is unidimensional 33

Exploratory model: 8 factors BIC= 1633.78 SCI 1 0.95 0.9 0.77 0.83 SCI 8 0.36 0.73 0.94 0.56 0.49 0.64 0.05 0.

37 Exploratory model: 8 factors BIC= SCI SCI SCI SCI 7 SCI 3 SCI 6 * 8 MIs are not statistically significant for the model SCI 4 SCI 5 34

38 Thank you for your kind attention.

BUILDING SOUND AND COMPARABLE METRICS FOR SDGS: THE CONTRIBUTION OF THE OECD DATA AND TOOLS FOR CITIES AND REGIONS

BUILDING SOUND AND COMPARABLE METRICS FOR SDGS: THE CONTRIBUTION OF THE OECD DATA AND TOOLS FOR CITIES AND REGIONS STATISTICAL CAPACITY BUILDING FOR MONITORING OF SUSTAINABLE DEVELOPMENT GOALS Lukas Kleine-Rueschkamp