Introductory compositional data (CoDa)analysis for soil

Transcription

1 Introductory compositional data (CoDa)analysis for soil 1 scientists Léon E. Parent, department of Soils and Agrifood Engineering Université Laval, Québec

2 2 Definition (Aitchison, 1986) Compositional data are strictly positive data constrained to a closed space (0,1) defined by a d-dimensional simplex S d Open simplex: S D = u u d < 1 Closed simplex: S D = u u d+1 = 1 where u d+1 is the filling value to 1 As part of some entity, compositional data are intrinsically multivariate and interrelated, some part «filling» information holes left by others

3 3 Source of biais in CoDa Sources of bias (distortion) when analyzing CoDa Redundancy: there are D-1 degrees of freedom for a D-part composition (one component is redundant as computed by difference) Subcompositional incoherence (scale dependency): results of statistical analyses depend on the basis (e.g. dry, organic or wet mass): adding more components should not influence the conclusion about any subcomposition Non-normal distribution (confidence intervals may scan beyond the compositional space, i.e. below 0 or above 100%

4 4 Redundancy The compositional space is constrained between 0 and some whole such as 1, 100%, 1000 g kg -1, 10 6 mg kg -1, 1 m 3 m -3, etc. Soil texture can be defined as %sand + %silt + %clay. Hence, %sand can be computed by difference between 100% and the sum of %silt and %clay Soil texture can be illustrated by a ternary diagram or a 2-D scatter diagram

5 5 Redundancy bubble for a D-part composition (Parent et al., 2012) Dual ratio a b: D D 1 2 a b + c Parent et al., 2012)

6 6 Subcompositional incoherence (scale dependency) Results of a statistical analysis should not depend on the scale of measurement, but this is too often ignored Sand + silt + clay + OM + water = 100% Sand + silt + clay + OM = 100% Sand + silt + clay = 100% Soil texture is 40% sand, 35% silt, and 25% clay; SOM = 4%; water content = 20% Percentage of textural components depends on the basis Subcomposition (scale) Sand Silt Clay SOM Water Sum Texture Texture + SOM Texture + SOM + water

7 7 Spurious correlations Correlation coefficients vary with measurement scale or subcomposition Closure to 100% depends on what composition makes up this 100% (scale dependency) CoDa are relative to each other, not asbolute values Mixture contains12 g of B and 8 g of E hence a 20 g subcomposition made of B+E B makes 60% of this subcomposition closed to 20 g and E, 40% B E = 12 8 = = 1,5 for this subcomposition (scale invariance) If total mass of the mixture is 80 g, the B E ratio remains the same (1.5) although the proportion of B in the whole mixture becomes 15% and that of E, 10% (scale-dependency). To back transform ratios into operational units (here 20 g), solve 2 equations for 2 unknowns B E = 1,5 and B + E = 20g

8 8 Spurious correlations One component as basis: a random selection of a 3-part composition (correlation expected to be zero) will generate spurious correlation if two parts are ratioed on the third part (Pearson, 1897) x 1 = w 1 w 3 ; x 2 = w 2 w 3 Spurious CORR α 3 2 α 1 2 +α 3 2 α 2 2 +α 3 2 (Aitchison, 1986) α 1, α 2, α 3 =coefficients of variations of w 1, w 2, w 3 Composition as basis (e.g. concentrations): large source of spurious correlations (Aitchison, 1986): x 1 = w 1 w 1 + w 2 + w 3 ; x 2 = w 2 w 1 + w 2 + w 3 ; x 3 = w 3 w 1 + w 2 + w 3, where the same part is at numerator and denominator Arctic Lake data (sand=w 1, silt=w 2, clay=w 3 ) (Aitchison, 1986, p. 359) Correlation between x 1 and x 2 = 0.50 Spurious correlation = 0.45, hence 90% of total correlation is meaningless (see excell sheet)

9 Weltje, G.J., Quantitative analysis of dentrial modes: statistically rigourous confidence regions in ternary diagrams and their use in sedimentatry petrology. Earth-Science Reviews, 57, Normally distributed data and their CI are within the real space (± ) Non normality

10 10 Log ratios (log contrast) Data transformation: Compositional space (raw) Real space (log ratio) lr = ln g r i ; if g r g s i +, lr + ; if g s i +, lr. Hence, the lr and CI i can scan the real space CoDa transformation methods: alr, clr, ilr

11 11 Additive log ratio transformation (Aitchison, 1986) alr i = ln u i u d+1 = ln u i ln u d+1 u i = some proportion of the whole d u d+1 = filling value (F v ) =1-1=1 u d Ingestad s (1987) stoichiometric ratios: P/N, K/N,, where N relpaces u d+1 D-1 degrees of freedom alrs are at 60 o angle from each other, hence not directly compatible witheuclidean geometry Silt = 35%; clay = 25%; hence sand = 100% - 35% - 25% = 40% alr silt = ln = 0,134 alr clay = ln = 0,470 Back-transform alr means into familiar units from the alr values and the bounded sum (here 100%) after statistical analysis

12 12 Centred log ratio transformation (Aitchison, 1986) clr i = ln u i g u i u i = some proportion of the whole d u d+1 =1-1=1 u d = filling value (F v ) Biplot analysis Provided proper geometry to the DRIS approach (Parent & Dafir, 1992) Silt = 35%; clay = 25%; hence sand = 100% - 35% - 25% = 40% clr sand = ln clr silt = ln clr clay = ln = = = i=1 clr 1 = 0, hence singularity (one clr excluded from statistical analysis, in general u d+1 )

13 13 Nutrient diagnosis (theories in plant nutrition) based on Law of minimum Liebig s barrel h=834&tbm=isch&imgil=bzoajpmonm_jum%253a%253bogdix tcpt9d9m%253bhttp%25253a%25252f%25252fwww.bestseedbank.com%25252f%25253fattachment_id% d8642&source=i u&pf=m&fir=bzoajpmonm_jum%253a%252cogdixtcpt9d9m%252c_&usg= NB76ewraB4PpBOa7oVzkqo2NAU%3D&ved=0CDcQyjdqFQoTCLOz5NrDxsYCFQ ptpgodqayheg&ei=l3eavfolnyra- QGpzZ6QAQ#imgrc=9RvzYO1EnSl3FM%3A&usg= NB76ewraB 4PpBO-a7oVzkqo2NAU%3D

14 14 Source of distortion in ceteris paribus (Law of minimum) shown by clr If the distance between the two compositions is computed as ln u i ln v i, all other nutrients are assumed to be equal and nutrient interactions to be negligible. Compositional vector u clr i = ln u i g u i = ln u i ln g u i Compositional vector v clr i = ln v i g v i = ln v i ln g v i The ceteris paribus assumption holds if and only if ln g u i ln g v i Clr is a geometrically correct formulation for DRIS (Parent and Dafir, 1992)

15 15 Isometric log ratio transformation (Egozcue et al., 2003) ilr i = rs ln g r i r+s balances g s i, rs r+s = normalization coefficient to compare different Log contrast between two non-overlaping sub-compositions (orthogonality), also called «orthonormal balance» Sequential binary partition (SBP): arrangement of balances D-1 degrees of freedom and Euclidean geometry Silt = 35%; clay = 25%; hence sand = 100% - 35% - 25% = 40% ilr clay silt,sand] = 2 1 ln = ilr silt sand] = ln =

16 Balance domain 16 Sequential binary partition and mobilefulcrums-buckets design to define balances Partition Sand Silt Clay r s Hierarchy Concentration domain

17 17 Chemical equilibrium as log contrast α 1 x 1 +α 2 x 2 α 3 x 3 +α 4 x 4 K e = ln x α1 1 x α2 2 x α3 3 x α4 4 K e or lr Ke = ln x 1 α 1 x 2 α 2 ln x3 α 3 x 4 α 4 = α1 ln x 1 + α 2 ln x 2 α 3 ln x 3 + α 4 ln x 4 Reactives: lr reactives = α 1 ln x 1 α 2 ln x 2 Products: lr products = α 3 ln x 3 α 4 ln x 4 Ilr x 1 x 2 x 3 x x 1 x 2 x 3 x 4 Reactives Products

18 Balance domain 18 Scale-independent orthonormal balance: subcompositional coherence Concentration (proportion) domain

19 19 Understanding behind balances Multivariate distances are independent from the way balances are arranged in the SBP or the mobile No knowledge: unstructured balances Meaningful balances Scientific theories or hypotheses (e.g. anions vs. cations) and hierarchical arrangement: literature review Nutrient management fertilizers to feed crops(n, P, K) or lime (to neutralize exchangeable acidity Ca and Mg carbonates) Nutrient budgets: input and output components Bi-plot (correlations): data driven, no cause-and-effect validation

20 20 Brasileira receita equilibrada do bolo de fubá (Marisa N.)

21 21 Mobile with 6 balances and 7 ingredients centred to recipe 3 4 ln Baking powder 3 Sugar Maize Wheat 2 3 ln Sugar Maize Wheat 1 Maize ln 2 Wheat

22 22 ILR as data transformation technique Univariate analysis ANOVA for each variable Multivariate analysis (all ilrs or concentrations included) Measure of bias from ordinary log-transformed or raw proportions or concentrations Holistic classification of specimens (no ad hoc SBP needed) Diagnosis (ad hoc SBP needed to interpret balances) Others Regression analysis Ilrs vs. ordinary log-transformed or raw concentrations as independent variables: performance influenced by correlations and overfitting Variates (variables combined across compositons such as PC) : ilrs remove bias

23 ilr1 23 Multivariate distance ilr i ilr i φ 1 ilr i ilr i If φ=identity matrix, Euclidean (Aitchison) distance If φ=variance matrix, chi-square variable If φ=covariance matrix, Mahalanobis distance Any composition can be compared holistically to a reference composition (e.g. initial composition or the composition of a performing system) for classification or diagnostic purposes Euclidean distance (hypothenuse) ilr2

24 24 Distortion (departure from 45 o line)in raw nutrient CoDa due to nutrient interactions

25 25 Sound balances between anionic and cationic macronutrients o o o o

26 26 Complex nutrient balances in plants avoiding ceteris paribus

27 27 Receiver operation characteristic (ROC) for specimen classification

28 28 Means and SD of balances at fulcrums and concentrations in buckets for diagnostic purposes

29 29 Subcompositional N, P, K diagnosis where P is varied to reach the reference cloud

30 30 Nutrient signature of mango varieties (left) and of soil properties (right) to test genetic vs. environment effects on foliar compositions

31 31 Classification of nutrient signatures (N, P, K, Ca, Mg) among wild (left) and domesticated fruit species

32 32 Challenges Revisit non-compositional models with balances Soil & Plant Sciences (equations, quality indices, classification, diagnosis, ) Geosciences, Agronomy, Ecology (countings, proprotions, nutrient cycles and budgets, sustainability indices, ) Support paradigm change from interpreting a posteriori computer-driven results to a priori sound understanding of the relational data structure Contribute to special issues using CoDa tools Teach compositional models and train a new generation of researchers

33 33 Key papers Aitchison, J., Shen, S.M Logistic-normal distributions: some properties and uses. Biometrika 67(2): Aitchison, J The statistical analysis of compositional data. Chapman Hall, London. Aitchison, J., Greenacre, M Biplots of compositional data. J. Royal Stat. Soc. Series C (Appl. Stat.) 51(4): Filzmoser, P., Hron, K., Reimann, C Univariate statistical analysis of environmental (compositional) data: Problems and possibilities. Sci. Total Environ. 407(23): Egozcue, J.J., Pawlowsky-Glahn, V., Mateu-Figueras, G., Barceló-Vidal, C Isometric log-ratio transformations for compositional data analysis. Math. Geol. 35: Egozcue, J.J., Pawlowsky-Glahn, V Groups of parts and their balances in compositional data analysis. Mathematical Geology 37: Parent, S.- É., Parent, L. E. Rozane, D. E. Hernandes, A., Natale, W Nutrient balance as paradigm of plant and soil chemometrics. Chapter 4 (32 pp.). In Issaka, R.N. (ed.). Soil Fertility, InTech Publ., Pearson, K., Mathematical contributions to the theory of evolution. On a form of spurious correlation which may arise when indices are used in the measurement of organs. Proceedings of the Royal Society of London LX,

34 34 Some of our open-access compositional papers in agronomy of-plant-and-soil-chemometricsnutrient-balance-as-paradigm-of-soil-and- w_zealand_kiwifruit_%28_actinidia_deliciosa_%29_at_high_yield_level

35 35 Codapack Import data Log ratio transformations Balance dendogram Biplots