Exploratory Data Analysis. CERENA Instituto Superior Técnico

Transcription

1 Exploratory Data Analysis CERENA Instituto Superior Técnico

2 Some questions for which we use Exploratory Data Analysis: 1. What is a typical value? 2. What is the uncertainty around a typical value? 3. What is a representative value? 4. A certain histogram is spatially representative of the reservoir? EDA should give answers to relevant and basic questions related to the characterization of the reservoir

3 Exploratory Data Analysis vs. Classical analysis Classical Analysis Problem => Data => Model => Analysis => Conclusions EDA Analysis Problem => Data => Analysis => Model => Conclusions

4 In classical analysis, data collection is followed by the imposition of a model (normality, linearity, etc.). and all the analysis that follows refers to the estimation and testing of the model parameters. In EDA data collection is not followed by the imposition of a model; the analysis that is done is to choose and inference of the best model for the data we have. The EDA approach takes as its priority the set of available data to suggest the most appropriate models to be fitted to those data.

5 Example : global statistics, local statistics, representativity,.

6 N=99 N=98

7 N=96 N=95

8 CMRP-Instituto Superior Técnico N=92 N=89

9 Var var(n)/var(n=99) (%) Nsamples/Variance Relaltive Unitary Ratio of Variance Series Series1 Samples Nsamples Relative decrease of the variance

10 Estimation of global statistics Statistics have to be robust and representative. Ex: variance (N=98) of 667 ppm 2 is more representative of the study area than variance (N=99) of 1000 ppm 2-30% of this value is due to a single sample and to the area it represents. In these situations, when samples are removed to encounter more robust estimators one should take into account the representative area of these samples. Ex: variance (N=89) de 212 is not representative of the central area of the highest values.

11 Local statistics But the spatial (local) characterization of permeability must necessarily have all the values

12 Exploratory Data Analysis: Univariate Description Raw Data North ing Easting Depth k(md) Mining the data: Visualize, describe, analyze

13 Univariate Description Histograms Class Limit Abs. Freq. Rel. Freq. Cum.Freq Box-plot Representation

14 Univariate Description

15 Symmetrical Histogram with moderated tails Symmetrical Histogram with short tails Symmetrical Histogram with long tails Bi-modal Symmetrical Histogram

16 Histogram resulting from the mixture of two normal distribuitions Histogram with tail to the right Histogram with tail to the left Symmetrical Histogram with outlier

17 Estimation of a Histogram Most ususal Estimator: equal weight to all samples Which can lead to a biased estimator. This sample has the same weight of all the others, then the histogram, mean, variance,, are skewed Solution: desagregated histograma

18 Univariate Description Mean m (arithmetic mean ) Measures of Center m 1 n n z i i1 Median M - is the z value corresponding to a cumulative percentage of 50% of the total values M m

19 Univariate Description Measures of Location quartiles: Q 1 - z value corresponding to a cumulative percentage of 25% Q 3 - z value corresponding to a cumulative percentage of 75% minimum: min- z value corresponding to a cumulative percentage of 0% maximum: max - z value corresponding to a cumulative percentage of 100% quantiles q(p) - z value corresponding to a cumulative percentage of 100.p %

20 Univariate Description max Q 2 M Q 1 min

21 Univariate Description variance 2 Measures of Spread 1 N N 11 ( z i m) 2 Interquartil range IQR=Q 3 -Q Q1 Q2 f(z) Zi Q1 Q2

22 Univariate Description Sensitivity to extreme values high sensitivity No. of Samples Variance Coef. Variation Mean low sensitivity No. of Samples Median Q1 Q

23 1- Univariate Description Extreme values and local uncertainty High local mean and low local variance High local mean and high local variance

24 Sample weights for the calculus of Disaggregated Spatial Statistics Weights proportional to the area of influence of samples Influence Polygons Regular Polygons Z(x 2 ) p(x 2 ) Z(x 1 ) p(x 1 ) Directly proportional to the area of each polygon Inversely proportional to the number of samples contained in each polygon

25 Univariate Description Distribution Models y m ln 2 Pros: Simplicity of representation (2 parameters ) and analysis. Cont: Representation to simplistic of important details of the histogram.

26 Bivariate Description x y z (%) K(md) Relation between Porosity and Permeability

27 Bivariate Description Bi-plots (14.4,13.2)

28

29

30 Bivariate Description Bi-Histograms % v 2 % Conditional Histograms v 1

31 Bivariate Description Quantile-Quantile Plots q-q plot: two marginal distributions can be compared by plotting theirs quantiles against one another : Cu Sn q(0) 0 10 q(0.1) 0 75 q(0.2) q(1.) Cu Sn If the q-q plot appears as a straight line, the 2 marginal distributions have the same shape.

32 Bivariate Description Regression Methods to summarize and visualize the behavior between two variables. Linear regression y=ax+b summarizes the behaviour between the two variables

33 Bivariate Description The regression model should have a qualitative relationship with the physical phenomenon under study. A polynomial regression can reproduce the particularities of sample data rather than the details of the relationship between the two variables and the physical phenomenon.

34 y x N i i i N y i N i x i N m m y x m y m x.. cov Bivariate Description Measures of correlation between the 2 variables Covariance between variables xi and yi Mean of x i N i i N m x x 1 1 Mean of y i N i i N m y y 1 1

35 Bivariate Description Measures of correlation between the 2 variables Correlation Coefficient (Pearson) cov x y Standard Deviation of the two variables xi e yi N 1 N i1 2 N 2 x m x i x y 1 N i1 y i m y

36 Bivariate Description Correlation Coefficient V2 Positive Correlation Coefficient V2 Negative Correlation Coefficient V1 V1 V2 Null Correlation Coefficient V1

37 Bivariate Description Correlation Coefficient The Correlation Coefficient is extremely sensitive to points which are located far from the main cloud.

38 Bivariate Description Correlation Coefficient Add just one pair of values: (200,200)

39 Bivariate Description Correlation Coefficient The correlation coefficient measures the linear dependence between two variables

40 Univariate and Bi-variate statistics Should reflect the most relevant geological patterns Should explain the main relationships between data Should always focus on the principle of maximum parsimony The use of statistic to summarize the behavior of key variables must be balanced against the drawbacks, too condensed information, sensitivity to extreme values, limited description in the case of bivariate,

41 Spatial Description Definition of Lithotypes

42 Spatial Description Data spatial representation

43 Spatial Description

44 Spatial Description Moving window statistics m 2 m 2 m 2

45 Spatial Description Variances Moving window statistics Mean Window Window

46 Introduction to GeoMS- Geostatistical Modelling Software

47 Geostatistic Software for Windows 2000, NT CMRP/IST Exploratory data analysis Spatial continuity analysis Modelling of variograms Kriging (SK, OK, KED,..) Co-Kriging Stochastic Simulation (DSS, SGS, SIS) Multi-phase classification Simulated Annealing Visualization Data transformation

48 Probabilities Distribuitions

49 Random variable (RV): Z Distribution function (cdf): F(z) = F Z (z)= Prob{Zz} Density Function (pdf): f(z)=f Z (z)=f (z)= lim dz0 F z dz Fz dz

50 Discrete probability function A discrete probability function, p(x), is a function that satisfies the following properties: 1. The probability of x taking a specific value is p(x) 2. p(x) is non-negative for all real x. 3. The sum of all possible p(x) values of x is 1, i.e. j1 p( Where j represents all the possible values of x and p j is the probability of x j. Consequently, 0 <= p(x) <= 1. N x j ) 1

51 Continuous probability functions A continuous probability functions, f(x), is a function that satisfies the following properties: The probability of x being between two points a and b is p a x b f xdx b Is non-negative for all real x. The integral of the probability function is one: a f xdx 1

52 Density function, pdf : f(z) f z 0 f z dz 1 z f z z f z F' z Z f zdx Fz ProbZ z lim dz0 F z dz Fz dz

53 Distribution function, cdf : 1. F(b) 0.5 F(a) z F z ProbZ z F Z F z is non decreasing Fz0,1 F 0 F 1 a 0. b Prob Z a, b Prob Z b Prob Z a Fb Fa

54 Normal Distribuition g x 1 2 xm 1 2 e 2 Prob{X<x}= F x x m G

55 Standard Normal Distribution X>0 N (m, ) Y= (X-m)/ N (0, 1) X=m+ Y g x x 2 e 2 Prob{X<x} = Prob{m+ Xx} = prob {Y(x-m)/ } Prob{Y y} G y x m G

56 Uniform Distribution f x B 1 Ax=B A Standard Uniform Distribution f x 1

57 Exponential Distribution f x 1 e x x e >0 Standard Exponential Distribution f x x e x0

58 a variable X as a lognormal distribution if Y = ln(x) is normally distributed X>0 lnn (m, ) Y=ln(X) N (,) f x 1 x ln g x

59 Central Limite Theorem Theorem: The sum of a large number of independent standardized random variables - evenly distributed - tend to be normally distributed. n RVs Zi, equally distributed (not necessarily normal) with zero mean: Y=Z i Normal, when n

60 Central Limite Theorem Corollary: The product of a large number of independent standardized random variables - equally distributed - tend to have a log-normal distribution. n RVs Zi, equally distributed (not necessarily normal) with zero mean : X=Z i Y=lnX= lnzi Normal, when n

61 Bi-variate Distributions Bi-Histograms % v 2 % Conditional Histograms v 1

62 Conditional Probability E{X Y=y} E{Y X=x} E{X Y=y} mean value of X in class Y=y. E{Y X=x} mean value of Y in class de X=x.

63 Conditional Probability f (x y)=prob{x=x;y=y}/prob{y=y}=f(x,y)/f(y) Prob{A B}=Prob{A e B}/Prob{B} A B Prob of A conditioned to the occurrence of event B

64 Bayes Relation Prob{A B}=Prob{A B}.Prob{B} Prob{A B}=Prob{B A}.Prob{A} Prob{A B} =(Prob{B A}/Prob{B}).Prob{A} (Prob{B A}/Prob{B}) is the likelihood of B given A

65 Bayes thinking - 1 (Savage S. 2005) There is 98% effective diagnostic test for SRHD: 98% of people who are infected display positive results; 98% of people who are not infected display negative results. Someone who in routine has just tested positive for SRHD. What is the chance he is actually infected? 1% of population that is SRHD+ Gut response is 98% of chance to be infected 99% of population that is SRHD- Bayesian analysis says it is only one out of three

66 Bayes Thinking - 1 A positive test can occur in two ways: true positive (98% of 1%) or false positive (2% of 99%) False negative: 2% of 1% = 0.02% True positive: 98% of 1% =.98% True negative: 98% of 99% = 97.02% False positive: 2% of 99% = 1.98% Real question: what is the chance of hitting a green area, given that you know you have hit a hatched area. Clearly about one third