Meelis Kull Autumn Meelis Kull - Autumn MTAT Data Mining - Lecture 03

Transcription

1 Meelis Kull Autumn

2 Demo: Data science mini-project CRISP-DM: cross-industrial standard process for data mining Data understanding: Types of data Data understanding: First look at attributes Types of attributes First look at a nominal attribute First look at a ordinal attribute First look at a numeric attribute 2

3 Demo: Data science mini-project CRISP-DM: cross-industrial standard process for data mining Data understanding: Types of data Data understanding: First look at attributes Types of attributes First look at a nominal attribute First look at a ordinal attribute First look at a numeric attribute 3

4 4

5 All the same applies as for nominal attributes Need to make sure that the order is retained in histograms Additional ways to look at the attribute: Calculate the min, max, median of the attribute 5

6 All the same applies as for nominal attributes Need to make sure that the order is retained in histograms Additional ways to look at the attribute: Calculate the min, max, median of the attribute More generally, calculate quantiles 6

7 There are 3 quartiles: lower quartile, median, upper quartile They divide a sorted data set into 4 equal parts Percentiles divide into 100 equal parts Lower quartile is the 25 th percentile Median is the 50 th percentile Deciles divide into 10 equal parts Quantiles generalise to any location over sorted data: Lower quartile = 25 th percentile = quantile at th decile = quantile at 0.4 7

8 A. 1 st and 2 nd decile B. 2 nd and 3 rd decile C. 3 rd and 4 th decile D. 4 th and 5 th decile E. 5 th and 6 th decile F. 6 th and 7 th decile G. 7 th and 8 th decile H. 8 th and 9 th decile 8

9 A. Nominal attributes B. Ordinal attributes C. Both D. Neither E. Not sure 9

10 Same aspects as in nominal attributes Additionally: Is the order specified in the documentation? (meta-data) If all possible values specified in the meta-data: Check if all values present in the data If not, make sure that the empty bars in histograms are in the right location corresponding to the order 10

11 Demo: Data science mini-project CRISP-DM: cross-industrial standard process for data mining Data understanding: Types of data Data understanding: First look at attributes Types of attributes First look at a nominal attribute First look at a ordinal attribute First look at a numeric attribute 11

12 12

13 Is it really numeric? E.g. if contains only values 0.0 and 1.0 then perhaps nominal is more appropriate? Is it discrete (only integers)? E.g. if contains integers 0 to 100 then can have a first look as if it were an ordinal attribute Calculate quantiles as for ordinal attributes Calculate the mean (arithmetic average): 13

14 Is it really numeric? Is it discrete (only integers)? Calculate quantiles as for ordinal attributes Calculate the mean (arithmetic average) Plot a histogram (with default binning) 14

15 Check if some value occurs many times With real numbers often no number occurs more than once If a value is frequent then: Possibly can denote missingness (e.g. value 0), should be treated as N/A (not available, missing) Perhaps can denote an extremal value (more extremal values represented as this value) Check for rounding E.g., if all numbers rounded to 2 nd digit and one has more digits then it might be a typing error 15

16 A. Nominal attributes B. Ordinal attributes C. Numeric attributes D. All of the above E. None of the above F. Not sure 16

17 Demo: Data science mini-project CRISP-DM: cross-industrial standard process for data mining Data understanding: Types of data Data understanding: First look at attributes Types of attributes First look at a nominal attribute First look at a ordinal attribute First look at a numeric attribute 17

18 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 18

19 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 19

20 20

21 We have now looked at each attribute A key question we can now answer: How do the items in this dataset look like? Now we have at least 3 answers to this: 21

22 How do the items in this dataset look like? Let us just pick up one as an example: Age = 22 Workclass = Private Education = 11 th Occupation = Other-service Capital.gain =

23 How do the items in this dataset look like? Let us just provide the ranges of attributes Age: {17,18,19,,90} Workclass: {Federal-gov, Local-gov, Private, } Education: {1 st -4 th,5 th -6 th,, Doctorate) Occupation: {Adm-clerical, Exec-managerial, } Capital.gain: [0,99999]... 23

24 How do the items in this dataset look like? Let us just provide the histograms Age: <histogram> Workclass: <histogram> Education: <histogram> Occupation: <histogram> Capital.gain: <histogram>... 24

25 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 25

26 26

27 Histograms on discrete data Nominal Ordinal Numeric with few different values E.g. small number of different integers Histograms on continuous data Numeric with many different values 27

28 Frequency histogram Frequency = count of items with each value ggplot(data) + geom_histogram(aes(x=age),stat="count") 28

29 Frequency histogram Frequency = count of items with each value ggplot(data) + geom_histogram(aes(x=workclass),stat="count") 29

30 Relative frequency histogram Relative frequency = proportion (0..1) or percentage (0..100%) of items with each value Heights of bars sum up to 1 ggplot(data) + geom_bar(aes(x=workclass,..count../sum(..count..)),stat="count") 30

31 Depends on the goal Frequency histogram Gives actual counts in the data Relative frequency histogram Gives proportions in the data Interpretable as probability distribution of a randomly chosen item 31

32 Continuous attribute: Usually value is different in each item Need to introduce bins (a.k.a. intervals, ranges) Histogram not informative without bins: ggplot(data) + geom_histogram(aes(x=factor(salaries)),stat="count") 32

33 Frequency histogram of binned data Frequency = count of items in each bin ggplot(data) + geom_histogram(aes(x=salaries),stat="bin",boundary=0,binwidth=10000) 33

34 Relative frequency histogram of binned data Relative frequency = proportion of items in bins Heights of bars add up to 1 ggplot(data) + geom_histogram(aes(x=salaries,..count../sum(..count..)),stat="bin",boundary=0,binwidth=10000) 34

35 Density histogram of binned data Density = Y-axis such that areas of bars in the histogram add up to 1 Density scale is invariant to the sizes of bins ggplot(data) + geom_histogram(aes(x=salaries,..density..),stat="bin",boundary=0,binwidth=10000) 35

36 Density histogram of binned data Density = Y-axis such that areas of bars in the histogram add up to 1 Density scale is invariant to the sizes of bins + geom_histogram(aes(x=salaries,..density..),stat="bin",boundary=0,binwidth=1000,alpha=0,colour="red") 36

37 Density histogram of binned data Density = Y-axis such that areas of bars in the histogram add up to 1 Density scale is invariant to the sizes of bins + geom_histogram(aes(x=salaries),stat="density",colour="blue") 37

38 Represented by the probability density function (pdf) Area under the curve is equal to 1 Areas represent probabilities Area = P(a<X<b) = probability that X is between a and b 38

39 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 39

40 40

41 Statistic measure calculated from all values Mode of the distribution: The most probable value (or values) The most frequent value if discrete Value with highest density if continuous Median and other quantiles We have defined earlier Mean of the distribution: Average value of the attribute Arithmetic average if discrete Expected value if continuous Centre of mass of the distribution 41

42 Consider attribute with values: 1,2,2,2,3,3,4,7 Mode: 2 because occurs three times Median 2.5 because 2 & 3 are in the middle, (2+3)/2=2.5 Mean: 3.0 because ( )/8 = 24/8 =

43 A. 0 B. 0.5 C. 1 D. 1.5 E. None of the above 43

44 A. 0 B. 0.5 C. 1 D. 1.5 E. None of the above 44

45 A. 0 B. 0.5 C. 1 D. 1.5 E. None of the above 45

46 Variance of a distribution is: average squared deviation from the mean Standard deviation is square root of variance Quadratic average deviation from the mean Example: Values: 1,2,2,2,3,3,4,7 Mean: 3.0 Variance: ((1-3)^2+(2-3)^2+ +(7-3)^2) / 8 = 3.0 Standard deviation: sqrt(3.0)=

47 Symmetric Skewed / right-skewed / left-skewed Heavy-tailed Bimodal Multi-modal Capped / right-capped / left-capped 47

48 Simple definitions, not fully correct: Unimodal 1 mode Bimodal 2 modes Multimodal multiple modes Actually: Bimodal usually means that the density (pdf) has two local maxima: Multimodal means pdf has multiple maxima (2 or more) 48

49 Symmetric distribution: Symmetric around a vertical axis of symmetry Left and right side are mirror-images of each other Left- (or right-)skewed distribution: Non-symmetric and mean below (above) mode Usually used only for unimodal distributions sym negati positiv m vely ely et ske ske ri wed we c d 49

50 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 50

51 51

52 Statisticians have names for many different families of probability distributions Why need to know some of them? In practice these are used to communicate the distribution without having to visualise it We will talk about: Uniform distributions Normal distributions Power law distributions 52

53 All options are equally probable 53

54 All values in [a,b] are equally probable: The pdf is constant between a and b, and 0 elsewhere 54

55 Represent data dispersion, spread Represent central tendency 55

56 N(mean,variance) The most common non-uniform continuous distribution Why? Sum of many independent and identically distributed (i.i.d.) random variables is approximately normally distributed Standard normal distribution: N(0,1) 56

57 Heavy-tailed (or long-tailed) distribution Very high or low values are likely More likely than in case of normal distribution Technical definition: Tails are not exponentially bounded 57

58 Alternative names: scale-free / scale-independent Distributions on positive real values The probability of M times bigger value is K times smaller (power law; M, K - parameters) Linearly descending pdf when drawn in loglog-scale (both x and y logarithmic) Examples: Number of connections in many real-world graphs Frequencies of words in languages Sizes of craters on the moon 58

59 A. Uniform B. Unimodal symmetric C. Unimodal right-skewed D. Unimodal left-skewed E. Bimodal symmetric F. Bimodal asymmetric G. Multi-modal symmetric H. Multi-modal asymmetric I. Normally distributed J. Power-law distributed 59

60 A. Uniform B. Unimodal symmetric C. Unimodal right-skewed D. Unimodal left-skewed E. Bimodal symmetric F. Bimodal asymmetric G. Multi-modal symmetric H. Multi-modal asymmetric I. Normally distributed J. Power-law distributed 60

61 A. Uniform B. Unimodal symmetric C. Unimodal right-skewed D. Unimodal left-skewed E. Bimodal symmetric F. Bimodal asymmetric G. Multi-modal symmetric H. Multi-modal asymmetric I. Normally distributed J. Power-law distributed 61

62 A. Uniform B. Unimodal symmetric C. Unimodal right-skewed D. Unimodal left-skewed E. Bimodal symmetric F. Bimodal asymmetric G. Multi-modal symmetric H. Multi-modal asymmetric I. Normally distributed J. Power-law distributed 62

63 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 63

64 64

65 Compactly visualise many distributions Density plots rotated by 90º and mirrored Distribution of salaries for each education level ggplot(data) + geom_violin(aes(x=education,y=salaries)) 65

66 Compact visualise many distributions marked median, upper and lower quartile, and outliers (R ggplot outlier = more than 1.5x interquartile range from quartile) ggplot(data) + geom_boxplot(aes(x=education,y=salaries)) 66

67 Violin plots and box plots can also be shown together: ggplot(data) + geom_violin(aes(x=education_factor,y=salaries)) + geom_boxplot(aes(x=education_factor,y=salaries),alpha=0.3) 67

68 Often too crowded, but sometimes provide extra insights compared to box plots and violin plots Over-crowded with too many points ggplot(data) + geom_point(aes(x=education_factor,y=salaries)) 68

69 Often too crowded, but sometimes provide extra insights compared to box plots and violin plots More useful, here only 100 points ggplot(data) + geom_point(aes(x=education_factor,y=salaries)) 69

70 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 70

71 71

72 2 categorical attributes: Cross-table (workclass & education) table(data$education,data$workclass) 72

73 2 categorical attributes: Cross-table (workclass & education) Heatmap (workclass & education) ggplot(data %>% group_by(education,workclass) %>% summarise(count=length(education))) + geom_tile(aes(y=education,x=workclass,fill=count)) + scale_fill_gradient(low='white',high='black',trans="log",breaks=c(1,10,100,1000)) + theme_bw() 73

74 2 categorical attributes: Cross-table (workclass & education) Heatmap (workclass & education) Cross-table and heatmap combined + geom_text(aes(x=workclass,y=education,label=count),color="red") 74

75 Scatter plot, box plot, violin plot 75

76 Scatter plot ggplot(data) + geom_point(aes(x=age,y=salaries)) 76

77 Scatter plot ggplot(data) + geom_point(aes(x=capital.gain,y=salaries)) 77

78 Scatter plot Discretise one (or both) of the attributes Discretise = make into categorical For instance, introduce bins library(arules) data$capital.gain.discretised = discretize(data$capital.gain,method="fixed",categories=c(0,5,10,20,50,100)*1000) ggplot(data) + geom_boxplot(aes(x=capital.gain.discretised,y=salaries)) 78

79 Make all pairwise visualisations and organise in a cross-table Perform dimensionality reduction Introduced later in the course Projects the data into a 2-dimensional space Then visualise Use colours, shapes, etc. 79

80 Four attributes visualised together ggplot(data) + geom_point(aes(x=capital.gain,y=salaries,color=workclass,shape=gender)) + scale_colour_brewer(type="qual") 80

81 Four attributes visualised together Often hard to read when many attributes visualised this way ggplot(data) + geom_point(aes(x=capital.gain,y=salaries,color=workclass,shape=gender)) + scale_colour_brewer(type="qual") 81

82 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 82

83 83

84 There are many statistical tests about this, we will cover some in the next lecture Visual inspection can reveal some relations For example, university graduates have higher median salaries than others 84

85 There are many statistical tests about this, we will cover some in the next lecture Visual inspection can reveal some relations For example, university graduates have higher median salaries than others Correlation is a statistic on 2 numeric attributes, quantifying linear relationships 85

86 Mean (actually sample mean as explained in next lecture) x 1 n n i 1 x i Correlation (Pearson correlation coefficient) 86

87 Ranges from -1 to +1: R=-1: perfectly anti-correlated R=+1: perfectly correlated R=0: absolutely uncorrelated Positively correlated Negatively correlated 87

88 Uncorrelated Uncorrelated Uncorrelated 88

89 89

90 Several sets of (x, y) points, with the Pearson correlation coefficient of x and y for each set. Note that the correlation reflects the noisiness and direction of a linear relationship (top row), but not the slope of that relationship (middle), nor many aspects of nonlinear relationships (bottom). N.B.: the figure in the center has a slope of 0 but in that case the correlation coefficient is undefined because the variance of Y is zero. 90

91 91

92 A. All equal B. All different C. Some equal, some different 92

93 93

94 94

95 95

96 96

97 97

98 405 - Feb CO2 Noise Pressure 98

99 Feb 2017 Changed the scale of noise CO2 Pressure Noise 99

100 80 70 CO2 vs Noise in 405 y = x R² = Noise CO

101 80 70 Not the same as correlation! CO2 vs Noise in 405 y = x R² = Noise CO

102 r Pearson correlation coefficient R 2 coefficient of determination Proportion of variance in the target attribute that is predictable from the source attribute Basically, measures how well the data follow the regression line In simple linear regression R 2 is equal to the square of r 102

103 Source: 103

104 Source: 104

105 ????????? Source: 105

106 Source: 106

107 ????????? 107

108 108

109 Relation Correlation Causation 109

110 Relation Correlation Causation Spurious correlations! Multiple testing problem (next lecture) 110

111 Data understanding: distribution of attributes Types of histograms How to describe probability distributions? Some standard probability distributions More ways to visualise distributions Visualising relations of attributes Are the attributes related? 111

112 112

113 The most merciful thing in the world... is the inability of the human mind to correlate all its contents. H. P. Lovecraft The correlation of quality of life and cost of energy is huge Sam Altman Visualization is daydreaming with a purpose. Bo Bennett Source: 113