Data mining/machine learning large data sets. STA 302 or 442 (Applied Statistics) :, 1

Size: px

Start display at page:

Download "Data mining/machine learning large data sets. STA 302 or 442 (Applied Statistics) :, 1"

Hector Woods
5 years ago
Views:

1 Data mining/machine learning large data sets STA 302 r 442 (Applied Statistics) :, 1

2 Data mining/machine learning large data sets high dimensinal spaces STA 302 r 442 (Applied Statistics) :, 2

3 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure STA 302 r 442 (Applied Statistics) :, 3

4 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive STA 302 r 442 (Applied Statistics) :, 4

5 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive plts are essential, but cmplicated STA 302 r 442 (Applied Statistics) :, 5

6 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive plts are essential, but cmplicated emphasis n means and variances STA 302 r 442 (Applied Statistics) :, 6

7 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive plts are essential, but cmplicated emphasis n means and variances STA 302 r 442 (Applied Statistics) small data sets :, 7

8 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive plts are essential, but cmplicated emphasis n means and variances STA 302 r 442 (Applied Statistics) small data sets lw dimensin :, 8

9 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive plts are essential, but cmplicated emphasis n means and variances STA 302 r 442 (Applied Statistics) small data sets lw dimensin lts f infrmatin n the structure :, 9

10 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive plts are essential, but cmplicated emphasis n means and variances STA 302 r 442 (Applied Statistics) small data sets lw dimensin lts f infrmatin n the structure plts are useful, and easy t use :, 10

11 Data mining/machine learning large data sets high dimensinal spaces ptentially little infrmatin n structure cmputatinally intensive plts are essential, but cmplicated emphasis n means and variances STA 302 r 442 (Applied Statistics) small data sets lw dimensin lts f infrmatin n the structure plts are useful, and easy t use emphasis n likelihd and inference using prbability distributins :, 11

12 Examples (Read Chapter 1, ) spam 4601 messages, frequencies f 57 cmmnly ccurring wrds Next week: r 4 (3.2.0 shuld be familiar) :, 12

13 Examples (Read Chapter 1, ) spam 4601 messages, frequencies f 57 cmmnly ccurring wrds prstrate cancer 97 patients, 9 cvariates Next week: r 4 (3.2.0 shuld be familiar) :, 13

14 Examples (Read Chapter 1, ) spam 4601 messages, frequencies f 57 cmmnly ccurring wrds prstrate cancer 97 patients, 9 cvariates DNA micrarray data 64 samples, 6830 genes Next week: r 4 (3.2.0 shuld be familiar) :, 14

15 Examples (Read Chapter 1, ) spam 4601 messages, frequencies f 57 cmmnly ccurring wrds prstrate cancer 97 patients, 9 cvariates DNA micrarray data 64 samples, 6830 genes wine data 178 wines, 3 cultivars, 13 cvariates Next week: r 4 (3.2.0 shuld be familiar) :, 15

16 Elements f Statistical Learning c Hastie, Tibshirani & Friedman 2001 Chapter 1 lpsa lcavl lweight age lbph svi lcp gleasn :, 16

17 Sme f the wine data: 1,14.23,1.71,2.43,15.6,127,2.8,3.06,.28,2.29,5.64,1.04,3.92,1065 1,13.2,1.78,2.14,11.2,100,2.65,2.76,.26,1.28,4.38,1.05,3.4,1050 1,13.16,2.36,2.67,18.6,101,2.8,3.24,.3,2.81,5.68,1.03,3.17,1185 1,14.37,1.95,2.5,16.8,113,3.85,3.49,.24,2.18,7.8,.86,3.45,1480 1,13.24,2.59,2.87,21,118,2.8,2.69,.39,1.82,4.32,1.04,2.93,735 1,14.2,1.76,2.45,15.2,112,3.27,3.39,.34,1.97,6.75,1.05,2.85,1450 1,14.39,1.87,2.45,14.6,96,2.5,2.52,.3,1.98,5.25,1.02,3.58,1290 1,14.06,2.15,2.61,17.6,121,2.6,2.51,.31,1.25,5.05,1.06,3.58,1295 1,14.83,1.64,2.17,14,97,2.8,2.98,.29,1.98,5.2,1.08,2.85,1045 1,13.86,1.35,2.27,16,98,2.98,3.15,.22,1.85,7.22,1.01,3.55,1045 1,14.1,2.16,2.3,18,105,2.95,3.32,.22,2.38,5.75,1.25,3.17,1510 1,14.12,1.48,2.32,16.8,95,2.2,2.43,.26,1.57,5,1.17,2.82,1280 1,13.75,1.73,2.41,16,89,2.6,2.76,.29,1.81,5.6,1.15,2.9,1320 1,14.75,1.73,2.39,11.4,91,3.1,3.69,.43,2.81,5.4,1.25,2.73,1150 1,14.38,1.87,2.38,12,102,3.3,3.64,.29,2.96,7.5,1.2,3,1547 1,13.63,1.81,2.7,17.2,112,2.85,2.91,.3,1.46,7.3,1.28,2.88,1310 1,14.3,1.92,2.72,20,120,2.8,3.14,.33,1.97,6.2,1.07,2.65,1280 1,13.83,1.57,2.62,20,115,2.95,3.4,.4,1.72,6.6,1.13,2.57,1130 1,14.19,1.59,2.48,16.5,108,3.3,3.93,.32,1.86,8.7,1.23,2.82,1680 1,13.64,3.1,2.56,15.2,116,2.7,3.03,.17,1.66,5.1,.96,3.36,845 1,14.06,1.63,2.28,16,126,3,3.17,.24,2.1,5.65,1.09,3.71,780 1,12.93,3.8,2.65,18.6,102,2.41,2.41,.25,1.98,4.5,1.03,3.52,770 1,13.71,1.86,2.36,16.6,101,2.61,2.88,.27,1.69,3.8,1.11,4,1035 1,12.85,1.6,2.52,17.8,95,2.48,2.37,.26,1.46,3.93,1.09,3.63,1015 1,13.5,1.81,2.61,20,96,2.53,2.61,.28,1.66,3.52,1.12,3.82,845 1,13.05,2.05,3.22,25,124,2.63,2.68,.47,1.92,3.58,1.13,3.2,830 :, 17

18 ash :, 18

19 alchl malic acid :, 19

20 Sme basic definitins input X (features), utput Y (respnse) :, 20

21 Sme basic definitins input X (features), utput Y (respnse) data (x i, y i ), i = 1,... N :, 21

22 Sme basic definitins input X (features), utput Y (respnse) data (x i, y i ), i = 1,... N use data t assign a rule (functin) taking X Y :, 22

23 Sme basic definitins input X (features), utput Y (respnse) data (x i, y i ), i = 1,... N use data t assign a rule (functin) taking X Y gal is t predict a new value f Y, given X: Ŷ (X) :, 23

24 Sme basic definitins input X (features), utput Y (respnse) data (x i, y i ), i = 1,... N use data t assign a rule (functin) taking X Y gal is t predict a new value f Y, given X: Ŷ (X) regressin if Y is cntinuus classificatin if Y is discrete :, 24

25 Hw t knw if the rule wrks? cmpare ŷ i t y i n data training errr :, 25

26 Hw t knw if the rule wrks? cmpare ŷ i t y i n data training errr cmpare Ŷ t Y n new data test errr :, 26

27 Hw t knw if the rule wrks? cmpare ŷ i t y i n data training errr cmpare Ŷ t Y n new data test errr In rdinary linear regressin, the least squares estimates minimize Σ(y i β 0 β 1 x i ) 2 and the minimized value is the sum f the squared residuals Σ(y i ŷ i ) 2 = Σ(y i ˆβ 0 ˆβ 1 x i ) 2 :, 27

28 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) :, 28

29 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) daily mrtality cunts frm NCHS (Natinal Center fr Health Statistics) :, 29

30 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) daily mrtality cunts frm NCHS (Natinal Center fr Health Statistics) hurly temperature and dewpint data frm Natinal Climatic data Center :, 30

31 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) daily mrtality cunts frm NCHS (Natinal Center fr Health Statistics) hurly temperature and dewpint data frm Natinal Climatic data Center data n pllutants PM 10, O 3, CO, SO 2, NO 2 frm EPA :, 31

32 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) daily mrtality cunts frm NCHS (Natinal Center fr Health Statistics) hurly temperature and dewpint data frm Natinal Climatic data Center data n pllutants PM 10, O 3, CO, SO 2, NO 2 frm EPA utput: Y t mrtality rate n day t :, 32

33 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) daily mrtality cunts frm NCHS (Natinal Center fr Health Statistics) hurly temperature and dewpint data frm Natinal Climatic data Center data n pllutants PM 10, O 3, CO, SO 2, NO 2 frm EPA utput: Y t mrtality rate n day t inputs: X t pllutin n day t 1, plus varius cnfunders: age and size f ppulatin, weather, day f the week, time :, 33

34 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) daily mrtality cunts frm NCHS (Natinal Center fr Health Statistics) hurly temperature and dewpint data frm Natinal Climatic data Center data n pllutants PM 10, O 3, CO, SO 2, NO 2 frm EPA utput: Y t mrtality rate n day t inputs: X t pllutin n day t 1, plus varius cnfunders: age and size f ppulatin, weather, day f the week, time a mdel was fit fr each city, and aggregated ver cities :, 34

35 A cmplex regressin mdel: NMMAPS (Natinal Mrbidity, Mrtality and Air Pllutin Study) 90 largest cities in US by ppulatin (US Census) daily mrtality cunts frm NCHS (Natinal Center fr Health Statistics) hurly temperature and dewpint data frm Natinal Climatic data Center data n pllutants PM 10, O 3, CO, SO 2, NO 2 frm EPA utput: Y t mrtality rate n day t inputs: X t pllutin n day t 1, plus varius cnfunders: age and size f ppulatin, weather, day f the week, time a mdel was fit fr each city, and aggregated ver cities Cnclusin 0.41% increase in mrtality fr a 10 µg increase in PM 10 :, 35

36 :, 36

37 :, 37

38 ... the mdel lg µ at = βx t 1 + γdow + S 1 (t, 7) + S 2 (temp 0, 6) + S 3 (temp 1 3, 6)+S 4 (dew 0, 3)+S 5 (dew 1 3, 3)+α a +S 6a (t, 8) :, 38

39 ... the mdel lg µ at = βx t 1 + γdow + S 1 (t, 7) + S 2 (temp 0, 6) + S 3 (temp 1 3, 6)+S 4 (dew 0, 3)+S 5 (dew 1 3, 3)+α a +S 6a (t, 8) a indexes age grups, t time (days) :, 39

40 ... the mdel lg µ at = βx t 1 + γdow + S 1 (t, 7) + S 2 (temp 0, 6) + S 3 (temp 1 3, 6)+S 4 (dew 0, 3)+S 5 (dew 1 3, 3)+α a +S 6a (t, 8) a indexes age grups, t time (days) S(z, 8) is a nn-specified, but smth, functin f z with 8 degrees f freedm, can think f it as a spline with a pre-specified number f knts. Large df means wiggly functin, 1 df is linear :, 40

41 ... the mdel lg µ at = βx t 1 + γdow + S 1 (t, 7) + S 2 (temp 0, 6) + S 3 (temp 1 3, 6)+S 4 (dew 0, 3)+S 5 (dew 1 3, 3)+α a +S 6a (t, 8) a indexes age grups, t time (days) S(z, 8) is a nn-specified, but smth, functin f z with 8 degrees f freedm, can think f it as a spline with a pre-specified number f knts. Large df means wiggly functin, 1 df is linear mrtality rates change with seasn, weather, changes in health status,... :, 41

42 ... the mdel lg µ at = βx t 1 + γdow + S 1 (t, 7) + S 2 (temp 0, 6) + S 3 (temp 1 3, 6)+S 4 (dew 0, 3)+S 5 (dew 1 3, 3)+α a +S 6a (t, 8) a indexes age grups, t time (days) S(z, 8) is a nn-specified, but smth, functin f z with 8 degrees f freedm, can think f it as a spline with a pre-specified number f knts. Large df means wiggly functin, 1 df is linear mrtality rates change with seasn, weather, changes in health status,... Is there anything left fr pllutin? :, 42

43 the new analysis is highly likely t delay the final review f new regulatins n small-particle pllutin :, 43

44 the new analysis is highly likely t delay the final review f new regulatins n small-particle pllutin industry fficials said the new findings called int questin the validity f sme research underlying the new federal standards :, 44

45 the new analysis is highly likely t delay the final review f new regulatins n small-particle pllutin industry fficials said the new findings called int questin the validity f sme research underlying the new federal standards It certainly brings int questin the precisin f the data, said Dr. Jane Q. Kenig :, 45

46 the new analysis is highly likely t delay the final review f new regulatins n small-particle pllutin industry fficials said the new findings called int questin the validity f sme research underlying the new federal standards It certainly brings int questin the precisin f the data, said Dr. Jane Q. Kenig The health risk psed by particulates is a surce f fierce envirnmental cntrversy in the United States :, 46

47 the new analysis is highly likely t delay the final review f new regulatins n small-particle pllutin industry fficials said the new findings called int questin the validity f sme research underlying the new federal standards It certainly brings int questin the precisin f the data, said Dr. Jane Q. Kenig The health risk psed by particulates is a surce f fierce envirnmental cntrversy in the United States Oppnents f tighter rules are likely t seize n the revisins as evidence that the research linking st in the air t harmful effects n health is nt t be trusted :, 47

48 the new analysis is highly likely t delay the final review f new regulatins n small-particle pllutin industry fficials said the new findings called int questin the validity f sme research underlying the new federal standards It certainly brings int questin the precisin f the data, said Dr. Jane Q. Kenig The health risk psed by particulates is a surce f fierce envirnmental cntrversy in the United States Oppnents f tighter rules are likely t seize n the revisins as evidence that the research linking st in the air t harmful effects n health is nt t be trusted A default setting that prduced errneus results went unchecked fr years, despite significant statistical expertise in all the grups :, 48

49 The findings d nt challenge what is nw a well established link between air pllutin and premature death :, 49

50 The findings d nt challenge what is nw a well established link between air pllutin and premature death The wrk has been published fr several years in a variety f the leading jurnals like the New England Jurnal f Medicine and the American Jurnal f Epidemilgy :, 50

51 The findings d nt challenge what is nw a well established link between air pllutin and premature death The wrk has been published fr several years in a variety f the leading jurnals like the New England Jurnal f Medicine and the American Jurnal f Epidemilgy The prject, the Natinal Mrbidity, Mrtality and Air Pllutin Study, was given extra weight by plicy makers because f the reputatin f the Health Effects Institute and the Jhns Hpkins grup :, 51

52 The findings d nt challenge what is nw a well established link between air pllutin and premature death The wrk has been published fr several years in a variety f the leading jurnals like the New England Jurnal f Medicine and the American Jurnal f Epidemilgy The prject, the Natinal Mrbidity, Mrtality and Air Pllutin Study, was given extra weight by plicy makers because f the reputatin f the Health Effects Institute and the Jhns Hpkins grup Nt as well knwn that the effect was first discvered at Health Canada, by Tim Ramsay and Rick Burnett :, 52

53 The findings d nt challenge what is nw a well established link between air pllutin and premature death The wrk has been published fr several years in a variety f the leading jurnals like the New England Jurnal f Medicine and the American Jurnal f Epidemilgy The prject, the Natinal Mrbidity, Mrtality and Air Pllutin Study, was given extra weight by plicy makers because f the reputatin f the Health Effects Institute and the Jhns Hpkins grup Nt as well knwn that the effect was first discvered at Health Canada, by Tim Ramsay and Rick Burnett their wrk als drew attentin t the incrrect calculatin f standard errrs in the gam sftware :, 53

54 The findings d nt challenge what is nw a well established link between air pllutin and premature death The wrk has been published fr several years in a variety f the leading jurnals like the New England Jurnal f Medicine and the American Jurnal f Epidemilgy The prject, the Natinal Mrbidity, Mrtality and Air Pllutin Study, was given extra weight by plicy makers because f the reputatin f the Health Effects Institute and the Jhns Hpkins grup Nt as well knwn that the effect was first discvered at Health Canada, by Tim Ramsay and Rick Burnett their wrk als drew attentin t the incrrect calculatin f standard errrs in the gam sftware Original estimate 0.41% increase in mrtality rate assciated with increase f 10µg/m 3 increase in PM 10. :, 54

55 The findings d nt challenge what is nw a well established link between air pllutin and premature death The wrk has been published fr several years in a variety f the leading jurnals like the New England Jurnal f Medicine and the American Jurnal f Epidemilgy The prject, the Natinal Mrbidity, Mrtality and Air Pllutin Study, was given extra weight by plicy makers because f the reputatin f the Health Effects Institute and the Jhns Hpkins grup Nt as well knwn that the effect was first discvered at Health Canada, by Tim Ramsay and Rick Burnett their wrk als drew attentin t the incrrect calculatin f standard errrs in the gam sftware Original estimate 0.41% increase in mrtality rate assciated with increase f 10µg/m 3 increase in PM 10. Revised estimate 0.22%. :, 55

56 :, The findings d nt challenge what is nw a well established link between air pllutin and premature death The wrk has been published fr several years in a variety f the leading jurnals like the New England Jurnal f Medicine and the American Jurnal f Epidemilgy The prject, the Natinal Mrbidity, Mrtality and Air Pllutin Study, was given extra weight by plicy makers because f the reputatin f the Health Effects Institute and the Jhns Hpkins grup Nt as well knwn that the effect was first discvered at Health Canada, by Tim Ramsay and Rick Burnett their wrk als drew attentin t the incrrect calculatin f standard errrs in the gam sftware Original estimate 0.41% increase in mrtality rate assciated with increase f 10µg/m 3 increase in PM 10. Revised estimate 0.22%. these are small effects; apprximately 12 additinal deaths per year in Mntreal, perhaps 15 in Trnt 56

CHAPTER 24: INFERENCE IN REGRESSION. Chapter 24: Make inferences about the population from which the sample data came.

MATH 1342 Ch. 24 April 25 and 27, 2013 Page 1 f 5 CHAPTER 24: INFERENCE IN REGRESSION Chapters 4 and 5: Relatinships between tw quantitative variables. Be able t Make a graph (scatterplt) Summarize the