Diversity and Team Science

Transcription

1 Diversity and Team Science Scott E Page Santa Fe Institute University of Michigan

2 Outline

3 A Great Big Complex World Diversity and Prediction Diversity Prediction Theorem Model Diversity Theorem Categorical Diversity Theorem Big Science

4 A Great Big (Complex) World

5

6

7

8 Authors Per Paper: Computer Science

9 Identity and Cognitive Diversity

10 Authors Per Paper: Computer Science Courtesy: Jacob Foster

11 ``Few if any funding programs support research on the effectiveness of science teams and larger groups.

12 Conclusions 1&2 #1: Team composition matters. Diversity is critical #2: Team composition systematic

13 Prediction

14

15 All Models Are Wrong - George Box

16 Hence, our truth is the intersection of independent lies. - Richard Levins

17

18

19

20 1197

21 Average = 1,

22 Diversity Prediction Theorem

23 Diversity Prediction Theorem Crowd Error = Average Error - Diversity

24 Crowd Error = Average Error - Diversity

25 Crowd Error = Average Error Diversity 0.6 = 2,

26 ctual Number 103 lass Prediction Diversity Prediction Theorem Crowd = Average Diversity 1086 = Number Finnish Athletes

27 ctual Number 103 lass Prediction 65.6 Diversity Prediction Theorem Crowd = Average Diversity 1401 = Number Finnish Athletes

28 Actual Number 319 Class Prediction Latvian Cars Per 1000 Diversity Prediction Theorem Crowd = Average Diversity 0.1 = 202, ,323.5

29 Actual Number 319 Class Prediction 369 Latvian Cars Per 1000 Diversity Prediction Theorem Crowd = Average Diversity 2542 = 89,743 87,201

30 Christina Romer

31

32 Model Diversity Theorem

33 N Models Distribution across models: (P 1, P 2, P 3 P n ) P j = probability someone uses model j F = probability two individuals have the same model

34 Probability two people use the same model. Match(P) = (P 1 ) 2 + (P 2 ) 2 + (P n ) 2

35 Diversity Index: Δ = 1/Match(P) Δ = Effective number of parties (political science) firms (economics) species (ecology)

36 Diversity Index Δ = (p p p 32 ) -1 (1/3) 2 + (1/3) 2 + (1/3) 2 = 3(1/9) = 1/3 Δ = 3 (1/2) 2 + (1/4) 2 + (1/4) 2 = 6/16 Δ = 2.66

37 Model Diversity Theorem Claim: N independent predictive models with a diversity index of Δ and an average variance of V, have an expected squared error equal to: V/Δ Economo, Hong, and Page (2015)

38 Size DIVERSITY (Δ) Matters

39 Goncola Abecasis

40

41 Categorical Diversity Theorem

42 Partition the set of possible instances into categories Make a prediction for each category

43 Computer Science PAC Learning: Valient Robust Classification: Provost and Foster Ensemble Learning: Desarthy and Sheela

44 Categorical Predictive Models Partition the set of possible instances into categories Make a prediction for each category

45 Chloride in Water: mg/l A B C D

46 Chloride in Water: mg/l A B C D

47 Variation in Data: 20,000 20,000 = ( ) 2 +( ) 2 +( ) 2 +( ) 2

48 Variation in Data: 20,000 20,000 = ( ) 2 +( ) 2 +( ) 2 +( ) 2 Residual Variation: 12,000 20,000 = ( ) 2 +( ) 2 +( ) 2 +( ) 2

49 Variation in Data: 20,000 20,000 = ( ) 2 +( ) 2 +( ) 2 +( ) 2 Residual Variation: 11,000 20,000 = ( ) 2 +( ) 2 +( ) 2 +( ) 2 R 2 : 0.45

50 Total Variation (20,000)

51 Chloride in Water: mg/l A B C D

52 Categorization Loss: 10,000 Mean of Category 1: 250 Categorization Loss 1 : = ( ) 2 +( ) 2 Mean of Category 2: 150 Categorization Loss 2 : = ( ) 2 +( ) 2

53 Possible Variation Explained 10,000 Categorization Categorization Error Loss 10,000

54 Chloride in Water: mg/l A B C D

55 Prediction Error: 1000= Prediction Category 1: 240 Actual Value: 250 Prediction Error: 200 = ( ) 2 +( ) 2 Prediction Category 2: 170 Actual Value: 150 Prediction Error: 800 = ( ) 2 +( ) 2

56 Variation Categorization Explained Categorization Error Loss 9,000 10,000 Prediction Error 1000

57 Categorical Diversity Theorem Variation = Explained + Category Loss + Predictive Error

58 Value of Distinct Categories Distinct Categories result in diverse categorization losses and by the Diversity Prediction Theorem lower error.

59

60

61 Six years of data Half million users 17,700 movies Data divided into (training, testing) Testing Data dived into (probe, quiz, test)

62 Singular Value Decomposition Each movie represented by a vector: (p 1,p 2,p 3,p 4 p n ) Each person represented by a vector: (q 1,q 2,q 3,q 4 q n )

63 Robert Bell Christina and David Rom

64 BellKor 50 dimensions 107 models Best Model: 6.8%

65 BellKor 50 dimensions 107 models Best Model: 6.8% Combination of Models: 8.4%

66 BellKor s Pragmatic Chaos Best Model 8.4% Ensemble: 10.1%

67 Enter ``The Ensemble 23 Teams 30 Countries

68 And the Winner is Ensemble 10.06% Bellkor 10.06%

69 But, the Real Winner is Ensemble 10.06% Bellkor 10.06% Combination 10.19%

70 Combine accurate and diverse models to make good predictions.

71 Big Science

72 Leaving Our Silos

73 Medicine Sociology Chemistry Economics

74 Freeman and Huang Citations Impact Factor # Addresses + + # References + + # Past papers + + Homophily - -

75 0.14% 0.12% Team Papers > 100 Cites 0.10% 0.08% 0.06% Team 0.04% 0.02% Solo Solo 0.00% Science & Engineering Social Science

76 Inter-University Collaboration Increases Impact Jones B, Wuchty S, Uzzi B (2008) Multi-University Research Teams: Shifting Impact, Geography, and Stratification in Science. Science 322: 1259

77 Cummings, J. N., Kiesler, S., Zadeh, R., & Balakrishnan, A. (2013). Group heterogeneity increases the risks of large group size: A longitudinal study of productivity in research groups. Psychological Science, 24(6),

78 Cummings, J. N., Kiesler, S., Zadeh, R., & Balakrishnan, A. (2013). Group heterogeneity increases the risks of large group size: A longitudinal study of productivity in research groups. Psychological Science, 24(6),

79 Cummings, J. N., Kiesler, S., Zadeh, R., & Balakrishnan, A. (2013). Group heterogeneity increases the risks of large group size: A longitudinal study of productivity in research groups. Psychological Science, 24(6),

80 Best patents have low proximity Best papers have low proximity

81

82 Atypical Connections Melissa Schilling Variable Odds Ratio Years since PHD 1.14 Prior Cites 2.25 Author Count 0.8 Depth (HHI) 3.29 Atypical Connect ``Recombinant search and breakthrough idea generation: An analysis of high impact papers in the social sciences Melissa A. Schilling and Elad Green Research Policy, 2011, vol. 40, issue 10, pages

83 Q?