Modeling Like It Matters. Ty Ferre Hydrology and Atmospheric Sciences University of Arizona Tucson

Transcription

1 Modeling Like It Matters Ty Ferre Hydrology and Atmospheric Sciences University of Arizona Tucson 1

2 Science Psychology The Game 2

3 The Scientific Method Observe Predict Theorize Hypothesize 3

4 Changing Your Mind Observe Predict Theorize Hypothesize Opportunity for lateral thinking 4

5 Predict Scientific Progress Observe Observe Theorize Observe Predict Theorize Hypothesize Observe Predict Theorize Hypothesize Predict Hypothesize Hypothesize Theorize 5

6 The Path of Science Observe Predict Observe Theorize Observe Predict Hypothesize Theorize Predict Observe Theorize Hypothesize Predict Hypothesize Theorize Hypothesize 6

7 Predict The Paths of Science Observe Observe Predict TheorizeObserve Observe Predict Hypothesize Predict Theorize Observe Theorize Hypothesize Observe Hypothesize Predict Hypothesize Theorize Predict Observe Theorize Hypothesize Predict Hypothesize Theorize Theorize Hypothesize 7

8 Predict Changing Your Mind Observe Observe Predict TheorizeObserve Observe Predict Hypothesize Predict Theorize Observe Theorize Hypothesize Observe Hypothesize Predict Hypothesize Theorize Predict Observe Theorize Hypothesize Predict Hypothesize Theorize Opportunity for lateral thinking Hypothesize Theorize 8

9 The Paths of Science Time 9 Model Space???

10 Splitting, Dying, Merging, Emerging Model Space Time 10

11 Visualizing Current Model Space 11 Model Space Model Space Time

12 Single Model Space (Discrete and Continuous Differences) Concepts Processes Structures Parameterizations Solution Schemes Parameter Values Boundary Conditions Space/Time Resolutions Model Space Model Space 12

13 Multi-Model Space Concepts Processes Structures Parameterizations Solution Schemes Parameter Values Boundary Conditions Space/Time Resolutions Model Space Model Space 13

14 Assessing Model Quality (Focus on A Few Models) Observe Predict Theorize Hypothesize Model Space 14

15 Assessing Model Quality Observe Predict Theorize Hypothesize Predicted Value One of all possible things that you could measure and that the models can predict. Observable Quantity 15

16 Assessing Model Quality Observe Predict Theorize Hypothesize Predicted Value Measured value Observable Quantity 16

17 Model Likelihood L = f(1/mismatch) ΣL = 1 Predicted Value Likelihood Sorted Models Observable Quantity 17

18 Model Likelihood L = f(1/mismatch) ΣL = 1 zero sum game Predicted Value Likelihood Sorted Models Observable Quantity 18

19 Model Likelihood L = f(1/mismatch) ΣL = 1 Predicted Value Likelihood Sorted Models Observable Quantity 19

20 Data Quality Predicted Value One of all possible things that you could measure and that the models can predict. Observable Quantity 20

21 Data Quality Predicted Value One of all possible things that you could measure and that the models can predict. Observable Quantity 21

22 Data Quality Better/Worse Data Likelihood Sorted Models Predicted Value We can assess their likely value BEFORE we collect the measurement. Observable Quantity 22

23 Discriminatory Data Better/Worse Data? Predicted Value Observable Quantity 23

24 Discriminatory Data Better/Worse Data? Depends on your objective Predicted Value Observable Quantity 24

25 Discriminatory Index DI = f(intergroup difference/ intragroup spreads) Better/Worse Data Even Worse Data Predicted Value Observable Quantity 25

26 Discriminatory Index DI = f(intergroup difference / intragroup variance) Better/Worse Data Predicted Value Observable Quantity 26

27 Model Importance Predicted Value Forecastable Quantity 27

28 Predictions of Interest Predicted Value Stakeholder 1 Stakeholder 2 Stakeholder 3 Forecastable Quantity 28

29 Outcomes of Concern Predicted Value Stakeholder 1 Stakeholder 2 Stakeholder 3 Forecastable Quantity 29

30 Models vs Outcomes of Concern Predicted Value Stakeholder 1 Forecastable Quantity 30

31 Models of Concern / Other Models Predicted Value Stakeholder 1 Forecastable Quantity 31

32 Discriminatory Data for a Stakeholder s Models of Concern Better/Worse Data Even Worse Data Predicted Value Predicted Value Observable Quantity Stakeholder 1 32 Forecastable Quantity 32

33 Models of Concern / Other Models Predicted Value Stakeholder 2 Forecastable Quantity 33

34 Decision Confidence Index Likelihood Models of Concern Other Models Sorted Models 34

35 Decision Confidence Index LOW CONFIDENCE Likelihood Models of Concern Other Models Sorted Models 35

36 Decision Confidence Index HIGH CONFIDENCE but unhappy Likelihood Models of Concern Other Models Sorted Models 36

37 Decision Confidence Index HIGH CONFIDENCE and happy Likelihood Models of Concern Other Models Sorted Models 37

38 Summary of Talk One Multiple plausible models can be built for any system; 38

39 Summary of Talk One Multiple plausible models can be built for any system; Model likelihoods can be defined based on fit to data; 39

40 Summary of Talk One Multiple plausible models can be built for any system; Model likelihoods can be defined based on fit to data; The expected value of data for any prediction(s) of interest can be assessed before the data are collected; 40

41 Summary of Talk One Multiple plausible models can be built for any system; Model likelihoods can be defined based on fit to data; The expected value of data for any prediction(s) of interest can be assessed before the data are collected; Stakeholders must define the prediction(s) of interest, which will identify the models of concern; 41

42 Summary of Talk One Multiple plausible models can be built for any system; Model likelihoods can be defined based on fit to data; The expected value of data for any prediction(s) of interest can be assessed before the data are collected; Stakeholders must define the prediction(s) of interest, which will identify the models of concern; Decision support relies on segregating eithermodels of concern or other models as high likelihood. 42

43 Talk Two: Time for Some Psychology Predicted Value Stakeholder 2 Forecastable Quantity 43

44 Stakeholders Must Drive the Process Each stakeholder must define their outcomes of concern. Predicted Value Stakeholder 2 Forecastable Quantity 44

45 Stakeholders Must Drive the Process (Stakeholders are Human) Predicted Value It is not a scientist s job to tell a stakeholder what to care about. Forecastable Quantity 45

46 Outcomes of Concern Are Biased (Stakeholders are Human) esp. loss aversion Predicted Value Forecastable Quantity 46

47 Modeling Is Biased (Modelers are Human) esp. confirmation bias Model Space Modeler 1 Modeler 2 Modeler 3 Time 47

48 Embrace the Bias It is not a scientist s job to inject false objectivity but, rather, to test hypotheses especially those that matter to stakeholders. Model Space Modeler 1 Time 48

49 Develop Plausible Models of Concern Predicted Value Stakeholders, with their experts (modelers), are best able to identify their outcomes of concern and find discriminatory data. Model Space Time Forecastable Quantity 49

50 An Ensemble of Diverse Biased Models Is Not Biased Stakeholder 1 Stakeholder 2 Stakeholder 3 Model Space Time Each stakeholder s modelers should be encouragedto form plausible models of concern to address their interests. 50

51 An Ensemble of Diverse Biased Models Is Not Biased Stakeholder 1 Stakeholder 2 Stakeholder 3 Models of Concern Other Models Model Space Likelihood Time Sorted Models Each stakeholder s modelers should be encouragedto form plausible models of concern to address their interests. Decisions should be made using a combined model ensemble. 51

52 View the Model Ensemble As a Team of Rivals Likelihood Likelihood Sorted Models Models of Concern Other Models Sorted Models Where advisors (models) agree, a clear decision can be made. 52

53 View the Model Ensemble As a Team of Rivals Likelihood Models of Concern Other Models Sorted Models Where advisors (models) agree, a clear decision can be made. Where models disagree, more information (data) is needed. 53

54 Summary of Talk Two Decision-making is inherently biased; 54

55 Summary of Talk Two Decision-making is inherently biased; Model construction is inherently biased; We should embrace bias to form a diverse ensemble of biased models to consider all stakeholders concerns; 55

56 Summary of Talk Two Decision-making is inherently biased; Model construction is inherently biased; We should embrace bias to form a diverse ensemble of biased models to consider all stakeholders concerns; Discriminatory data should be chosen collectively. 56

57 The Game 57 Higher Head Lower Head Higher Head Lower Head E Lower K Higher K S Lower K Higher K Higher Head Lower Head 40% B Lower K Higher K 60% 70%

58 The Game 58 Higher Head Lower Head Higher Head Lower Head E Lower K Higher K S Lower K Higher K Higher Head Lower Head 40% K eff = 0.04 B Lower K Higher K 60% K eff = % K eff = 0.46

59 59 Normalized K eff

60 60 Normalized K eff

61 Most time spent 61 Normalized Keff

62 90% Percolating 10% 62

63 90% Harmonicish 10% 63

64 90% Harmonicish 10% 64

65 Inferred K eff 0,5 0,45 0,4 0,35 0,3 0,25 0,2 0,15 0,1 0,05 0 R² = 0, ,1 0,2 0,3 0,4 0,5 True K eff

66

67 1/

68 1/ /

69 1/ / /

70 Summary of Talk Three Use models to train your intuition; 70

71 Summary of Talk Three Use models to train your intuition; Students rock. 71

72 Summary of the Three Talks A scientist s job is to propose different plausible models; Then to find discriminatory data; This can be guided by stakeholder s interests; This requires trained intuition. 72