Rigorous Science - Based on a probability value? The linkage between Popperian science and statistical analysis

Transcription

1

2 Rigorous Science - Based on a probability value? The linkage between Popperian science and statistical analysis

3 The Philosophy of science: the scientific Method - from a Popperian perspective Philosophy How we understand the world Science How we expand that understanding Design How we implement science Arguments over how we understand and expand our understanding are the basis of debates over how science has been, is and should be done

4 The Philosophy of science: the scientific Method - from a Popperian perspective Terms: 1. Science - A method for understanding rules of assembly or organization a) Problem: How do we, (should we) make progress in science 2. Theory - a set of ideas formulated to explain something 3. Hypothesis - supposition or conjecture (prediction) put forward to account for certain facts, used as a basis for further investigations 4. Induction or inductive reasoning - reasoning that general (universal) laws exist because particular cases that seem to be examples of those laws also exist 5. Deduction or deductive reasoning - reasoning that something must be true because it is a particular case of a general (universal) law Induction Deduction Particular General

5 The Scientific Method - from a Popperian perspective Extreme example 1. Induction Every swan I have seen is white, therefore all swans are white 2. Deduction All swans are white, the next one I see will be white Compare these statements: 1. Which can be put into the form of a testable hypothesis? (eg. prediction, if - then statement) 2. Which is closer to how we operate in the world? 3. Which type of reasoning is most repeatable? Is there a difference between ordinary understanding and scientific understanding (should there be?)

6 The Scientific Method - from a Popperian perspective Hypothetico - deductive method 1. Conception - Inductive reasoning a. Observations b. Theory c. Problem d. Regulation e. Belief Conception - Inductive reasoning Previous Observations Perceived Problem Belief 2. Leads to Insight and a General Hypothesis 3. Assessment is done by a. Formulating Specific hypotheses b. Comparison with new observations 4. Which leads to: a. Falsification - and rejection of insight, and specific and general hypotheses, or Existing Theory Confirmation H A supported (accepted) H Orejected INSIGHT General hypothesis Specific hypotheses (and predictions) Comparison with new observations Assessment - Deductive reasoning Falsification H A rejected H Osupported (accepted) b. Confirmation - and retesting of alternative hypotheses

7 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Questions and Notes 1. Is there any provision for accepting the insight or working hypothesis? 2. Propositions not subject to rejection by contrary observations are not scientific 3. Confirmation does not end hypothesis testing - new hypotheses should always be put forth for a particular observation, theory, belief 4. In practice but rarely reported, alternatives are tested until only one (or a few) are left (not rejected). Then we say things like: suggest, indicates, is evidence for 5. Why is there no provision for accepting theory or working hypotheses? Conception - Inductive reasoning Previous Observations Perceived Problem INSIGHT Existing Theory General hypothesis All swans are white Confirmation H A supported (accepted) H Orejected Comparison with new observations Belief Specific hypotheses The next swan I see will be white) Assessment - Deductive reasoning Falsification H A rejected H Osupported (accepted) a) Because it is easy to find confirmatory observations for almost any hypothesis, but one negative result refutes it absolutely (this assumes test was adequate - the quality of falsification is important)

8 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Considerations - problems with the Popperian hypothetico -deductive approach) 1) This type of normal science may rarely lead to revolutions in Science (Kuhn) A) Falsification science leads to paradigms - essentially a way of doing and understanding science that has followers B) Paradigms have momentum - mainly driven by tradition, infrastructure and psychology C) Evidence against accepted theory is considered to be exceptions (that prove the rule) D) Only major crises lead to scientific revolutions 1) paradigms collapse from weight of exceptions - normal science - crisis - revolution - normal science Conception- Inductive reasoning Previous Observations Perceived Problem INSIGHT Existing Theory General hypothesis All swans are white Specific hypotheses Comparison with new observations Belief The next swan I see will be white) Confirmation Falsification H A supported (accepted) H O rejected H A rejected H Osupported (accepted) Assessment- Deductive reasoning

9 The paradigm: The earth must be the center of the universe 350 BC By adjusting the velocities of these concentric spheres, many features of planetary motion could be explained. However, the troubling observations of varying planetary brightness and retrograde motion could not be accommodated: the spheres moved with constant angular velocity, and the objects attached to them were always the same distance from the earth because they moved on spheres with the earth at the center.

10 Exceptions are explained The "solution" to these problems came in the form of a mad, but clever proposal: planets were attached, not to the concentric spheres themselves, but to circles attached to the concentric spheres, as illustrated in the adjacent diagram. These circles were called "Epicycles", and the concentric spheres to which they were attached were termed the "Deferents". Then, the centers of the epicycles executed uniform circular motion as they went around the deferent at uniform angular velocity, and at the same time the epicyles (to which the planets were attached) executed their own uniform circular motion.

11 Ptolemy and the paradigm However, in practice, even this was not enough to account for the detailed motion of the planets on the celestial sphere! In more sophisticated epicycle models further "refinements" were introduced: In some cases, epicycles were themselves placed on epicycles, as illustrated in the figure below. In actual models, the center of the epicycle moved with uniform circular motion, not around the center of the deferent, but around a point that was displaced by some distance from the center of the deferent. That ancient astronomers could convince themselves that this elaborate scheme still corresponded to "uniform circular motion" is testament to the power of three ideas that we now know to be completely wrong, but that were so ingrained in the astronomers of an earlier age that they were essentially never questioned: 1.All motion in the heavens is uniform circular motion. 2.The objects in the heavens are made from perfect material, and cannot change their intrinsic properties (e.g., their brightness). 3.The Earth is at the center of the Universe. These ideas concerning uniform circular motion and epicycles were catalogued by Ptolemy in 150 A.D. His book was called the "Almagest" (literally, "The Greatest"), and this picture of the structure of the Solar System has come to be called the "Ptolemaic Universe". Paradigm nears scientific collapse

12 Religion Intervenes By the Middle Ages, such ideas took on a new power as the philosophy of Aristotle (newly rediscovered in Europe) was wedded to Medieval theology in the great synthesis of Christianity and Reason undertaken by philosophertheologians such as Thomas Aquinas. The Prime Mover of Aristotle's universe became the God of Christian theology, the outermost sphere of the Prime Mover became identified with the Christian Heaven, and the position of the Earth at the center of it all was understood in terms of the concern that the Christian God had for the affairs of mankind. Thus, the ideas largely originating with pagan Greek philosophers were baptized into the Catholic church and eventually assumed the power of religious dogma: to challenge this view of the Universe was not merely a scientific issue; it became a theological one as well, and subjected dissenters to the considerable and not always benevolent power of the Church.

13 The Copernican Revolution 1543 AD

14 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Considerations - problems with the Popperian hypothetico -deductive approach) 1) This type of normal science may rarely lead to revolutions in Science (Kuhn) A) Falsification science leads to paradigms - essentially a way of doing and understanding science that has followers B) Paradigms have momentum - mainly driven by tradition, infrastructure and psychology C) Evidence against accepted theory is considered to be exceptions (that prove the rule) D) Only major crises lead to scientific revolutions 1) paradigms collapse from weight of exceptions - normal science - crisis - revolution - normal science Aristotle - Ptolemian universe Copernican Universe

15 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Considerations - problems with the Popperian hypothetico -deductive approach) 2) Philosophical opposition - (e.g. Roughgarden 1983) A) Establishment of empirical fact is by building a convincing case for that fact. B) We don t use formal rules in everyday life, instead we use native abilities and common sense in building and evaluating claims of fact C) Even if we say we are using the hypothetico - deductive approach, we are not, instead we use intuition and make it appear to be deduction Conception- Inductive reasoning Previous Observations Perceived Problem INSIGHT Existing Theory General hypothesis All swans are white Specific hypotheses Comparison with new observations Belief The next swan I see will be white) Confirmation Falsification H A supported (accepted) H O rejected H A rejected H Osupported (accepted) Assessment- Deductive reasoning

16 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Considerations - problems with the Popperian hypothetico -deductive approach) 3) Practical opposition - (e.g. Quinn and Dunham 1983) A) In practice ecology and evolution differ from Popperian science 1) they are largely inductive 2) although falsification works well in physical and some experimental areas of biology - it is difficult to apply in complex systems of multiple causality - e.g. Ecology and Evolution 3) Hypothetico - deductive reasoning works well if potential cause is shown not to work at all (falsified) but this rarely occurs in Ecology or Evolution - usually effects are of degree. Conception- Inductive reasoning Previous Observations Perceived Problem INSIGHT Existing Theory General hypothesis All swans are white Specific hypotheses Comparison with new observations Belief The next swan I see will be white) Confirmation Falsification H A supported (accepted) H O rejected Assessment- Deductive reasoning H A rejected H Osupported (accepted)

17 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Considerations - problems with the Popperian hypothetico -deductive approach) 1) Choice of Method for doing science. Platt (1964) reviewed scientific discoveries and concluded that the most efficient way of doing science consisted of a method of formal hypothesis testing he called Strong Inference. A) Apply the following steps to every problem in Science - formally, explicitly and regularly: 1) Devise alternative hypotheses 2) Devise critical experiments with alternative possible outcomes, each of which will exclude one or more of the hypotheses (rejection) 3) Carry out procedure so as to get a clean result 1 ) Recycle the procedure, making subhypotheses or sequential ones to define possibilities that remain Conception- Inductive reasoning Previous Observations Perceived Problem INSIGHT Existing Theory General hypothesis All swans are white Specific hypotheses Comparison with new observations Belief The next swan I see will be white) Confirmation Falsification H A supported (accepted) H O rejected Assessment- Deductive reasoning H A rejected H Osupported (accepted)

18 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Considerations - problems with the Popperian hypothetico -deductive approach) 1) Choice of Method for doing science. Platt (1964) reviewed scientific discoveries and concluded that the most efficient way of doing science consisted of a method of formal hypothesis testing he called Strong Inference. A) Apply the following steps to every problem in Science - formally, explicitly and regularly: 1) Devise alternative hypotheses 2) Devise critical experiments with alternative possible outcomes, each of which will exclude one or more of the hypotheses (rejection) 3) Carry out procedure so as to get a clean result 1 ) Recycle the procedure, making subhypotheses or sequential ones to define possibilities that remain Conception- Inductive reasoning Previous Observations Perceived Problem INSIGHT Existing Theory General hypothesis All swans are white Specific hypotheses Comparison with new observations Belief The next swan I see will be white) Confirmation Falsification H A supported (accepted) H O rejected Assessment- Deductive reasoning H A rejected H Osupported (accepted) Very much like the Popperian method!

19 The Scientific Method - from a Popperian perspective Hypothetico - deductive method Considerations - problems with the Popperian hypothetico -deductive approach) 3) Practical opposition - (e.g. Quinn and Dunham 1983) A) In practice ecology and evolution differ from Popperian science 1) they are largely inductive 2) although falsification works well in physical and some experimental areas of biology - it is difficult to apply in complex systems of multiple causality - e.g. Ecology and Evolution 3) Hypothetico - deductive reasoning works well if potential cause is shown not to work at all (falsified) but this rarely occurs in Ecology or Evolution - usually effects are of degree. Conception- Inductive reasoning Previous Observations Perceived Problem INSIGHT Existing Theory General hypothesis All swans are white Specific hypotheses Comparison with new observations Belief The next swan I see will be white) Confirmation Falsification H A supported (accepted) H O rejected Assessment- Deductive reasoning H A rejected H Osupported (accepted) This may be a potent criticism and it leads to the use of inferential statistics

20 Absolute vs. measured differences A) Philosophical underpinnings of Popperian Method is based on absolute differences 1) E.g. All swans are white, therefore the next swan I see will be white - If the next swan is not white then the hypothesis is refuted absolutely. B) Instead, most results are based on comparisons of measured variables 1) not really true vs. false but degree to which an effect exists Example - Specific hypothesis number of Oak seedlings is higher in areas outside impact sites than inside impact sites Observation 1: Number inside Number outside Mean 0 13 What counts as a difference? Are these different? Observation 2: Number inside Number outside Mean 5 9.2

21 Statistical analysis - cause, probability, and effect I) What counts as a difference - this is a statistical philosophical question of two parts A) How shall we compare measurements - statistical methodology B) What will count as a significant difference - philosophical question and as such subject to convention II) Statistical Methodology - General A) Null hypotheses - H o 1) In most sciences, we are faced with: a) NOT whether something is true or false (Popperian decision) b) BUT rather the degree to which an effect exists (if at all) - a statistical decision. 2) Therefore 2 competing statistical hypotheses are posed: a) H A : there is a difference in effect between (usually posed as < or >) b) H O : there is no difference in effect between Specific hypothesis H A : Number of oak seedlings is Greater in areas outside impact Sites than inside impact sites Conception Previous Observations Existing Theory Confirmation H A supported (accepted) H O rejected - Inductive reasoning Perceived Problem INSIGHT General hypothesis Specific hypotheses (and predictions) Comparison with new observations Belief Falsification H A rejected H O supported (accepted) Assessment - Deductive reasoning

22 Statistical analysis - cause, probability, and effect Statistical tests - a set of rules whereby a decision about hypotheses is reached (accept, reject) 1) Associated with rules - some indication of the accuracy of the decisions - that measure is a probability statement or p-value 2) Statistical hypotheses: a) do not become false when a critical p-value is exceeded b) do not become true if bounds are not exceeded c) Instead p-values indicate a level of acceptable uncertainty d) critical p-values are set by convention - what counts as acceptable uncertainty e) Example - if critical p-value = 0.05 this means that we are unwilling to accept the posed alternative hypothesis unless: 1) we 95% sure that it is correct, or equivalently that 2) we are willing to accept an error rate of 5% or less that we are wrong when we accept the hypothesis

23 Statistical analysis - cause, probability, and effect The logic of statistical tests - how they are performed 1) Assume the Null Hypothesis (H o ) is true (e.g. no difference in number of Oak seedlings in impact and non-impact sites). 2) Compare measurements - generally this means comparing two sample distributions (determined from the numbers from the experiment or survey) A) Comparison of distributions - generally by comparing means and the estimate of error associated with the sampling of the means simplest case is the Standard Error of the Mean (SE or SEM) = S x.5 sd/n = standard deviation / square root of level of replication 3) Determine the probability that distributions are similar/different 4) Compare with a critical p-value to assign significance.

24 Calculation of statistical distributions Distribution of Oak Seedlings - pre-impact VALUES Sites = 100 Mean per site = 25 Total seedlings = 2500

25 Evaluate effect of sample size on calculation of and confidence in Mean Compare for sample size's of 5,10, 20, 50, 99 cells Iterate 50 times Example: sample 10 sites to determine mean x fifty iterations 31 X=21.5 X=25.8

26 Mean = True Mean = Means Mean = 25.8 Number of cases Estimate of Mean

27 Effect of number of observations on estimate of Mean # of cases Proportion per Bar Estimate of Mean

28 Effect of number of observations on estimate of Mean Number of Cases 2.5% 2 SE 2 SE 2.5% # of cases Estimate of Mean Proportion per Bar Estimate of Mean

29 Effect of number of observations on estimate of Mean 16 Frequency distributions of sample means Number of cases # of cases Estimate of Mean Proportion per Bar Estimate of Mean Mean based on X observations

30 How to estimate optimal sample size 1) Do a preliminary study of variables that will be evaluated in project 2) Plot the mean and some estimate of variability of data as a function of sample size 3) Look for sweet spot where estimates of mean and variance (or standard deviation) converge on a stable value Value Mean Standard Deviation 4) Calculate a bang for buck relationship to determine if a robust design (sufficient sampling) can be paid for SAMPLES

31 Trade off between accuracy and cost Accuracy 1 Bang per buck + - Cost Sample Size e.g. number of replicate sites ( 1 Accuracy = index of approach to most reliable estimate of mean)

32 Types of statistical error Type 1 and II Type 1 and Type II error. 1) By convention, the hypothesis tested is the null hypothesis (no difference between) a) In statistics, assumption is made that a hypothesis is true (assume H O true = assume H A false) b) accepting H O (saying it is likely to be true) is the same as rejecting H A (falsification) c) Scientific method is to falsify competing alternative hypotheses (alternative H A s) 2) Errors in decision making Decision Truth Accept H O Reject H O H O true no error (1-α) Type I error (α) H O false Type II error ( β ) no error (1- β) Type I error - probability α that we mistakenly reject a true null hypothesis (H O ) Type II error - probability β that we mistakenly fail to reject (accept) a false null hypothesis Power of Test - probability (1-β) of not committing a Type II error - The more powerful the test the more likely you are to correctly conclude that an effect exists when it really does (reject H O when H O false = accept H A when H A true).

33 Conception - Inductive reasoning Perceived Problem Scientific method and statistical errors Previous Observations INSIGHT Existing Theory General hypothesis Belief Confirmation H A supported (accepted) H O rejected Specific hypotheses (and predictions) Comparison with new observations Falsification H A rejected H O supported (accepted) Assessment - Deductive reasoning Decision Truth Accept H O Reject H O H O true no error (1-alpha) Type I error (alpha) H O false Type II error (beta) no error (1-beta)

34 Conception - Inductive reasoning Perceived Problem Scientific method and statistical errors Oil leaking on a number of sites with Oaks Existing Theory INSIGHT Oil affects Oak seedlings Belief - case example Confirmation Seedling # will be higher In control sites than on impact sites H A supported (accepted) H Orejected More seedlings in control sites Compare Seedling # on impact and control sites Falsification H A rejected H Osupported (accepted) No difference Assessment - Deductive reasoning Decision Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta)

35 Error types and implications in basic and environmental science Biological Truth No Impact Impact Monitoring No Impact Correct decision No impact detected Type II Error Failure to detect real impact; false sense of security Conclusion Impact Type 1 Error False Alarm Correct decision Impact detected Decision Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta) What type of error should we guard against?

36 Statistical Power, effect size, replication and alpha Statistical comparison of two distributions X1 X2

37 Statistical Power, effect size, replication and alpha Area under the curve = Count Proportion per Bar Estimate of Mean

38 Distributions are truly the same assign critical p (assume = 0.05) Distribution of Sample Means X1 X2 Decision Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta) If distributions are truly the same then: area to right of critical XC represents the Type 1 error rate (Blue); any calculated X2 > XC will cause incorrect conclusion that distributions are different xc

39 Distributions are truly different assign critical p (assume = 0.05) Distribution of Sample Means X1 X2 Decision Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta) If distributions are truly different then: area to left of critical XC represents the Type II error rate (Red); any calculated X2 < XC will cause incorrect conclusion that distributions are same xc

40 How to control Type II error (distributions are truly different) This will maximize statistical power to detect real impacts 1) Vary critical P-Values 2) Vary Magnitude of Effect 3) Vary replication Decision Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta) Critical p X1 X2 Type II error xc Type I error

41 How to control Type II error (distributions are truly different) 1) Vary critical P-Values (change blue area) Critical p X1 X2 Decision Reference Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta) Type II error xc Type I error Critical p A) Make critical P more stringent (smaller) Type II error increases Power decreases xc Critical p A) Relax critical P (larger values) Type II error decrease Power increases xc

42 How to control Type II error (distributions are truly different) 2) Vary magnitude of effect (vary distance between X1 and X2) Critical p X1 X2 Decision Reference Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta) Type II error xc Type I error X1 X2 A) Make distance smaller Type II error increases Power decreases xc A) Make distance greater X1 X2 Type II error decreases Power increases xc

43 How to control Type II error (distributions are truly different) 3) Vary replication (which controls estimates of error) Critical p X1 X2 Decision Reference Truth Accept HO Reject HO HO true no error (1-alpha) Type I error (alpha) HO false Type II error (beta) no error (1-beta) Type II error xc Type I error A) Decrease replication X1 X2 Type II error increases Power decreases xc A) Increase replication X1 X2 Type II error decrease Power increases xc

44 POWER High Power Low Low Replication High High High Power Power Low Small Large Low Low High Magnitude of Effect Critical alpha

45 Replication and independence Control sites Impact sites Pseudoreplication in space Pseudoreplication in time Pseudoreplication in time and space

46 The fallacy of the primacy of alpha for environmental work The trade-off between error and cost an obvious conflict (I) Cost of assessment Sample size

47 The fallacy of the primacy of alpha for environmental work The trade-off between error and cost an obvious conflict (II) Probability (statistical) of detecting true impact Probability (statistical) of concluding no impact Cost of assessment Therefore, the cynical conclusion is that the best strategy for an applicant would be to under-fund an assessment A) The lower the cost the more likely that a conclusion of no impact will be reached (regardless of the truth)

48 The fallacy of the primacy of alpha for environmental work The trade-off between error and cost an obvious conflict (III) Recommendations: 1) Determine the importance of error rates as a ratio For example what is the relative importance of: correctly concluding there is an impact (I) vs. correctly concluding there is no impact (N) 2) Determine the maximum Type 1 error acceptable (probability of concluding there is an effect (impact) when there is not one) 3) Estimate cost of assessment 4) Recycle procedure if necessary 5) Do not do the assessment if acceptable ratios and maximum error rates cannot be attained

49 General schematic for the design of a monitoring program 1 Basic Sampling Design Select Variables Effect Size Cost of errors Background Variation Accept Greater Error Rate Y N Refine design Achievable? Y Go 1 from Keough and Mapstone 1993