Imprint WHY I AM NOT A LIKELIHOODIST. Greg Gandenberger. volume 16, no. 7 may University of Pittsburgh / University of Bristol.

Transcription

1 Philosophers Imprint volume 16, no. 7 may 2016 WHY I AM NOT A LIKELIHOODIST Greg Gandenberger University of Pittsburgh / University of Bristol 2016, Gregory Gandenberger < Abstract Frequentist statistical methods continue to predominate in many areas of science despite prominent calls for statistical reform. They do so in part because their main rivals, Bayesian methods, appeal to prior probability distributions that arguably lack an objective justification in typical cases. Some methodologists find a third approach called likelihoodism attractive because it avoids important objections to frequentism without appealing to prior probabilities. However, likelihoodist methods do not provide guidance for belief or action, but only assessments of data as evidence. I argue that there is no good way to use those assessments to guide beliefs or actions without appealing to prior probabilities, and that as a result likelihoodism is not a viable alternative to frequentism and Bayesianism for statistical reform efforts in science. 1. Introduction The Open Science Collaboration (2015) recently published the results of attempts to replicate 100 findings from studies that were published in prominent psychology journals. Only 36 of the 100 replication attempts yielded statistically significant results, compared to 97 of the 100 original studies. Moreover, the average effect size in the replication attempts was approximately half that in the original studies. Many psychologists have argued that replication rates could be improved through some kind of statistical reform. Most publications in psychology (and many other areas of science) use frequentist statistical methods. Those methods have some appealing characteristics. For instance, a frequentist test of a hypothesis is designed to reject that hypothesis with low probability (often 5%) if it is true and with high probability (often 80%) if it is false to a substantial degree. However, frequentist methods violate a plausible claim about evidence called the Likelihood Principle, in the sense that they are sensitive to aspects of experimental outcomes that the Likelihood Principle entails are evidentially irrelevant to the set of hypotheses in question. The Likelihood

2 Principle is supported by many lines of reasoning, including axiomatic arguments (Birnbaum, 1962; Berger and Wolpert, 1988; Gandenberger, 2015). In addition, results from studies like the Open Science Collaboration suggest that frequentist methods have not been producing reliable results, at least among studies published in prominent journals in certain fields. Many advocates of statistical reform promote Bayesian methods as superior alternatives to frequentist approaches (e.g. Wagenmakers et al., 2008; Dienes, 2011; Wetzels et al., 2011; Kruschke, 2013). Those methods involve assigning prior probabilities to hypotheses and then updating those probabilities by conditioning on experimental results. Unlike frequentist methods, Bayesian conditioning conforms to the Likelihood Principle. However, the results it yields in a given case depend on not only the evidence that is obtained, but also the prior probability distribution that is used. There are many objective Bayesian proposals for specifying prior probability distributions by formal rules, but none of those proposals are generally accepted. As a result, Bayesian methods face the objection that their dependency on prior probability distributions makes them too subjective for use in science. Advocates of those methods have proposed ways to mitigate this dependency and have argued that it is no worse than the problems that other approaches face, 1 but they generally admit that it is a genuine source of difficulties. There is a third approach called likelihoodism that conforms to the Likelihood Principle without appealing to prior probability distributions, thereby combining some of the most attractive features of frequentism and Bayesianism. However, unlike frequentist and Bayesian approaches, likelihoodist methods do not attempt to address questions 1. For instance, many advocates for Bayesian methods argue that fully objective methods are impossible and that the subjective elements of frequentist methods are more pernicious than those of Bayesian methods because they are less open to inspection (e.g. Wagenmakers et al., 2008, 198 9). They often propose mitigating the influence of subjective factors on the outputs of Bayesian methods in science through some form of robustness analysis, such as methods developed by Berger (1994). about what one should believe or do; they only provide assessments of data as evidence. Likelihoodists claim that this circumspection is a virtue. Their methods are particularly intended for cases in which objectively well-grounded prior probabilities are not available. In such cases, Bayesian guidance for belief and action is not objectively well-grounded, but likelihoodist assessments of evidence can be. Likelihoodists claim that we should restrict our attention to likelihoodist assessments of evidence in such cases, rather than making claims about belief or action that outrun our legitimate epistemic resources (Sober, 2008, 32). In this paper, I argue that likelihoodism is not a viable alternative to frequentism and Bayesianism for statistical reform efforts in science. I briefly explain the frequentist, Bayesian, and likelihoodist methodologies in Section 2. Then in Section 3, I argue that a viable candidate for statistical reform efforts would provide good guidance for belief or action. In Section 4, I argue that any likelihoodist attempt to provide such guidance would have to conform to one of two simple principles. I then argue in Sections 5 7 that those principles have unacceptable consequences. Likelihoodist ideas might be useful for some purposes despite the arguments given here. For instance, subjective Bayesian scientists could communicate their results by sharing likelihoodist characterizations of their data as evidence rather than their own posterior degrees of belief (Barnard et al., 1962). In this way, likelihoodist ideas could provide a practically significant supplement to Bayesianism. One might think that evidential favoring plays a role in inductive inference that is roughly analogous to the role that validity plays in deductive inference; just as the deductive strength of an argument for an individual depends on whether or not that individual believes its premises as well as on whether or not the argument is valid, so too the inductive strength of an argument for an individual depends on his or her assessment of the prior probabilities of the relevant hypotheses as well as the associated degrees of evidential favoring This analogy was brought to my attention by an anonymous referee. philosophers imprint vol. 16, no. 7 (may 2016)

3 My point is that the use of likelihoodist methods does nothing to mitigate the need to appeal to prior probabilities in order to put likelihoodist characterizations of data as evidence to practical use. Thus, the likelihoodist promise to provide a methodology for science that can avoid the pitfalls of Bayesianism is empty. Some advocates of likelihoodist ideas only intended them to be supplements to Bayesianism (e.g. Good, 1985, 266), but others present them as alternatives to Bayesian methods to be used when objectively well-grounded prior probabilities are not available (e.g. Edwards, 1972; Royall, 1997; Sober, 2008). 2. Frequentist, Bayesian, and Likelihoodist Methods In this section, I briefly explain how frequentist, Bayesian, and likelihoodist methods work. Historically, frequentist statistical methods have been associated with frequentist views about probability statements that is, views according to which statements about probabilities should be understood as statements about hypothetical longrun frequencies. Likewise, Bayesian statistical methods have been associated with the view that probability statements should be understood in terms of rational degrees of belief. However, certain hybrid views are possible. For instance, one could maintain that it is perfectly legitimate to interpret probability statements in terms of degrees of belief, but that scientists should use frequentist methods because degrees of belief should not play the role in scientific inquiry that Bayesian methods give to them. In this paper, I am concerned with questions about what kinds of methods scientists should use, and not directly with questions about how probability statements should be understood. Consider the following example. Example 1. Between 1973 and 1986, a team of researchers led by Robert Bartlett used a novel technique called extracorporeal membrane oxygenation (ECMO) to treat one hundred newborns suffering from severe respiratory problems. Each patient was estimated to have no more than a 20% chance of surviving under then-conventional ventilator therapy, but 72 of the 100 survived under ECMO (Bartlett et al., 1986). In a randomized follow-up trial, all eleven patients who were assigned to ECMO survived, and the only patient who was assigned to ventilator therapy died (Bartlett et al., 1985). Suppose one wished to evaluate the hypothesis that ECMO is more effective than ventilator therapy. A typical frequentist approach to this case would begin with designating the claim that ECMO is no more effective than conventional ventilator therapy the null hypothesis. Before looking at the data from a given trial, the frequentist would specify a rule for deciding on the basis of that data whether or not to reject the null hypothesis in favor of the alternative hypothesis that ECMO is more effective than conventional therapy. This rejection rule is to be chosen so that the probability of rejecting the null hypothesis is as large as possible if that hypothesis is false without rising above a specified threshold α (often.05) if it is true. This approach, called null hypothesis significance testing, is ubiquitous in most areas of science. Frequentists take it to be justified, at least in part, by guarantees about its long-run performance in repeated applications with varying data. 3 For instance, with probability one the frequency with which a scientist who used a fixed value for α in a series of null hypothesis significance tests would reject true null hypotheses in that series would approach α in the long run as the number of true null hypotheses increased without bound. Frequentists appeal to guarantees about long-run performance in repeated applications to justify other kinds of techniques as well as null hypothesis significance tests, including methods of estimation. One important source of objections to frequentist methods is that they violate the Likelihood Principle. The Likelihood Principle says 3. There are longstanding disagreements among frequentists about whether their methods should be understood simply as rules for obtaining good performance in repeated applicaitons in the long run or as having a deeper epistemic justification (see e.g. Fisher, 1955; Neyman, 1957). philosophers imprint vol. 16, no. 7 (may 2016)

4 roughly that the evidential import of a body of data for a set of hypotheses depends only on how probable that body of data would be if each hypothesis in the set were true. Null hypothesis significance tests violate the Likelihood Principle by being sensitive to the probabilities that the null hypothesis assigns to possible but non-actual observations. For instance, suppose that a coin lands heads nine times in twelve flips. A null hypothesis significance test with α =.05 would reject the hypothesis that the coin is fair on this outcome if it had been determined in advance that data collection would cease after the third tail. However, the same kind of test would not reject that same hypothesis on that same outcome if it had been determined in advance that data collection would cease after the twelfth toss. The difference between these two cases lies in the data that could have been but were not observed: in the first case, any sequence containing three tails of which one is in the final position is possible, while in the second case, any sequence containing twelve outcomes is possible. The Likelihood Principle s implication that this difference in merely possible data is irrelevant to the evidential import of the actual data is not only intuitively plausible, but also supported by formal proofs from principles that seem to be even more immediately compelling (Birnbaum, 1962; Berger and Wolpert, 1988; Gandenberger, 2015). Formally, the Likelihood Principle says that the evidential import of a body of data E for a set of hypotheses H depends only on the likelihood function Pr(E H) as a function of H on H. Unlike frequentist methods, Bayesian conditioning conforms to this principle. Bayesians claim that scientists should assign prior probabilities to hypotheses and update them on new data by Bayes s rule: Pr(H E) = Pr(E H) Pr(H) Pr(E) The denominator on the right-hand side can also be written i Pr(E H i ) Pr(H i ), where {H i } is a set of mutually exclusive and exhaustive hypotheses that includes H (or as a corresponding integral if the hypothesis space is continuous). This expansion makes it clear that Bayesian conditioning conforms to the Likelihood Principle in the following sense: for any H in H = {H i }, the posterior probability of H given E depends on E only through the likelihood function Pr(E H i ) considered as a function of H i on H. In the ECMO case, a Bayesian might put a probability distribution over the set of hypotheses H p according to which the probability that a given patient in the randomized trial receiving ECMO would survive is p (where p [0, 1]). 4 He or she would then update those probabilities upon learning that all eleven patients who received ECMO survived by plugging Pr(11 of 11 patients receiving ECMO survive H p ) as a function of p into Bayes s theorem. He or she could proceed similarly for hypotheses about the effectiveness of conventional ventilator therapy, and then use the results of those procedures to calculate the probability that ECMO is more effective than ventilator therapy (in terms expected survival rates). This approach is attractive because it seems that this probability (along with information about side effects) is exactly what we want to learn from the ECMO trials for the sake of deciding whether to use ECMO or ventilator therapy in future cases. However, getting a posterior probability out of Bayes s theorem requires choosing a prior probability distribution to put into it, and in this case, as is the usual case in science, we have no generally accepted method for doing so. Like Bayesian methods, likelihoodist methods also conform to the Likelihood Principle. However, they do so without appealing to prior probabilities. They are based on a claim about evidence called the Law of Likelihood that goes beyond the Likelihood Principle. Whereas the Likelihood Principle says that evidential import depends only on likelihood functions, the Law of Likelihood says something about how it does so. Specifically, the Law of Likelihood says that a body of data E favors a hypothesis H 1 over another hypothesis H 2 if and only if 4. I assume that patient outcomes are independent and that p is the same for each patient merely for the sake of illustration. philosophers imprint vol. 16, no. 7 (may 2016)

5 the likelihood ratio L = Pr(E H 1 )/ Pr(E H 2 ) is greater than 1, with L measuring the degree of favoring. This Law has good Bayesian credentials. By Bayes s rule, the ratio of the posterior probability of H 1 given E to that of H 2 given E equals the product of the corresponding ratio of prior probabilities and the corresponding likelihood ratio: Pr(H 1 E)/ Pr(H 2 E) = Pr(E H / 1) Pr(H 1 ) Pr(E H2 ) Pr(H 2 ) Pr(E) Pr(E) = Pr(E H 1) Pr(H 1 ) Pr(E H 2 ) Pr(H 2 ) = L Pr(H 1) Pr(H 2 ) In this way, the Law of Likelihood quantifies the degree to which acquiring a body of data would change one s relative degrees of belief in a pair of hypotheses under Bayesian conditioning. Thus, Bayesians can accept the Law of Likelihood. However, they do not have to do so: a Bayesian could decline to adopt any account of evidential favoring at all, for instance. Conversely, one can also accept the Law of Likelihood without accepting Bayesian conditioning. Many likelihoodists do so in a limited way, by restricting the use of Bayesian conditioning to cases in which an objectively well-grounded prior probability distribution is available. They have arguments for accepting the Law of Likelihood even outside of those cases (Royall, 1997; Gandenberger, MS). The main advantage that the likelihoodist practice of reporting facts about evidential favoring in accordance with the Law of Likelihood has over using Bayesian conditioning is that the likelihood ratio Pr(E H 1 )/ Pr(E H 2 ) is often objectively well-grounded in cases in science in which a prior probability distribution over H 1 and H 2 is not. For instance, Pr(11 of 11 patients receiving ECMO survive H p ) is objectively well-defined for any p, because the corresponding hypothesis H p entails a probability distribution over outcomes of the experiment in the ECMO group. 5 One can show, for instance, that the Law of Likelihood says that the data from the randomized trial favors H 80% over H 20% to the degree 4, 194, 304. This degree of favoring is enormous: likelihoodists often take a likelihood ratio of 8 to indicate fairly strong favoring and a likelihood ratio of 32 to indicate strong favoring (Royall, 1997). 6 It might seem reasonable to conclude from this result that ECMO is superior to ventilator therapy in the relevant patient population, at least in terms of survival rates. However, the Law of Likelihood does not provide this conclusion, for two main reasons. First, even if we knew that the probability of survival under ventilator therapy was 20%, for instance, the relevant comparison would not be between the simple hypotheses H 80% and H 20%, but between the composite hypotheses p [20%,100%] H p and p [0%,20%) H p, i.e. the hypotheses that the chance of survival for a given patient receiving ECMO is at least 20% and the hypothesis that it is less than 20%. 7 Unfortunately, composite hypotheses do not entail definite, objective probability distributions over possible observations. 5. The claim that Pr(11 of 11 patients receiving ECMO survive H p ) is objectively well-defined for any p is a slight oversimplification, in two respects. First, H p entails entails a probability distribution over outcomes in the ECMO group only given the experimental design (for instance, the rule that determines how many patients will be treated). Second, <the conditional probability given H p that x of n 11 patients receiving ECMO survive> is conceptually distinct from <the probability entailed by H p that x of n patients receiving ECMO survive>, as noted by (Fitelson, 2007, 475). These quantities can come apart, for instance when the conditional probability reflects inadmissible information about the outcome of the experiment (Lewis, 1981). However, it seems that they should coincide in typical cases. For the purposes of this paper, I will leave aside the anomalous cases in which they do not. 6. These thresholds are admittedly somewhat arbitrary. They are somewhat analogous to the p =.05 threshold that frequentists often use to decide whether or not to declare a result statistically significant. 7. A hypothesis is called statistically simple with respect to a sample space if it entails a particular probability distribution over that sample space. It is called statistically composite if it entails a disjunction of multiple simple statistical hypotheses. philosophers imprint vol. 16, no. 7 (may 2016)

6 Pr(11 of 11 patients receiving ECMO survive p [20%,100%] H p ), for instance, depends on the relative probabilities of the various possibilities for p within the interval [20%, 100%]. Thus, the Law of Likelihood does not apply to composite hypotheses in the straightforward, objective way in which it applies to simple hypotheses such as H 80% and H 20% (see Royall, 1997, Ch. 7). Second, even regarding simple hypotheses, the Law of Likelihood only addresses questions about evidential favoring and not questions about what to believe or do. For instance, it can tell you that a given body of data favors H 80% over H 20% to a given degree, but it does not thereby tell you that you should believe H 80% over H 20% or act on the assumption that the former rather than the latter is true. Likelihoodists such as Royall and Sober are quite clear on this point: they insist that questions about what the evidence say are distinct from questions about what to believe or do and that the Law of Likelihood directly addresses only the former (Royall, 1997, 4; Sober, 2008, 3). One might wonder how they could provide a viable alternative to frequentism and Bayesianism for statistical reform efforts in science, given this limitation. In subsequent sections, I argue that they cannot. 3. A Dilemma for Likelihoodism Likelihoodists claim that one should refrain from addressing questions about belief or action when objectively well-grounded prior probabilities are not available. They take different views about what is required for a prior probability distribution to be objectively well-grounded. Some maintain that a prior probability distribution over a set of hypotheses can be objectively well-grounded only if it comes from a model of the chance set-up by which those hypotheses obtained their truth values (Edwards, 1972, 52 3). For instance, the probability that a ball drawn from an urn is a particular color could be objectively well-grounded if it comes from a model of the drawing process and the composition of the urn. Others adopt agnostic (Royall, 1997, 13) or more liberal views (Sober, 2008, 32). No likelihoodist believes that a prior probability distribution is objectively well-grounded simply by virtue of conformity to general constraints on rational degrees of belief, such as those that objective Bayesians provide. Whatever view one takes about which prior probability distributions are objectively well-grounded, the claim that we should not use distributions that do not meet this standard in science might sound quite reasonable. However, I will argue that, for likelihoodists, it leads to a dilemma. If a prior probability distribution over a set of hypotheses is objectively well-grounded only if it comes from a model of the chance set-up by which those hypotheses obtained their truth values, as Edwards mantains, then objectively well-grounded prior probability distributions are very rare in science. In physics, for instance, it is possible that fundamental laws and constants were generated by a chance process that created our universe. However, we do not know that they were so generated, nor do we know what their objective chances would be if they had been so generated. Thus, it does not seem possible to use a model of a chance set-up to assign prior probabilities to hypotheses about fundamental laws and constants. In general, it is very rare that we know enough about the process by which a scientific hypothesis obtained its truth value to derive an objective chance for that hypothesis from a credible model of that process. As a result, on the view that prior probabilities are objectively well-grounded only if they are derived in this way, the claim that we should restrict our attention to likelihoodist assessments of evidence when objectively well-grounded prior probabilities are not available would mean the end of using science to guide our beliefs and actions for the foreseeable future in most domains. We would not be able to use science to guide policy decisions, and using public funds to support scientific research would be very difficult to justify. I take it that a methodological program that would have such dire consequences for science is not a viable candidate for statistical reform efforts. Thus, likelihoodists need to adopt a more liberal view about what is required for a prior probability distribution to be objectively well-grounded. However, it is not clear how they can do so philosophers imprint vol. 16, no. 7 (may 2016)

7 without lapsing into Bayesianism. A likelihoodist might be willing to allow prior probabilities to be based on physical symmetries in cases in which the relevant symmetries are known to exist (for instance, in cases involving devices such as dice, coins, or roulette wheels), but again it is rarely possible to use such appeals to justify assignments of prior probabilities to scientific hypotheses. They could try to derive prior probabilities for hypotheses from features of those hypotheses that are available to inspection without reference to any actual data. For instance, they could choose the prior probability distribution that maximizes the expected value of a measure of the distance between the prior and posterior probability distributions for a given experiment, on the thought that the posterior probability distribution should depend on the data as much as possible. Alternatively, they could appeal to cardinality or topological considerations or to other information-theoretic criteria. However, deriving prior probabilities from considerations of this kind is what objective Bayesians do, and likelihoodists reject their approaches as excessively aprioristic (e.g. Edwards, 1972, Ch. 4; Royall, 1997, Section 8.6; and Sober, 2008, 27). It seems doubtful that objective Bayesians have missed an approach of this kind that likelihoodists would regard as acceptable. Alternatively, likelihoodists could try to derive prior probabilities for a given application from evidence obtained in previous studies. Some likelihoodists seem to endorse this approach; Sober, for instance, says only that prior probability distributions must be empirically defensible (2008, 32). However, at present we have no account of how prior probability distributions can be derived from past evidence other than the Bayesian account that requires providing a further prior probability distribution. This Bayesian account must ultimately ground out in a prior probability distribution that is not based on evidence, which is exactly what a likelihoodist wishes to avoid. 8 Likelihoodists 8. One could try to mitigate this problem by appealing to sets of initial probability distributions, but there is still the problem of choosing non-trivial initial sets. Moreover, this approach would not distinguish likelihoodism from imprecise Bayesianism, to which it is supposed to be an alternative. who claim that prior probabilities can be objectively well-grounded by virtue of being derived from evidence obtained in previous studies owe us an alternative, non-bayesian account of how such derivations are supposed to work. It is not obvious that a satisfactory account of this kind is possible. Thus, likelihoodists face a dilemma. Given a strict view of what is required for a prior probability distribution to be objectively wellgrounded, their account would deprive science of almost all practical value, because it would entail that we cannot use data to decide what to believe or do in the kinds of cases that we typically encounter in science. A more liberal view is needed. However, it is not clear that there is a more liberal view that yields a well-functioning theory without encountering the same objections that likelihoodists raise against Bayesian approaches. Likelihoodists could try to escape this dilemma by providing a way to use likelihoodist assessments of data as evidence to guide beliefs or actions without appealing to prior probability distributions at all. There are two approaches to doing so that they could take. First, they could develop methods based on likelihoodist assessments of data as evidence that provide some kind of assurance of good performance in repeated applications. In fact, likelihoodists have developed some seemingly favorable results concerning the performance of their procedures (e.g. Royall, 2000). However, the idea that one can justify a particular conclusion or inference by appealing to operating characteristics of a repeatable procedure that generated it is a frequentist notion that likelihoodists reject. One of the key points of distinction between frequentists on the one hand and likelihoodists and Bayesians on the other hand is that the latter insist that questions about what conclusion or action data warrant in a particular case are logically separate from questions about the performance of a method in repeated applications (Hacking, 1965, ). A second way to escape the dilemma would be to provide a rule of belief based on likelihoodist characterizations of data as evidence alone, or a rule of action based on those characterizations together philosophers imprint vol. 16, no. 7 (may 2016)

8 with considerations of value. It might be possible to provide a rule that somehow incorporates other considerations as well, without being effectively equivalent to an objective Bayesian approach, but it is far from obvious how to do so. The burden is on likelihoodists to provide such a rule and to show that it functions properly. In the absence of any plausible proposals, trying to provide a purely likelihood-based rule of belief or purely likelihood- and value-based rule of action seems to be the most promising way for a likelihoodist to proceed. In the next section, I argue that any rule of belief or action based on likelihoodist characterizations of data as evidence alone would have to satisfy one of two constraints, depending on whether it is a rule of belief or a rule of action. In the subsequent three sections, I show that these constraints are sufficient to generate problems that I take to be fatal to the associated rules. In the absence of any alternative way to derive guidance for belief and action from likelihoodist characterizations of data as evidence in the usual case in science in which prior probabilities based on known objective chances are not available, I conclude that likelihoodism is not a viable alternative to frequentism and Bayesianism for statistical reform efforts in science. Two possible objections are worth addressing here. First, one could claim that if science requires guidance for belief or action when objectively well-grounded prior probabilities in a strict sense are not available, then so much the worse for science. We simply do not have a good rational basis for providing such guidance under those circumstances. One problem with this objection is that we use scientific data to guide our beliefs and actions in the absence of objectively wellgrounded prior probabilities all the time, often with very good results. For instance, the massive improvements in medical care that occurred during the twentieth century would not have been possible if physicians had refused to change their practices in light of new research until appropriate prior probabilities had been derived from models of the chance set-ups that gave the relevant hypotheses their truth values. In addition, given that scientific inferences from data are not qualitatively different from everyday inferences from experience, and that we rarely have objectively well-grounded prior probability distributions for the latter, this approach would lead to a rather severe and debilitating form of skepticism. It would be difficult indeed to convince working scientists that statistical reform efforts should take this form! At least for purposes of practical scientific inquiry, giving up on providing guidance for belief or action is simply a non-starter. A second possible objection to my argument so far is that one could claim that while science does require more than just assessments of data as evidence, it does not require guidance for belief or action. For instance, one might settle for guidance for accepting hypotheses in a sense that does not involve believing that they are true. According to van Fraassen (1980), for instance, acceptance of a theory does not require belief in that theory, but only a belief that the theory is empirically adequate and commitments to use the theory in certain ways. My argument can accommodate this idea. Van Fraassen s notion of acceptance still has a belief component (belief in empirical adequacy) and is still related to action, in terms of how one uses the theory to guide future inquiry. Thus, providing good guidance for acceptance would have to involve providing good guidance for belief or action. If likelihoodists cannot do the latter, as I argue, then it seems to follow that they cannot do the former either. It seems doubtful that there is some concept still weaker than acceptance, not involving either belief or action, such that providing guidance for the application of that concept is sufficient for scientific purposes. 4. Getting Guidance for Beliefs or Actions from Evidence Alone I argued in the previous section that likelihoodists face a dilemma. If they adopt a strict view about what is required for a prior probability distribution to be objectively well-grounded, then they deprive science of nearly all of its practical value. If they adopt a more liberal view, then they are in danger of collapsing the distinction between likelihoodism and Bayesianism. One way to escape from this dilemma would be to provide a rule of belief based only on likelihoodist characterizations of data as evidence, or a rule of action based only on those philosophers imprint vol. 16, no. 7 (may 2016)

9 characterizations together with considerations of value. In this way, a likelihoodist could retain a strict view about what is required for a prior probability distribution to be objectively well-grounded without depriving science of practical value. In this section, I argue that such a rule would have to satisfy one of two principles, depending on whether it is a rule of belief or a rule of action. The proposal that beliefs should be based on evidence alone has some plausibility. It accords with the intuitively appealing evidentialist view that the epistemic justification of one s doxastic states depends only on one s evidence (Conee and Feldman, 2004). The more specific claim that one should proportion one s beliefs to one s evidence is sometimes stated as if it were a truism, or at least a promising starting point for an epistemological theory. 9 Perhaps likelihoodists could adapt this idea to their approach by saying that one should proportion one s relative beliefs (that is, one s beliefs in one hypothesis over another) to one s evidence, at least when objectively well-grounded prior probabilities are not available. This proposal might seem to be an obvious non-starter, given cases like the following (from Sober, 2008, 10). Suppose you hear a banging sound from your attic. Let us say that plumbing problems in the attic would not usually produce such a sound, but a gremlin in the attic would. Thus, Pr(sound gremlins)/ Pr(sound plumbing problems) 1, so the Law of Likelihood says that the sound favors the hypothesis of gremlins in the attic over the hypothesis of plumbing problems in the attic to a high degree. Nevertheless, the posterior probability of plumbing problems is presumably much higher than that of gremlins, simply because plumbing problems are (at least!) much more common than gremlins. 9. For instance, Hume famously claims that a wise man proportions his belief to his evidence (EHU 10.1/SBN 87). Tal and Comesaña echo this claim, asserting that rational subjects proportion their beliefs to their evidence (2015, 1). Similarly White suggests as a first stab that one s evidence determines what it is epistemically permissible for one to believe (2005, 445). However, this example simply exploits the fact that the evidential import of one piece of evidence can be quite different from the evidential import of a total body of evidence that contains it. Presumably, your total body of evidence contains many credible experiences or reports of plumbing problems but none of gremlins. Thus, even if the sound from the attic favors the gremlin hypothesis over the plumbing-problems hypothesis, your total evidence as a whole points quite strongly in the opposite direction. As a result, proportioning your relative beliefs to your total evidence would yield the sensible result of believing the plumbing-problems hypothesis over the gremlin hypothesis. Thus, this example is not enough to show that proportioning one s relative beliefs to one s total evidence in accordance with the Law of Likelihood will yield bad outcomes. However, I will show in the next sections that proportioning one s beliefs to one s total evidence can lead to paradoxes, at least on a likelihoodist account of evidence. In fact, the paradoxes arise from a principle much weaker than the claim that one should proportion one s beliefs to one s evidence. I call this principle minimal comparative proportionalism (MCP). This principle says that there is a real number d > 1 such that for any pair of hypotheses H 1 and H 2, a rational agent believes H 1 over H 2, either in an absolute sense or at least to some degree, if its total evidence favors H 1 over H 2 to degree d or greater. In other words, there is some degree of evidence that, for any pair of hypothesis, suffices for (some degree of) relative belief. When evidential favoring is understood in accordance with the Law of Likelihood, MCP is equivalent to the claim that there is some d > 1 such that a rational agent believes H 1 over H 2, at least to some degree, if the likelihood ratio of H 1 to H 2 on its total evidence exceeds d (i.e., Pr(T H 1 )/ Pr(T H 2 ) > d). Following Chandler (2013), I have argued elsewhere that the notion of evidential favoring that the Law of Likelihood explicates applies only to mutually exclusive hypotheses (Gandenberger, MS), so MCP understood in accordance with the Law of Likelihood applies only to mutually exclusive hypotheses. I do not presuppose any substantial account of relative belief. philosophers imprint vol. 16, no. 7 (may 2016)

10 One natural approach is to interpret a person s belief in H 1 over H 2 probabilistically, as Pr(H 1 )/ Pr(H 2 ) understood as a ratio of that person s degrees of belief. On such an interpretation, one could implement MCP by setting the posterior probability ratio for H 1 and H 2 given one s total evidence T equal to the associated likelihood ratio: Pr(H 1 T)/ Pr(H 2 T) = Pr(T H 1 )/ Pr(T H 2 ). However, MCP permits many other approaches. In fact, it allows Pr(H 1 T)/ Pr(H 2 T) to be any function of Pr(T H 1 )/ Pr(T H 2 ) whatsoever, as long as there is a threshold d > 1 such that Pr(H 1 T)/ Pr(H 2 T) > 1 whenever Pr(T H 1 )/ Pr(T H 2 ) > d. Moreover, it does not require that relative belief be understood probabilistically, or even as coming in degrees. It simply requires that there be some threshold in the likelihood ratio on one s total evidence that suffices for some degree of relative belief, or relative belief simpliciter, for any pair of mutually exclusive propositions. I also do not assume that beliefs are related to actions in any particular way. One might wish to assume, for instance, that one believes H 1 over H 2 (to some degree) if and only if one would choose a bet that yields a prize if H 1 is true and nothing otherwise over one that yields the same prize if H 2 is true and nothing otherwise, given various provisos such as that the value of the prize is the same in the two cases, there are no other significant consequences of choosing one bet rather than the other, and one s judgment is not impaired. However, such assumptions are contentious and difficult to specify adequately, and I do not need them. I will appeal only to pre-theoretically compelling constraints that any account recognizable as an account of rational relative belief would have to satisfy. It does not seem possible for an account of relative belief that violates MCP to do the job that likelihoodists need such an account to do. Such an account would have to allow that for any constant c, there are some pairs of hypotheses such that the threshold for (some degree of) relative belief in those hypotheses is greater than c. (MCP is not violated if there are different thresholds for different pairs of hypotheses but they are all below some common threshold.) The problem with this proposal is that likelihoodists seem to have no principled way to pick out particular pairs of hypotheses to receive this special treatment. They cannot pick them out on the basis of evidence, because all of the evidence is already reflected in the likelihood ratio. They cannot pick them out on the basis of the kinds of considerations to which objective Bayesians appeal, without running afoul of their own criticisms of objective Bayesianism. There does not seem to be any other rational basis on which to pick them out. Thus, if MCP fails, then the connection likelihoodists seem to need between evidence and relative belief does not exist. MCP is a principle of belief, but I have said that likelihoodists need to provide only a principle of belief or action. Perhaps they could say that in a case in which H 1 and H 2 are the only epistemic possibilities, one should proceed as if H 1 rather than H 2 were true for the purposes of a particular decision if and only if the likelihood ratio of H 1 against H 2 on one s total evidence exceeds some d (not necessarily greater than one) that varies only with the (dis)utility of proceeding as if H 1 (or H 2 ) were true if in fact H 1 (or H 2 ) were true. Call this principle minimal decision proportionalism (MDP). It seems to be a minimal commitment for a likelihoodist who wishes to give guidance for action, as MCP is a minimal commitment for a likelihoodist who wishes to give guidance for belief. A nontrivial account of the relationship between evidential favoring and action that did not satisfy MDP would have to make decisions depend on more than just one s total evidence and the relevant utilities, but there does not seem to be anything else to which likelihoodists can appeal, given their rejection of Bayesian approaches. Likelihoodism appears to be an attractive candidate for statistical reform in science because it combines the objectivity of the frequentist approach with the conformity to the Likelihood Principle of the Bayesian approach. However, I have argued that it is a viable alternative to frequentist and Bayesian approaches only if MCP or MDP is a good principle. At the least, the burden is on the likelihoodist who rejects MCP and MDP either to propose an alternative approach to using likelihoodist characterizations of evidence to guide beliefs or actions or to find another way to escape the dilemma presented in the previous philosophers imprint vol. 16, no. 7 (may 2016)

11 section. In the next three sections, I argue that MCP and MDP are not good principles, and thus that likelihoodism is not a viable alternative to frequentism and Bayesianism in science. 5. Problem 1 for MCP and MDP The Law of Likelihood says that an experimental outcome favors one hypothesis over another if the first hypothesis ascribes a higher probability to that outcome than the second. As many commenters have pointed out (e.g. Barnard, 1972, 129), it follows that the outcome of an experiment always favors the maximally likely hypothesis that the experiment was bound to produce the outcome it actually produced over any hypothesis that makes the outcome of the experiment a matter of chance. (That a hypothesis is maximally likely does not mean that it is more probable than any other hypothesis, but rather that it makes the data more probable than any other hypothesis.) Some commentators claim that this fact is a problem for the Law of Likelihood itself (e.g. Birnbaum, 1969, 127 8), while others defend it against this objection (e.g. Royall, 1997, 13 5). I argue that it is a problem for MCP and MDP even if it is not a problem for the Law of Likelihood. Consider the following example. Example 2. Suppose you observe ten radioactive isotopes labeled i 1 to i 10 of a particular species for a period equal to their half-life. You are sure that each of those isotopes decays or not independently of the others, but you are not sure about whether radioactive decay is deterministic. Thus, you consider the hypothesis H 50% that each of the ten isotopes has a 50% chance of decaying independently of the others and the 2 10 hypotheses each of which says that exactly some subset of the isotopes is bound to decay. You have no evidence about those hypotheses prior to the experiment. Suppose you observe E : only the isotopes i 2, i 3, i 4, i 6, and i 9 decay. How does this outcome bear on the deterministic hypothesis H that exactly those isotopes were bound to decay, against the indeterministic hypothesis H 50%? According to the Law of Likelihood, it favors the former over the latter to the substantial degree This result might look bad for the Law of Likelihood. After all, E does not seem to make H substantially more believable than H 50%. The standard likelihoodist response to this objection is that the Law of Likelihood does not address the question of which hypothesis is more believable in light of the data, but only the question of which hypothesis the data favor and by how much (Royall, 1997, 13 5). However, I have argued that likelihoodism is not a viable alternative to frequentism and Bayesianism for statistical reform efforts in science if it cannot provide guidance for belief or action. Thus, even if this response saves the Law of Likelihood as an account of evidential favoring, it does not save likelihoodism as a candidate for statistical reform. Now, MCPers (i.e., those who believe in accordance with MCP) are free to set the likelihood ratio threshold d that always suffices for believing one hypothesis over another as high as they like, including to values higher than But however high they set it, one can describe an experiment like the one described in Example 2 the result of which will inevitably favor some deterministic hypothesis over H 50% to some degree greater than d: an experiment like the one described in Example 2 will inevitably yield a result that favors the deterministic hypothesis that correctly predicts the data (call it H ) over H 50% to the degree 2 n, where n is the number of isotopes observed, so for a fixed n, one simply needs to increase n until 2 n > d. Thus, for any MCPer, one can describe an experiment like the one described in Example 2 (though possibly larger) the result of which will inevitably lead him or her to believe some deterministic hypothesis H over H 50%. The decision principle MDP is difficult to apply in this case. The relevant utilities are not obvious because whether one accepts H or H 50% need not make any difference to one s predictions about future radioactive decay events. As a result, it is not clear what the likelihood ratio 10. H gives E probability 1, while H 50% gives it probability 1/2 10, so Pr(E H )/ Pr(E H 50% )) = 1/(1/2 10 ) = philosophers imprint vol. 16, no. 7 (may 2016)

12 threshold for proceeding as if some deterministic hypothesis rather than H 50% were true should be. However, it would be hard to maintain that this threshold should grow at a rate at least 2 n as the number of isotopes n increases, because changing n seems to change the relevant utilities very little, if at all. As a result, it seems that for any MDPer, there must be a sample size sufficiently large that a radioactive decay experiment with that sample size would inevitably lead that MDPer to behave as if the maximally likely hypothesis H were true, rather than H 50%. To make matters worse, given background knowledge and the result of the experiment, H and H 50% are respectively equivalent to the hypotheses H d that the experiment is deterministic and the hypothesis H ind that it is indeterministic. Thus, assuming that relative belief is closed under known single-premise entailment conditional on one s evidence and background knowledge, it follows that the experiment will inevitably lead an MCPer who knows that H is equivalent to H d and H 50% to H ind to believe (to some degree) H d over H ind. However, the result of the experiment seems completely irrelevant to the issue of determinism. 11 To make matters worse still, because an MCPer will inevitably be led to believe (to some degree) H d over H ind, it seems that he or she should not have to wait for the outcome of the experiment at 11. For a Bayesian, the result of the experiment is irrelevant to the issue of determinism if all of the deterministic hypotheses have equal prior probabilities. If some are more probable than others, then a result that agrees with one of the more probable deterministic hypotheses will raise the overall probability of determinism, while a result that agrees with one of the less probable deterministic hypotheses will lower the probability of determinism. The intuition that the result is completely irrelevant plausibly arises from the fact that one has no reason to prefer one deterministic hypothesis over another prior to the experiment. In any case, a Bayesian would not approve of believing H d over H ind regardless of the experimental outcome if one is neutral between H d and H ind before the experiment. all in accordance with an analogue of van Fraassen s reflection principle (1995), 12 he or she should simply believe (to some degree) determinism over indeterminism from the beginning. Analogous statements apply to MDPers. It seems that these plainly mistaken conclusions follow inexorably from MCP and MDP, given the Law of Likelihood, so that likelihoodists must reject those principles. They are then left with no way to provide guidance for belief or action. Again, they could maintain that guidance for belief or action is inappropriate when objective prior probabilities are not available, but I have argued that this position is untenable because it would make science practically useless in most cases. For this reason, likelihoodism is not a viable candidate for statistical reform. Bayesians address the problem of maximally likely hypotheses through prior probabilities. For instance, suppose that a Bayesian assigned some prior probability 0 < p < 1 to H 50% and equal probabilities (1 p)/2 n to the 2 n deterministic hypotheses. Then the posterior probability of H 50% would remain p, those of the 2 n 1 refuted deterministic hypotheses would be 0, and that of the single unrefuted deterministic hypothesis would be 1 p. Thus, the probability of the unrefuted deterministic hypothesis would increase by a factor of 2 n, but it would not rise above 1 p. As a result, the probabilities of H d and H ind would not change. Thus, for a Bayesian, the prior probability of each deterministic hypothesis can decrease with the sample size in a way that exactly compensates for the increase in the likelihood ratio. This way of dealing with maximally likely hypotheses is not available to likelihoodists, of course, because they forego the use of prior probabilities in this problem. A likelihoodist might attempt to avoid the problem of maximally likely hypotheses in either of two ways. First, he or she might put some 12. Van Fraassen s General Reflection Principle says that for any future time t, one s current opinion about a hypothesis must lie in the span of what one currently regards as the opinions that one might hold about that hypothesis at t. philosophers imprint vol. 16, no. 7 (may 2016)