Divergence Based priors for the problem of hypothesis testing

Transcription

1 Divergence Based priors for the problem of hypothesis testing gonzalo garcía-donato and susie Bayarri May 22, 2009 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

2 Jeffreys and testing: the essentials Much of the material presented in this talk is based on the work of Jeffreys. gonzalo garcı a-donato and susie Bayarri () DB priors May 22, / 46

3 Jeffreys and testing: the essentials Much of the material presented in this talk is based on the work of Jeffreys. Sir Harold Jeffreys born in England in 1891 and died in 1989 (almost at the age of hundred). He was Professor of Astronomy at Cambridge. gonzalo garcı a-donato and susie Bayarri () DB priors May 22, / 46

4 Jeffreys and testing: the essentials Much of the material presented in this talk is based on the work of Jeffreys. Sir Harold Jeffreys born in England in 1891 and died in 1989 (almost at the age of hundred). He was Professor of Astronomy at Cambridge. His contributions to statistics through a Bayesian thinking are very popular (eg. Jeffreys prior) and are the basis of many important methods. gonzalo garcı a-donato and susie Bayarri () DB priors May 22, / 46

5 Jeffreys and testing: the essentials We focus here on his ideas on testing gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

6 Jeffreys and testing: the essentials We focus here on his ideas on testing specially on the solutions he provided to testing problem, but also and very importantly, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

7 Jeffreys and testing: the essentials We focus here on his ideas on testing specially on the solutions he provided to testing problem, but also and very importantly, on his particular perception of the problem. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

8 Jeffreys and testing: the essentials We focus here on his ideas on testing specially on the solutions he provided to testing problem, but also and very importantly, on his particular perception of the problem. This has motivated the design of this presentation: PART I. Looking at testing problem through Jeffreys eyes, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

9 Jeffreys and testing: the essentials We focus here on his ideas on testing specially on the solutions he provided to testing problem, but also and very importantly, on his particular perception of the problem. This has motivated the design of this presentation: PART I. Looking at testing problem through Jeffreys eyes, PART II. Priors for testing, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

10 Jeffreys and testing: the essentials We focus here on his ideas on testing specially on the solutions he provided to testing problem, but also and very importantly, on his particular perception of the problem. This has motivated the design of this presentation: PART I. Looking at testing problem through Jeffreys eyes, PART II. Priors for testing, PART III. Generalizations of Jeffreys ideas, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

11 Jeffreys and testing: the essentials We focus here on his ideas on testing specially on the solutions he provided to testing problem, but also and very importantly, on his particular perception of the problem. This has motivated the design of this presentation: PART I. Looking at testing problem through Jeffreys eyes, PART II. Priors for testing, PART III. Generalizations of Jeffreys ideas, PART IV. Divergence based priors. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

12 PART I. Looking at testing problem through Jeffreys eyes PART I. Looking at testing problem through Jeffreys eyes gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

13 PART I. Looking at testing problem through Jeffreys eyes Testing a point null hypothesis Let y = (y 1, y 2,..., y n ) be a random sample from f (y θ, ν). We are interested in testing the hypotheses H 1 : θ = θ 0 (null), vs H 2 : θ θ 0 (alternative). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

14 PART I. Looking at testing problem through Jeffreys eyes Testing a point null hypothesis Let y = (y 1, y 2,..., y n ) be a random sample from f (y θ, ν). We are interested in testing the hypotheses H 1 : θ = θ 0 (null), vs H 2 : θ θ 0 (alternative). This is equivalent to selecting between the competing models M 1 : f 1 (y ν) = f (y θ 0, ν) vs M 2 : f 2 (y θ, ν ) = f (y θ, ν ). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

15 PART I. Looking at testing problem through Jeffreys eyes Testing a point null hypothesis Let y = (y 1, y 2,..., y n ) be a random sample from f (y θ, ν). We are interested in testing the hypotheses H 1 : θ = θ 0 (null), vs H 2 : θ θ 0 (alternative). This is equivalent to selecting between the competing models M 1 : f 1 (y ν) = f (y θ 0, ν) vs M 2 : f 2 (y θ, ν ) = f (y θ, ν ). Jeffreys (1961) calls θ the new parameter; gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

16 PART I. Looking at testing problem through Jeffreys eyes Testing a point null hypothesis Let y = (y 1, y 2,..., y n ) be a random sample from f (y θ, ν). We are interested in testing the hypotheses H 1 : θ = θ 0 (null), vs H 2 : θ θ 0 (alternative). This is equivalent to selecting between the competing models M 1 : f 1 (y ν) = f (y θ 0, ν) vs M 2 : f 2 (y θ, ν ) = f (y θ, ν ). Jeffreys (1961) calls θ the new parameter; ν, ν are the old (or common) parameters. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

17 PART I. Looking at testing problem through Jeffreys eyes Initial considerations Due to the statement in H 1, this problem has received, in the literature, the name of punctual, precise, sharp hypothesis testing or significance tests (Jeffreys 1961). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

18 PART I. Looking at testing problem through Jeffreys eyes Initial considerations Due to the statement in H 1, this problem has received, in the literature, the name of punctual, precise, sharp hypothesis testing or significance tests (Jeffreys 1961). A precise hypothesis is (in many situations) a convenient approximation of the more realistic hypothesis H : θ = θ 0 H : θ θ 0 < ɛ, ɛ <<. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

19 PART I. Looking at testing problem through Jeffreys eyes Initial considerations Due to the statement in H 1, this problem has received, in the literature, the name of punctual, precise, sharp hypothesis testing or significance tests (Jeffreys 1961). A precise hypothesis is (in many situations) a convenient approximation of the more realistic hypothesis H : θ = θ 0 H : θ θ 0 < ɛ, ɛ <<. See Berger and Delampady (1987) for a quantification of this correspondence. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

20 PART I. Looking at testing problem through Jeffreys eyes Initial considerations Due to the statement in H 1, this problem has received, in the literature, the name of punctual, precise, sharp hypothesis testing or significance tests (Jeffreys 1961). A precise hypothesis is (in many situations) a convenient approximation of the more realistic hypothesis H : θ = θ 0 H : θ θ 0 < ɛ, ɛ <<. See Berger and Delampady (1987) for a quantification of this correspondence. Common parameters ν in M 1 and ν in M 2 do not (necessarily) identify the same magnitude. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

21 PART I. Looking at testing problem through Jeffreys eyes Testing precise hypothesis is not an estimation problem gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

22 PART I. Looking at testing problem through Jeffreys eyes Testing precise hypothesis is not an estimation problem Many authors, including Jeffreys, have repeatedly argued that estimation and testing problem are statistical problems of different nature. Jeffreys, in his book Theory of probability right at the beginning of the chapter devoted to testing problems says gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

23 PART I. Looking at testing problem through Jeffreys eyes Testing precise hypothesis is not an estimation problem Many authors, including Jeffreys, have repeatedly argued that estimation and testing problem are statistical problems of different nature. Jeffreys, in his book Theory of probability right at the beginning of the chapter devoted to testing problems says In the last chapters we were concerned with the estimation of the parameters in a law, the form of the law itself being given. We are now concerned with the more difficult question: in what circumstances do observations support a change of the form of the law itself? gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

24 PART I. Looking at testing problem through Jeffreys eyes (genuine) Testing Testing (genuine) problems The proposal θ = θ 0, is (a priori) plausible and represents, in a broad sense, a theory that needs to be tested. As opposed to gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

25 PART I. Looking at testing problem through Jeffreys eyes (genuine) Testing versus estimation Testing (genuine) problems The proposal θ = θ 0, is (a priori) plausible and represents, in a broad sense, a theory that needs to be tested. As opposed to Estimation problems There is no uncertainty about what is the right model and there is no a special value of θ. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

26 PART I. Looking at testing problem through Jeffreys eyes Different problems According to these ideas, it is not that surprising that the tools used for estimation problems do not work for testing problems: gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

27 PART I. Looking at testing problem through Jeffreys eyes Different problems According to these ideas, it is not that surprising that the tools used for estimation problems do not work for testing problems: Improper priors produce arbitrary answers, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

28 PART I. Looking at testing problem through Jeffreys eyes Different problems According to these ideas, it is not that surprising that the tools used for estimation problems do not work for testing problems: Improper priors produce arbitrary answers, same (or worst) about proper flat priors, Gamma( , ) gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

29 PART I. Looking at testing problem through Jeffreys eyes Different problems According to these ideas, it is not that surprising that the tools used for estimation problems do not work for testing problems: Improper priors produce arbitrary answers, same (or worst) about proper flat priors, Gamma( , ) Credible intervals to solve hypothesis testing [Example]. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

30 PART I. Looking at testing problem through Jeffreys eyes Different problems According to these ideas, it is not that surprising that the tools used for estimation problems do not work for testing problems: Improper priors produce arbitrary answers, same (or worst) about proper flat priors, Gamma( , ) Credible intervals to solve hypothesis testing [Example]. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

31 PART I. Looking at testing problem through Jeffreys eyes Jeffreys tools Jeffreys based the solution of the problem on what we call now Bayes factors: M 1 : f 1 (y ν) vs M 2 : f 2 (y θ, ν ), B 12 = m 1(y) m 2 (y), where, the prior marginals are m 1 (y) = f 1 (y ν)π 1 (ν)dν, m 2 (y) = f 2 (y θ, ν )π 2 (θ, ν )dν dθ. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

32 PART I. Looking at testing problem through Jeffreys eyes Other approaches There are many other Bayesian measures used in the problem of hypothesis testing, like the BIC, DIC, etc. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

33 PART I. Looking at testing problem through Jeffreys eyes Other approaches There are many other Bayesian measures used in the problem of hypothesis testing, like the BIC, DIC, etc. Principle (Berger y Pericchi 2001) Testing and model selection methods should correspond, in some sense, to actual Bayes factors, arising from reasonable default prior distributions. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

34 PART I. Looking at testing problem through Jeffreys eyes Other approaches There are many other Bayesian measures used in the problem of hypothesis testing, like the BIC, DIC, etc. Principle (Berger y Pericchi 2001) Testing and model selection methods should correspond, in some sense, to actual Bayes factors, arising from reasonable default prior distributions.? The question of what priors should be used is an open question, of which only partial answers are known. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

35 PART II. Objective testing priors PART II Objective testing priors gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

36 PART II. Objective testing priors The normal scenario Y i (iid)n(µ, σ 2 ), i = 1, 2,..., n, σ unknown. H 1 : µ = 0 (M 1 : N(0, σ 2 )), H 2 : µ 0 (M 2 : N(µ, τ 2 )). B 12 = p1 (y σ)π 1 (σ)dσ p2 (y µ, τ)π 2.1 (τ)π(µ τ)dτdµ. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

37 PART II. Objective testing priors Jeffreys proposal π 1 (σ) = h(σ), π 2 (µ, τ) = h(τ)π(µ τ). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

38 PART II. Objective testing priors Jeffreys proposal π 1 (σ) = h(σ), π 2 (µ, τ) = h(τ)π(µ τ). Arguing, with respect to the common parameters µ and τ are orthogonal so τ and σ have the same meaning, justifying the assignment of the same prior, h in this case. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

39 PART II. Objective testing priors Jeffreys proposal π 1 (σ) = cσ 1, π 2 (µ, τ) = cτ 1 π(µ τ). Arguing, with respect to the common parameters µ and τ are orthogonal so τ and σ have the same meaning, justifying the assignment of the same prior, h in this case. Bayes factor is quite robust to the election of h, and hence we can use h(σ) = σ 1 (constants cancel out). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

40 PART II. Objective testing priors Jeffreys proposal π 1 (σ) = cσ 1, π 2 (µ, τ) = cτ 1 π(µ τ). With respect to the new parameter: π(µ τ) has to be proper,centered at zero (the null model),scaled by τ, symmetric around zero, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

41 PART II. Objective testing priors Jeffreys proposal π 1 (σ) = cσ 1, π 2 (µ, τ) = cτ 1 π(µ τ). With respect to the new parameter: π(µ τ) has to be proper,centered at zero (the null model),scaled by τ, symmetric around zero, with heavy tails. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

42 PART II. Objective testing priors Jeffreys proposal π 1 (σ) = cσ 1, π 2 (µ, τ) = cτ 1 Ca(µ 0, τ). With respect to the new parameter: π(µ τ) has to be proper,centered at zero (the null model),scaled by τ, symmetric around zero, with heavy tails. The simpler function with these properties is the Ca(0, τ). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

43 PART II. Objective testing priors Jeffreys proposal have resisted the passage of time. It has been used as a benchmark for other priors. Why? It provides reasonable responses. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

44 PART II. Objective testing priors Jeffreys proposal have resisted the passage of time. It has been used as a benchmark for other priors. Why? It provides reasonable responses. It is information consistent: as the information against the null becomes overwhelming the Bayes factor against the null tends to infinity. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

45 PART II. Objective testing priors Jeffreys proposal have resisted the passage of time. It has been used as a benchmark for other priors. Why? It provides reasonable responses. It is information consistent: as the information against the null becomes overwhelming the Bayes factor against the null tends to infinity. Surprisingly the Ca(0, τ) has arisen, approximately, using other modern methodologies: the intrinsic priors (Berger and Pericchi, 1996) and the expected posterior priors (Pérez and Berger, 2002). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

46 PART II. Objective testing priors Diving a little bit deeper: a key slide gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

47 PART II. Objective testing priors Diving a little bit deeper: a key slide Implicitly recognizes the null model as a source of objective information: M 0 gives the center and the scale. This should be interpreted as of unitary size, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

48 PART II. Objective testing priors Diving a little bit deeper: a key slide Implicitly recognizes the null model as a source of objective information: M 0 gives the center and the scale. This should be interpreted as of unitary size,...the mere fact that it it has been suggested that the parameter is zero corresponds to some presumption that it is fairly small. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

49 PART II. Objective testing priors Diving a little bit deeper: a key slide Implicitly recognizes the null model as a source of objective information: M 0 gives the center and the scale. This should be interpreted as of unitary size,...the mere fact that it it has been suggested that the parameter is zero corresponds to some presumption that it is fairly small. The importance of the tails (information consistency). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

50 PART III. Generalizations of Jeffreys ideas PART III Generalizations of Jeffreys ideas gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

51 PART III. Generalizations of Jeffreys ideas Jeffreys proposals are very specific to the problem at hand (a normal mean). They do not arise through a rule which can be reasonably applied to other testing scenarios. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

52 PART III. Generalizations of Jeffreys ideas Jeffreys proposals are very specific to the problem at hand (a normal mean). They do not arise through a rule which can be reasonably applied to other testing scenarios. Is it always a Cauchy a sensible prior? gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

53 PART III. Generalizations of Jeffreys ideas Jeffreys proposals are very specific to the problem at hand (a normal mean). They do not arise through a rule which can be reasonably applied to other testing scenarios. Is it always a Cauchy a sensible prior? What is a good scale? gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

54 PART III. Generalizations of Jeffreys ideas Jeffreys proposals are very specific to the problem at hand (a normal mean). They do not arise through a rule which can be reasonably applied to other testing scenarios. Is it always a Cauchy a sensible prior? What is a good scale? What about problems with multivariate parameters or problems with some design? gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

55 PART III. Generalizations of Jeffreys ideas Jeffreys proposals are very specific to the problem at hand (a normal mean). They do not arise through a rule which can be reasonably applied to other testing scenarios. Is it always a Cauchy a sensible prior? What is a good scale? What about problems with multivariate parameters or problems with some design? Although not too many, there are some interesting ideas in the literature about the extension of Jeffreys ideas. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

56 PART III. Generalizations of Jeffreys ideas Jeffreys proposals are very specific to the problem at hand (a normal mean). They do not arise through a rule which can be reasonably applied to other testing scenarios. Is it always a Cauchy a sensible prior? What is a good scale? What about problems with multivariate parameters or problems with some design? Although not too many, there are some interesting ideas in the literature about the extension of Jeffreys ideas. We start focusing on priors for new parameters. These ideas are the basis for the full problem with nuisance parameters (more on this at the end) gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

57 PART III. Generalizations of Jeffreys ideas Idea 1: Jeffreys extending Jeffreys gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

58 PART III. Generalizations of Jeffreys ideas Idea 1: Jeffreys extending Jeffreys An omnipresent quantity in Jeffreys work is the divergence: D = log f 1 d ( ) f 1 f 2. f 2 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

59 PART III. Generalizations of Jeffreys ideas Idea 1: Jeffreys extending Jeffreys An omnipresent quantity in Jeffreys work is the divergence: D = log f 1 d ( ) f 1 f 2. f 2 For the general problem: H 1 : y iid f (y θ 0 ),, H 2 : y iid f (y θ), θ R, Jeffreys proposed the rule gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

60 PART III. Generalizations of Jeffreys ideas Idea 1: Jeffreys extending Jeffreys An omnipresent quantity in Jeffreys work is the divergence: D = log f 1 d ( ) f 1 f 2. f 2 For the general problem: H 1 : y iid f (y θ 0 ),, H 2 : y iid f (y θ), θ R, Jeffreys proposed the rule π J (θ) = 1 π d dθ tan 1 (D[θ, θ 0 ]) 1/2, where D[θ, θ 0 ] is the divergence between f (y θ 0 ) and f (y θ). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

61 PART III. Generalizations of Jeffreys ideas Jeffreys extending Jeffreys (cont ) π J (θ) reduces to Jeffreys Cauchy proposal when θ is a normal mean. Noticeable, when θ θ 0 is small, π J (θ) 1 π ( 1 + D[θ, θ0 ] ) 1 π NJ (θ), where π NJ (θ) is Jeffreys (estimation) prior (i.e. the squared root of the expected Fisher information). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

62 PART III. Generalizations of Jeffreys ideas Jeffreys extending Jeffreys (cont ) π J (θ) reduces to Jeffreys Cauchy proposal when θ is a normal mean. Noticeable, when θ θ 0 is small, π J (θ) 1 π ( 1 + D[θ, θ0 ] ) 1 π NJ (θ), where π NJ (θ) is Jeffreys (estimation) prior (i.e. the squared root of the expected Fisher information). Unfortunately Note that π J can lead to improper priors and at least in principle can not be applied for multivariate parameters. However, the approximation was a main inspiration for the definition of DB priors, with clear similarities between them. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

63 PART III. Generalizations of Jeffreys ideas Idea 2: Zellner and Siow (1980, 1984) For the linear model y N(αX 1 + βx, σ 2 I), X t 1X = 0, with hypotheses they propose H 1 : β = 0, H 2 : β 0, β Ca k (0, nσ 2 (X t X) 1 ), the matrix (X t X) 1 suggested by the information matrix. This is a proposal followed by many others: Liang et al (2008), Bayarri and García-Donato (2007). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

64 PART III. Generalizations of Jeffreys ideas Idea 3: Kass and Wasserman (1995) Along these same lines, for a general multidimensional parameter β, Kass and Wasserman (1995) propose β Ca k (0, I β (α, β = 0) 1 ), where I β is the block of the Fisher information matrix corresponding to β (one observation). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

65 PART III. Generalizations of Jeffreys ideas Notice These generalizations focus on the Fisher information, a second order approximation of the Kullback-Leibler divergence. Jeffreys did a similar exercise. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

66 PART III. Generalizations of Jeffreys ideas Notice These generalizations focus on the Fisher information, a second order approximation of the Kullback-Leibler divergence. Jeffreys did a similar exercise. They are primarily concerned with location-type parameters. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

67 PART III. Generalizations of Jeffreys ideas Notice These generalizations focus on the Fisher information, a second order approximation of the Kullback-Leibler divergence. Jeffreys did a similar exercise. They are primarily concerned with location-type parameters. Our work retakes the essence of the Jeffreys ideas using directly divergence measures as a central quantity. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

68 PART IV. Divergence based priors PART IV Divergence based priors gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

69 PART IV. Divergence based priors Motivation: θ is a location parameter (in R) H 1 : θ = θ 0 vs. H 2 : θ θ 0 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

70 PART IV. Divergence based priors Motivation: θ is a location parameter (in R) H 1 : θ = θ 0 vs. H 2 : θ θ 0 M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), where f is a given model. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

71 PART IV. Divergence based priors Motivation: θ is a location parameter (in R) H 1 : θ = θ 0 vs. H 2 : θ θ 0 M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), where f is a given model. Let D[θ, θ 0 ] be a measure of unitary distance between M 1 and M 2. We prefer use the Kullback-Leibler divergence divided by n. n D[θ, θ 0 ] = log f (y θ 0) ( f (y θ) f (y θ0 ) ) dy. f (y θ) gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

72 PART IV. Divergence based priors Motivation: θ is a location parameter (in R) H 1 : θ = θ 0 vs. H 2 : θ θ 0 M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), where f is a given model. Let D[θ, θ 0 ] be a measure of unitary distance between M 1 and M 2. We prefer use the Kullback-Leibler divergence divided by n. n D[θ, θ 0 ] = log f (y θ 0) ( f (y θ) f (y θ0 ) ) dy. f (y θ) Let h q ( ) be a real valued decreasing function, possibly indexed by a real argument q > 0. Simple possibilities: h(x) = e q x, h q (x) = (1 + x) q, h q (x) = (1 + x/q) q. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

73 PART IV. Divergence based priors Definition gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

74 PART IV. Divergence based priors Definition Define (θ location) being q such that π D is proper. π D (θ) h q (D[θ, θ 0 ]), gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

75 PART IV. Divergence based priors Definition Define (θ location) being q such that π D is proper. π D (θ) h q (D[θ, θ 0 ]), Interpretation π D is a robust prior which uses M 1 as a guess for M 2 : those values of θ such that f 1 (y) and f 2 (y θ) are close, have the greatest probability. This idea goes a step beyond other proposals, under which the prior is (an inverse) function of (θ θ 0 ) 2 or I (θ 0 ) (θ θ 0 ) 2. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

76 PART IV. Divergence based priors Basic property π D is a unimodal density, with mode at θ 0, symmetric around θ 0. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

77 PART IV. Divergence based priors Testing a normal mean H 1 : µ = 0, revisited (σ known) We have D[µ, 0] = µ 2 /σ 2, and hence π D (µ) = Γ(q) Γ(q.5)σ ( µ 2 ) q, 1 + q > 0.5 π σ 2 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

78 PART IV. Divergence based priors Testing a normal mean H 1 : µ = 0, revisited (σ known) We have D[µ, 0] = µ 2 /σ 2, and hence π D (µ) = Γ(q) Γ(q.5)σ ( µ 2 ) q, 1 + q > 0.5 π σ 2 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

79 PART IV. Divergence based priors Testing a normal mean H 1 : µ = 0, revisited (σ known) We have D[µ, 0] = µ 2 /σ 2, and hence π D (µ) = Γ(q) Γ(q.5)σ ( µ 2 ) q, 1 + q > 0.5 π σ 2 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

80 PART IV. Divergence based priors Testing a normal mean H 1 : µ = 0, revisited (σ known) We have D[µ, 0] = µ 2 /σ 2, and hence π D (µ) = Γ(q) Γ(q.5)σ ( µ 2 ) q, 1 + q > 0.5 π σ 2 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

81 PART IV. Divergence based priors Testing a normal mean H 1 : µ = 0, revisited (σ known) We have D[µ, 0] = µ 2 /σ 2, and hence π D (µ) = Γ(q) Γ(q.5)σ ( µ 2 ) q, 1 + q > 0.5 π σ 2 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

82 PART IV. Divergence based priors Scale parameters: introducing the general rule The above proposal can be easily applied to scale parameters, σ: M 1 : f 1 (y) = f (y σ 0 ) vs. M 2 : f 2 (y σ) = f (y σ). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

83 PART IV. Divergence based priors Scale parameters: introducing the general rule The above proposal can be easily applied to scale parameters, σ: M 1 : f 1 (y) = f (y σ 0 ) vs. M 2 : f 2 (y σ) = f (y σ). Reparameterize in terms of θ = log σ, a location parameter, derive the DB prior π D (θ) h q (D [θ, log σ 0 ]), and transform it to the original parameterization. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

84 PART IV. Divergence based priors Scale parameters: introducing the general rule The above proposal can be easily applied to scale parameters, σ: M 1 : f 1 (y) = f (y σ 0 ) vs. M 2 : f 2 (y σ) = f (y σ). Reparameterize in terms of θ = log σ, a location parameter, derive the DB prior π D (θ) h q (D [θ, log σ 0 ]), and transform it to the original parameterization.one obtains π D (σ) h q (D [log σ, log σ 0 ]) 1 σ. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

85 PART IV. Divergence based priors Scale parameters: introducing the general rule The above proposal can be easily applied to scale parameters, σ: M 1 : f 1 (y) = f (y σ 0 ) vs. M 2 : f 2 (y σ) = f (y σ). Reparameterize in terms of θ = log σ, a location parameter, derive the DB prior π D (θ) h q (D [θ, log σ 0 ]), and transform it to the original parameterization.one obtains Note, if D is invariant π D (σ) h q (D [log σ, log σ 0 ]) 1 σ. π D (σ) h q (D[σ, σ 0 ])π N (σ). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

86 PART IV. Divergence based priors The general case (without nuisance par.): construction gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

87 PART IV. Divergence based priors The general case (without nuisance par.): construction M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), with θ a k-dimensional parameter. Let π N (θ) the reference prior. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

88 PART IV. Divergence based priors The general case (without nuisance par.): construction M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), with θ a k-dimensional parameter. Let π N (θ) the reference prior. Parameterize in terms of ξ = g(θ) such that π N (ξ) = 1. Remarkably ξ behaves, asymptotically, as location parameter (Bernardo 2005). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

89 PART IV. Divergence based priors The general case (without nuisance par.): construction M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), with θ a k-dimensional parameter. Let π N (θ) the reference prior. Parameterize in terms of ξ = g(θ) such that π N (ξ) = 1. Remarkably ξ behaves, asymptotically, as location parameter (Bernardo 2005). Apply the definition of DB priors for location parameters. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

90 PART IV. Divergence based priors The general case (without nuisance par.): construction M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), with θ a k-dimensional parameter. Let π N (θ) the reference prior. Parameterize in terms of ξ = g(θ) such that π N (ξ) = 1. Remarkably ξ behaves, asymptotically, as location parameter (Bernardo 2005). Apply the definition of DB priors for location parameters. Obtain the prior in the original parameterization (apply the transformation of variables formula). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

91 PART IV. Divergence based priors The general case (without nuisance par.): construction M 1 : f 1 (y) = f (y θ 0 ) vs. M 2 : f 2 (y θ) = f (y θ), with θ a k-dimensional parameter. Let π N (θ) the reference prior. Parameterize in terms of ξ = g(θ) such that π N (ξ) = 1. Remarkably ξ behaves, asymptotically, as location parameter (Bernardo 2005). Apply the definition of DB priors for location parameters. Obtain the prior in the original parameterization (apply the transformation of variables formula). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

92 PART IV. Divergence based priors General definition gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

93 PART IV. Divergence based priors General definition Of course, all these steps are not at all needed. By construction, we have arrived at the general definition: DB prior π D (θ) h q (D[θ, θ 0 ])π N (θ). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

94 PART IV. Divergence based priors Properties: The DB priors in words They capture the information contained into the problem, making use of the distance between the models. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

95 PART IV. Divergence based priors Properties: The DB priors in words They capture the information contained into the problem, making use of the distance between the models. The information contained is equivalent to the information in a sample of unitary size. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

96 PART IV. Divergence based priors Properties: The DB priors in words They capture the information contained into the problem, making use of the distance between the models. The information contained is equivalent to the information in a sample of unitary size. DB priors generalize the work of Jeffreys (1961) and Zellner and Siow (1980). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

97 PART IV. Divergence based priors Properties: More theoretically If θ is a location parameter then π D approximately behaves as a Student centered at θ 0 and scaled by the Fisher information matrix evaluated at θ 0. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

98 PART IV. Divergence based priors Properties: More theoretically If θ is a location parameter then π D approximately behaves as a Student centered at θ 0 and scaled by the Fisher information matrix evaluated at θ 0. π D is invariant under reparameterizations. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

99 PART IV. Divergence based priors Properties: More theoretically If θ is a location parameter then π D approximately behaves as a Student centered at θ 0 and scaled by the Fisher information matrix evaluated at θ 0. π D is invariant under reparameterizations. π D satisfies the sufficiency principle (e.g. do not change if the problem is reduced via sufficient statistics). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

100 PART IV. Divergence based priors Properties: More theoretically If θ is a location parameter then π D approximately behaves as a Student centered at θ 0 and scaled by the Fisher information matrix evaluated at θ 0. π D is invariant under reparameterizations. π D satisfies the sufficiency principle (e.g. do not change if the problem is reduced via sufficient statistics). For the example analyzed it is consistent in information gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

101 PART IV. Divergence based priors Properties: More theoretically If θ is a location parameter then π D approximately behaves as a Student centered at θ 0 and scaled by the Fisher information matrix evaluated at θ 0. π D is invariant under reparameterizations. π D satisfies the sufficiency principle (e.g. do not change if the problem is reduced via sufficient statistics). For the example analyzed it is consistent in information The (DB) Bayes factor can be expressed as B D 21 = B N 21 Correction factor (FC), where FC is a expectation with respect to the posterior distribution. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

102 PART IV. Divergence based priors The choice of q Clearly, π D depends on the election of q, a parameter which needs to be assigned. This parameter regulates the heaviness of the tails. Our proposal is q = q + 0.5, with q = inf{q > 0 : π D (θ) is proper. This reproduces the Jeffreys-Zellner-Siow priors in the normal scenario. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

103 PART IV. Divergence based priors DB priors in practice Leaving aside formal criteria... gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

104 PART IV. Divergence based priors DB priors in practice Leaving aside formal criteria... the best way of knowing if a proposal is reasonable is studying its behaviour in scenarios, representative of the practice of statistics. Scenario Standard type Improper likelihood Irregular models gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

105 PART IV. Divergence based priors DB priors in practice Leaving aside formal criteria... the best way of knowing if a proposal is reasonable is studying its behaviour in scenarios, representative of the practice of statistics. Scenario Standard type Improper likelihood Irregular models gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

106 PART IV. Divergence based priors The problem with nuisance parameters: some few words M 1 : f 1 (y ν 1 ) = f (y θ 0, ν) vs. M 2 : f 2 (y θ, ν ) = f (y θ, ν ), gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

107 PART IV. Divergence based priors The problem with nuisance parameters: some few words M 1 : f 1 (y ν 1 ) = f (y θ 0, ν) vs. M 2 : f 2 (y θ, ν ) = f (y θ, ν ), Assume that θ and ν are orthogonal. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

108 PART IV. Divergence based priors The problem with nuisance parameters: some few words M 1 : f 1 (y ν 1 ) = f (y θ 0, ν) vs. M 2 : f 2 (y θ, ν ) = f (y θ, ν ), Assume that θ and ν are orthogonal. Use π 1 (ν) = π1 N (ν) and, gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

109 PART IV. Divergence based priors The problem with nuisance parameters: some few words M 1 : f 1 (y ν 1 ) = f (y θ 0, ν) vs. M 2 : f 2 (y θ, ν ) = f (y θ, ν ), Assume that θ and ν are orthogonal. Use π 1 (ν) = π1 N (ν) and, π 2 (θ, ν ) = π1 N(ν )π D (θ ν ), where ) π D (θ ν) h q (D[(θ, θ 0 ) ν] π N (θ ν). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

110 PART IV. Divergence based priors To be highlighted... DB priors have a simple form and are easy to explain! gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

111 PART IV. Divergence based priors To be highlighted... DB priors have a simple form and are easy to explain! They have appealing theoretical properties and in standard scenarios DB priors produce similar answers to other well established methods. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

112 PART IV. Divergence based priors To be highlighted... DB priors have a simple form and are easy to explain! They have appealing theoretical properties and in standard scenarios DB priors produce similar answers to other well established methods. In complex scenarios, where the other proposals fail, DB priors are well defined and produce quite reasonable results. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

113 PART IV. Divergence based priors To be highlighted... DB priors have a simple form and are easy to explain! They have appealing theoretical properties and in standard scenarios DB priors produce similar answers to other well established methods. In complex scenarios, where the other proposals fail, DB priors are well defined and produce quite reasonable results. DB priors are quite general and, we hope, can be considered a reasonable extension of Jeffreys ideas on testing. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

114 PART IV. Divergence based priors Full technical details available at: gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

115 PART IV. Divergence based priors Full technical details available at: THANKS! gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

116 PART IV. Divergence based priors Simple example: credible intervals and testing Suppose: From a sample y i N(µ, 1), of size n = 50 I get ȳ = 0.25 from which I derive the 95% credible interval using π(µ) = 1. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

117 PART IV. Divergence based priors Simple example: credible intervals and testing Suppose: From a sample y i N(µ, 1), of size n = 50 I get ȳ = 0.25 from which I derive the 95% credible interval using π(µ) = 1. I obtain CI=[0.11,0.39]. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

118 PART IV. Divergence based priors Simple example: credible intervals and testing Suppose: From a sample y i N(µ, 1), of size n = 50 I get ȳ = 0.25 from which I derive the 95% credible interval using π(µ) = 1. I obtain CI=[0.11,0.39]. I am interested in knowing if H : µ = 0 = no effect (although I did not explicitly say). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

119 PART IV. Divergence based priors Simple example: credible intervals and testing Suppose: From a sample y i N(µ, 1), of size n = 50 I get ȳ = 0.25 from which I derive the 95% credible interval using π(µ) = 1. I obtain CI=[0.11,0.39]. I am interested in knowing if H : µ = 0 = no effect (although I did not explicitly say). From the interval above, I would decide Bayesianly, to reject H (say has a probability which is less than 0.05). But gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

120 PART IV. Divergence based priors Simple example: credible intervals and testing Suppose: From a sample y i N(µ, 1), of size n = 50 I get ȳ = 0.25 from which I derive the 95% credible interval using π(µ) = 1. I obtain CI=[0.11,0.39]. I am interested in knowing if H : µ = 0 = no effect (although I did not explicitly say). From the interval above, I would decide Bayesianly, to reject H (say has a probability which is less than 0.05). But Prob(H y) = 0.66 Come back! gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

121 PART IV. Divergence based priors Standard scenarios In linear models, the DB priors coincide with the proposals of Zellner and Siow (1980, 1984). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

122 PART IV. Divergence based priors Standard scenarios In linear models, the DB priors coincide with the proposals of Zellner and Siow (1980, 1984). In other traditional problems (Bernoulli, exponential, Poisson or Normal), DB priors produce Bayes factors very similar to the intrinsic. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

123 PART IV. Divergence based priors Standard scenarios In linear models, the DB priors coincide with the proposals of Zellner and Siow (1980, 1984). In other traditional problems (Bernoulli, exponential, Poisson or Normal), DB priors produce Bayes factors very similar to the intrinsic. H 1 : (µ, σ) = (0, 1) vs.h 2 : (µ, σ) (0, 1) Come back! gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

124 PART IV. Divergence based priors Scenario with improper likelihood Let f (y θ, p) = p N(y 0, 1) + (1 p) N(y θ, 1), (p known) The problem: H 1 : θ = 0 vs. H 2 : θ 0. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

125 PART IV. Divergence based priors Scenario with improper likelihood Let f (y θ, p) = p N(y 0, 1) + (1 p) N(y θ, 1), (p known) The problem: H 1 : θ = 0 vs. H 2 : θ 0. Intrinsic Bayes factors do not exist. Berger and Pericchi (2001) recommend using π2 BP (θ) = Ca(θ 0, 1). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

126 PART IV. Divergence based priors Scenario with improper likelihood Let f (y θ, p) = p N(y 0, 1) + (1 p) N(y θ, 1), (p known) The problem: H 1 : θ = 0 vs. H 2 : θ 0. Intrinsic Bayes factors do not exist. Berger and Pericchi (2001) recommend using π2 BP (θ) = Ca(θ 0, 1). The DB prior does not have a closed form, although a good approximation is π D 2 (θ) Ca(θ 0, 1/(1 p)). gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

127 PART IV. Divergence based priors Comparisons π D (solid line), Ca(0, 1/1 p) (dashed) and π2 BP (µ) = Ca(µ 0, 1) (dots) mu mu p=0.50 Come back! p=0.75 gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

128 PART IV. Divergence based priors Scenario with irregular models Sea f (y θ) = exp{ (y θ)}, y > θ, and the problem H 1 : θ = θ 0 vs. H 2 : θ > θ 0. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

129 PART IV. Divergence based priors Scenario with irregular models Sea f (y θ) = exp{ (y θ)}, y > θ, and the problem H 1 : θ = θ 0 vs. H 2 : θ > θ 0. These models pose in serious problems to the intrinsic priors. They conjecture that it is π AI 2 (θ) = ( e θ θ 0 log(1 e θ 0 θ ) 1 ), θ > θ 0. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

130 PART IV. Divergence based priors Scenario with irregular models Sea f (y θ) = exp{ (y θ)}, y > θ, and the problem H 1 : θ = θ 0 vs. H 2 : θ > θ 0. These models pose in serious problems to the intrinsic priors. They conjecture that it is π AI 2 (θ) = ( e θ θ 0 log(1 e θ 0 θ ) 1 ), θ > θ 0. DB prior is π D 2 (θ) = ( 1 + 2(θ θ 0 ) ) 3/2, θ > θ0. gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46

131 PART IV. Divergence based priors Comparisons (θ 0 = 0) For, π D 2 (solid line) y πai 2 (dashed line) Come back! gonzalo garcía-donato and susie Bayarri () DB priors May 22, / 46