Testable Forecasts. Luciano Pomatto. Caltech

Transcription

1 Testable Forecasts Luciano Pomatto Caltech

2 Forecasting Every day a forecaster announces a probability p of rain. We want to understand if and how such predictions can be tested empirically. Probabilistic predictions: Weather and climate (Gneiting and Raftery (2005)), aggregate output and inflation (Diebold, Tay and Wallis (1997)), epidemics (Alkema, Raftery and Clark (2007)), seismic hazard (Jordan et al. (2011)), financial risk (Timmermann (2000)), demographic variables (Raftery et al. (2012)), elections (Tetlock (2005)), etc.

3 Calibration Test Many variations across fields: density forecasts, Value-at-Risk, etc.

4 Fundamental property Let P be the true law governing the data. P is unrestricted and unknown. Dawid (1982) A forecaster who predicts according to P passes the calibration test P-a.s. Hence: Type-I error free: No risk of rejecting the correct predictions of an expert who knows the true law. The tester is not required to have any preconceived theory about the problem at hand. The forecaster can be evaluated on purely empirical ground.

5 Tests and incentive problems Two main approaches: 1 Contract theory: forecasters as agents advising a principal about the best course of action. 2 Statistical tests: alternative to standard contracts. Used when: Forecasts lack an easily identifiable user. (e.g. National Weather Service, Macroeconomics) Contracts are impractical. The decision problem is not well defined. (e.g. testing of scientific theories) Key issue: Forecasters may be concerned about their reputation.

6 Adverse selection Consider: An expert informed about the true probabilistic law governing the data. A forecaster who is ignorant about the data generating process but is interested in passing the test. The calibration test cannot discriminate between the two.

7 Adverse selection Foster and Vohra (1998) There exists a randomized forecasting algorithm that requires no knowledge about the data generating process and makes the forecaster calibrated with high probability, no matter what data is realized.

8 Adverse selection Foster and Vohra (1998) There exists a randomized forecasting algorithm that requires no knowledge about the data generating process and makes the forecaster calibrated with high probability, no matter what data is realized. Sandroni (2003) The result extends to essentially any test that is Type-I error free.

9 This paper I consider the problem of testing in the presence of a theory about the data generating process. Theory a restriction over the domain of possible laws. Forecasters are required to make predictions that belong to such domain. Q1: What domains allow for tests that cannot be manipulated? Q2: What tests are non-manipulable? Q3: What does it mean to be an expert?

10 Literature Negative Results: Sandroni (2003), Al-Najjar and Weinstein (2008), Shmaya (2008), Olszewski and Sandroni (2008), etc. Positive results for non-finite tests: Dekel and Feinberg (2006), Olszewski and Sandroni ( ), Stewart (2011), Feinberg and Stewart (2008), Feinberg and Lambert (2015), etc.

11 Literature Results for finite tests: Olszewski and Sandroni (2009): examples and negative results. Al-Najjar, Smorodinsky, Sandroni, Weinstein (2010). Computational constraints: Fortnow and Vohra (2009). Scoring Rules: Babaioff, Blumrosen, Lambert and Reingold (2011).

12 Model: Basic Ingredients In each period an outcome from a finite set X is publicly observed. Ω = X : set of all paths. (Ω) : set of all Borel probability measures on (Ω, B).

13 Model: Empirical Tests A forecaster claims to know the law P (Ω) governing the data. A tester is interested in evaluating this claim. Timing: 1 The tester designs a test T : Ω (Ω) {0, 1} 2 The forecaster observes T and reports a prediction P. 3 Nature produces a path ω Ω. 4 T (ω, P) determines acceptance or rejection.

14 Adverse Selection The forecaster is either: A true expert who knows the true law P governing the data, and reports it truthfully. A strategic forecaster uninformed but interested in passing the test. A (mixed) strategy is a randomization ζ ( (Ω))

15 Example: Likelihood-Ratio Test 1 Fix a benchmark measure P with full support and a time n. 2 The forecaster announces P (Ω). 3 T (ω, P) = 1 if and only if P(ω n ) P (ω n ) > 1

16 Example: Likelihood-Ratio Test 1 Fix a benchmark measure P with full support and a time n. 2 The forecaster announces P (Ω). 3 T (ω, P) = 1 if and only if P(ω n ) P (ω n ) > 1 There exists a strategy ζ ( (Ω)) such that for every ω Ω. ζ{p : T (ω, P) = 1} n

17 Paradigms A paradigm is a subset Λ (Ω). It represents a theory about the data generating process. Forecasts outside Λ are rejected a priori. We want to understand: What paradigms allow for tests that do not reject true expert and cannot be manipulated?

18 Desiderata I : The test does not reject a true expert. Definition Given a paradigm Λ, a test T passes the truth with probability 1 ε if for all P Λ P{ω : T (ω, P) = 1} 1 ε

19 Desiderata II : Rejecting strategic forecasters is feasible. Definition Given a paradigm Λ a test T is ε-nonmanipulable if for every strategy ζ there is a law P ζ Λ such that (P ζ ζ){(ω, P) : T (ω, P) = 1} ε

20 Desiderata Payoffs: 0 outside option w > 0 if T = 1 l < 0 if T = 0 Maxmin expected payoff: inf P Λ E P ζ [wt + l (1 T )] < 0 whenever T is ε-nonmanipulable for ε small enough. So, the test can screen between informed and uninformed forecasters.

21 Desiderata III : The test decides in finite time. Definition A test T is finite if for every P (Ω) there exists a time N P such that T (, P) is measurable with respect to F NP.

22 Testable Paradigms Definition A paradigm Λ is testable if for every ε > 0 there exists a test T such that: 1 T passes the truth with probability 1 ε; 2 T is ε-nonmanipulable; 3 T is finite.

23 Testable Paradigms Definition A paradigm Λ is testable if for every ε > 0 there exists a test T such that: 1 T passes the truth with probability 1 ε; 2 T is ε-nonmanipulable; 3 T is finite. Q: What paradigms are testable?

24 A Subjectivist Perspective Consider an outside observer (a voter, an analyst or a consumer) who is uncertain about the data generating process, as expressed by a prior belief µ ( (Ω)) The observer trusts the paradigm if µ(λ) = 1 His predictions are given by Q µ (E) = (Ω) P(E)dµ(P) for every event E

25 Characterization Theorem A paradigm Λ is testable if and only if for every ε > 0 there exists a prior µ (Λ) such that sup Q µ (E) P(E) 1 ε for all P Λ. E Ω

26 Characterization Theorem A paradigm Λ is testable if and only if for every ε > 0 there exists a prior µ (Λ) such that sup Q µ (E) P(E) 1 ε for all P Λ. E Ω Consider a belief µ. Two polar cases: Q µ Λ : the observer predicts as a potential expert. Q µ P 1 ε for all P Λ : the predictions of the observer are far from the true law, whatever that is.

27 Characterization Theorem A paradigm Λ is testable if and only if for every ε > 0 there exists a prior µ (Λ) such that sup Q µ (E) P(E) 1 ε for all P Λ. E Ω Consider a belief µ. Two polar cases: Q µ Λ : the observer predicts as a potential expert. Q µ P 1 ε for all P Λ : the predictions of the observer are far from the true law, whatever that is. The fact that Λ is taken to be true should not exhaust all possible opinions that a rational agent can entertain.

28 Equivalent Characterization Index of compactness & convexity: (Shapley-Folkman-Starr) I(Λ) = sup inf Q co w (Λ) P Λ Q P 0 I(Λ) 1 I(Λ) = 0 = back to impossibility results. I(Λ) = 1 Λ is testable.

29 Non-Manipulable Tests Theorem Let Λ be testable. Let µ ( Λ ) satisfy Q µ P 1 ε for all P Λ. There exist integers (n P ) such that the test 1 if P Λ and P (ω n P ) > Q µ (ω n P ) T (ω, P) = 0 otherwise does not reject the truth with probability 1 ε and is ε-nonmanipulable.

30 Non-Manipulable Tests Theorem Let Λ be testable. Let µ ( Λ ) satisfy Q µ P 1 ε for all P Λ. There exist integers (n P ) such that the test 1 if P Λ and P (ω n P ) > Q µ (ω n P ) T (ω, P) = 0 otherwise does not reject the truth with probability 1 ε and is ε-nonmanipulable. Equivalent to a Neyman-Pearson hypothesis test, where P is the null and Q µ the alternative.

31 Ranking Tests The result leaves open the possibility that likelihood-ratio tests are inefficient in the number of observations they require. Q: Do there exist test that for a fixed sample size are more efficient than likelihood-ratio tests?

32 Ranking Tests Definition Fix a paradigm Λ. Given T 1 and T 2, say that T 1 is less manipulable than T 2 if sup ζ (Λ) inf E P ζ[t 1 ] P Λ sup ζ (Λ) inf E P ζ[t 2 ] P Λ Any uninformed forecaster who is screened out under T 2 is also screened out under T 1.

33 Ranking Tests Definition Fix a paradigm Λ. Given T 1 and T 2, say that T 1 is less manipulable than T 2 if sup ζ (Λ) inf E P ζ[t 1 ] P Λ sup ζ (Λ) inf E P ζ[t 2 ] P Λ Any uninformed forecaster who is screened out under T 2 is also screened out under T 1. Comparisons are more informative if we fix: 1 A bound α for the probability of not rejecting the truth. 2 Testing times (n P )

34 A Neyman-Pearson Lemma Theorem Fix a paradigm Λ, testing times (n P ) and a probability α [0, 1]. There exists a prior µ ( Λ ), thresholds (λ P ), and a test T such that: 1 T (ω, P) = 1 if P Λ and P (ω n P ) > λ P Q µ (ω n P ), 2 T (ω, P) = 0 if P / Λ or P (ω n P ) < λ P Q µ (ω n P ), 3 T is less manipulable than any other test that is (i) bounded by (n P ), (ii) does not reject the truth with probability α.

35 Testing Forecasts and Testing Knowledge Definition (Doob (1949)) A paradigm Λ is identifiable if there exists a measurable map f : Ω Λ such that P{ω : f (ω) = P} = 1 for every P Λ.

36 Testing Forecasts and Testing Knowledge Definition (Doob (1949)) A paradigm Λ is identifiable if there exists a measurable map f : Ω Λ such that P{ω : f (ω) = P} = 1 for every P Λ. Examples: i.i.d. Markov irreducible stationary ergodic

37 Testing Forecasts and Testing Knowledge Definition (Doob (1949)) A paradigm Λ is identifiable if there exists a measurable map f : Ω Λ such that P{ω : f (ω) = P} = 1 for every P Λ. Remarks: An estimation problem. Experts are more relevant when the paradigm is not identifiable.

38 Testability and Identifiability Fact Every infinite, identifiable paradigm is testable. Fact Not all testable paradigms are identifiable. E.g. Markov. A procedure for creating testable domains: enlarge a rich identifiable paradigm.

39 Testability and Identifiability Theorem If there exists a prior µ (Λ) such that Q µ P = 1 for all P Λ, then there exists a subset Λ Λ that is identifiable and uncountable. Any well-behaved testable paradigm is obtained by enlarging a rich identifiable paradigm. Relies on a result of Burgess and Mauldin (1981).

40 Large Paradigms Any theory, if incorrect, exposes the tester to the risk of rejecting informed experts. Several papers study tests which do not reject the truth except for a small set of distributions: Feinberg and Stewart (2008), Olszewski and Sandroni (2009), Stewart (2011), Feinberg and Lambert (2015), etc.

41 Maximal Paradigms Theorem Let ε (0, 1). Given a law Q (Ω) the paradigm Λ Q = {P (Ω) : Q P > 1 ε} is ε-testable. It is not included in any testable paradigm.

42 Maximal Paradigms Theorem Let ε (0, 1). Given a law Q (Ω) the paradigm Λ Q = {P (Ω) : Q P > 1 ε} is ε-testable. It is not included in any testable paradigm. Corollary Any testable paradigm can be enlarged to a maximal ε-testable paradigm.

43 On and off-path predictions Key property: The forecaster reports a fully specified law P. Unconventional. Under the tests considered so far, an expert cannot prove his knowledge without revealing it. As a result, even informed experts might not be willing to participate to the test.

44 A new test Fix a prior µ. Test: At time 0, the forecaster announces a deadline d N. In each period n = 0,..., d 1, given history ω n 1 = x 1,..., x n 1 the forecaster provides a prediction p n (X ). At time d we obtain a history ω d = x 1,..., x d and a sequence of predictions (p 0,..., p d 1 ). The forecaster passes the test iff d 1 n=0 p n (x n+1 ) > 1 ε Q µ ( ω d).

45 Test Definition Fix µ ( Λ ) be a prior. Given ε, the forecasts-based likelihood-ratio test is defined as T µ,ε (d, ω, P) = for all d N, ω Ω and P (Ω). 1 if P ( ω d) > 1 ε Q ( µ ω d ) 0 otherwise A strategy is a randomization ζ (N (Ω)).

46 Properties Theorem For every testable paradigm Λ and every ε there is a prior µ (Λ) such that: 1 For every P in Λ there is a deadline d P such that for every d d P P ({ω : T µ,ε (d, ω, P) = 1}) 1 ε. 2 For every strategy ζ (N (Ω)) there is a law P ζ Λ such that (P ζ ζ) {(ω, (d, P)) : T µ,ε (d, ω, P) = 1} ε.