Lecture 4 How to Forecast

Transcription

1 Lecture 4 How to Forecast Munich Center for Mathematical Philosophy March 2016 Glenn Shafer, Rutgers University 1. Neutral strategies for Forecaster 2. How Forecaster can win (defensive forecasting) 3. Must Skeptic s tests of Forecaster be continuous? 4. The meaning of probability 1

2 Four ways of using probability games 1. Statistical testing. Put a theory in the role of Forecaster. Take the role of Skeptic and test the theory. 2. Forecasting. Put ourselves in the role of Forecaster and try to make good probability predictions. 3. Probability judgement. Use a battery of probability games as a scale of canonical examples to which to compare evidence. 4. Causal investigation. Hypothesize a hidden game in which Nature plays Forecaster. We see only some of Reality s moves. 2

3 Game theoretic probability. On each round: 1. Forecaster offers bets (i.e., probabilistic predictions). 2. Skeptic decides which offers to accept. 3. Reality decides the outcome. Probability theory and statistical testing are about strategies for Skeptic. Prediction is about strategies for Forecaster. 3

4 How to forecast Part 1. Strategies for Forecaster Forecaster s moves never determine global probabilities. But a global probability distribution provides a strategy for Forecaster. Bayes provides only one way of constructing strategies for Forecaster. Part 2. How Forecaster can win (defensive forecasting) Winning when Skeptic s strategy is known Beating an all purpose strategy for Skeptic Part 3. Must Skeptic s tests of Forecaster be continuous? Putnam s and Dawid s objection Randomization as a response Part 4. The meaning of probability Statistical models mean less than we thought. Neyman s inductive behavior Probability judgement 4

5 Part 1. Strategies for Forecaster Forecaster s actual moves fall far short of specifying a probability distribution for the sequence of moves made by Reality. But a global probability distribution for Reality s moves provides a strategy for Forecaster. Bayes provides only one way of constructing strategies for Forecaster. 5

6 Part 1. Strategies for Forecaster Game theoretic probability. On each round: 1. Forecaster offers bets (i.e., probabilistic predictions). 2. Skeptic decides which offers to accept. 3. Reality decides the outcome. Example: Each evening for a year, Bob the weather forecaster gives probabilities for rain the next day. 6

7 Part 1. Strategies for Forecaster 1 Path taken

8 Part 1. Strategies for Forecaster

9 Part 1. Strategies for Forecaster 9

10 Part 1. Strategies for Forecaster Two ways of finding a relatively robust strategy for Forecaster This can be called Bayesian. Collect a variety of strategies for Forecaster and average them. Asymptotically, the average may be as successful as the most successful of the individual strategies. We call this defensive forecasting. Collect a variety of strategies for Skeptic, which enforce different properties we expect when we say y n has probability p n. Average these strategies for Skeptic and play against them. 10

11 Part 1. Strategies for Forecaster p 1,y 1,p 2,y 2, the different properties we expect when we say y n has probability p n. This phrase calls out for acknowledgement of Phil Dawid s insights. Dawid s outstanding insight was that these expectations do not depend on further details of any purported joint distribution for y 1,y 2,. In particular, they do not require independence. Alexander Philip Dawid Born

12 Part 2. How Forecaster can win Winning when Skeptic s strategy is known Beating an all purpose strategy for Skeptic Karl Menger Leonid Levin born 1948 Jean Ville invented the notion of an all purpose betting test after hearing about von Mises s ideas from Abraham Wald in Menger s seminar in Vienna in Leonid Levin developed the idea of playing against such a test in the 1970s. 12

13 Part 2. How Forecaster can win Basic intuition For each way that we expect Reality s moves to conform to Forecaster s predictions, Skeptic has a strategy that multiplies his capital if Reality does not conform. Forecaster can keep Skeptic s strategy from making money by adapting to Reality s past moves. Forecast can adapt so well that Reality conforms to his forecasts better than we would expect by chance. Example 13

14 Part 2. How Forecaster can win Defensive forecasting The name was introduced in the working paper Defensive Forecasting, by Vovk, Takemura, and Shafer (September 2004). See also Working Papers 7, 9, 10, 11, 13, 14, 16, 17, 18, 20, 21, 22, and 30 at Akimichi Takemura Born 1952 Volodya Vovk Born

15 Part 2. How Forecaster can win Important idea: Any test (betting strategy for Skeptic) Forecaster wants to pass is continuous. 1. The strategies for Skeptic used by Shafer and Vovk in Probability and Finance were continuous (& computable & relatively simple). 2. Conjecture: every high probability or probability one result in classical theory can be proven by a strategy for Skeptic that is computable in Brouwer s sense. 3. L. E. J. Brouwer s continuity principle (1916): only continuous functions are computable. Why can t discontinuous functions be computed? Because you would need infinite precision. 15

16 How Forecaster can beat any single strategy S Part 2. How Forecaster can win 16

17 Defensive Forecasting, page 4: Part 2. How Forecaster can win 17

18 Part 2. How Forecaster can win Suppose E is an event in this protocol: Suppose Skeptic has a continuous strategy that does not risk bankruptcy and makes him infinitely rich if his opponents do not make E happen. Corollary of theorem: In this case, Forecaster can guarantee E. Corollary of conjecture: Forecaster can guarantee any interesting high probability E. 18

19 Part 2. How Forecaster can win Crucial idea: all the tests (betting strategies for Skeptic) Forecaster needs to pass can be merged into a single all purpose test for Forecaster to pass. 1. If Skeptic has two strategies for multiplying capital risked, he can average them (i.e., divide his capital between them). 2. Formally: average the strategies. 3. You can average countably many strategies. 4. There are only countably many tests (Wald). 5. Forecaster can beat any single test (including the all purpose test). Abraham Wald 1936 {1937 Laurent BIENVENU, Glenn SHAFER and Alexander SHEN : On the history of martingales in the study of randomness, Electronic Journal for History of Probability and Statistics, 5(1), June

20 Part 2. How Forecaster can win Constructing an all purpose test In practice, we want to test calibration (y=1 happens 30% of the times you say p=.3) resolution (also true just for times when it rained yesterday) For simplicity, consider only calibration. Use Bernoulli s theorem to test calibration for each p. Merge the tests for different p. 20

21 Part 2. How Forecaster can win 21

22 Part 2. How Forecaster can win 22

23 Part 2. How Forecaster can win 23

24 Part 2. How Forecaster can win 24

25 Part 2. How Forecaster can win 25

26 Part 2. How Forecaster can win 26

27 Part 2. How Forecaster can win 27

28 Part 3. Must tests of Forecaster be continuous? Putnam s and Dawid s objection Randomization as a response 28

29 Part 3. Must tests of Forecaster be continuous? Hilary Putnam s counterexample Hilary Putnam, born 1926, on the right, with Bruno Latour, born

30 Part 3. Must tests of Forecaster be continuous? 30

31 Part 3. Must tests of Forecaster be continuous? 31

32 Part 3. Must tests of Forecaster be continuous? 32

33 Part 3. Must tests of Forecaster be continuous? Two paths to successful probability forecasting 1. Insist that tests be continuous. Conventional tests can be implemented with continuous betting strategies (Shafer & Vovk, 2001). Only continuous functions are constructive (L. E. J. Brouwer). 2. Allow Forecaster to hide his precise prediction from Reality using a bit of randomization. 33

34 Part 4. The meaning of probability Statistical models mean less than we thought. Neyman s inductive behavior Probability judgement 34

35 Part 4. The meaning of probability Giving probabilities for successive events. Think stochastic process, unknown probabilities, not iid. Can I assign probabilities that will pass statistical tests? 1. If you insist that I announce all probabilities before seeing any outcomes, NO. 2. If you always let me see the preceding outcomes before I announce the next probability, YES. 35

36 Part 4. The meaning of probability We knew that a probability can be estimated from a random sample. But this depends on the idd assumption. Defensive forecasting tells us something new. 1. Our opponent is Reality rather than Nature. (Nature follows laws; Reality plays as he pleases.) 2. Defensive forecasting gives probabilities that pass statistical tests regardless of how Reality behaves. 3. The notion of a stochastic process with unknown probabilities loses its empirical content. 4. But the prediction of y n from x n depends on the sequence in which we have placed it. 36

37 Part 4. The meaning of probability Jeyzy Neyman s inductive behavior A statistician who makes predictions with 95% confidence has two goals: be informative be right 95% of the time Why isn t this good enough for probability judgment? Answer: Two statisticians who are right 95% of the time may tell the court different and even contradictory things. They are placing the current event in different sequences. 37

38 Part 4. The meaning of probability Three settings for probability 38