Efficient Minimax Strategies for Square Loss Games

Transcription

1 Efficient Minimax Strategies for Square Loss Games Wouter M. Koolen Queensland University of Technology and UC Berkeley Alan Malek University of California, Berkeley Peter L. Bartlett University of California, Berkeley and Queensland University of Technology Abstract We consider online prediction problems where the loss between the prediction and the outcome is measured by the squared Euclidean distance and its generalization, the squared Mahalanobis distance. We derive the minimax solutions for the case where the prediction and action spaces are the simplex this setup is sometimes called the Brier game and the l ball this setup is related to Gaussian density estimation. We show that in both cases the value of each sub-game is a quadratic function of a simple statistic of the state, with coefficients that can be efficiently computed using an explicit recurrence relation. The resulting deterministic minimax strategy and randomized maximin strategy are linear functions of the statistic. Introduction We are interested in general strategies for sequential prediction and decision making a.k.a. online learning that improve their performance with experience. Since the early days of online learning, people have formalized such learning tasks as regret games. The learner interacts with an adversarial environment with the goal of performing almost as well as the best strategy from some fixed reference set. In many cases, we have efficient algorithms with an upper bound on the regret that meets the game-theoretic lower bound up to a small constant factor. In a few special cases, we have the exact minimax strategy, meaning that we understand the learning problem at all levels of detail. In even fewer cases we can also efficiently execute the minimax strategy. These cases serve as exemplars to guide our thinking about learning algorithms. In this paper we add two interesting examples to the canon of efficiently computable minimax strategies. Our setup, as described in Figure, is as follows. The Learner and the Adversary play vectors a A and x X, upon which the Learner is penalized using the squared Euclidean distance a x or its generalization, the squared Mahalanobis distance, a x W = a x W a x, parametrized by a symmetric matrix W 0. After a sequence of T such interactions, we compare the loss of the Learner to the loss of the best fixed prediction a A. In all our examples, this best fixed action in hindsight is the mean outcome a = T T t= x t, regardless of W. We use regret, the difference between the loss of the learner and the loss of a, to evaluate performance. The minimax regret for the T -round game, also known as the value of the game, is given by V := a x a T x T t= a t x t W a t= a x t W

2 where the a t range over actions A and the x t range over outcomes X. The minimax strategy chooses the a t, given all past outcomes x,..., x t, to achieve this regret. Intuitively, the minimax regret is the regret if both players play optimally while assuming the other player is doing the same. Our first example is the Brier game, where the action and outcome spaces are the probability simplex with K outcomes. The Brier game is traditionally popular in meteorology [Bri50]. Our second example is the ball game, where the action and outcome spaces are the Euclidean norm ball, i.e. A = X = {x R K x = }. Even though we measure loss by the squared Mahalanobis distance, we play on the standard Euclidean norm ball. The ball game is related to Gaussian density estimation [TW00]. In each case we exhibit a strategy that can play a T -round game in OT K time. The algorithm spends OT K + K 3 time pre-processing the game, and then plays in OK time per round. Outline Given: T, W, A, X. For t =,,..., T Learner chooses prediction a t A Adversary chooses outcome x t X Learner incurs loss a t x t W. Figure : Protocol We define our loss using the squared Mahalanobis distance, parametrized by a symmetric matrix W 0. We recover the squared Euclidean distance by choosing W = I. Our games will always last T rounds. For some observed data x,..., x n, the value-to-go for the remaining T n rounds is given by V x,..., x n := a n+ x n+ a T x T t=n+ a t x t W a t= a x t W. By definition, the minimax regret is V = V ɛ where ɛ is the empty sequence, and the value-togo satisfies the recurrence { T V x,..., x n = a t= a x t W if n = T, an+ xn+ a n+ x n+ W + V x,..., x n+ if n < T. Our analysis for the two games proceeds in a similar manner. For some past history of plays x,..., x n of length n, we summarize the state by s = n t= x t and σ = n t= x t W x t. As we will see, the value-to-go after n of T rounds can be written as V s, σ, n; i.e. it only depends on the past plays through s and σ. More surprisingly, for each n, the value-to-go V s, σ, n is a quadratic function of s and a linear function of σ under certain conditions on W. While it is straightforward to see that the terminal value V s, σ, T is quadratic in the state this is easily checked by computing the loss of the best expert and using the first case of Equation, it is not at all obvious that propagating from V s + x, σ + x W x, n + to V s, σ, n, using the second case of, preserves this structure. This compact representation of the value-function is an essential ingredient for a computationally feasible algorithm. Many minimax approaches, such as normalized maximum likelihood [Sht87], have computational complexities that scale exponentially with the time horizon. We derive a strategy that can play in constant amortized time. Why is this interesting? We go beyond previous work in a few directions. First, we exhibit two new games that belong to the tiny class admitting computationally feasible minimax algorithms. Second, we consider the setting with squared Mahalanobis loss which allows the user intricate control over the penalization of different prediction errors. Our results clearly show how the learner should exploit this prioritization.. Related work Repeated games with minimax strategies are frequently studied [CBL06] and, in online learning, minimax analysis has been applied to a variety of losses and repeated games; however, computa-

3 tionally feasible algorithms are the exception, not the rule. For example, consider log loss, first discussed in [Sht87]. Whiile the minimax algorithm, Normalized Maximum Likelihood, is well known [CBL06], it generally requires computation that is exponential in the time horizon as one needs to aggregate over all data sequences. To our knowledge, there are two exceptions where efficient NML forecasters are possible: the multinomial case where fast Fourier transforms may be exploited [KM05], and very particular exponential families that cause NML to be a Bayesian strategy [HB], [BGH + 3]. The minimax optimal strategy is known also for: i the ball game with W = I [TW00] our generalization to Mahalanobis W I results in fundamentally different strategies, ii the ball game with W = I and a constraint on the player s deviation from the current empirical minimizer [ABRT08] for which the optimal strategy is Follow-the-Leader, iii Lipschitz-bounded convex loss functions [ABRT08], iv experts with an L bound [AWY08], and v static experts with absolute loss [CBS]. While not guaranteed to be an exhaustive list, the previous paragraph demonstrates the rarity of tractable minimax algorithms. 3 The Offline Problem The regret is defined as the difference between the loss of the algorithm and the loss of the best action in hindsight. Here we calculate that action and its loss. Lemma 3.. Suppose A convx this will always hold in the settings we study. For data x,..., x T X, the loss of the best action in hindsight equals a A t= a x t W = T x t W x t T T x t W x t, 3 T t= and the minimizer is the mean outcome a = T T t= x t. Proof. The unconstrained minimizer and value are obtained by equating the derivative to zero and plugging in the solution. The assumption A convx ensures that the constraint a A is inactive. The best action in hindsight is curiously independent of W, A and X. This also shows that the follow the leader strategy that plays a t = t t s= x s is independent of W and A as well. As we shall see, the minimax strategy does not have this property. 4 Simplex Brier Game In this section we analyze the Brier game. The action and outcome spaces are the probability simplex on K outcomes; A = X = := {x R K + x = }. The loss is given by half the squared Mahalanobis distance, a x W. We present a full minimax analysis of the T -round game: we calculate the game value, derive the maximin and minimax strategies, and discuss their efficient implementation. The structure of this section is as follows. In Lemmas 4. and 4., the conclusions value and optimizers are obtained under the proviso that the given optimizer lies in the simplex. In our main result, Theorem 4.3, we apply these auxiliary results to our minimax analysis and argue that the maximizer indeed lies in the simplex. We immediately work from a general symmetric W 0 with the following lemma. Lemma 4.. Fix a symmetric matrix C 0 and vector d. The optimization problem has value d Cd Cd p = C C provided that p is in the simplex. max p p C p + d p d C C C C = d Cd C = t= t= d + Cd C attained at optimizer C C C C d + C C 3

4 Proof. We solve for the optimal p. Introducing Lagrange multiplier λ for the constraint k p k =, we need to have p = C d λ which results in λ = Cd C. Thus, the maximizer equals p = C d Cd C which produces objective value d + Cd C C d Cd C. The statement follows from simplification. This lemma allows us to compute the value and saddle point whenever the future payoff is quadratic. Lemma 4.. Fix symmetric matrices W 0 and A such that W + A 0, and a vector b. The optimization problem min max a x a x W + x Ax + b x achieves its value c W c W c W where c = diag W + A + b at saddle point the maximin strategy randomizes, playing x = e i with probability p i a = p = W W W c + W W W provided p 0. Proof. The objective is convex in x for each a as W + A 0, so it is maximized at a corner x = e k. We apply min-max swap see e.g. [Sio58], properness of the loss which implies that a = p and expand: min max a x a x W + x Ax + b x = min max a k = max min p a = max E p k p a e k W + e k Ae k + b e k [ E k p a e k W + ] e k Ae k + b e k [ p e k W + ] e k Ae k + b e k = max p p W p + diag W + A p + b p The proof is completed by applying Lemma Minimax Analysis of the Brier Game Next, we turn to computing V s, σ, n as a recursion and specifying the minimax and maximin strategies. However, for the value-to-go function to retain its quadratic form, we need an alignment condition on W. We say that W is aligned with the simplex if W W W W diagw W W, 4 where denotes an entry-wise inequality between vectors. Note that many matrices besides I satisfy this condition: for example, all symmetric matrices. We can now fully specify the value and strategies for the Brier game. Theorem 4.3. Consider the T -round Brier game with Mahalanobis loss a x W with W satisfying the alignment condition 4. After n outcomes x,..., x n with statistics s = n t= x t and σ = n t= x t W x t the value-to-go is V s, σ, n = α ns W s σ + nα n diagw s + γ n, 4

5 and the minimax and maximin strategies are given by a s, σ, n = p s, σ, n = W W + α n+ where the coefficients are defined recursively by s nw W + nα n+ W W W W diagw α T = T γ T = 0 α n = α n+ + α n+ γ n = nα n+ + γ n+. 4 diagw W diagw W diagw W Proof. We prove this by induction, beginning at the end of the game and working backwards in time. Assume that V s, σ, T has the given form. Recall that the value at the end of the game is V s, σ T, T = a t= a x t W and is given by Lemma 3.. Matching coefficients, we find V s, σ, T corresponds to α T = T and γ T = 0. Now assume that V has the assumed form after n rounds. Using s and σ to denote the state after n rounds, we can write V s, σ, n = min max a x a x W + α ns + x W s + x σ + x W x + nα n diagw s + x + γ n. Using Lemma 4. to evaluate the right hand side produces a quadratic function in the state, and we can then match terms to find α n and γ n and the minimax and maximin strategy. The final step is checking the p 0 condition necessary to apply Lemma 4., which is equivalent to W being aligned with the simplex. See the appendix for a complete proof. This full characterization of the game allows us to derive the following minimax regret bound. Theorem 4.4. Let W satisfy the alignment condition 4. The minimax regret of the T -round simplex game satisfies V + lnt 4 diagw W diagw W diagw. W Proof. The regret is equal to the value of the game, V = V 0, 0, 0 = γ 0. First observe that nα n+ = nα n+ + n α n+ = nα n+ + n α n α n+ = α n+ + n + α n+ + n α n. After summing over n the last two terms telescope, and we find γ 0 nα n+ = T α T + + α n+ = n=0 n=0 α n. Each α n can be bounded by /n, as observed in [TW00, proof of Lemma ]. In the base case n = T this holds with equality, and for n < T we have α n = α n+ + α n+ n= n + + n + = nn + n n + n. It follows that γ 0 T n= α n T n= n + lnt as desired. 5

6 5 Norm Ball Game This section parallels the previous. Here, we consider the online game with Mahalanobis loss and A = X = := {x R K x }, the -norm Euclidian ball not the Mahalanobis ball. We show that the value-to-go function is always quadratic in s and linear in σ and derive the minimax and maximin strategies. Lemma 5.. Fix a symmetric matrix A and vector b and assume A + W 0. Let λ max be the largest eigenvalue of W +A and v max the corresponding eigenvector. If b λ max I A b, then the optimization problem a x W + x Ax + x b a x has value b λ max I A b + λ max, minimax strategy a = λ max I A b and a randomized maximin strategy that plays two unit length vectors, with Pr x = a ± a a v max = ± a v max a a, where a and a are the components of a perpendicular and parallel to v max. Proof. As the objective is convex, the inner optimum must be on the boundary and hence will be at a unit vector x. Introduce a Lagrange multiplier λ for x x to get the Lagrangian a λ 0 x a x W + x Ax + x b + λ x x. This is concave in x if W + A λi 0, that is, λ max λ. Differentiating yields the optimizer x = W + A λi W a b, which leaves us with an optimization in only a and λ: a λ λ max a W a W a b W + A λi W a b + λ. Since the imums are over closed sets, we can exchange their order. Unconstrained optimization of a results in a = λi A b. Evaluating the objective at a and using W a b = W λi A b b = W + A λi λi A b results in λ λ max b λi A b + λ = λ λ max i u i b λ λ i + λ using the spectral decomposition A = i λ iu i u i. For λ λ max, we have λ λ i. Taking derivatives, provided b λ max I A b, this function is increasing in λ λ max, and so obtains its imum at λ max. Thus, when the assumed inequality is satisfied, the a is minimax for the given x. To obtain the maximin strategy, we can take the usual convexification where the Adversary plays distributions P over the unit sphere. This allows us to swap the imum and remum see e.g. Sion s minimax theorem[sio58] and obtain an equivalent optimization problem. We then see that the objective only depends on the mean µ = E x and second moment D = E xx of the distribution P. The characterization in [KNW3, Theorem.] tells us that µ, D are the first two moments of a distribution on units iff trd = and D µµ. Then, our usual min-max swap yields V = P = E a x P µ,d a [ a W a a W x + x W x + x Ax + b x a W a a W µ + tr W + AD + b µ = µ,d µ W µ + tr W + AD + b µ = a W a + b a + D a a trd= tr W + AD, ] 6

7 µ v max Figure : Illustration of the maximin distribution from Lemma 5.. The mixture of red unit vectors with mean µ has second moment D = µµ + µ µv max v max. where the second equality uses a = µ and the third used the saddle point condition µ = a. The matrix D with constraint trd = now seeks to align with the largest eigenvector of W + A but it also has to respect the constraint D a a. We now re-parameterize by C = D a a. We then need to find C 0 tr W + AC. trc= a a By linearity of the objective the maximizer is of rank, and hence this is a scaled maximum eigenvalue problem, with solution given by C = a a v max v max, so that D = a a + a a v max v max. This essentially reduces finding P to a -dimensional problem, which can be solved in closed form [KNW3, Lemma 4.]. It is easy to verify that the mixture in the theorem has the desired mean a and second moment D. See Figure for the geometrical intuition. Notice that both the minimax and maximin strategies only depend on W through λ max and v max. 5. Minimax Analysis of the Ball Game With the above lemma, we can compute the value and strategies for the ball game in an analogous way to Theorem 4.3. Again, we find that the value function at the end of the game is quadratic in the state, and, surprisingly, remains quadratic under the backwards induction. Theorem 5.. Consider the T -round ball game with loss a x W. After n rounds, the valueto-go for a state with statistics s = n t= x t and σ = n t= x t W x t is The minimax strategy plays V s, σ, n = s A n s σ + γ n. a s, σ, n = λ max I A n+ W An+ s and the maximin strategy plays two unit length vectors with Pr x = a ± a a v max = ± a v max a a, where λ max and v max correspond to the largest eigenvalue of A n+ and a and a are the components of a perpendicular and parallel to v max. The coefficients A n and γ n are determined recursively by base case A T = T W and γ T = 0 and recursion A n = A n+ W + λ max I A n+ An+ + A n+ γ n = λ max + γ n+. 7

8 Proof outline. The proof is by induction on the number n of rounds played. In the base case n = T we find see 3 A T = T W and γ T = 0. For the the induction step, we need to calculate V s, σ, n = a x a x W + V s + x, σ + x W x, n +. Using the induction hypothesis, we expand the right-hand-side to a x a x W + s + x A n+ s + x σ + x W x + γ n+. which we can evaluate by applying Lemma 5. with A = A n+ W and b = A n+ s. Collecting terms and matching with V s, σ, n = s A n s σ + γ n yields the recursion for A n and γ n as well as the given minimax and maximin strategies. As before, much of the algebra has been moved to the appendix. Understanding the eigenvalues of A n As we have seen from the A n recursion, the eigensystem is always the same as that of W. Thus, we can characterize the minimax strategy completely by its effect on the eigenvalues of W. Denote the eigenvalues of A n and W to be λ i n and ν i, respectively, with λ n corresponding to the largest eigenvalue. The eigenvalues follow: λ i n = λ i n+ ν i + λ n+ λi n+ + λ i n+ = λi n+ν i + λ n+ ν i + λ n+, λi n+ which leaves the order of the λ i n unchanged. The largest eigenvalue λ n satisfies the recurrence λ T /ν = /T and λ n/ν = λ n+/ν + λ n+ /ν, which, remarkably, is the same recurrence for the α n parameter in the Brier game, i.e. λ max n = α n ν max. This observation is the key to analyzing the minimax regret. Theorem 5.3. The minimax regret of the T -round ball game satisfies γ 0 = +lnt λ max W Proof. We have V = V 0, 0, 0 = γ 0 = / T n= λmax n proof of Theorem 4.4 gives the bound on T n= α n. = /λ max W T n= α n. The Taking stock, we find that the minimax regrets of the Brier game Theorems 4.3 and ball game Theorems 5. have identical dependence on the horizon T but differ in a complexity factor arising from the interaction of the action space and the loss matrix W. 6 Conclusion In this paper, we have presented two games that, unexpectedly, have computationally efficient minimax strategies. While the structure of the squared Mahalanobis distance is important, it is the interplay between the loss and the constraint set that allows efficient calculation of the backwards induction, value-to-go, and achieving strategies. For example, the squared Mahalanobis game with l ball action spaces does not admit a quadratic value-to-go unless W = I. We emphasize the low computational cost of this method despite the exponential blow-up in state space size. In the Brier game, the α n coefficients need to be precomputed, which can be done in OT time. Similarly, computation of the eigenvalues of the A n coefficients for the ball game can be done in OT K + K 3 time. Then, at each iteration of the algorithm, only matrix-vector multiplications between the current state and the precomputed parameters are required. Hence, playing either T round game requires OT K time. Unfortunately, as is the case with most minimax algorithms, the time horizon must be known in advance. There are many different future directions. We are currently pursuing a characterization of action spaces that permit quadratic value functions under squared Mahalanobis loss, and investigating connections between losses and families of value functions closed under backwards induction. There is some notion of conjugacy between losses, value-to-go functions, and action spaces, but a generalization seems difficult: the Brier game and ball game worked out for seemingly very different reasons. 8

9 References [ABRT08] Jacob Abernethy, Peter L. Bartlett, Alexander Rakhlin, and Ambuj Tewari. Optimal strategies and minimax lower bounds for online convex games. In Servedio and Zhang [SZ08], pages [AWY08] Jacob Abernethy, Manfred K. Warmuth, and Joel Yellin. When random play is optimal against an adversary. In Servedio and Zhang [SZ08], pages [BGH + 3] Peter L. Bartlett, Peter Grunwald, Peter Harremoës, Fares Hedayati, and Wojciech Kotłowski. Horizon-independent optimal prediction with log-loss in exponential families. CoRR, abs/ , 03. [Bri50] [CBL06] [CBS] [HB] [KM05] [KNW3] [Sht87] Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78: 3, 950. Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge University Press, 006. Nicolò Cesa-Bianchi and Ohad Shamir. Efficient online learning via randomized rounding. In J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F.C.N. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 4, pages , 0. Fares Hedayati and Peter L. Bartlett. Exchangeability characterizes optimality of sequential normalized maximum likelihood and bayesian prediction with jeffreys prior. In International Conference on Artificial Intelligence and Statistics, pages , 0. Petri Kontkanen and Petri Myllymäki. A fast normalized maximum likelihood algorithm for multinomial data. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence IJCAI-05, pages 63 66, 005. Wouter M. Koolen, Jiazhong Nie, and Manfred K. Warmuth. Learning a set of directions. In Shai Shalev-Shwartz and Ingo Steinwart, editors, Proceedings of the 6th Annual Conference on Learning Theory COLT, June 03. Yurii Mikhailovich Shtar kov. Universal sequential coding of single messages. Problemy Peredachi Informatsii, 33:3 7, 987. [Sio58] Maurice Sion. On general minimax theorems. Pacific J. Math., 8:7 76, 958. [SZ08] Rocco A. Servedio and Tong Zhang, editors. st Annual Conference on Learning Theory - COLT 008, Helsinki, Finland, July 9-, 008. Omnipress, 008. [TW00] Eiji Takimoto and Manfred K. Warmuth. The minimax strategy for Gaussian density estimation. In 3th COLT, pages 00 06,

10 A Proofs Proof of Theorem 4.3. We prove by induction. Recall that V s, σ, T, the value at the end of the game, already has V s, σ, T = T s W s σ, corresponding to α T = T and γ T = 0. Adding a constant to the objective in Lemma 4. only adds the same offset to the value without changing the saddle point. Now, assume the induction hypothesis for n T rounds. Using the abbreviation d = diagw, we then need to evaluate V s, σ, n = min max a x a x W + α ns + x W s + x σ + x W x + nα n d s + x + γ n. = min max a x a x W + α nx W x x W x + nα n d x + α n s W x + α ns W s + nα n d s σ + γ n. Applying Lemma 4. with A = α n W and b = nα n d + α n W s produces the value where all time indices are n V s, σ, n = α + αs W n α s + α + nα d s + n α d W d n α W d + α s W and the strategy p = σ + γ W W W W n α nd + α n W s + W W. Using the fact that s = n, the value can be further simplified to V s, σ, n = α + αs W s + n α + α d s + n α 8 d W d W d W σ + γ. This establishes the induction step upon setting α n = α n+ + α n+, γ n = γ n+ + nα n+ 8 d W d W d. W Finally, we need to verify that the strategies above respect the constraints of the game, i.e. that a and p are in the simplex. Otherwise, the above calculations do not correspond to the minimax strategy. We need to show for all s 0 with sum n that α n s + n α n W d α n n + W n α n W d W. 0

11 As this needs to hold for all s 0, we need in fact that component-wise that is n α n W d α n n + W n α n W d W, W W W diagw n α n W W n α n W = W W. Proof of Theorem 5.. Recall that we needed to calculate V s, σ, n = a x a x W + s + x A n+ s + x σ + x W x + γ n+ which may be reorganized to s A n+ s σ + γ n+ + a x a x W + x A n+ W x + s A n+ x. We now apply Lemma 5. with A = A n+ W and b = A n+ s. Let λ max n+ be the largest eigenvalue of A n+ W + W = A n+. If we have then the value equals s A n+ W + λ max n+i A n+ An+ s, 5 s A n+ s σ + γ n+ + λmax n+ + s A n+ W + λ max n+i A n+ An+ s. So if we assert that V s, σ, n = s A n s σ + γ n, we find that we must have the correspondence: A n γ n = A n+ W + λ max n+i A n+ An+ + A n+, = λmax n+ + γ n+. The final piece is to check Equation 5. To see this, let ν denote the eigenvalues of W. Observe that the largest eigenvalue of A n+ W + λ max n+i A n+ An+ is, λ max n+/ν max because the mapping λ λ ν + λ max n+ λ is monotonic in λ λ max. The proof now follows by combining the recurrence for the largest eigenvalue from Section 5. with the bound on α n from the proof of Theorem 4.4: s A n+ W + λ max λ max n+i A n+ An+ s s αn+n n+ ν max n n + <.