Self-normalized Martingale Tail Inequality

Transcription

1 Online-to-Confience-Set Conversions an Application to Sparse Stochastic Banits A Self-normalize Martingale Tail Inequality The self-normalize martingale tail inequality that we present here is the scalar-value version of the more general vector-value results obtaine by Abbasi-Yakori et al. 2011b,a. We inclue the proof for completeness. Theorem 7 Self-normalize boun for martingales. Let F t be a filtration. Let τ be a stopping time w.r.t. to the filtration F t+1 i.e. the event τ t belongs to F t+1. Let Z t be a sequence of real-value variables such that Z t is F t -measurable. Let η t be a sequence of real-value ranom variables such that η t is F t+1 -measurable an is conitionally R-sub-Gaussian. Let > 0 be eterministic. Then, for any > 0, with probability at least 1, τ η tz t 2 τ + + τ R 2. Proof. Pick λ R an let ηt λz t exp R 1 2 λ2 Zt 2, t S t η s λz s, D λ t M λ t exp λs t R 1 2 λ2 t Z 2 t. We claim that Mt λ is an F t+1 -aapte supermartingale. That Mt λ the efinitions. By sub-gaussianity, EDt λ F t ] 1. Further, F t+1 for t 1, 2,... is clear from EM λ t F t ] EM λ D λ t F t ] showing that M t is inee a supermartingale. M λ ED λ t F t ] M λ, Next we show that Mτ λ is always well-efine an EMτ λ ] 1. First efine M Mτ λ an note that Mω Mτω λ ω. Thus, when τω, we nee to argue about M ω. λ By the convergence theorem for nonnegative supermartingales, lim t Mt λ ω is well-efine, which means Mτ λ is well-efine, inepenently of whether τ < hols or not. Now let Q λ t Mminτ,t λ be a stoppe version of M t λ. We procee by using Fatou s Lemma to show that EMτ λ ] Elim inf t Q λ t ] lim inf t EQ λ t ] 1. Let F be the σ-algebra generate by F t i.e. the tail σ-algebra. Let Λ be a zero-mean Gaussian ranom variable with variance 1/ inepenent of F. Define M t EMt Λ F ]. Clearly, we still have EM τ ] EMτ Λ ] EEMτ Λ ] Λ] E1 Λ] 1. 1 Let us calculate M t. We will nee the ensity λ which is fλ e λ2 /2. Now, it is easy to write M t 2π/ explicitly M t EM Λ t F ] 2π exp M λ t fλ λ exp λs t R λ2 2 S 2 t 2R 2 + t t where we have use that expaλ bλ2 expa 2 /4b π/b. Z 2 t e λ2 /2 λ + t,

2 Yasin Abbasi-Yakori, Dávi Pál, Csaba Szepesvári To finish the proof, we use Markov s inequality an the fact that EM τ ] 1: τ Pr η tz t 2 ] + + τ 2R 2 Sτ 2 + Pr 2R 2 + τ Sτ 2 Pr exp 2R 2 + τ Pr M τ 1 ]. ] + ] The theorem can be bootstrappe to a stronger statement or at least one, that looks stronger at the first sight that hols uniformly for all time steps t as oppose to only a particular stopping time τ. The iea of the proof goes back at least to Freeman Corollary 8 Uniform Boun. Uner the same assumptions as the previous theorem, for any > 0, with probability at least 1, for all n 0, n η t Z t R Zt 2. Proof. Define the ba event B t ω Ω : t η sz s 2 + t Z2 s > 2R 2 + t Z2 s. We are intereste in bouning the probability that t 0 B t happens. Define τω mint 0 : ω B t, with the convention that min. Then, τ is a stopping time. Further, B t ω : τω <. Thus, by Theorem 7 it hols that t 0 Pr B t Pr τ < ] t 0 Pr Pr. τ η tz t 2 + τ τ η tz t 2 + τ ] + > 2R 2 an τ < ] + > 2R 2 B Some Useful Tricks Proposition 9 Square-Root Trick. Let a, b 0. If z 2 a + bz then z b + a.

3 Online-to-Confience-Set Conversions an Application to Sparse Stochastic Banits Proof of the Proposition 9. Let qx x 2 bx a. The conition z 2 a + bz can be expresse as qz 0. The quaratic polynomial qx has two roots x 1,2 b ± b 2 + 4a. 2 The conition qz 0 implies that z x 1, x 2. Therefore, z x 1, x 2 b + b 2 + 4a 2 where we have use that u + v u + v hols for any u, v 0. c+f b + a, Proposition 10 Logarithmic Trick. Let c 1, f > 0, 0, 1/4]. If z 1 an z c + f z/ then z c + f 2. Proof of the Proposition 10. Let gx x c f x/ for any x 1. The conition z c + f z/ can be expresse as gz 0. For large enough x, the function gx is increasing. This is easy to see, since g x 1. Namely, it is not har see gx is increasing for x 1, f/2 since for any such x, g x is positive. Clearly, c + f f 2x x/ 2 c+f 1, f/2 since c 1 an 0, 1/4]. Therefore, it suffices to show that c + f g c + f 2 0. This is verifie by the following calculation c + f c + f g c + f 2 c + f 2 c f c + f 2 c + f/ c + f f 2 f c + f 2 c + f/ 2 c + f f f c + f 2 c + f/ 2 c + f f f c + f 2 c + f/ f A 2 f A 2 A 0, where have efine A c + f/ an the last inequality follows from that A 2 A 2 A for any A > 0. C Proof of Theorem 3 In this section we will nee the following notation. For a given positive efinite matrix A R we enote by x, y A x Ay the inner prouct between two vectors x, y R inuce by A. We enote by x A x, xa x Ax the corresponing norm. The following lemma is a from Dani et al We reprouce the proof for completeness.

4 Yasin Abbasi-Yakori, Dávi Pál, Csaba Szepesvári Lemma 11 Elliptical Potential. Let x 1, x 2,..., x n R an let t I + t x s x s for t 0, 1, 2,..., n. Then it hols that min 1, x t 2 Furthermore, if x t 2 X for all t 1, 2,..., n then et n et n. 1 + nx2. Proof of Lemma 11. We use the inequality x 1 + x vali for all x 0, 1]: n min 1, x t x t x t 2. We now show that et n n 1 + x t 2 : et n et n + x n x n et n I + /2 n x n /2 n x n et n et I + /2 n x nn /2 x n et n 1 + x n 2 n n 1 + x t 2. since 0 I In the above calculation we have use that eti + zz 1 + z 2 2 since all but one eigenvalue of I + zz equals to 1 an the remaining eigenvalue is 1 + z 2 2 with associate eigenvector z. To prove the secon part, consier the eigenvalues α 1, α 2,..., α of n. Since n is positive efinite, the eigenvalues are positive. Recall that et n i1 α i. The boun on x t X implies a boun on the trace of n : Trace n TraceI + Tracex t x t + x t nx 2. Recalling that Trace n i1 α i we can apply the AM-GM inequality: α1 α 2 α α 1 + α α Trace n from which the secon inequality follows by taking logarithm an multiplying by. Proof of Theorem 3. Consier the event A when θ t0 C t. By Corollary 2, the event A occurs with probability at least 1. The set C is an ellipsoi unerlying the covariance matrix I + X s X s an center θ t argmin θ Ŷs θ, X s 2. θ R The ellipsoi C is non-empty since θ lies in it on the event A. Therefore θ t C. We can thus express the ellipsoi as C θ R : θ θ t θ θ 2 t + θ t Ŷs θt, X s β.,

5 Online-to-Confience-Set Conversions an Application to Sparse Stochastic Banits The ellipsoi is containe in a larger ellipsoi C θ R : θ θ t θ θ t β θ R : θ θ t β. First, we boun the instantaneous regret using that X t, θ t arg x,θ Dt C x, θ: x X t, θ x, θ X t, θ X t, θ t X t, θ X t, θ t θ X t, θ t θ t X t, θ t θ X t, θ t θ t + X t, θ t θ θ t θ t + X t X t θ t θ Cauchy-Schwarz β X t. because θ t, θ C Since we assume that x, θ G for any x D t an any t 1, 2,..., n, we can upper boun x X t, θ 2 ming, β X t. Summing over all t we upper boun regret R n x X t, θ min G, β X t β min G, X t βt n min G, X t βt 1, G βt 1, G 1, G 2n log min n n 1, X t 1 + nx2 β t, min 1, X t 2 since β 1 Cauchy-Schwarz where the last inequality follows from Lemma 11. Proof of Theorem 4. Summing over all t we upper boun regret R n x X t, θ 1 x X t, θ 2, where the last inequality follows from the fact that either x X t, θ 0 or x X t, θ >. Then we take

6 Yasin Abbasi-Yakori, Dávi Pál, Csaba Szepesvári similar steps as in the proof of Theorem 3 to obtain R n 1 x X t, θ 2 4 β t 1, G 2 8 β t 1, G 2 log finishing the proof of the problem epenent boun. min 1, X t nx2,