O forward improvemet iteratio for stoppig problems Mathematical Istitute, Uiversity of Kiel, Ludewig-Mey-Str. 4, D-24098 Kiel, Germay irle@math.ui-iel.de Albrecht Irle Abstract. We cosider the optimal stoppig problem for g(z ), where Z, 1, 2... is a homogeeous Marov sequece. A algorithm, called forward improvemet iteratio, is ivestigated by which a optimal stoppig time ca be computed. Usig a iterative step, this algorithm computes a sequece B 0 B 1 B 2... of subsets of the state space such that the first etrace time ito the itersectio F of these sets is a optimal stoppig time. Keywords. Optimal stoppig, Marov sequece, forward improvemet iteratio. 1 Itroductio We recall the geeral situatio for a problem of optimal stoppig. Startig with a probability space ad a time set T [0, ], we have a filtratio (A t ) t T ad a adapted real-valued stochastic process (X t ) t T. A stoppig rule is a mappig τ : Ω T satisfyig {τ t} A t for all t T, ad we let T deote the set of all stoppig rules. We assume that EX τ exists (possibly ifiite) for all τ T. The aim is to fid a stoppig rule τ satisfyig EX τ sup EX τ τ T ad to compute this supremum, called the value of the stoppig problem. There is a wealth of literature o the theory of optimal stoppig ad its applicatios, see e.g. the recet moograph by Pesir ad Shiryaev (2006), curret research ofte triggered by the applicability i optio pricig problems of mathematical fiace. Differig from the well-ow bacwards iductio approach, the followig algorithm called forward improvemet iteratio (FII) was itroduced i Irle (1980) as a geeral theoretical tool ad was the used for fidig explicit solutios i best choice problems. For FII we let T {0, 1, 2,..., m} for m thus icludig the ifiite time case m. For ay sequece C (C ) T of C A with C m Ω we defie the stoppig rule For C we defie C by C m Ω ad τ (C)(ω) if { : ω C }. C {E(X τ+1 (C) A ) X } C, < m. FII proceeds i the followig way. Let C 0 (Ω) ad by iductio C (C 1 ). Defie D by D C. It is show i Irle (1980) that τ 0 (D) is a optimal stoppig rule uder the coditio, trivially true for m <, that EX lim σ lim EX σ for ay icreasig sequece (σ ) of stoppig rules. This approach was adapted i Irle (2006) to Marovia stoppig problems. Differet to the geeral case where FII wors i the set of sample paths, the adapted algorithm ow wors i the space set of a Marov chai which allows for efficiet umerical calculatios. We cosider T {0, 1, 2,..., } ad a homogeeous Marov process (Z ) < with respect to the uderlyig filtratio. The measurable state space is deoted by (S, S). Let g : S R be measurable. We loo at the optimal stoppig problem for X g(z ), <, X lim sup X,
2 Irle formally writig g(z ) for X i various expressios. We use P z, E z for P ( Z 0 z), E( Z 0 z) ad assume that E z g(z τ ) exists for all stoppig rules τ ad all z S. We are looig for a stoppig rule τ such that for all z S E z g(z τ ) sup E z g(z τ ) V (z), say, τ V referred to as value fuctio. FII wors i the followig way. For a measurable B S set τ (B) if {j : Z j B} ad defie B {z : g(z) E z g(z τ1 (B)} B. Let B 0 S ad by iductio B (B 1 ), furthermore F B. Uder the coditio E z g(z lim σ ) lim E z g(z σ ) for all z ad all icreasig sequeces (σ ) of stoppig rules, τ 0 (F ) is a optimal stoppig rule. The above coditio may be omitted i the case of a fiite state space S ad the the algorithm termiates after at most S steps. I the case of discoutig with 0 < α 1 we cosider the stoppig problem for X α g(z ). For α 1 this reduces to X g(z ), but o the other had, the discouted case may also be viewed as a o-discouted stoppig problem for the space-time chai (Z, ). The algorithmic step of goig from B to B is provided by B {z : g(z) E z α τ1(b) g(z τ1 (B))} B. We refer to Irle (2006) where also various examples with discoutig were treated. I these examples, the algorithmic step of goig from B to B was performed by providig umerical values for the quatities E z α τ 1(B) g(z τ1 (B)) by path wise simulatios of the Marov chai. As show theoretically ad demostrated by actual computatios, FII fids, for fiite state space S, the value fuctio ad the optimal stoppig time i a fiite umber of iteratios. I the examples of Irle (2006), this umber was surprisigly low. The purpose of this ote is twofold. Firstly we shall show that the algorithmic step may be described i terms of a liear equatio which allows the use of fast methods of umerical liear algebra ad the hadlig of very large state spaces. Secodly we shall show how the algorithm may be adapted to cotiuous time Marov chais. 2 The algorithmic step as a liear equatio We cosider a discouted Marovia stoppig problem as described i the itroductio with a fiite state space S, g : S R, 0 < α 1. We assume g 0 for α < 1 to rule out that ifiite observatio is optimal due to discoutig. Let p zy P (Z 1 y Z 0 z), y, z S. For shorter otatio we write h i (B)(z) E z α τ i(b) g(z τi (B)), z S, i 0, 1. Propositio 1 Let B S with P z (τ 0 (B) < ) 1 for all z S. The h 0 (B) is the uique solutio of h(z) g(z), z B, h(z) α y p zy h(y), z S \ B.
Forward improvemet iteratio 3 Proof. Obviously h 0 (B) fulfills h 0 (B)(z) g (z), z B, by defiitio. Furthermore h 0 (B)(z) p zy α h 0 (B)(y), z S \ B, as a well-ow cosequece of the Marov property which is ofte y deoted as the α-harmoicity of h 0 (B) o S \ B. So h 0 (B) provides a solutio. Usig the coditio P z (τ 0 (B) < ) 1, it is immediate from the α-harmoicity that the maximum of h 0 (B) is attaied at some poit i B, this fact ofte beig referred to as discrete maximum priciple. Uiqueess is a immediate cosequece of this priciple. Cosider two solutios h, h ad set f. The f ad f provide solutios correspodig to g(z) 0, z B, ad by the maximum priciple f 0, f 0, hece h h. Corollary 1 Let B 0 S, B (B 1 ), 1, 2,.... The for all, h 0 (B ) is the uique solutio of h(z) g(z), z B, h(z) α y p zy h(y), z S \ B. Furthermore h 1 (B )(z) h 0 (B )(z), z S \ B, h 1 (B )(z) α y p zy h 0 (B )(y), z B. Proof. The relatio betwee h 0 (B ) ad h 1 (B ) is obvious. Hece it is eough to show that P z (τ 0 (B ) < ) 1 for all z. For this defie F {z : g(z) V (z)}, where V is the value fuctio of the discouted stoppig problem. It is well-ow that τ 0 (F ) is a optimal stoppig time ad P z (τ 0 (F ) < ) 1 for all z, see Shiryayev (1978), Chapter 2, Theorem 4. By iductio, B F for all, hece P z (τ 0 (B ) < ) 1 for all z. This corollary shows that the iterative step may be performed by solvig h(z) g(z), z B,h(z) α p zy h(y), z S, ad the usig h(z) h(z), z S \ B, h(z) α p zy h(y), z B y y for the compariso with g(z). Usig appropriate pacages of umerical liear algebra this may be doe for large state spaces. 3 Cotiuous time Marov chais ad radom discoutig We ow treat a cotiuous time Marov chai (Ẑt) t [0, ) with a fiite state space S ad loo at the discouted optimal stoppig problem for X t α t g(ẑt), t [0, ), X lim sup X t, assumig g 0 for α < 1. It is well-ow that a optimal stoppig time exists ad is of the form τ if {t [0, ) : Ẑt B} for some B S, see Shiryayev (1978). So we oly have to loo at first etrace times i this stoppig problem. Deote the jump times of the chai by T 0 0 < T 1 < T 2 <... ad the embedded chai by Z ẐT with trasitio probabilities p zy, z, y S. Give Z 0 z 0,..., Z z, the waitig times T i T i 1, i 1,...,, are idepedet expoetially distributed with parameters λ(z 1 ),..., λ(z ). The trasitio probabilities p zy ad the parameters are computed from the Q-matrix of the chai.
4 Irle Now loo at ay a.s. fiite first etrace time τ if {t : Ẑt B}. The for D (B c ) 1 B E X τ ( )) E 1 {τt} α T g (ẐT ( ) E 1 {(Z0,...,Z ) D } α T T 1 g (Z ) ( ( )) E 1 {(Z0,...,Z ) D } g(z )E α T T 1 Z 1,..., Z ( E 1 {(Z0,...,Z ) D } g(z ) 1 ) α(z i ) ( E g(z τ ) 1 ) α(z i ), where τ if{ : Z B} ad α(z) λ(z) λ(z) log α is the momet geeratig fuctio of a expoetial distributio with parameter λ(z). So the cotiuous time problem is equivalet to a discrete time problem with radom discoutig where X g(z ) 1 α(z i ). The algorithmic step is doe i the followig way. For B S set B {z : g(z) E z (g(z τ1 (B)) τ 1 (B) 1 α(z i ))}. Agai let B 0 S, B (B 1 ) ad F B. Uder the coditio E z X lim σ lim E z X σ for all z ad all icreasig sequeces (σ ) of stoppig rules, τ 0 (F ) agai is a optimal stoppig time. We give a proof of this result i the Appedix. Writig as before h i (B)(z) E z (g(z τi (B)) it follows as i Sectio 2 that h 0 (B) is the uique solutio of τ i (B) 1 α(z i )) h(z) g(z), z B, h(z) α(z) y p zy (y), z S \ B. Furthermore h 1 (B)(z) h 0 (B)(z), z S \ B, h 1 (B)(z) α(z) y p zy h 0 (B)(y), z B. So we may use agai umerical liear algebra for FII. We ow state ad prove the validity of FII i the case of radom discoutig. Theorem 1 Cosider a Marovia stoppig problem with radom discoutig where g : S R, α : S [0, 1] are measurable fuctios. Let 1 X g(z ) α(z i ), 0, 1,..., X lim sup X. Assume that E z X τ exists for all stoppig rules τ, z S ad E z lim X σ lim E z X σ for all z S ad all icreasig sequeces (σ ) of stoppig rules. Defie B, 0, 1, 2,... ad F B as i Sectio 3. The for all z S (i) E z X τ(b 0 ) E z X τ(b 1 )... E z X τ(f ), (ii) E z X τ(f ) V (z), i.e. τ(f ) is optimal.
Forward improvemet iteratio 5 Proof. The proof uses the basic ideas of the proof of Theorem 1 i Irle (1980) adapted to the Marovia settig with discoutig. (a) Let B S, B as above. For σ T (B) set σ if{ σ : Z B } with T (B) {τ : Z τ B o {τ < }. We shall show E z X σ E z X σ from which (i) immediately follows. Let σ < τ +1 (B) 1 {σ,z / B } + σ1 {Zσ B } + 1 {σ }. Obviously σ T (B). For ay <, with a idepedet copy (Z ) of (Z ) {σ,z / B } X σ dp z This shows E z X bσ E z X σ. Next let {σ,z / B } {σ,z / B } {σ,z / B } {σ,z / B } {σ,z / B } g(z ) 1 [ E Z g E ( g(z τ+1 (B) (Z τ 1(B) ) τ 1 (B) 1 ) τ +1(B) 1 i τ +1 (B) 1 g(z τ+1 (B)) X τ+1 (B)dP z. σ 0 σ ad by iductio σ σ 1, α(z i ) ] 1 α(z i ) A ) 1 so that E z X σ E z X σ E z X σ+1... for all. We have σ σ 1 σ 2 σ. Assume that sup σ (ω) < σ (ω) for some ω. The σ (ω) σ +1 (ω) <, for some, hece Z σ (ω) B ad σ +1 (ω) σ (ω). This cotradictio implies σ lim σ, so that by assumptio E z X σ lim E z X σ E z X σ. (b) We shall ow show that for ay stoppig time σ there exists τ T (F ) such that E z X τ E z X σ. Defie σ if{ σ : Z B }, τ if{ σ : Z F }. The σ 0 σ σ 1 σ 2 τ, σ +1 (σ ). It follows from (a) that E z X σ E z X σ E z X σ +1 for all. Sice F B, we obtai lim σ τ, hece by assumptio E z X σ lim E z X σ E z X τ. (c) Let ρ, τ T (F ) such that ρ τ. We shall show E z X ρ E z X τ. Let ρ ρ1 {ρτ} + τ +1 (B )1 {ρ<τ}, ρ ρ1 {ρτ} + τ +1 (F )1 {ρ<τ}. < <
6 Irle The ρ ρ ρ +1 ρ τ, ρ lim ρ. Sice ρ T (F ) we have for ay implyig τ 1 (B ) 1 {ρ < τ} {g(z ) E Z g(z τ 1 (B ) ) α(z i)}, {ρ<τ} X ρ dp z {ρ<z} {ρ<τ} {ρ<τ} g(z ) 1 α(z i ) dp z [E Z g(z τ τ 1 (B ) ) 1 (B ) 1 α(z i 1 )] τ +1 (B ) 1 g(z τ+1 (B )) α(z i ) dp z α(z i ) dp z {ρ<τ} ρ dp z, hece E z X ρ E z X ρ. Lettig we obtai E z X ρ E z X ρ. Defie ρ 0 ρ, ad ρ (ρ 1 ) for 1. The obviously ρ ρ ρ +1, ρ T (F ), ad lim ρ τ. It follows E z X ρ lim E z X ρ E z X τ. (d) Let σ be ay stoppig time. The by (b) there exists τ T (F ) such that E z X σ E z X τ. By defiitio τ 0 (F ) τ, hece by (c) E z X τ0 (F ) E z X τ E z X σ. Refereces Irle, A. (1980). O the Best Choice Problem with Radom Populatio Size. Zeitschrift f. Operatios Research, 24, 177 190. Irle, A. (2006). A Forward Algorithm for Solvig Optimal Stoppig Problems. J. Appl. Probab., 43, 102 113. Pesir, G. ad Shiryaev, A.N. (2006). Optimal Stoppig ad Free-Boudary Problems. Birhaeuser, Basel. Shiryaev, A.N. (1978). Optimal Stoppig Rules. Spriger, New Yor.