Universal Confidence Sets for Solutions of Optimization Problems

Transcription

1 Universal Confidence Sets for Solutions of Optimization Problems Silvia Vogel Abstract We consider random approximations to deterministic optimization problems. The objective function and the constraint set can be approximated simultaneously. Relying on concentration-of-measure results we derive universal confidence sets for the constraint set, the optimal value and the solution set. Special attention is paid to solution sets which are not single-valued. Many statistical estimators being solutions to random optimization problems, the approach can also be employed to derive confidence sets for constrained estimation problems. Keywords: random optimization problems, universal confidence sets, convergence rate, constrained estimation MSC2000: 90C15, 90C31, 62F25, 62F30 1 Introduction Random approximations of deterministic or random optimization problems come into play if unknown quantities are replaced with estimates or for numerical reasons. Qualitative stability results, which make assertions on the (semi-)convergence of the constraint sets, the optimal values, and the solution sets, are available for convergence almost surely, in probability and in distribution, c.f. [11], [16], [7], [6], [18], [2]. Furthermore, there are quantitative results which estimate the distance between the optimal values and/or solutions sets by suitable probability metrics, see [13] for an overview. Confidence bounds for optimal values and solution sets provide valuable additional information. In the traditional way confidence bounds are derived from the distribution of the statistic under consideration. However, the exact distribution is available only in rare cases. Therefore, often knowledge about the limit distribution is used as a surrogate. Hence qualitative stability 1

2 results for convergence in distribution can be employed to derive asymptotic confidence sets, see [11] and [2]. In [12] Pflug suggests an approach which can be used to derive even nonasymptotic confidence sets without knowledge about the exact distribution. The starting point are assertions on convergence in probability plemented with a suitable convergence rate and tail behavior. In [12] uniform convergence with known convergence rate and tail behavior of the objective functions together with a growth condition are assumed and results on inner approximations of the solution sets are proved. In this way non-asymptotic confidence sets can be obtained if the solution set is single-valued. We will pursue the way proposed in [12] farther, take into account also the approximation of the constraint set, and show how one can proceed if the solution set is not single-valued. The results are formulated in a general way, allowing e.g. for ε n -relaxed constraint sets and ε n -optimal solutions. We will also assume suitable assertions on the (one-sided) uniform convergence in probability of the objective functions and/or the constraint functions with a convergence rate and tail function are available, albeit these conditions are rather restrictive. It is, however, also possible to derive similar, yet more technical assertions if one assumes some kind of continuous convergence in probability with convergence rate and tail behavior. This so-called pointwise approach opens the possibility to employ directly suitable concentration inequalities. This method will be investigated elsewhere. The results will be illustrated by three examples. Firstly, we discuss a simple example in order to show how one can deal with the uniform convergence assumptions. Secondly, we consider the approximation for a chance constraint. The third example was chosen to emphasize the applicability of our results in statistics. Many statistical estimators being solutions to random optimization problems, our assertions can be employed to derive confidence bounds for estimators, even if one does not have full knowledge about the distribution of the underlying statistic. We will provide universal confidence bounds for quantiles, allowing for distribution functions which are not continuous. The paper is organized as follows. In Chapter 2 we introduce the mathematical model and show how universal confidence sets can be derived. In Chapter 3 and Chapter 4 we prove the needed convergence assertions for the constraint sets, the optimal values, and the solutions sets. Chapter 5 contains the examples. 2

3 2 Universal confidence sets Let (E, d) be a complete separable metric space and [Ω, Σ, P ] a complete probability space. We assume a deterministic optimization problem (P 0 ) min x Γ 0 f 0 (x) is approximated by a sequence of random problems (P n ) min x Γ n (ω) f n(x, ω), n N. Additionally to (P n ), for a given ε > 0, we consider so-called ε-relaxations (P ) min x Γ (ω) f (x, ω), n N. The relaxed problems offer the possibility to deal with approximate constraint sets, objective functions and/or solution sets. Consequently, the approach can be applied, e.g., to constraint sets and functions which are obtained by Monte Carlo methods (c.f. [14], [10]) or to methods which use plug-in estimators. Moreover, the relaxed problems occur in a natural manner if we aim at deriving outer approximations of constraint sets and solution sets. The following results will be formulated for (P ). The problem (P n ) is then regarded as a special case of (P ) with objective functions and constraint sets do not depend on ε. Γ 0 is a nonempty closed subset of E, and f 0 E R 1 is a lower semicontinuous function. Γ Ω 2 E are closed-valued measurable multifunctions, and f E Ω R 1 are lower semicontinuous random functions which are posed to be (B(E) Σ, B 1 )-measurable. B(E) denotes the Borel-σ-field of E and B 1 the σ-field of Borel sets of R1. The measurability conditions imposed here do not have the weakest form. We use them for sake of simplicity. They are satisfied in many applications and guarantee all functions of ω needed in the following have the necessary measurability properties. Furthermore, it should be mentioned the lower semicontinuity assumption of the objective functions f can be dropped. Imposing this condition, however, we can omit some technical details in the proofs. Eventually, we assume all objective functions are (almost surely) proper functions, i.e. functions with values in (, + ] which are not identically. Our main concern will be with the solution sets Ψ 0 of (P 0 ) and Ψ of (P ). We aim at proving assertions of the form ε > 0 : P {ω : Ψ (ω) \ U β Ψ 0 } K(ε) (1) 3

4 and ε > 0 : P {ω : Ψ 0 \ U β Ψ (ω) } K(ε). Here (β ) is a sequence of nonnegative numbers which tends to zero for each ε > 0 and K R + R + is a function with lim K(ε) = 0. U α X ε denotes an open neighborhood of the set X with radius α: U α X := {x E : d(x, X) < α}. Because of the similarity with inner approximations in probability of sequences of random sets (c.f. [7], [19]) we will call a sequence (Ψ ) fulfilling the first relation an inner approximation in probability to Ψ 0 with convergence rate β and tail behavior function K (in short, an inner (β, K)- approximation). Correspondingly a sequence (Ψ ) fulfilling the second relation will be called an outer approximation in probability to Ψ 0 with convergence rate β and tail behavior function K (in short, an outer (β, K)- approximation). In order to derive universal confidence sets to the level ε 0, i.e. a sequence of random sets (C n ) with P {ω : Ψ 0 \ C n (ω) } ε 0, we can n n 0 proceed as follows: Suppose an outer (β, K)-approximation (Ψ ) to Ψ 0 is available and choose, to ε 0 > 0, an ε > 0 such K(ε) ε 0. The set C n := U β Ψ has the desired property. Of course, one is interested in small confidence sets, hence (β ) should go to zero as fast as possible and K should converge to zero as fast as possible if ε tends to infinity. If one considers approximating problems (P n ) without relaxation, then under reasonable conditions one obtains inner approximations only. This is satisfying if all approximating problems (P n ) have solutions which are uniformly bounded and the solution set to the problem (P 0 ) is single-valued, because in this case inner approximations are also outer approximations and we can proceed as above. What can be done if the solution set to (P 0 ) is not single-valued? Taking into account we need knowledge about convergence rates anyway, we can exploit this knowledge to determine suitable relaxing sequences (κ ), which tend to zero for each ε > 0 and consider κ -optimal solutions Ψ κn. 4

5 The problem,, without relaxing, only inner approximations can be obtained is also apparent for the constraint sets. There are problems, where κ -relaxations are needed to obtain outer approximations of the constraint sets. An example will be considered in Chapter 5. In order to obtain reasonable confidence sets one would like to have lim n β(i) = 0 and lim ε K i 0 for all sequences (β (i) ) and functions K i which occur in the following. As mentioned, in order to obtain small confidence sets, the limits should go to zero as fast as possible. These properties are, however, not needed to prove the results in Chapter 3 and Chapter 4. We only assume throughout the paper the sequences (β ) (i) are non-increasing sequences of positive numbers and the functions K i R + R + are non-increasing. As a by-product, from the results presented in the following, one can also derive confidence sets for the optimal values, proceeding as described above for the solution sets. Hence the approach can help to assess the quality of a solution to an approximate optimization problem and may be of interest also for model selection. 3 Approximation of the constraint set In this section we consider constraint sets, which are given by inequality constraints, and their approximations. Results of kind are of course needed if approximation of the constraint set is inherent in the problem under consideration. Moreover, the statements will be employed to derive assertions on the behavior of the solution sets, regarding the difference between the true objective function and the optimal value as constraint function. As the objective values for problems which κ -relaxed constraint sets may depend on ε, the resulting constraint function may depend on ε, too. Furthermore, when applying Theorem 1 below to the solution sets, the multifunctions Q n will be interpreted as constraint sets, hence we have to allow they depend on ε, too. Consequently, we have to make sure, even the results for the constraint sets hold for relaxed constraint functions and multifunctions. Let Q 0 a closed non-empty subset of E and J = {1,..., j M } a finite index set. We consider functions g0 E j R 1, j J, which are lower semicontinuous in all points x E, and define Γ 0 := {x : g0(x) j 0, j J} Q 0. Γ 0 is assumed to be nonempty. For each ε > 0, the set Q 0 is approximated by a sequence (Q ) of 5

6 closed-valued measurable multifunctions, and the functions g j 0, j J, are approximated by sequences (g j ) of functions g j E R 1, j J, which are (B(E) Σ, B 1 )-measurable. Furthermore, we assume the functions g (, ω) are lower semicontinuous for all ω Ω. Also these measurability and semicontinuity properties could be weakened. Eventually Γ is defined by Γ (ω) := {x E : g j (x, ω) 0, j J} Q (ω). Under our assumptions Γ is a closed-valued measurable multifunction. If we have Q 0 = Q (ω) E, we will use the denotation ˆΓ 0 and ˆΓ, respectively: ˆΓ0 = {x E : g j 0(x) 0, j J} and ˆΓ (ω) := {x E : g j (x, ω) 0, j J}. In the following we use functions ν, µ and λ. We assume they map R + into R +, fulfill ν(0) = µ(0) = λ(0) = 0, are increasing and non-constant. By the erscript 1 we denote their right inverses: ν 1 (y) := inf{x R : ν(x) > y}. Theorem 1 (Inner Approximation of the Constraint Set) Assume the following conditions are satisfied: (CI1) There exist a function K 1 and to all ε > 0 a sequence (β (1) P {ω : Q (ω) \ U Q (1) β 0 } K 1 (ε). ) (CI2) There exist a function K 2 and to all ε > 0 a sequence (β P {ω : inf (g(x, j ω) g0(x)) j β } K 2 (ε) j J x UQ 0 \Γ 0 for a suitable neighborhood UQ 0. ) such such (CI3) There exists a function ν such for all ε > 0 U ε Γ 0 U ν(ε) Q 0 U ν(ε)ˆγ0. (CI4) There exists a function µ such for all ε > 0 x U ε Q 0 \ U εˆγ0 j J : g j 0(x) µ(ε). Then for all ε > 0, β = max{ν 1 (β ), (1) ν 1 (µ 1 (β ))}, and n 0 (ε) = min{k : U β Q 0 UQ 0 } the relation k,ε 6

7 P {ω : Γ (ω) \ U Γ β 0 } K 1 (ε) + j M K 2 (ε) Proof. Assume for given ε > 0, n n 0 (ε), and ω Ω the relation P {ω : Γ (ω) \ U Γ β 0 } Then there is x (ω) Γ (ω) which does not belong to U β Γ 0. Because of (CI3) we have x (ω) / U ν(β ) Q 0 or x (ω) U ν(β ) Q 0 and x (ω) / U ν(β )ˆΓ 0. In the first case we obtain Q \ U Q ν(β 0. Hence, because of ν(β ) β, (1) ) Q (ω) \ U Q (1) β 0, and we can employ (CI1). In the second case we obtain by (CI4) for at least one j J, g0(x j (ω)) µ(ν(β )) β, hence, because of U Q ν(β 0 UQ 0, inf (g ) (x, j ω) g0(x)) j β. It remains x UQ 0 \Γ 0 to employ (CI2). The conditions (CI2) and (CI4) can be replaced using strict inequalities in the following form without changing the conclusion of Theorem 1. This holds correspondingly for further assertions. (CI2-s) There exist a function K 2 and to all ε > 0 a sequence (β ) such P {ω : j J for a suitable neighborhood UQ 0. inf (g(x, j ω) g0(x)) j < β } K 2 (ε) x UQ 0 \Γ 0 (CI4-s) There exists a function µ such for all ε > 0 x U ε Q 0 \ U εˆγ0 j J : g0(x) j > µ(ε). If the set Q 0 remains fixed in all approximations, i.e. Q Q 0, the assumptions in Theorem 1 can be weakened. Corollary 1.1 Assume the following assumptions are satisfied: (CI2-W) There exist a function K 2 and to all ε > 0 a sequence (β P {ω : inf (g(x, j ω) g0(x)) j β } K 2 (ε). j J x Q 0 \Γ 0 ) such 7

8 (CI4-W) There exists a function µ such for all ε > 0 x Q 0 \ U εˆγ0 j J : g0(x) j µ(ε). Then for all ε > 0 and β = µ 1 (β ) the relation P {ω : Γ (ω) \ U β Γ 0 } j M K 2 (ε) Proof. We can choose ν(ε) = ε. (CI1) is satisfied with K 1 0 and an arbitrary β. (1) As we are interested in small values of β, we use the form given in the assertion. If we want to employ Theorem 1 for solution sets we have to deal with one constraint function only. The following corollary is a specialization of Theorem 1 to j M = 1 and g0 1 =: g 0, g 1 =: g. Additionally, we replace (CI4) with a growth condition. Corollary 1.2 Assume (CI1), (CI2), (CI3), and the following condition (Gr-g 0 ) are satisfied. (Gr-g 0 ) There exist an increasing function ψ 1 R + R + and constants c 1 > 0, δ 1 > 0, and θ 1 > 0 such x UQ 0 : g 0 (x) ψ 1 (d(x, ˆΓ 0 )) and 0 < θ < θ 1 : ψ 1 (θ) c 1 θ δ 1. Then for all ε > 0, β = max{ν 1 (β (1) n 0 (ε) = min{k : U β k,ε ), ν 1 (( β δ 1 )}, and Q 0 U θ1 Q 0 UQ 0 } the relation c 1 ) 1 P {ω : Γ (ω) \ U β Γ 0 } K 1 (ε) + K 2 (ε) If the functions g do not depend on ε, Q Q 0 and β = ε γ n instead of (CI2) the following condition can be used: (CI2 ) There exist a function K 2 and a sequence (γ n ) such P {ω : γ n ( inf (g n (x, ω) g 0 (x)) ε} K 2 (ε). x Q 0 \Γ 0 In this case Corollary 1.2 can be simplified in the following way: holds, 8

9 Corollary 1.3 Assume (CI2 ) and (Gr-g 0 ) with Q 0 instead of UQ 0 are satisfied. Then for all ε > 0 the relation P {ω : Γ n (ω) \ U ε Γ 1 0 } K 2 (c 1 ε δ 1 ) (γn) δ 1 Proof. With Corollary 1.1 and Corollary 1.2 we obtain P {ω : Γ n (ω) \ U Γ β 0 } K 2 (ε) with β = ( ε c 1 γ n ) 1 δ 1. Replacing ε with c 1 η δ 1 yields the conclusion. As mentioned in the introduction, in general, the sequence (Γ n ) approximates a subset of Γ 0 only. In order to obtain outer approximations, additional assumptions have to be imposed. Qualitative stability theory usually assumes the condition Γ 0 cl{x Q 0 : g0(x) j < 0, j J} is fulfilled where cl denotes the closure. In order to obtain also a convergence rate and a tail function, we impose a quantified version of this assumption, see (CO3) below. Unfortunately, a condition of kind is useless if one intends to employ the result for the solution set, because (CO3) can not be satisfied by g n := f n Φ n where Φ n denotes the optimal value to the problem (P n ). Hence we will provide a second approach which considers inequality constraints of the form g(x, j ω) < κ, j J, where (κ ) is a suitable sequence of positive reals with lim κ = 0 ε > 0. n For the formulation of (CO3) we need the ε-interior of Γ 0. Let, for a given ε > 0, CI(ε) := Γ 0 \ U ε (E \ Γ 0 ). Ū ε denotes the closure of U ε. Theorem 2 (Outer Approximation of the Constraint Set, Inner Point) Assume the following conditions are satisfied: (CO1) There exist a function K 1 and to all ε > 0 a sequence (β (1) P {ω : Q 0 \ U Q (1) β (ω) } K 1 (ε). ) such (CO2) There exist a function K 2 and to all ε > 0 a sequence (β P {ω : (g(x, j ω) g0(x)) j β } K 2 (ε). j J x Γ 0 ) such 9

10 (CO3) There exist ε > 0 and a function µ such for all 0 < ε ε Γ 0 ŪεCI(ε) and x CI(ε) j J : g j 0(x) µ(ε). Then for all 0 < ε ε, β = max{β (1), µ 1 (2β n 0 (ε) = min{k : β k,ε P {ω : Γ 0 \ (U β )}, and 2 ε} the relation ˆΓ (ω) U Q (1) β (ω)) } K 1 (ε) + j M K 2 (ε) Proof. Assume for given ε > 0, n N, and ω Ω the relation Γ 0 \(U β ˆΓ (ω) U Q (1) β (ω)) Then there is x (ω) Γ 0 which does not belong to U β ˆΓ (ω) U Q (1) β (ω). If x (ω) / U Q (1) β (ω) we can employ (CO1). If x (ω) / U β ˆΓ (ω) we can choose x (ω) CI( β 2 ) with x (ω) / ˆΓ (ω), i.e. g j 0 ( x (ω), ω) > 0 for at least one j 0 J. Because of g j 0 0 ( x (ω)) µ( β ) 2 β we can employ (CO2). Now we consider κ -relaxed inequality constraints. Let ˆΓ κ n (ω) := {x E : g(x, j ω) < κ, j J} and Γ κ n (ω) = ˆΓ κ n (ω) Q. Theorem 3 (Outer Approximation of the Constraint Set, Relaxation) Assume (CO1) and (CO2) are satisfied. Then for all ε > 0, κ = β and β = max{β, (1) β } the relation P {ω : Γ 0 \ (ˆΓ κn (ω) U β (1) Q (ω)) } K 1 (ε) + j M K 2 (ε) Proof: Assume for given ε > 0, n N, and ω Ω the relation Γ 0 \ (ˆΓ κn (ω) U Q (1) β (ω)) Then there is x (ω) Γ 0 which does not belong to ˆΓ κ n (ω) U Q (1) β (ω). Hence g j 0(x (ω)) 0 j J and x (ω) Q 0 (ω), but either x (ω) / U Q (1) β (ω) or g j (x (ω), ω) > β = κ for at least one j J. In the first case we obtain Q 0 \ U Q (1) β (ω). The second case yields (g(x, j ω) g0(x)) j > β for at least one j J. x Γ 0 10

11 A corresponding result holds if U β (1) Q is replaced with Q in the condition (CO1) and in the assertion. The question arises under what conditions (Γ κ n ) is also an inner approximation. Results of kind will help to assess the quality of an outer approximation. An inspection of the proof to Theorem 1 shows with κ = β the following statement can be obtained. Theorem 4 (Inner Approximation of the Constraint Set, Relaxation) Assume (CI1), (CI2), (CI3), and (CI4) are satisfied. Then for all ε > 0, κ = β, β = max{ν 1 (β ), (1) ν 1 (µ 1 (2β ))}, and n 0 (ε) = min{k : Q 0 UQ 0 } the relation U β k,ε ε > 0 P {ω : Γ κ n (ω) \ U Γ β 0 } K 1 (ε) + j M K 2 (ε) Finally, we will summarize what we obtain for one constraint function under a growth condition. Theorem 5 (Approximation of the Constraint Set, Relaxation) Assume j M = 1 and (CI1), (CO1) (CI2), (CO2), (CI3), and (Gr-g 0 ) are satisfied. Then for all ε > 0, κ = β, β = max{β, (1) β, ν 1 (β ), (1) ν 1 (( 2β c 1 ) 1 δ 1 )}, and n 0 (ε) = min{k : U β Q 0 UQ 0 } the relation k,ε P {ω : (Γ κn (ω) \ U Γ β 0) (Γ 0 \ (ˆΓ κn (ω) Q (ω)) } 2K 1 (ε) + 2K 2 (ε) Proof. We have P {ω : (Γ κ n (ω) \ U Γ β 0) (Γ 0 \ (ˆΓ κ n (ω) Q (ω)) } P {ω : Γ κn (ω) \ U Γ β 0 } + P {ω : Γ 0 \ (ˆΓ κn (ω) Q (ω)) }. The assumptions of Theorem 3 and Theorem 4 are satisfied with µ(ε) = c 1 ε δ 1 (1) and β = β = β. Unfortunately, the result is not symmetric. In order to derive a result of the form 11

12 P {ω : (Γ κn (ω) \ U Γ β 0) (Γ 0 \ U β Γκn (ω)) } K(ε) we would need functions ν and conditions similar to (CI3) for each n. Now assume there is only one constraint function g n, which does not depend on ε, and Q Q 0 If β = ε γ n, instead of (CO2) the following condition can be used: (CO2 ) There exist a function K 2 and a sequence (γ n ) such P {ω : γ n (g n (x, ω) g 0 (x)) ε} K 2 (ε). x Γ 0 Then the following result can be derived. Corollary 5.1 Assume (CI2 ), (CO2 ), and (Gr-g 0 ) with Q 0 instead of UQ 0 are satisfied. Then for all ε > 0, κ = ε γ n, β = max{ ε γ n, ( 2ε c 1 γ n ) 1 δ 1 }, and n 0 (ε) = min{k : β k,ε θ 1} the relation P {ω : (Γ κ n (ω) \ U Γ β 0) (Γ 0 \ Γ κ n ) (ω)) } 2K 2 (ε) Proof. The proof follows by Theorem 5 proceeding in a similar way as in the derivation of Corollary 1.3. If max{ ε γ n, ( 2ε 1.3. c 1 γ n ) 1 δ 1 } = ( 2ε c 1 γ n ) 1 δ 1 the result can be rewritten as in Corollary 4 Approximation of the optimal values and the solution sets We turn to the optimal values and the solutions sets of the problems (P 0 ) and (P ). Let Φ (ω) := inf f (x, ω) and Ψ (ω) denote the corresponding solution set. In the following, the constraint sets and their approximations x Γ (ω) are not posed the have a special form. Especially, Γ can have the form which was used in Chapter 3, but it can also denote a set originating from a relaxation like Γ κn. 12

13 We do not impose compactness conditions on Γ 0 and Γ. Instead we assume, for sake of simplicity, the original and the approximating problems have a solution. Theorem 6 (Lower Approximation of the Optimal Value) Assume the following conditions are satisfied: (VL1) There exist a function K 1 and to all ε > 0 a sequence (β (1) P {ω : Γ (ω) \ U Γ (1) β 0 } K 1 (ε). ) such (VL2) There exist a function K 2 and to all ε > 0 a sequence (β ) such P {ω : inf (f (x, ω) f 0 (x)) β } K 2 (ε) for a suitable x UΓ 0 neighborhood UΓ 0. (VL3) There exists a function λ such for all ε > 0 x U λ(ε) Γ 0 UΓ 0 : f 0 (x) Φ 0 ε. Then for all ε > 0, β = max{2λ 1 (β (1) n 0 (ε) = min{k : Uλ( β k,ε ), 2β }, and Γ 0 UΓ 0 } the relation 2 ) P {ω : Φ (ω) Φ 0 β } K 1 (ε) + K 2 (ε) Proof. Assume for given ε > 0, n n 0 (ε), and ω Ω the relation Φ (ω) Φ 0 β Then there exists x (ω) Γ (ω) such f (x (ω), ω) = Φ (ω) Φ 0 β. Firstly, let x (ω) Γ Uλ( β 0. Then 2 ) inf (f (x, ω) f 0 (x)) f (x (ω), ω) f 0 (x (ω)) Φ (ω) Φ 0 + β x UΓ 0 β β 2. Eventually, if x (ω) / Γ Uλ( β 0, we have Γ (ω) \ U Γ (1) β 2 ) 0. The proof shows also the following assertion 2 13

14 Corollary 6.1 Assume Γ Γ 0 and the following condition is satisfied: (VL2 ) There exist a function K 2 and to all ε > 0 a sequence (β P {ω : inf (f (x, ω) f 0 (x)) β } K 2 (ε). x Γ 0 Then for all ε > 0 the relation P {ω : Φ (ω) Φ 0 β } K 2 (ε) ) such In the following we consider upper approximations and distinguish two cases according to whether ε > 0 : P {ω : Γ 0 \ U Γ (1) β (ω) } K 1 (ε) or ε > 0 : P {ω : Γ 0 \ Γ (ω) } K 1 (ε) is imposed. As expected, in the second case we obtain a better tail behavior. Theorem 7 (Upper Approximation of the Optimal Value I) Assume the following conditions are satisfied: (VU1) There exist a function K 1 and to all ε > 0 a sequence (β (1) P {ω : Γ 0 \ U Γ (1) β (ω) } K 1 (ε). ) such (VU2) There exist a function K 2 and to all ε > 0 a sequence (β ) such P {ω : (f (x, ω) f 0 (x)) β } K 2 (ε) for a suitable x UΨ 0 neighborhood UΨ 0. (VU3) There exists a function λ such for all ε > 0 x U λ(ε) Ψ 0 UΨ 0 : f 0 (x) Φ 0 + ε. Then for all ε > 0, β = max{2λ 1 (β (1) n 0 (ε) = min{k : Uλ( β k,ε ), 2β }, and Ψ 0 UΨ 0 } the relation 2 ) P {ω : Φ (ω) Φ 0 β } K 1 (ε) + K 2 (ε) 14

15 Proof. Assume for given ε > 0, n n 0 (ε), and ω Ω the relation Φ (ω) Φ 0 + β Then there exists x (ω) Γ (ω) such f (x (ω), ω) = Φ (ω) Φ 0 + β. To x (ω) we select x (ω) Γ (ω) such d( x (ω), Ψ 0 ) = min d(x, Ψ 0). x Γ (ω) Firstly, assume x (ω) Ψ Uλ( β 0. Then 2 ) f ( x (ω), ω) Φ (ω) Φ 0 + β f 0 ( x (ω)) + β and consequently, (f (x, ω) f 0 (x)) β β 2. If x (ω) / x UΨ 0 Ψ Uλ( β 0, we have Γ 0 \ U Γ (1) β (ω). 2 ) If we impose the special upper semicontinuity condition (UCon) for f 0, we obtain the following corollary. Corollary 7.1 (Upper Approximation of the Optimal Value I) Assume (VU1), (VU2) and the following condition (UCon) are satisfied: (UCon) There exist an increasing function ψ 2 R + R + and constants c 2 > 0, δ 2 > 0 and θ 2 > 0 such x UΨ 0 : f 0 (x) Φ 0 + ψ 2 (d(x, Ψ 0 )) and 0 < θ < θ 2 : ψ 2 (θ) c 2 θ δ 2. Then, for all ε > 0, β = max{2c 2 (β (1) ) δ 2, 2β }, and n 0 (ε) = min{k : U λk,ε Ψ 0 UΨ 0 U δ2 Ψ 0 } with λ k,ε = ( β k,ε the relation P {ω : Φ (ω) Φ 0 β } K 1 (ε) + K 2 (ε) 2 2c 2 ) 1 δ 2 Proof. We employ Theorem 7. Because of (UCon) we can choose λ(θ) = ( θ c 2 ) 1 δ 2 and consequently λ 1 (ε) = c 2 ε δ 2. A similar Corollary can be proved for the lower approximation of the optimal values. We give only the result for the upper approximation because we will use it in the following. Furthermore, the assertions can be plemented by results similar to Corollary 1.3 and Corollary

16 Theorem 8 (Upper Approximation of the Optimal Value II) Assume the following conditions are satisfied: (VU1-R) There exist a function K 1 and to all ε > 0 a sequence (β (1) P {ω : Γ 0 \ Γ (ω) } K 1 (ε). (VU2-R) There exist a function K 2 and to all ε > 0 a sequence (β P {ω : (f (x, ω) f 0 (x)) β } K 2 (ε). x Ψ 0 Then for all ε > 0 the relation P {ω : Φ (ω) Φ 0 β } K 1 (ε) + K 2 (ε) ) ) such such Proof. Assume for given ε > 0, n N, and ω Ω the relation Φ (ω) Φ 0 + β Then there exists x (ω) Γ (ω) such f (x (ω), ω) = Φ (ω) > Φ 0 + β. To x (ω) we select x (ω) Γ (ω) such d( x (ω), Ψ 0 ) = min d(x, Ψ 0). x Γ (ω) If x (ω) Ψ 0 we have f ( x (ω), ω) Φ (ω) Φ 0 + β = f 0 ( x (ω)) + β and consequently, (f (x, ω) f 0 (x)) β. x Ψ 0 Otherwise we have Γ 0 \ Γ (ω) and can employ the first assumption. Now we turn to the solution sets. We use the abbreviation ˆΨ = {x E : f 0 (x) Φ 0 }. Theorem 9 (Inner Approximation of the Solution Set) Assume (VL1), (VL2) and the following assumptions are satisfied: (SI3) There exist a function K 3 and to all ε > 0 a sequence ( ˆβ n 0 (ε) such P {ω : Φ (ω) Φ 0 ˆβ } K 3 (ε). ) and 16

17 (SI4) There exists a function ν such for all ε > 0 U ε Ψ 0 U ν(ε) Γ 0 U ν(ε) ˆΨ0. (SI5) There exists a function µ such for all ε > 0 x U ε Γ 0 \ U ε ˆΨ0 : f 0 (x) Φ 0 + µ(ε). Then for all ε > 0, β = max{ν 1 (β (1) n 1 (ε) = min{k n 0 (ε) : U β k,ε n n 1 (ε) ), ν 1 (µ 1 (β Γ 0 UΓ 0 } the relation + P {ω : Ψ (ω) \ U β Ψ 0 } K 1 (ε) + K 2 (ε) + K 3 (ε) ˆβ ))}, and Proof. Let g (x, ω) := f (x, ω) Φ (ω), g 0 (x) := f 0 (x) Φ 0. Then Ψ (ω) = Γ (ω) {x E : g (x, ω) 0} and Ψ 0 = Γ 0 {x E : g 0 (x) 0}. Furthermore, P {ω : P {ω : inf ( g (x, ω) g 0 (x)) β x UΓ 0 \Ψ 0 inf (f (x, ω) f 0 (x)) β } x UΓ 0 \Ψ 0 + P {ω : Φ (ω) + Φ 0 It remains to apply Theorem 1 with ˆβ } ˆβ } K 2 (ε) + K 3 (ε) =: K 2 (ε). β = β + ˆβ and K 2. We emphasize we can choose ν(ε) = ε if Γ Γ 0. Furthermore, if U ε Ψ 0 Γ 0 for a suitable ε > 0, we can also deal with ν(ε) = ε for all ε ε. Corollary 9.1 (Inner Approximation of the Solution Set) Assume (VL1), (VL2), (VU1), (VU2), (SI4), (UCon), and the following condition are satisfied: (Gr-f 0 ) There exist an increasing function ψ 1 R + R + and constants c 1 > 0, δ 1 > 0, and θ 1 > 0 such x UΓ 0 : f 0 (x) Φ 0 ψ 1 (d(x, ˆΨ 0 )) and 0 < θ < θ 1 : ψ 1 (θ) c 1 θ δ 1. Then for all ε > 0, β = max{ν 1 (β ), (1) ν 1 (( ˆβ = max{2c 2 (β ) (1) δ 2, 2β }, and n 1 (ε) = min{k n 0 (ε) : U β k,ε Γ 0 UΓ 0, U β k,ε 17 ˆβ +β c 1 ) 1 δ 1 )}, Ψ 0 UΨ 0, β k,ε max{θ 1, θ 2 }}

18 the relation n n 1 (ε) P {ω : Ψ (ω) \ U β Ψ 0 } 2K 1 (ε) + 2K 2 (ε) Proof: We apply Theorem 9 together with Corollary 7.1. (SI3) is satisfied with ˆβ and K 3 = K 1 + K 2. (VL2) is satisfied with ˆβ and K 2. Theorem 9 with µ 1 (ε) = ( ε c 1 ) 1 δ 1 yields the conclusion. If for all ε > 0 the condition (CK-R) There exist a function K 1 and to all ε > 0 a sequence (β ) (1) such P {ω : (Γ (ω) \ U β (1) Γ 0) (Γ 0 \ Γ (ω)) } K 1 (ε) is satisfied, then also (VL1) and (VU1) are fulfilled with K 1 and β. (1) Consequently, if Γ = Γ κ n we can employ Theorem 5 or Corollary 5.1 in order to determine a suitable κ and formulate sufficient conditions for (VL1) and (VU1). Now we consider outer approximations of the solution set via κ -optimal solutions of the approximating problems. Let ˆΨ κ n (ω) := {x E : f (x, ω) < Φ (ω) + κ }. Γ can, e.g., be specified as U Γ (1) β n or as Γ κ n. Theorem 10 (Outer Approximation of the Solution Set, Relaxation) Assume for all ε > 0 there exist sequences (β ) (i), i = 1, 2, and functions K i, i = 1, 2, such (VU1-R), (VU2-R) and the following assumption are satisfied: (SO3) There exist a function K 3 and to all ε > 0 a sequence ( ˆβ n 0 (ε) such P {ω : Φ (ω) Φ 0 ˆβ } K 3 (ε). ) and Then for all ε > 0, κ = β relation + ˆβ and β = max{β, (1) β + ˆβ } the 18

19 κn P {ω : Ψ 0 \ ( ˆΨ (ω) U Γ (1) β (ω)) } K 1 (ε) + K 2 (ε) + K 3 (ε) (ε) Proof. We apply Theorem 3 and label the denotations in Theorem 3 with a tilde. With g (x, ω) := f (x, ω) Φ (ω) and g 0 (x) := f 0 (x) Φ 0 we have Ψ 0 = {x Γ 0 : g 0 (x) 0} and ˆΨ κ n (ω) := {x E : g (x, ω) < κ }. Furthermore, let Q = Γ, ˆΓκ n = ˆΨ κ n, Q0 = Γ 0, and ˆΓ0 = ˆΨ 0. Because of P {ω : inf ((f (x, ω) Φ (ω)) (f 0 (x) Φ 0 )) β x Ψ 0 P {ω : inf (f (x, ω) f 0 (x)) β } + P {ω : Φ (ω) + Φ 0 ) x Ψ 0 K 2 (ε) + K 3 (ε) =: K2 (ε) condition (CO2) is satisfied with and K 2. + β = β + ˆβ } ˆβ } ˆβ 5 Examples We will illustrate the approach by three examples. When applying our results to problems in decision theory or estimation theory, the most critical assumption is probably (SI4). Fortunately, there are several important applications where some quantities do not vary with n and ν is not needed at all or, as in our third example, (SI4) is easy to verify. In the general case, however, if the constraint set and the objective function are approximated simultaneously and the solution lies on the boundary of the constraint set, ν can not be ignored. Only in rare cases one should have enough knowledge to determine it exactly. One way out are adaptive methods for successive approximation of ν. However, even if one does not succeed in determining ν with a satisfactory accuracy, our results still yield assertions on the convergence rate, albeit without a reliable constant. Results of kind can be used to derive asymptotic confidence sets if a limiting distribution is not available. Firstly, we will discuss a simple example which is intended to give an idea of how one can deal with the uniform convergence assumption for the objective functions. At the first glance this assumption seems to be rather restrictive. There is, however, a growing number of results from probability theory yielding assertions of kind, c.f. [15], [1], [12]. A more refined investigation, which relies on stability assertions adjusted to the direct utilization of 19

20 concentration-of measure inequalities for sequences of random variables, will be given elsewhere. Moreover, we would like to refer the reader to [12] where several sufficient conditions are gathered. We assume E = R p and consider a fixed compact constraint set K and a linear objective function q(z) T x with x = (x 1,..., x p ) T, q R m R p. z is the realization of a random vector Z with given distribution P Z on the sigmafield of Borel sets of R m. The range of q(z) is posed to be bounded. For sake of simplicity we deal with the metric d(x, y) = max x i y i. The i=1,...,p problem (P 0 ) min x K Eq(Z)T x is approximated, replacing the expectation with respect to P Z by the expectation with respect to the empirical distribution based on a sequence Z (j), i = 1, 2,..., of i.i. P Z -distributed random vectors: n 1 (P n ) min q(z (j) ) T x. x K n j=1 We assume Eq(Z) 0, because otherwise the problem becomes trivial, and abbreviate m := max q i (Z(ω)). i=1,...,p ω We consider the sets K k = {x K : k 1 d(0, x) k}, k = 1, 2,... Let I K := {k : K k K }. Hence we obtain by Hoeffding s inequality ([3], [1]): P {ω : 1 n x K n j=1 k I K P {ω : x K k 1 n q(z (j) (ω)) T x Eq(Z) T x ε n } n j=1 q(z (j) (ω)) T x Eq(Z) T x ε n } P {ω : max n 1 q k I i=1,...,p n i (Z (j) (ω)) Eq i (Z) ε k } n K j=1 2 e ε 2 2k 2 m 2 =: K 2 (ε). k I K Of course this inequality can be further improved. For example, due to the linearity, it is enough to consider a cover of the boundary of K only. Employing other concentration inequalities, also the boundedness condition for q(z) can be overcome. Moreover, if the functions have a more involved form, one can often proceed in a similar way. We aim at employing Theorem 9 and (in case of a non-unique solution) Theorem 10. (VL1), (VU1), and (VU1-R) are satisfied with K 1 0. (VL2), (VU2), and (VU2-R) are fulfilled with K 2. (SI4) is satisfied with ν(ε) = ε. 20

21 It remains to investigate the growth condition (Gr-f 0 ), where we can choose UΓ 0 = Γ 0, and the semicontinuity condition (UCon). We have f 0 (x) Φ 0 max E q i(z) d(x, Ψ 0 ), hence (UCon) is satisfied with i=1,...,p c 2 = max E q i(z) and δ 2 = 1. i=1,...,p Let I := {i {1,..., p} : Eq i (Z) 0}. According to our assumption I is not empty. To x with d(x, ˆΨ 0 ) > ε there are ˆx ˆΨ 0 and i I such x i ˆx i > ε. Consequently (Gr-f 0 ) is satisfied with c 1 = min E q i(z) and i I δ 1 = 1. Secondly, we consider the approximation of a constraint set which is determined by a probabilistic constraint. Again, replacing the true probability measure with the empirical measure, we obtain a sequence of approximating constraint functions. In detail, we assume the constraint function has the special form g 0 (x) = α P Z ((, γ(x)]) = α F Z (γ(x)). Here Z and P Z are defined as above; F Z denotes the corresponding distribution function. Additionally we restrict the considerations to p = 1. α (0, 1) denotes a probability level and γ E R 1 a given concave function. The inequality constraint g 0 (x) 0 then reads as P (Z γ(x)) α. We assume Γ 0 = ˆΓ 0 = {x E : g 0 (x) 0} =. The approximating constraint set has the form Γ n (ω) = {x E : α F n (γ(x), ω) 0} with the empirical distribution function F n. In order to fulfil (CI2 ) and (CO2 ) we can directly apply the Dvoretzky- Kiefer-Wolfowitz inequality with Massart s bound ([9], [1]) and we obtain P {ω : n (α F n (γ(x), ω)) (α F Z (γ(x)) > ε} 2e 2ε2. x R 1 (CI3) is not needed. In order to fulfill (Gr-g 0 ), we will impose growth conditions for F Z and γ. Assume, for the given probability level α, the α-quantile q α of F Z is unique and consider a compact set K such q α int K. Furthermore, let X K := {x E : γ(x) K} and pose the following conditions are satisfied: (IG) There exist positive constants c 1,γ, c 1,F, δ 1,γ, δ 1,F such y K with y < q α : α F Z (y) > c 1,F (d(y, q α )) δ 1,F and x X K : γ(x) < q α c 1,γ d(x, Γ 0 ) δ 1,γ. (Gr-g 0 ) with K instead of UQ 0 and a strict inequality is then satisfied with 21

22 c 1 = c F (c 1,γ ) δ F and δ 1 = δ 1,γ δ F. Of course only one inequality in (IG) has to be strict. If Γ 0 is single-valued, it remains to apply Theorem 1. Otherwise we employ Theorem 2 and assume the following condition is satisfied: (OG) There exist positive constants c 2,γ, c 2,F, δ 2,γ, δ 2,F such y K with y > q α : F Z (y) α > c 2,F (d(y, q α )) δ 2,F. Furthermore, there exists an ε > 0 such CI( ε) and x Γ 0 \ CI( ε) : γ(x) > q α + c 2,γ d(x, (E \ Γ 0 ) δ 2,γ. Hence, with respect to (CO3) we obtain for all 0 < ε ε Γ 0 ŪεCI(ε) and x Γ 0 \ CI( ε) : g 0 (x) < c 2 (d(x, E \ Γ 0 ) δ 2 with c 2 = c F (c 2,γ ) δ F and δ 2 = δ 2,γ δ F. Thus in Theorem 2 we can choose µ(ε) = c 2 ε δ 2. Consequently, for all ε ε, β = max{( ε c 1 ) 1 δ1 n 1 2 δ 1, (2 ε c 2 ) 1 δ2 n 1 2 δ 2 }, and n 0 (ε) = min{k : β k,ε 2 ε, γ(γ 0 \ CI( β k,ε )) K} the relation 2 P {ω : (Γ n (ω) \ U Γ β 0) (Γ 0 \ (U Γ β n(ω)) } 4e 2ε2 Eventually, we consider quantile estimation because here relaxation of the constraint set comes into play in a natural way. Papers dealing with quantile estimation usually assume the distribution function is strictly increasing in a neighborhood of the quantile (c.f. [4], [5]). There are, however, applications where one can not a priori assume, the lower and the upper quantile coincide. We consider - as in the foregoing example - a real-valued random variable Z with distribution P Z and distribution function F Z. We will, for a fixed α (0, 1), investigate the lower α-quantile qα l := inf{x R 1 : F Z (x) α}. We consider the constraint set Γ 0 := {x R : F Z (x) α} and the optimization problem (P 0 ) min x. x Γ 0 As F Z is upper semicontinuous by definition, the set Γ 0 is closed and the minimum qα l will be attained. (P 0 ) could be approximated replacing F Z by the empirical distribution function F n. Unfortunately, the set {x R 1 : F n (x) α}, in general, does not approximate the whole set Γ 0. In [19] we showed with a suitable relaxation κ the solutions to the approximate problems convergence in 22

23 probability to the desired quantile. Here we can proceed in a similar way, consider the modified constraint set Γ with Γ (ω) := {x R : F n (x, ω) > α ε n } and investigate the approximating optimization problems (P ) min x. x Γ (P ) has a unique solution, too. In order to obtain a convergence rate, we need some knowledge about F Z, e.g. a growth condition. Theorem 11 (Quantile Estimation) Assume there exist constants c > 0, δ > 0, and θ > 0 such x with 0 < d( x, Γ 0 ) θ : F 0 ( x) < α cd( x, Γ 0 ) δ. Then for all ε > 0, β β k,ε θ} the relations hold. = max{εn 1 2, ( 2ε c ) 1 δ n 1 2δ }, and n 0 (ε) = min{k : P {ω : (Γ (ω) \ U β Γ 0) (Γ 0 \ Γ (ω)) } 2e 2ε2 P {ω : Ψ (ω) \ U β Ψ 0 } 2e 2ε2 Proof. We employ Corollary 5.1 and, since the solution is unique, Theorem 9. The objective function is not approximated, hence (VL2) and (VU2) are satisfied with K 2 0. The conditions (UCon) and (Gr-f 0 ) are fulfilled with c i = δ i = 1, i = 1, 2. Due to the Dvoretzky-Kiefer-Wolfowitz inequality with Massart s bound ([9], [1]), strict variants of (CI2 ) and (CO2 ) are satisfied with γ n = n 1 2 and K 2 (ε) = e 2ε2. The first assertion then follows by Corollary 5.1. Furthermore, taking into account (SI4) is satisfied with ν(ε) = ε, the second assertion is implied by Theorem 9. and References [1] L. Devroye and G. Lugosi. Combinatorical Methods in Density Estimation. Springer, [2] O. Gersch. Convergence in Distribution of Random Closed Sets and Applications in Stability Theory of Stochastic Optimisation. Dissertation thesis, Technical University Ilmenau

24 [3] W. Hoeffding. Probability inequalities for sums of bounded random variables. J. Amer. Statistical Association., 58:13 30, [4] A.I. Kibzun and Y.S. Kan. Stochastic Programming Problems. Wiley [5] K. Knight. What are the limiting distibutions of quantile estimators? Technical Report, University of Toronto, [6] P. Lachout, E. Liebscher and S. Vogel. Strong convergence of estimators as ε n -estimators of optimization problems. Ann. Inst. Statist. Math., 57: , [7] P. Lachout and S. Vogel. On continuous convergence and epi-convergence of random functions. Part I: Theory and relations. Kybernetika, 39:75 98, [8] P. Lachout and S. Vogel. On continuous convergence and epi-convergence of random functions. Part II: Sufficient conditions and applications. Kybernetika, 39:99 118, [9] P. Massart. The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. Ann. of Probability, 18: , [10] A. Nemirovski and A. Shapiro. Scenario approximations of chance constraints. SIAM J. Optimization, to appear. [11] G.C. Pflug. Asymptotic dominance and confidence for solutions of stochastic programs. Czechoslovak J. Oper. Res., 1:21 30, [12] G.C. Pflug. Stochastic Optimization and Statistical Inference. In: Stochastic Programming (A. Ruszczynski, A. Shapiro, eds.), Handbooks in Operations Research and Mangement Science Vol. 10, Elsevier 2003, [13] W. Römisch. Stability of Stochastic Programming. In: Stochastic Programming (A. Ruszczynski, A. Shapiro, eds.), Handbooks in Operations Research and Mangement Science Vol. 10, Elsevier 2003, [14] A. Shapiro. Monte Carlo Sampling Methods. In: Stochastic Programming (A. Ruszczynski, A. Shapiro, eds.), Handbooks in Operations Research and Mangement Science Vol. 10, Elsevier 2003,

25 [15] A.W. van der Vaart and J.A. Wellner. Weak Convergence and Empirical Processes. Springer, [16] S. Vogel. A stochastic approach to stability in stochastic programming. J. Comput. and Appl. Math., Series Appl. Analysis and Stochastics, 56: 65 96, [17] S. Vogel. On stability in stochastic programming - Sufficient conditions for continuous convergence and epi-convergence. Preprint TU Ilmenau, [18] S. Vogel. On semicontinuous approximations of random closed sets with application to random optimization problems. Ann. Oper. Res., 142: , [19] S. Vogel. Qualitative stability of stochastic programs with applications in asymptotic statistics. Statistics and Decisions, 23: ,