Generalized Pattern Search Algorithms : unconstrained and constrained cases

Size: px

Start display at page:

Download "Generalized Pattern Search Algorithms : unconstrained and constrained cases"

Arthur Johnston
6 years ago
Views:

1 IMA workshop Optimization in simulation based models Generalized Pattern Search Algorithms : unconstrained and constrained cases Mark A. Abramson Air Force Institute of Technology Charles Audet École Polytechnique de Montréal John Dennis Rice University IMA, Jan 2003

2 2 Presentation outline Generalized Pattern Search (GPS) Unconstrained problem min x R n f(x)

3 2 Presentation outline Generalized Pattern Search (GPS) Unconstrained problem min x R n f(x) Bound and linear constraints min x X f(x) where X = {x Rn : Ax b} [l, u]

4 2 Presentation outline Generalized Pattern Search (GPS) Unconstrained problem min x R n f(x) Bound and linear constraints min x X f(x) where X = {x Rn : Ax b} [l, u] General constraints min x X Ω f(x) where Ω = {x : c i(x) 0, i =, 2,..., m} IMA, Jan 2003

5 3 Unconstrained optimization min x R n f(x) where f : R n R { } may be discontinuous or infinite valued, and: f is usually given as a black box (typically a computer code), f is expensive and have few correct digits, f(x) may fail expensively and unexpectedly. IMA, Jan 2003

6 4 Ancestor of GPS : Coordinate search Initialization: x 0 : initial point in R n 0 > 0 : initial step size.

7 4 Ancestor of GPS : Coordinate search Initialization: x 0 : initial point in R n 0 > 0 : initial step size. Poll step: For k = 0,,... If f(t) < f(x k ) for some t P k := {x k ± k e i : i N}, set x k+ = t and k+ = k ;

8 4 Ancestor of GPS : Coordinate search Initialization: x 0 : initial point in R n 0 > 0 : initial step size. Poll step: For k = 0,,... If f(t) < f(x k ) for some t P k := {x k ± k e i : i N}, set x k+ = t and k+ = k ; otherwise x k is a minimizer over the set P k, set x k+ = x k and k+ = k 2. IMA, Jan 2003

9 5 Coordinate search run x 0 =(2,2) T, 0 = f=440

10 5 Coordinate search run x 0 =(2,2) T, 0 = f=440 f=29286

11 5 x 0 =(2,2) T, 0 = Coordinate search run f=4772 f=440 f=29286

12 5 Coordinate search run f=4772 x 0 =(2,2) T, 0 = f=66 f=440 f=29286

13 5 Coordinate search run f=4772 x 0 =(2,2) T, 0 = f=66 f=440 f=29286 f=476 IMA, Jan 2003

14 6 Coordinate search run x =(,2) T, = f=66

15 6 Coordinate search run f=262 x =(,2) T, = f=8 f=66 f=440 f=06 IMA, Jan 2003

16 7 Coordinate search run x 2 =(2,2) T, 2 = f=8

17 7 Coordinate search run f=52 x 2 =(2,2) T, 2 = f=2646 f=8 f=66 f=36 IMA, Jan 2003

18 8 Coordinate search run x 3 =(0,) T, 3 = f=36

19 8 Coordinate search run x 3 =(0,) T, 3 = f=8 f=2466 f=36 f=06 f=7 IMA, Jan 2003

20 9 Coordinate search run x 4 =(0,0) T, 4 = f=7

21 9 Coordinate search run x 4 =(0,0) T, 4 = f=36 f=2402 f=7 f=82 f=24

22 9 Coordinate search run x 4 =(0,0) T, 4 = x 4 is called a mesh local optimizer f=36 f=2402 f=7 f=82 f=7 f=24 IMA, Jan 2003

23 0 Coordinate search run x 5 =x 4 =(0,0) T, 4 = 2 f=7 IMA, Jan 2003

24 Coordinate search run A budget of 20 function evaluations produces x = (0, 0) T with f(x) = 7. Can we do better with the coordinate search? IMA, Jan 2003

25 2 Coordinate search : opportunistic run x 0 =(2,2) T, 0 = f=440

26 2 Coordinate search : opportunistic run x 0 =(2,2) T, 0 = f=440 f=29286

27 2 Coordinate search : opportunistic run f=4772 x 0 =(2,2) T, 0 = f=440 f=29286

28 2 Coordinate search : opportunistic run f=4772 x 0 =(2,2) T, 0 = f=66 f=440 f=29286 IMA, Jan 2003

29 3 Coordinate search : opportunistic run x =(,2) T, = f=66

30 3 Coordinate search : opportunistic run f=262 x =(,2) T, = f=8 f=66 f=440 IMA, Jan 2003

31 4 Coordinate search : opportunistic run x 2 =(2,2) T, 2 = f=8

32 4 Coordinate search : opportunistic run f=52 x 2 =(2,2) T, 2 = f=2646 f=8 f=66 f=36 IMA, Jan 2003

33 5 Coordinate search : opportunistic run x 3 =(0,) T, 3 = f=36

34 5 Coordinate search : opportunistic run x 3 =(0,) T, 3 = f=8 f=2466 f=36 f=06 f=7 IMA, Jan 2003

35 6 Coordinate search : opportunistic run x 4 =(0,0) T, 4 = f=7

36 6 Coordinate search : opportunistic run x 4 =(0,0) T, 4 = f=36 f=2402 f=7 f=82 f=24 IMA, Jan 2003

37 7 Coordinate search : opportunistic run x 5 =x 4 =(0,0) T, 4 = 2 f=7

38 7 Coordinate search : opportunistic run x 5 =x 4 =(0,0) T, 4 = 2 f=7 f=.0625 IMA, Jan 2003

39 8 Coordinate search : opportunistic run x 6 =(0,0.5) T, 4 = 2 f=.0625 IMA, Jan 2003

40 9 Coordinate search : opportunistic run A budget of 20 function evaluations produces x = (0.5, 0) T with f(x) = Can we do better with the coordinate search? IMA, Jan 2003

41 20 Coordinate search : dynamic run x 0 =(2,2) T, 0 = f=440

42 20 Coordinate search : dynamic run x 0 =(2,2) T, 0 = f=440 f=29286

43 20 Coordinate search : dynamic run f=4772 x 0 =(2,2) T, 0 = f=440 f=29286

44 20 Coordinate search : dynamic run f=4772 x 0 =(2,2) T, 0 = f=66 f=440 f=29286

45 20 Coordinate search : dynamic run f=4772 x 0 =(2,2) T, 0 = f=66 f=440 f=29286 IMA, Jan 2003

46 2 Coordinate search : dynamic run x =(,2) T, = f=66

47 2 Coordinate search : dynamic run x =(,2) T, = f=8 f=66 IMA, Jan 2003

48 22 Coordinate search : dynamic run x 2 =(2,2) T, 2 = f=8

49 22 Coordinate search : dynamic run x 2 =(2,2) T, 2 = f=2646 f=8

50 22 Coordinate search : dynamic run x 2 =(2,2) T, 2 = f=2646 f=8 f=66

51 22 Coordinate search : dynamic run f=52 x 2 =(2,2) T, 2 = f=2646 f=8 f=66

52 22 Coordinate search : dynamic run f=52 x 2 =(2,2) T, 2 = f=2646 f=8 f=66 f=36 IMA, Jan 2003

53 23 Coordinate search : dynamic run x 3 =(0,) T, 3 = f=36

54 23 Coordinate search : dynamic run x 3 =(0,) T, 3 = f=36 f=7 IMA, Jan 2003

55 24 Coordinate search : dynamic run x 4 =(0,0) T, 4 = f=7

56 24 Coordinate search : dynamic run x 4 =(0,0) T, 4 = f=36 f=2402 f=7 f=82 f=24 IMA, Jan 2003

57 25 Coordinate search : dynamic run x 5 =x 4 =(0,0) T, 4 = 2 f=7

58 25 Coordinate search : dynamic run x 5 =x 4 =(0,0) T, 4 = 2 f>4. f=7 f=7.25 f=.0625 IMA, Jan 2003

59 26 Coordinate search : dynamic run x 6 =(0,0.5) T, 4 = 2 f=.0625

60 26 Coordinate search : dynamic run x 6 =(0,0.5) T, 4 = 2 f=.0625 f=82 f=0.375 IMA, Jan 2003

61 27 Coordinate search : 3 strategies Complete Fixed order Dynamic order x T f(x) x T f(x) x T f(x) After 20 function evaluations (0, 0) 7 (.5, 0).0625 (.5,.5) After 50 function evaluations (.375,.375).8e-2 (.375,.32) 5.7e-3 (.375,.344) 3.e-3 IMA, Jan 2003

62 28 Convergence of coordinate searches if the sequence of iterates {x k } belongs to a compact set lim k k = 0 there is an ˆx which is the limit of a sequence of mesh local optimizers If f is continuously differentiable at ˆx, then f(ˆx) = 0.

63 28 Convergence of coordinate searches if the sequence of iterates {x k } belongs to a compact set lim k k = 0 there is an ˆx which is the limit of a sequence of mesh local optimizers If f is continuously differentiable at ˆx, then f(ˆx) = but we do not know anything about f. We need to work with less restrictive differentiability assumptions. IMA, Jan 2003

64 29 Coordinate search on a non-differentiable function f(x) = x with x 0 = (, ) T. Level set: {x R 2 : f(x) = } b x 0 = (, ) T a

65 29 Coordinate search on a non-differentiable function f(x) = x with x 0 = (, ) T. Level set: {x R 2 : f(x) = } b x 0 = (, ) T a A directional algorithm IMA, Jan 2003

66 30 Clarke Calculus If f is Lipschitz near x R n, then Clarke s generalized derivative at x in the direction v R n is f ( x; v) = lim sup y x, t 0 f(y + tv) f(y). t there exists a nonnegative scalar K such that f(x) f(x ) K x x for all x, x in some open neighborhood of x. IMA, Jan 2003

67 3 Facts on Clarke calculus The generalized gradient of f at x is the set f(x) := {ζ R n : f (x; v) v T ζ for all v R n }. Let f be Lipschitz near x, then f(x) = co{lim f(x i ) : x i x and f(x i ) exists }. Generalized derivative can be obtained from the generalized gradient : f (x; v) = max{v T ζ : ζ f(x)}. If x is a minimizer of f, and f is Lipschitz near x, then 0 f(x). Generalizes the st order necessary condition for continuously differentiable f: 0 = f(x). IMA, Jan 2003

68 32 If f is differentiable (Hadamard, Gâteaux, or Fréchet) at x, then the derivative of f at x is in the generalized gradient f(x). When f is convex, f(x) = subdifferential. f is regular at x if for all v R n, the one-sided directional derivative f (x; v) exists and equals f (x; v). f is strictly differentiable at x if for all v R n, lim y x,t 0 f(y + tv) f(y) t = f(x) T v. If f is Lipschitz near x and f(x) reduces to a singleton {ζ}, then f is strictly differentiable at x and f(x) = ζ. IMA, Jan 2003

69 33 Differentiable, not strictly differentiable y = x 2 (2+sin(π/x)).4.2 y x f is Lipschitz and differentiable, near 0: y (0) = 0 and y = 2x(2 + sin( π x )) π cos(π x )

70 33 Differentiable, not strictly differentiable y = x 2 (2+sin(π/x)).4.2 y x f is Lipschitz and differentiable, near 0: y (0) = 0 and y = 2x(2 + sin( π x )) π cos(π x ) The derivative is not continuous at 0: y ( 2k ) = 2 k π

71 33 Differentiable, not strictly differentiable y = x 2 (2+sin(π/x)).4.2 y x f is Lipschitz and differentiable, near 0: y (0) = 0 and y = 2x(2 + sin( π x )) π cos(π x ) The derivative is not continuous at 0: y ( 2k ) = 2 k π f is not strictly differentiable : f(0) = [ π, π]

72 33 Differentiable, not strictly differentiable y = x 2 (2+sin(π/x)).4.2 y x f is Lipschitz and differentiable, near 0: y (0) = 0 and y = 2x(2 + sin( π x )) π cos(π x ) The derivative is not continuous at 0: y ( 2k ) = 2 k π f is not strictly differentiable : f(0) = [ π, π] f is not regular : f (0, ±) = π f (0, ±) = 0 IMA, Jan 2003

73 34 Strictly, not continuously differentiable Example from Hiriart Urruty and Lemarechal φ(x) x x/(+x) f(x) = x 0 0 /3 /2 x ϕ(u)du where ϕ(u) = u if u 0 +κ if κ + > u k IMA, Jan 2003

74 35 Strictly, not continuously differentiable f 0.35 Ex: Hiriart Urruty and Lemarechal /4 /3 /2 x f is Lipschitz near ˆx = 0, has kinks at κ with f( κ ) = [ κ+, κ ]

75 35 Strictly, not continuously differentiable f 0.35 Ex: Hiriart Urruty and Lemarechal /4 /3 /2 x f is Lipschitz near ˆx = 0, has kinks at κ with f( κ ) = [ κ+, κ ] f is not strictly differentiable, nor continuously differentiable, in any neighborhood of ˆx = 0.

76 35 Strictly, not continuously differentiable f 0.35 Ex: Hiriart Urruty and Lemarechal /4 /3 /2 x f is Lipschitz near ˆx = 0, has kinks at κ with f( κ ) = [ κ+, κ ] f is not strictly differentiable, nor continuously differentiable, in any neighborhood of ˆx = 0. f(0) = {0} therefore f is strictly differentiable at ˆx = 0. IMA, Jan 2003

77 36 Convergence of coordinate searches if the sequence of iterates {x k } belongs to a compact set. lim k k = 0 there is an ˆx which is the limit of a sequence {x k } k K of mesh local optimizers (f(x k ± k e i ) f(x k ) for all e i ) If f is Lipschitz near ˆx, then f (ˆx; ±e i ) 0 for every e i

78 36 Convergence of coordinate searches if the sequence of iterates {x k } belongs to a compact set. lim k k = 0 there is an ˆx which is the limit of a sequence {x k } k K of mesh local optimizers (f(x k ± k e i ) f(x k ) for all e i ) If f is Lipschitz near ˆx, then f (ˆx; ±e i ) 0 for every e i Proof:f f(y + te i ) f(y) (ˆx; e i ) := lim sup y ˆx, t 0 t

79 36 Convergence of coordinate searches if the sequence of iterates {x k } belongs to a compact set. lim k k = 0 there is an ˆx which is the limit of a sequence {x k } k K of mesh local optimizers (f(x k ± k e i ) f(x k ) for all e i ) If f is Lipschitz near ˆx, then f (ˆx; ±e i ) 0 for every e i Proof:f f(y + te i ) f(y) (ˆx; e i ) := lim sup y ˆx, t 0 t f(x k + k e i ) f(x k ) lim sup k K k

80 36 Convergence of coordinate searches if the sequence of iterates {x k } belongs to a compact set. lim k k = 0 there is an ˆx which is the limit of a sequence {x k } k K of mesh local optimizers (f(x k ± k e i ) f(x k ) for all e i ) If f is Lipschitz near ˆx, then f (ˆx; ±e i ) 0 for every e i Proof:f f(y + te i ) f(y) (ˆx; e i ) := lim sup y ˆx, t 0 t f(x k + k e i ) f(x k ) lim sup 0 k K k IMA, Jan 2003

81 37 Convergence of coordinate searches if the sequence of iterates {x k } belongs to a compact set lim k = 0 k there is an ˆx which is the limit of a sequence {x k } k K of mesh local optimizers If f is Lipschitz near ˆx, then f (ˆx; ±e i ) 0 for every e i If f is regular at ˆx, then f (ˆx; ±e i ) 0 for every e i If f is strictly differentiable at ˆx, then f(ˆx) = 0 IMA, Jan 2003

82 38 The two phases of Pattern search algorithms Global search on the mesh Flexibility, User heuristics, Knowledge of the model, Surrogate functions. Local poll around the incumbent solution More rigidly defined. Ensures appropriate first order optimality conditions. IMA, Jan 2003

83 39 Positive spanning sets D R n is a positive spanning set if non-negative linear combinations of the elements of D span R n.

84 39 Positive spanning sets D R n is a positive spanning set if non-negative linear combinations of the elements of D span R n.

85 39 Positive spanning sets D R n is a positive spanning set if non-negative linear combinations of the elements of D span R n.

86 39 Positive spanning sets D R n is a positive spanning set if non-negative linear combinations of the elements of D span R n. D contains at least n + directions IMA, Jan 2003

87 Basic pattern search algorithm for unconstrained optimization Positive spanning directions: D k = {d, d 2,..., d p } D Current best point: x k R n Current mesh size parameter: k R + x k + k d x k x k + k d 3 x k + k d 2

88 Basic pattern search algorithm for unconstrained optimization Positive spanning directions: D k = {d, d 2,..., d p } D Current best point: x k R n Current mesh size parameter: k R + x k 2

89 Search step (global) x k Given k, x k : Search anywhere on M k.

90 Search step (global) x k Given k, x k : Search anywhere on M k. If an improved mesh point is found i.e. f(x k+ ) < f(x k ) then set k+ k, 3

91 Search step (global) Given k, x k : Search anywhere on M k. If an improved mesh point is found i.e. f(x k+ ) < f(x k ) x k+ then set k+ k, and restart the search from this improved point. 4

92 Poll step (local) x k If search fails, Poll at mesh neighbors:

93 Poll step (local) If search fails, Poll at mesh neighbors: If an improved mesh point is found i.e. f(x k+ ) < f(x k ), set k+ k, and restart search from improved point. x k 5

94 Poll step (local) x k+ If search fails, Poll at mesh neighbors: If an improved mesh point is found i.e. f(x k+ ) < f(x k ), set k+ k, and restart search from improved point. Else x k+ = x k is a mesh local optimizer. Set k+ < k, and restart search from this point. 6

95 Convergence results If all iterates are in a compact set then lim inf k k = 0 (the proof is non trivial since k+ may increase)

96 Convergence results If all iterates are in a compact set then lim inf k = 0 (the proof is non trivial since k+ may increase) k For every limit point ˆx of any subsequence {x k } k K of mesh local optimizers where { k } k K 0, and for the set ˆD of poll directions used infinitely many times in this subsequence

97 Convergence results If all iterates are in a compact set then lim inf k = 0 (the proof is non trivial since k+ may increase) k For every limit point ˆx of any subsequence {x k } k K of mesh local optimizers where { k } k K 0, and for the set ˆD of poll directions used infinitely many times in this subsequence If f is Lipschitz near ˆx, then f (ˆx; d) 0 for every d ˆD If f is regular at ˆx, then f (ˆx; d) 0 for every d ˆD. If f is strictly differentiable at ˆx, then f(ˆx) = 0. 7

98 The convergence results do not state that if the sequence of iterates {x k } belongs to a compact set lim k k = 0

99 The convergence results do not state that if the sequence of iterates {x k } belongs to a compact set lim k k = 0 If f is continuously differentiable everywhere then f(ˆx) = 0 for any limit point ˆx of the sequence of iterates.

100 The convergence results do not state that if the sequence of iterates {x k } belongs to a compact set lim k k = 0 If f is continuously differentiable everywhere then f(ˆx) = 0 for any limit point ˆx of the sequence of iterates. If the entire sequence of iterates converges (at ˆx say), and if f is differentiable then f(ˆx) = 0.

101 The convergence results do not state that if the sequence of iterates {x k } belongs to a compact set lim k k = 0 If f is continuously differentiable everywhere then f(ˆx) = 0 for any limit point ˆx of the sequence of iterates. If the entire sequence of iterates converges (at ˆx say), and if f is differentiable then f(ˆx) = 0. If the entire sequence of iterates converges (at ˆx say), and if f is Lipschitz near ˆx then 0 f(ˆx). Thus, the method does not necessarily produce a stationary point in the Clarke sense. 8

102 Basic pattern search algorithm for linearly constrained optimization Infeasible trial points are pruned (the objective function is not evaluated and set to infinity). When x k is close to the boundary of the feasible region, the poll directions must conform to the boundary. 9

103 Convergence results linearly constraints If all iterates are in a compact set then lim inf k k = 0 For every limit point ˆx of any subsequence {x k } k K of mesh local optimizers where { k } k K 0, and for the set ˆD of poll directions used infinitely many times in this subsequence

104 Convergence results linearly constraints If all iterates are in a compact set then lim inf k k = 0 For every limit point ˆx of any subsequence {x k } k K of mesh local optimizers where { k } k K 0, and for the set ˆD of poll directions used infinitely many times in this subsequence If f is Lipschitz near ˆx, then f (ˆx; d) 0 for every d ˆD T X (ˆx) If f is regular at ˆx, then f (ˆx; d) 0 for every d ˆD T X (ˆx). If f is s.d. at ˆx, then f(ˆx) T v 0 for every v T X (ˆx) 0

105 Positive spanning sets A richer set of D increases the number of directions in ˆD, and thus the directions for which f (ˆx; d) 0. User s domain specific knowledge can help choose D k D. Theory limited to a finite number of directions in D, so the barrier approach (reject infeasible points) does not apply to general constraints because a finite number can not conform to the boundary of Ω.

106 General nonlinear constrained optimization min x X f(x) s.t. x Ω {x C(x) 0}, where C = (c, c 2,..., c m ) T X is defined by bound and linear constraints. Define the nonnegative constraint violation function h : R n R h(x) = j max(0, c j (x)) 2. Note that h(x) = 0 iff x Ω, and h inherits smoothness from C.

107 General nonlinear constrained optimization min x X f(x) s.t. x Ω {x C(x) 0}, where C = (c, c 2,..., c m ) T X is defined by bound and linear constraints. Define the nonnegative constraint violation function h : R n R h(x) = j max(0, c j (x)) 2. Note that h(x) = 0 iff x Ω, and h inherits smoothness from C. We look at the biobjective optimization problem where a priority is given to the minimization of h over the minimization of f. 2

108 Pattern Search with filter f h

109 Pattern Search with filter f 40 f= h

110 Pattern Search with filter f 40 f= h

111 Pattern Search with filter f 40 f= h

112 Pattern Search with filter f 40 f=28 f= h

113 Pattern Search with filter f 40 f=28 f= h

114 Pattern Search with filter f=38 f 40 f=28 f= h

115 Pattern Search with filter f=38 f 40 f=28 f= h

116 Pattern Search with filter f=24 f=38 f 40 f=28 f= h

117 Pattern Search with filter f=24 f=38 f 40 f=28 f= h

118 Pattern Search with filter f=24 f=38 f 40 f=28 f= h

119 Pattern Search with filter f=24 f=38 f 40 f=28 f= f= h

120 Pattern Search with filter f=24 f=38 f 40 f=28 f= f= h

121 Pattern Search with filter f=24 f=38 f 40 f=28 f= f= h

122 Pattern Search with filter f=24 f=38 f 40 f=28 f= f=20 f= h

123 Pattern Search with filter f=24 f=38 f 40 f=28 f= f=20 f= h

124 Pattern Search with filter f=24 f=38 f 40 f=28 f= f=20 f=5 f= h

125 Pattern Search with filter f=24 f=38 f 40 f=28 f= f=20 f=5 f= h 3

126 Pattern Search with filter f 40 f= f=5 f= h 4

127 Pattern Search with filter f=24 f=8 f h

128 Pattern Search with filter f=24 f=8 f=6 f h

129 Pattern Search with filter f=24 f=8 f=6 h= f h

130 Pattern Search with filter f=24 f=7 f=8 f=6 h= f h

131 Pattern Search with filter f=24 f=7 f=8 f=6 h= f h 5

132 Pattern Search with filter f=24 f=7 f h

133 Pattern Search with filter f=24 f=25 f=7 f h

134 Pattern Search with filter f=24 f=20 f=25 f=7 f h

135 Pattern Search with filter f=24 f=20 f=25 f=7 f=8 f h

136 Pattern Search with filter f=24 f=20 f=25 f=7 f=8 h= f Mesh isolated filter point h 6

137 Convergence results general constraints Polling around least infeasible point gives priority to the minimization of h versus the minimization of f. If all iterates are in a compact set then lim inf k = 0 k For every limit point ˆp of any subsequence {p k } k K of mesh isolated poll centers where { k } k K 0, and for the set ˆD of poll directions used infinitely many times in this subsequence

138 Convergence results general constraints Polling around least infeasible point gives priority to the minimization of h versus the minimization of f. If all iterates are in a compact set then lim inf k = 0 k For every limit point ˆp of any subsequence {p k } k K of mesh isolated poll centers where { k } k K 0, and for the set ˆD of poll directions used infinitely many times in this subsequence If h is Lipschitz near ˆp, and ˆp is feasible then h (ˆx; v) 0 for every v T X (ˆp)

139 Convergence results general constraints Polling around least infeasible point gives priority to the minimization of h versus the minimization of f. If all iterates are in a compact set then lim inf k = 0 k For every limit point ˆp of any subsequence {p k } k K of mesh isolated poll centers where { k } k K 0, and for the set ˆD of poll directions used infinitely many times in this subsequence If h is Lipschitz near ˆp, and ˆp is feasible then h (ˆx; v) 0 for every v T X (ˆp) If h is Lipschitz near ˆp, then h (ˆp; d) 0 for every d ˆD T X (ˆp) 7

140 If h is regular at ˆp, then h (ˆx; d) 0 for every d ˆD T X (ˆp) If h is s.d. at ˆp, then h(ˆx) T v 0 for every v T X (ˆx) Note: The same convergence results hold for f, with an additional requirement: ˆp must be strictly feasible with respect to the general constraints. 8

141 A planform filter : 7 vars, 3 ctrs, kriging F ilter after 5 evaluations Zoom of filter on left f(x) 9.7 f(x) h(x) h(x) 9

142 A planform filter : 7 vars, 3 ctrs, kriging F ilter after 50 evaluations Zoom of filter on left f(x) 9.75 f(x) h (x) h(x) 20

143 A planform filter : 7 vars, 3 ctrs, kriging F ilter after 7 evaluations Zoom of filter on left f(x) f(x) h(x) h(x) 2

144 Discussion GPS algorithms are directional algorithms: Analysis is easy with Clarke calculus

145 Discussion GPS algorithms are directional algorithms: Analysis is easy with Clarke calculus Optimality results depend on local differentiability The analysis can be extended by considering other directions

146 Discussion GPS algorithms are directional algorithms: Analysis is easy with Clarke calculus Optimality results depend on local differentiability The analysis can be extended by considering other directions Linear and bound constraints are treated by appropriate polling directions

147 Discussion GPS algorithms are directional algorithms: Analysis is easy with Clarke calculus Optimality results depend on local differentiability The analysis can be extended by considering other directions Linear and bound constraints are treated by appropriate polling directions General constraints are treated by the filter

148 Discussion GPS algorithms are directional algorithms: Analysis is easy with Clarke calculus Optimality results depend on local differentiability The analysis can be extended by considering other directions Linear and bound constraints are treated by appropriate polling directions General constraints are treated by the filter GPS for problems with categorical variables Tomorrow s talk by Mark Abramsom 22

A PATTERN SEARCH FILTER METHOD FOR NONLINEAR PROGRAMMING WITHOUT DERIVATIVES

A PATTERN SEARCH FILTER METHOD FOR NONLINEAR PROGRAMMING WITHOUT DERIVATIVES CHARLES AUDET AND J.E. DENNIS JR. Abstract. This paper formulates and analyzes a pattern search method for general constrained