Performance Measures for Ranking and Selection Procedures

Transcription

1 Rolf Waeber Performance Measures for Ranking and Selection Procedures 1/23 Performance Measures for Ranking and Selection Procedures Rolf Waeber Peter I. Frazier Shane G. Henderson Operations Research & Information Engineering, Cornell University Tuesday December 07, Winter Simulation Conference, Baltimore, MD

2 Rolf Waeber Performance Measures for Ranking and Selection Procedures 2/23 Motivation

3 Rolf Waeber Performance Measures for Ranking and Selection Procedures 2/23 Motivation Question: What is a good policy to allocate measurements and ultimately to make a selection?

4 Rolf Waeber Performance Measures for Ranking and Selection Procedures 2/23 Motivation Question: What is a good policy to allocate measurements and ultimately to make a selection?

5 Rolf Waeber Performance Measures for Ranking and Selection Procedures 2/23 Motivation Question: What is a good policy to allocate measurements and ultimately to make a selection?

6 Rolf Waeber Performance Measures for Ranking and Selection Procedures 2/23 Motivation Question: What is a good policy to allocate measurements and ultimately to make a selection?

7 Rolf Waeber Performance Measures for Ranking and Selection Procedures 3/23 Set-Up Each alternative produces random outcomes Y i F i F, i = 1,..., k, where F is a set of distribution functions. Interested in a functional on F, i.e. θ(f i ) = θ i R. Goal: Find the best system i, such that θ i = min i {1,...,k} θ i.

8 Rolf Waeber Performance Measures for Ranking and Selection Procedures 3/23 Set-Up Each alternative produces random outcomes Y i F i F, i = 1,..., k, where F is a set of distribution functions. Interested in a functional on F, i.e. θ(f i ) = θ i R. Goal: Find the best system i, such that θ i = min i {1,...,k} θ i. A R&S policy π = (x n, τ, i π ) consists of allocation, stopping and decision rules. Π is set of all policies.

9 Rolf Waeber Performance Measures for Ranking and Selection Procedures 3/23 Set-Up Each alternative produces random outcomes Y i F i F, i = 1,..., k, where F is a set of distribution functions. Interested in a functional on F, i.e. θ(f i ) = θ i R. Goal: Find the best system i, such that θ i = min i {1,...,k} θ i. A R&S policy π = (x n, τ, i π ) consists of allocation, stopping and decision rules. Π is set of all policies. Assume fixed total simulation budget N.

10 Rolf Waeber Performance Measures for Ranking and Selection Procedures 4/23 Selecting a Selection Procedure Normal R&S, F i = N(µ i, σ 2 i ) for unknown µ i and σ i - Kim and Nelson (2001), frequentist approach - Chick and Inoue (2001), Value of Information (VIP) - Chen (1996), Optimal Computing Budget Allocation (OCBA) -...

11 Rolf Waeber Performance Measures for Ranking and Selection Procedures 4/23 Selecting a Selection Procedure Normal R&S, F i = N(µ i, σ 2 i ) for unknown µ i and σ i - Kim and Nelson (2001), frequentist approach - Chick and Inoue (2001), Value of Information (VIP) - Chen (1996), Optimal Computing Budget Allocation (OCBA) -... Problem: Selecting a selection procedure.

12 Rolf Waeber Performance Measures for Ranking and Selection Procedures 4/23 Selecting a Selection Procedure Normal R&S, F i = N(µ i, σ 2 i ) for unknown µ i and σ i - Kim and Nelson (2001), frequentist approach - Chick and Inoue (2001), Value of Information (VIP) - Chen (1996), Optimal Computing Budget Allocation (OCBA) -... Problem: Selecting a selection procedure. Goal: Introduce preference order on the set of all policies Π.

13 Rolf Waeber Performance Measures for Ranking and Selection Procedures 4/23 Selecting a Selection Procedure Normal R&S, F i = N(µ i, σ 2 i ) for unknown µ i and σ i - Kim and Nelson (2001), frequentist approach - Chick and Inoue (2001), Value of Information (VIP) - Chen (1996), Optimal Computing Budget Allocation (OCBA) -... Problem: Selecting a selection procedure. Goal: Introduce preference order on the set of all policies Π. Branke, Chick and Schmidt (2007), extensive testbed. Here: Robust approach inspired by statistical decision theory and convex risk measures.

14 Rolf Waeber Performance Measures for Ranking and Selection Procedures 5/23 Three-Layer Performance Analysis 1 Loss of decision: L(i, θ) R, where θ = (θ 1,..., θ k ). Examples: - L(i, θ) = I {θi θ i }, 0-1 loss. - L(i, θ) = θ i θ i, linear regret. - L(i, θ) = f (θ i θ i ), f is a convex and increasing function on R +.

15 Rolf Waeber Performance Measures for Ranking and Selection Procedures 5/23 Three-Layer Performance Analysis 1 Loss of decision: L(i, θ) R, where θ = (θ 1,..., θ k ). Examples: - L(i, θ) = I {θi θ i }, 0-1 loss. - L(i, θ) = θ i θ i, linear regret. - L(i, θ) = f (θ i θ i ), f is a convex and increasing function on R +. 2 Configuration-specific risk: R(π, F) R, a functional of the loss distribution with respect to the induced probability measure P π, where F = (F 1,..., F k ) F k. Examples: - R(π, F) = E π [L(i π, θ(f))], expected loss.

16 Rolf Waeber Performance Measures for Ranking and Selection Procedures 6/23 Three-Layer Performance Analysis cont. 3 Risk of a policy: Mapping ρ : Π R. Need to make some assumptions on F.

17 Rolf Waeber Performance Measures for Ranking and Selection Procedures 6/23 Three-Layer Performance Analysis cont. 3 Risk of a policy: Mapping ρ : Π R. Need to make some assumptions on F. Most popular choices: - Worst-Case: - Indifference Zone: ρ WC(π) := sup F F k R(π, F). - Bayes Risk: ρ IZ(π) := sup R(π, F). F F k \IZ ρ P0 (π) := E P0 [R(π, F)], P 0 is a risk weighting (prior) on F k.

18 Rolf Waeber Performance Measures for Ranking and Selection Procedures 7/23 Overview

19 Rolf Waeber Performance Measures for Ranking and Selection Procedures 7/23 Overview

20 Rolf Waeber Performance Measures for Ranking and Selection Procedures 8/23 Convex Risk Measures Applications in Finance: - Quantify the risk of a financial portfolio. - Determine appropriate risk capital for financial institutions.

21 Rolf Waeber Performance Measures for Ranking and Selection Procedures 8/23 Convex Risk Measures Applications in Finance: - Quantify the risk of a financial portfolio. - Determine appropriate risk capital for financial institutions. Risk Measure: a mapping ρ : X R. Convex Risk Measures: A risk measure ρ on L is called a convex risk measure if for all X, Y L : 1 Monotonicity: If X Y, then ρ(x) ρ(y). 2 Cash Invariance: If m R, then ρ(x + m) = ρ(x) m. 3 Convexity: ρ(λx + (1 λ)y) λρ(x) + (1 λ)ρ(y), for 0 λ 1. (Föllmer and Schied (2002)).

22 Rolf Waeber Performance Measures for Ranking and Selection Procedures 9/23 Representation Theorem Theorem: (Föllmer and Schied (2002)) Let Q denote the set of all finitely additive probability measures on a measurable space (Ω, G ), and α : Q R { } be a functional with inf Q Q α(q) R. Then the mapping ρ(x) := sup (E Q [ X] α(q)), Q Q defines a convex risk measure on L.

23 Rolf Waeber Performance Measures for Ranking and Selection Procedures 9/23 Representation Theorem Theorem: (Föllmer and Schied (2002)) Let Q denote the set of all finitely additive probability measures on a measurable space (Ω, G ), and α : Q R { } be a functional with inf Q Q α(q) R. Then the mapping ρ(x) := sup (E Q [ X] α(q)), Q Q defines a convex risk measure on L. Theorem: (W., Frazier, Henderson (2010)) WC, IZ and Bayes performance measures for R&S procedures can each be represented as ρ(π) = sup (E Q [R(π, F)] α(q)), Q Q where Q is an appropriate set of probability measures on (F, σ(f )), and α : Q R { } is some penalty function.

24 Rolf Waeber Performance Measures for Ranking and Selection Procedures 10/23 Examples Worst-Case: Q WC = { all point measures in F k}, α 0. Indifference zone: Q IZ = { all point measures in F k \ IZ }, α 0. Bayes: Q Bayes = {P 0 }, α 0.

25 Rolf Waeber Performance Measures for Ranking and Selection Procedures 10/23 Examples Worst-Case: Q WC = { all point measures in F k}, α 0. Indifference zone: Q IZ = { all point measures in F k \ IZ }, α 0. Bayes: Q Bayes = {P 0 }, α 0. Combination of Indifference Zone and Bayes formulations: Q = Q IZ Q Bayes, - α 0. - α(q) = 0 for Q Q Bayes, and α(q) = c > 0 for Q Q IZ.

26 Rolf Waeber Performance Measures for Ranking and Selection Procedures 10/23 Examples Worst-Case: Q WC = { all point measures in F k}, α 0. Indifference zone: Q IZ = { all point measures in F k \ IZ }, α 0. Bayes: Q Bayes = {P 0 }, α 0. Combination of Indifference Zone and Bayes formulations: Q = Q IZ Q Bayes, - α 0. - α(q) = 0 for Q Q Bayes, and α(q) = c > 0 for Q Q IZ. Bayes Robustness: Q = {P : P = (1 ε)p 0 + εc, C C }, α 0.

27 Rolf Waeber Performance Measures for Ranking and Selection Procedures 11/23 Performance Measure Based on Acceptance Set The runlength N can be used as a control of the policy. All reasonable policies should satisfy ρ(π N ) 0, N.

28 Rolf Waeber Performance Measures for Ranking and Selection Procedures 11/23 Performance Measure Based on Acceptance Set The runlength N can be used as a control of the policy. All reasonable policies should satisfy ρ(π N ) 0, N. Assume Q = m <, define E Q1 [R(π N, F)] α(q 1 ) f (π N ) =. E Qm [R(π N, F)] α(q m ) Define performance measure given an acceptance set A R m based on runlength ρ A (π) := inf {n N f (π n ) A }.

29 Rolf Waeber Performance Measures for Ranking and Selection Procedures 12/23 Performance Measure Based on Acceptance Set cont.

30 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

31 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

32 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

33 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

34 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

35 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

36 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

37 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

38 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

39 Rolf Waeber Performance Measures for Ranking and Selection Procedures 13/23 Axioms for Acceptance Sets Every coherent acceptance set A should satisfy (after appropriate normalization): (For p, q R m, [p, q] m := [p 1, q 1] [p m, q m] R m.) 1 0 A 2 ( k 1 k 1, 1] m A = 3 For any point q A [0, q] m A

40 Rolf Waeber Performance Measures for Ranking and Selection Procedures 14/23 Voting Machines in Ohio How many voting machines should be allocated to each precinct in Franklin County, Ohio? Allen and Bernshteyn (2008) advised the board of elections to better allocate voting machines. 4,600 machines, 788 precincts, forecasts of turnout exist. Here, only as visualization example of our framework.

41 Rolf Waeber Performance Measures for Ranking and Selection Procedures 15/23 R&S Application Can use simulation optimization routines to find a set of promising allocations. Then use R&S to decide which allocation is the best.

42 Rolf Waeber Performance Measures for Ranking and Selection Procedures 15/23 R&S Application Can use simulation optimization routines to find a set of promising allocations. Then use R&S to decide which allocation is the best. Objective: Minimize the probability of having long waiting times.

43 Rolf Waeber Performance Measures for Ranking and Selection Procedures 15/23 R&S Application Can use simulation optimization routines to find a set of promising allocations. Then use R&S to decide which allocation is the best. Objective: Minimize the probability of having long waiting times. S 1,..., S k are k different allocation rules. Performance of rule S i θ i := Φ(q i, µ, σ), q i = Fi 1 (α), where F i is the distribution of the longest waiting time across all precincts under allocation S i, α is some confidence level (e.g. 0.95).

44 Rolf Waeber Performance Measures for Ranking and Selection Procedures 16/23 Exit Option Include an exit option 0 with performance θ 0 = 1 Φ(min q i, µ, σ). Goal: Find i such that θ i = min {i=0,...,k} θ i.

45 Rolf Waeber Performance Measures for Ranking and Selection Procedures 17/23 R&S Performance To evaluate R&S procedures, we will consider: 1 Loss of decision: Linear regret, i.e., L(i, θ) = θ i min i {0,...,k} 2 Configuration-specific risk: Expected loss, i.e., R(π, F) = E π [L(i π, θ(f))]. θ i.

46 Rolf Waeber Performance Measures for Ranking and Selection Procedures 17/23 R&S Performance To evaluate R&S procedures, we will consider: 1 Loss of decision: Linear regret, i.e., L(i, θ) = θ i min i {0,...,k} 2 Configuration-specific risk: Expected loss, i.e., R(π, F) = E π [L(i π, θ(f))]. θ i. Compare two policies: 1 Equal allocation 2 2-stage allocation

47 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

48 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

49 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

50 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

51 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

52 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

53 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

54 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

55 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

56 Rolf Waeber Performance Measures for Ranking and Selection Procedures 18/23 Compare Policies

57 Rolf Waeber Performance Measures for Ranking and Selection Procedures 19/23 Preference Orders 1 Worst-Case performance: N Equal Allocation Stage

58 Rolf Waeber Performance Measures for Ranking and Selection Procedures 19/23 Preference Orders 1 Worst-Case performance: 2-stage equal N Equal Allocation Stage

59 Rolf Waeber Performance Measures for Ranking and Selection Procedures 19/23 Preference Orders 1 Worst-Case performance: 2-stage equal 2 Uniform Risk Weighting: 2-stage equal N Equal Allocation Stage N Equal Allocation Stage

60 Rolf Waeber Performance Measures for Ranking and Selection Procedures 19/23 Preference Orders 1 Worst-Case performance: 2-stage equal 2 Uniform Risk Weighting: 2-stage equal 3 N(1,1) Risk Weighting: N Equal Allocation Stage N Equal Allocation Stage N Equal Allocation Stage

61 Rolf Waeber Performance Measures for Ranking and Selection Procedures 19/23 Preference Orders 1 Worst-Case performance: 2-stage equal 2 Uniform Risk Weighting: 2-stage equal 3 N(1,1) Risk Weighting: 2-stage equal N Equal Allocation Stage N Equal Allocation Stage N Equal Allocation Stage

62 Rolf Waeber Performance Measures for Ranking and Selection Procedures 20/23 Preference Order Based on Acceptance Sets

63 Rolf Waeber Performance Measures for Ranking and Selection Procedures 21/23 Preference Order Based on Acceptance Sets cont. A 1 A 2 A 3 A 1 A 2 A 1 A 2 Equal Allocation (0.1) (1.2) (0.5) (0.1) (1.2) 2-Stage (0.0) (0.8) (0.5) (0.8) (0.0)

64 Rolf Waeber Performance Measures for Ranking and Selection Procedures 22/23 Summary

65 Rolf Waeber Performance Measures for Ranking and Selection Procedures 23/23 THANK YOU!

66 Rolf Waeber Performance Measures for Ranking and Selection Procedures 24/23 Robustness of Preference Order Blue: 2-stage equal Red: 2-stage equal

67 Rolf Waeber Performance Measures for Ranking and Selection Procedures 25/23 Parameter Selection for 2-Stage Procedure βn is spent on first stage for β [0, 1]. Remaining budget is spent on second stage.

68 Rolf Waeber Performance Measures for Ranking and Selection Procedures 25/23 Parameter Selection for 2-Stage Procedure βn is spent on first stage for β [0, 1]. Remaining budget is spent on second stage.

69 Rolf Waeber Performance Measures for Ranking and Selection Procedures 25/23 T. ALLEN and M. BERNSHTEYN (2008): Helping Franklin County Vote in 2008: Waiting Lines. Report to the Franklin County Board of Elections. J. BRANKE, S. CHICK and C. SCHMIDT (2007): Selecting a selection procedure. Management Science 53: C. CHEN (1996): A lower bound for the correct subset-selection probability and its application to discrete-event system simulations. IEEE transactions on automatic control 41: S. CHICK and K. INOUE (2001): New two-stage and sequential procedures for selecting the best simulated system. Operations Research 49: H. FÖLLMER and A. SCHIED (2002): Convex measures of risk and trading constraints. Finance and Stochastics 6: S. KIM and B. NELSON (2001): A fully sequential procedure for indifference-zone selection in simulation. ACM Transactions on Modeling and Computer Simulation (TOMACS) 11:273. R. WAEBER, P. I. FRAZIER and S. G. HENDERSON (2010): Performance Measures for Ranking and Selection Procedures Winter Simulation Conference, Conference Proceedings.