2 Situation A (Two-valued Model) Describing the Situation Reaching the rst goal

Transcription

1 A Probabilistic Stopping Criterion for the Evaluation of Benchmarks (Extended Version) Michael Greiner, Manfred Schramm z Keywords Evaluation of benchmarks, heuristic comparison of algorithms, general steps for stochastic modelling, subjective probability, Principle of Indierence, Rule of Succession Abstract We present general steps of modelling a problem within the frame of subjective probabilities and explain some connections to the well-known rule of succession. For practical relevance we present the solution of a small example concerning the ecient testing of benchmarks, including a discussion about the advantage of producing further data. We present the choices of modelling and obtain solutions to dierent questions and dierent information states. The way the solutions are obtained is therefore not specic and can be used for very dierent problem situations. Contents 1 Introduction Problem description Construction of the frame of discernment (Fr) Incorporation of background assumptions and knowledge (Ba) Incorporation of update information (Up) Incorporation of assumptions and knowledge about the process update information is delivered (Pr) Terminology (Urn) Goals (expressed in the Urn-Terminology) Siemens AG, Corporate Technology ZT PP 2, Otto-Hahn-Ring 6, D Munchen, Germany. michael.greiner@mchp.siemens.de z Fachhochschule Ravensburg-Weingarten, Fachbereich Elektrotechnik/Informatik, D Weingarten, Germany. schramma@fbe.fh-weingarten.de

2 2 Situation A (Two-valued Model) Describing the Situation Reaching the rst goal Example 1: Worst Case (P bn2c+1;n for N 200) Example 2: Best Case (P n;n for N 200) Example 3: Determining a minimal majority of red balls Connection with the Rule of Succession Reaching the second goal Example 5: Worst Case (P bn2c+1;n for N 200) Example 6: Best Case (P n;n for N 200) The advantage of (further) drawings { an analysis by decision theoretic considerations Example 7: D m r;n for r 15, n 25, and N Application: A priori calculation of the necessary sample size for a given security level Situation B Describing the situation Reaching the goal Example 8: Average Case (P maj for N 200) Example 9: Studying the qualitative shapes for varying values of N Considering the mean Situation C (Three-valued Model) Describing the situation Reaching the goal Example 10: Comparison to the two-valued case Conclusion/Summary 21 Appendix { Simulation Results 22 References 25

3 1 Introduction The real problem of using probability theory for solving a (e.g. decision-) problem does not consist in using ready made statistical formulas but in answering some important questions about the construction of a corresponding model. (Fr) What frame of discernment should we use? (Ba) How can we encode our background assumptions and knowledge? (Up) How can we encode new (update) information? (Pr) How can we encode assumptions and knowledge about the process in which new (update) information is delivered? We now briey explain some of these ideas and apply them to our example. 1.1 Problem description Let N be the total number of available benchmarks for two algorithms, call them A 1 respectivly A 2. Any algorithm might behave dierently on a single benchmark: there might be a very useful solution (e.g. very quickly obtained), there might be a less useful solution (e.g. very slowly obtained), there might be a wrong solution or there might be no solution at all (e.g. core dump oder eternal search). Generally, we are interested in arguing for \A 1 is better than 1 A 2 " or vice versa, based on the information of how the two algorithms behave on the whole set of benchmarks. As the evaluation of benchmarks may be expensive we want to test as few benchmarks as possible in order to reach the desired conclusion with a certain security. Our questions are therefore as follows: (1) Increasing the size n of the tested sample of benchmarks, what is the value of A 1 better than A 2 in r cases of the sample n which is necessary to reach a specied average correctness of our decision on the total set of benchmarks? (2) What is the relation if (only) incomplete information about a sample is available? (3) What kind of relation holds between the accuracy of guessing the value of and the sample size n? Number of cases where A 1 performs better than A 2 (4) How can we estimate the value of further experiments (data)? How can we estimate their inuence on our decision? 1 Where \better" has to be dened by the specic application. N

4 1.2 Construction of the frame of discernment (Fr) Calculations of probabilities are very sensitive to the choice of the frame, i.e. the set of elementary events. For example, drawing from an urn which may contain two kinds of balls is not the same as drawing from an urn which may contain three kinds of balls (especially if the dierence is not evident in some sample). We may call these states of information dierent states of possibility. Further, we have to justify the choice of our frame of discernment and to show how dierent choices of the frame might inuence our results. Generally we choose the minimal frame which allows the specication of our information (background and update) and our goals. In our case, we use (also for stochastic convenience) the model of an urn containing N balls at the beginning. A single ball may be red (which means that A 1 performs better than A 2 on a specic benchmark), green (in the opposite case) or blue (if neither happens). The following Situations A and B (see Section 2 and Section 3) show the answers if only red and green balls are expected to be in the urn (two-valued model), Situation C exerts only a very small inuence on our goals if blue balls are also expected (three-valued model). 1.3 Incorporation of background assumptions and knowledge (Ba) Given a frame of discernment we have to specify our background (a priori) information. Whenever this kind of knowledge does not lead to a completely specied probability model (P-model), we might have to determine a most probable model (in analogy of calculating the center of gravity in physics) to simplify our calculations. Principles like Indierence, Independence and the method of Maximum Entropy (see [Greiner & Schramm, 1994]) can be shown to support such calculations of the most probable P-model whenever commitments to a single P-model are necessary. In our case we make the assumption (justied by the Principle of Indierence, because we have no specic information, especially no statistics, how tests of benchmarks are distributed) that any possible P-model (i.e. any number of red, green and blue balls in the urn) has the same possibility (weight, probability) at the beginning. 1.4 Incorporation of update information (Up) Here we specify the (update) information which is delivered by the process. This information may contain parameters which are xed in any specic problem situation. In our case this means to get the information of a sample of size n (which is drawn without replacement) containing r red, g green and b blue balls. 1.5 Incorporation of assumptions and knowledge about the process update information is delivered (Pr) Identical samples can be obtained by dierent processes. Specically, we may have obtained identical samples from dierent sets of events by dierent processes. In the famous

5 Monty Hall problem (see e.g. [Savant, 1992] or [Randow, 1992]) it makes a dierence to assume a random experiment on the set of closed doors or on the set of closed and empty doors, where the result (update information) is in both cases an empty door. In the famous problem of the two children (see e.g. [Grinstead & Snell, 1997]) it makes a dierence whether we assume (in the lack of further knowledge) the child to be selected by a random process or whether we have some information (e.g. if the family has boys they are known to be more likely presented to the visitor). In our cases we do not have such knowledge 2. To end up with a distribution about the process the update information was generated, we have to use the same principles as above. In our case the use of the Principle of Indierence (no knowledge) leads to a uniform distribution Terminology (Urn) N n R (G; B) r (g; b) U R;N U R;G;N S r;n S r;g;n the known number of balls in the urn (i.e. number of available benchmarks) the size of the sample the unknown number of red (green, blue) balls in the urn at the beginning. In the two-valued case we have (N R + G). In the three-valued case we have (N R + G + B). the number of red (green, blue) balls in the sample. In the two-valued case we have (n r + g). In the three-valued case we have (n r + g + b). the urn contains R red and N? R green balls the urn contains R red, G green and N? R? G blue balls the sample contains r red and n? r green balls the sample contains r red, g green and n? r? g blue balls 1.7 Goals (expressed in the Urn-Terminology) The main goal (Go) is of course the estimation of the value RN (resp. GN) and the use of this for a two-valued decision (R > G or G R). More rened, our questions are as follows: (1) What is the average correctness of the decision given a sample of size n with r red balls and g green balls and r > g? Given a minimum value of an expected average correctness (e.g or 0.95) of our decision and a sample of size n, what is the smallest value of r that we need to reach this correctness? Knowing this value will allow us to make a decision about R > G (given a specic sample of size n) without further calculation. 2 Though this statement is not trivial, because any attempt at knowledge mining will very probably show that the benchmarks are somehow grouped together and include some history and relationship. 3 This is a way to avoid the classical demand that the sample has to be representative.

6 We will answer these questions for the two-valued case in the equations and examples of Situation A, goal 1, and for the three-valued case in Situation C. (2) Instead of using the Principle of Indierence we can obtain the same result on the questions of item (1) by consecutively using the rule of succession of Laplace. This application can be found in Section 2.3. (3) Given a sample of size n with r red balls, what is the (minimal) condence interval of level for our decision? We give the answer to this question in answering goal 2 in Situation A. (4) Given a sample of size n with r red balls and g green balls. How will further draws increase the average correctness for our decision R > G? We will discuss this in Section 2.5 for Situation A. (5) How can we determine the size n of the sample a priori, if we demand a certain level of average correctness for our decision? We will discuss this in Section for Situation A. (6) What is the correctness of our decision if only incomplete information r > g about the sample is available? In Situation B (Section 3) we deal with this kind of knowledge for the condence levels 0.90 and (7) Finally we expand the basic model to the three-valued case and give an answer to question (1) for this model (see Situation C, Section 4). 2 Situation A (Two-valued Model) 2.1 Describing the Situation ( N\ ) (Fr) i1 e i j e i 2 fr i ; g i g ball in the i-th draw" (two-valued model), where r i \red ball in the i-th draw" and g i \green (Ba) For convenience of notation, let U R;N : U R;N [ U R+1;N [ : : : [ U N;N. In the beginning no situation is preferred. We therefore apply the Principle of Indierence to the dierent llings of the urn. This yields P (U I;N ) P (U J;N ) 8 I; J 2 f0; 1; : : : ; Ng (Up) S r;n (Pr) Random draws (Go) First Goal: We are interested in the probability : P U bn2c+1;n j S r;n ; r bn2c + 1; : : : ; n; (1) P r;n

7 where bxc denotes the largest integer value less than or equal to x 2 R. z Second Goal: Given S r;n we are interested in a condence interval I ;r of minimal length and condence niveau 2 (0; 1) for the possible values of R. 2.2 Reaching the rst goal In order to determine (1) we rst consider P (U R;N j S r;n ) P (S r;n \ U R;N ) P (S r;n ) P (S r;n j U R;N ) P (U R;N ) ; R 0; : : : ; N NP 4 : (2) P (S r;n j U J;N ) P (U J;N ) Using the well-known formula for hypergeometric distributions, we can transform (2) into P (U R;N j S r;n ) (Ba) J0 P (S r;n j U R;N ) P (S r;n j U R;N ) NP J0 P (S r;n j U J;N ) R r N?R N n R r NP J0 n?r ; (3) N?R n?r J r N?J n?r R r N?R n?r N+1 n+1 : (4) The combinatorial identity used in the last equation of (4) can be found e.g. in [Jaynes, 1996], eq Substituting (2) into (1) nally gives P r;n NX RbN2c+1 P (U R;N j S r;n ) (4) 1 N+1 n+1 NX RbN2c+1 R r! N? R n? r! : (5) Example 1: Worst Case (P bn2c+1;n for N 200) Figure 1 shows P r;n for the extremal value r bn2c + 1, plotted versus the fraction nn (sample size over urn size). From this gure it is obvious that we should always choose an even sample size n in order to get a higher probability P bn2c+1;n. For an explanation of this result remember that a majority of red balls in a sample with odd (even) size is achieved when the dierence between red and green balls is least equal or greater than 1 (2). However, we will never get P bn2c+1;n to be greater than 76%, except for the uninteresting sample sizes 2; 4; 6 and 200. z The Gaussian brackets b c are used to avoid the distinction between even and odd urn or sample sizes, respectively. 4 The values 0 R < r denote impossible events with probability 0. For simplicity of the equations these events are included.

8 n even n odd n/n Figure 1: P bn2c+1;n for N 200 (Worst case) n/n Figure 2: P n;n for N 200 (Best case). Here, a distinction between even and odd n is obviously not necessary.

9 Let us now check the consistency of the stochastic model with qualitative considerations in the vicinity of nn 1 (again for N 200 and r bn2c + 1) : n 198, i.e. r 100 : U bn2c+1;n occurs, if a red and a green ball, or a green and a red ball, or two red balls appear in the last two draws; it does not occur, if two green balls are drawn. Therefore, the probability should be approximately 34 75%. n 199, i.e. r 100 : U bn2c+1;n occurs, if a red ball appears in the last draw; it does not occur, if a green ball is drawn. Therefore, the probability should be approximately 12 50%. n 200, i.e. r 101 : P 101;200 1 as the sample contains all balls of the urn Example 2: Best Case (P n;n for N 200) Figure 2 shows P r;n for the extremal value r n, plotted versus the fraction nn (sample size over urn size). Obviously, P n;n 1 for n bn2c + 1. At rst glance it is astonishing how rapidly the curve reaches the maximum value 1. From a qualitative point of view we argue as follows: The more red balls that are in the sample (in this special case only red balls were drawn), the more likely is the appearance of red balls in the following draws. A detailed and quantitative explanation of this eect could be obtained by the following considerations: In order to study the gradual behavior 5 of P n;n we dene a family of partial sums f m g of P n;n by m : P (\at least bn2c + 1? n red balls in the following m draws") ; (6) m bn2c + 1? n; : : : ; N? n, given that we already drew a sample of size n only consisting of red balls. Obviously, bn2c+1?n bn2c+2?n : : : N?n P n;n (7) and bn2c+1?n n + 1 n + 2 n + 2 bn2c+2?n (8). n + 1 bn2c + 2 n + 3 : : : n + bn2c + 1? n n + bn2c + 2? n n + 1 bn2c + 2 " bn2c + 2 bn2c (bn2c + 2? n) 1 bn2c + 3 # (8) (9) (see Figure 3 for visualisation; again for the urn size N 200). Now it should be clear why we chose m instead of other values: The m 's allow an interpretation by themselves 6, they can easily be obtained and they \converge" to P n;n [ see (7) ]. 5 Analogous to the construction of power-tail distributions as limits of truncated tails, see [Greiner et al., 1998]. 6 For instance, the rst sum bn2c+1?n, i.e. the probability for a majority of red balls in the urn after the minimal number of additional draws, is almost twice as large as the fraction nn, though it is only a small part of P n;n itself.

10 n/n Figure 3: 101?n (), 102?n (2), 103?n (+), and P n;n () for N Example 3: Determining a minimal majority of red balls for a given security level Normally we will be interested not in the extremal values of the last examples, but in the value of r : min r f r j bn2c + 1 r n ; P r;n g ; (10) i.e. the minimal majority of red balls in the sample that cause a majority of red balls in the urn with a probability equal or higher than 2 (0; 1). 7 From Figure 4 we see that the minimal fraction r n is sectionally monotonic decreasing to a value greater than 0.5 (both for the even and the odd case). Further, for n 1; 2 there is no r with P r;n 0:9, for n N2 (the sample size is half the urn size) P r;n 0:9 if at least 55% of the balls in the sample are red. It should be noted that r is a monotonic non-decreasing function in, i.e. 1 > 2 ) r 1 r ; 2 2 (0; 1) : 2.3 Connection with the Rule of Succession Let us rst quote Laplace's rule of succession (RoS): A sample of size n is drawn out of an urn containing an unknown number of red and green balls. Then the probability RoS r;n of getting a red ball in the 7 For completeness of notation we set r : 2 n (> n) if there is no r 2 fbn2c + 1; : : : ; ng fullling P r;n for a given probability level, say 2 f0:90; 0:95g.

11 n even n odd n/n Figure 4: The fraction r n for N 200 and 0:90. following draw is r + 1 n + 2 ; (11) where r again denotes the number of red balls in the sample. We will now show an alternative way of computing P r;n [ cf. (5) ] by consecutively using (RoS). Obviously, P r;n 1 for r bn2c + 1. So in the following we assume r < bn2c + 1: P r;n N?n?r X RbN2c+1 N?n?r X RbN2c+1 N?n?r X RbN2c+1 n + 1 N N+1 n+1 P (\exactly R? r red balls in the following N? n draws") N? n R? r! {z } Number of arrangements (r + 1) : : : (r + R? r) (n? r + 1) : : : (N? r) (n + 2) : : : (n N? n) [ by using (RoS) N? n times ] (N? n)! (R? r)! (N? n? R + r)! R! (n + 1)! (N? R)! r! (N + 1)! (n? r)! N?n?r X RbN2c+1 NX RbN2c+1 R r N?R R r! N n n?r N? R n? r! 1 N+1 n+1 (5) : N?n?r X RbN2c+1 R r! N? R n? r!

12 89 rr ED BC < :; rrr ED BC rrg ED BC rrrr ED BC rrrg ED BC ED BC BC BC BC BC ED BC GF?> ED BC < ED BC < GF ED 89 rrrrrr:; 89 rrrrrg:; 89 rrgggg ED BC Figure 5: A visualisation of P n;n for N 6 and n 2. Note that we did not explicitly use (Ba) in this derivation. However, (Ba) is included in (RoS) itself (the total number of balls in the urn does not appear in (RoS) but in its derivation, see e.g. [Greiner & Schramm, 1994]). 2.4 Reaching the second goal Given S r;n we are now interested in a condence interval I ;r of minimal length and con- dence niveau 2 (0; 1) for the possible values of R. As P (U R;N j S r;n ) is unimodal [ see (4) ] with a maximum value at hr r i [ see (18) ], I ;r can be determined by the following simple algorithm: 1 Let fp (0) ; P (1) ; : : : ; P (N) g be an ordering of fp (U R;N j S r;n ) j 0 R Ng 2 such that P (0) P (1) : : : P (N) 3 J : N 4 SUM : 0 5 REPEAT 6 SUM : SUM + P (J) 7 J : J? 1 8 UNTIL (SUM ) It is easily seen that I ;r : [R;r lower ; R;r upper ] is the union of all values of R that were used in the construction of SUM.

13 n/n Figure 6: The condence interval I 0:9; r for r bn2c + 1 and N Example 5: Worst Case (P bn2c+1;n for N 200) In the worst case r bn2c + 1 Figure 6 shows a typical condence interval, i.e. increasing lower bounds and diminishing upper bounds Example 6: Best Case (P n;n for N 200) In the best case r n, we can obtain from Figure 7 the dierent values for the lower condence bound R;r lower while the upper condence bound R;r upper is equal to N The advantage of (further) drawings { an analysis by decision theoretic considerations Let us now consider the advantage of further drawings, given a sample S r;n. In discussions we came across two dierent intuitive views: By drawing additional balls we increase our knowledge about the distribution in the whole urn. As we reduce the possible llings of the urn, we therefore expect more correct decisions when guessing the content of the urn. If we have a sample S r;n with r b n 2 c + 1 red balls and we draw another red ball the condence in our decision for R > N? R increases. On the other hand, if we draw a green ball our condence obviously is weakened. So why should we draw another ball if we cannot guarantee that we will at least keep our current level of condence?

14 n/n Figure 7: The condence interval I 0:9;r for r n and N 200. At a rst glance these intuitions seem to be inconsistent. We will therefore show in detail that both describe a correct aspect of the given situation. We rst recall some terms from Sections 2.2 and 2.3. By the theorem of total probability and (11) we immediately get P r;n P r+1;n+1 RoS r;n + P r;n+1 RoS n?r;n (12) in analogy to the probability tree in Figure 5. Please note that in (12) and in the further context we implicitly extend the denition of P r;n (cf. (1)) to arbitrary values of r between 0 and n. In contrast to the previous sections we are not interested in the probability P r;n itself but in a measure of the probability of a correct decision (more red than green balls in the urn or otherwise) based on a given sample. We chose D r;n : maxfp r;n ; P n?r;n g P maxfr;n?rg;n (13) as a suitable measure for that purpose, because if P r;n < 0:5 we will of course decide that N? R > R and be correct with probability P n?r;n. Using (12) and (13) we nally get an inequality for the D r;n D r;n D r+1;n+1 RoS r;n + D r;n+1 RoS n?r;n (14) that can be regarded as a decision tree. The sign immediately results from the maximum operator in (13).

15 m Figure 8: D m r;n for r 15, n 25, and N 200. Now we are able to determine the gain of correctness D m r;n for m additional drawings, m 1; : : : ; N? n, as [(the actual correctness after n + m drawings) - (the actual correctness after the initial n drawings)], i.e. D m r;n 2 mx 6 4 s0 m s! sq i1 Q (r + i) m?s (n? r + i) i1 mq D r+s;n+m 7 5? D r;n ; (15) (n i) i1 3 where s denotes the number of red balls in the additional sample of size m. It is obvious that such a gain can only be achieved if m is greater or equal to 2r? n Example 7: D m r;n for r 15, n 25, and N 200 It can easily be seen in Figure 8 that indeed m 5 ( 2r? n) is some sort of threshold value: Only above that value is a gain achieved (as supposed in the rst intuitive view), whereas below that value positive and negative eects neutralize each other resulting in no further gain (compare the second intuitive view). 8 As there is a majority of red balls in the rst sample of size n, i.e. r b n 2 c + 1, an additional sample of size m < 2r? n would be of no inuence to the decision whether there is a majority of red balls in the urn or not.

16 N m 0:85 m 0:90 m 0: ? z Table 1: m for dierent values of and N Application: A priori calculation of the necessary sample size for a given security level Up to now we assumed that a sample S r;n was already drawn. However, it can be of considerable interest 9 to determine a priori the minimal number of balls to be drawn in order to gain a certain level of security for the chosen decision. In order words: we want to determine o m : min nm j D m 0;0 : 1mN Table 1 shows how rapidly m stabilizes for increasing N; for N > 1000 it is almost independent of N. In the appendix we will further present simulation results that support these analytic calculations. 3 Situation B 3.1 Describing the situation (Fr) as in Situation A (Ba) as in Situation A (Up) A sample of size n with rg > 0:5 (incomplete knowlegde about the sample) 10 (Pr) random draws (as in Situation A) (Go) We are interested in the probability P maj : P U bn2c+1;n j S bn2c+1;n (16) that an arbitrary majority of red balls in the sample means that we have a majority of red balls in the urn. 9 e.g. for organizers of software or hardware competitions (see [Sutcliffe & Suttner, 1996]) 10 For example, this question occurs when someone else draws the sample and only tells us that there was a majority of red balls in the sample z A? denotes that the demanded level of correctness,, could not be reached.