Assessing bandwidth selectors with visual error criteria

Transcription

1

2

3 Assessing bandwidth selectrs with visual errr criteria J. s. Marrn* Department f Statistics University f rth Carlina Chapel Hill,.C U. S. A. May 12, 1993 Abstract "Visual errr criteria" have been develped as a means f measuring the distance between curves, in a way that crrespnds mre clsely t "what the eye sees", than the usual nrms n functin spaces. The main cntributin f this paper, is the applicatin f this idea t the cmparisn f data based bandwidth selectin methds. t is seen that which bandwidth selectr is "best" is ften much different in this sense, but clser t "visual best", than what is btained frm classical methds f measuring errr. n additin, new insights int visual errr criteria are prvided. 1 ntrductin Chice f the smthing parameter, i.e. bandwidth, is crucial t the effective use fnnparametric curve estimatrs. The prblem is als f current interest This research partially supprted by.s.f. Grant DMS The authr is very grateful fr the chance t present this material at the Cnference spnsred by the Academia Sinica, and fr the many useful cmments made during the cnference. 1

4 because it has clse relatinships t ther imprtant prblems, such as mdel selectin. There is a large literature n data based smthing parameter selectin. See Jnes, Marrn and Sheather (1992) [6] fr a recent survey. The deepest research n data based smthing parameter selectin has been dne in the cntext f kernel density estimatin, s that case will be studied here t. Hwever, the main ideas and lessns f this paper clearly apply t ther types f curve estimatin as well. Kernel density estimatin can be viewed as an attempt t use a randm sample Xl,., X" frm a prbability density f (x) t try t recver the curve f (x). Given a kernel functin K(x) (taken in all examples here as the Gaussian prbability density), and a bandwidth h, the kernel density estimatr is: where Kh (.) = kk (x). The bandwidth is represented as a separate parameter because "width f the kernel windw" is crucial t the perfrmance f the estimatr. See Silverman (1986) [12] fr illustratin f this pint, as well as discussin f many ther useful aspects f this estimatr. n this paper the bandwidth selectin prblem is studied with careful attentin paid t the issue f hw t assess the perfrmance f varius bandwidths. There has been cnsiderable past cntrversy n this issue, mainly centered arund whether ne shuld measure errr based n the Ll r n the L2 nrms. Hwever, it is seen in Marrn and Tsybakv (1993) [7] that in sme situatins neither f these is particularly attractive, and indeed "visual chice f bandwidth" (the methd used mst cmmnly in actual data analyses) can be quite different than either the Ll r the L2 ptimal chices. Marrn and Tsybakv g n t develp and investigate alternative measures f errr which crrespnd mre clsely t "what the eye sees". n this paper, their prpsed errr criterin V E 2 (h + J) is studied. Figure 1 shws hw V E2(!h + J) assesses bandwidth selectrs in a way that is clser t visual perceptin than L2. [put figure 1 abut here] FGURE 1: A target density (heavy line) and kernel density estimates (thin lines), frm ten simulated data sets (same fr each 2

5 bandwidth selectr), each fsize n = 100, fr fur majr methds f data based bandwidth selectin: (a) LSCV, (b) ROT2, (c) SJP, (d) RSB. Bandwidth selectin methds are cmpared by bth L2 and V E2(!h + J), averaged ver the 10 data sets shwn here. The errr criterin V E 2 (!n + J) is defined in the next sectin. Precise frmulatin f the bandwidth selectrs; LSCV (Least Squares Crss Validatin), ROT2 (Rule Of Thumb 2), SJP (Sheather Jnes Plug n), and RSB (Rt Smthed Btstrap), with suitable references, is given in the appendix. te that LSCV is better than S J P and RS B in the sense f L2, but wrse with respect t V E2(!n + J). But the visual perfrmances f SJP and RSB are clearly superir. SJP lks slightly better than RSB, which is als reflected by V E2(!h + J). The main pint f this paper is t study bandwidth selectin methds thrugh criteria which wrk differently than the usual nes, but are mre useful fr determining perfrmance in real data situatins. This example is clearly a case where sme type f lcatin varying bandwidth is called fr. n particular, a rather larger bandwidth is desirable n the left side, while a simultaneusly smaller bandwidth will imprve perfrmance n the right side. Hwever, such methds are cmplicated, and are typically used nly after a need fr them is understd thrugh a preliminary analysis with glbal bandwidth methds. Hence "best" (albeit ut f an unsatisfactry set) chice f glbal bandwidth is still a vital issue even in situatins such as this. n sectin 3, previusly unknwn facts abut V E2(!h + J) are described. Different types f cmparisns are made between L2 and V E2(!n + J). One f these is a study f their expected values as functins f nand h. Behavir f their respective minimizers is als studied. n sectin 4, it is seen that the measure VE 2 (!h + J) prvides an imprtant new viewpint fr the study f data based bandwidth selectin, with sme substantially different answers than thse btainable frm the L2 viewpint. The basic recmmendatin f Jnes, Marrn and Sheather (1992) [6], that the SJP methd is very gd all arund still remains the same. Hwever it is nw mre clear that LSCV is nt apprpriate. A surprising feature (at least t the authr) is that ROT2 is mre frequently nt s bad as had been previusly reprted. The RS B is als nw seen t be better 3

6 than had previusly been thught, which is mre cnsistent with its excellent asympttic prperties. 2 Errr criteria The gal f ptimizing the L2 nrm is the same as ptimizing it's square, the ntegrated Squared Errr, SE(h) =J(lh(x) f(x)f dx. Here and in the fllwing, the integral symbl is understd t mean definite integratin ver the whle real line. Sme researchers prefer t wrk with the expected SE, ften called the Mean ntegrated Squared Errr, MSE(h) = E [SE(h)]. As nted abve, the visual errr criterin V E 2 (lh + J) was develped in Marrn and Tsybakv (1993) [7]. See that paper fr mtivatin and discussin. This is defined as fllws. A cntinuus functin f : [a, b] + R can be represented by its "graph in R2", G J = {(x, y) : x E [a, b], y = f(x)} C 1R 2 A basic building blck fr cnstructing "distances" between tw graphs is the distance frm a pint (x, y) t the graph G, d«x,y), G) = inf l(x,y) (x',y')112 (x',y')eg i.e. the shrtest distance frm the given pint (x, y) t any pint in the clsed set G, where dentes the usual Euclidean distance. These distances are summarized, in a squared integrated way t yield the asymmetric "visual errr" criterin Many variatins n this idea are pssible, but reasns behnd this particular chice are detailed in Marrn and Tsybakv (1993) [7]. n that paper, a 4

7 symmetrized versin f V E 2 (h + J) als is seen t have useful prperties. Hwever, that is nt treated here because V E 2 (lh + J) is seen there t be superir at tracking an experienced data analyst's desire that the estimate shuld shw n mre features than can reasnably be btained frm the data. Fr cmputatinal feasibility, a discretized versin f V E 2 (h + J), invlving bth discretized graphs, and als a Riemann sum apprximatin, ver an equally spaced fine grid f 400 pints has been used in all examples here. See Marrn and Tsybakv (1993) [7] fr details. There has been an interesting cntrversy ver which shuld be called the "ptimal bandwidth in the L2 sense", between the respective minimizers f SE and MSE. Passins run surprisingly strng n bth sides, even t the pint f peple n each side being unwilling t even admit that a cntrversy exists, feeling that their view is s bviusly crrect. See Grund, Hall and Marrn (1992) [3] fr references and discussin. n that paper it is seen that despite imprtant mathematical and cnceptual differences, there is ften little difference in terms f assessing bandwidth selectrs in mst practical situatins. Hwever MSE des give sme what better "reslutin" in cmparing bandwidth selectrs than SE, s that is used in the rest f this paper. n the empirical wrk dne fr this paper, it has been bserved that this same lessn hlds fr V E 2 (lh + 1)2 vs. EVE 2 (h + 1)2, s the latter will be used fr assessment f bandwidth selectrs thrughut this paper. Explicit details are nt presented here, because the main ideas are essentially the same as thse f Grund, Hall and Marrn (1992) [3], and thus d nt seem t be wrth the jurnal space. Hwever detailed summaries are available frm the authr. 3 ew facts n visual errr Part f this research invlves studying the errr criterin in a variety f senses, in examples where sme nrmal mixture densities were used fr the underlying. n particular analgs f all figures have been cnstructed fr all f the 15 target densities f Marrn and Wand (1992) [8], but nly sme representative nes are presented here. The underlying density in Figure 1 is density number 14 frm that paper. 5

8 nsight int hw the average curves MSE(h) and EVE2(h) (as functins f h) cmpare may be btained frm Figure 2. [put figure 2 abut here] FGURE 2: Plts f MSE(h) (slid curves) and EVE2(h) (dashed curves) as functins f the bandwidth h, fr n = 100, 1000, 10,000 and 100,000. The underlying densities are: (a) number 2, (b) number 4, frm Marrn and Wand. Fr each n, a grid f 49 (lgarithmically) equally spaced bandwidths, frm lhmse t 9hM1SE is used. Mnte Carl errr in calculatin f EVE 2 (lh f)2 is seen t be negligible thrugh the inclusin f 95% pintwise cnfidence bands, as dtted curves. Vertical lines shw minimizers f MSE (slid) and EVE2 (dtted). There is n errr in the calculatin f the MJSE curves, since the exact frmulas in Marrn and Wand (1992) [8] were used. Hwever the EVE2 curves were calculated by simulatin, in particular by averaging V E 2 (h J) ver 500 pseud data sets (fr each n) at each bandwidth. S there is sme Mnte Carl errr t take int accunt. This is dne in Figure 2, thrugh shwing upper and lwer 95% pintwise asympttic cnfidence bands (cnstructed using the classical asympttic nrmality frmula: X ±1.96). te that these dtted lines are, at mst h lcatins, nearly indistinguis able frm the average curve, s the 500 replicatins is enugh fr at least mst visual lessns t be crrect. Figure 2a shws this cmparisn fr density number 2. Similar pictures were btained fr density numbers 1, 6, 7,8, and 9. These are essentially the "easy t estimate densities", where mst f the features f the underlying f can be recvered well with nly mderate sample sizes. n these situatins, there will be little difference between assessing the perfrmance f a data driven bandwidth, say h, by MSE(h) r by EVE2(h). This is in keeping with the findings f Marrn and Tsybakv (1993) [7] that in many situatins, all errr criteria wrk in essentially the same way. Als, as expected frm that paper, MSE(h) is cnsistently larger than EVE2(h). Figure 2b shws the results fr density number 4, and is fairly representative f the results fr the remaining densities. These are mstly characterized as "hard t estimate" densities, which have at least sme features which ne 6

9 can nt expect t cmpletely recver frm a sample f nly mderate size. te that fr n = 100, 1000 and 10,000, majr differences will shw up when bandwidth selectrs are cmpared by the 2 methds. Hwever, fr n = 100,000 the answers will be clser. This is because fr such large n, this density n lnger has features that are "t hard t btain", and hence jins the first grup, represented in Figure 2a. te that EVE2(h) tends t be "flatter" in these situatins (which means nt such large penalties fr being further frm the minimizer), and als much smaller than MSE(h), which als is nt surprising in view f the intuitin in Marrn and Tsybakv (1993) [7]. A disturbing feature f Figure 2b is the "wiggles" n the right hand sides f the EVE2(h) curves fr large n. te that the cnfidence bands indicate this is systematic, and nt due t Mnte Carl variability (as is indicated by the fact that these wiggles "line up" fr n = 10,000 and 100,000). This has nt been carefully investigated, but seems t be due t the discretizatin in the numerical calculatin f V E 2 (h, + ). n particular, all f these densities have regins where 1!'(x)1 is extremely large (the "slpes f the spikes"). n such regins there can be substantiallr distance between successive pints in the discretized graphs. This appears t make sme bandwidths less desirable than thers. te this effect nly ccurs n the right sidefthe curves, where bias is dminant. On the left side, large randm variability in the estimates entails that in averaging this effect disappears. An unpleasant feature f these "wiggles" in EVE2(h) is that many lcal minima are created. Lcal minima can als appear in MSE(h), as seen in Marrn and Wand (1992) [8], but this happens far less frequently. Even mre lcal minima were encuntered in the randm curves V E 2 (lh + ). This is cnsistent with the fact that SE tends t have mre lcal minima than MSE, as quantified in Hall and Marrn (1991) [4]. A cmpletely different way f lking at V E 2 (lh + ) is t study hw its minimizer (where ties are brken by taking the smallest f the set f minimizers), h V E2' cmpares fr given data sets t ther ntins f "ptimal" bandwidth. Here it is cmpared with hse, the minimizer f SE, and hae, the minimizer f the ntegrated Abslute Errr, AE =Jh,(x)!(x)1 dx, and als hsup the minimizer f the supremum nrm, supz lh(x)!(x)l 7

10 Figure 3 gives an indicatin f the multivariate distributin f the randm vectr (hae hse hsup hye2 )T. [put Figure 3 abut here] FGURE 3: Draftsman's scatterplts f the multivariate distributin f hae, hse, hsup, hye2. Frm 500 pseud data sets fsize n = 100, frm (aj density 2, (bj density 3. All scatterplt axes are n scale flg 3(h) Og3(hM1SE ). Diagnal plts shw kernel density estimates f univariate marginal density fr each bandwidth, tgether with vertical dtted lines shwing the quartiles f these distributins. Figure 3a is representative f the "easier t estimate densities", numbers 1, 2, 6, 7, 8, and 9. te that all ntins f "ptimal bandwidth" rughly cincide here, with a very strng crrelatin between them, althugh less s fr hsup. This is because fr these densities all ntins f ptimal "feel rughly the same aspects f the data and the density" (althugh this is least true fr hsup). Figure 3b shws typical behavir fr the "harder t estimate densities", numbers 3,4, 10, 12, 13, 14 and 15. te that the varius ptimal bandwidths nw behave much differently frm each ther, bth in terms f much smaller crrelatins, and als different mean behavirs. This is because "each feels different aspects f the variability in the data", in particular hsup is driven by behavir at the peak, while h Y E2 is mre strngly influenced by the flat parts f density, while hse and hae feel different things in between these extremes. One interesting exceptin t the abve pattern was density number 5. This is nt shwn explicitly, t save space, but the selectrs hae, hse, and hsup behave much the same as in Figure 3a, with very strng crrelatin and similar means. Hwever the distributin f hye2 is substantially larger than the thers, and als has essentially n crrelatin, because this methd feels behavir in the tails f distributin. Anther interesting exceptin is density number 11, in the case n = 100. Here hae, hse, and hye2 wrk much as in Figure 3a, since with respect t these criteria, this density is essentially the same as number 6. hwever the sup nrm feels behavir f the estimate at the thin tall spikes almst 8

11 exclusively. Hence it is nt surprising that hsup is uncrrelated with the thers, and als spread ver a very wide range. 4 Cmparisn f Bandwidth Selectrs A criterin which has been used fr the simulatin cmparisn f data driven bandwidth selectrs, see fr example Jnes, Marrn and Sheather (1992) [6], S MSE GRT =avg MSE(h) MSE(hM1SE ), h MSE(hM1SE ) where the average is taken ver simulated realizatins f h. A direct analg f this, based n EVE2(h) is EVE2 _ GRT =avg EVE(h) EVE2(hEVE2 ), h EVE2(hEVE2) where heve2 is the h t minimize EVE2(h). These tw ways f cmparing data driven bandwidths are cmpared in Figures 4 and 5. These are visual summaries, intended t prvide mre ready cmparisns f several types than the usual set f tables (whse presentatin wuld be quite vluminus and tedius t interpret in this particular cntext). [put figure 4 abut here] FGURE 4: Visual displays f summary statistics fr cmparisn f bandwidth selectrs, ver alli5 target densities in Marrn and Wand, fr (a) n = 100 and (b) n = n each case the densities are rdered by MSE GRT fr SJP. Means and standard deviatins are n the scale flg3(h) lg3(hm1se ). Since the density numbering in Marrn and Wand (1992) [8] was nt intended t be relevant fr bandwidth selectin, varius rerderings have been experimented with. Mst infrmative seems t be t rder in terms f MSE GRT fr a bandwidth selectr f interest. te that in the MSE sense, the methds are nt cmparable, with n methd being dminant in all situatins. Hwever, Jnes, Marrn and Sheather (1992) [6] suggest SJP 9

12 as a reasnable "all arund favrite", because it is never t far frm the best (while each f the thers is far frm best fr at least sme densities). Reasns fr the varius perfrmances are seen clearly by studying the means and variances f the given methds. Fr example, when LSGV gives pr perfrmance in the MSE sense, it is because it has t much variability (which is already well knwn). When LSGV wrks well, it is because it has substantially less bias than the thers. The bandwidth selectr ROT2 gives generally pr perfrmance because, while it has smaller than usual variance (because the nly estimatin is f "scale" versus the mre challenging estimatin that is at least implicitly attempted by the ther methds). Further insight int this "variance bias" effect cmes frm Figure 5. [put figure 5 abut here] FGURE 5: Kernel density estimates shwing the distributins f data driven bandwidths (with same line types as in Figure 4), fr n = 100, and each f the target densities, again standardized t the scale lg3(h) lg3(h M1SE )' The verlayed slid curve is MSE(h) and the dashed curve is EVE2(h). MSE GRT and EVE2 GRT behavir is understd by "plugging the densities int the crrespnding curve". te that sme f the MSE GRT summaries in Figure 4 are nt particularly useful, because the MSE sense f a gd bandwidth is different frm visual impressin. This is rughly the same phenmenn bserved in Figure 1. T see hw the lessns f Figure 4 change, when MSE GRT is replaced by EVE2 GRT, see Figure 6. [put figure 6 abut here] FGURE 6: Visual displays f summary statistics fr cmparisn fbandwidth selectrs, ver all 15 target densities in Marrn and Wand, fr (a) n = 100 and (b) n = n each case the densities are rdered by EVE2 GRT fr S J P. Means and standard deviatins are n the scale flg3(h) lg3(h M1SE )' The means and standard deviatins are again displayed, because the rdering f the densities is nw different frm that f Figure 4. te that 10

13 SJP is still never t far frm best, s the previus recmmendatin f this as a general purpse default seems t remain acceptable. Hwever, the relative standing f LSCV has changed a lt. t ges frm being best a fair number f times in the MSE sense, t being almst unifrmly wrst in the EVE2 sense, and ften by quite a lt. Figure 5 shws that it's difficulties are a cmbinatin f "unbiased with respect t h M1SE " actually being a drawback in this sense, and it's very large variance. This interpretatin f the perfrmance f LSCV is much mre in keeping with intuitive and practical pinin f its behavir, see fr example Sheather (1992) [10]. Anther nticeable difference between Figures 4 and 6, is that ROT2 lks much better nw, at least fr n = 100. This is als in keeping with anecdtal evidence several researchers have expressed in persnal cnversatin. An inspectin f Figure 5 shws that the main reasn fr this gd perfrmance is the lw variance f ROT2 cmbined with its ften being frtunately clse t h EvE2 This clseness disappears fr n = 1000, where this gd perfrmance falls ff significantly. Finally nte that RS B nw lks better with respect t S J P than it did in the MSE sense. This is interesting because RS B has superir asympttic perfrmance, and it has been previusly thught that these asympttics required unreasnably large sample sizes befre the superir perfrmance f RS B appeared. A fascinating area fr future research is t design bandwidth selectin methds which specifically target the criterin EVE2. References [1] Bwman, A. W. (1984) An alternative methd f crssvalidatin fr the smthing f density estimates, Bimetrika, 71, [2] Fan, J. and Marrn, J. S. (1993) Fast implementatins f nnparametric curve estimatrs, unpublished manuscript. [3] Grund, B., Hall, P. and Marrn, J. S. (1992) Lss and risk in smthing parameter selectin, unpublished manuscript. [4] P. Hall and Marrn, J. S. (1991) Lcal minima in crssvalidatin functins, Jurnal fthe Ryal Statistical Sciety, Series B, 53,

14 [5] Jnes, M. C., Marrn, J. S. and Park, B.U. (1991) A simple rt n bandwidth selectr, Annals f Statistics, 19, [6] Jnes, M. C., Marrn, J. S. and Sheather, S. J. (1992) Prgress in databased bandwidth selectin fr density estimatin, unpublished manuscript. [7] Marrn, J. S. and Tsybakv, A. B. (1993) Visual errr criteria fr qualitative smthing, unpublished manuscript. [8] Marrn, J. S. and Wand, M. P. (1992) Exact mean integrated squared errr, Annals f Statistics, 20, [9] Rudem, M. (1982) Empirical chice f histgrams and kernel density estimatrs, Scandinavian Jurnal f Statistics, 9, [10] Sheather, S. J. (1992) The perfrmance f six ppular bandwidth selectin methds n sme real data sets, Cmputatinal Statistics, 7, [1l] Sheather, S. J. and Jnes, M. C. (1991) A reliable databased bandwidth selectin methd fr kernel density estimatin, Jurnal f the Ryal Statistical Sciety, Series B, 53, [12] Silverman, B. W. (1986) Density Estimatin fr statistics and data analysis, Chapman and Hall, ew Yrk. 5 Appendix: Bandwidth selectrs n all cases, calculatins were dne using the ideas behind the fast "binned" implementatins described in Fan and Marrn (1993) [2]. This invlves evaluatin at an equally spaced grid f pints, which was taken t be 400 pints n [3,3]. Fr each f these bandwidth selectrs, nly a quick definitin, and a few imprtant facts are mentined. See Jnes, Marrn and Sheather (1992) [6] fr further discussin. 12

15 5.1 Least Squares Crss Validatin This methd was prpsed by Rudem (1982) [9] and Bwman (1984) [1]. This bandwidth is taken t be the minimizer (in the case f multiple lcal minimizers, the largest h value is used here, as that seems t give the best perfrmance) f the functin LSGV(h) = Jh (x)2 dx 2n 1 t h,i (x) i=1 where h,i(x) is the kernel density estimate based n the sample with Xi deleted. This minimizer has an intuitively attractive cnnectin t h/se. Hwever, it is knwn t have rather pr practical and asympttic prperties. 5.2 Sheather Jnes Plug n This methd was prpsed in Sheather and Jnes (1991) [11]. This bandwidth selectr is the slutin (taken t be the largest h value when there are multiple slutins) f the fixed pint equatin (in the variable h) where 92(h) = G(K)D(hl)h 5 / 1 where C(K) and D(f) are apprpriate cnstants, and h is a pilt estimatr, whse bandwidth is chsen by the nrmal reference distributin methd. This bandwidth selectr is knwn t have much better asympttic and als applied perfrmance than h LSCV 5.3 Rule Of Thumb 2 This methd is prpsed in Silverman (1986) [12].This is defined as 13

16 where r.p; dentes the nrmal prbability density with variance 02 = min (82,)' where 8 2 is the usual sample variance, QR is the sample nter Quartile Range, and QRp is the nter Quartile Range f the standard nrmal distributin. Asympttically, this methd is far wrse than any f the thers here, because (fr f different frm the Gaussian in shape) it des nt cnverge t any ntin f "ptimal bandwidth". Hwever, it has the advantage that it has less variability, because nly the scale is estimated. 5.4 Rt Smthed Btstrap The versin f this used here is that f Jnes, Marrn and Park (1991) [5]. This is the minimizer (taken as the glbal minimizer in the case f lcal minima) f the functin where g2(h) = C(<)D(hl )n23/4sh2 where C(<) and DU) are different apprpriate cnstants, and hi is a pilt estimatr, whse bandwidth is chsen by a nrmal reference methd. This bandwidth selectr has excellent asympttic prperties, even a fast "rt n" rate f cnvergence usually encuntered nly in parametric cntexts. 14

17 10 01 Figure 1a, i 'A i, i, i i i, 'i i i i, i i,,, i j 0 t i i, 10 Figure 1b Avg Lz = 0,110,. "':[ _ Ava VE., = ,, >0 >0 +J +J 'iii 'iii c c u U "0 " x x Figure 1c ci i,,, i,,, i i,,, i i, i,, i,,, i,,, 1 10,. 0 0,, >0 >0 +J /) 'iii c c u U "0 "0 0 0 Figure 1d i i 0 t Avg Lz = 0,091 "':l /. _Ava VE., = x x

18 Figure 2a O t,_.r_,,,._,,.,rt._,._,t._. tj.!.2&5""""''''''2.0''''''''''''''1''.501.''''''''''''''''1''''.0''''''''''''''''''''''0.5''''''''' L0910(h) Figure 2b O t.,_,.,.r,,.,.,,..,r.,t._. "i:'c('., lu ọ Ot.3 '7 T. tj.!2&5"" ol L0910(h)

19 ,.!. ", ri.;.!.. " i.i.,. + *. : " ; '''':";":";';'':'':''':''':':''l f t!' e..!.r :.'._;.,.!. "...;<>" '!"!.,!.....:,.İ ; L..!. " E İ.!,;;:;;.,,.,,.,.,,,.,,.Ji ( f. (!'!. ; e ",.!.;.,;_i t.,. : İ ; r Jw,.!.r;.,;_i " l:: ;.: !.r.,.;;:::".,.} l { " :.:'.;.0 ;!.r......:. :.'. " 0 ;!.r,.. + *... Jw :.' ;!.r.'. +.!',!, J!. 1.İ E.İ E.İ E

20 2 1 ". dntr.. "'"".!..!..!..! { { { i. f!. { ++ [.!> f. l:: C +., " "'. a. a a. a "'""., +.!..!..!..!. t:...++ t i +.t c i.,.+, t "., + [ + + f s.,!>!> iii!!: :::i g. 0. "'"".!..!..,. "' a., +A ' ' t + + i "'+ ( (!> J t (. +'. t: G!!: (.!. ".s: dntr ". "' a. a.. a.!..!..!..!. 141 i f!l. 1!> ;; 1++:+:; *

21 Figure 4a. Summary f MSECRTs Figure 4a. Summary f Means <' iii iii iii iii iii' p u w. en '(). D>. K, "" / 1/ '/. V V"' /1 / :..!., fr", / V....../ V V,. ". t c:. u 0 U 0 c:.b 0 en,., B density number 1"' h,! 2 J 5 b 4 4! density number t d Figure 4a. Summary f Standard Deviatins. d en,., 0 d u. 0 5 d J en d V '"., r / ' /./ ' t' '" density number : : : : :.. ' t' K v. V t M f ' 1': 7 ' n... '".. LSCV ROT2 SJP RSB

22 Figure 4b, Summary f MSECRTs Figure 4b, Summary f Means, c 0 u w!!!, 0 C',,., ". ) / 1 / / 1 [ / 1 tl./"" 1 r y 1 / 1. 'j V.""'. V '..f / V r../ V V. /". ) B 9 < density'number '" c D ) _ '0 ) D '0C.B 0 ) 1 / /, / 1'".. 1'" ' density'number // / l V l, r, r / ) '" V. b Figure 4b, Summary f Standard Deviatins ci r.,.j",,,,r"'jrjr."""'""j"rr,,.,i,,,.,. ci 0 ) t') '0 ci ).!::! 'E D 'tj C. D 0 ) LSCV ROT2 SJP RSB ci L+r"""":J:+:!""",*+!+.;:*h:h."'"

23 .0' / / ""'",,. ",.. u..,... _ t law " / / 1. /"" ,..,,,,. law. law law T' / / ""'" ""'" ""'"./' "" " /, a ,, law _ r;,.. _. law. law. ""'" ""'" ""'" / law law 11j " " / ""'" ""'" / ""'" 't., c r /1 ' /,, 1, " 0 ' ",t,,,c,t.c...,... law " ""'" / ""'" ""'" " "P.,. _u,., 0'+.., /

24 Figure 6a, Summary f EVE2CRTs Figure 6a, Summary f Means 'P c u W. D.Q r. 1 0 V.. / V /,,/ Jr / [, 0/. " / / / /' f t1,. /' J C. u :::E 'tl U 'tl C.B 0 til.,. V... V r/' "'.!==.,. V "',, 4 J:. 4.., 4"1r 4 4.,.,.. "" density number,. n. 0 r ' 'Q density number "til, 'J 'tl t) 10 ci ci :i c: 'tl j 0 til ci ci Figure 6a, Summary f Standard Deviatins r T., Til ; ' 1 T.. '. '. '. : : : : ' densitynumber ", ' :. ; ':'1./ ".V 1/ : L> f ' A ft 0, /,/ if" _111 /...n! _!! i. LSCV ROT2 SJP RSB

25 'P c u [;j W. ''" 01.2, 15 ST:r1l, Figure 6b. Summary f EVE2CRTs Figure 6b. Summary f Means l t,. 1'. 1, ", ': '":' " :,,. J. j "".. '/ " ll/,., y ',, ':.., ':, /: / } : : vi.b:?' r r! density number./, ' / V 101+ B3 t:. ) "0 ) "0 l: E 0 lj).., 15 5 " 1/.,f 1/ '.",7 / 1 V.. /. V. V. " "":': V, 13T density number B43 ci Figure 6b. Summary f Standard Deviatins ci lj), "0 ci ). "E "0 5 ci lj) ci,.,..,.,' V b. r V, 1" '.., ' V'. r rt V..... density number ", LSCV ROT2 SJP RSB