Contents. 2 groups n groups (n > 2) (independent) unpaired. paired t -test. one-way ANOVA ANOVA. (related) paired. two-way ANOVA.

Transcription

1 Statstcal ests for Computatonal Intellgence Research and Human Subjectve ests Sldes are downloadable from Hdeyuk AKAGI Kyushu Unversty, Japan ver. March 6, 0 ver. July, 0 ver. July, 0 ver. Aprl, 0 un un un t -test t -test Contents groups n groups (n > ) sgn test Wlcoxon sgned-ranks test + Scheffé's method of comparson for Human Subjectve ests ftness How to Show Sgnfcance? Just compare averages vsually? It s not scentfc. ftness conventonal conventonal EC EC proposed EC proposed EC generatons generatons Fg. XX Average convergence curves of n tmes of tral runs. How to Show Sgnfcance? Sound desgn concept: extng sound made by conventonal IEC sound made by proposed IEC sound made by proposed IEC Whch method s good to make extng sound? How to show t? You cannot show the superorty of your method wthout statstcal tests. Papers wthout statstcs tests may be rejected. statstcal test My method s sgnfcantly! un un Whch est Should We Use? un t -test t -test groups n groups (n > ) Whch est Should We Use? groups n groups (n > ) Whch est Should we Use? groups n groups (n > ) Whch est Should We Use? groups n groups (n > ) un un un t -test t -test Mann-Whtney U-test Kruskal-Walls test Fredman test un un un t -test Normalty est Anderson-Darlng test t -test D'Agostno-Pearson test Kolmogorov-Smrnov test Shapro-Wlk test Mann-Whtney U-test Jarque Bera test Kruskal-Walls test un un un un t -test group A group B t. -test Mann-Whtney U-test ntal conven proposed # tonal Kruskal-Walls test Fredman test

2 Whch est Should We Use? groups n groups (n > ) Whch est Should We Use? Q: Whch tests are more senstve, those groups for un orn groups? (n > ) A: Statstcal tests for because of more nformaton. Whch est Should We Use? Q: How should you desgn your expermental condtons groups to use statstcal ntests groups for (n > ) and reduce the # of tral runs? A: Use the same ntalzed for the set of (method A, method B) at each tral run. un un un t -test A group B group t -test ntal # GA proposed un un un t -test A group B group t -test ntal # GA proposed un un t -test t -test sgnfcant? un un un Whch est Should we Use? Q: Whch statstcal tests are senstve, parametrc groups tests or non-parametrc n groups ones (n > ) and why? A: Parametrc tests whch can use nformaton of assumed. t-est groups n groups (n > ) t-est How to Show Sgnfcance? un un t -test t -test un un t -test t -test t -test sgnfcant? un un t-est t-est t-est F-est Excel ( bts verson only?) has t-tests and n Data Analyss ools. You must nstall ts add-n. (Fle -> opton -> add-n, and set ts add-n.) A B est ths dfference wth assumng no dfference. (null hypothess) sgnfcant dfference? Condtons to use t-tests: () normalty () equal varances (not essental though ) A B When 0 (p > 0.0), we assume that 9 there s no sgnfcant dfference between σ A and σ B est ths dfference wth assumng no dfference. (null hypothess) Normalty est sgnfcant Anderson-Darlng test dfference? D'Agostno-Pearson test Kolmogorov-Smrnov test Shapro-Wlk test Jarque Bera test Condtons to use t-tests: () normalty () equal varances (not essental though )

3 t-est t-est t-est () t-est: Pars two sample for means sgnfcant? () t-est: wo-sample assumng equal varances hs s a case when each par of two methods wth the same ntal condton. () t-est: wo-sample assumng unequal varances: Welch's t-test sample A B t-est: Pared wo Sample for Means Varable Varable Mean.897. Varance Observatons 0 0 Pearson Correlaton Hypotheszed Mean Dfference 0 df 9 t Stat.7996 P(<=t) one-tal t Crtcal one-tal.89 P(<=t) two-tal t Crtcal two-tal.676 sample When p-value s less t-est: than 0.0 Pared or 0.0, wo we Sample assume that for Means there s sgnfcant dfference wth the level A B Varable of Varable of (p < 0.0) or (p < 0.0)... Mean.897. Varance Observatons % Pearson.% Correlaton %. A. > B A BHypotheszed A < B Mean When A>B never 0happens, Dfference you may use a one-tal test df 9 t Stat P(<=t) one-tal t Crtcal one-tal P(<=t) two-tal t Crtcal two-tal.676 t-est : Analyss of Varance groups n groups (n > ) : Analyss of Varance () t-est: Pars two sample for means () t-est: wo-sample assumng equal varances Dfference between two groups s sgnfcant (p < 0.0). We cannot say that there s a sgnfcant dfference between two group. un un un t -test t -test Mann-Whtney U-test Kruskal-Walls test Fredman test sgnfcant? : Analyss of Varance. Analyss of more than two groups.. Normalty and equal varance are requred. : Analyss of Varance. Analyss of more than two groups.. Normalty and equal varance are requred. : Analyss of Varance Excel has n Data Analyss ools. Excel has n Data Analyss Check ools. t usng the Bartlett test. A B C A B C C A B C A B When are ndependent, use (sngle factor ). When correspond each other, use (two-factor ) = three t-tests one hree tmes of t-test wth (p<0.0) equvalent one (p<0.). -(-0.0) = 0.

4 : Analyss of Varance : Analyss of Varance : Analyss of Varance Q: What are "sngle factor" and "two factors"? A: A column factor (e.g. three groups) and a sample factor (e.g. ntalzed condton). When are ndependent, use column (sngle factor factor ). When correspond each other, use column (two-factor factor ). sample factor one-factor () column factor group A group B group C We cannot say that three groups are sgnfcantly dfferent. (p=0.089) sample factor two-factor () column factor ntal group A group B group C condton #...0 #...89 #.6.7. #...9 # # # #8..9. here are sgnfcant dfference somewhere among three groups. (p<0.0) Output of the Source of Varaton SS df MS F P-value F crt Between Groups E-0. Wthn Groups otal.0 9 When (p-value < 0.0 or 0.0), there s(are) sgnfcant dfference somewhere among groups. Output of the Source of Varaton SS df MS F P-value F crt Sample Columns Interacton Wthn otal Sgnfcant dfference among Sample (e.g. ntal condtons) cannot be found (p > 0.0). Sgnfcant dfference can be found somewhere among Columns (e.g. three methods) (p < 0.0). We need not care an nteracton effect between two factors (e.g. ntal condton vs. methods) (p > 0.0). Sample factor Column factor A B C : Analyss of Varance Non-s Mann-Whtney U-test Q: Where s sgnfcant among A, B, and C? A: Apply multple comparsons between all pars among columns. (Fsher's PLSD method, Scheffé method, Bonferron-Dunn test, Dunnett method, Wllams method, ukey method, Nemeny test, ukey-kramer method, Games/Howell method, Duncan's new multple range test, Student-Newman-Keuls method, etc. Each has dfferent characterstcs.) Column factor Source of Varaton SS df MS F P-value F crt Sample Columns Interacton Wthn otal sgnfcant? Sample factor A B C un un un t -test t -test groups n groups (n > ) If normalty and equal varances are not guaranteed, use non-parametrc tests. un un un t -test t -test groups n groups (n > ) Mann-Whtney U-test (Wlcoxon-Mann-Whtney test, two sample Wlcoxon test) Mann-Whtney U-test (Wlcoxon-Mann-Whtney test, two sample Wlcoxon test) Mann-Whtney U-test (cont.) (Wlcoxon-Mann-Whtney test, two sample Wlcoxon test). Comparson of two groups.. Data have no normalty.. here are no correspondng between two groups.?? no normalty. Calculate a U value. 0 U = = 9 U' = (U + U' = n n ) when two values are the same, ( count as 0.. ). See a U-test table. Use the smaller value of U or U'. When n 0 and n 0, see a Mann-Whtney test table. (where n and n are the # of of two groups.) Otherwse, snce U follows the below normal roughly, nn nn ( n n ) N U, U N, U U normalze U as z and check a standard normal table U nn nn ( n n ) wth the z, where U and. U Use an Excel functon to calculate the p-value for the z-value: p-value = - NORM.S.DIS( z )

5 Examples: Mann-Whtney U-test (Wlcoxon-Mann-Whtney test, two sample Wlcoxon test) Ex. 0 Ex Ex. U = 9 U' = (p < 0.0) n n 6 ー 0 U = U' = U =. U' =.. n 6 n (p < 0.0) ー ー ー 0 Exercse: Mann-Whtney U-test (Wlcoxon-Mann-Whtney test, two sample Wlcoxon test) (p < 0.0) U = 9. U' = 6. n n 6 7 ー Snce U' >, (p > 0.0): ( s not found) n 6 7 n (p < 0.0) ー ー ー ー ー ー un un un t -test t -test Sgn est groups n groups (n > ) sgn test Wlcoxon sgned-ranks test ()Sgn est test between the # of wnnngs and losses ()Wlcoxon's Sgned Ranks est test usng both the # of wnnngs and losses and the level of wnnngs/losses of groups Sgn est # of wnnngs and losses the level of wnnngs/losses Sgn est. Calculate the # of wnnngs and losses by comparng runs wth the same ntal.. Check a sgn test table to show of two methods. Sgn est Fg. n Y. Pe and H. akag, "Fourer analyss of the ftness landscape for evolutonary search acceleraton," IEEE Congress on Evolutonary Computaton (CEC), pp.-7, Brsbane, Australa (June 0-, 0). he (+,-) marks show whether our proposed methods converge sgnfcantly or poorer than normal DE, respectvely, (p 0.0). Fg. n the same paper. Generatons F: DE_N vs. DE_LR F: DE_N vs. DE_LS F: DE_N vs. DE_FR_GLB_nD F: DE_N vs. DE_FR_LOC_nD F: DE_N vs. DE_FR_GLB_D F: DE_N vs. DE_FR_LOC_D F: DE_N vs. DE_LR F: DE_N vs. DE_LS F: DE_N vs. DE_FR_GLB_nD F: DE_N vs. DE_FR_LOC_nD F: DE_N vs. DE_FR_GLB_D F: DE_N vs. DE_FR_LOC_D F: DE_N vs. DE_LR F: DE_N vs. DE_LS F: DE_N vs. DE_FR_GLB_nD F: DE_N vs. DE_FR_LOC_nD F: DE_N vs. DE_FR_GLB_D F: DE_N vs. DE_FR_LOC_D + + F: DE_N vs. DE_LR F: DE_N vs. DE_LS F: DE_N vs. DE_FR_GLB_nD F: DE_N vs. DE_FR_LOC_nD F: DE_N vs. DE_FR_GLB_D F: DE_N vs. DE_FR_LOC_D F: DE_N vs. DE_LR F: DE_N vs. DE_LS F: DE_N vs. DE_FR_GLB_nD F: DE_N vs. DE_FR_LOC_nD ++ F: DE_N vs. DE_FR_GLB_D +++ F: DE_N vs. DE_FR_LOC_D + F6: DE_N vs. DE_LR F6: DE_N vs. DE_LS F6: DE_N vs. DE_FR_GLB_nD F6: DE_N vs. DE_FR_LOC_nD F6: DE_N vs. DE_FR_GLB_D F6: DE_N vs. DE_FR_LOC_D F7: DE_N vs. DE_LR + F7: DE_N vs. DE_LS F7: DE_N vs. DE_FR_GLB_nD F7: DE_N vs. DE_FR_LOC_nD + F7: DE_N vs. DE_FR_GLB_D + F7: DE_N vs. DE_FR_LOC_D F8: DE_N vs. DE_LR F8: DE_N vs. DE_LS F8: DE_N vs. DE_FR_GLB_nD F8: DE_N vs. DE_FR_LOC_nD + F8: DE_N vs. DE_FR_GLB_D F8: DE_N vs. DE_FR_LOC_D ask Example Whether performances of pattern recognton methods A and B are sgnfcantly dfferent? level of % % level of Sgn est % % Sgn est Let's thnk about the case of N = 7. level of % % Exercse: Sgn est Check the of: level of % % n cases: Both methods succeeded. n cases: Method A succeeded, and method B faled. n cases: Method A faled, and method B succeeded. n cases: Both methods faled. How to check?. Set N = n + n.. Check the rght table wth the N.. If mn(n, n ) s smaller than the number for the N, we can say that there s sgnfcant dfference wth the sgnfcant rsk level of XX. Exercse Whether there s sgnfcant dfference for n = and n = 8? o say that n and n are sgnfcantly dfferent, (n vs. n ) = (7 vs. 0), (6 vs. ), or ( vs. ) (p < 0.0) or (n vs. n ) = ( vs. ) or ( vs. ) (p < 0.0) 6 vs. vs. 9 vs. 8 vs. ANSWER: Check the rght table wth N = 0. As n s bgger than and smaller than, we can say that there s a sgnfcant dfference between two wth (p < 0.0) but cannot say so wth (p < 0.0).

6 un un Wlcoxon Sgned-Ranks est un t -test t -test groups n groups (n > ) sgn test Wlcoxon sgned-ranks test Wlcoxon Sgned-Ranks est Q: When a sgn test could not show, how to do? A: ry the Wlcoxon sgned-ranks test. It s more senstve than a smple sgn test due to more nformaton use. Wlcoxon Sgned-Ranks est ()Sgn est test between the # of wnnngs and losses ()Wlcoxon's Sgned Ranks est test usng both the # of wnnngs and losses and the level of wnnngs/losses of groups # of wnnngs and losses the level of wnnngs/losses Wlcoxon Sgned-Ranks est Example: (step ) (step ) (step ) (step ) v (system A) v (system B) dfference d rank of d add sgn to the ranks rank of fewer # of sgns n 8 (step ) # of ( Step) (step 6) Wlcoxon test table (step 6) n = 8 = = (n=8, p<0.0), then dfference between systems A and B s sgnfcant. = > 0 (n=8, p<0.0), then we cannot say there s a sgnfcant dfference. When n > As follows the below normal roughly, n( n ) n( n )(n ) N, N, normalze as the below and check a standard normal table wth the z; see and n the above equaton. z Wlcoxon est able: one-tal p < 0.0 p < 0.00 two-tal p < 0.0 p < 0.0 n = Wlcoxon Sgned-Ranks est (step ) (step ) (step ) (step ) v (system A) v (system B) dfference d rank of d add sgn to the ranks rank of fewer # of sgns p # 6. 0 p # p # ps:. When d = 0, gnore the.. When there are the same ranks of d, gve average ranks. Gve the average rank 6. = (+6+7+8)/ Exercse : Wlcoxon Sgned-Ranks est (step ) (step ) (step ) (step ) v (system A) v (system B) dfference d rank of d add sgn to the ranks rank of fewer # of sgns n = (step ) # of ( Step) (step 6) = Wlcoxon test table Exercse : Wlcoxon Sgned-Ranks est (step ) (step ) (step ) (step ) v (system A) v (system B) dfference d rank of d add sgn to the ranks rank of fewer # of sgns n = 8 (step 6) Wlcoxon test table (step ) # of ( Step) = As (=) <, there s a sgnfcant dfference between A and B (p<0.0). But, as 0 < (=), we cannot say so wth the level of (p<0.0). Exercse : Wlcoxon Sgned-Ranks est v (system A) v (system B) dfference d rank of d n = (step ) (step ) (step ) (step ) add sgn to rank of fewer the ranks # of sgns (step 6) Wlcoxon test table (step ) # of ( Step)

7 Exercse : Wlcoxon Sgned-Ranks est (step ) (step ) (step ) (step ) v (system A) v (system B) dfference d rank of d add sgn to the ranks rank of fewer # of sgns (No need to care the case of d = 0.) n = 8 (no count for d = 0.) (step 6) Wlcoxon test table As >, we cannot say that there s a sgnfcant dfference between A and B. (step ) # of ( Step) = Exercse : Wlcoxon Sgned-Ranks est Explan how to apply ths test to test whether two groups are sgnfcantly dfferent at the below generaton? un un Kruskal-Walls est un t -test t -test groups n groups (n > ) Fredman test Kruskal-Walls est Kruskal-Walls est Kruskal-Walls est. Comparson of more than two groups.. Data have no normalty.. here are no correspondng among groups.??? no normalty Let's use ranks of N: total # of k: # of groups n : # of of group R : sum of ranks of group R = 8 R = 69 R = 6 How to est. Rank all.. Calculate N, k, n and R.. Calculate statstcal value H. k R H ( N ) N( N ) n. If k = and N 7, compare the H wth a sgnfcant pont n a Kruskal-Walls test table. Otherwse, assume that H follows the χ and test the H usng a χ table of (k-) degrees of freedom Example: Kruskal-Walls est N = n +n +n = 7 k = groups (n, n, n ) = (6,, 6) (R, R, R ) = (8, 69, 6) H N( N ) 7(7 = k R ( N ) n 8*8 69*69 6*6 ) 6 6 (7 ) Snce sgnfcant ponts of (p<0.0) and (p<0.0) for (n, n, n ) = (6,, 6) are.76 and 8., respectvely, there are sgnfcant dfference(s) somewhere among three groups (p<0.0) (p<0.0) 8. (p<0.0) Kruskal-Walls est able (for k = and N 7) n n n p < 0.0 p < 0.0 n n n p < 0.0 p < Example: Kruskal-Walls est N = n +n +n = 7 k = groups (n, n, n ) = (6,, 6) (R, R, R ) = (8, 69, 6) Q: Where s sgnfcant among A, B, and C? k R H A: Apply multple ( Ncomparsons ) between all pars N( among N ) columns. n (Fsher's PLSD method, Scheffé method, Bonferron-Dunn test, Dunnett method, R 8*8 69*69 6*6 Wllams method, ukey method, Nemeny (7 test, ) ukey-kramer method, 7(7 Games/Howell ) n 6 method, Duncan's 6 new multple range test, Student-Newman-Keuls method, etc. Each has dfferent characterstcs.) = Snce sgnfcant ponts of (p<0.0) and (p<0.0) for (n, n, n ) = (6,, 6) are.76 and 8., respectvely, there are sgnfcant dfference(s) somewhere among three groups (p<0.0) (p<0.0) 8. (p<0.0) Kruskal-Walls est able (for k = and N 7) n n n p < 0.0 p < 0.0 n n n p < 0.0 p < Exercse: Kruskal-Walls est R = R = R = N = n +n +n = samples k = groups (n, n, n ) = (,, ) (R, R, R ) = (,, ) H N( N ) = 6.7 k R ( N ) n here s/are sgnfcant dfference(s) somewhere among three groups (p<0.0) (p<0.0) (p<0.0)

8 Fredman est Fredman est Fredman est un un un t -test t -test groups n groups (n > ) Fredman test When () more than two groups, () have correspondence (not ndependent), but () the condtons of are not satsfed, Let' use ranks of and Fredman test. (ex.) Comparson of recognton rates. benchmark methods tasks a b c d A B C D methods a b c d Step : Make a rankng table. Step : Sum ranks of the factor that you want to test. benchmark tasks method a b c d A B C D Σ 6 7 # of methods (k = ) # of (n = ) Step : Calculate the Fredman test value, χ r. k r R n( k ) nk( k ) where (k, n) are the # of levels of factors and. Step : If k = or, compare χ r wth a sgnfcant pont n a Fredman test table. Otherwse, use a χ table of (k-) degrees of freedom. methods a b c d rankng among methods Example: Fredman est Step : Make a rankng table. Step : Sum ranks of the factor that you want to test. benchmark method tasks a b c d A B C D Σ 6 7 # of methods (k = ) Step : Calculate the Fredman test value, χ r. k r R n( k ) nk( k ) 6 7 ** ** 8. Step : Snce sgnfcant pont for (k,n) = (,) s7.80, there s/are sgnfcant dfference(s) somewhere among four methods, a, b, c, and d (p<0.0) (p<0.0) (p<0.0) # of (n = ) Fredman test table. k n p<0.0 p< Example: Fredman est Step : Make a rankng table. Step : Sum ranks of the factor that you want to test. benchmark method tasks a b c d Q: Where A s sgnfcant among a, b, c, or d? B C A: Apply Dmultple comparsons between all pars among columns. Σ 6 7 (Fsher's PLSD method, Scheffé method, Bonferron-Dunn test, Dunnett Fredman test table. method, Wllams method, # of methods ukey method, (k = ) Nemeny test, ukey-kramer Step : Calculate the Fredman test value, χ k n p<0.0 p<0.0 method, Games/Howell method, Duncan's new r. multple range test, Student-Newman-Keuls k method, etc. Each has dfferent characterstcs.) r R n( k ) nk( k ) ** ** 8. Step : Snce sgnfcant pont for (k,n) = (,) s7.80, there s/are sgnfcant dfference(s) somewhere among four methods, a, b, c, and d (p<0.0) (p<0.0) (p<0.0) # of (n = ) Multple Comparsons When there s sgnfcant dfference among groups, multple comparson s used to know whch group s sgnfcantly dfference from others. Example C = 6 tmes of par comparsons wth (p < 0.0) - ( - 0.0) 6 = level 6.%! Multple Comparsons When there s sgnfcant dfference among groups, multple comparson s used to know whch group s sgnfcantly dfference from others. Soluton s to apply multple par comparsons wth more strct level. Multple Comparsons -- Bobferron method -- When par comparsons are appled m tmes, let's use a level of p / m. Multple Comparsons -- Holm method -- Corrected Bonferron method to detect s easly. Example par comparsons vs. p-value corrected p-value eqn. corrected p-value = p-value* vs = p-value* 0.07 Example C = 6 tmes of par comparsons wth (p < 0.0) 0.0 C = 6 tmes of par comparsons wth (p < ) 6 vs. vs = p-value* = p-value* ( - 0.0) 6 = level 6.%! Features: () Smple. () Rather strct,.e. showng s s rather hard. vs. vs = p-value* = p-value* 0.00

9 normalty (parametrc) no normalty (non-parametrc) sgn test groups n groups (n > ) t -test Wlcoxon Sgned-Ranks est (Analyss of Varance) kruskal-walls test + Scheffé's method of comparson for Human Subjectve ests lghtng desgn of -D CG Corrdor W K L B Verenda Wall room layout plannng desgn MEMS desgn arget System Evolutonary Computaton measurng mental scale room lghtng desgn by optmzng LED assgnments subjectve evaluatons Interactve Evolutonary Computaton IEC mage enhancement processng Can you hear me??? hearng-ad fttng geologcal smulaton based on n C comparsons for n objects. check usng a yardstck Order Effect order effect () and then Orgnal method and three modfed methods yes no () and then may result dfferent evaluaton. All subjects must evaluate all pars. no orgnal ( 原法, 9) Haga's varaton ( 芳賀の変法 ) yes Ura's varaton ( 浦の変法, 96) Nakaya's varaton ( 中屋の変法, 970). Ask N human subjects to evaluate t objects n, or 7 grades.. Assgn [-, +], [-, +] or [-, +] for these grades.. hen, start calculaton (see other materal). Questonnare otal row O O O O O O6 A - A A - A strap for a moble phone A - A 0-0 Ex. Q. Sx subjects (N = 6) Pared comparsons for t= objects. Applcaton Example: What s the best present to be her/hs boy/grl frend? [SIUAION] He/he s my longng. I want to be her/hs boy/grl frend before we graduate from our unversty. o get over my love, I decded to present somethng of about,000 JPY and express my heart. I show you C pars of presents. Please compare each par and mark your relatve evaluaton n fve levels. nvtaton to a dnner tea /coffee stuffed anmal fountan pen I thnk effectve. Realty s... Results of (Nakaya's varaton) What s the best present to be her/hs boy/grl frend? less effectve less effectve (sgnfcant dfference) present from a male I wll catch her heart by dnner more effectve I hestate to accept t as we have not gone about wth hm more effectve less effectve less effectve present from a female How about tea leave or a stuffed anma? Eat! Eat! Eat! more effectve more effectve Orgnal method and three modfed methods All subjects must evaluate all pars. no yes order effect Modfed methods by Ura and Nakaya yes no orgnal ( 原法, 9) Haga's varaton ( 芳賀の変法 ) Ura's varaton ( 浦の変法, 96) Nakaya's varaton ( 中屋の変法, 970) Parwse comparsons for objects whch are effected by dsplay order (order effect)

10 Ask N human subjects to evaluate t C pars for t objects n, or 7 grades and assgn [-, +], [-, +] or [-, +], respectvely Step : Make comparson table of each human subject. A A A A A A A A A A A A 0 0 A A A A A A A A x Step : Make comparson table of each human subject. jl : evaluaton value when the l-th human subject compares the -th object wth the j-th object. Subject O Subject O Subject O Step : Make a table summng all subjects' and calculate the average evaluatons for all objects. ˆ ( x tn A A A A ˆ ˆ Average of four objects ˆ ˆ x where t: # of object () N: # of human subjects () ) S ( x x ) tn S ( x l x l ) S t l S ( xj x j) S N j S x Nt( t ) S S S S t( t ) S S xjl l j Step : Make a table. x S l S S S unbased varance = S/f where S S, S, S, S, S, S, S and f degree of freedom. unbased varance F = unbased varance of S for F tests. S tn S t S N l ( x x ) j table. ( x x ) S ( x x ) S j l j l S x Nt( t ) S xl S t( t ) S S S S S S S S l j x jl table. A A A A ˆ ˆ ˆ ˆ here are sgnfcant dfference among A -A Step : Apply multple comparsons. Q: Where s sgnfcant among A, A, and A? A: Apply multple comparsons between all pars. (Fsher's PLSD method, Scheffé method, Bonferron-Dunn test, Dunnett method, Wllams method, ukey method, Nemeny test, ukey-kramer method, Games/Howell method, Duncan's new multple range test, Student-Newman-Keuls method, etc. Each has dfferent characterstcs.) Step : Apply multple comparsons between all pars and fnd whch dstance s sgnfcant. (Fsher's PLSD method, Scheffé method, Bonferron-Dunn test, Dunnett method, Wllams method, ukey method, Nemeny test, ukey-kramer method, Games/Howell method, Duncan's new multple range test, Student-Newman-Keuls method, etc. Each has dfferent characterstcs.) Example of a smple multple comparson. Calculate a studentzed yardstck When a dfference of average > a studentzed yardstck, the dstance s sgnfcant. A A A A ˆ ˆ ˆ ˆ

11 Step : Example of a smple multple comparsons. Y q t f ˆ (, ) / tn (studentzed yardstck) where ( ˆ, t, N ) are an unbased varance of S ε, the # of objects, and the #of human subjects; q ( t, f ) s a studentzed range obtaned s a statstcal test table for t, the degree of freedom of S ε ( f ), and the sgnfcant level of φ; see these varables n an table. When (t, f) = (,), studentzed yardstcks for levels of % and % are: (See q 0.0 (,) n the next slde.) Studentzed yardstck q ( 0.0 t, f ) f t Step : Example of a smple multple comparsons. Modfed methods by Ura and Nakaya order effect Orgnal method and three modfed methods yes no All subjects must evaluate all pars. no orgnal ( 原法, 9) Haga's varaton ( 芳賀の変法 ) yes Ura's varaton ( 浦の変法, 96) Nakaya's varaton ( 中屋の変法, 970) Modfed method by Nakaya Parwse comparsons for objects that can be compared wthout order effect Modfed method by Nakaya. Ask N human subjects to evaluate t objects n, or 7 grades.. Assgn [-, +], [-, +] or [-, +] for these grades, respectvely.. hen, start calculaton (see other materal). Questonnare Pared comparsons for t= objects. Sx human subjects (N = 6) O O O O O O 6 A - A 0 A - A A - A Modfed method by Nakaya Step : Make comparson table of each human subject. x jl : evaluaton value when the l-th human subject compares the -th object wth the j-th object. Modfed method by Nakaya Step : Make a table summng all subjects' and calculate the average evaluatons for all objects. Average of four objects ˆ tn x where t: # of object () N: # of human subjects (6) Modfed method by Nakaya Step : Make a table. S x.. S x. l S tn t l S S S S S S x. l S t l Unbased varance F S x Unbarased varance of S.. tn here are sgnfcant dfference among A -A table.

12 Modfed method by Nakaya Step : Apply multple comparsons. table. Q: Where s sgnfcant among A, A, and A? A: Apply multple comparsons between all pars among columns. (Fsher's PLSD method, Scheffé method, Bonferron-Dunn test, Dunnett method, Wllams method, ukey method, Nemeny test, ukey-kramer method, Games/Howell method, Duncan's new multple range test, Student-Newman-Keuls method, etc. Each has dfferent characterstcs.) Modfed method by Nakaya Step : Apply multple comparsons between all pars and fnd whch dstance s sgnfcant. (Fsher's PLSD method, Scheffé method, Bonferron-Dunn test, Dunnett method, Wllams method, ukey method, Nemeny test, ukey-kramer method, Games/Howell method, Duncan's new multple range test, Student-Newman-Keuls method, etc. Each has dfferent characterstcs.) Example of a smple multple comparson. Calculate a studentzed yardstck When a dfference of average > a studentzed yardstck, the dstance s sgnfcant. Y Y Modfed method by Nakaya Step : Example of a smple multple comparsons. Y q ( t, f ) ˆ / tn.79 / / (studentzed yardstck) where ( ˆ, t, N ) are an unbased varance of S ε, the # of objects, and the #of human subjects; q ( t, f ) s a studentzed range obtaned s a statstcal test table for t, the degree of freedom of S ε ( f ), and the sgnfcant level of φ; see these varables n an table. (See q 0.0 (,) n the next slde.) Studentzed yardstck q ( 0.0 t, f ) f t SUMMARY. We overvew whch statstcal test we should use for whch case. un un un t -test t -test groups n groups (n > ) Fredman test + Scheffé's method of comparson for Human Subjectve ests. We can appeal the effectveness of our experments wth correct use of statstcal tests.