(Monte Carlo) Resampling Technique in Validity Testing and Reliability Testing

Transcription

1 Iteratoal Joural of Computer Applcatos ( ) (Mote Carlo) Resamplg Techque Valdty Testg ad Relablty Testg Ad Setawa Departmet of Mathematcs, Faculty of Scece ad Mathematcs, Satya Wacaa Chrsta Uversty Jl. Dpoegoro 5-60 Salatga Idoesa ABSTRACT Ths paper descrbes the use of resamplg techque valdty testg ad relablty testg that are wdely used order to make the measuremet tool the feld of psychology ad educato research. Resamplg techque s the techque of resamplg sample wth replacemet or wthout replacemet. Resamplg techque ca be used to determe whether the tem s vald or ot by usg the percetle cofdece terval. The same techque ca also be used to determe the sgfcace of relablty coeffcets order to obta a relable measuremet tool. Ths techque ca also be used to obta a hgher relablty coeffcet by reducg the sample sze or by reducg the umber of tems used the calculato. I ths paper, the techque/method s descrbed the m-data ad case studes usg real data that has 40 tems ad 48 respodets. Keywords Resamplg techque, valdty testg, relablty testg, Pearso coeffcet of correlato, relablty coeffcet. 1. INTRODUCTION I the makg of measuremet tool the feld of psychology ad educato research, t s ofte ecessary to do valdty testg ad relablty testg. To test the valdty of tem usually use crtcal value 0.3 as a vald tem wthout deped o the umber of used respodet [1]. I addto, to test the relablty of measuremet tool, t s ofte used 0.7 as the crtcal value of sgfcace of relablty coeffcet []. The crtcal value s ot deped o the sze of the sample used the research. Resamplg techque has bee dscussed ad used several recet papers (e.g, [3]; [4]; [5]). I ths paper t wll be dscussed o how to use the resamplg techque wth replacemet or wthout replacemet to test the valdty tem ad relablty of measuremet tool.. LITERATURE REVIEW I the lterature revew t s descrbed about how to calculate the Pearso correlato coeffcet that s used as a tool to test the valdty of the tems, relablty coeffcet that s used relablty testg. I addto, t s also explaed about resamplg techque ad examples of how ths techque s used o small data (m data). Suppose ( 1,Y 1 ), (,Y ),., (,Y ) are bvarate radom sample sze that s take from a certa populato. Pearso coeffcet of correlato s defed by, Y E ( )( Y Y ) Y where E[ ], Y E[Y ], V() ad Y V(Y) ad estmato of Pearso coeffcet of correlato based o the sample ca be foud by [6] : where r 1 1 ( ( 1 ) 1 ad )( Y Y) 1 1 Y. Y 1 ( Y Y) Suppose data of 10 respodets ad 5 tems that have a score of 1 through 5 are preseted Table 1. Pearso coeffcets of correlato for each of the tems are 0.95, 0.71, 0.66, 0.93 ad 0.51 respectvely. Pearso coeffcet of correlato s sad to be sgfcat wth level of sgfcace 5% f t s bgger tha 0.63 [7]. Thus, all tems except tem 5 are sad to be sgfcat. I other words tem 1, tem, tem 3 ad tem 4 are vald tems, whle tem 5 s sad to be vald. The vald tem meas that people who have hgh total scores wll ted to gve a hgh score o the tem ad people who have low total scores wll ted to gve a low score. Table 1. Respose of 10 persos o 5 tems wth score 1 through 5. Perso Total Mea Varace Source : [6] page 89. I 1951, Crobach preseted a method to estmate the teral cosstecy wth a formula that became kow as Crobach's Alpha. The alpha relablty coeffcet was calculated by the formula [8] 6

2 Frequecy Frequecy Frequecy Frequecy Frequecy Frequecy Iteratoal Joural of Computer Applcatos ( ) k k 1 1 k 1 r wth k specfes the umber of tems used the calculatos the aalyss, s the varace of the -th tem ad s the total score varace. Relablty coeffcet rages betwee 0 ad 1 [8]. Relablty coeffcet s postve ad sgfcat meas that the measuremet tool s relable otherwse the measuremet tool s ot relable. Based o Table 1, shows that all vald tems except tem 5 ad that the relablty coeffcet s If t s used all the tems the data t wll be obtaed the relablty coeffcet Resamplg Techque Resamplg techque wth replacemet ca be explaed as follows. Suppose a sample wth sample sze 4.e. {1,, 3, 4 }. Based o ths sample, ew sample wth sample sze ca be made based o the sample sze 4 by takg oe by oe wthout replacemet such that the ew samples {, 4, 1, 1} for sample sze = 4; {1,, } for sample sze = 3 ad {4, 1 } for sample sze = are obtaed. If the procedure of the radom samplg for the above expermet s repeated, a dfferet result would be obtaed radomly. Suppose Table 1, the sample of the 10 people { 1,, 3, 4, 5, 6, 7, 8, 9, 10 } are draw wth replacemet such that a ew sample sze 10 wll be obtaed.e. {3, 9, 1,, 8, 5, 3, 3, 4, 7 } ad Table as the result of replcato. Based o Table, the coeffcet of correlato ca be foud as follows 0.98, 0.65, 0.58, 0.96 ad 0.60 ad the relablty coeffcet for the replcato sample s If the procedure s repeated a large umber of tmes B the t s obtaed a matrx whch each colum represets the values of the Pearso coeffcet of correlato for each tem wth the total score. Fgure 1 presets the coeffcet of correlato for each tem ad relablty coeffcet for replcated samples usg the descrbed procedure ad B = 10,000. Pot estmate of the correlato coeffcet usg the mea (or meda) for each of the tems s (0.9560), (0.7157) (0.6771) (0.9363), (0.58) respectvely. It s see that the pot estmate usg the meda teds to be closer to the actual Pearso coeffcet of correlato. Furthermore, pot estmato of the coeffcet of relablty by usg the mea (meda) s (0.8066). Table. The result of replcato based o sample Table 1. Perso Total Mea Varace The 95% percetle cofdece terval for the Pearso coeffcet of correlato coeffcet for each tem respectvely (0.8971, ), (0.3153, ), (0.3634, ), (0.8434, 0.984) ad (0.0699,0.8038) whle the 95% percetle cofdece terval for the relablty coeffcet s (0.6301, ). It s see that the lower lmt for the Pearso correlato coeffcet 5 tems ted to be close to 0 such that tem 5 s almost vald by usg ths method. These results are le wth the results, f we use the Pearso coeffcet of correlato table at a sgfcace level α = 5%. However, usg the resamplg techque, tem 5 s stll sad to be vald. Hstogram of Pearso Correlato Coef. Item 1 Hstogram of Pearso Correlato Coef. Item Hstogram of Pearso Correlato Coef. Item Hstogram of Pearso Correlato Coef. Item 4 Hstogram of Pearso Correlato Coef. Item 5 Hstogram of Relablty Coef Fg 1. Hstogram of Pearso coeffcet of correlato ad relablty coeffcet based o ew sample by usg resamplg techque. 7

3 Iteratoal Joural of Computer Applcatos ( ) Resamplg techque wthout replacemet ca also be used to select tems that provde relablty coeffcets were dfferet from prevous relablty coeffcet. If t s used 4 tems from 5 tems the calculato of the relablty coeffcet the there wll be 5 possble combatos (.e. combato of 4 tems from 5 tems). The order of the tems does ot gve dfferet result of relablty coeffcet. That s oly determed by the umber of tems ad tems whch are used the calculato. For example, f t s used a combato of tems { 1,, 3, 4 }, { 1,, 3, 5}, { 1,, 4, 5 }, { 1, 3, 4, 5 } ad {, 3, 4, 5 } the t s obtaed relablty coeffcet , , , ad , respectvely. I the same way, the values of the coeffcet of relablty for the 3 tems from 5 tems avalable are 10 possble values.e , , , , 0.696, , , , 0.883, Furthermore, the relablty coeffcet ca be determed theoretcally for tems from 5 tems avalable.e , , 0.487, , , , 0.657, , 0.840, There s a egatve relablty coeffcet s whch volates to the assumpto that the coeffcet of relablty should always les betwee 0 ad 1. It s obtaed f there are oly tems that are used the calculato of the coeffcet of relablty,.e. tem ad tem 5 ad that s caused by the Pearso coeffcet of correlato betwee the two tems s egatve (bg scores o tems teds to related to lttle score the tem 5 ad vce versa) such that t results egatve relablty coeffcet. The selecto of tems also rema to be based o the assumpto that the selected tems ca stll measure what s to be measured such that more or less tems wll also affect the relablty coeffcet. Thus, t would be urelable to use measurg devces wth small umber of tems. The umber of tems that s used as measuremet tool depeds o the varable that wll be measured. If the umber of tems s large ad the umber of tems used the calculato of the relablty coeffcet s relatvely small, t s ureasoable to make all possble combatos. If the umber of tems that wll be used the calculato are kow, the smulato method s oe method that ca be used to select tems whch wll provde relablty coeffcets that s ear to maxmum value. I ths case t s used the termology ear to the maxmum because there are possble combatos of tems that cause maxmum relablty coeffcets ad they are ot draw the smulato. Resamplg techque wthout replacemet techque ca also be used the selecto of respodets that wll be used the calculato of statstcs (Pearso coeffcet of correlato or the relablty coeffcet). Suppose the example of Table 1 above, f all tems are used the calculato of the relablty coeffcet but uses less tha 10 respodets. By usg respodets m = 9, 8, 7, 6, 5, 4 ad 3 respectvely, t would be obtaed maxmum relablty coeffcet , 0.881, , 0.914, 0.931, ad That meas, the umber of respodets ca be reduced to obta hgher relablty coeffcet. I the same way, by usg resamplg techques wth replacemet ca also be used the statstcal calculatos by usg more tha 10 respodets.e. m = 0, 30, 50, 100, 500, 1000, respodets order to obta maxmum relablty coeffcet , , 0.880, , 0.840, 0.885, respectvely. It s see that the umber of replcato sample wll ted to gve lower relablty coeffcet. 3. RESEARCH METHODOLOGY The data used s the data obtaed from 48 respodets the measuremet scale cossts of 40 tems [9]. Case 1 Resamplg techque wth replacemet s used for 48 respodets to fd the Pearso coeffcet of correlato by usg all tems. Because t s used the resamplg techque wth replacemet, t s possble that respodets are draw more tha 1 tme a sample by usg = 48 respodets. I ths case, t s lmted to sample sze equal to the umber of respodets the prevous sample. If the procedure s repeated a large umber of tmes B t wll be determed statstcal pot estmates for the Pearso correlato coeffcet by usg the mea or meda values of Pearso correlato coeffcets were formed. Percetle cofdece terval usg cofdece coeffcet ( 1 - α ) 100% ca also be determed based o the values of the Pearso correlato coeffcet. Resamplg techque procedure statstcal calculato of the Pearso coeffcet of correlato ca be descrbed as follows : 1. Suppose 1,,..., m s - varate sample wth sample sze m where m s the umber of respodets ad s the umber of tems the data.. A sample wth sample sze m s draw by usg resamplg techques wth replacemet to obta a ew sample 1 *, *,..., m *. 3. Based o the ew sample, the statstcs s calculated Pearso correlato coeffcets to obta T 1 *, T *,..., T * for each tem that s the Pearso coeffcet of correlato betwee the scores of each tem ad the total umber of scores for each respodet ( ths case there are m respodets). 4. If steps 1 through 3 were repeated a large umber of tmes B t s obtaed the Pearso coeffcet of correlato matrx as follows : T 11 *, T 1 *,..., T 1 *; T 1 *, T *,..., T *;... T 1B *, T B *,..., T B *. 5. Dstrbuto of the Pearso coeffcet of correlato for each tem (.e. tems) ca be determe based o statstcal values each colum of the matrx step 4. Pot estmato of Pearso coeffcet of correlato ca be foud by calculatg the mea or meda of each colum of the matrx. Furthermore, ( 1 - α ) 100 % percetle cofdece terval ca foud based o ordered value of each colum wth order α/ 100 % B (roudg value earest teger) for the lower lmt ad ( 1 - α / ) 100 % B (roudg value earest teger) for the upper lmt. Case Resamplg techque wthout replacemet s used for the 40 tems that are avalable whe s draw k = 39, 38, 35, 30, 5, 0, 10 tems from avalable tems. By usg tems selected for each k tems, the relablty coeffcet was calculated ad whe the above procedure s repeated B = 1000, 10000, ad tmes the relablty coeffcet ca the be determed maxmum or ear- maxmum. Resamplg techque procedure the calculato of the relablty coeffcet ca be explaed as follows : 1. Suppose 1,,..., k s m - varate sample wth sample sze k where k s the umber of tems ad m s the umber of respodets the data. 8

4 Iteratoal Joural of Computer Applcatos ( ). A sample wth sample sze k s draw by usg resamplg techques wth replacemet to obta a ew sample 1 *, *,..., k *. 3. Based o the ew sample statstcs s calculated relablty coeffcets to obta T*. 4. If steps 1 through 3 were repeated a large umber of tmes B we wll obta a relablty coeffcet vector as follows : T 1 *, T *,..., T B *. 5. Dstrbuto of relablty coeffcet s the vector result step 4. Furthermore, (1 - α) 100 % percetle cofdece terval of relablty coeffcet ca foud based o ordered value of each colum wth order α/ 100 % B (roudg value earest teger) for the lower lmt ad (1 - α/) 100 % B (roudg value earest teger) for the upper lmt. Case 3 Resamplg techque wth replacemet s used for 40 avalable tems whe k = 10, 0, 5, 30, 35, 38, 39, 40, 50, 60, 70, 80, 90, 100, 00 tems are draw. By usg selected tems for each k the relablty coeffcet was calculated. Based o the table obtaed Case, t ca be chose whch tems resultg relablty coeffcets more tha ts maxmum value ( ths case we oly use dgts umber after the decmal pot). Resamplg techque procedure used ths case s aalogous to Case but ths case t s used resamplg techque wth replacemet. Case 4 The procedure used ths case s aalogous to Case 1. I case 1 t s used oly 48 respodets, f the respodets used the calculato of relablty coeffcet s m < 48 t s used resamplg techque wthout replacemet, f, however, t s used respodets m > 48 the t s used resamplg techque wth replacemet. The assumpto used ths case that the tems used calculato are vald. 4. RESULT AND DISCUSSION Case 1 Table 3 presets the coeffcet of correlato based o the real data for each tem ad the mea, meda ad lower lmt ad upper lmt of the 95 % percetle cofdece terval. We ca see that the tem s vald f the Pearso coeffcet of correlato s larger tha 0.8 [7]. Thus, tem 14, 17, 18, 7, 30, 31 ad 36 are ot vald. By usg a resamplg techque ca be cocluded that the tem be vald f the 95 % percetle cofdece terval does ot cota the pot 0 such that tems that are ot vald by usg ths method are tem 14, 17, 7, 30, 35 ad 36. Most of the coclusos obtaed are the same as the prevous method except tems 18, 31 ad 35. Items 1 ad 35 due to exact o the border. Items 18 ad 31, however, have a lower lmt that s very close to the pot 0.e Probably, ths s happeed due to the value of B used the procedure s ot qute large. If the vald tems are used the calculato the the relablty coeffcet s (f, however, t s used all the tems the the relablty coeffcet s 0.803). If t s oly used the vald tems the calculato, the 95 % percetle cofdece terval for the relablty coeffcet s the tems are dscarded vald (0.7407, ) whereas f t s used all the tems are (0.7184, ). Table 3. Table of Pearso coeffcet of correlato based o the real data wth mea, meda, upper lmt ad lower lmt of Pearso coeffcet of correlato value by usg resample techque. Item Correlato Mea Meda Lower Lmt Upper Lmt Item Correlato Mea Meda Lower Lmt Upper Lmt Case Based o avalable tems t wll be selected freely k tems that vary from to 39 tems such that the measuremet tool has relablty coeffcet maxmum or ear to maxmum. If k = 39 tems s used the calculato of the relablty coeffcet there wll be 40 combatos of tems. The maxmum value of the relablty coeffcet ca be foud by usg oly a relatvely small replcato, let B =

5 Iteratoal Joural of Computer Applcatos ( ) replcatos. However, f t s chose k = 35 tems the calculato of the relablty coeffcet the there wll be 658,008 combatos of tems such that there s o reaso to use the umber of replcato B = If, however, t s chose the umber of replcato B = 50,000, t wll take a log tme the calculato. For the same reaso, for moderate k betwee ad 39 tems, t wll be better chose B aroud B = 50,000 or 100,000 such that the relablty coeffcet chose close to the actual maxmum value. Case 3 Whe 10 tems s draw, t wll be foud the maxmum relablty coeffcet close to wth the selected tems are {7, 7, 7, 9, 13, 17, 0, 0, 31, 31}. I the same way t ca be take as k = 0, 5, 30, 35, 40, 50, 60, 70, 100, 00, 500, 1000 ad 10,000 such that t s obtaed relablty coeffcets , , , , 0.91, , , , , , respectvely. It s see that the more the sample sze used t wll ted to the greater maxmum relablty coeffcet ca be obtaed. Case 4 I ths case the procedure s aalogous to Case 1. I ths case, however, t s used less tha 48 or more tha 48 respodets. If t s used respodets m = 10, 0, 5, 30, 35, 40, 45, 46, 47 the calculato of the relablty coeffcet ad wthout vald tems ( { 14, 17, 18,, 7, 30, 31, 36 } ) the we ca obta the maxmum relablty coeffcet 0.951, , 0.901, , , ad respectvely. We see that the relablty coeffcet teds to decrease f the umber of used respodet m creases. If t s used m = 50, 100, 500, 1000 ad 10,000 respodets the calculato of relablty coeffcet ad wthout vald tems { 14, 17, 18,, 7, 30, 31, 36 } the t s obtaed maxmum relablty coeffcet , , , ad It s see that the creasg of the umber of replcato tems m teds to decrease the relablty coeffcet. Table 4. Maxmum value or close to maxmum value of relablty coeffcet gve the umber of tems k that s used the calculato of relablty coeffcet ad the umber of replcato B. k The umber of combato B Maxmum or Close to maxmum Relablty coeffcet k The umber of combato B Maxmum or Close to maxmum Relablty coeffcet More tha More tha More tha More tha More tha CONCLUSION I ths paper t s descrbed how to use a resamplg techque wth or wthout replacemet the valdty ad relablty testg. Resamplg techque ca be used to determe whether the tem s vald or ot by usg the percetle cofdece terval that s deped o the sample sze. The same techque ca also be used to determe whether relablty coeffcets s sgfcat or ot. Ths techque ca 6. REFERENCES [1] Sugyoo, 01, Metode Peelta Kuattatf da Kualtatf, R & D. Badug, ALFABETA. [] Sudjoo, A., 007, Pegatar Evaluas Peddka. Jakarta: PT. Kg Grafdo Persada. [3] Berkowtz, D. & Caer, M. & Fag, Y., 01. "The valdty of strumets revsted," Joural of Ecoometrcs, Elsever, vol. 166 (), pages also be used to obta hgher relablty coeffcet by reducg the sample sze or by reducg the umber of tems used the calculato. The research ca be doe also by usg other statstc to test the relablty coeffcet of measuremet toll such as KR-0, KR-1, Spearma-Brow etc. Istead of Pearso coeffcet of correlato, t ca be used Spearma correlato or Kedall correlato valdty testg. [4] Setawa, A., 01, Resamplg Based o Bvarate Kerel Desty Estmato, Proceedgs of the Natoal Semar o Mathematcs Ues October 13, 01. [5] Huo, Mg, Heyvaert, Meke; Va de Noortgate, Wm; Oghea, Patrck, 013, Permutato Tests the Educatoal ad Behavoral Sceces : A Systematc Revew, Methodology : Europea Joural of Research Methods for the Behavoral ad Socal Sceces Vol. 1, No. 1,

6 Iteratoal Joural of Computer Applcatos ( ) [6] Croker, L & J. Alga, 008, Itroducto to Classcal ad Moder Test Theory, Cegage Learg, Maso Oho. [7] Crawshaw, J. & J. Chambers, 001, A Cocse Course Advaced Level Statstcs 4 th Ed., Publsher: Nelso Thores Ltd, Delta Place. [8] De Gujter, D. N., M. & L. J. Th. va der Kamp, 008, Statstcal Test Theory for the Behavoral Sceces, Chapma & Hall/CRC, Boca Rato. [9] Bma, S. A. 014, The formato of New Sample that s Vald ad Relable by Usg Resamplg Techque, Bachelor Thess, Faculty of Scece ad Mathematcs, Satya Wacaa Chrsta Uversty. IJCA TM : 11