Chapter - 5 Reliability, Validity & Norms

Transcription

1 Chapter - 5 Reliability, Validity & Norms

2 Chapter - 5 Reliability, Validity & Norms Introduction Concept of the Reliability Methods of Estimation of reliability Method of equivalent form Test Retest Method Split-half Method Rulon Formula Flanagan Formula Method of Rational Equivalence (KR20) Reliability by Cronbach s α(alpha) Validity The Concept of validity The Method of Determining validity Face Validity Congruent Validity Concurrent Validity Construct Validity Norms Norms for present test Test Manual Rules for the administration of the test Conversion of raw scores into PR Grade and level fixing Conclusion

3 5.1.0 Introduction : Chapter 5 Reliability, Validity & Norms In the previous chapter, the process of finalization of the test has been described in detail. The test constructor should present the evidences of reliability and validity of the said test. Hence, it is obligatory on the part of the investigator to establish the reliability and validity of the test. This chapter deals with reliability, validity and norms Concept of the Reliability : The main purpose of the measurement is to arrive at some standard and precise judgment about an individual. The judgment would be of good value, if it is based on dependable scores earned on dependable test. The dependable test means nothing but a reliable test. The term reliability denotes truest worthiness, or consistency Ebel(1972) 1 has given an operational definition of a test reliability as follows : The reliability co-efficient for a set of a scores from a group of examines is the co-efficient of co-relation between that set of scares and another set of scores on a equivalent test obtained independently from the members of the same group This definition implies that reliability is not a property of a test by itself but rather of a test, when applied to a particular group of respondents. The more appropriate, the test is to the level of abilities in the group, the higher is the reliability of the scores it yields Methods of Estimation of Reliability : they are : There are different methods of estimating the reliability of the test and 1. Equivalent form 2. Test-retest method. 3. Split-half method 4. Rulon method 90

4 5. Flanagan Method 6. Method of Rational Equivalence. (KR 20 ) 7. Reliability by Cronbach s α (Alpha) Method of equivalent form : This method is simple. From the very beginning, the investigator has to prepare two equal form of the test. These two forms must be very close in similarity i.e. number of items, matter of contents, traits to be measured and procedure for responding the test item etc. Over and above the items must have equal discriminative power, equal facility value, internal consistency and unidimensional ability. The examinee, say subject, takes one form of the test and then the other soon after first. In order to control some error variance, the turns of the forms should be rotated. The agreement between the two forms is determined by means of correlation co-efficient. This method is rarely used because it is very difficult to prepare (Construct) two absolute parallel forms of the same test construction of two equivalent form needs a lots of time and energy on the part of the investigator. Therefore, this method is rarely used for the purpose of determining the reliability of the test. Due to these constrains, the investigator has not used this method Test Retest Method : In this method, the test is given to the same individuals twice. The agreement between the two sets of scores from the two administrations of the same test is determined by means of a correlation of coefficient. There are certain factors like practices, confidence, growth, physical conditions which play certain role in these two different occasions of test administrations. To minimize the effects of these variables, a fairly large sample would be needed. Besides, the time interval should be adequate. The investigator, after studying the pros and cons of this method, used it for estimating the reliability of the present test. The scores obtained on two different administration of the same test were used as two sets of scores for estimating the correlation of coefficient. In this method the time interval for two successive administration should not be too long or too short because it affects on the coefficient of 91

5 correlation. The investigator had selected thirty-day time interval for two successive administration which is supported by many researches. After final run of the present test, a sub sample of 118 students form main sample was drown (See. Chapter 3 para 3.4.5). After 30 days final the test was administrated to the sample as per rules and regulations. After scoring response of the respondents, true scores were entered in a excel spreadsheet which is appended as appendix-viii. The coefficient of correlation was computed by using NRTVB-99 software. The said software was developed by Navnit Rathod in year of (Prof. Bhavnagar University, Bhavnagar, Gujarat State). The coefficient of correlation was found 0.90 Hence, the reliability estimated by this method is satisfactory Split-half Method : There are some practical difficulties which are associated with the determination of reliability coefficient by the test-retest and equivalent forms. These difficulties could be overcome by split-half method. One of the practical alternative is to spilt up the entire form of the test into two reasonably equivalent halves. For making two equivalent halves, usually pooling the odd numbered items for one set of scores and pooling the even numbered items for second set of scores. If controls such factors as practice, fatigue, distraction and mental set. After the test has been given to a representative sample, two scores are obtained for each subject, one of the odd numbered and the other on the even numbered items. The agreement between these set of scores on the same test as determined by a correlation of coefficient, reflects the reliability of half the test. The reliability of the whole that could be obtained by spearman brown s formula 2 which is as under 2 1 The answer-sheet of retest were used for this purpose. Item wise scores were entered in Excel spreadsheet there after score earned by the respondent on odd number item and even number item were split into two halves. The entire table is appended as appendix-ix The NRTVB-99 software 92

6 was used to compute coefficient of correlation. The computed coefficient of correlation was found there after the coefficient of correlation for whole test was computed by abovementioned formula by the said software It is observed from the value of r = This value shows that the present test is reliable Rulon Formula : Rulon has developed a simple formula for establishing the reliability of the total test using the basic definition of reliability. Rulon equation expresses the complimentary statement that reliability is equal to unity minus the proportion of error variance. Hence computing error variance is a main thing in his work. The entire table is appended as appendix-x the rulon formula 3 is given below : 1 During the split-half reliability student wise score earned on odd part and even part were computed from excel spread sheet there after difference between these to scores (d) and (d 2 ) were computed for the same student and were computed. On that basis of and variance for difference ( ) and variance for total ( ) were computed. They were as under , = 0.79 The coefficient of correlation was found The value shows that correlation between two halves is high and reliable. 93

7 5.3.5 Flanagan Formula : Flanagan s formula for establishing reliability is very close to Rulon s formula. In Flanagan s formula. The variance of two halves are added, instead of difference of division of two sets. In other words it estimates the error variance in a sense as the sum of the variance of the two halves. The Flanagan formula 4 is as under. 21 = Variance of s of odd numbered items. = Variance of of even numbered items. = Variance of total scores During the computation of split-half reliability value for each student for odd number item, even number item, d, d 2, X t and were received. With the help of these data,,, were computed by using excel (spread sheet) program. The entire table is appended as appendix-xi They ware as under 46.17, 41.20, These values were used to compute coefficient of correlation by using Flanagan formula. The computation is as under It is observed that the computed value of r was found This results also indicates that the test is reliable Method of Rational Equivalence (KR 20 ) : This method was developed by Kuder and Richardson, it is also known as K.R. method. This method is useful for estimating the internal consistency or homogeneity of a test. The Kuder Richardson formulas, was developed because of dissatisfaction with Split-half method. A test can be split 94

8 into two equal halves in great many way and such split might yield some what different estimate of rtt. The use of items statistics set away from such basis as may arise from arbitrary splitting into two halves. Finally the most accurate and practical KR 20 formula 5 was developed as under. 1 Σ. n p q = number of items in the test = Proportion of passing on items = (1-P) proportion of the group answering the item incorrectly = The variance of the entire test score. = reliability of the whole test To compute reliability by using KR 20. The responses of 118 student to whom the retest was administrated were used. The present test carries = 360 item. For each item total score, number of correct items and number of incorrect items were worked out form the excel spread sheet. The proportion of correct responses for each item and its proportion (p) and proportion of incorrect responses (q = 100-p) were also computed for each item. With the help of p and q, pxq for each item was also computed. The sum of p x q i.e. Σpq and were also computed. The entire table is appended as appendix-xii. The required statistic i.e. n, Σpq and are as under. 360, 20.41, Σ These value were inserted in the formula and tt was computed. And computation is as under = x 0.97 = 0.97 The computed value of was found 0.97 and this value shows that resent test is highly reliable Reliability by Cronbach s (alpha) : Cronbach s α is the coefficient of reliability. It is commonly used as measure of the internal consistency or reliability of a test. It was 95

9 denoted as alpha by lee cronbach in According to him, he was intended to continue with further coefficients. The measure can be viewed as an extension of the Kunder-Rechardson formula (KR 20 ). Which is an equivalent measure of dichotomous items. Cronbach has given a formula to work out the reliability by using variance of the scores not only on odd and even items but also the total number of items. The software NRTVB-99 is based on the following formula. Cronbach s α = 2[1 ( ) ] The formula transformation is built in the NRTVB-99 software. So that when we run the software, we get value of cronbach s alpha directly. Hence, after running these software the investigator got the value of Cronbach s alpha, which was A commonly accepted rule of thumb for describing internal consistency using cronbach s alpha is as follows : Cronbach s alpha α α 0.9 Good 0.7 α 0.8 Acceptable 0.6 α 0.7 Questionable Internal consistency Excellent 0.5 α 0.6 Poor α 0.5 Unacceptable The cronbach's & value for present test was As per norms laid down by cronbach, the internal consistency say the reliability of the present test is excellent. Summary of the Reliablility :- The investigator has used six methods of estimating reliability of the present test. It is summarised in table Summary of Reliability Sr No. Method Correlation Remark 1 Test-retest 0.90 Highly Reliable 2 Splif-half 0.78 Reliable 3 Rulon formula 0.79 Reliable 4 Flanagun Method 0.76 Reliable 5 Rational Equivalence (KR 20 ) Method 0.97 Highly Reliable 6 Cronbach's & (alpha) Method 0.93 Highly Reliable 96

10 It is observed from the table that the range of coefficient of correlation worked out by five different methods of the present test is 0.76 to Looking to different values one can dare to say that the speed and Accuracy test is reliable. As per cronbach s α the reliability of the said test is trustable, reliable and no harm to used for the population Validity : A test is expected to Prove its worth. If the test does not fullfill its worth, it is not worth of anything. Ross and Stanley (1963) 6 have rightly Pointed out : Although high reliability is no guarantee that the test is good, low reliability does not indicate that it is poor. Validity is always the first to be sought in a test, and granted that reliability is a valuable auxiliary. The test for measuring mental abilities, aptitude, attitude, etc., must justify their Purpose. In the process of justification of purpose is known as the test validation. Consequently, validation of a test score is the most important and significant step in the process of standardization of any test. Most of the users, before selecting the test for the use, looks carefully in to the values of validity. Therefore, the constructor of the test should make clear the concept of the validity The Concept of validity : The validity is an important characteristic of the test. It depends upon the efficiency with which it measures that it attempts to measure. In other words it is defined as the accuracy with which the test measures what it claims to measure. The term validity and purpose are very closely associated with each other. A test which fulfills the purpose for which it is designed is called a valid test. This led to say that test of a Speed and Accuracy (SA) measures the Speed and Accuracy and nothing else. Therefore, in the course of valid Speed & Accuracy test the student reading for their grade XII in the various higher secondary schools who have speed and accuracy should get more scores than those who have less S.A. Garrett (1965) 7 has rightly put it as : The validity of the test or of any measuring instrument, depends upon the fidelity with which it measures what it proposes to measure 97

11 In such situation the validity of the test must be established even the test is highly reliable The Method of Determining validity : Procedures for determining validity of tests are primarily concerned with relationship between performance on the test and other independently observable facts about the speed and accuracy characteristics under consideration. There are numerous techniques that are employed for investigating these relationships. They are described as various techniques. They are as under : 1. Face Validity 2. Congruent Validity 3. Concurrent Validity 4. Construct Validity The investigator has used all the four techniques for estimation validity of the Speed & Accuracy test Face Validity : This method is very simple and provides valid image of the test which can be treated as validity. In this method, experts having long experience to judge the test for which it has been constructed. The test was given to experts and requested them to judge whether it has a capacity to measure the Speed and Accuracy of the students of higher secondary school. The investigator have selected 5 experts. Who have large experience in test construction. For the validation of item content two Teachers were also involved for judging item content. Over and above both the expert groups were requested to scale out level of difficulty of each item by assigning 1, 2, and 3 for too easy, difficult and too difficult respectively. The judges were also requested to make language correction if needed and if he / she fills to add more items in the given list of items. The reactions of experts were analysed and found that, there were no much more suggestions in editing items. All the experts were opined that items have good power to measure speed and Accuracy of the students under investigation. Hence, it was concluded that the first draft of test carries good face validity. 98

12 5.4.4 Congruent Validity : This type of validity is estimated by the means of a statistical technique. For this, the set of scores on the present test is correlated with the set of criteria of a similar measure. It means, it is correlated with some available well known, fully valid and powerful test of the similar nature. The correlation of the new test with the existing test would show to what extent the two tests measure the same speed & Accuracy. For this purpose, the present investigator has selected a speed and accuracy test constructed and standardized by K. G. Desai. The said test was based on Minesota claricde Apptitude test. The said test was constructed and presently widely used by many researchers. The reliability range of the said test was between 0.85 to Its validation was made by administering the test to criterian groups like Accountants, Clarks and persons busy with such work. More detail is given in chpt. 3 Para After selection of valid criterion test, both tests were administered to a representative randomized sample of 118 students and two sets of scores were obtained for each student please see appendix VII. The agreement between these scores were determined by a correlation of coefficient. Both the set of scores were entered in a excel spread sheet. The correlation of co-efficient was computed using the NRTVB 99 soft-ware. The value of coefficient of correlation was Hence, the present test can be treated as a valid test Concurrent Validity : The evidence of the validity be obtained from the relationship with other currently obtainable information about an individual. Anne Anastasi (1976) 8 has defined the concurrent validity as follows : The relationship between test scores and indices of criterion status obtained at approximately the same time is known as concurrent validity. Asking students to provide percentage of last annual examination of the students included in the sample of re-test. For the students 99

13 of Std. XI, their board examination result of std X were taken into consideration while for the students of XII, the annual examination of their std XI were used. The stated results were verified by the investigator from school record for its validation. Two set of scores earned by each student on speed and accuracy test and his /her percentage of annual examination (Educational achievement) were entered in Excel - spread sheet please see appendix - VIII. The coefficient of correlation between these two set of scores was computed by using Excel program. The said values were The value of coefficient of correlation indicated that the concurrent validity of the present test is satisfactory one Construct Validity : There are different methods for examining construct validity. The factor analysis is a well known method of construct validity. In the present investigation, the method of cliff s consistency Indices C was used due to its strongness. In the present study cliff s consistency Indices C was calculated using the computer program NRTVB 99. During computation of reliability by KR 20 (See para : 5.3.6), item wise correct and incorrect responses were entered in a Excel Spread Sheet. The spread sheet prepared by data entry was directly supporting to NRTVB-99. So that these soft-ware was run to get the value of cliff s consistency C. The computed cliff s consistency C was 0.36 As per criteria, if the value of C is 0.36 or above it, then the test is treated as valid one. The computed value for different methods employed by the investigator is present in table

14 Table Validity Summary Method Values Remark 1) Face Validity - Good 2) Congruent Validity 0.60 Valid 3) Concurrent Validity 0.50 Satisfactory 4) Construct Validity 0.36 Valid It is observed form table that the present test is found valid and useful to measure Speed and Accuracy of the higher secondary school students Norms : A norm represent a typical level of performance for a particular group. A raw score on any Psychological test alone is meaningless unless we have additional interpretive data. So the scores on psychological test are most commonly interpreted by reference to norms that represent the test performance of the standardized sample. Norms are empirically established by determining what parsons in a representative group actually do on a test. In order to ascertain more precisely the individual s exact position with reference to the standardized sample, the raw score is converted into some relative measure. These derived scores serve two purposes. 1. They indicate the individuals relative standing in the normative sample and facilitate evaluation of performance. 2. They provide comparable measures that permit a direct comparison of the individuals performance on different tests. Fundamentally, derived scores are expressed in one of two major ways (1) developmental Norms and (2) within group Norms. Developmental Norms :- These type of norms generally indicate the normal developmental path the individual has progressed. They are very helpful for descriptive purpose but they are not compatible to precise statistical treatment. The types of developmental norms are Mental age Norms Grade Equivalent Norms. Ordinal Scale Norms. 101

15 Within Group Norms :- Such type of norms help in comparing the individual s performance with the most nearly comparable standardised group s performance. Within group norms have a uniform and clearly defined quantitative meaning and can be appropriately employed in most types of statistical analyses. Percentiles :- Percentile scores represent the percentage of persons in the standardised sample who fall below a given raw score. They indicate an individual s relative position in the standardized sample. In case of percentiles, the counting begins from the bottom so lower the percentile, poorer the standing / rank. Standard :- Standard score express the individual s distance from the mean in terms of the standard deviation of the distribution. They are obtained by linear or nonlinear transformation of the original raw scores. Age Norms :- To establish age norms, the mean of raw scores obtained by all in the same age group within a standardized sample is taken. So mean raw score of 12 year old students would represent the 12 years norm. Grade Norms :- Grade norms are found by computing the mean row score obtained by students in particular grade Norms for the present test :- To gage the performance of the testee, norms provide such facility to its users. In the present study, Area, Gender, Stream of the study are naturally divisible. The educational achievement levels and Occupation of the parent father are not naturally divisible. As per chapter six levels of each variable except standard has significant effect on Speed and Accuracy (SA) of the students included in the sample. According to theory, one has to computed norms for each variable level wise. In such situation, one has to incorporate all the rows for each raw-score. If we include all the variables than there will be 144 rows (2 x 2 x 3 x 4 x 3) which may create complication for reading 102

16 equivalence for raw-score. To minimize the rows, it was decided to establish norms for naturally divisible variables. i.e. Area, Gender and streams only. To avoid confusions, again only norms were computed in form of PR. PR will help to identify the position of the testee in the entire population. It will also help the researchers, teachers and counsellors to decide the level of speed & Accuracy in terms of letter grade and qualitative division of the testees. e.g. Very Good (A), Good (B), Average (C) Below Average (D) and Poor (E). For dividing into levels or letter grade following criteria were used. Table Letter grade and qualitative division Level Grade Coverage of PR Very Good A Above 80 PR Good B From 61 to 80 PR Average C From 41 to 60 PR Below Average D From 21 to 40 PR Poor E Below 21 Total true score earned by each student was classified according to stratum. Area wise, Gender wise and Stratum wise frequency distributions were prepared accordingly by using excel program. PR was calculated for each raw score in the context of each variable with its levels. The minimum score was 192 earn by the students, on present test. Hence, from 191 raw score, PR table was prepared : To make easy reading PR corresponding to raw score stream wise, i.e. three tables were prepared. These PRs are presented in table say table - (A), table - (B) and table - (C) for stream wise. They are as follow. 103

17 Table (A) Value of PR for each row score in the context of variable & its levels Science Stream Rural Rural

18 Rural Rural

19 Rural Rural Table (B) Value of PR for each row score in the context of variable & its levels Commerce Stream Rural Rural

20 Rural Rural

21 Rural Rural Table (C) Value of PR for each row score in the context of variable & its levels Arts Stream Arts Arts

22 Arts Arts

23 Arts Arts

24 Arts Arts Test Manual :- Test manual provides guide line about how to use the present test scientifically. The present test is a speed test so that proper administrative rules are needed. The manual also provides guide line to evaluate testee's score in terms of his / her PR and also how to categories PR into different levels say assigning letter grade to the Speed and Accuracy of the testee. The manual also provides important hints to counsellor for student's counselling Rules for the administration of the test : Population : This test would be used for the students studying in the higher secondary schools having all the three streams : Science, Commerce and Arts situated in the Gujarat state such as Ahmedabad, Gandhinagar, Mehsana, Surendranagar and Vadodara districts having Gujarati Medium. Administration of the test : o Proper sitting arrangement should be made. For that distance between to testees must be 3 feet. o For Section I and Section II, the time limit must be eight-eight minutes, so everyone has to finish the work in that time. To maintain the said time one has to use stop-watch for that. o Be sure that every student must fill up necessary primary information before starting the test. o Give detailed explanation with example how student has to register their response in the test-booklet. Check personally for illustrative items. o Clarify if the pair is not proper then one has not to put any sign in the space given between them. 111

25 o Calculate true scores by subtracting numbers of incorrect responses from correct numbers of responses for each section and there after for the whole test Conversion of raw scores into PR : Identify the Stream, Area and Gender of the respondent. if o Science stream than use Norm table No (A). o Commerce stream than table no (B). o Arts stream than table no (C) Decide the Area and Gender of the student and select appropriate column. Traced-out the position of True-score say row scores of the testee and read PR from the norm-table for corresponding true score. Obtained PR suggests the position of the testee in that particular variable group. e.g. If the student having PR 70, he/she is superior than 69% of the students in that particular group Letter grade and level fixing : To decide letter grade or level of Speed and Accuracy follow the is to be used. Sr. No. Range of PR Table No Letter grade and Level of SA Letter Grade Level of Speed and Accuracy 1 80 & above A Excellent B Good C Average D Poor 5 below 40 E Very Poor Conclusion : The present chapter dealt with the Reliability, Validity and Norms. The reliability was measured by using Test-retest, split-half and Cronebatch α which was running between 0.85 to A Congruent validity was estimated and was high one. A guide line for users along with norms were discussed at length in this chapter. The next chapter is based on data analysis in the light of objectives stated for phase II. 112

26 References 1. Robert L. Ebel, (1972); Measuring Educational Achievement, New Delhi : Prentice Hall. P Kothari R. C. (2009), Research Methodology, Method and Techniques, (New Delhi : New Age International (P) ltd.), pp Rulon, P. J. A. (1939), Simplified procedure for determining the reliability of a test by split-halves theory, Edu. Pr. 9, pp Garrelt, J. H. E. (1985), Statistics in Psychology and Education; Vakils, (Bombay : Feffer and Simons Pvt. Ltd.), p Freeman, S.F.(1968), Theory and Practice of Psychological Testing, (New York : Holt, Rinehart and Winston. Inc. No.4), p Ross and Stanley, (1963), Measurement in today's school, New York, Prentice Hall, P H.E. Garrtl, (1965); 'Statistical in Psychology and Education, Bombay; Vakils, Feffer and Simoson Pvt. Ltd. P Anne Anastasi (1976); Psychological Testing, Fourth edition, London, The McMillan Co. Colliar MacMillan Ltd. P