CRDTS EXAMINER PROFILE SERVICE. Examiner Profile Statistics 2017

Transcription

1 CRDTS EXAMINER PROFILE SERVICE Examiner Profile Statistics 2017 CLINICAL PERIODONTAL EXAMINATION Prepared by: Lynn Ray, Director of Analysis Kimber Cobb, Director of Dental Examination Development and Data Management Central Regional Dental Testing Service, Inc.

2 EXPLANATION OF PERIODONTAL PROFILES The CRDTS Examination Review Committee maintains an "Examiner Profile Service" specifically designed to provide each CRDTS examiner with information which the Committee hopes individual examiners will use to self-assess and, where necessary, improve their individual examining skills. The profiles are prepared by CRDTS Director of Analysis and reviewed by the Examiner Development Committee to assist them with examiner assignments. No identities are attached to the examiner code numbers until the review is complete. The Examination Review Committee assumes that this individualized self-assessment approach will be the most preferable feedback mechanism for our examiners. The usefulness of the service ultimately depends upon the interest shown by each individual examiner in reviewing his/her profile and assessing their personal contribution to the CRDTS testing program. DESCRIPTION OF EXAMINER INDICES This preface describes the various indices which have been used to assemble each examiner's individual profile. No one index is indicative of an examiner's approach to their testing responsibilities and each individual is encouraged to look for patterns and relationships among the indices which may reveal certain trends in their approach to the examination. It should be noted that all data in the profiles is analyzed in the context in which the examiner worked. In all instances, examiners are compared to their examining partners who were evaluating the same cases. Overall, there could be significant variations in examiner profiles based on examination assignments and possible differences in groups of candidates that are being examined. When reviewing your individual profiles, please note that the indices used often include these numbers: 1. Your confidential examiner profile code. As of 2017, these code numbers begin at 600 and are randomly changed by the computer every year. Examiners who evaluated a variety of skills, such as manikin, restorative, perio and hygiene will have the same profile code number in each separate profile. 2. a number or percent which reflects your performance as an individual examiner or the combined performance of you and your examining partners; 3. a number or percentage that represents the average for all individual examiners. In reviewing your data, a comparison should be made to determine where you stand in relation to the average of your colleagues and also to how much "higher" or "lower" than the average you actually stand. This can be assessed most readily in the bar graphs. Individuals at or near the extreme limits of the bar graphs should perhaps assess the reasons that could account for their ratings and decide whether or not corrective action would be appropriate for the next testing cycle. Table 1A & B: These tables report the total number of evaluations the examiners completed (Table 1A) and their average time per candidate (Table 1B). The average number of evaluations per test site is not a definitive measure for comparing examiners productivity; the number of evaluations is heavily dependent on the size of the exam and whether the test site was a one or two day examination. The more evaluations that were completed, the more reliable the examiner s profile can be considered as a measure of their trends. Table 1B reports the average time that the examiner spent in each evaluation that was completed. The times are reported for the Treatment Selection Evaluation, the Final Evaluation

3 and the total average time per candidate. It should be recognized that these times are per individual examiner, and not a computation of the total time required for three examiners to complete their evaluations. Table 2A, B, C, D: These tables report the percentage of agreement between an examiner and his/her partners for the final evaluation of the periodontal procedures in which the examiner marks Yes or No Oral Assessment, Scaling or Calculus Removal, Periodontal Probing and Removal of Supragingival Deposits. For all these procedures, the examiner marks Yes in every instance where they can confirm the candidate fulfilled the criteria and No for every instance that they identify a deficiency in the candidate s performance. Tables 2A, 2B, 2C and 2D report the same information for different procedures so the heading of each table should be noted to identify which procedure is being analyzed. Whether the examiner marked Yes or No is not a factor in these tables only whether the examiners independent judgments are in agreement. There are two levels of agreement that can be measured: 1. All three independent examiners assigned an identical score. 2. Two of the three independent examiners assigned an identical score; the third examiner is an outlier and assigned the opposite score from his/her two partners. These two levels of agreement are all reported in Tables 2A (Oral Assessment), 2B (Scaling), 2C (Periodontal Probing and Gingival Recession) and 2D (Removal of Supragingival Deposits). Some procedures are combined because there is a much lower incidence of disagreement in those sections. The percentage of the examiners three identical and two identical scores are reported and those two factors are combined to show the percentage of confirmed scores the examiner assigned. The percentage of times that one of the examiners disagreed with the confirmed score is also displayed. These would be the occasions that the examiner marked No when the other two examiners marked Yes, or marked Yes when the other two examiners marked No. In these instances, the examiner is considered the odd-manout or outlier, so the percentage of times that each examiner was an outlier is also reported on Tables 2A, B, C, D. Bar Charts: Immediately following these tables 2A, B, C, and D is a bar chart that graphically displays the degree of the disagreements between the examiners; these charts take the instances in which the examiner was an outlier and shows the degree and direction of their disagreements. The bars to the left of the vertical axis are the percentage of times the examiner assigned a rating that was higher (YNN, easier, more lenient) than the confirmed score. The bars to the right of the vertical axis indicate the percentage of times the examiner assigned a rating that was lower (NYY, harder, more critical) than the confirmed score. The heavy black broken lines on either side of the vertical axis are the means or averages for all examiners. Another measure of examiner disagreement is the standard deviation of each of their scores from the mean. The standard deviation has been calculated and is shown on each bar graph. A green dotted line has been added to the bar graph one standard deviation beyond the mean. Those examiners whose bar graph lies between or very close to the heavy black, broken lines representing the means are considered to be within acceptable ranges of agreement. Those examiners whose bar graph extends beyond those dotted lines that represent one standard deviation beyond the mean, are considered by the EEAC to be at an extreme. These examiners

4 should review the criteria for that procedure very carefully, participate attentively during calibration and reconsider the way in which they are applying the criteria. Table 3: This table and its accompanying chart presents the same information in the same format as Tables 2A, B, C and D; only it combines all the data from Tables 2 and reports each examiner s data for the entire examination. Immediately following this table is a bar chart that graphically displays the degree and the direction of the disagreements between the examiners; these charts show the instances in which the examiner was an outlier for the overall exam. The bars to the left of the vertical axis are the percentage of times the examiner assigned a Yes (easier, more lenient) when the confirmed score from the other two examiners was No. The bars to the right of the vertical axis indicate the percentage of times the examiner assigned a No (harder, more critical) when the confirmed score from the other two examiners was a Yes. Table 4: At the end of every examination, examiners are asked to evaluate their fellow examiners with whom they worked. In 2017, examiners assigned an overall rating for each examiner on a scale of 1 to 4, with 4 being the highest rating. The examiners could also support their ratings with comments. Table 4 displays the average rating for each examiner. This Table is distributed separately.

5 List of Tables Table 1A - Summary of Examiner Assignments # Test Sites and # Candidates Evaluated Table 1B - Average Examiner Evaluation Time Per Candidate Table 2 Examiners Agreements and DISagreements with their Partners by Procedure Table & Chart 2A Oral Assessment Table & Chart 2B Scaling/Calculus Removal Table & Chart 2C Periodontal Probing and Gingival Recession Table & Chart 2D Supragingival Deposits Table 3 Examiners Agreements and DISagreements with their Partners Overall Examination Table 4 Peer Evaluations (Distributed Separately)

6 Table 1A EXAMINER ASSIGNMENTS Examiner # Test # Treatment Total # Final ID # Sites Selections Evaluations AVG:

7 Table 1A (Continued) EXAMINER ASSIGNMENTS Examiner # Test # Treatment Total # Final ID # Sites Selections Evaluations AVG:

8 Table 1B AVERAGE EXAMINER EVALUATION TIMES PER CANDIDATE Examiner # Test Treatment Final Total Average ID # Sites Selections Evaluations Time/Candidate :03 5:03 15: :33 3:51 8: :35 6:47 10: :48 5:35 13: :28 3:22 8: :37 6:54 11: :50 4:30 11: :42 4:42 9: :41 4:32 8: :49 6:42 11: :20 5:23 9: :46 4:35 7: :47 6:47 11: :54 7:05 13: :53 1:14 3: :36 4:46 8: :05 7:58 16: :19 5:51 8: :52 5:14 10: :23 4:06 8: :26 5:16 9: :42 2:54 8: :00 8:12 19: :01 3:25 6: :49 3:41 7: :06 4:03 8: :02 6:46 9: :04 4:09 8: :17 5:00 10: :02 7:55 13: :01 5:16 9: :28 7:13 14: :45 5:00 8: :01 6:16 11: :58 5:03 10: :29 6:27 10: :30 8:13 13: :24 6:28 17: :11 1:00 3: :00 1:16 3: :41 10:12 21: :18 6:19 12: :21 4:20 8: :15 10:29 20: :48 8:05 11: :18 5:52 11:10 AVG: :25 5:42 11:04

9 Table 1B (Continued) AVERAGE EXAMINER EVALUATION TIMES PER CANDIDATE Examiner # Test Treatment Final Total Average ID # Sites Selections Evaluations Time/Candidate :03 6:57 16: :52 4:42 8: :52 10:07 17: :22 6:03 9: :05 13:04 22: :55 3:36 6: :44 4:22 8: :38 4:24 8: :38 6:38 13: :40 8:23 17: :43 3:45 11: :11 3:21 7: :18 3:38 7: :21 4:10 7: :43 5:34 13: :55 2:28 6: :39 7:42 13: :20 5:30 11: :43 6:43 11: :57 7:36 14: :28 6:43 12: :00 12:15 12: :43 7:11 11: :42 5:12 10: :13 3:19 6: :26 7:04 13: :03 6:50 12: :56 3:56 11: :39 6:11 11: :33 7:19 14: :34 6:07 9: :17 6:52 11: :24 4:09 8: :28 7:48 12: :59 8:13 15: :06 7:22 14: :53 1:20 5: :51 2:53 8: :27 4:13 9: :50 1:31 4: :39 6:31 12: :12 5:21 12: :11 4:47 7: :33 5:45 11: :49 6:02 11: :33 6:07 12: :29 7:16 13:45 AVG: :25 5:42 11:04

10 Table 2A ORAL ASSESSMENT Individual Examiner Agreements with Partners Results (%) Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

11 Table 2A (Continued) ORAL ASSESSMENT Individual Examiner Agreements with Partners Results (%) Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

12 CHART 2A DEGREE OF NON-MATCHING SCORES 2017 PERIO ORAL ASSESSMENT Axis Title EASIER THAN PARTNERS (YNN) HARDER THAN PARTNERS (NYY) MEAN = 0.42% MEAN = 3.17% SD = 0.94 SD =10.38

13 Table 2B Individual Examiner Agreements with Partners Results (%) SCALING Calculus Removal Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

14 Table 2B (Continued) Individual Examiner Agreements with Partners Results (%) SCALING Calculus Removal Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

15 CHART 2B DEGREE AND DIRECTION OF NON-MATCHING SCORES 2017 PERIO SCALING-CALCULUS REMOVAL Axis Title EASIER THAN PARTNERS (YNN) HARDER THAN PARTNERS (NYY) MEAN = 1.46% MEAN = 6.33% SD = 1.50 SD = 11.50%

16 Table 2C PERIODONTAL PROBING AND GINGIVAL RECESSION Individual Examiner Agreements with Partners Results (%) Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

17 Table 2C (Continued) PERIODONTAL PROBING AND GINGIVAL RECESSION Individual Examiner Agreements with Partners Results (%) Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

18 CHART 2C DEGREE AND DIRECTION OF NON-MATCHING SCORES 2017 PERIO PROBING AND GINGIVAL RECESSION EASIER THAN PARTNERS (YNN) HARDER THAN PARTNERS (NYY) MEAN = 0.39% MEAN = 2.27% SD = 0.74 SD = 4.96

19 Table 2D Individual Examiner Agreements with Partners Results (%) SUPRAGINGIVAL DEPOSITS Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

20 Table 2D (Continued) Individual Examiner Agreements with Partners Results (%) SUPRAGINGIVAL DEPOSITS Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

21 CHART 2D DEGREE AND DIRECTION OF NON-MATCHING SCORES 2017 PERIO SUPRAGINGIVAL DEPOSIT EASIER THAN PARTNERS (YNN) HARDER THAN PARTNERS (NYY) MEAN = 0.08% MEAN = 1.16% SD = 0.24 SD = 1.65

22 Table 3 Individual Examiner Agreements with Partners Results (%) OVERALL PERIODONTAL EXAMINATION Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

23 Table 3 Individual Examiner Agreements with Partners Results (%) OVERALL PERIODONTAL EXAMINATION Examiner Total # % 3 Identical % 2 Identical % Confirmed % Outlier ID # Scores Scores Scores Scores (YNN or NYY) AVG:

24 CHART 3 DEGREE AND DIRECTION OF NON-MATCHING SCORES 2017 OVERALL PERIODONTAL EXAMINATION EASIER THAN PARTNERS (YNN) HARDER THAN PARTNERS (NYY) MEAN = 0.63% MEAN = 3.27% SD = 0.73 SD = 2.23