Removing Leniency Error in Student Clinical Evaluations

Background: Evaluating clinical performance is an essential component of nuclear medicine technology programs. Previously, our program evaluated clinical performance using a Likert scale of 1-4, four being favorable. Over the past 2 years, the overall student evaluation scores ranged from 3.79 to 4.0 making it difficult to differentiate student performance. Further, scores did not illustrate substantial growth over time, with an average score of 3.90 in semester 1 and 3.95 in semester 2. Leniency error, in which students’ abilities are rated higher than performance metrics(1), removed the ability to draw meaningful conclusions from student scores. Our objective was to create a clinical assessment that would allow for differentiation of clinical performance among students and demonstrate student growth over the program duration.

Methods: Students were evaluated weekly at the end of each clinical rotation. Students selected a clinical instructor to complete the evaluation based on which technologist they worked most with during the clinical rotation. Electronic evaluation forms were completed using an online clinical management system. Students were assessed on five categories of clinical skill including collaboration and teamwork, preparedness and initiative, communication, clinical accuracy, and clinical confidence. A new scale was developed to indicate the student’s clinical skill in the previously mentioned categories and included the following classifications: entry level student (score of 0), advanced entry level student (25), intermediate student (50), advanced intermediate student (75), and entry level CNMT (100). Additional yes (score of 100) or no (0) questions were implemented to reflect whether the student upheld the following mandatory skills throughout the entire clinical rotation: punctuality, integrity, patient and procedural verification, dose verification, patient-focused care, and safety guidelines. Individual student evaluation scores were calculated using a weighted scale. Clinical skill scores were averaged for each week and weighted at 70% of the overall score, while the mandatory skill scores were averaged and weighted at 30%. Overall evaluation scores were available for students to view on the online clinical management system and were reviewed with the student at the completion of each semester (quarters 2 and 4).

Results: Between November 2017 and August 2018, a total of 258 evaluations were completed for 7 students in the Mayo Clinic Nuclear Medicine Technology Program. Evaluations that were completed more than 14 days following rotation completion were excluded from analysis (n=12). The average overall student score for quarter one was 40.52+ 6.19(range 34.04-50.63). Average student score for quarter two was 53.22+11.31 (range 33.25-67.33). Average student score for quarter three was 75.9+ 9.94(range 65.0-87.07). The average score for quarter four was 89.92+7.14 (range 78.13-96.03). The relationship between the average student evaluation scores and corresponding quarter showed a strong linear correlation (R=0.99), demonstrating a student’s progression of skill over time (Figure 1).

Conclusions: Improving both the criteria and scale used for clinical evaluation of nuclear medicine technology students allowed for differentiation of clinical performance among students. The ability to assess student growth highlighted the opportunity to provide coaching as a way to improve student performance and thus improve evaluation scores. As a result, quarterly and as-needed student coaching has been implemented. One limitation to be addressed is that students self-select the clinical instructor to complete the evaluation, and may only select technologists that provide positive feedback. Future work includes the development of a rubric for clinical instructors to more consistently define entry level student, advanced entry level student, intermediate student, advanced intermediate student, and entry level CNMT.