An experimental platform to evaluate self-assessment skills for 3^rd year dental students

INTRODUCTION

At the level of undergraduate dental education, students “do not know what they do not know,” because of their inability to self-assess. This prevents them from avoiding unwarranted errors in their performance.^[1,2] Self-assessment was defined by Asadoorian and Batty as an active process, which comprises learning requirements consciousness, and highlighted its importance during the learning phase for obtaining professional skills.^[3]

The importance of self-assessment skill has been underlined by dental schools and educational regulatory bodies, including the Commission on Dental Accreditation (CODA), which stated that dental graduates should be capable of assessing their work.^[4,5] Consequently, the students should be armed with the appropriate training tools enabling them to critically recognize and distinguish ideal from error.

The aim of this approach is to enhance their ability to self-assess^[6] and thus actively participate in their learning.^[1] Moreover, Burrows presumed that faculty staff assessments are often one-sided and could be frequently subjective, inconsistent, and misunderstood, and most of the time do not involve students in critical thinking and reflection.^[1]

The Association for Dental Education in Europe advocates that cognitive and psychomotor capabilities should constitute a part of the curriculum, besides the learning and evaluation methods.^[7]

Needless to say, continuous development of innovative methods and improvements to the overall learning process are required, to attain a positive impact on the learning process as a whole.^[8]

Self-assessment should therefore, be used as a pedagogic tool throughout undergraduate education and across all learning domains. It should help students to develop into reflective practitioners and not just be used as an assessment tool for grading students.^[1,9]

On the other hand, Mays and Branch-Mays in their review of self-assessment use in pre-clinical and clinical dental education observed that most studies did not consider self-assessment training during their curriculum.^[6]

Correspondingly, the use of the internet and e-learning formative assessments can enhance self-assessment and critical appraisal abilities.^[6] Indeed, recent studies have shown that the use of smartphones particularly for educational reasons would be most beneficial.^[10]

In the present study, an experimental platform was established to advance the self-assessment skills of 3^rd-year dental students. To investigate if the experimental platform had a positive impact, a questionnaire was put forward to monitor the students’ appraisal of the platform, and a quiz was prepared to assess the students’ ability to evaluate examples of preparations and treatments according to the standards established by the faculty graders. The students’ assessments were then, compared to those obtained by the faculty graders.

The aim of the present study was to evaluate the educational model of this experimental platform. The hypothesis was that no differences would be found in the assessment results between the students and the faculty graders.

MATERIAL AND METHODS

The experimental platform

The assignments required from 3^rd-year dental students during their academic year practical sessions (held weekly) ranged from cavity preparation, endodontic treatment, or crown and bridge preparations. Students were asked to take photographs of their performed task using their (or their colleagues) smartphone devices, from two angles. These photographs would then be inserted by the students into a slide (PowerPoint or a PDF format), with the comments and observations of each student on the quality of his or her work.

An experimental educational platform was created, to which students were required to send their slides. Students would indicate their name, forename, and group so that they could be identified by the faculty graders.

An example is presented in Figure 1, and remarks and self-evaluation comments about the imperfections and flaws present are included in the slide, showing the deviations and shortcomings of these in relation to the ideal.

Close

Figure 1:

Example of presentation delivered by students commenting on shortcomings and imperfections of a crown preparation.

Export to PPT

Finally, the obtained slides were stored and presented as a form indicating the name, the date, and the task required [Figure 2]. The corresponding link is present in each cell of the table so that at any time faculty graders and staff members can click on the link and access the denoted slide [Figure 2].

Close

Figure 2:

The table deposited on the platform indicating the name, group, and the task required.

Export to PPT

The platform was made accessible to all the faculty graders in charge, with the list of names of students in each group. Consequently, they could consult it, whether to appreciate the overall progress of the group on a weekly or monthly basis or observe the common mistakes that are prevalent for a certain type of cavity or preparation. Now and again, these can be useful to make comparisons between groups or certain students or to observe the evolution during the semester whether positive or negative taking place to the learning curve of certain students, or for the group in general.

It should be noted that the evaluation attributed to each student was ascribed to the level of self-assessment and analysis of his/her work and not to the number of errors present in the task since the students were still in the training phase.

RESULTS

The following tables [Tables 1-2, Figure 3] represent the results obtained from 3^rd-year dental students’ group (n = 86), as compared to the results obtained from the faculty graders group (n = 8).

Close

Figure 3:

Ranking evaluation percentage for students and graders concerning different criteria. (0: Very bad, 1: Bad, 2: Good, 3: Very good, 4: Excellent).

Export to PPT

The crown preparation evaluation results seemed to differ according to the quality of the preparation; for the metal-ceramic preparation, statistical differences were found between the faculty graders and students’ evaluation concerning depth (P = 0.001), Homothetic (P = 0.002), Limits, (P = 0.001), Finishing (P = 0.008) respectively. On the other hand, the evaluation results for the preparation margin were not significantly different between the students and the faculty graders groups (P = 0.074). As for the ceramic preparation which was of a better quality (the overall evaluation marks were higher for both groups), no significant differences were found except for the finishing criterion (P = 0.002).

Concerning the endodontic access cavity, no significant differences were found except for the finishing criterion (P = 0.001). The endodontic canal obturation evaluation showed significant differences regarding all the points with the exception of the limit criterion (P = 0.147). The overall grade attributed was only significantly different for the metal-ceramic preparation and the endodontic canal obturation [Table 1].

The grader’s evaluation percentage scores for each of the photograph slides were compared to the students’ assessment percentage scores. The difference between the two was referred to as the student-faculty (S-F) gap [Table 2]. The S-F gap for metal-ceramic crown preparation and the endodontic canal obturation were 9 and 10%, respectively.

The results of the students’ questionnaire concerning the self-assessment platform are presented in Figure 4 and were analyzed using non-parametric Chi-square test. The replies were significantly positive concerning the students’ perception of the platform (P = 0.001–P = 0.028), and the only exceptions were question number three (P = 0.317), concerning the time consumption, and question number seven (P = 0.316) regarding the students’ photograph comparison.

Close

Figure 4:

Replies for the evaluation questionnaire of the self-assessment platform.

Export to PPT

DISCUSSION

The experimental platform developed in the current study supports the CODA standard, requiring that graduated dental students are able to assess their work, and demonstrate professional skills and values, essential for their self-directed learning.^[4]

It was truly assumed that an image is worth a thousand words. In the context of teaching, photographs can be of great value in conveying information to the students about the required form or outcome of their work.^[11] Moreover, the storage of these photographs as a point of reference can be beneficial for the students to monitor their progress.

The presumed advantage of the platform utilized in this work, is its simplicity, ease of use, and that no complicated tools or expensive software are needed for its implementation. It represents such an accessible option for institutions that lack the resources for elaborate, complex, and costly equipment. Evaluating preparations and cavities only visually is suboptimal, yet the objective evaluation of the task is not the aim, the main objective was to develop the students’ awareness and self-assessment skills, in that regard the platform accomplished the aim of training the students in these skills. The significantly positive outcome of the replies to the questionnaire by the students provides further proof.

Several studies have demonstrated factors influencing self-assessment skills, and these include pre-clinical performance, gender, personality traits, and the use of digital dentistry.^[5,12]

Earlier studies cast some doubt on portfolios and their role in the improvement of knowledge, understanding, and increased self-awareness for the students, yet the same studies acknowledged that they encourage students’ engagement in analytical reflection of their work.^[13] These studies also observed that students tend to be reluctant to participate unless required to do so.

In general, portfolio assessment requires time and commitment from both faculty graders and students, and as more recent studies have shown, they result in an excellent student-mentor relationship, facilitate students’ feedback so that they can learn from their mistakes, and build on accomplishments. Moreover, they instill self-reflection and lifelong learning.^[14,15]

The experimental platform presented in this work and the portfolio share many features, yet the main difference between the two is that the platform is more interactive and allows continued surveillance and communication between the grader and the students. It also permits the appraisal of students’ evolution during the academic year and observes common errors and trends in a more systematic way.

Another potential benefit of this platform was to help students cope with the problem of stress management, which is quite prevalent among students in the dental field^[16,17] and mostly after the lockdown and COVID-19 episodes. We presume that the fact that students can better assess their work, communicate easily with their faculty graders, and understand in a better way the manner of reviewing and evaluating the performed work, would make them more confident and less anxious in operating the tasks required from them.

Another interesting aspect is that through photograph evaluation, the students realize and appreciate more details that they could not recognize at the first look, and that through the magnified photographs on their computer screens, they could examine their work and its shortcomings in a more efficient manner. This was again confirmed through the students’ replies to the survey questionnaire.

Concerning the experimental platform presented in this study, the level of participation throughout the year was not optimal (65–70%), taking into consideration that there were no notes assigned to rendering the photographs. This was probably due to the fact that they knew beforehand that this evaluation did contribute to the final note for the semester.

Previous studies have shown that there was a tendency for students to be less critical in studies in which a grade was connected to the process. In addition, there was a trend for better-performing students to underrate themselves and poorer-performing students to overrate themselves and, overall, for students to score themselves higher than their faculty evaluators.^[6,18] The same observations were found in the current investigation.

The results obtained in the current study show more agreement and less variation concerning certain criteria, while other criteria reflected greater variability.

The evaluations submitted by the students concerning the crown preparation seem to suggest that they were less able to detect imperfections in the preparation with the lower quality (the metal-ceramic preparation) (significant differences in evaluating margin, depth, homothetic, limits, and finishing P = 0.074, P = 0.001, P = 0.002, P = 0.001, P = 0.008, respectively) while they were able to confirm the better quality of the second preparation and no discernable differences could be found except for the finishing step (margin, depth, homothetic, limits, P = 0.585, P = 0.533, P = 0.568, P = 110, respectively, and finishing P = 0.002). As expected, students tended to grade their work higher compared to the faculty graders, and this tendency has previously been recorded in other studies.^[19]

Moreover, some evaluation criteria seem to be more easily understood and appreciated, as can be shown clearly in the corresponding figures. For example, it has been previously shown that students tend to have more tapering crown preparations and they are less conscious about the amount of reduction required for crown preparations, especially for posterior teeth, this was evidenced in this investigation as well as in earlier studies.^[20]

Gratton et al. found that correlations between visual and digital assessment scores for students were modest concerning tooth preparation,^[21] yet they maintain the importance of such assessment skills for monitoring dental students’ progression. Earlier studies have shown that measured evolution in self-assessment skills could be obtained with regular monitoring.^[22] Moreover, a recent study retains that portfolio assessment, which observes the continued progress and evolution during the academic year is more beneficial than a single final examination or evaluation.^[23]

Habib found intergrader variation for ceramic crown preparation,^[24] this can be compared to the assessment variation between faculty graders present in the current work; it also highlights the importance of better defining the validation criteria.

Concerning the results for the access cavity preparation and endodontic obturation, results showed that the students had less critically evaluated the latter. Thus, while no significant differences were found in evaluating the access cavity between the two groups, students seemed to be less insistent concerning the finishing phase (Form, Undercut P = 0.361, P = 0.358, respectively, Finishing P = 0.001).

On the other hand, the results showed that students failed to be sufficiently critical concerning the endodontic canal preparation and obturation (Density, Conicity, and Homothetic P = 0.001, P = 0.001, P = 0.002, respectively), except for the length (Limit) criterion (P = 0.147). Indeed, several studies found that students are more stressed when performing endodontic requirements, which would impact their ability to judge and evaluate. The same studies indicated that students tend to seek more innovative resources and training methods to obtain more confidence in their operating skills.^[25,26]

The evaluation exercise presented in the current study shows that some criteria for assessment revealed a high degree of variability; these findings suggest the need for greater attention to systematically teach critical-assessment as part of the curriculum and for further research on its impact.^[6]

Unfortunately, the experimental platform used in the current study lacked most of the features that would be present in an advanced website mounted, especially for that purpose. A specialized website would include more data, allow easier access, and be more interactive, allowing to add recommendations or to perform studies on the upward or downward trends of the collective students’ work.

More innovative methods could be used to demonstrate deficiencies in students’ operative skills,^[27-29] whether virtual simulators or those based on computer-aided design and computer-aided manufacturing technology.^[30] Other approaches even use intra-professional education programs to boost students’ ability to self-assess and communicate efficiently.^[31] Lee et al. found that digital self-assessments were more accurate than conventional self-assessments and that students when grading themselves digitally were more critical of themselves.^[32] On the other hand, self-assessment programs and strategies provide an efficient and easily accessible means to attain its required pedagogic role.^[33] Moreover, students found portfolio assignments beneficial and increased their self-reflection in pre-clinical dental courses.^[34]

The current study had some limitations; more variation in the type and quality of the photographs would have been more useful. Moreover, separating the students into two groups: Those who participated in the experimental platform during the academic year and those who did not would have been useful in measuring the actual impact on the critical evaluation of students. The grading of photographs could have been more simplistic and differentiated more between acceptable and non-acceptable shortcomings and imperfections in the performed task. Nonetheless, the study revealed some interesting points and highlighted some important aspects to be investigated in the process of preparing future dentists.

Moreover, a long-term study should be established, to measure the impact of such tools on the performance of students during successive years, and whether they continue to apply self-assessment measures in their practice or not.

Moreover, the current developed platform would help open up discussion between the staff members, around certain general observations regarding the curriculum in general.

CONCLUSION

The evaluation exercise presented in the current work shows a higher degree of variability concerning some criteria. This could be due to either a lack of assessment experience or failure to recognize the indicated criteria, in both cases this issue has to be addressed to achieve the required objectives of the training process.

The current platform tool provides several beneficial features including ease of setup, simplicity, and low cost. The students’ perception of the platform was significantly positive, and they found it to be a useful and efficient tool to enhance self-assessment, communication, and the pedagogical process in general.

Moreover, a more advanced version with additional features would increase the effectiveness of the teaching process.