RESEARCH

 

Facilitating technology-enhanced external examination moderation during the Covid-19 pandemic

 

 

N Potgieter^I; N Mohamed^II; RJ Vergotine^III; CW Peck^IV

^IBChD, PDD, PGDip (Dent), MSc (Dent), Senior lecturer, Department of Paediatric Dentistry, Faculty of Dentistry, University of the Western Cape, South Africa. ORCID:0000-0003-4061-3322ac.za
^IIBChD, BSc Hons, MSc, PhD, MPhilHPE, 2 Professor and HOD, Department of Paediatric Dentistry, Faculty of Dentistry, University of the Western Cape, South Africa. ORCID: 0000-0003-2184-2648
^IIIBChD, MSc (Dent), FAAPD, FIADT, FABPD, Clinical Associate Professor, Department of Orthodontics and Pediatric Dentistry, School of Dentistry, University of Michigan, United States of America. ORCID: 0000-0001-8445-1121
^IVBMedSc, BChD, MPhil HPE, Lecturer, Department of Paediatric Dentistry, Faculty of Dentistry, University of the Western Cape, South Africa. ORCID: 0000-0001-6695-9554

Correspondence

 

 

---

ABSTRACT

INTRODUCTION: Due to the Covid-19 pandemic and associated travel restrictions, the physical presence of international external examiners was a challenge when assessing the exit level outcomes of the MSc (Dent) in Paediatric Dentistry at the University of the Western Cape
External moderation of final examinations ensure an acceptable standard, coverage of content as specified by the programme outcomes and eliminates bias during assessment. Internationalization of the moderation and examination process allows countries to compare and maintain international standards and graduate attributes expected for professional qualifications.Qualifications requiring assessment of skills often rely on Objective Structured Clinical Examinations, Objective Structured Practical Examinations and simulated cases in combination with an oral examination, which requires the presence of all examiners to assess the student.
AIMS AND OBJECTIVES: This paper describes how the final examination in this MSc (Dent) degree was adapted and conducted in order to overcome the challenges of the Covid-19 pandemic, to maintain the academic integrity and rigour of the programme.
Design and Methods: A narrative essay-style approach was adopted, which reflects on the challenges and opportunities created by Covid-19.
CONCLUSION: The adapted assessment method proved to be an effective alternative to the more traditional assessment approaches employed pre-Covid.

Key words: International External Examiner, Virtual Oral Examination, Adapted Assessment, Covid-19 pandemic, Assessment Challenges.

---

 

 

INTRODUCTION

The MSc (Dent) Degree in Paediatric Dentistry is a full-time two year clinical programme that not only requires in-depth theoretical knowledge but also proficient clinical skills and professional competency during patient interactions. It is essential that the application of theoretical knowledge is assessed together with clinical skills throughout the course in the form of continuous assessment, but specifically at the end of the course by means of a final exit level examination. This MSc (Dent) differs from most other MSc programmes as well as the MSc (Dent) in Paediatric Dentistry by-thesis, in that it has an integrated clinical component as part of the degree over the two years.

The assessment practices implemented were guided by the assessment policy for post-graduate students of the University of the Western Cape (UWC), South Africa.¹ Continuous assessment was weighted at 60% and the final summative assessment at 40% of the final mark. This has potential of enhancing fairness of the departmental assessment practices, in that it places more emphasis on student achievement and progression over the two-year degree than on a final high-stakes examination.

The final examination was structured to include a written theoretical examination (40%), a case-based written examination (20%), an online e-portfolio (20%) and a comprehensive oral examination including a simulated case (10%) and a presentation of a prepared case (10%). An independent external examiner plays a crucial role in ensuring an objective analysis of examinations and student performance. The Department of Paediatric Dentistry at UWC utilised the services of an external examiner from the University of Michigan in the United States of America. since 2016. The external examiner moderated and participated in all the components of the final examination as explained below.

Written theoretical and Case-based examinations

Sound theoretical knowledge is essential for application in the clinical context. The written theoretical examination consisted of essay questions, which were marked independently by more than one assessor to improve reliability, and short questions that had a clinical slant but with the focus on in-depth theoretical knowledge. Questions covered the entire spectrum of content addressed in seminar discussions and article critiques during the two years and are aligned with the learning outcomes as stipulated in the study guide. Content validity is thus ensured.² Consequential validity is not an issue as blueprinting of the content ensures that topics are evenly distributed.³ All content is weighted equally² as students are required to have a broad idea of all aspects of Paediatric Dentistry.

A case-based written examination, continuous clinical assessments and an e-portfolio (which will be discussed later) cover the clinical competency assessment in this module. The questions in the case-based examination were similar to case summaries,⁴ where students are presented with a clinical scenario including a history with accompanying photographs and radiographs. A series of clinical questions was based on the case. Students were then required to make deductions regarding the management and treatment options based on the information provided. The main focus of this exam paper was to assess the student's ability to critically apply their knowledge to the clinical context, by requiring them to interpret and diagnose the clinical problem and draw up a treatment plan. Both the written examinations were compiled by a minimum of three departmental staff members and were then moderated by the external examiner. The external examiner was totally independent in that he was not involved with the training of the students who were being assessed. Making use of more than one examiner improves reliability and fairness.^{2, 5 6}

All examination papers and memoranda were moderated in advance of the examinations purely via email using password protected documents to maintain confidentiality. Open discussion and exchange of opinions allowed for moderation and adaptation of the papers and memoranda before the examination. Question papers were also moderated to ensure clarity of wording, degree of difficulty and content validity.⁵ This moderation process was a positive experience which reinforces the standard of training by ensuring that people are being assessed in a consistent, accurate, and well-designed manner and that assessors use comparable assessment tools and methods. This translates to making analogous and consistent judgements about learners' performance and the scope of work covered. Both the internal and external examiners were satisfied with the papers.

The quality of handwritten answer sheets is always a challenge due to legibility of writing. For the first time in 2021, answers were typed in a word document instead of being handwritten. Each answer sheet was set up in a word document and was labelled with the student number as a footer on each page. Question numbers and section breaks separated the different sections. This method was preferred by both students and examiners and allowed for greater clarity, ability to organize thoughts more coherently and removed the issue of illegible handwriting. Sharing of the document was also easier.

Students were allowed adequate time to complete the written examinations, which took variations in typing speeds into account, for example. This therefore ensures validity³ and fairness, by ensuring the assessment is student-focussed. According to the university's assessment policy, a minute should be allocated per mark, but additional time was factored in to accommodate for the fact that most international students in this programme only had English as a second or third language. As most of the cases required interpretation, planning and formulation of treatment plans, additional time was allocated. Students were also encouraged to ask questions during the examination if clarity was required on any aspect.

After completion of the examination, the document was saved as both a word and pdf version in the presence of the student to ensure that no changes could be made after the examination. The different sections were emailed to the various internal examiners and combined again after marking. The use of a memorandum and the fact more than one assessor graded the papers, improves the reliability of the assessment. Of the examination components, the written examinations carry the highest weighting. This is justified as more value may be placed on a reliable written assessment compared to an oral examination.⁷ The completed, marked scripts were then sent to the external examiner for moderation. The moderation of the written papers prior to the examination as well as after marking, was not experienced as a challenge as it could all be done online.

e-Portfolio

Students shared their e-portfolios with all the internal and external examiners three weeks before the oral examination. The portfolios included their seminar presentations, written assignments, and case presentations that were completed during the course of the programme. Portfolios have been shown to be a good way for monitoring student progress and assessing competence development.⁸ The e-portfolios were easily accessed internationally, making it possible for the quality of the programme to be assessed in greater depth.

The e-portfolios gave insight into the depth of understanding and application the students have regarding theoretical knowledge, but also showcased their clinical competency. In this programme, it is used to demonstrate a variety of cases and students have to reflect on the treatment options with motivation from the literature. It is therefore an indicator of the student's progress and growth in the discipline.^{7, 9}

The patient cases that are included in the portfolio have either been presented in the clinic or during the case presentations. Students have therefore received feedback on these cases prior to inclusion in the e-portfolio. These cases have also been assessed by different supervisors in the clinics. Portfolios are assessed by at least three examiners (two internal and one external) which addresses some of the subjectivity associated with portfolio assessment¹⁰ and improves reliability. The use of a rubric and multiple examiners, limits subjectivity and ensures reliability and validity of this assessment method.¹¹

Scientific reasoning and referencing were applied in all assignments and case discussions. It was interesting that the e-portfolio marks awarded by the examiners correlated with the Continuous Clinical Assessment marks, which validates the use of an e-portfolio as a final assessment tool. This was not tested statistically, but merely an anecdotal observation.

Oral examination

The oral examination included an unseen simulated case and a presentation of a completed comprehensive case followed by questions and discussion. In preparation for the oral examination, a Google drive folder was shared with all the examiners participating in the examination. The folder contained all continuous assessments up to date, the examination papers, the oral examination simulated case interview script (with memorandum, supplemental images and radiographs), and the rubrics for the oral examination. All examiners had time to familiarize themselves with the contents before the oral examination.

Information Technology played a key role in enabling the oral examination. All the internal examiners and the students were present in one venue with the external examiner joining remotely from abroad. The decision to host the examination both online and on site contributed to creating an examination environment which was conducive to observing the students as well as facilitating discussion. Having more people in a room with a limited number of electronic devices could reduce the chances of unforeseen complications with connections, such as interference and echo on the international call. Background noise and acoustic and audio feedback were experienced when multiple devices were tested in one room at the same time. Only two laptops were set up for communication with the external examiner. The main laptop was used by the module coordinator to share the screen of the simulated case / rubrics or whatever was necessary to share with the examiners. This main laptop was also connected to the projector in the venue, thereby allowing all the examiners to see the images / presentations.

Oral examination - simulated case

The oral examination was structured to start with the unseen simulated case. The clinical examination is intended to assess the students' ability to formulate a diagnosis and motivate their treatment options taking all factors into consideration. A simulated case was considered instead of a patient case which was previously used, as the latter was not standardised and students were exposed to different patients with varied needs. This is therefore not ideal and negatively affects the fairness and validity of the assessment. A child cannot be expected to serve as an exam patient for more than one student where the same procedures are repeated and the same questions are asked. Hence, different patients had to be booked for the students. Additionally, the ethical aspects of using a child as an examination subject remain a challenge.

The student was invited into the examination room and positioned so that the one laptop's camera was facing the student and the actor, who posed as the mother. The student conducted a history by interviewing "the mother". The student followed a systematic approach, obtaining all the necessary information from "the mother". "The mother" followed a script, ensuring that the same information was given to both candidates. This interview process allowed the examiner to evaluate the student's interaction with "the mother", history taking skills, and the ability to identify areas that required further interrogation. The internal examiners observed the interview process directly while the external examiner observed the interview through the aligned laptop.

After the interview, the student was allowed 30 minutes to compile a diagnosis and treatment plan based on the history, additional images and radiographs that were made available to them. This took place in an adjacent office under supervision. Thereafter, students returned to the examination room and presented their diagnosis and treatment plan to the entire panel using the same setup as before. The external examiner primarily interacted with the students, asking questions and discussing the case. All the internal examiners could however also join in on the discussions throughout. All the examiners used a rubric for assessment of this case.

Oral examination - prepared patient case presentation

For this component of the oral examination, the student presented a comprehensive patient case which showcased the quality of the work they have done. These cases are chosen by the student from the cases already presented during the course of the programme. Due to the stressful and intimidating nature of an oral examination, its reliability and fairness has been questioned,¹² especially when used to make a decision about whether a student should pass or fail.⁷ This examination was thus just a formality to confirm their continuous assessment performance. The familiarity of the patient case mitigates the stressful situation to some extent and contributes to fairness of the assessment.

The case was presented as a PowerPoint presentation and included the history, clinical photographs, radiographs, diagnosis and treatment plan as well as a summary of all the treatment completed. Students included their rationale for the treatment and motivation for the choice of materials. After the presentation, the student answered questions posed by the external examiner for this specific case. The external examiner led the discussion and asked specific questions regarding motivation for particular procedures and the case in general. Integration and application of content to the clinical context could thus be assessed. A different rubric was used for this presentation.

A discussion between the internal examiner panel and external examiner concluded the exam. All examiners agreed on the standard of the students although the marks were not discussed. Individual examiner marks were tallied after the oral examination. All marks were within a 5% range indicating an agreement between all examiners regarding the standard of the students. In cases where marks were not within a 10% range, an online discussion would have been initiated between the examiners to allow for motivation and consensus.

Table 1 below summarizes the challenges, how they were overcome and ongoing challenges of an oral examination for a clinical module, requiring the services of an international external examiner.

 

Table 1 - Click to enlarge

 

The students' feedback regarding this final examination was obtained through an anonymous online questionnaire. Their expectations of the final examination were met and was expressed as follows by one of the students: "The exam had a lot of variety and we were given a chance to present a case of ours. I liked the variety of assessment tools. And especially the ability to add short cases to the e-portfolio added a lot in terms of giving justice to our clinical work during the program." Students experienced the combination of live and virtual interaction during the oral examination to be effective, however, they would have liked to have more interaction with the external examiner. One of the comments was "not knowing his response or expression through the online connection", which indicates the value of live interaction. The authors acknowledge that although the student cohort was small (two), their comments are of value and useful but using these comments to guide change may be problematic.

All students indicated that, of all the assessment components, they enjoyed the Clinical Written examination the most and that the e-portfolio was an effective tool to showcase their competencies and skills. When students were asked how the examination can be improved, the comments included to have longer time for completing the written examinations and to have mock exams throughout the year to better prepare them for the final examination.

 

CONCLUSION

1. The COVID-19 pandemic presented significant problems with utilizing an international external examiner for the final examination assessment of the Paediatric Dentistry Masters students at UWC.

2. The use of a remote internet platform alleviated most of the issues that travel restrictions travel placed on the utilization of an international external examiner.

3. Technological issues should be expected when using remote platforms and the development of work-around solutions ahead of time can result in a speedy resolution of issues and alleviate stresses associated with this process.

4. Students are receptive to using remote options for examinations, but would prefer the presence of examiners to improve interactions through face-to-face contact.

5. The combination of an e-portfolio and virtual oral examination are acceptable examination tools for clinical competence while also allowing assessment by an international external examiner.

6. Virtual simulated cases can be an acceptable substitute for live patients. This would address issues created by lockdowns as well as the ethical issues created by using actual patients for examination purposes. Standardisation of the patient case also enhances fairness of the assessment as all students are exposed to the same case.

Acknowledgement: With thanks to the students who willingly participated in the examination and feedback processes.

Declaration of Interest: None to declare. All authors have contributed to the conception, writing and editing of this manuscript. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Funding statement: We wish to confirm no funding was received or used for the purpose of this publication.

 

REFERENCES

1. The University of the Western Cape. UWC Assessment Policy. 2021. Available from https://www.uwc.ac.za/flles/files/General-Calendar-2021-Assessment.pdf. pp 88-90.         [ Links ]

2. Schuwirth LW, Van der Vleuten CP. Understanding medical education: evidence, theory, and practice. 3rd Edition. New Jersey: Wiley-Blackwell; 2018. Chapter 20. How to Design a Useful Test: The Principles of Assessment. pp 275-89. Available from https://doi.org/10.1002/9781119373780.ch20        [ Links ]

3. Patil SY, Gosavi M, Bannur HB, Ratnakar A. Blueprinting in assessment: A tool to increase the validity of undergraduate written examinations in pathology. International Journal of Applied and Basic Medical Research [Internet]. 2015 [cited 2022 Feb 8];Aug;5(Suppl 1):S76-9. doi: 10.4103/2229-516X.162286.         [ Links ]

4. Dory V, Gagnon R, Charlin B, Vanpee D, Leconte S, Duyver C, et al. In brief: Validity of case summaries in written examinations of clinical reasoning. Teaching and Learning in Medicine. 2016;28(4):375-84.         [ Links ]

5. Hays RB, Hamlin G, Crane L. Twelve tips for increasing the defensibility of assessment decisions. Medical Teacher. 2015;37(5):433-6.         [ Links ]

6. Van Der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Medical Education. 2005;39(3):309-17.         [ Links ]

7. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. The Lancet. 2001;357(9260):945-9.         [ Links ]

8. Driessen EW, Overeem K, Van Tartwijk J, Van Der Vleuten CP, Muijtjens AM. Validity of portfolio assessment: which qualities determine ratings? Medical Education. 2006;40(9):862-6.         [ Links ]

9. Sackett PR, Schmitt N, Ellingson JE, Kabin MB. High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative-action world. American Psychologist. 2001;56(4):302.         [ Links ]

10. McMullan M, Endacott R, Gray MA, Jasper M, Miller CM, Scholes J, et al. Portfolios and assessment of competence: a review of the literature. Journal of Advanced Nursing. 2003;41(3):283-94.         [ Links ]

11. Gadbury-Amyot CC, McCracken MS, Woldt JL, Brennan RL. Validity and reliability of portfolio assessment of student competence in two dental school populations: a four-year study. Journal of Dental Education. 2014;78(5):657-67.         [ Links ]

12. Memon MA, Joughin GR, Memon B. Oral assessment and postgraduate medical examinations: establishing conditions for validity, reliability and fairness. Advances in Health Sciences Education. 2010;15(2):277-89.         [ Links ]

 

 

Correspondence:
Nicoline Potgieter
Department of Paediatric Dentistry
University of the Western Cape, Faculty of Dentistry
Francie Van Zijl Drive
Parow, 7505, South Africa
Telephone: (work) +27 21 937 3107; Cell: 082 921 9645
E-mail: nipotgieter@uwc.ac.za

 

 

Source of funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author contributions:
1 . Nicoline Potgieter: 25%
2 . Nadia Mohamed: 25%
3 . Rodney J Vergotine: 25%
4 . Craig W Peck: 25%