RESEARCH

 

Assessing orthodontic treatment outcome of patients treated by orthodontic residents - using the Peer Assessment Rating index

 

 

JC Julyan^I; H Bellardie^II; A Harris^III

^IUniversity of the Western Cape, Orthodontic resident. ORCID ID: 0000-0002-6186-5724
^IIUniversity of the Western Cape, Consultant. ORCID ID: 0000-0002-8632-9960
^IIIUniversity of the Western Cape, Consultant. ORCID ID: 0000-0001-6237-1200

Correspondence

 

 

---

ABSTRACT

BACKGROUND: Assessing the degree of improvement is important for establishing the standard of care provided by an individual orthodontist or tertiary care institution.
OBJECTIVE: This study aimed to assess the orthodontic treatment outcomes of patients treated by residents in a postgraduate orthodontic programme at a University in South Africa.
DESIGN: Retrospective assessment of orthodontic treatment outcomes using the Peer Assessment Rating (PAR) index.
SETTING: Department of Orthodontics at a University in South Africa. Participants Patients who completed fixed orthodontic treatment between May 2016 and May 2021.
METHODS: The PAR index was used to assess pre- (T0) and post-treatment (T1) orthodontic study models. Additional outcome measures that were assessed included extraction, non-extraction, orthodontic bracket prescription, impactions, Angle classification and duration of treatment.
RESULTS: Seventy-four patients were included in the study, with a mean age of 16 years and 6 months, and a sex distribution of 47 (63.5%) females and 27 (36.5%) males. The mean treatment time was 32 months, with a mean weighted score reduction of 28.1 (86%). The sample had 72 (97.3%) patients categorised as "improved" and 2 (2.7%) as "worse or no different". Of the "improved" patients, 52 (70.2%) were "greatly improved".
CONCLUSION: The mean percentage weighted PAR score reduction of 86% for the sample reflects a high standard of care provided by the orthodontic residents at the tertiary care centre.

Keywords: Orthodontics, Occlusal Index, PAR Index, Treatment outcome, Orthodontic residents, Standard of care

---

 

 

INTRODUCTION

Orthodontic treatment aims to provide patients with worthwhile improvement in the overall alignment and occlusion of their teeth. In addition, it aims to improve the facial appearance, which contributes to the psychological and physical wellbeing of patients.¹ Continuous assessment of the quality of outcomes is essential for the development and maintenance of optimal standards of orthodontic care.¹ According to Richmond et al.,² it has been a common practice to grade orthodontic treatment outcomes at study groups, resulting in an increased interest in recent years in the development of techniques or methods to reduce subjectivity when assessing orthodontic outcomes.

Occlusal indices have fulfilled this role in orthodontics and they are used to record traits of malocclusion numerically or categorically, to enlist a degree of objectivity when assessing malocclusions compared to the subjective method.³ Objective assessment is important in orthodontics and can reflect whether patients finish treatment with a worthwhile improvement in the overall alignment of their teeth, as well as proper occlusion. According to Onyeaso and BeGole,⁴ the objective assessment of orthodontic treatment results should not be limited to individual patients but also include the greater proportion of the orthodontist's caseload.

When assessing the treatment outcome of an orthodontist's caseload, a greater proportion of cases should show improvement. The quality of future orthodontic treatment can be improved when individuals grade their treatment results.² According to Richmond et al.,² when different orthodontists use their own set of criteria, it becomes difficult to evaluate orthodontic treatment results accurately. To overcome this subjective evaluation, a standardised objective index for assessing orthodontic treatment outcomes was deemed necessary.

The Peer Assessment Rating (PAR) index developed by Richmond et al.² is an example of an occlusal index. It offers orthodontists a reliable and standardised tool to evaluate treatment outcomes. The PAR index is a useful tool when evaluating orthodontic treatment and assessing the standard of care.⁵ The reduction in PAR index scores of greater than 70% reflects orthodontic treatment success.² Orthodontic success was determined by the percentage reduction in the PAR score or by using a nomogram graph. The PAR index enables clinicians and researchers to evaluate outcomes achieved through orthodontic treatment by relating the study models before and after treatment.⁶ The PAR index is an objective assessment tool that allows evaluation of the standard of orthodontic care provided.²

 

METHODS

Design

Retrospective assessment of orthodontic treatment outcomes of patients treated by orthodontic residents, using the Peer Assessment Rating (PAR) index.

Population

The convenience sample comprised 74 patients who were treated with fixed orthodontic appliances by residents between May 2016 and May 2021. The inclusion criteria included patients with no previous history of orthodontic treatment, with good quality pre- and post-treatment orthodontic study models and complete treatment records. The following excluding criteria was used: patients treated by permanent staff members, patients with poor quality or missing orthodontic study models, uncompleted treatments, patients with craniofacial anomalies and patient who underwent orthognathic surgery. Patients with incomplete treatment records and premature removal of orthodontic appliances due to poor oral hygiene or dental concerns were also excluded. All patients were treated by orthodontic residents at the institution under the supervision of qualified orthodontists, who were either full- or part-time faculty members in the Department of Orthodontics at the time.

Setting

All treatments were performed at one of the South African tertiary care institutions, and treatments were provided by residents in the Department of Orthodontics.

Intervention

The pre-and post-treatment orthodontic study models and treatment records of patients treated using fixed orthodontic appliances were collected retrospectively. The PAR index was used to determine the success of orthodontic treatment. Additional outcome measures for the treated cases were also assessed using treatment records and included extraction, non-extraction, orthodontic bracket prescription, impactions, Angle classification and duration of treatment.

All PAR index measurements were performed at two different times: before treatment (T0) and after treatment (T1). The measurements included alignment of the maxillary and mandibular anterior segments; buccal occlusion in the anteroposterior, transverse and vertical planes; overbite; overjet; and centerline alignment. The measurements of the pre- (T0) and post-treatment (T1) study models were blinded. Pre-treatment models were measured initially and post-treatment models were measured one month later. The names of patients and treating residents were concealed to exclude bias. The degree of improvement after orthodontic treatment was organised into three categories: "greatly improved", "improved" and "worse/no improvement". Two methods exist for the assessment of orthodontic treatment outcomes using the PAR index: (1) the numerical reduction in the weighted PAR score or (2) using the percentage reduction in the weighted PAR score. A reduction of the PAR score of at least 30% is considered as being an improved case. When there is a reduction of 22 PAR points or more the case is considered greatly improved. High standards of orthodontic treatment refer to situations where the proportion of caseload being "worse/no different" is less than 5% and the mean reduction is above 70%. If the majority of patients are "greatly improved" it suggests that the treating practitioner is providing treatment of a high standard to a large proportion of patients. When the reduction in the PAR score is calculated, the amount of improvement is influenced by the pre-treatment PAR score. Not every patient has a pre-treatment PAR score of 22, which means that a proportion of cases cannot be classified as greatly improved, according to Richmond et al.. ²

Outcomes

The orthodontic study models were scored by three examiners (principal investigator and both supervisors). The main supervisor underwent training in calibration using the PAR index at one of Prof Stephen Richmond's PAR calibration courses. The study models used were the pre-and post-treatment (at the time of debonding) of each patient in the sample. Patient records, including clinical notes, were retrospectively assessed for other outcome measures. The treatment time was calculated from the application of the orthodontic fixed appliances until their removal.

 

DATA MANAGEMENT AND DISPOSAL

Data was collected using a Microsoft Excel spreadsheet to facilitate analysis and graphical output. All patient data were recorded anonymously and presented in aggregate. Any data exported into statistical programs (such as Excel or Stata) was stored in the institutional research data repository.

 

ETHICS APPROVAL AND CONSENT

The research protocol was presented to the Higher Degrees Committee and the Biomedical Research and Ethics Committee of the university and was approved as a research project (BM22/4/9). All patients signed an informed consent form stating that their records may be used for academic purposes. All the information obtained was handled with strict confidentiality. Patients were anonymised and each patient was allocated a number - for example, Subject 1. All data was stored on a password protected computer.

 

STATISTICAL ANALYSIS

Pre-treatment orthodontic study models were scored initially by all three examiners. To blind the examiners, the post-treatment orthodontic study models were scored a month later. At both pre-treatment and post-treatment scoring, the samples were randomised and anonymised. The complete data set was used by the statistician to assess inter-examiner reliability. Intra-examiner reliability of the pre-treatment and post-treatment PAR scores was carried out by random rescoring of 30 study models, one month after the last scoring session by the examiners.

Inter-and intra-rater reliabilities were assessed using a two-way random-effects model. For the pre-treatment readings, the inter-rater reliability was excellent (ICC 0,87; 95% CI, 0,88-0,94) and the intra-rater reliability was moderate (ICC 0,87; 95% CI, 0,76-0,91). The inter-rater reliability (ICC 0,59; 95% CI, 0,65-0,81) and intra-rater reliability (ICC 0,59; 95% CI, 0,18-0,67) of the post-treatment readings were both moderate. For the difference between pre-and post-treatment ratings, the inter-rater reliability (ICC 0,84; 95% CI, 0,85-0,93) and the intra-rater reliability (ICC 0,84; 95% CI, 0,72-0,89) were both excellent.

 

RESULTS

Baseline characteristics

The sample of 74 patients had a mean age of 16 years and 6 months (ranging from 8 years 9 months to 31 years 5 months) at the start of treatment. Of the sample, 27 (36.5%) were male and 47 (63.5%) were female.

Occlusal outcomes of the patient sample (PAR index results)

The mean pre-treatment PAR score was 31,2 (SD 10,38) and the mean post-treatment PAR score was 3,1 (SD 3,60). There was a mean score reduction of 28,1 (SD 11,99). The mean reduction in PAR percentage was 86%, with 97.3% (n=72) of the cases being improved, and of the improved cases 70.2% (n=52) were greatly improved. The percentage of patients showing worse or no improvement was 2.7% (n=2). The results are detailed in both a nomogram and a traffic light bar chart, as suggested by Bellardie⁷ (Figures 1 and 2). The cases with improved and greatly improved treatment outcomes constituted 97.3% of the sample, indicating a high standard of orthodontic treatment.

In the pre-treatment group, 14 (18.9%) had PAR scores between 0 and 20, 21 (28.4%) had scores between 21 and 30, and 39 (52.7%) had PAR scores greater than 30. In the post-treatment group, 64 (86.5%) patients had PAR scores between 0 and 5, while 8 (10.8%) had scores between 6 and 10, and 2 (2.7%) had a PAR score greater than 11.

Results of the individual PAR index components

There was a significant reduction in the maxillary anterior alignment, mandibular anterior alignment, buccal occlusion in the transverse plane, overjet, overbite and midline components of 83.3% to 94.8%. Overjet showed the greatest improvement (94.8%) and buccal occlusion in the anteroposterior plane showed the least improvement (57.1%). The buccal occlusion in the vertical plane component had a zero percent reduction, but also had a zero pre-treatment score, making it insignificant. The upper anterior segment had the highest mean (6,1 ± 3,2) for the unweighted pre-treatment PAR score, and the overjet had the highest mean (13,6 ± 7,5) for the weighted pre-treatment PAR score. In the post-treatment PAR scores, the buccal occlusion in the anteroposterior plane component had the highest mean for both the unweighted and weighted PAR scores (0,9 ± 0,7) (Figure 3).

 

ADDITIONAL OUTCOME MEASURES

Distribution of extraction and non-extraction cases

In 33 patients (44.6%), orthodontic treatment was performed without extractions; in 41 patients (55.4%), orthodontic treatment was performed in combination with extractions. The average reduction in the PAR score was 28,1 (86%) for the non-extraction group and 29,0 (87%) for the extraction group.

Orthodontic bracket prescription

The appliances used for the sample included McLaughlin, Bennett and Trevisi (MBT) in 38 (51.4%) cases, Alexander in 29 (39.1%) cases and TipEdge in 7 (9.5%) cases. The difference in the PAR reduction scores of the different prescriptions used was not included because the number of cases was not equal and did not represent an accurate score.

Impactions

When the space for a tooth was less than or equal to 4mm, an impaction was recorded. Impacted canines were recorded in the anterior segment component. In the study sample, 14 (19%) patients had impactions according to the PAR index criteria. The impacted teeth were canines in 12 (86%) and lateral incisors in 2 (14%) cases.

Angle classification of malocclusion

The study sample comprised patients with Angle Class I (n=26, 35%), Class II (n=35, 47%) and Class III (n=13, 18%) malocclusions. Class II malocclusions were further divided into 26 (74.3%) Class II Division 1 malocclusions and 9 (25.7%) Class II Division 2 malocclusions. The average reduction in the PAR score was 28,6 (86%), 28,5 (87%) and 28,8 (87%) for Class I, II, and III malocclusions, respectively. Duration of treatment The mean duration of treatment for the sample was 32 ± 6,9 months. The minimum treatment duration was 22 months, and the maximum duration was 49 months.

 

DISCUSSION

Assessing orthodontic treatment outcomes helps to establish standards of care and set improvement goals, and is useful in postgraduate education clinics. Using the PAR index, a more objective assessment of the final treatment outcome was possible in the present study.

Deguchi et al.⁶ assessed orthodontic treatment outcomes at two postgraduate orthodontic clinics by using the PAR index. They found an average pre-treatment PAR score of 32 and 28 and post-treatment PAR scores of 7 and 4 for the two postgraduate clinics, respectively. Turbill et al.⁸ evaluated patients treated with removable appliances and showed mean pre-treatment weighted PAR scores of 26,94; 26,74; and 24,74 and post-treatment weighted PAR scores of 12,79; 15,19; and 11,40, respectively. The mean pre-treatment PAR score in the present study was 31,2 (SD 10,38) and the mean post-treatment PAR score was 3,1 (SD 3,60), with a mean score reduction of 28,1 (SD 11,99). The pre-and post-treatment PAR scores were comparable to those reported by Deguchi et al.⁶ The present study showed higher pre-treatment and lower post-treatment PAR scores than those reported by Turbill et al.⁸ It is noteworthy that the study by Turnbill et al.⁸ evaluated removable orthodontic appliances whereas, in the present study, patients were treated with fixed orthodontic appliances that provide three dimensional tooth movement. According to Richmond et al. ,^2,9 the cut-off point for treatment needs according to the PAR score was 10. In the present study, only 1 (1.3%) patient had a pre-treatment score of less than 10, indicating that one patient had a mild malocclusion with a low treatment need.

Not every patient has a pre-treatment PAR score of 22, which means that a proportion of cases cannot be classified as greatly improved, according to Richmond et al.² In the study by Kerr et al.,¹⁰ this was applied to one-third of patients. In the present study, 15 (20%) patients had pre-treatment scores of less than 22 points, resulting in their inability to be classified as greatly improved after treatment.

A score of zero is not always achievable because of the complexity of certain cases. A post-treatment PAR score of 10 or less indicates an acceptable occlusion and alignment, while a score of 5 or less suggests an almost ideal occlusion.² In the present study, the number of cases with a final score of zero was 4 (5.4%), less than or equal to five was 63 (85.1%) and more than five was 11 (14.9%). Therefore, a high percentage (90.5%) of cases finished with close-to-ideal occlusions.

O'Brien et al.¹¹ used the PAR index with the British weighting system, to assess 1,630 patients treated by the Regional Consultant Orthodontic Service in England and Wales. They found a mean reduction in PAR score of 67.62% with 8% of patients categorised as worse/no improvement, 48.6% as improved and 43.4% greatly improved. A similar study by Richmond and Andrews¹² in Norway found a mean reduction in PAR score of 78% for 220 patients treated by orthodontists, with only 4% of patients categorised as worse/no different. Dyken et al.¹³ found a mean percentage reduction in PAR score of 81.7% for graduate students for 51 patients assessed. Richmond¹⁴ assessed 51 consecutively treated cases and found a mean reduction in PAR percentage of 74%, with 8% of the patients categorised as worse/no different, 39% as improved and 53% as greatly improved. Buchanan et al.¹⁵ assessed 82 patients who underwent fixed orthodontic treatment with either pre-adjusted Edgewise or Begg appliances and also found a mean PAR reduction score of 74%. The results of the present study revealed that for the sample caseload, the mean reduction in PAR score was 86% with 97.3% (n=72) of cases being improved and 70.2% (n=52) of cases greatly improved. The percentage of patients showing a worse or no-improvement result was 2.7% (n=2). The present study showed more PAR improvement than the literature.^{11,12,13,14,15} The reduction in PAR percentage was similar to that reported by Onyeaso and BeGole.⁴ This indicates that the residents at this tertiary care centre provided this sample of patients with a high standard of orthodontic care when compared with other similar studies.

It should be noted that the study samples of both O'Brien et al.¹¹ and Richmond and Andrews¹² were significantly larger than those in the present study and the aforementioned studies. This motivates the need for more comprehensive studies in which all consecutive orthodontic treatments are assessed. The studies conducted by Richmond¹⁴ and Dyken et al.¹³ included patients who underwent orthognathic surgery, which resulted in a higher reduction in PAR scores according to Richmond et al.² The present study did not include any patients who underwent orthognathic surgery.

The proportion of caseloads that fall into the worse/no improvement category should be consistently negligible and the mean PAR score reduction should be as high as possible for a practitioner to demonstrate a high standard of treatment. In the literature, patients categorised as worse/no Improvement were 3%^4,11,12 and 8%.¹⁴ In the present study, 2 (2.7%) patients were in the worse/no improvement category, and both cases were multidisciplinary treatments that still required interventions from other disciplines to establish their final results.

According to Richmond et al.,² a high treatment standard is achieved when the proportion of cases that fall in the "worse or no different" category is less than 5% and the mean percentage reduction in the weighted PAR score is greater than 70%. A high proportion of cases (>50%) should also fall into the "greatly improved" category.¹⁴ When this is achieved, the practitioner or treatment centre can provide high-quality care to a substantial proportion of patients with a clear need for treatment. The present study achieved these goals by having 2.7% of cases in the worse/no improvement category, a mean PAR reduction percentage of 86%, and 70.2% of cases in the greatly improved category.

 

LIMITATIONS

The retrospective nature of this study was a major limitation. Future prospective studies can overcome this limitation by using randomised clinical trials to assess the outcomes of different treatment modalities in larger consecutively treated samples from different treatment centres.

An additional limitation of the present study was the use of study models that were taken on the day of debonding to conduct post-treatment PAR scoring and therefore do not reflect the long-term stability of the treatment outcomes. Future studies should include additional scoring of the patient results during the retention period. This will provide the opportunity to both assess the stability of the treatment results and evaluate different retention protocols to improve the field of orthodontics, which still requires more evidence based on long-term studies.

 

FUTURE WORK

The results of this study reflect the treatment outcomes and standard of care of patients treated by orthodontic residents at a tertiary care centre in South Africa. More comprehensive future studies are encouraged to evaluate the standard of care for all consecutively completed treatments in all orthodontic departments or treatment centres in South Africa. The documentation of the reduction in PAR score of all the treated cases in the various departments can subsequently be used in future accreditation meetings to reveal and improve the standard of orthodontic care for public sector patients in South Africa.Future prospective studies should also incorporate long-term follow-up measurements during retention to evaluate the stability of the results and standards of the retention protocols used.

 

CONCLUSIONS

The results of this study indicate an overall improvement in the treatment outcomes of the study sample. Patients with a clear need for treatment received care of exceptional quality.

This is a reflection of the standard of orthodontic treatment at the tertiary care centre.

Documenting the reduction in PAR scores of all consecutively treated patients in the different departments of orthodontics in South Africa should be used in future studies to improve the standard of orthodontic care provided to patients in the public sector.

Conflict of interest

The authors declare no conflicts of interest.

 

REFERENCES

1. Campbell CL, Roberts WE, Hartsfield JK, Qi R. Treatment outcomes in a graduate orthodontic clinic for cases defined by the American Board of Orthodontics malocclusion categories. Am J Orthod Dentofacial Orthop. 2007;132:822-829        [ Links ]

2. Richmond S, Shaw WC, O'Brien KD, Buchanan IB, Jones R, Stephens CD, Roberts CT, Andrews M. The development of the PAR index (peer assessment rating): Reliability and validity. Eur J Orthod. 1992a;14:125-139        [ Links ]

3. Buchanan I, Shaw W, Richmond S, O'Brien KD, Andrews M. A comparison of the reliability and validity of the PAR Index and Summers' Occlusal Index. Eur J Orthod. 1993;15:27-31        [ Links ]

4. Onyeaso CO, BeGole EA. Orthodontic treatment - Improvement and standards using the peer assessment rating index. Angle Orthod. 2006;76:260-264        [ Links ]

5. Stonehouse-Smith D, Rahman A, Mooney J, Bellardie H. Occlusal outcome of orthodontic treatment for patients with complete cleft lip and palate. Cleft Palate Craniofac J. 2022;59:79-85        [ Links ]

6. Deguchi T, Honjo T, Fukunaga T, Miyawaki S, Roberts WE, Takano-Yamamoto T Clinical assessment of orthodontic outcomes with the peer assessment rating, discrepancy index, objective grading system, and comprehensive clinical assessment. Am J Orthod Dentofacial Orthop. 2005;127:434-443        [ Links ]

7. Bellardie H. Letter to the editor: Peer Assessment Rating (PAR) scoring of cleft patients treated within a regional cleft centre in the United Kingdom. J Orthod. 2022; 49:368        [ Links ]

8. Turbill E, Richmond S, Wright J. A closer look at General Dental Service orthodontics in England and Wales 1: factors influencing effectiveness. Br Dent J. 1999;187: 211-216        [ Links ]

9. Richmond S, Shaw WC, Roberts CT, Andrews M. The PAR index (peer assessment rating): Methods to determine outcome of orthodontic treatment in terms of improvement and standards. Eur J Orthod. 1992b;14:180-187        [ Links ]

10. Kerr W, Buchanan I, McColl J. Use of the PAR Index in assessing the effectiveness of removable orthodontic appliances. Br J Orthod. 1993;20:351-357        [ Links ]

11. O'Brien K, Shaw W, Roberts C. The use of occlusion indices in assessing the provision of orthodontic treatment by the hospital orthodontic service of England and Wales. Br J Orthod. 1993;15:7-15        [ Links ]

12. Richmond S, Andrews M. Orthodontic treatment standards in Norway. Eur J Orthod. 1993;15:7-15        [ Links ]

13. Dyken RA, Sadowsky PL, Hurst D. Orthodontic Outcomes Assessment Using the Peer Assessment Rating Index. Angle Orthod. 2001;71:164-169        [ Links ]

14. Richmond S. Personal audit in orthodontics. Br J Orthod. 1993;20:135-145        [ Links ]

15. Buchanan I, Russell J, Clark J. Practical Application of the P.A.R. Index: An Illustrative Comparison of the Outcome of Treatment Using Two Fixed Appliance Techniques. British J Orthod. 1996;23:351-357        [ Links ]

 

 

Correspondence:
Name: Dr. JC Julyan
Tel: 021 975 7478 Cell: 074 136 3505
E-mail: jcjulyan@gmail.com