RESEARCH

 

Predictive scores for critically ill obstetric patients in a resource-limited setting: A retrospective validation of the Obstetric Early Warning Score

 

 

N Kheswa^I; R Rodseth^II; R Wise^III; D G Bishop^IV

^IMB BCh, DA, FCA, MMed; Department of Anaesthesia and Critical Care, School of Clinical Medicine, College of Health Sciences, University of KwaZulu-Natal, Durban, South Africa
^IIMB ChB, DCH, DA, FCA, MMed, Cert Crit Care, MSc, PhD; Perioperative Research Group, Department of Anaesthetics, Critical Care and Pain Management, University of KwaZulu-Natal, Pietermaritzburg, South Africa; and Consistency of Care Division, Netcare Ltd, Johannesburg, South Africa
^IIIMB ChB, DipObst, DipPEC, FCA, MMed, Cert Crit Care; Faculty Medicine and Pharmacy, Vrije Universiteit Brussel, Brussels, Belgium; Discipline of Anesthesiology and Critical Care, School of Clinical Medicine, University of KwaZulu-Natal, Durban, South Africa; and Adult Intensive Care, John Radcliffe Hospital, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
^IVMB ChB, DA, FCA, PhD; Perioperative Research Group, Department of Anaesthetics, Critical Care and Pain Management, University of KwaZulu-Natal, Pietermaritzburg, South Africa

Correspondence

 

 

---

ABSTRACT

BACKGROUND. In South Africa (SA), an unacceptably high institutional maternal mortality rate persists due to failure to recognise critically ill patients. Early warning systems could assist in identifying these patients sooner.
OBJECTIVES. We evaluated the Obstetric Early Warning Score (OEWS) as a predictor of maternal outcomes in an intensive care unit (ICU) and compared its prognostic validity with the Acute Physiology and Chronic Health Evaluation (APACHE) II and the quick Sequential Organ Failure Assessment score (qSOFA).
METHODS. Data were extracted on pregnant and post-partum ICU-admitted patients at a tertiary and regional centre in SA between October 2015 and April 2020. Clinical characteristics and outcomes were used to compare the three scoring systems.
RESULTS. Among 251 eligible patients, the mortality rate was 8.5%. The OEWS score failed to differentiate between survivors and non-survivors (odds ratio 1.13, 95% confidence intervals (CI) 0.972 - 1.311, p=0.113). APACHE II outperformed the OEWS (area under the receiver operating characteristics curve (AUROC) 0.69, 95% CI 0.540 - 0.846 v. 0.55, 95% CI 0.430 - 0.674). The OEWS (AUROC 0.55, 95% CI 0.430 - 0.674) and qSOFA (AUROC 0.60, 95% CI 0.500 - 0.703) showed no differences. Further analysis revealed that positive scoring for diastolic blood pressure and high systolic blood pressure weakened OEWS performance. Removing these variables improved OEWS prediction (AUROC 0.68).
CONCLUSION. In a SA obstetric population, OEWS did not predict mortality in ICU-admitted patients and offered no advantages over APACHE II or qSOFA scores. Further research should identify critical outcome predictors for low-resource populations.

---

 

 

The institutional maternal mortality rate (iMMR) for preventable deaths in South Africa (SA) remains unacceptably high (70 per 100 000 live births).^[1] The 2017 Saving Mothers: Annual Report on Confidential Enquiries into Maternal Deaths in South Africa identified clinicians' failure to recognise critically ill patients as an avoidable factor contributing to maternal mortality.^[2] Risk prediction scoring systems are appealing as objective measures to detect these patients, but traditional intensive care unit (ICU) models like the Acute Physiology and Chronic Health Evaluation (APACHE), Simplified Acute Physiology Score (SAPS), Sequential Organ Failure Assessment score (SOFA) and Mortality Probability Model (MPM), tend to overestimate mortality in obstetric patients.^[3-5] Early warning systems (EWS) are simple-to-use bedside algorithms that aid in the early recognition of patients at risk of deterioration or critical illness, or those who may benefit from early intervention.^[6] They offer a potential solution to this problem. Carle et al.^[7] described and validated the Obstetric Early Warning Score (OEWS) (Table 1), in response to the 2003 - 2005 Report on Confidential Enquiries into Maternal Deaths in the UK, which recommended the use of EWS in obstetric patients to identify patients at risk of deterioration.^[7,8]

Initial OEWS validation studies showed excellent discrimination between survivors and non-survivors among obstetric patients admitted to an ICU in the UK, with an area under the receiver operating characteristic (AUROC) curve of 95.7%.^[7] In a 2017 South American retrospective study, OEWS showed good survival prediction in patients admitted to the ICU where the cause of admission was directly related to pregnancy or the puerperium (AUROC 87%).^[4]

The primary outcome of this study was to evaluate the prognostic validity of OEWS as a predictor of mortality in obstetric patients admitted to the ICU. We further aimed to compare the efficacy of the OEWS with the APACHE II and the Quick Sequential Organ Failure Assessment (qSOFA) in predicting mortality among ICU-admitted obstetric patients.

 

Methods

We conducted a retrospective, observational study including all pregnant and post partum women (<42 days post partum), admitted to Greys Hospital and Harry Gwala Regional Hospital (HGRH) ICUs. Both hospitals form part of the Pietermaritzburg Metropolitan Hospital Complex, located in Pietermaritzburg, KwaZulu-Natal, South Africa. Greys Hospital offers tertiary-level services and has a 13-bed intensivist-led ICU. HGRH offers regional-level hospital services and has a 6-bed ICU and 3-bed high care led by specialist anaesthesiologists. We included patients admitted between 21 October 2015 and 30 April 2020.

The University of KwaZulu-Natal (UKZN) Biomedical Research Ethics Committee granted ethics approval (BREC/00001972/2020) with a waiver ofconsent. Approval was obtained from the institutions and the KwaZulu-Natal Provincial Department of Health (KZ_202011_015).

Data were extracted on all obstetric patients found on the Integrated Critical Care Electronic Database (ICED) database,^[9] a UKZN BREC-approved database. All patients admitted to Greys Hospital and HGRH ICUs are entered into the ICED database as part of the normal workflow. Patient data are entered into the ICED database at four discrete time points during the ICU admission. The first data entry is at the time of referral to the ICU, followed by the time of admission to the ICU, then at the end of the first 24 hours after ICU admission and finally at the time of discharge. The APACHE II score is calculated using the variables at the end of the first 24 hours. Patients were excluded if primary outcome data were missing. An OEWS for every patient enrolled was determined from the data collected at admission. In determining the OEWS, the level of consciousness was defined as 'alert' (Glasgow Coma Scale [GCS] 15/15) or 'not alert' (GCS <15). For each patient, we collected the following information: age, the reason for admission, diagnosis, information required to calculate the OEWS score (i.e., systolic blood pressure, diastolic blood pressure, heart rate, respiratory rate, temperature, fraction of inspired oxygen and level of consciousness), comorbidities, gestational age, days post-delivery, duration of ICU admission, patient outcomes (deceased or alive at discharge) and the APACHE II score.

Data analysis

Data analysis was conducted using STATA version 15.1 (StataCorp, USA). In-ICU mortality was defined as death due to any cause while admitted to the ICU. ICU length of stay was defined as the number of calendar days for which a patient was admitted to the ICU. Gestational age was defined as weeks post conception. For patients admitted to the ICU after delivery or after the termination of pregnancy, the gestational age at the time of delivery or termination of pregnancy was recorded.

Data were reported as mean (standard deviation [SD]) for continuous normally distributed variables or median (range) for non-normally distributed variables. Categorical variables were described by counts and percentage frequencies. Comparisons between normally distributed data were performed using Student's t-test and the Wilcoxon Mann-Whitney (7-test was used for non-normally distributed data. Categorical data were analysed using the χ² test. We used Fisher's exact test where an expected cell count in the cross-tabulation was less than five. The Shapiro-Wilk test was used for normality testing.

To measure the efficacy of OEWS as a predictor of mortality in obstetric patients admitted to the ICU, we first determined the proportion of patients who died in the ICU, together with the associated 95% confidence intervals (CI). We then performed a univariate analysis of OEWS variables for mortality. We conducted a logistic regression analysis and calculated the AUROC to determine the efficacy of the OEWS in predicting mortality. We performed individual receiver operating characteristic curves (ROC) for each of the OEWS scoring variables to determine their contribution to mortality prediction. Similar approaches were used for the APACHE II and qSOFA scores of each patient. We compared the AUROC of the three scores (OEWS, APACHE II and qSOFA) to assess their performance in predicting mortality in obstetric patients admitted to ICU. This comparison aimed to assess the binary dependent variable (mortality) with the continuous independent ordinal variable (OEWS) in each category. This was followed by an assessment of model discrimination (c-statistic) and calibration. Binary logistic regression was used to describe the patient variables as risk factors for mortality. A p-value of <0.05 defined statistical significance for all analyses.

We used the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement guidelines to report our study. The completed STROBE checklist is attached as Appendix 1.

 

Results

The patient flow diagram is shown in Fig. 1.

 

 

Patient characteristics are shown in Table 2.

The unit mortality of all admissions during the study period (including obstetric and non-obstetric admissions) was 815/5 279 (15.4%). The mortality rate of patients included in this study was 8.4% (21/251).

Univariable analysis of OEWS variables showed that low systolic blood pressure (SBP) and high fraction of inspired oxygen (FiO2) were significant discriminators between survivors and non-survivors (Table 3).

 

 

The comparison between the OEWS (AUROC 0.55, 95% CI 0.430 - 0.674) and APACHE II (AUROC 0.69, 95% CI 0.540 - 0.846) favoured the APACHE II. There was no difference between the OEWS (AUROC 0.55, 95% CI 0.430 - 0.674) and qSOFA (AUROC 0.60, 95% CI 0.500 - 0.703) (Table 4).

The individual ROCs for the variables used to calculate OEWS revealed that the positive scoring diastolic blood pressure (DBP) (AUROC 0.39; 95% CI 0.256 - 0.531) and high SBP (AUROC 0.33, 95% CI 0.206 - 0.450) weakened the performance of the OEWS in our population. The DBP was also statistically performed poorly in our population, possibly due to differences in population characteristics. In the study by Khergade et al..^[10] the OEWS' performance was equivalent to that of the APACHE II and SOFA but not superior.^[10]

Our study mortality rate of 8.4% was markedly higher than the 2.8% reported insignificant in the univariable analysis. We then determined the ROC for a new OEWS, which excluded DBP, the positive scoring of high SBP and included a low SBP variable (SBP <90 mmHg). This score performed better than the original OEWS (AUROC 0.68, 95% CI 0.563 - 0.794) (Fig. 2).

 

Discussion

Our study found that the OEWS does not perform well as a predictor of mortality in obstetric patients admitted to two SA ICUs.

Further, it was not superior to qSOFA or APACHE II. The OEWS was developed in the UK and showed excellent discrimination between survivors and non-survivors (AUROC 0.957).^[7] Paternina-Caicedo et al..^[4] in Colombia and Khergade et al..^[10] in India have since also evaluated the OEWS performance in mortality prediction and reported favourable results of the performance of the OEWS (AUROC 0.84 and 0.89, respectively).^[4,10] The OEWS in the original study by Carle et al..^[7] and different from other studies evaluating the OEWS (4.1% in Paternina-Caicedo et al..^[4] and 26% in Khergade et al..^[10]). Further, the median APACHE II scores of 25 for survivors and 32 for non-survivors in our study were significantly higher than the median scores (10 for survivors and 17 for non-survivors) reported by Carle et al..^[7] in their internal validation group for the OEWS. These findings suggest that our patients present with more severe illness, which may account for the differences in score performances. While our mortality rate was lower than the 11.1% mortality rate reported in a similar study in Johannesburg, SA, we looked at all obstetric admissions rather than just those associated with sepsis.^[11] Due to challenges with access to healthcare services, patients in resource-limited countries may present to the hospital later with more severe illnesses.^[6,12] This partially explains the higher mortality rate in our units than in the studies by Carle et al..^[7] and Paternina-Caicedo et al..^[4] The biochemical and chronic health parameters used in addition to physiological parameters to determine the APACHE II score may contribute to its superior performance compared with the OEWS, which only uses physiological parameters, in our population. However, this has not been consistent with other studies evaluating the OEWS.^[4,710] The performance of the APACHE II in our setting requires further investigation.

Causes of mortality seem to influence the performance of the OEWS, as shown in the study by Paternina-Caicedo et al.. ^[4] In their study, with a reported mortality of 4.1%, the authors found that OEWS performed worse (AUROC 0.77) when the cause of mortality was indirectly related to pregnancy or the puerperium rather than directly related causes (AUROC 0.87).^[4] Our study did not attempt to differentiate between the causes of mortality. In the study by Khegade et al..^[10] with a mortality rate of 26%, the OEWS did not outperform APACHE II. This suggests that in populations with higher mortality rates, the OEWS and APACHE II may have similar performance.

Our univariate analysis of OEWS variables shows that DBP and high SBP were not associated with mortality. These findings are akin to those of Carle et al..^[7] reported while developing the OEWS. The authors initially developed a statistical model of the OEWS that excluded DBP and zero-weighted the contribution of elevated SBP. However, these parameters were included in the final version of the OEWS as clinicians expected their inclusion in an obstetric EWS score, aiming to aid in the diagnosis of hypertensive diseases during pregnancy.^[7] The decision to include positive scoring for high SBP and scoring for DBP in the OEWS seems to be the reason this score did not perform well in our setting. Our explorative modification of the OEWS excluded DBP and scoring for high SBP, but included low SBP scoring, which improved the performance of the OEWS (Fig. 2). Umar et al..^[13] developed an EWS for obstetric patients in a low-resource environment and found that DBP was collinear with SBP (R² 0.9) in predicting severe maternal outcomes. They thus excluded DBP from their EWS.^[13] Aoyoma et aL^[14] recommend using the Collaborative Integrated Pregnancy High-dependency Estimate of Risk (CIPHER) model and Maternal Severity Index rather than the OEWS for prospective studies and clinical trials and quality improvement in critically ill pregnant and post partum women. They argued that these scores demonstrated a low risk of bias in studies in which they were developed and validated.^[14] However, these models require laboratory data, which may not be available at the bedside, particularly in a resource-limited setting.

The limitations of our study include the retrospective design, which led to missing data and reduced the number of study participants, particularly for our secondary outcomes. However, the data collected was entered contemporaneously as part of normal workflow, improving reliability. Interventions preceding ICU admission and those performed in the first 24 hours following ICU admission are not routinely entered in the ICED database, precluding an analysis of their impact on the results of our study. Our study examined mortality as the primary outcome and did not consider morbidity. Assessing the ability of the OEWS to predict morbidity may make it a more clinically applicable score. Our study is the first to evaluate the performance of the OEWS in Africa and, while the OEWS under-performed in our setting, our exploratory analysis of modifications required to improve the performance of OEWS may inform future prospective studies exploring EWS in an obstetric population. Our study was not designed to develop a new scoring system, and the exploratory analysis discussed should provide a theoretical basis for future research, rather than clinical application.

The OEWS was created and validated in the critical care setting, but it was intended for clinical use in ward patients.^[7] Our study followed a similar methodology to Carle et al..'s ^[7] original work on the OEWS and the subsequent study by Paternina-Caicedo et al..^[4] A flaw in the methodology is that it does not evaluate the OEWS validity as an EWS. As such, both the OEWS and our modification of it should be evaluated in non-critically ill patients to assess their utility as EWS, if this model is to fulfil its intended use. Our modification of the OEWS suggests that prospective studies should investigate alternate clinical parameters. Future research in resource-limited settings should also take into consideration Umar's EWS which was developed in and for a low-resource setting.^[13] Further, while clinical risk scores are commonly used to develop standardised risk scores to allow comparison between different patient populations, they are also able to identify high-risk patients who may benefit from changes in management like additional monitoring modalities, increased monitoring frequency or more aggressive therapeutic interventions. While not explored in this study, future studies should explore the utility of these scores in directing or changing patient management.

 

Conclusion

In a low-to-middle-income African country, the OEWS is an unreliable predictor of mortality in obstetric patients admitted to the ICU and offered no advantages over traditional scores like the APACHE II and SOFA. Modifications to the OEWS are required to improve its performance in this clinical context.

Declaration. This study was conducted as part of the requirements for MMED degree completion for NK.

Acknowledgements. None.

Author contributions. NK contributed to the conceptualisation, design, and draft of the manuscript. RR contributed to the conceptualisation, design, statistical analysis and manuscript revisions. RW contributed to the conceptualisation, design and manuscript revisions. DB contributed to the conceptualisation, design, primary statistical analysis and interpretation of data and manuscript revisions. All authors approved the final version for publication.

Funding. None.

Conflicts of interest. None.

 

References

1. Moodley J, Fawcus S, Pattinson R. 21 years of confidential enquiries into maternal deaths in South Africa: reflections on maternal death assessments. Obstet Gynaecol Forum 2020;30(4):4-7. https://hdl.handle.net/10520/ejc-medog-v30-n4-a2        [ Links ]

2. Saving Mothers 2017: Annual Report on Confidential Inquiries into Maternal Death in South Africa. 2018. https://www.westerncape.gov.za/assets/departments/health/saving_mothers_annual_report_2017.pdf        [ Links ]

3. Oliveira-Neto AF, Parpinelli MA, Costa ML, et al.. Prediction of severe maternal outcome among pregnant and puerperal women in obstetric ICU. Crit Care Med 2019;47(2):e136-e43.         [ Links ]

4. Paternina-Caicedo A, Miranda J, Bourjeily G, et al.. Performance of the Obstetric Early Warning Score in critically ill patients for the prediction of maternal death. Am J Obstet Gynecol 2017;216(1):58. e1-e8. https://doi.org/10.1016/j.ajog.2016.09.103        [ Links ]

5. Rojas-Suarez J, Paternina-Caicedo AJ, Miranda J, Mendoza R, Dueñas-Castel C, Bourjeily G. Comparison of severity-of-illness scores in critically ill obstetric patients: A 6-year retrospective cohort*. Crit Care Med 2014;42(5):1047-54.         [ Links ]

6. Vasco M, Pandya S, Van Dyk D, Bishop DG, Wise R, Dyer RA. Maternal critical care in resource-limited settings. Narrative review. Int J Obstet Anesthesia 2019;37:86-95. https://doi.org/10.1016/j.ijoa.2018.09.010        [ Links ]

7. Carle C, Alexander P, Columb M, Johal J. Design and internal validation of an obstetric early warning score: Secondary analysis of the Intensive Care National Audit and Research Centre Case Mix Programme database. Anaesthesia 2013;68(4):354-367. https://doi.org/10.1111/anae.12180        [ Links ]

8. Lewis G. The confidential enquiry into maternal and child health (CEMACH). Saving mothers' lives: Reviewing maternal deaths to make motherhood safer 2003-2005. The seventh report of the confidential enquiries into maternal deaths in the United Kingdom. London: CEMACH; 2007.         [ Links ]

9. Allorto NL, Wise RD. Development and evaluation of an integrated electronic data management system in a South African metropolitan critical care service. S Afr J Anaesth Analg 2015;21:31-35. http://dx.doi.org/10.1080/22201181.2015.11        [ Links ]

10. Khergade M, Suri J, Bharti R, Pandey D, Bachani S, Mittal P. Obstetric Early Warning Score for prognostication of critically ill obstetric patient. Indian J Crit Care Med 2020;24(6):398-403.         [ Links ]

11. Lafon J, Buga E, Nethathe G. Characteristics and outcomes of obstetric patients with maternal sepsis requiring admission to a South African intensive care unit: A retrospective review. S Afr J Obstet Gynaecol 2021;26(3). https://hdl.handle.net/10520/ejc-m_sajog-v26-n3-a2        [ Links ]

12. Stafford RE, Morrison CA, Godfrey G, Mahalu W. Challenges to the provision of emergency services and critical care in resource-constrained settings. Glob Heart 2014;9(3):319-323. https://doi.org/10.1016/j.gheart.2014.08.005        [ Links ]

13. Umar A, Manu A, Mathai M, Ameh C. Development and validation of an obstetric early warning system model for use in low resource settings. BMC Pregnancy Childb 2020;20(1):531. https://doi.org/10.1186/s12884-020-03215-0        [ Links ]

14. Aoyama K, D'Souza R, Pinto R, et al.. Risk prediction models for maternal mortality: A systematic review and meta-analysis. PLOS ONE 2018;13(12):e0208563. https://doi.org/10.1371/journal.pone.0208563        [ Links ]

 

 

Correspondence:
N Kheswa
ndumisokheswa@live.co.za

Accepted 2 April 2024.