GUEST EDITORIAL

 

How good are our guidelines? Four years of experience with the SAMJ's AGREE II review of submitted clinical practice guidelines

 

 

The South African Medical Journal (SAMJ) is an established source of clinical practice guidelines (CPGs) serving the local healthcare community. CPGs link professional societies and clinicians by guiding best practice through the collation and interpretation of the best available evidence. Not only are CPGs important in standardising the quality of patient care, but they also assist with medicine selection and resource allocation decisions, adjudicating medicolegal claims, and promoting equity by influencing medicine access and health system organisation.^[1]

In 2014, the SAMJ appointed an editorial subcommittee to review CPGs submitted for publication.^[2] This was in response to several concerns, including the global shift in CPG quality requirements, the potential effect of poor-quality CPGs on healthcare quality and cost, and the challenges South African (SA) CPG developers face in meeting new standards. This editorial reflects on the SAMJ CPG review subcommittee's experience over the past 4 years and describes the value of more robust CPG development.

 

Evolution of CPG development

Early CPGs were essentially narrative reviews linked to recommendations of indeterminate validity. Over the pastfew decades, CPG development has become more rigorous by transparently considering evidence of efficacy and safety alongside factors such as cost, patients' values and context.^[3-5] The institute of Medicine's concept of a CPG has also evolved since it was first published in 1990 to the current version: 'statements that include recommendations intended to optimise patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options'.^[3]

The World Health Organization (WHO) is a recognised source of CPGs. However, in 2007, an audit of the WHO's CPG development processes uncovered several shortcomings. M Methods for development were not transparent and guidance was biased towards expert opinion rather than reflecting published evidence.^[6] This audit led to new CPG standards across all WHO programmes, as well as the introduction of a guidelines review committee. The committee assesses quality and approves CPGs at several stages of development, from planning to final dissemination, and requires the use of a systematic approach to grade the overall quality of evidence (Grading of Recommendations Assessment, Development and Evaluation (GRADE)).^[7] As a result a repeat audit in 2010 found improvements in the quality and credibility of WHO CPGs.^[8] The framework used for the audit was the Appraisal of Guidelines for Research and Evaluation (AGREE II) tool, a validated, peer-reviewed method for evaluating the quality of development and reporting of CPGs!'] and the same tool adopted by the SAMJ in its CPG editorial review process.^[2]

In SA, interviews with national primary care CPG stakeholders indicate that there is a perception that CPG processes are progressing and transitioning for the better; however, many challenges remain.^[10] These include transparent use of evidence, management of potential conflicts of interest, stakeholder consultation and co-ordination across guideline groups and sectors, and processes for adaptation to local context. I"! The underlying reasons include financial constraints and limited technical capacity.^[10,11] These issues highlight potential vulnerabilities in local CPG development, recognised by local developers, further substantiating the SAMJ decision to implement its CPG review process and the adoption of the AGREE II tool.

 

AGREE II as a tool to audit guideline quality and vulnerabilities

The AGREE II tool includes six domains of CPG quality:^[9]

• Domain 1. Scope and purpose 'is concerned with the overall aim of the CPG, the specific health questions, and the target patient population'

• Domain 2. StakrlinliW involvement 'focuses on the extent to which the CPG was developed by the appropriate stakeholders and represents the views of its intended users'

• Domain 3. Rigour of development 'relates to the process used to gather and synthesise the evidence, the methods to formulate the recommendations, and to update them'

• Domain 4. Clarity of presentation 'deals with the language, structure, and format of the CÍG'.

• Domain 5. Applicability 'pertains to the likely barriers and facilitators to implementation, strategies to improve uptake, and resource implications of applying the CPG.

• Domain 6. Editorial independence 'is concerned with the formulation of recommendations not being unduly biased by competing interests'

Of these domains, the editorial subcommittee considers two 'non-negotiable' - that is, two domains that, if not addressed, are likely to undermine CPG validity by the presentation of potentially inaccurate or biased recommendations. These are domain 3 (rigour of development) and domain 6 (editorial independence and CPG funding).

Several studies systematically evaluating the quality of CPGs from SA have consistently found moderate to poor reporting, particularly in these domains.^[12,14] Given the major concern that poor-quality CPGs may result in inefficient use of limited resources or potential harm, and the global progress in standards for CPG reporting, these issues cannot be overlooked

 

Recognising and addressing challenges to credible guidelines

A study comparing two iterations of the American College of Chest Physicians Antithrombotic Guidelines found that when interests are present, recommendations are likely to be swayed in biased ways.^[15] In CPG development, some CPG panels may aim to avoid all conflicts of interest; however, more often steps are taken to ensure that the chair of a CPG process has no potential interests in the CPG and that the rest of the committee's interests (be they financial or non-financial) are actively managed.^[16] This includes recusal from decisions involving interventions that they may favour by virtue of research done or funding received

Reviewing the CPG submissions since the editorial subcommittee's inception, the overwhelming observation has been that several of the AGREE II domains are poorly reported. Of the seven CPGs reviewed, the domains describing the 'scope and purpose' and clarity and presentation attained good average scores of 70%. However, domains describing the rigour of the process or methods, or editorial independence from funding sources, or suggesting considerations for applicability or implementation, all scored below 20% (Fig. 1). Of the CPGs submitted, only a minority addressed peer review and were subsequently published in the SAMJ, with the others opting for alternative publication options.

The editorial subcommittee also found that many local CPGs are adapted from CPGs developed in other countries, or from multilateral organisations such as the WHO. Adapting CPGs is a well-recognised and expedient approach to CPG development, and reduces cost and time if new systematic reviews are not required. This is an appropriate approach for countries such as SA with limited CPG resources. However, adapting a CPG to a local setting does not obviate the need for transparent and explicit methods, including the management of interests.^[17-18] Specifically, rather than wholesale and uncritical adoption of a CPG, adaptation requires a concerted and documented process considering locally relevant issues affecting anticipated health benefits and risks, as well as costs and feasibility. Where possible, SA CPGs should also note any major differences between their recommendations and those stated in already existing standard treatment CPGs or policies in the public sector. Convergence between private and public sector CPGs is an important component of SAs trajectory towards universal health coverage.

 

Expert opinion and gufdelinedevelopment

Expert opinion is at the core of the CPG development process. Experts play a key role in defining priorities, leading CPG development and supporting implementation. They should also lead the consideration of the balance of benefits and harms in making recommendations, including review of evidence about efficacy, harm, costs, stakeholders' values/ preferences, and context.^[19,20] Many tools are available to assist in the development of rigorous and transparent CPGs, including the Guidelines 2.0 checklist, a project that systematically compiled all potential steps from 'Guidelines for guidelines' manuals.^[5] Given that few groups have the time, technical expertise or funds to follow every step, there are simpler guides. For example, the AGREE II tool can be used as a checklist for key quality features.^[9] There is also an online CPG tool developed with SA CPG developers in mind that describes approaches for identifying, appraising and adapting CPGs (https://guidelinetoolkit.org.za/).

 

Guideline users: Recommendations and reviews v. CPGs

The users of CPGs should be mindful of the difference between articles that present recommendations or reviews of clinical practice, which often simply reflect the viewpoint of an interested party or group of specialists, and a CPG that has undergone the rigorous development and evaluation process required to meet current standards.

 

Conclusions

Meeting international standards for local CPG development is challenging, but basic issues such as evidence scrutiny and understanding of potential conflicts of interest are important. The CPG editorial subcommittee aims to assist the SAMJ in ensuring that clinicians and readers receive credible, trust-worthy CPGs to inform SA clinical care. There is a further undertaking to support CPG developers by sharing best practice and tools needed for the construction and publication of sound, locally relevant CPGs. The subcommittee looks forward to ongoing collaboration with stakeholders as we collectively seek to improve the delivery of healthcare in SA.

Author contributions. All authors conceptualised the editorial. TK drafted the editorial, and all authors reviewed and provided feedback. All authors approved the final version.

Funding. None.

Conflicts of interest. All members of the guidelines editorial subcommittee are involved with clinical guideline development, implementation and/or training. No members have conflicts to declare in respect of this editorial.

 

Tamara Kredo

Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa; and Division of Clinical Pharmacology, Department of Medicine, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
tamara.kredo@mrc.ac.za

Roger Wiseman

Liberty Health Holdings, Cape Town, South Africa

Andy Gray

Division of Pharmacology, Discipline of Pharmaceutical Sciences, School of Health Sciences, University of KwaZulu-Natal, Durban, South Africa

Andy Parrish

Department of Internal Medicine, Faculty of Health Sciences, Walter Sisulu University, Mthatha, Eastern Cape, South Africa; and Cecilia Makiwane Hospital, East London, Eastern Cape, South Africa

Jacqui Miot

Health Economics and Epidemiology Research Office, Department of Internal Medicine, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa

Karen Cohen

Division of Clinical Pharmacology, Department of Medicine, Faculty of Health Sciences, University of Cape Town, South Africa

 

References

1. Kredo T, Bernhardsson S, Machlngaldze S, et al Guide to clinical practice guidelines. The current state of play. Int J Qual Health Care 2016-,28(1).122-128. https://doi.org/10.1093/intqhc/nizv115        [ Links ]

2. Wiseman R Cohen K, Gray A, et al. AGREE to disagree: Critical appraisal and the publication of practice guidelines. S Afr Med J 2014-,104(5).345-346. https://doi.org/10.7196/SAMJ.8215        [ Links ]

3. Graham R Mancher M, Wolman DM, Greenfield S, Steinberg E, eds. Clinical Practice Guidelines We Can Trust. Washington, DC. National Academies Press, 2011.         [ Links ]

4. Qaseem A, Forland F, Macbeth F, et al Guidelines International Network Toward International standards for clinical practice guidelines. Ann Intern Med 2012,156(7).525-531. https://doi.org/10.7326/0003-4819-156-7-201204030-00009        [ Links ]

5. Schünemann HJ, Wiercioch W Etxeandia I, et al Guidelines 2.0. Systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ 2014,186(3).E123-E142. https://doi.org/10.1503/cmaj.131237        [ Links ]

6. Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet 2007:369(9576):1883-1889. https://doi.org/10.1016/S0140-6736(07)60675-8        [ Links ]

7. World Health Organization. WHO handbook for guideline development 2014. http.//www.who.lnt/publications/guidellnes/handbook_2nd_ed.pdf?ua(accessed 15 July 2018).         [ Links ]

8. Sinclair D, Isba R Kredo T, Zani B, Smith H, Garner P. World Health Organization guideline development. An evaluation. PLoS One 2013,8(5):e63715. https://doi.org/10.1371/journal.pone.0063715        [ Links ]

9. Brouwers MC, Kho ME, Browman GP, et al. AGREE II. Advancing guideline development, reporting and evaluation in health care. CMAJ 2010;182(18):E839-E842. https://doi.org/10.1503/cmaj.090449        [ Links ]

10. Kredo T, Abrams A, Young T, Louw Q, Volmlnk J, Daniels K. Primary care clinical practice guidelines in South Africa. Qualitative study exploring perspectives of national stakeholders. BMC Health Serv Res 2017,17(1).1-12. https://doi.org/10.1186/sl2913-017-2546-z        [ Links ]

11. Kredo T, Cooper S, Abrams A, Daniels K, Volmlnk J, Atkins S. National stakeholders' perceptions of the processes that inform the development of national clinical practice guidelines for primary healthcare In South Africa. Health Res Policy Syst 2018,16(1).1-14        [ Links ]

12. Kredo T, Gerritsen A, Van Heerden J, Conway S, Siegfried N. Clinical practice guidelines within the Southern African Development Community. A descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases. Health Res Policy Sys 2012,10(1).1-13. https://doi.org/10.1186/1478-4505-10-1        [ Links ]

13. Machlngaldze S, Zani B, Abrams A, et ai Series. Clinical Epidemiology in South Africa. Paper 2. Quality and reporting standards of South African primary care clinical practice guidelines. J Clin Epidemiol 2017,83.31-36. https://doi.org/10.1016/j.jdlnepi.2016.09.015        [ Links ]

14. Wilkinson M, Wilkinson T, Kredo T, et al. South African clinical practice guidelines. A landscape analysis. S Afr Med J 2017,108(1).23-27. https://doi.org/10.7196/SAMJ.2017.v10811.12825        [ Links ]

15. Agoritsas T, Neumann I, Mendoza C, Guyatt GH. Guideline conflict of Interest management and methodology heavily Impacts on the strength of recommendations. Comparison between two iterations of the American College of Chest Physicians Antithrombotic Guidelines. J Clin Epidemiol 2017,81.141-143. https://doi.org/10.1016/j.jclinepi.2016.07.007        [ Links ]

16. Schünemann HJ, Al-Ansary LA, Forland F, et al Guidelines International Network. Principles for disclosure of Interests and management of conflicts In guidelines. Ann Intern Med 2015,163(7).548-553. https://doi.org/10.7326/M14-1885        [ Links ]

17. Dtzon JM, Grimmer K, Louw Q, Kredo T, Young T, Machlngaldze S. South African Guidelines Excellence (SAGE). Adopt, adapt, or contextualise? S Afr Med J 2016,106(12).1177-1178. https://doi.org/10.7196/SAMJ.2016.vl06.H2.11374        [ Links ]

18. Schünemann HJ, Wiercioch W Brozek J, et al GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations. GRADE-ADOLOPMENT. J Clin Epidemiol 2017:81:101-110. https://doi.org/10.1016/j.jcllnepi.2016.09.00S        [ Links ]

19. Andrews J, Guyatt G, Oxman AD, et al. GRADE guidelines. 14. Going from evidence tc recommendations. The significance and presentation of recommendations. J Clin Epidemiol 2012,66(7).719-725. https://doi.org/10.1016/j.jclinepi.2012.03.013        [ Links ]

20. Guyatt GH, Oxman AD, Vist GE, et al. GRADE. An emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008,336(7650).924-926. https://doi.org/10.1136/ bmj.39489.470347.AD        [ Links ]