RESEARCH LETTER

 

Does the DHET research output subsidy model penalise high-citation publication? A case study

 

 

Yolande X. Harley; Esmari Huysamen; Carlette Hlungwani; Tania Douglas

Faculty of Health Sciences, University of Cape Town, Cape Town, South Africa

Correspondence

 

 

---

ABSTRACT

South African universities are awarded annual subsidy from the Department of Higher Education and Training (DHET) based on their research publication output. Journal article subsidy is based on the number of research publications in DHET-approved journals as well as the proportional contribution of authors from the university. Co-authorship with other institutions reduces the subsidy received by a university, which may be a disincentive to collaboration. Inter-institutional collaboration may affect the scientific impact of resulting publications, as indicated by the number of citations received. We analysed 812 journal articles published in 2011 by authors from the University of Cape Town's Faculty of Health Sciences to determine if there was a significant relationship between subsidy units received and (1) citation count and (2) field-weighted citation impact. We found that subsidy units had a significant inverse relationship with both citation count (r= -0.247; CI = -0.311 - -0.182; p<0.0001) and field-weighted citation impact (r= -0.192; CI= -0.258 - -0.125; p <0.0001). These findings suggest that the annual subsidy awarded to universities for research output may inadvertently penalise high-citation publication. Revision of the funding model to address this possibility would better align DHET funding allocation with the strategic plans of the South African Department of Science and Technology, the National Research Foundation and the South African Medical Research Council, and may better support publication of greater impact research.

Keywords: higher education subsidy; collaboration; impact; citation; scientometrics

---

 

 

Introduction

South African universities are awarded annual subsidy from the Department of Higher Education and Training (DHET) based on research publication output - a significant proportion of which is composed of journal article publications. The journal article subsidy is based on the number of journal output units generated by the university, calculated from the number of research publications in DHET-approved journals and the proportional contribution of authors from the university. This subsidy provides financial incentive to increase research output.

Given that the DHET subsidy rewards and intends to stimulate research for the benefit of the country, the manner of awarding subsidy should align with strategies to maintain or improve the impact of South Africa's research. Ideally, university researchers would publish high-quality research that makes an impact in the scientific field, and, where appropriate, work collaboratively with other groups to add value to studies and aid further development and translation of the research.

The goals of South Africa's National Research Foundation (NRF) Strategic Plan¹ incorporate not only research output, but also 'citation intensity', emphasising the importance of the impact of the country's research (not only the volume). The NRF system of rating researchers is also based primarily on the quality and impact of their outputs.² The South African Medical Research Council (SAMRC) Strategic Plan 2014/15-2018/19³ highlights the need to publish in high-impact journals, and includes the number of articles published in the 'top four' journals (New Engl J Med, Lancet, Science and Nature) as an indicator towards meeting its objectives. The SAMRC also encourages its scientists to work collaboratively, as 'no single group can respond alone to the priorities'³. This sentiment is echoed by the NRF Strategic Plan¹, which advocates for 'promoting and enhancing international networks and partnerships', as well as by the South African Department of Science and Technology's (DST's) Ten-Year Innovation Plan⁴, which states that greater networking and collaboration (domestic and international) is needed for the country's biotechnology industry to grow.

However, the current subsidy model does not factor in research quality or impact (other than specifying that journals must be DHET accredited). In addition, the greater the co-authorship with other institutions (domestic or international), the lower the subsidy received by a university. This consequence may result in a disincentive to collaboration. It has been argued that the current system may lead to 'non-virtuous practices in research'⁵, such as writing short 'salami-sliced' papers, targeting low-tier journals with high acceptance rates, and avoiding collaboration^5,6 to increase subsidy. Cautioning that the drive to increase research volume had come at the expense of the pursuit of excellence, the 2014 Report of the Ministerial Committee for the Review of the Funding of Universities argued it was time to change the funding framework.⁵ Yet, the new Research Outputs Policy published in March 2015⁷ did not make any changes to the journal article subsidy formula to address these concerns.

Inter-institutional collaboration may affect the scope and quality of research as well as the impact of the resulting publications in the scientific field, as indicated by the number of citations received. Indeed, it has been shown that research that is more collaborative is associated with higher citation rates.^8,9 The inverse relationship between DHET subsidy units received by an institution for a paper and the proportion of authors from outside that university may therefore lead to greater subsidy being awarded to articles of lower citation impact than those of higher citation impact. We hypothesised that greater subsidy (as a result of fewer 'outside' authors) would in fact be associated with lower citation impact. We analysed a set of journal articles published in 2011 to determine if there was a significant relationship between subsidy units received and (1) the number of citations received (citation count) and (2) the field-weighted citation impact. The latter measure is the ratio of citations received by a publication and the average number of citations received by all other similar publications¹⁰ (i.e. with the same publication year, publication type and discipline), and so takes into account differences in citation patterns across disciplines or publication types.

 

Materials and methods

We analysed journal articles published in 2011 by authors from the University of Cape Town's (UCT's) Faculty of Health Sciences that were audited and approved for subsidy by DHET. For each article, we identified the subsidy units assigned by DHET to UCT and the proportion of non-UCT authors. For each article, we extracted (on 15 December 2015) the citation count (total number of citations received) and the field-weighted citation impact for the period 2011-2015 using data drawn from Scopus¹¹ accessed via SciVal¹². Articles not listed on the Scopus database were excluded from the analysis.

The relationship between subsidy units assigned to articles and their citation count, as well as between subsidy units and the articles' field-weighted citation impact, were examined using a Pearson correlation, with a two-tailed p-value and a 95% confidence interval.

 

Results

Following exclusion of 38 articles not listed on the Scopus database, 812 articles were included in the analysis, with a mean subsidy unit assignment of 0.53±0.35 units (Table 1). Of these, 589 (72.5%) articles had non-UCT co-authorship.

Both citation count and field-weighted citation impact were negatively correlated with subsidy units (Figure 1). While the shared variance was small, the correlation was significant in both cases.

 

 

Discussion

DHET subsidy units assigned to UCT for journal articles published by the Faculty of Health Sciences in 2011 had a significant inverse relationship with both citation count and field-weighted citation impact. This finding implies that subsidy allocation is smaller for articles receiving a greater number of citations than it is for those receiving a lower number of citations, whether in absolute terms (citation count) or when compared with the average number of citations received by similar publications¹⁰ (i.e. with the same publication year, publication type and discipline).

Greater collaboration is associated with greater citation rates and, author affiliation aside, citation count tends to increase with the number of authors (with self-citation likely to play only a minor role).^8,9,13,14 It is therefore not unexpected that lower-subsidy-earning publications, which will have been more collaborative, are more highly cited. The analysis has confirmed our hypothesis. By directly relating subsidy to the proportional contribution of authors from a university, and therefore penalising universities for collaborative research, the annual subsidy awarded by DHET to South African universities for research publication output may also be inadvertently penalising high-citation, 'high-impact' publication.

Given the financial benefit of subsidy unit assignment, the existing model discourages inter-institutional collaboration. This situation seems particularly punitive in the case of international collaboration. Universities only receive subsidy for the proportional contribution of authors from that university, whether the external authors are based at other South African institutions or international ones. The annual research publication output subsidy distributed by DHET is currently valued at about ZAR1.6 billion.^15,16 Such large-scale funding should align with the strategic goals of the government, i.e. should incentivise collaborative research that may be associated with high-impact science.

Collaborative publishing has increased in recent years in African institutions, and Pouris and Ho¹⁷ suggest that the large proportion of inter-institutional articles from South African universities indicates that factors encouraging collaboration outweigh the adverse impact of the funding model. Indeed, we found 72.5% of the publications in our analysis had non-UCT co-authorship. That noted, South African universities have different methods of internally allocating subsidy received, and this internal funding distribution could influence their researchers' publishing behaviour. We also note that the nature of research output in the university sector is differentiated with respect to volume, journals, level of collaboration, citation rates and scientific field. Our findings are representative of health sciences articles from a research-intensive university that does not directly allocate publication subsidy to researchers.

Our findings support the recommendation of the 2014 Report from the Review of the Funding of Universities⁵ that the funding framework be revised. The Report proposed that subsidy units be divided only among South African authors of articles, so the model no longer actively discourages international collaboration. This revision would better align the DHET model with the DST's Innovation Plan⁴, the SAMRC's Strategic Plan³ and the NRF Strategic Plan¹, which encourage collaboration.

DHET research output subsidy is a means of distributing government funding in a way that factors productivity. Research output has been steadily increasing in South Africa over the last decade and the subsidy system is thought to have contributed to this increase.^5,18 However, quantity should not be emphasised at the expense of quality.¹⁹ It is perhaps notable that while South Africa's medical publication output increased during 1996-2010, the number of citations per document declined.²⁰ The 2014 Report from the Review of the Funding of Universities⁵ also recommends that quality and scientific impact of publications be directly factored in the model; preferential weighting of journals with higher impact factors was suggested. The notion of a journal's impact factor being a measure of the quality of papers published in it has been contested, and deficiencies of this measure as a tool for research assessment have been highlighted.²¹ The impact factor may have some utility in the funding framework as an indicator of the quality of the journals in which South African universities publish, especially given research output is aggregated at an institutional level. This is in line with the original intention of the impact factor, namely as a measure of journal (rather than article) quality.²² Revising the subsidy model to include a weighting for research impact would potentially better align it with the NRF Strategic Plan¹ and SAMRC Plan³, which advocate for targeting impact. As bibliometric measures of quality and impact evolve in an attempt to minimise inherent flaws (e.g. through the addition of weighting by field), their utility in directing funding allocation should be evaluated on a regular basis.

In summary, the annual subsidy awarded by DHET to South African universities for research publication output may be inadvertently penal ising high-citation publication. Revision of the funding model to address this effect would better align DHET funding allocation with government strategic plans and may better support publication of greater impact research.

 

Authors' contributions

Y.X.H. was the project leader; YX.H., E.H., C.H. and T.D. were responsible for experimental design; E.H., C.H. and YX.H. collected and compiled the data; T.D. made conceptual contributions; YX.H. performed the analysis; Y.X.H. wrote the manuscript; and E.H., C.H. and TD. gave input on the draft manuscript.

 

References

1. South African National Research Foundation (NRF). NRF vision 2015: Strategic plan of the National Research Foundation. Pretoria: NRF; 2008.         [ Links ]

2. South African National Research Foundation (NRF). Evaluation and rating 2016: Guidelines and instructions for completing the application for evaluation and rating via the NRF Online Submission System. Pretoria: NRF; 2015. Available from: https://www.cput.ac.za/storage/research/research_directorate/funding/apply_nrf_rated_ researcher_18022016.pdf        [ Links ]

3. The South African Medical Research Council (MRC) Strategic Plan for the fiscal years 2014/15-2018/19. South African Medical Research Council; 2014.         [ Links ]

4. South African Department of Science and Technology (DST). Innovation towards a knowledge-based economy: Ten-Year Plan for South Africa (2008-2018). Pretoria: DST; 2007.         [ Links ]

5. Report of the Ministerial Committee for the Review of the Funding of Universities. Pretoria: Department of Higher Education and Training; 2013.         [ Links ]

6. Macleod C. The incentive system for research is bad for scholarship. University World News. 2010 June 06. Available from: http://www.universityworldnews.com/article.php?story=20100605063654105        [ Links ]

7. Research outputs policy. Pretoria: Department of Higher Education and Training; 2015.         [ Links ]

8. Khor KA, Yu LG. Influence of international collaboration on the research impact of young universities. In: Salah AA, Tonta Y, Akdag Salah AA, Sugimoto C, Al U, editors. Proceedings of the 15th International Society of Scientometrics and Informetrics Conference; 2015 June 29 - 04 July; Istanbul, Turkey. Istanbul: Bogaziçi University; 2015. p. 278-279. Available from: http://www.issi2015.org/files/downloads/all-papers/0278.pdf        [ Links ]

9. Sooryamoorthy R. Do types of collaboration change citation? Collaboration and citation patterns of South African science publications. Scientometrics. 2009;81(1):177-193. http://dx.doi.org/10.1007/s11192-009-2126-z        [ Links ]

10. Colledge L. Snowball Metrics recipe book. 2nd ed. Amsterdam: Snowball Metrics program partners; 2014. Available from: http://www.snowballmetrics.com/wp-content/uploads/snowball-recipe-book_HR.pdf        [ Links ]

11. Scopus [database on the Internet]. No date [cited 2015 Dec 15]. Available from: http://www.scopus.com/.         [ Links ]

12. SciVal [database on the Internet]. c2015 [cited 2015 Dec 15]. Available from: http://scival.com/.         [ Links ]

13. Aksnes DW. A macro study of self-citation. Scientometrics. 2003;56(2):235-246. http://dx.doi.org/10.1023/A:1021919228368        [ Links ]

14. Van Raan AFJ. The influence of international collaboration on the impact of research results - Some simple mathematical considerations concerning the role of self-citations. Scientometrics. 1998;42(3):423-428. http://dx.doi.org/10.1007/BF02458380        [ Links ]

15. Ministerial statement on university funding: 2015/16 and 2016/17. Pretoria: Department of Higher Education and Training; 2014.         [ Links ]

16. University performance within block grants: Public report. Pretoria: Department of Higher Education and Training; 2015.         [ Links ]

17. Pouris A, Ho YS. Research emphasis and collaboration in Africa. Scientometrics. 2014;98(3):2169-2184. http://dx.doi.org/10.1007/s11192-013-1156-8        [ Links ]

18. Pouris A. Scientometric research in South Africa and successful policy instruments. Scientometrics. 2012;91(2):317-325. http://dx.doi.org/10.1007/s11192-011-0581-9        [ Links ]

19. Ioannidis JP Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166-175. http://dx.doi.org/10.1016/S0140-6736(13)62227-8        [ Links ]

20. Pillay TS. Subject and discipline-specific publication trends in South African medical research, 1996-2011. S Afr J Sci. 2013;109(9/10), Art. #2012-0054, 4 pages. http://dx.doi.org/10.1590/sajs.2013/20120054        [ Links ]

21. San Francisco Declaration on Research Assessment (DORA) [document on the Internet]. c2012 [cited 2015 Dec 15]. Available from: http://www.ascb.org/files/SFDeclarationFINAL.pdf        [ Links ]

22. Alberts B. Impact factor distortions. Science. 2013;340(6134):787. http://dx.doi.org/10.1126/science.1240319        [ Links ]

 

 

Correspondence:
Yolande Harley
Faculty Research Office, Faculty of Health Sciences, Barnard Fuller Building
University of Cape Town
Anzio Road, Observatory 7925
South Africa
Email:yolande.harley@uct.ac.za

Received: 08 Sep. 2015
Revised: 07 Jan. 2016
Accepted: 04 Apr. 2016