ARTICLES

 

Do words matter: Investigating the association between linguistic features of accounting examinations and marks

 

 

Juan Mendelsohn Ontong

School of Accountancy, Faculty of Economic and Management Sciences, Stellenbosch University, Stellenbosch, South Africa ontongj@sun.ac.za

 

 

---

ABSTRACT

The use of linguistic features in school assessments, as well as the impact of these factors on the outcome of assessments have received limited attention in the literature. With this study I aimed to analyse linguistic features of accounting examinations. A quantitative technique, using tests for correlation, was employed to analyse the Grade 12 National Senior Certificate accounting examinations from 2011 until 2021. Advanced textual analysis software was used to identify connections between specific linguistic features and the marks obtained in these examinations. The findings of this study suggest that increasing the levels of linguistic features associated with analytical thinking, emotional tone and big words, which may be assumed to promote comprehension, may in reality be counterproductive, consequently potentially resulting in poorer marks attained in assessments in the future. The findings of this study are important for the creators of assessment to consider when developing accounting assessment. Given the effect of linguistic features on assessment results as identified in this study, it contributes to the debate on the use of certain linguistic features in assessment.

Keywords: accounting; assessment; linguistics; National Senior Certificate; performance

---

 

 

Introduction

Accounting as subject is commonly referred to as the language of business. Accounting and its related processes and information systems establish a common language for businesses that allows investors and other stakeholders to assess financial information and make decisions based on that financial information (Abreu, 2015; International Accounting Standards Board [IASB], 2022).

Although accounting has been referred to as a business language for many decades, Avery (1953) suggests that there appears to be some debate about whether the accountant's vocabulary has achieved a level of comprehension such that accounting may be regarded as an art or a branch of scientific knowledge in and of itself. Although the concept of accounting as a language of business continues to be a relevant topic, the effect of language and the use of linguistic features within the accounting subject have, however, received limited attention in the literature. When evaluating the concept of whether the use of specific linguistic features has an effect on assessment, the literature which is predominantly from outside South Africa (such as; Cruz Neri, Guill & Retelsdorf, 2021; Feser & Höttecke, 2021; Höttecke, Feser, Heine & Ehmke, 2018) focuses heavily on the subjects of mathematics and sciences, while literature on the use of linguistic features in accounting and accounting assessment is scarce both within South Africa and internationally. Furthermore, at the school level, this area of research has received even less attention. Accounting as subject differs significantly from science, technology, engineering, and mathematics (STEM)-related subjects. Accounting as subject is a distinct discipline based on the nature of the learning outcomes and, therefore, requires specific focus.

Turning the focus to learner performance in the Grade 12 National Senior Certificate (NSC) accounting assessment in South Africa, it is noted that the average mark earned by learners who acquire a score of at least 30% in these examinations has historically fluctuated significantly when compared year on year (Department of Basic Education [DBE], Republic of South Africa [RSA], 2022). The impact of linguistic features on assessment and their impact on learner performance in Grade 12 NSC examinations are largely unknown. Although this study focused on the South African setting, the purpose of this study was to determine whether the use of specific linguistic features in the question papers had any statistically significant correlation with the average scores achieved in the accounting Grade 12 year-end examinations for those learners achieving a mark of 30% and above, the results could be applied within a South African and international context. The research question was: What is the relationship between the averages obtained in the Grade 12 NSC accounting examination and specific linguistic features used in those assessment question papers? This study contributes to the current body of knowledge by examining a variety of linguistic features over an extended period in order to determine their impact. These findings may aid creators of Grade 12 NSC accounting examinations and other accounting assessment in gaining a better understanding of the effect of specific linguistic features on the outcome in terms of learner marks achieved.

The remaining structure of this article is as follows. The introduction is followed by the literature review in which I examine literature on the effect of using linguistic features in assessment. This is followed by the methodology, in which I describe the process followed to answer the research question, which is followed by the results, findings, and a discussion of the identified correlations, with the conclusion ending the article.

 

Literature Review

The literature review starts with a discussion of the theoretical underpinning of the research question, which is followed by an understanding of the literature on the effect of comprehension on academic achievement in assessment.

Theoretical underpinning

Understanding the effect of linguistic features on academic performance, literature notes that to be academically effective, learners must develop linguistic competencies beyond the use of daily language registers (Schleppegrell, 2004). From a theoretical perspective, Just and Carpenter (1980), through the theory of reading, suggest impact of certain words on comprehension. It is suggested that theoretically certain words may require the reader to use a greater cognitive processing capacity level. It is suggested that certain words such as infrequent words may impact processing levels and comprehension (Just & Carpenter, 1980). This study extends the theory of reading to examine specific linguistic features and their impact on comprehension and consequential learner performance in accounting assessment.

Teachers must support learners in developing their competencies by delivering texts, such as textbooks, workbooks, and assessment, at an appropriate linguistic level that enables both comprehension of specific topics and the acquisition of new linguistic repertoires (Eberts, Hollenbeck & Stone, 2002). Additionally, assessment must include text, and therefore linguistic features, that are understandable by all learners to avoid construct-irrelevant variation and item bias. Beinborn's (2016) research on text comprehension and readability demonstrates that the difficulty of a text is determined not only by its lexical and syntactic structures, but also by its cognitive structuring, cohesion, semantic redundancy, and the reader's prior knowledge.

The effect of comprehension on academic achievement in assessment

Numerous studies have demonstrated that reading comprehension has an effect on overall academic achievement (Cooper, Moore, Powers, Cleveland & Greenberg, 2014; McGee, Prior, Williams, Smart & Sanson, 2002). While it is self-evident that simplifying texts improves their readability, research to date has been inconsistent. It has been demonstrated that while great textual cohesion should boost a text's comprehensibility, it might cause comprehension issues for learners with a high degree of knowledge (Cruz Neri et al., 2021). It is, therefore, important for creators of assessment, who may have good intentions with adding additional linguistic features, to understand within a specific subject that the effect of these linguistic features may be detrimental to learners' academic performance.

Because comprehension of a written task is important for completion, learners must have sufficient reading comprehension to process assignments, beginning with letter and word decoding, and moving to connecting sentences to form a cohesive mental image (Francis, Snow, August, Carlson, Miller & Iglesias, 2006; Hall, Kowalski, Paterson, Basran, Filik & Maltby, 2015). Reading comprehension has been identified as a necessary component of science education and scientific literacy (Martin, 1993; Yore, Hand, Goldman, Hildebrand, Osborne, Treagust & Wallace, 2004) as science can only be built, altered, and transmitted via the use of language (Yore et al., 2004). Numerous studies have established a substantial correlation between learners' reading comprehension and performance in science modules (Bird & Welford, 1995; Cromley, 2009; O'Reilly & McNamara, 2007). Good readers routinely beat struggling readers in science, despite the fact that their scientific knowledge may be comparable, which highlights the critical importance of reading comprehension for science success. However, limited research has been conducted on the comprehension and performance of accounting learners as a result of linguistic features used in assessment.

Additionally, some data indicate that specific textual qualities, such as grammar or word count, have a substantial effect on the association between reading comprehension and science performance, which means that item difficulty varies when linguistic features are altered (Prophet & Badede, 2009; Rivera & Stansfield, 2004). Contributing to the mixed results seen in the literature, studies such as that by Llosa, Lee, Jiang, Haas, O' Connor, Van Booven and Kieffer (2016) demonstrate that avoiding difficult-to-understand terminology consistently throughout an assessment has no effect on the difficulty of the test. Only the use of dictionaries and glossaries was found to be useful in reducing item bias. While linguistic simplification had no significant effect on mathematical items, Haag, Heppt, Roppelt and Stanat (2015) found that it provided a small benefit to learners with intermediate language proficiency.

The literature shows the importance of comprehension in an assessment to achieve learning outcomes in terms of measurable academic performance. Key to understanding the value of linguistic features in assessment performance is understanding that mixed results have been noted across disciplines and subjects. The complexity of linguistic features, although potentially perceived to have a positive potential effect on learner comprehension, may result in the opposite occurring and the learners instead not being able to understand the assessment.

 

Methodology

Research Approach, and Sample and Analysis In order to answer the research question, What is the relationship between the averages obtained on the Grade 12 NSC accounting examinations and specific linguistic features used in those assessment question papers?, the linguistic features were quantified using the Linguistic Inquiry and Word Count (LIWC)-22 textual analysis software program. The statistical analysis program, Statistica, was then used to determine whether any correlations existed between the averages obtained in the South African Grade 12 NSC accounting assessment for learners who earned a final grade of 30% and above and the linguistic features used in those examination question papers. A scale of -1 to 1 is used as the descriptor for the correlation analysis. In this study I examined learners who earned a final grade of at least 30%, as this group represents the majority of accounting learners.

The sample for this study was the Grade 12 NSC accounting assessment question papers from 2011 to 2021, spanning an extended period to ensure statistical power in identifying correlations. Additionally, the period for analysis included a pre-COVID-19 (coronavirus disease) pandemic and a post-COVID-19 period. Accounting was chosen as the focus of this study because it represents a unique type of subject. By focusing on a single subject, I was able to identify the effect of specific linguistic features that could be used in the future for accounting assessment. As a result of this study, it is suggested that further research into other subjects is warranted.

Textual Processing Module

LIWC-22 has been used in various research papers as an appropriate tool to measure narratives and linguistic tones (Boyd, Ashokkumar, Seraj & Pennebaker, 2022). For the purposes of this study, various linguistic features in the form of summary features, linguistic dimensions, and psychological processes were evaluated; their meaning and sub-categories are summarised in Tables 1, 2, and 3.

 

 

 

 

 

 

Sources of Data

NSC examination results and assessment question papers were obtained from the DBE's website (DBE, RSA, 2022), which ensures reliability and validity of the data. The LIWC dictionary is integrated into the LIWC software program, which I purchased under a research licence.

Ethical Considerations

I obtained ethical clearance from the Research and Ethics Committee of my university of employment. All information used in this study was publicly available on the South African DBE's website at the time of this study.

 

Results

The results for the Grade 12 NSC accounting assessment in terms of the average mark achieved for those achieving a mark of 30% or above in the assessment fluctuated significantly over the period under review (cf. Table 4). In addition, the total word count per assessment also differed significantly, which suggests on a surface level that the creators of the assessments may have engaged in various linguistic features.

 

Discussion

The results of the textual analysis are presented in three sections: firstly, based on summary features, followed by analysis using linguistic dimensions, and then concluded by an analysis using psychological processes in assessment. The interpretation of the results from the correlation as shown in Table 4 can be interpreted in two ways: firstly, a positive coefficient indicates that an increase in the level of the specific linguistic feature results in an increase in the average assessment mark, while a negative coefficient indicates that an increase in the level of the specific linguistic feature results in a decrease in the average assessment mark; secondly, an asterisk indicates a significant correlation at a 95% confidence level.

Summary Features

The results of the summary features, as shown in Table 5, provide insight into a variety of linguistic features measured as part of the Grade 12 NSC accounting examinations. A total of six summary features are reported. The results firstly show negative associations between the features analytical thinking, which refers to the use of logical and formal thinking features; emotional tone, which refers to the degree of positive or negative tone used; and big words, being words with seven or more characters. These negative associations suggest that perhaps perceived logic and formal thinking, and the use of emotional tone and big words, do not contribute to increased performance when the perceived language of accounting is muddled by the use of these linguistic features. This is consistent with the notion of Eberts et al. (2002) who suggest that teachers support learners in order to enable comprehension, however, specific attention must be placed on appropriate linguistic features to aid comprehension.

 

 

Although the metric, authentic, which refers to the perceived use of honesty, may suggest a positive connotation with comprehension, a significant negative association has been noted. This suggests that learners perhaps approach the subject of accounting with a cautious outlook and question the intention of the assessment creator regarding whether their intention is to challenge the learner. Although noted as insignificant, positive associations are noted with the use of the clout and words-per-sentence features, which suggests that when assessment creators use language of leadership and status and reduce the words per sentence, learners are better able to comprehend the question. The reduction of words per sentence is consistent with the findings of Prophet and Badede (2009) that show an association between reading comprehension and academic performance in the subject, science.

Although various studies have identified an effect of comprehension on overall academic achievement (Cooper et al., 2014; McGee et al., 2002), specific features seem to have greater importance compared to others with specific reference to understanding the assessment of accounting.

Linguistic Dimension Analysis

The results of the evaluation of linguistic dimensions, as shown in Table 6, indicate mixed results for the various linguistic dimensions. The only significant association noted is for auxiliary verbs, which show a significant negative association between the use of this feature and the assessment averages achieved. Noteworthy is the juxtaposition between quantities and numbers, where quantities appear to provide clarity that results in a positive association, whereas numbers do not necessarily provide sufficient context to aid the learner in completing a question.

 

 

Analysis using Psychological Processes

The final step of the linguistic feature analysis involved examining the psychological processes features used in the assessment, the results of which are shown in Table 7. Significant associations, in the form of negative associations, are only noted in cognition and its sub-component features. These results suggest that the use of psychological processes in Grade 12 NSC accounting examinations are ineffective linguistic features to employ to increase assessment marks.

 

 

Conclusion

The application of linguistics and the dimensions associated with the use of particular texts in assessments is an area of literature that has received little attention in South African secondary education. This study adds to understanding the effect of certain linguistic features on performance in assessment. The findings may help assessment creators in and outside South Africa to appreciate the significance of word choice for an assessment within a subject such as accounting. In accounting learners are frequently given instructions based on which they are then required to produce a product that is frequently highly dependent on the set of facts provided. With this study I found that certain linguistic features may result in negative effects on assessment results, which include the use of words associated with analytical thinking, emotional tone and big words. This study contributes to linguistic research outside of traditional language and instead focused on accounting, which in turn is often referred to as the universal business language.

Although this study was performed over an extensive period of time, the study focused only on accounting examinations in the National Senior Certificate; the results may, therefore, differ from other subjects. A further limitation is the specific word lists used in order to identify the linguistics features being assessed; this could be made extended in future research projects. As an exploratory study, I aimed to spark discussion about the use of linguistics in non-language modules and to call into question the perceived benefit of adding more words to result in easier assessment, while it has been discovered that this is not the case. However, a thorough understanding of the specific types of words and linguistics is critical for making assessment more understandable to learners which might result in higher grades.

 

Notes

i. Published under a Creative Commons Attribution Licence.

 

References

Abreu R 2015. Accounting for citizenship: The role of accountant. Procedia Economics and Finance, 26:933-941. https://doi.org/10.1016/S2212-5671(15)00909-0        [ Links ]

Avery HG 1953. Accounting as a language. The Accounting Review, 28(1):83-87.         [ Links ]

Beinborn LM 2016. Predicting and manipulating the difficulty of text-completion exercises for language learning. PhD dissertation. Darmstadt, Germany: Technische Universität Darmstadt. Available at https://tuprints.ulb.tudarmstadt.de/5647/1/DissertationBeinborn_publishedVersion_September20_online.pdf. Accessed 17 April 2024.         [ Links ]

Bird E & Welford G 1995. The effect of language on the performance of second-language students in science examinations. International Journal of Science Education, 17(3):389-397. https://doi.org/10.1080/0950069950170309        [ Links ]

Boyd RL, Ashokkumar A, Seraj S & Pennebaker JW 2022. The development and psychometric properties of LIWC-22. Austin, TX: University of Texas at Austin. Available at https://www.liwc.app/static/documents/LIWC-22%20Manual%20-%20Development%20and%20Psychometrics.pdf. Accessed 18 April 2024.         [ Links ]

Cooper BR, Moore JE, Powers CJ, Cleveland M & Greenberg MT 2014. Patterns of early reading and social skills associated with academic success in elementary school. Early Education and Development, 25(8): 1248-1264. https://doi.org/10.1080/10409289.2014.932236        [ Links ]

Cromley JG 2009. Reading achievement and science proficiency: International comparisons from the Programme on International Student Assessment. Reading Psychology, 30(2):89-118. https://doi.org/10.1080/02702710802274903        [ Links ]

Cruz Neri N, Guill K & Retelsdorf J 2021. Language in science performance: Do good readers perform better? European Journal of Psychology of Education, 36(1):45-61. https://doi.org/10.1007/s10212-019-00453-5        [ Links ]

Department of Basic Education, Republic of South Africa 2022. 2021 National Senior Certificate. Diagnostic report Part 1: Content subjects. Pretoria: Author. Available at https://www.education.gov.za/Portals/0/Documents/Reports/2021NSCReports/2021%20Diagnost ic%20Report%20Part%201%20Content%20Subjects.pdf?ver=2022-02-08-103527-000. Accessed 18 April 2024.         [ Links ]

Eberts R, Hollenbeck K & Stone J 2002. Teacher performance incentives and student outcomes. The Journal of Human Resources, 37(4):913-927. https://doi.org/10.2307/3069621        [ Links ]

Feser MS & Höttecke D 2021. Exploring the role of language in physics teachers' everyday assessment practice. Journal of Science Teacher Education, 32(6):686-704. https://doi.org/10.1080/1046560X.2021.1890926        [ Links ]

Francis DJ, Snow CE, August D, Carlson CD, Miller J & Iglesias A 2006. Measures of reading comprehension: A latent variable analysis of the diagnostic assessment of reading comprehension. Scientific Studies of Reading, 10(3):301-322. https://doi.org/10.1207/s1532799xssr1003_6        [ Links ]

Haag N, Heppt B, Roppelt A & Stanat P 2015. Linguistic simplification of mathematics items: Effects for language minority students in Germany. European Journal of Psychology of Education, 30:145-167. https://doi.org/10.1007/s10212-014-0233-6        [ Links ]

Hall SS, Kowalski R, Paterson KB, Basran J, Filik R & Maltby J 2015. Local text cohesion, reading ability and individual science aspirations: Key factors influencing comprehension in science classes. British Educational Research Journal, 41(1):122-142. https://doi.org/10.1002/berj.3134        [ Links ]

Höttecke D, Feser MS, Heine L & Ehmke T 2018. Do linguistic features influence item difficulty in physics assessments? Science Education Review Letters, 2018:1-6. Available at https://edoc.huberlin.de/bitstream/handle/18452/19939/2018_1.pdfsequence=1. Accessed 28 April 2022.         [ Links ]

International Accounting Standards Board 2022. International financial reporting standards (IFRS): Including international accounting standards (IASs) and interpretations as at 2022. London, England: Author.         [ Links ]

Just MA & Carpenter PA 1980. A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4):329-354. https://doi.org/10.1037/0033-295X.87A329        [ Links ]

Llosa L, Lee O, Jiang F, Haas A, O'Connor C, Van Booven CD & Kieffer MJ 2016. Impact of a large-scale science intervention focused on English language learners. American Educational Research Journal, 53(2): 395-424. https://doi.org/10.3102/000283121663734        [ Links ]

Martin JR 1993. Literacy in science: Learning to handle text as technology. In MAK Halliday & JR Martin (eds). Writing science: Literacy and discursive power. Washington, DC: Falmer Press.         [ Links ]

McGee R, Prior M, Williams S, Smart D & Sanson A 2002. The long-term significance of teacher-rated hyperactivity and reading ability in childhood: Findings from two longitudinal studies. The Journal of Child Psychology and Psychiatry, 43(8):1004-1017. https://doi.org/10.1111/14697610.00228        [ Links ]

O'Reilly T & McNamara DS 2007. The impact of science knowledge, reading skill, and reading strategy knowledge on more traditional "high-stakes" measures of high school students science achievement. American Educational Research Journal, 44(1): 161-196. https://doi.org/10.3102/0002831206298171        [ Links ]

Prophet RB & Badede NB 2009. Language and student performance in junior secondary science examinations: The case of second language learners in Botswana. International Journal of Science and Mathematics Education, 7:235-251. https://doi.org/10.1007/s10763-006-9058-3        [ Links ]

Rivera C & Stansfield CW 2004. The effect of linguistic simplification of science test items on score comparability. Educational Assessment, 9(3-4):79-105. https://doi.org/10.1080/10627197.2004.9652960        [ Links ]

Schleppegrell MJ 2004. The language of schooling: A functional linguistics perspective. New York, NY: Routledge. https://doi.org/10.4324/9781410610317        [ Links ]

Yore LD, Hand B, Goldman SR, Hildebrand GM, Osborne JF, Treagust DF & Wallace CS 2004. New directions in language and science education research. Reading Research Quarterly, 39(3):347-352.         [ Links ]

 

 

Received: 5 May 2022
Revised: 14 December 2023
Accepted: 4 March 2024
Published: 31 May 2024