ARTICLES

 

Evaluating quality mechanisms in collaborative open distance learning (ODL) course development using lecturers' perspectives

 

 

Estelle van Rensburg; Anneke Venter; Michele van Wyk; Jane Ndinisa; Albert Michau

University of South Africa, South Africa

 

 

---

ABSTRACT

Course development at online distance learning (ODL) universities is frequently undertaken as a formalised collaborative process. In line with this, the University of South Africa (Unisa) has adopted a framework for a 'team approach' to course development, to which lecturers are key contributors, and who are supported by other specialists. The course development process has several embedded quality assurance mechanisms, which were derived from relevant literature on collaborative course development practices and quality standards. At Unisa, there has been no systematic research into the effectiveness of the quality mechanisms from lecturers' perspective. An exploratory study was conceptualised to obtain feedback from lecturers on the value of different quality mechanisms. The research posed the question: Which quality mechanisms promote successful course development, in the Unisa context, from a lecturer's perspective? A mixed-method approach was used. It involved a survey among lecturers to gauge the relative value attached to different quality mechanisms. Follow-up focus group sessions were conducted to further explore emerging issues. The results of the study highlighted the importance of formative feedback, including feedback from knowledgeable peers inside and outside the university. Furthermore, the allocation of sufficient resources was regarded as critical. On the basis of the findings, a model for quality assurance in collaborative course development that integrates the lecturers' perspectives is suggested.

Keywords: collaborative course development, quality assurance in higher education, distance education, lecturers' perceptions

---

 

 

INTRODUCTION

In traditional higher education institutions, the task of preparing course notes or course materials is the responsibility of the lecturer, who may complete the task without significant consultation or support from peers or other experts. On the other hand, in online distance learning (ODL) institutions, the design of materials is typically undertaken as a collaborative process, using input from a range of experts such as curriculum developers, instructional designers, graphic designers and educational technologists. A large-scale collaborative development process of this nature requires a systematic and integrated approach to achieve overall strategic goals such as the development of student-centred learning environments, which are mediated through appropriate technologies. An effective application of the collaborative development model needs to include quality mechanisms to ensure that quality courses are produced on a consistent and scalable basis (Chao, Saj & Hamilton, 2010).

In line with this approach, South Africa's largest ODL provider, the University of South Africa (Unisa), has adopted a cross-functional course development process, undertaken by selected design and development teams. At Unisa, the teams typically include education consultants (ECs), who manage the development process and advise on curriculum and instructional design matters. The teams that work on the preparation of the learning materials also include lecturers, graphic artists, electronic originators, subject librarians and language practitioners. Lecturers (in the literature often referred to as 'faculty' or 'course developers') are considered to be critical role players in the development process since they are responsible for authoring the teaching text and facilitating the learning process.

In an attempt to ensure quality in a complicated course development process, several quality assurance mechanisms have been embedded at Unisa. The mechanisms are aligned with relevant literature on collaborative course development practices and quality standards (Chao, Saj & Hamilton, 2010; Sankey et al., 2014). However, to date, there has been no formal research to establish the lecturers' perspectives on the value added by the existing quality mechanisms. A group of ECs, who are responsible for assisting lecturers as part of the collaborative course development process, conceptualised a small-scale study to gain feedback from lecturers who participated in the cross-functional course development process. This small group of lecturers was purposively sampled from a larger group who had worked with the ECs in the year leading up to the study. The aim of this study was to explore lecturers' experiences and perspectives in order to evaluate the relevance and effectiveness of existing quality mechanisms, with a view to ultimately improving the overall quality of the course development process. The results of such a study could potentially benefit other ODL institutions that undertake collaborative course development.

The article provides the background to the study by describing the Unisa context in order to clarify the scale and complexity of the environment in which the study is located, with the concomitant need for effective quality assurance mechanisms. Subsequently, a literature review is provided to explore and motivate the implementation and evaluation of critical quality assurance mechanisms in the development process. Against this background, the article continues to explain how an appropriate explorative methodology was applied to gain an understanding of the lecturers' perspectives on the usefulness of the quality assurance mechanisms in an ODL context. A combination of a survey and follow-up focus group sessions was used to elicit feedback from the participating lecturers. The data were analysed and interpreted to derive key themes for insight and inform a suggested model for quality assurance in collaborative course development.

 

CONTEXT OF THE STUDY

Unisa is a mega-university, serving more than 350 000 students worldwide, the majority of which are from southern Africa. It has traditionally provided distance education, mainly via print-based delivery, and is currently transitioning to an online delivery mode. The curriculum and course materials are carefully designed, developed and pre-packaged by a team of experts to provide for timeous delivery.

The collaborative course design and development process at Unisa is facilitated according to procedures contained in an institutional document entitled 'Framework for the implementation of a team approach to curriculum and learning development at Unisa' (FTA), which was approved by the Executive Committee of Senate on 11 April 2013. The process described in this document is highly complex, involving multiple role players and cross-departmental sub-processes that extend across internal departmental boundaries.

The main contributors to the creative process are the lecturers, who author academic content and design assessment strategies. Their writing process is supported by specialists from auxiliary departments such as ECs from the Directorate for Curriculum Development and Transformation (DCDT). The ECs provide a range of supportive functions, including just-in-time and hands-on training, a project management function and the coordination of critical support services such as graphic design, language editing and electronic origination functions. The collaborative nature of the team approach implies that different role players need to commit to the project and attach value to a quality end product.

The process described in the document involves an initial phase of curriculum planning - undertaken collaboratively by the lecturer, the EC and other relevant role players - during which a 'module form' (course blueprint), with learning outcomes, assessment criteria and other information, is produced for every module (course). This is followed by planning the learning and assessment strategies for a module. Once these are in place, a development phase starts, during which authors generate any necessary materials, while graphic designers, multimedia experts and other role players contribute relevant elements of the course. Critical readers (academic peers) and DCDT ECs provide feedback on the various course components. The planning and development processes are not strictly linear and may follow an iterative process until the team is satisfied with the product. Subsequently, the materials are language edited and formatted for print or online delivery.

A variety of mechanisms is applied to promote the quality of the development process as well as the final product. For the purposes of this study, these aspects were called 'quality mechanisms', which are defined as regulating or evaluation procedures. They are conducted during or after the development process to assess whether the materials produced meet expected requirements or standards. The aim of quality mechanisms is to ensure that the development teams support the creation of meaningful learning experiences. In the Unisa context, these quality mechanisms include:

• The implementation of policies and standards for the design and development process

• The initialisation and conclusion of the course development process through the signing of a 'certificate of due diligence' (CDD) by the main parties involved

• The use of a project management system to coordinate and monitor the design and development process

• The dissemination of guideline documents or templates

• The use of critical readers and ECs to provide constructive feedback on curricula and materials

• The use of language practitioners to edit the learning materials

• The voluntary completion of an evaluation form ('service satisfaction survey') at the end of the process

• The implementation of an annual institutional evaluation based on feedback by students and other role players.

A literature review was undertaken to compare the Unisa quality assurance mechanisms against those currently used in other collaborative course development contexts. This provided more insight into how the Unisa quality assurance system for collaborative course development compares with international practice.

 

REVIEW OF LITERATURE ON QUALITY MECHANISMS IN THE COLLABORATIVE COURSE DEVELOPMENT PROCESS

In their seminal work on the nature of distance education, scholars like Peters (1993), Moore (1993, 1996), Keegan (1996) and Holmberg (2005) argued that distance education requires 'new forms of organization that are based on the application of principles of systems management' (Moore, 1993: 2). This would involve the design and development of courses, not by individual lecturers but rather by teams of specialist role players (Holmberg, 2005). A review of relevant literature indicated that higher education institutions, especially ODL institutions, have commonly adopted this approach and use a highly skilled team, consisting of members with different areas of expertise, to design and develop curricula and learning materials (Abdous, 2009; Bawa & Watson, 2017; Bronson, 2016; Butcher & Wilson-Strydom, 2013; Chao, Saj & Hamilton, 2010; Guri-Rosenblit, 2009; Herron et al., 2012; Holsombach-Ebner, 2013; Kalantzi et al., 2016; Mills, 2006; Puzziferro & Shelton, 2008; Thurab-Nkhosi & Marshall, 2009; Venable et al., 2009). They typically follow a process consisting of distinctive phases such as planning, curriculum development, learning design, generation of materials and media, production, and evaluation (Bawa & Watson, 2017; Chao, Saj & Hamilton, 2010; Herron et al., 2012; Holsombach-Ebner, 2013; Kalantzi et al., 2016; Mills, 2006; Puzziferro & Shelton, 2008; Thurab-Nkhosi & Marshall, 2009; Venable et al., 2009).

During this process, a number of quality mechanisms is usually involved. The mechanisms identified in the literature can be clustered into different categories, namely (i) management and resource allocation processes, (ii) good practice guidelines and (iii) evaluation processes.

Management and resource allocation processes

The institutional allocation of sufficient resources to ensure the quality of courseware design and development is generally highlighted as essential (Hansson, 2008; Butcher & Wilson-Strydom, 2013; Sankey et al., 2014). Furthermore, the organised and systematic nature of project management, typically consisting of five phases - (i) initiation, (ii) planning, (iii) execution, (iv) monitoring and control, and (iv) closure - is seen as promoting quality through its inbuilt emphasis of monitoring, control and meaningful collaboration (Da Silva, Diana & Catapan 2015; Bawa & Wilson, 2017).

A clear delineation of the various team members' roles and responsibilities during the early stages of a project ensures that everyone understands what is expected and supports the efficient implementation of the process (Abdous, 2009; Puzziferro & Shelton, 2008; Venable et al., 2009). Allocating resources for the training of staff in course development and other required skills is equally essential as this is a key element of achieving quality improvement (Herron et al., 2012; Mills, 2006). This is particularly true in online learning environments where staff systematically need to update their knowledge and strategies (Lenert & James, 2017; Sankey et al., 2014; Hansson, 2008).

Determination of a course lifecycle by specifying at which intervals a course should be revised - for example, every three years - encourages ongoing attention to quality and provides a basis for continuous improvement (Holsombach-Ebner, 2013).

Good practice guidelines

Numerous authors highlight the importance of implementing predetermined guidelines, standards, criteria, checklists or rubrics to assure the quality of the design and development process. Such guidelines are frequently based on international, regional or national standards for distance education or e-learning, for example, in Europe (Hansson, 2008; ENQA, 2015), Australasia (Sankey et al., 2014), the United States (Keil & Brown, 2014), and South Africa (CHE, 2014). Subscription-based rubrics are also widely used, particularly the University of Maryland's 'Quality Matters' (Rucker, Edwards & Frass, 2015; Debattista, 2017), but a study by Lenert and Janes (2017) suggested that many institutions use internally compiled rubrics. Standards should ideally (i) be compiled for the entire design and development process (Thurab-Nkosi & Marshall, 2009; Chao, Saj & Hamilton, 2010), (ii) clearly communicate what is expected of all the team members (Puzziferro & Shelton 2008), and (iii) be easy to use (Baldwin & Ching 2019).

Templates are another form of good practice identified by several authors. A template is a document that has been preformatted with a certain structure, layout or look, which can then be populated with content. Learning design templates typically specify the various elements that should form part of the learning experience as required by quality standards, for example, introductions, learning outcomes, activities, and online discussion questions (Herron et al., 2012). Abdous (2009: 287-288) proposes a quality assurance process for e-learning development that is centred around a combination of templates and checklists to ensure the 'appropriateness, comprehensiveness and consistency' of courses. Herron et al. (2012) and Holsombach-Ebner (2013) report that comprehensive templates increased consistency and learning effectiveness in their distance learning courses, while Albashiry et al. (2015: 408) indicate that templates provided 'structure and support'.

Actual examples of good practice can further improve the quality of course development. They facilitate the work process (Albashiry et al., 2016) and promote pedagogically sound designs by providing concrete models to work from (Sankey et al., 2014; Bower & Vlachopoulos, 2018).

Evaluation processes

Evaluation is considered an integral part of the traditional instructional design process (Briggs, 1991). In their well-known model for instructional design, Dick and Carey (1985, quoted in Obizoba, 2015) advocated the use both of formative evaluation - evaluation during the development process - and summative evaluation. Summative evaluation can be done by the course team itself at the end of the project or after its implementation by means of student and lecturer feedback on the course.

The literature suggests that reviewing is one of the most common forms of formative evaluation. Most sources consulted describe the involvement of academic peer reviewers, also called 'critical readers', 'subject matter experts' or 'moderators'. Academic reviewers evaluate the text generated by the course author, paying particular attention to its correctness, appropriateness for the context and level, and its currency and alignment between outcomes and assessment (Heron et al., 2012; Holsombach-Ebner, 2013; Ahmad Zabidi et al., 2017). In addition, ECs commonly give feedback on draft course materials from an educational perspective (Puzziferro & Shelton, 2008; Thurab-Nkhosi & Marshall, 2009; Holsombach-Ebner, 2013; Bawa & Watson, 2017).

In course piloting, another formative mechanism, prospective students are asked to complete a part of the course and to give feedback on it, which is then used to make improvements (Herron et al., 2012; Kartunnen & Juusola, 2019). This can be done with a prototype unit in the initial stages or at the end of development before the course is implemented (Bronson, 2016).

Formative evaluation of the process by the development team itself is not frequently described in the context of quality assurance in course development, but, in literature relating to reflection and action research (e.g., Boud et al., 2013; Zuber-Skerritt, 2001), evaluative reflection by the team on work-in-process is said to enhance the quality of the process and product as well as professional development. Taylor et al. (2016: 2) argue that reflective evaluation results in team members gaining greater ownership of the project, and enables them to 'adapt the way they work in an iterative manner throughout the life cycle of the project'. Reflective strategies can also be applied by the team at the end of the development process as a form of summative evaluation on both process and product (McKenney & Reeves, 2014). Furthermore, support departments typically conduct satisfaction surveys among their internal clients (Ahmad Zabidi et al., 2017).

Feedback by students on how they experienced a course is seen as an important form of summative evaluation (Thurab-Nkosi & Marshall, 2009; Holsombach-Ebner, 2013; Young & Hoerig, 2013; Nichols Hess & Greer, 2016; Ahmad Zabidi et al., 2017). Students can comment on factors such as tutoring and course interaction, but also on the structure of the course, and its 'learning outcomes, content (including background materials), teaching and learning methodologies, and online materials' (Butcher & Wilson-Strydom, 2013: 8).

Quality assurance in course development is typically regarded as a cycle, and summative evaluation results are used to inform future revisions of a course (Hansson, 2008; Abdous, 2009; Sankey et al., 2014; Ahmad Zabidi et al., 2017; Lenert & Janes, 2017).

An examination of the course development process at Unisa indicated that most of these mechanisms were used in one form or another. The only exception was formal course piloting, which was generally not undertaken owing to time constraints. However, the relative value of these quality mechanisms in the collaborative process needed to be explored from the perspective of the lecturers as key contributors to the development process.

 

THEORETICAL FRAMEWORK

The theoretical framework underlying this study is twofold. Firstly, the researchers' understanding of the course design and development process at distance institutions is grounded in a systems model, as described by Peters (1993) and Moore (1993). In this process, delivery of teaching and learning experiences is seen not as a function of a few simple elements only, but rather as a complex system constituted by the collective action of many role players, including institutional management, lecturers, instructional designers, media specialists and instructional technologists. The contributions of these individuals and technologies merge to create an integrated network '. of media specialists, knowledge specialists, instructional design specialists, and learning specialists. ... this process requires... large budgets, and long periods of design time' (Moore, 1993: 4). In such a complex system, quality assurance becomes essential to ensure that a course will be fit for its purpose and will effectively integrate the various role players' contributions.

The second theoretical perspective adopted in this study is an understanding of quality assurance as a cyclical process in which iterative, reflective evaluation promotes continuous quality enhancement, which is inherent in the Unisa context. This approach is rooted in a tradition of conceptualising human learning and activity as a cycle of experience and reflection, which has been proposed by several theorists (e.g., Dewey, 1986 (1938); Lewin, 1946; Freire, 1972; Kolb, 1984; Schön, 1987; Ford & Profetto-McGrath, 1994; Boud, Keogh & Walker, 2013; Brookfield, 2017). In the context of higher education, Biggs refers to such a cyclical conception of quality assurance as 'prospective'. 'Prospective QA is concerned with assuring that teaching and learning does now, and in future will continue, to fit the purpose of the institution' (Biggs, 2001: 222). This is a process in which both evaluation and reflection play an important part: 'The institution needs... to establish built-in mechanisms that allow it, like the individual reflective teacher, to continually review and improve current practice' (Biggs 2001: 223, our italics). In the collaborative design and development process, the entire development team has a stake in the enhancement of quality through the application of such quality mechanisms. Exploring the perspectives of a key role player in the team - i.e., the lecturer - may assist in ensuring that these mechanisms are an effective tool in successfully implementing the reflective quality assurance cycle.

 

RESEARCH QUESTION

A research question was formulated to explore the lecturers' perspectives and experiences of the value of the quality mechanisms embedded in the course development process at Unisa. The main research question was phrased as follows:

Which quality mechanisms promote successful course development in the Unisa context, from a lecturer's perspective?

The following sub-questions were formulated:

• Which quality mechanisms, used in the Unisa process of course development, do lecturers experience as adding a great deal of value to the process, and why?

• Which quality mechanisms do lecturers experience as adding little or no value to the process, and why?

• Which additional quality mechanisms do lecturers recommend, and why?

• How do the quality mechanisms used in the Unisa process of course development, and those recommended by lecturers, compare with those recommended in literature?

• How should quality mechanisms be amended or extended in order to establish an improved quality assurance process for course development at Unisa?

A suitable research methodology to address these questions needed to provide for an overall view of the relative value that lecturers attach to the different mechanisms, together with an in-depth exploration of their perceptions and of their authentic experiences.

 

RESEARCH METHODOLOGY

Lecturers' perspectives on and experience of the way quality is assured in course development is a complex phenomenon. While there is relevant research available, none of the previous studies focused directly on the lecturers' perceptions of the relative value attached to quality assurance mechanisms in the overall course development process. The researchers therefore considered that such a study could yield valuable information in an ODL context. Unisa is regarded as a good case study for such research as it is a large ODL institution and has been employing collaborative course team development for many years. A study of this nature needed to include an overall view of the relative value that lecturers attach to the different mechanisms, together with an in-depth exploration of their experiences and perceptions thereof.

The relevant Ethics Review Committee at Unisa granted ethics approval for the research project on 22 July 2016. The research project was aligned to the values and principles expressed in the Unisa Policy on Research Ethics. Care was taken to protect participants from any harm and all participants consented to participate in the research. Participants were also allowed to participate anonymously, and their identities were concealed throughout the research project. Permission to conduct the research at Unisa was granted by the Senate Research, Innovation, Postgraduate Degrees and Commercialisation Committee on 8 September 2016.

A qualitative explorative study was undertaken to gain an overview of the perceptions and experiences of lecturers involved in course development. A mixed-method data collection process (Creswell & Plano Clark, 2011) was deemed suitable to gain an overview of such views and practices. The mixed methods included the use of a survey, follow-up focus group sessions and individual interviews.

The study started with an online survey, comprising both closed- and open-ended questions, about the value the participants attached to the different quality mechanisms. The survey was followed up with a combination of qualitative methods, namely focus groups and individual interviews, to gather in-depth information about the perceptions and experiences of lecturers.

The data collection process thus included a two-tiered approach. Details about the methods and samples are

• The survey: A survey was distributed among all the lecturers who had participated in a collaborative course development process during a specific development cycle (265 lecturers). Qualtrics software was used to run the survey. The survey was developed by the research team, approved by the ethics committee, and piloted among a group of colleagues. As part of the survey, participants were given information about the study and asked to indicate their informed consent online. The responses were submitted anonymously. There were 63 respondents (a response rate of 23.7%).

• Focus group and individual interviews: Information emerging from the analysis of the survey data was validated and explored in more depth and detail by means of focus group sessions and individual interviews. The research team prepared the questions. The interview schedule was part of the ethics clearance process. The lecturers who were invited to participate in the focus group sessions and interviewees were recruited from the same larger group of lecturers surveyed.

Using a purposive sampling method, the survey was sent to all the lecturers who had worked with DCDT ECs according to the team approach in the year leading up to the study, and who would, therefore, have had relevant experience of the quality mechanisms in the process. These participants can be considered knowledgeable as they were actively involved in the course development process (Cresswell & Plano Clark, 2011). For the focus group interviews, a combination of voluntary and purposive sampling was applied: volunteers from the survey group were requested to take part, and, where there was an insufficient number of participants for a focus group, additional participants from the survey group were personally invited to participate. These sessions involved 14 participants.

The data obtained from the closed-ended survey questions were quantified to determine the relative value that lecturers attached to specific quality mechanisms. The data from the open-ended questions were subjected to a thematic analysis to obtain insight into lecturers' motivations for their various selections.

Pertinent information emerging from the analysis of the survey data was used as a basis for compiling follow-up questions in the focus group sessions and individual interviews. The research team did a manual thematic analysis of the content of the survey results and interview transcripts. The team compared and re-examined the analysis to identify main themes.

 

FINDINGS

The results of the survey question on the relative importance of listed quality mechanisms are shown in Table 1 below. In the survey, respondents were asked to identify what they considered to be the 'most important' and 'second most important' quality mechanisms. In analysing the results, these two categories were grouped into a single category, designated 'more important'. The same was done for the quality mechanisms considered to be 'least important' and 'second least important', which together were designated 'less important'. For every mechanism, the total for 'more important' was converted into a percentage to reflect the number of participants who had considered that mechanism as 'more important'. The same was done for mechanisms identified as 'less important'. The respondents were also asked to motivate their selection of the more and less important mechanisms.

 

 

Table 1 - Click to enlarge

 

The largest number of survey respondents indicated that they regarded the two most useful quality mechanisms as feedback on the learning materials by a DCDT EC and feedback on the materials by an academic peer (46% and 37%, respectively). Two other mechanisms that also received a fair amount of support were feedback on the course blueprint ('module form') by the EC and templates for writing (17% in both cases). Interestingly, an exactly equal number of respondents (17%) indicated that they regarded templates as not being important. A smaller number of respondents (10% or fewer) regarded some of the other listed quality mechanisms as more important.

Responding to the question on which of the quality mechanisms were seen as less important, 25% selected the option of the 'service satisfaction form' (evaluation form) used at the conclusion of projects while 19% chose the option referring to the 'certificate of due diligence'. As already mentioned, 17% indicated that templates were of little importance to them. The institutional evaluation of modules was regarded as less important by 13% of the respondents, with 8% or fewer of the respondents selecting the other listed options as less important.

The survey respondents were asked to motivate their selections. Relevant comments most frequently made in response to these open-ended questions referred to the following:

• The contribution of the EC was seen as enhancing quality, owing to the expertise of this role player in the field of teaching and learning, and materials development. (One respondent did, however, indicate that the EC he or she worked with, was 'not suited for her role'.)

• Similarly, feedback by an academic peer ('critical reader') was highlighted as valuable, owing to this person's academic expertise and ability to 'provide a different perspective and insight' into the content

• Templates were cited by several respondents as a means of providing guidelines for writing and promoting consistency between courses. On the other hand, a number of respondents criticised the use of templates, mainly for the reason as summarised by one respondent: 'templates kill creativity'

• A number of respondents mentioned that, in addition to templates, sharing of best practices such as good examples of course materials was of great assistance in enhancing quality

• The evaluation form, used at the end of the process, was considered of little use in promoting quality. More than one respondent indicated that they felt constrained in their responses, owing to a lack of anonymity. Furthermore, they felt that the fact that the form was completed after rather than during the development process did little to enhance the quality of that particular project, even though it could conceivably play a role in future projects

• The CDD was indicated by several respondents to be of no value in addressing quality as it was deemed as purely 'administrative' or as a 'rubber stamp'

• The institutional quality evaluation, conducted among students, was not considered helpful by most respondents, because the data gathered in this way reflected students' general impressions, with no specific recommendations for improvements

• With regard to the university's quality system in general, several respondents indicated that their workload and the timeframe imposed on the module development process were problematic issues that compromised the time they could spend on projects, and, hence, the projects' quality. A number of other respondents also mentioned that they saw limitations in the university's learning management system ('myUnisa') as an obstacle to creating good quality learning experiences, with one respondent commenting that 'myUnisa currently places a major damper on quality and creativity'.

The findings from the data obtained in the focus group and interview sessions largely corroborated those from the survey, while also providing further details that shed more light on many of the issues raised. In general, participants indicated that Unisa's procedure for course development, described in the FTA, did much to facilitate the process of module design and development and to enhance the quality of the end product as well as in building capacity among the participating lecturers. Lecturers who participated in the process also indicated, more than once, that they especially gained valuable skills related to module design and development.

Focus group participants concurred that they saw feedback by experienced specialists, both in the academic field concerned and in learning and teaching generally, as the most important contribution to the quality of learning material. The ECs' assistance in designing curricula by contributing to the formulation and institutional approval of the 'module forms' was also seen as particularly helpful, and the sentiment was that ECs with some experience in the subject field could add even more value. The participants nevertheless pointed out that the usefulness of the feedback varied among individuals, especially in the case of ECs. Some ECs provided extensive guidance and useful documents such as templates and exemplars, while there were a few whose feedback and assistance with the module form and module development were less helpful, and not all were equally experienced.

The participants noted that, in fact, the observation about inconsistency of service applied in general to the FTA: while the team approach definitely promoted quality, as opposed to developers working individually, the different departments involved and the different individuals within the departments tended to provide varying levels of service to the lecturers. Furthermore, some participants also confirmed the view of survey respondents that the timeframe and deadlines of the FTA process were problematic in terms of academic workload and allocation of resources.

Variability of feedback also applied to critical readers, but, in this case, it was determined to a greater extent by whether they were internal or external lecturing staff. While internal lecturing staff were familiar with the university's internal standards and conventions, and provided useful feedback from this point of view, external lecturing staff could frequently provide additional subject-related insights.

Focus group participants were as divided on the issue of templates as the survey respondents had been. Some regarded them as crucial, while others felt them unnecessary and involving a 'danger of boxing one in'. In general, more value was attached to good examples of materials and the participants were unanimous in their view that seeing good examples was one of the best means of enhancing quality and that it could be 'really inspiring'.

The focus group participants generally attached more value to project management and project plans than the survey respondents had done, pertaining specifically to the ECs' role in guiding the process, which was described by one participant as 'outstanding'. Several commented that such project management mechanisms provided good coordination, structure and guidance for projects, as well as a recordkeeping and document management system, which they argued played a valuable role in promoting quality. On the other hand, some commented that it should be ensured that all ECs clearly communicate boundaries and role definitions. There should also be more in-depth constructive feedback and reflection by the team at the end of a project. Furthermore, the results of this reflection should be shared within the university to contribute to the development of best practice.

The view that the evaluation form used at the conclusion of projects and the CDD were of little value in promoting quality was echoed by the focus group participants. They explained that the form was sent directly to the lecturer by the EC and had to be returned to both the EC and his or her manager. The fact that the EC would see the lecturers' feedback meant that most lecturers would be reluctant to include negative comments about the performance of the EC and that shortcomings would, therefore, not be addressed by this measure. The CDD, in turn, was a simple, standardised form, whose purpose was not explained on the form itself, and, as such, served no direct purpose in enhancing quality.

Another issue raised in the focus group and interview sessions was the value of academic peers in promoting quality during the writing process. One participant described how materials in her department were written collaboratively by a group of lecturers, with a 'team leader' serving to collate and check all contributions and round off the final product. In their view, this process significantly contributed to the good quality of materials and assisted lecturers to complete the material within the required timeframe.

Several respondents and participants in the focus groups suggested additional quality mechanisms that could be introduced. These included

• providing opportunities for participants to raise any problematic issues with team members during rather than only at the end of projects

• arranging training interventions to ensure that ECs and other role players are at the same level, and that ECs and other role players work consistently and professionally with lecturers

• involving more critical readers

• piloting of materials among students

• providing more training interventions for academics, specifically on module design and writing skills

• introducing 'author writing weeks' where academics may go off campus with ECs and other experts to write part of their materials

• making a set of resources available to lecturers on an interactive online site

• allocating dedicated writing time or other writing incentives for lecturers

• providing academic peer support during the writing process, for instance, a second lecturer to moderate the materials, more than one critical reader or an academic team to undertake the writing, with the final product collated and refined by a team leader

• setting up module form repositories held by academic departments (on the institutional database).

The findings corresponded to a large extent with elements of quality assurance that were highlighted in the literature reviewed. For example, the allocation of sufficient resources for course development (Hansson, 2008; Butcher & Wilson-Strydom, 2013; Sankey et al., 2014) was considered essential for the success of development projects. The central role of the EC or instructional designer in assuring quality (Chao, Saj & Hamilton, 2010; Bawa & Watson, 2017) was identified, while the importance of peer reviewers (Holsombach-Ebner, 2013; Ahmad Zabidi et al., 2017) was emphasised. The same does not apply to standardised templates: frequently considered crucial in literature, survey respondents regarded them as somewhat less important, although it was indicated that they did have a role to play. While it was recognised that summative evaluation was essential in the quality cycle, it was recommended that the way it is implemented be improved. Formative reflective evaluation, seen as significant in one strand of the literature (e.g., Taylor et al., 2016), was a suggested in addition to the quality assurance process.

 

A SUGGESTED MODEL FOR QUALITY ASSURANCE IN COLLABORATIVE COURSE DEVELOPMENT

Based on the findings of the study, a model for quality assurance in collaborative course development is suggested that integrates the lecturers' perspectives.

The model distinguishes three levels at which quality assurance mechanisms should be implemented: (i) institutional management level, (ii) the level of the academic department, and (iii) the level of the courseware development department that provides educational consultancy or instructional design services (DCDT, in Unisa's case) together with support departments offering services such as language editing, graphic design, and web uploading. It also distinguishes three phases in which quality mechanisms are applied, namely (i) before, (ii) during and (iii) after the actual development process. A summary of the model is shown in Figure 1.

At institutional management level, attention should be paid to formulating an integrated quality management system for courseware design and development that cuts across departments and sections. While every department will have its own internal quality management system and arrangements, higher-level management should ensure that the standards for product and processes are consistent and consistently implemented among all the departments, and that work flows seamlessly from one department to the next. These measures should be integrated into the institution's quality assurance policies and procedures.

Institutional management should also ensure that sufficient resources are allocated in terms of

• Time - The courseware development timeframe should be long enough for role players to dedicate sufficient time to the development of materials of high quality.

• Professional development - Institutional arrangements should be in place to offer lecturers professional development opportunities in course development and ODL.

• Technology - The institutional learning management system should be sufficiently advanced to allow for flexibility and innovation in online course design and development.

Finally, the institutional quality management system should make provision for an evaluation of modules that will generate student feedback that is specific enough to inform the improved redesign of individual modules. This could be done centrally by the institution or by the lecturers themselves in specific modules. Providing feedback can be done anonymously.

At the level of the academic department, all possible measures should be taken to support lecturers in the courseware design and development process in order to improve the quality of courses. The department should ensure that lecturer workload is of such a nature that it allows the lecturers sufficient time to attend available training opportunities in writing and other ODL-related skills. Academics should serve as critical readers for their colleagues, but, in addition, further academic peer support should be offered for writing, for example, by using a second lecturer as a collaborator or mentor for every project, or having materials written by a group of authors with a team leader. Ideally, incentives should also be provided for lecturers to produce good quality writing, for example, allowing them dedicated writing time with a corresponding reduction of workload in other areas, or conducting writing workshops extending over several days. Academics who are part of the development team should also participate in the development team's formative and summative evaluation processes.

At the level of support departments, all the following mechanisms should be implemented: (i) project management for each development project, unless this is provided by the academic department; (ii) support with the course design and development process by ECs, including constructive feedback on materials; (iii) language editing; (iv) proofreading; and (iv) support with building and checking the module websites. All the service providers should have specific quality policies and standards guiding their activities and should conduct satisfaction surveys at the end of their processes.

The courseware development department should play a key role in developing and implementing quality mechanisms. Its contribution should include:

• A project management service, if this is not provided by the academic department. As part of the project management process, it should be ensured that work boundaries and the role definitions of the different role players are clearly communicated

• Providing training for their own staff members to ensure that all are at the same level and their work with academics is consistent

• I n collaboration with lecturers, where possible, compiling internal standards for course design and development, as well as any further guideline documents, checklists, and templates; and making all these resources accessible for lecturers online, as well as real examples of good course materials and module forms (course blueprints)

• Apart from providing advice on the curriculum and course design and development process, also giving extensive feedback on draft module forms and course materials

• Providing opportunities for team members to evaluate the design and development process both during the process and at its end (i.e., conducting formative and summative evaluation)

• Conducting a satisfaction survey with lecturer 'clients' and ensure confidentiality of lecturer feedback, where this is required

• Where possible, involving students during course design and development, for example, by piloting materials with students

• Facilitating a process of reflective feedback by teams at the end of projects and sharing the results of the reflection to promote good practices.

In summary, the most important recommendations for a quality assurance model in the collaborative course development process, based on lecturers' needs and perceptions, are that the institution should (i) make sufficient resources available for the process, particularly in terms of time, human resources, training, availability of resource documents and technological infrastructure; (ii) ensure that collaboration with academic peers and education consultants is maximised; (iii) obtain feedback from a range of role players, including students; and (iv) conduct evaluation and reflection both during and after the process.

 

CONCLUSION

The study supports the positive contribution of the collaborative course development process and highlights key quality mechanisms in the development process. Input and feedback from knowledgeable peers, inside and outside the university, are regarded as significant evaluative factors in ensuring that quality learning experiences are provided to the student. Such evaluations are to take place both during and after the process. The allocation of necessary resources is critical to the overall functioning of the design team, particularly to support the lecturers through effective collaborative procedures. The study confirms that careful consideration needs to be given to the implementation of relevant quality mechanisms in a collaborative course development process.

 

REFERENCES

Abdous, M.H. (2009) E-Learning quality assurance: A process-oriented lifecycle model. Quality Assurance in Education 17(3) pp.281-295.         [ Links ]

Ahmad Zabidi, N., Woo, T.K., Rajesh Kumar, P., Fadzil, M. & Syed Husain, S.H. (2017) Quality assurance in learning material development at OUM. Asian Association of Open Universities Journal 12(1) pp.69-81.         [ Links ]

Albashiry, N.M., Voogt, J.M. & Pieters, J.M. (2016) Curriculum leadership in action: A tale of four community college Heads of Department leading a curriculum development project. Community College Journal of Research and Practice 40(5) pp.401-413.         [ Links ]

Baldwin, S.J. & Ching, J.Y. (2019) An online course design checklist: Development and users' perceptions. Journal of Computing in Higher Education 31(1) pp.156-172. http://dx.doi.org/10.1007/s12528-018-9199-8        [ Links ]

Bawa, P. & Watson, S. (2017) The chameleon characteristics: A phenomenological study of instructional designer, faculty, and administrator perceptions of collaborative instructional design environments. The Qualitative Report 22(9) pp.2334-2355.         [ Links ]

Biggs, J. (2001) The reflective institution: Assuring and enhancing the quality of teaching and learning. Higher Education 41(3) pp.221-238. https://link.springer.com/content/pdf/10.1023/A:1004181331049        [ Links ]

Boud, D., Keogh, R. & Walker, D. (Eds.) (2013) Reflection: Turning experience into learning. London: Routledge Falmer.         [ Links ]

Bower, M. & Vlachopoulos, P. (2018) A critical analysis of technology-enhanced learning design frameworks. British Journal of Educational Technology 49(6) pp.981-997.         [ Links ]

Briggs, L.J. (1991) Instructional design: Principles and applications. Englewood Cliffs, New Jersey: Educational Technology.         [ Links ]

Bronson, A.O. (2016) Writing materials: Insights into course design and writing processes. New Directions for Higher Education 173 pp.55-64. doi.org/10.1002/he.20179        [ Links ]

Brookfield, S.D. (2017) Becoming a critically reflective teacher. San Francisco, CA: John Wiley & Sons.         [ Links ]

Butcher, N. & Wilson-Strydom, M. (2013) A guide to quality in online learning. Dallas: Academic Partnerships.         [ Links ]

Chao, I.T., Saj, T. & Hamilton, D. (2010) Using collaborative course development to achieve online course quality standards. International Review of Research in Open and Distance Learning 11(3) pp.106-126, doi.org/10.19173/irrodl.v11i3.912        [ Links ]

Council on Higher Education. (2014) Distance higher education programmes in a digital era: Good practice guide. Pretoria: CHE.         [ Links ]

Creswell, J.W. & Plano Clark, V.L. (2011) Designing and conducting mixed method research (2nd ed.) Thousand Oaks, CA: Sage.         [ Links ]

Darojat, O. (2013) Quality assurance in distance teaching universities: A comparative study in Thailand, Malaysia, and Indonesia. Unpublished PhD dissertation, Burnaby, Simon Fraser University, Canada.         [ Links ]

Da Silva, A.R.L., Diana, J.B., Catapan, A.H. (2015) Management and instructional design: Building intersections. US-China Education Review 5(2) pp.133-138. https://www.researchgate.net/publication/311480522_Management_and_Instructional_Design_Building_Intersections        [ Links ]

Debattista, M. (2018) A comprehensive rubric for instructional design in e-learning. The International Journal of Information and Learning Technology 35(2) pp.93-104, doi.org/10.1108/IJILT-09-2017-0092        [ Links ]

Dewey, J. (1986) Experience and education. The Educational Forum 50(3) pp.241-252, doi. org/10.1080/00131728609335764        [ Links ]

European Association for Quality Assurance in Higher Education (ENQA). (2015) Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). Brussels. https://enqa.eu/index.php/home/esg/

Ford, J.S. & Profetto-McGrath, J. (1994) A model for critical thinking within the context of curriculum as praxis. Journal of Nursing Education 33(8) pp.341-344.         [ Links ]

Freire, P. (1972) Pedagogy of the Oppressed; Translated by Myra Bergman Ramos. Harmondsworth, Middlesex: Penguin.         [ Links ]

Guri-Rosenblit, S. (2009) Distance education in the digital age: Common misconceptions and challenging tasks. Journal ofDistance Education 23(2) p.105. https://files.enc.ed.gov/fulltext/EJ851907.pdf (Accessed 10 July 2019).         [ Links ]

Hansson, H. (2008) E-learning Quality. Aspects and criteria for evaluation of e-learning in higher education. http://www.diva-portal.org/smash/get/diva2:283764/FULLTEXT01.pdf (Accessed July 10, 2019).

Herron, R., Holsombach-Ebner, C., Shomate, A. & Szathmary, K. (2012) Large scale quality engineering in distance learning programs. Journal of Asynchronous Learning Networks 16(5) pp.19-35.         [ Links ]

Holmberg, B. (2005) Theory and practice of distance education. London: Routledge.         [ Links ]

Holsombach-Ebner, C. (2013) Quality assurance in large scale online course production. Online Journal of Distance Learning Administration 16(2) https://www.learntechlib.org/p/155617/ (Accessed 10 July 2019).         [ Links ]

Kalantzi, R., Sideris, D., Spyropoulou, N. & Androulakis, G. (2016) Changing the gear: Adopting inter-institutional collaborative course development as the policy for distance higher education in Greece. In Proceedings of the Online, Open and Flexible Higher Education Conference, pp.61-68. Rome, Italy.

Karttunen, M. & Juusola, S. (2019) Maintaining quality online: Piloting an online language course for immigrants in Finland. Languages 4(2) pp.25.         [ Links ]

Keegan, D. (1996) Foundations of distance education (3rd ed.) London: Routledge.         [ Links ]

Keil, S. & Brown, A. (2014) Distance education policy standards: A review of current regional and national accrediting organizations in the United States. Online Journal of Distance Learning Administration 17(3) pp.15-26.         [ Links ]

Kolb, D. (1984) Experiential learning: Experience as the source of learning and development. Englewood Cliffs, N.J.: Prentice-Hall.         [ Links ]

Lenert, K.A. & Janes, D.P. (2017) The Incorporation of Quality Attributes into Online Course Design in Higher Education. International Journal of E-Learning and Distance Education 32(1) https://files.eric.ed.gov/fulltext/EJ1146391.pdf (Accessed 10 July 2019).         [ Links ]

Lewin, K. (1946) Action research and minority problems. Journal of social issues 2(4) pp.34-46.         [ Links ]

McKenney, S. & Reeves, T. (2014) Methods of evaluation and reflection in design research. Zeitschrift für Berufs- und Wirtschaftspädagogiek 27 pp.141-153. https://ris.utwente.nl/ws/files/7007413/McKenneyReevesZBW2014-PrePrint.pdf        [ Links ]

Mills, R. (2006) Quality assurance in distance education - Towards a culture of quality: A case study of the Open University, United Kingdom (OUUK). In B.N. Koul & A. Kanwar (Eds.) Perspective on Distance Education. Towards a Culture of Quality Vancouver: Commonwealth of Learning.         [ Links ]

Moore, M.G. (1993) Editorial: Is teaching like flying? A total systems view of distance education. American Journal of Distance Education 7(1) pp.1-10, doi.org/10.1080/08923649309526806        [ Links ]

Moore, M.G. & Kearsley, G.G. (1996) Distance education: A system view. Wadsworth: Cengage Learning.         [ Links ]

Nichols Hess, A. & Greer, K. (2016) Designing for Engagement: Using the ADDIE model to integrate high-impact practices into an online information literacy course. Communications in Information Literacy 10(2) pp.265-282.         [ Links ]

Obizoba, C., (2015) Instructional design models-Framework for innovative teaching and learning methodologies. International Journal of Higher Education Management 2(1) pp.40-51.         [ Links ]

Peters, O. (1993) Distance education in a postindustrial society. In D. Keegan (Ed.) Theoretical principles of distance education, London: Routledge pp.39-58.         [ Links ]

Puzziferro, M. & Shelton, K. (2008) A Model for developing high-quality online courses: Integrating a systems approach with learning theory. Journal of Asynchronous Learning Networks 12(3-4) pp.119-136.         [ Links ]

Rossi, R. & Mustaro, P.N. (2011) A simplified quality model for e-learning development and evaluation. In Proceedings of e-Learn World Conference on e-Learning in Corporate, Government, Healthcare and Higher Education pp.878-883.

Rucker, R., Edwards, K. & Frass, L.R. (2015) Assessing faculty experiences with and perceptions of an internal quality assurance process for undergraduate distributed learning courses: A pilot study. Quarterly Review of Distance Education 16(4) pp.35-44. https://search.proquest.com/docview/1786249868?accountid=14648 (Accessed 27 September 2020).         [ Links ]

Sankey, M., Carter, H., Marshall, S., Obexer, R., Russell, C. & Lawson, R. (2014) Benchmarks for technology enhanced learning. Canberra: Australasian Council on Open, Distance and e-learning (Acode). https://ro.uow.edu.au/asdpapers/549/        [ Links ]

Schön, D.A. (1987) Educating the reflective practitioner: Towards a new design for teaching and learning in the professions. San Francisco: Jossey-Bass.         [ Links ]

Taylor, C., Cockburn, J., Rouget, M., Ray-Mukherjee, J., Mukherjee, S., Slotow, R., Roberts, D., Boon, R., O'Donoghue, S. & Douwes, E. (2016) Evaluating the outcomes and processes of a research-action partnership: The need for continuous reflective evaluation. Bothalia-African Biodiversity & Conservation 46(2) pp.1-16.         [ Links ]

Thurab-Nkhosi, D. & Marshall, S. (2009) Quality management in course development and delivery at the University of the West Indies Distance Education Centre. Quality Assurance in Education 17(3) pp. 264-280.         [ Links ]

University of South Africa (Unisa). (2013) Framework for the Implementation of a Team Approach to Curriculum and Learning Development at Unisa. Internal Unisa procedure document. Pretoria: Unpublished.         [ Links ]

Venable, M., McKimmy, P., Gose, E., Hutchinson, N., Lavolette, E., Huang, P.E., Nakamura, P., McCann, K., Meeder, R. & Pusal, E. (2009) Distance course design and consulting Group (DCDC): Online course development in higher education. In Proceedings of ED-MEDIA 2009 World Conference on Educational Multimedia, Hypermedia and Telecommunications pp.517-522. http://www.editlib.org/p731549/proceeding_31549.pdf (Accessed 10 July 2019).

Wang, H. (2008) Benchmarks and quality assurance for online course development in higher education. Online Submission 5(3) pp.31-34. https://files.enc.ed.gov/fulltext/ED503008.pdf (Accessed 10 July 2019).         [ Links ]

Young, A & Hoerig, B. (2013) Utilizing student feedback to inform faculty development activities for online course development and delivery. International Journal on E-Learning 12(4) pp.439-453. https://www.learntechlib.org/primary/p/38464/ (Accessed 10 July 2019).         [ Links ]

Zuber-Skerritt, O. (2001) Action learning and action research: paradigm, praxis and programs. Effective change management through action research and action learning: Concepts, perspectives, processes and applications. Southern Cross University Press, Lismore, Australia

 

 

Date of submission: 16 March 2020
Date of review outcome: 11 August 2020
Date of acceptance: 9 October 2020

 

 

1 This paper was first presented at the 2019 National Association of Distance Education and Open Learning in South Africa (NADEOSA) conference.