Evaluation of academic review in Wales 2002

Introduction

1 This report outlines the findings of the evaluation of the academic review of subjects in Wales, conducted by the Quality Assurance Agency for Higher Education (the Agency) in 2002-03. The reviews, known as subject engagements, were conducted in fourteen higher education institutions (HEIs), addressing 19 subject areas at both undergraduate and postgraduate level. As the individual review reports are confidential to the HEI and the Higher Education Funding Council for Wales (HEFCW), this report has been published to disseminate the results and learning from the 2002 engagements.

2 This evaluation report begins by setting the context for these engagements, the review method and the outcomes. The evaluation process is briefly described, before outlining the key issues and learning points from the experiences of participants in 2002.

3 This report has been written primarily for colleagues who are involved in higher education and those who have an interest in the standards and quality of higher education programmes. The results of the evaluation contribute to the discussions and consultation that are taking place in the first part of 2003 in relation to the development of the new process of institutional review.

Background to the subject engagements

4 The plans for quality assurance in Wales were suspended in March 2001. This came about when the UK-wide arrangements were dissolved and new methods were considered in England. As the first HEFCW cycle of quality assessment was a considerable distance from the likely implementation of new methods for quality assurance, interim arrangements were needed. The circular WO1/90HE updated the higher education sector on a range of issues related to higher education (Appendix 1). The document stated that every institution would undertake from one to three engagements at subject level, with the key purpose of assisting institutions to develop their procedures and processes to reflect Agency mechanisms.

5 The engagements were based on the Handbook for Academic Review 2000 (the Handbook) but were intended to be a substantially lighter burden of assessment on the HEIs. As the aim of the engagements was developmental, the outcomes of the reviews are confidential to the HEIs concerned and to the HEFCW. The Agency developed a document entitled Academic Review in Wales 2002 (Appendix 2). This document outlined a method that was greatly restricted in terms of the time required, the judgements made and the overall burden placed on subject providers. The judgements were focused on academic standards (intended outcomes, curriculum, assessment and student achievement), on student progression, including the quality of student support arrangements, and on the quality of the learning resources supporting the programmes of study. In addition, the reviewers were asked to provide a commentary on the maintenance and enhancement of the quality of the subject. The institutions were required to prepare a self-evaluation document (SED), incorporating programme specifications, for each review.

6 Teams of three Subject Specialist Reviewers (SSRs) normally carried out the reviews. Management of each review and liaison with the institution was the responsibility of a Review Coordinator (CR). The team was normally in the institution for a maximum of two and a half days. The reviewers analysed the self-evaluation documentation and any other materials provided by the institution off-site, and then normally visited the institution for two and a half days.

7 As Agency officers started to work with institutions, it became clear that some additional explanation of the method of engagement would help institutional staff. Two briefing seminars were held on 26 April and 7 May 2002. HEFCW officers and at least two representatives from each institution attended. The briefings were planned to help subject providers understand the requirements of the engagements. At these seminars, an additional synoptic paper was distributed (Appendix 3). This addressed operational issues and the implementation for the processes for the engagements. The participants evaluated both sessions positively.

The outcomes of subject engagements

8 The range of subject engagements

Subjects	Number of engagements	Number of SSRs involved in engagements

Single reviews
Computing	5	15
Law	2	6
Sports Science	2	6
Architecture	1	3
Theatre Design	1	3
Nursing	1	3
Archaeology	1	3
Occupational Therapy	1	2
Business and Management	1	3
Social Policy	1	3

Aggregate reviews
Computing/Electrical Engineering	1	4
Business/Accounting	1	4
IT/Business and Management	1	3
Total	19	58

The outcomes of the subject engagements

Academic Standards	Confidence in standards	19

Quality of Learning Opportunities Student progression	Commendable	17*
	Approved	2
	Failed	0

Learning resources	Commendable	13
	Approved	6
	Failed	0

*including the identification of exemplary features in Student Progression in one institution and exemplary features in Learning Resources in another.

Evaluation approach

9 The evaluation is based on the established Agency evaluation framework and draws upon a number of sources:

postal evaluation questionnaires;
visit support monitoring logs: these are records made by Agency officers when attending the reviews;
minutes/notes of meetings including the briefing seminars;
oral feedback from participants;
the Agency's web folders: these are electronic files of the review visits including documentation created by the review team;
the evaluation seminar in January 2003;
review judgements and reports.

10 The evaluation questionnaires were sent to the three key groups involved in each engagement: the HEIs, including the Institutional Nominee (IN); the CRs and the SSRs. The percentage returned was 74 per cent, 53 per cent, and 72 per cent respectively, with an overall total return of 69 per cent.

11 Most of the responses are given in a four-point scale ranging from 1 (strongly disagree) to 4 (strongly agree). The analysis provides insights into the level of agreement between the three groups and the level of satisfaction with the different aspects of the review. The majority of responses were positive, either 3 (agree) or 4 (strongly agree), 79 per cent for the HEIs, 86 per cent for the CRs and 88 per cent for the SSRs.

Evaluation results

The Handbook

12 The participants had four documents for reference: the Handbook for Academic Review, the Circular WO1/90HE, the Academic Review in Wales 2002 document and the synoptic paper. This number of documents increased the likelihood of staff, particularly at subject level, becoming confused and this happened in some institutions. The evaluation data indicated that HEIs would have benefitted from more preparation and support from the the Agency for writing SEDs.

13 Dissemination from the briefing seminars was more effective in some HEIs than others. This had some effect on the standard and structure of SEDs. A few of these reflected the process of subject review in England 1998-2000. In these cases, HEI staff had not fully understood the shift from evaluating the quality of student experience through a grading system (1998-2000) to one of consideration of standards and quality against self-evaluation and external reference points. The majority of HEIs considered the seminars helpful and relevant. The evaluations taken at the end of both events confirmed a high level of satisfaction with the content and approach of the briefings.

Schedule of visits

14 The HEIs were offered three possible modes for the engagement: an intensive visit of two and a half days, with the final half day devoted to meetings to confirm team judgements which were then conveyed to the institution in an informal oral feedback; or two days then a later half day for the meetings about judgements; or two and a half days used flexibly within a four-week period. The first two approaches had the advantage of brevity and less disruption for the institution and the last provided both the subject staff and the review team with greater opportunity to reflect upon and consider the information shared.

15 The number of HEIs that chose each of the modes was nine, six and four respectively. Most HEIs preferred a concentrated period of peer review activity.

Self-evaluation documents

16 These varied in quality and comprehensiveness. The evaluation feedback indicated that institutions were less than satisfied with the guidelines for the reviews (Appendix 2). This may have been linked to the multiplicity of documents, some published and some not. For future methods of review, a single published document, widely available to the HE sector and subject communities, will be preferable. Several institutions took the opportunity to directly reference key institutional and departmental level documents in the SED. This worked very well in practice as these references enabled reviewers to find documentation and evidence quickly and efficiently.

17 It soon became clear to the Agency that if HEIs supplied additional documentation with the SED, this would help the reviewers to prepare prior to the visit(s). Where institutions were careful and focused in their choice of additional material and where their SED had clear and direct references to materials, the whole process was more efficient and less intrusive.

18 Several HEIs made very good use of information technology to provide reviewers with information, for example, CD-ROM and web-based materials.

Engagement activity, process issues

19 The analysis of the questionnaires showed that the majority of participants were positive and satisfied with the process. The institutions generally found that the review teams were constructive and understood the developmental nature of the engagements. The approach of peer review against a critical and evaluative SED was clearly successful. The feedback indicated that the HEIs found this process more collegial than their early experiences of external quality assurance.

20 The length of time (2.5 days) for the engagements was sufficient for the review. The CRs and the SSRs expressed a high level of satisfaction in the evidence they gathered and the judgements they made. Institutions were satisfied with the communication between the review team and the HEI.

21 The HEIs provided the reviewers with materials that were adequate and appropriate to inform the judgements and the commentary on the maintenance and enhancement of quality. The material was particularly helpful if it reflected the SED and was focused in nature. This element of control and restraint was also important in the sample of student work provided by the institution. HEIs and review teams worked best when the sample was relevant and contained no more examples of work than were necessary to demonstrate the process of assessment, and the students' achievement of the required standard for the award.

Review teams

22 Review teams were normally composed of three reviewers in each subject area, selected to reflect the broad Joint Academic Coding System (JACs) codes of the subject. Each institution identified the key aspects that they were concerned that the review team should reflect and had an opportunity to comment on the proposed team membership.

23 Institutions varied in the extent to which they requested that review teams had an awareness of higher education in Wales. Wherever possible, at least one member of the team had direct experience of working in an HEI in Wales, or indirect experience such as validation panel member or external examiner. Many institutions thought it more important that the ethos of the HEI was reflected, for example, research-oriented institution, rather than direct experience of providing higher education in Wales.

Review Coordinators

24 CRs manage reviews and liaise with the institution, they do not act as subject specialists. The Agency introduced CRs to the key issues of engagements in Wales at the twice-yearly conferences which the Agency arranges. All CRs received the three guidance papers (Appendix 1.2.3). An Agency Assistant Director and an administrator were named contacts for the engagements. Fifteen CRs managed the 19 reviews. Their coordination of the reviews was generally well received by institutional staff and reviewers, but there were some concerns about the consistency of the conduct of the engagements. This was particularly the case where more than one engagement took place in an HEI and where each CR adopted a different approach. In each case, the guidance in the Handbook was followed, but the interpretations of the scope of the role varied.

Institutional Nominee

25 This role was introduced in the later stages of planning the engagements (Annex D, Appendix 2). The role was optional for the HEI. The person nominated was expected to have a thorough knowledge of their institution, its protocols and regulations as well as extensive knowledge and experience of HE and quality assurance. The Agency provided written briefing materials for INs and, in addition, the majority of INs attended the briefing seminars. All HEIs chose to have an IN, but the extent of their involvement in the engagement and particularly at judgement meetings varied. This was partly due to the interpretation of the role by some CRs. This was sometimes a reflection of their anxiety about that level of involvement of a member of the HEI in the process of the engagement. This individual interpretation of such a role needs careful risk assessment and management for future review processes in Wales.

26 The analysis of the questionnaires and other feedback found that the IN role was considered a very positive aspect of this activity and well received by review teams and institutions. In the briefing seminars, participants were not sure that INs should be present at the meetings when reviewers were discussing and confirming judgements. The main reasons given concerned the potential impact of the IN on the process of making judgements. Secondly, some participants considered that the IN would be under considerable pressure from colleagues in their institution both before and after judgements, particularly if the judgements were negative. The evaluation shows that where the HEIs and the review teams followed the written guidance provided by the Agency, the predicted problems of the INs' attendance at judgement meetings were not realised.

Reports

27 In all 19 engagements there was confidence in standards. There were 17 judgements of commendable for student progression and two approved. There were 12 judgements of commendable for learning resources. The reviewers also noted two exemplary features, one in student progression and one in learning resources, each at a different institution. As the subjects and programmes reviewed were so diverse, no conclusions can be reached about the standard or state of subjects.

28 The comments below are drawn from an analysis of the reports:

Academic standards

29 The intended learning outcomes specified by subject staff were usually clear and relevant, informed by external reference points. The outcomes were generally well communicated to students and other stakeholders.

30 Review teams found that curricula are generally current and supported by research and other scholarly activities of staff. The programmes were considered to have appropriate content for the level of the awards.

31 Reviewers found that assessments are well matched to outcomes, with assessment strategies and activities that are rigorous and secure. Some review teams commented that feedback to students could be improved and/or be more consistent in terms of rigour and helpfulness to the students. Several reports also commented that internal moderation strategies, including second marking, could be improved. Where good practice was evident in such strategies, this was of a particularly high standard.

32 Student achievement demonstrated added-value for many students. This aspect of standards was generally good, being at an appropriate level for most programmes.

Student progression

33 There was wide spread evidence of good practice of supporting students from their entry to the completion of their programmes of study and beyond. This positive outcome concerns both academic and pastoral support. Students reported that staff are consistently accessible, positive
and helpful.

34 A clear message from the reports on the engagements is that HEIs in Wales are actively engaged in widening participation. The reviewers found that staff understood the importance of support tailored to the needs of students from non-traditional backgrounds and met these
needs well.

35 The reviewers raised questions about the effectiveness of the management information systems in several HEIs. Reviewers found that HEIs need to improve the quality and breadth of the statistical information on students, and to enable both subject staff and the institutional administrative staff to share and use the information more effectively. This ability to provide clear and accurate information is important for the institutions' own management of standards and quality, as well as for future review processes.

Learning resources

36 The reviewers commented positively on staff qualifications and expertise. They reported that academic, administrative and other support staff are suitably qualified and knowledgeable. Generally, HEIs are able to provide support for staff development, promoting professional as well as educational development.

37 There are mixed messages about the learning resources of libraries and computing in the HEIs. While all institutions have a challenge to maintain and improve the quantity and quality of library stock, some institutions succeeded while others did not meet their students' needs as well. There was no discernable pattern within subjects or institutions as to why the level and quality of provision was so variable. Information and communication technology also varied, both for the students' direct use and when supporting teaching and learning processes.

38 The reviewers clearly identified pressure on staff and technical resources in several HEIs. They attributed this pressure to the increasing numbers of students in many subjects, and the lack of a corresponding increase in staffing and other resources.

Summary of key issues and recommendations arising from the engagements

39 The engagements in Wales consistently identified many positive messages about the standards and quality of higher education programmes in Wales. All subjects received judgements of confidence in standards and commendable or approved in the quality of the learning opportunities.

40 The programmes reviewed demonstrated considerable strengths in relation to learning outcomes, curriculum, assessment and achievement, and particularly to widening participation. Assessment remains the area of standards where the reviewers identified the most scope for improvement. In particular, the reviewers highlighted the need for improvements in second marking or internal moderation.

41 The reviewers found consistently very good levels of academic and pastoral support for students. Increasing student numbers continues to be a challenge for some HEIs, especially where there has not been a corresponding increase in staffing and other learning resources. In these cases, staff are under pressure to sustain this very good level of academic and pastoral support.

42 The overall commentary on maintenance and enhancement of quality and standards was positive and the HEIs were well aware of their responsibilities. Some institutions are advised to review their management information systems to ensure that programme and subject staff as well as administrative staff receive information which helps and supports them to achieve their institution's mission.

43 The reviewers find the use of references in SEDs very helpful. Where HEIs also cross-referenced to documents available on the institutions' web sites, CD-ROM and intranet or in the workroom for reviewers, the engagements with their external peers worked particularly well.

44 The role of the IN was welcomed by most reviewers and HEIs. Many recommended that this or a similar role should be included in future methods of review.

45 Preparation for the method of Institutional Review in Wales needs to be thorough and tailored to meet the needs of HEIs in Wales. The participants at the evaluation seminar recommended: workshops to assist with the preparation of SEDs; more guidance on documentation to be provided by the HEI both before the review and during the process; and more preparation for the role of the IN including training shared by reviewers and INs, with the benefit of ensuring a greater appreciation of the Wales higher education context during training.

Conclusion

46 The subject engagements in Wales were approached with a mix of some reservations and some enthusiasm by the HEIs. The aims for these activities were realised and achieved; the HEFCW, the Agency and the HEIs worked together to ensure that the subject engagements in Wales were a positive experience. This report has highlighted the key issues and learning points arising from the reviews and evaluation. There are lessons which are applicable to the new method of Institutional Review under development.

47 The successful completion of the process is in great part due to the hard work and enthusiasm of the HEIs' subject staff and students, and of the review teams.

Next>> Appendix 1