Academic review of subjects in HEIs – 2002-04: summary report

1 As part of the transitional arrangements in England during 2002-03 and 2003-04, QAA undertook 46 academic reviews in 14 subjects ¹ in higher education institutions (HEIs). This report identifies the key messages that emerge in terms of strengths and areas for improvement, and presents a synopsis of key points from the final evaluation of the review programme. Whilst the initial analysis of published reports was undertaken by subject, this report does not generally highlight features of provision for each subject area, although some illustrative references are made.

2 The outcomes of this review and evaluation process, and discussion of these in a number of fora, will inform project group discussion of principles and possible models for the future academic review of higher education in further education colleges (FECs).

Academic standards

Intended learning outcomes

3 Overall, institutions are clearly working with external reference points at the subject level in identifying intended learning outcomes for programmes. A significant number of programmes relate well to internal and external reference points, including the institution’s mission, subject benchmark statements and the Framework for higher education qualifications in England, Wales and Northern Ireland. Almost half the reviews report the appropriateness of the intended learning outcomes to the aims and level of awards as a strength. This is particularly evident in history. The fact that this is a positive feature in so many reviews is a measure of the extent to which institutions have embraced this area of quality management and enhancement.

4 A strength indicated in a small number of reviews is the way intended learning outcomes are communicated to students; for example, the use of matrices linking outcomes to specific modules.

5 Despite these strengths, the reports indicate a number of areas where further work needs to be done, such as in the differentiation of outcomes between pathways or levels, in representing the outcomes in a clearer and/or more consistent manner in programme specifications, and in a clearer relationship between module and programme outcomes.

Curricula

6 The judgements on curricula largely reflect the individual nature of the subject areas, with repeated comments tending to be of a generic nature. Almost half of the reports comment on coherent curricula, which have been developed with due consideration of external reference points, are progressively demanding and enable achievement of the learning outcomes. Such comments are in line with those in relation to learning outcomes and external reference points.

7 Substantial recognition is made in the reports of the positive impact of staff research and scholarship on curricula, particularly in history and English. Staff expertise significantly enriches the curricula, and the student experience is underpinned and enhanced by staff involvement in research. The curricula are also refreshed and kept current in a variety of ways, including internal staff discussion, consultation with external academics and industry, and staff involvement in a number of external organisations. In vocational areas, a high level of professional body exemption is mentioned as an indication of the vocational relevance of the curricula.

8 The ability of curricula to develop transferable skills in undergraduate programmes is highlighted in reports across the subject areas, and particularly in law. Breadth, appropriate balance of core and option modules and flexibility in curricula are strengths noted in four subject areas. Other reports indicate subject specific strengths in curricula such as the good use of fieldwork, live projects or work placements and relevance of modules for the local community, as well as praising individual programmes.

9 No single aspect of the curricula is repeatedly cited as an issue, suggesting that curricular design is satisfactory across the subject areas reviewed. However, the mapping of skills, formal links with employers and management of curriculum choice for students are encouraged in more than one report. Limited evidence of staff research informing the curricula is highlighted in two of the accountancy reviews.

10 Individual criticisms across the reviews include the quality of fieldwork undertaken, limits in curriculum design in relation to cognitive and transferable skills, and the balance between technical and theoretical content and specific aspects of the subject area (for example, finance).

Assessment

11 The wide range of assessment methods is praised across the reviews, although one report cited the need to increase variety, particularly in the assessment of transferable skills. Formative and summative methods have been chosen to motivate students and staff guidance before and after assignments. Written feedback on student work is highlighted as thorough, timely and constructive in just over a quarter of reports. Occasionally, individual mention is made of clear links between assessment criteria and feedback and student satisfaction. The clarity of assessment criteria and their communication to students is mentioned as a strength in only four reports, however. Good practice in marking and robust, rigorous assessment and moderation processes are frequently mentioned.

12 Other examples of good practice identified include clear assessment strategy at the subject level, progression in the complexity of assessment tasks, appropriate consideration of student assessment workloads, assignments mapped against intended learning outcomes and clear systems for the submission of assignments.

13 The area cited most frequently as an issue relates to the timeliness and consistency of feedback provided for students, closely followed by the need to make assessment methods and criteria more explicit to students. The need for a coherent assessment strategy and an explicit link between assessment criteria and learning outcomes is mentioned in six reports. The need for more rigorous internal moderation or simply to extend the good practice is identified in a small number of reports.

14 Other points identified for further development include the need to use the full marking range, to address the excessive student assessment workload, to involve external examiners in moderation and approval of assessment at the design stage, to ensure consistency in the application of late submission of assessments, and scope for improvement to ensure the integrity of the assessment process.

Student achievement

15 Overall, student achievement is highly satisfactory and appropriate to the level of the awards, with over half of the reports acknowledging this as a strength. External examiners endorse such achievement. In five instances, the achievement of master’s students and sandwich students is highlighted. Furthermore, in seven reports, the achievement of students from non-traditional academic backgrounds demonstrates significant added value. As a result of this positive picture of achievement, progression to employment is also noted as a strength, with one instance of praise for the comprehensive monitoring of destinations.

16 The low level of student completion and achievement is cited as an issue in three cases. However, the need to keep completion rates, degree classifications and achievement at levels one and two under review is also mentioned, as well as the fact that weak students are not always achieving the learning outcomes of modules. Such individual references across the reviews indicate that there is no room for complacency, despite the strengths noted above. Concerns relating to progression, and HEIs’ response to these, are discussed below in paragraphs 23-24. The achievement of evaluative and analytical skills is mentioned as needing particular attention in a small number of reviews.

Quality of learning opportunities

Teaching and learning

17 A key message emerging from a majority of reports is the high quality of teaching and learning experienced. Reports confirm that students are enthusiastic and highly motivated where high-quality varied and challenging teaching enables them to learn effectively. In these provisions, a good variety and range of methods are experienced, well matched to intended learning outcomes and providing students with a well-balanced and stimulating work environment. Interdisciplinary and interprofessional learning are a frequent indicator of vibrant teaching and learning. Moreover, there is strong student support for wide-ranging teaching methods conducted in very supportive learning environments. The regular review by course teams of their teaching methods in the light of student feedback and staff experience is a contributing factor in the maintenance of strength in teaching and learning. However, in a few cases, the lack of an explicit strategy to underpin the approach to teaching and learning, within and across programmes, is a weakness that leads to a wide range of teaching standards, which is indicated by student feedback.

18 Encouragement of student participation and student-centred learning are frequently recorded as important elements in the learning process, well supported through directed study, integrated case studies, dissertation and tutorial classes. Student-focused delivery methods are greatly appreciated by students. Year long modules promote deep learning and formative learning. Current and former students are positive about the quality of teaching, particularly their participation in seminars where they experience the dedication and approachability of academic and support staff. They are also positive about their development of oral presentation skills and group working; student-led assessed seminars was noted as one exemplary feature. The learning experience of students is enhanced by the effective use of postgraduate students and staff as mentors and on help-desks. Where there is a significant commitment to placement provision the quality is often high, with well-organised work structured to promote experiential learning. The preparation for, and support of students during placements is generally effective. However, there is scope to strengthen student guidance and support for the process of gaining placements in some provisions.

19 Where there is a strong research emphasis, the most able students are appropriately challenged and have ample opportunity to work at the frontiers of knowledge. A number of reports record provision in which staff research, scholarship and professional activity inform their teaching, injecting it with enthusiasm and energy. Many successful programmes successfully enable students to develop progressively to autonomous and, therefore, lifelong learning.

20 The quality of supporting materials and resources is an important accompaniment of high-quality teaching and learning. Many providers demonstrate the use of informative documentation. The quality of module handbooks is often considered to be good by students, staff and external examiners. A further frequently recorded feature is the use of information technology (IT) in teaching and learning, successfully underpinning programmes through the effective use of electronic learning materials. There are also a number of initiatives recorded of on-line learning and academic support. These include effective learning support materials on distance-learning degrees and for study skills to support undergraduates.

21 A broad range of university-wide staff development sessions available to support teaching and learning is frequently associated with strength in this area. Participation of academic staff in the national Teaching Fellowship Research Project is also recorded. Substantive commitment to developing teaching and learning is associated with the effective use of peer review and staff development practice that ensures the maintenance and enhancement of a high quality of teaching and learning. However, the arrangements for teaching observation are not always rigorously observed and sometimes there are no arrangements to share good practice.

Student progression

22 Excellent academic and personal support of students is a widely characteristic feature of HEIs. Many reports record that academic and pastoral support is comprehensive, extensive and very effective. Current and former students speak highly of the academic support that they receive. They report relaxed and friendly staff-student relations and staff ‘open-door’ policies. Effective learning is also promoted by the central contribution of fieldwork, for instance in geography and environmental sciences. Effective personal tutor systems inevitably underpin student academic and pastoral support, consistently applied at all programme levels. These are a crucial contribution to the effectiveness of the widening participation vision and agenda. The use of IT opportunities for support is increasingly characteristic of the best providers. There are a few reports of arrangements for academic guidance being not always clear to students.

23 In the best provisions, undergraduate progression and completion rates are very good, with high retention and low withdrawal rates. A quarter of reports cited the recruitment of a diverse body of students reflecting widening participation strategies, attracting large and growing bodies of appropriately qualified students. There are good retention rates on some part-time programmes. However, high rates of attrition on many courses are a significant area of weakness. Most often, this is represented by poor progression at Level 1 but, occasionally and more worryingly, this extends to Level 2 as well. This includes students who enrol but fail to attend. Reviewers perceive wider access policies related to poor rates of retention and progression as a cause for concern. Cases where there are no published admissions criteria for non-traditional entry students are likely not to promote good practice in this area. There are also recorded instances of a failure to collect and analyse data at programme level to identify reasons for non-completion and to inform future actions. Also, students in full-time employment cannot be expected to progress satisfactorily if they are on full-time programmes. A lack of overall monitoring of student performance is associated with poor-quality data and poor presentation of statistics relating to student progression.

24 Reports record commendable attempts to introduce strategies to improve retention rates. These often involve the effective monitoring of student withdrawals and comprehensive careers advice, while the support of students’ employers can be an important factor in students’ progression from Higher National Certificates or Diplomas (HNC/Ds) to degrees. In the latter case, a long history of successful links between HEIs and their local FECs is a factor that helps to ensure high rates of progression and completion. The appointment of student liaison officers is an innovative addition to the support retention systems. Also, induction programmes that are sensitive to individual student needs have a positive effect on retention in the early stages of Level 1. There is also scope recorded for more proactive student support systems. There are cases of postgraduate students being well supported from application to induction by clear documentation and appropriate services, while flexible attendance modes are of great benefit to students with changing personal circumstances. Students experiencing staff as approachable, helpful, and constantly aware of how they are performing also supports good retention rates. This also helps students from a variety of academic backgrounds to develop their full potential. The support for students from overseas, for which English is a second language, and support for disabled students and their special assessment requirements, are sometimes commended.

Learning resources

25 Academic staff are widely considered to be highly committed and student-centred in their approach to teaching. They are normally well qualified, both professionally and pedagogically, research active and appropriate in number for the provision. They are well matched to curricula need, benefit from appropriate staff development and receive good technical and administrative support, although in the modern languages review, difficulty was reported in academic staff provision. Reports frequently refer to strong, well-funded support for staff development, including part-time staff. However, occasionally, there is no evidence of a strategy enabling a department to overcome the difficulties posed by long-term staff absence, and in one case reviewers were concerned about the heavy reliance on hourly-paid staff. In another university, reviewers found no systematic appraisal for all staff to inform development and planning. Problems with the induction and support of part-time staff are also recorded.

26 In general, there are clear strategies for the maintenance and development of learning resources. The availability and range of learning resources responds to student needs and a majority of reports commend learning resource centres of good quality. The subject-specific book stock and electronic journals provide high-quality and current access to subject developments. Library collections are recorded as well matched to student needs, together with library staff that students experience as enthusiastic, professional and supportive. In a few provisions, the currency and range of book stock need improvement, with one instance where the resources available were deemed insufficient to meet students’ needs.

27 Good IT and information resources support the achievement of intended learning outcomes in most cases and include an extensive range of relevant
e-journals and databases, well supported by specialist staff. Only in a few cases are more effective strategies needed to encourage the better use of electronic resources. More often, the comprehensive range of support facilities provided include specific ‘skills zones’ to address student help needs. Setting aside quality areas for students to undertake independent and group work has been beneficial in several institutions. There is good student access to learning centres, which is further enhanced in cases where there is membership of UK Libraries Plus. Reports record the increasing use of virtual learning environments (VLE), while professionally equipped video-conferencing facilities connect dispersed campuses.

28 Students benefit from a typically high standard of physical environment for teaching, learning and private study. There are frequent examples of new, purpose-built buildings equipped with excellent laboratory resources in the case of science provision. Specialist teaching facilities, including an appropriate range of field equipment and well-equipped laboratories for earth sciences, workshops for engineering, and language suites for modern languages, are usually reported to be of high quality. Industry partnership with local manufacturers and other links with local companies provide an industrial focus to studies for students who wish it. Teaching accommodation is sometimes overcrowded, with planned enhancement of accommodation not being realised, indicating the need for a more coherent and strategic approach to a department’s accommodation needs.

Maintenance and enhancement of quality and standards

29 A majority of HEIs have clearly defined and robust systems of quality maintenance and enhancement, applied systematically and thoroughly to programmes. Consequently, they are able to monitor and assure the quality and standards of their provision. Reviews generally recorded appropriate and well-documented procedures and effective regulatory frameworks at discipline level, with clear lines of communication between all levels. These features give confidence that standards and quality can be maintained and enhanced in most institutions. However, in a few universities, although comprehensive quality procedures are in place, there is some variability in their application. One university recognised that its quality management mechanisms needed revising and was carrying out a major overhaul of its quality systems. In another, quality processes were in a state of transition, withy quality maintenance mechanisms currently stronger than quality enhancement ones. Sometimes, the reviewers recognised that restructuring has given rise to a review of quality and enhancement procedures and a redrafting of policy documentation that was currently incomplete. The management of relationships with partner colleges is often an important consideration. Reviewers note that, in a few cases, colleges operate largely independently with little academic interaction.

30 In a few institutions, while there is an extensive and thoroughly documented system of quality review and enhancement at university level, there is less evidence at programme or unit level of critical reflection and response to quality issues. Aggregative evaluation of student achievement at module level would enhance current procedures. There is sometimes the need to ensure that quality assurance procedures are fully implemented at subject level and owned by all staff. In these cases, there is a failure at university level to ensure that appropriate actions are taken to remedy identified shortcomings at programme level.

31 Part of an effective quality maintenance and enhancement system is the use of staff, student and external examiner feedback in self-evaluative processes, with proper attention being given to the views of students. Sound mechanisms in place for seeking student views, with appropriate action often being taken in response to them, are commonplace. In four provisions, reviewers note that a more formalised system of staff-student consultation would be beneficial. On occasions, the effectiveness of student representation is inconsistent, with staff-student liaison committees not showing systematic reporting back of actions taken. The legitimate concerns of external examiners and professional bodies generally, but not always, receive prompt and full attention.

32 The reports focus little on the enhancement agenda. In a few cases there are effective mechanisms for disseminating good practice, including peer review and mentoring, but there is often scope for improving academic quality and improvement action plans.

Evaluation of the programme of review

33 In November 2004, QAA finalised a summary report based on both quantitative and qualitative data captured from questionnaires and focus groups in 2002-03 and 2003-04. Information was collected from institutions, subject specialist reviewers and contract reviewers.

34 Although we would have welcomed a higher response rate, those that completed the questionnaires were, on the whole, satisfied with all aspects of the academic review process. However, scope for improvements was identified. A number of these points have been addressed in the revised version of the Handbook for academic review: England, 2004 for the review of directly-funded HE in FECs (published January 2005). We are also looking at ways of balancing institutions’ wish to engage in a negotiated, bespoke process with their wish for prescription in some areas.

35 Out of 46 reviews, 45 were awarded ‘confidence’ judgements for Academic standards; there was one ‘no confidence’ judgement. For the Quality of learning opportunities - with three separate judgements awarded for Teaching and Learning, Student Progression and Learning Resources - only one ‘failing’ judgement was awarded (for Student Progression). All others were ‘commendable’ (the overwhelming majority) or ‘approved’.

36 This indicates a very positive picture within the scope of this review programme. Colleagues may wish to note the:

effective use of the Academic Infrastructure in setting and maintaining academic standards and quality
positive impact of staff involvement in research in underpinning and enhancing the student experience
high-quality teaching and learning facilitated by appropriate strategies and wide-ranging methods, supported by good quality documentation and materials
excellent academic and personal support
well supported staff development
learning support resources, including IT, library services, specialist facilities and the physical environment
strong, robust systems for the maintenance and enhancement of standards and quality that generally seek and make good use of evaluative feedback.

37 Areas for improvement or continued careful attention include the need to:

differentiate outcomes between pathways or levels more clearly, and to clarify the relationship between module outcomes and programme outcomes
improve the timeliness and consistency of feedback to students on assessed work and to make assessment methods and criteria more explicit to students
monitor retention and progression systematically and to develop explicit strategies to address problems
apply good staff development practice evenly amongst all categories of staff, and ensure that peer observation schemes are rigorously observed to promote dissemination of best practice and secure confidence in the quality of teaching
secure ownership of quality assurance and enhancement strategies and processes at the discipline level and in the context of partnership arrangements.

1 Accountancy; Architecture; Business and Management; Computing; Drama; Earth, Environmental Sciences and Environmental Studies; Engineering; English; Geography; History; Law; Modern Languages; Music; Social Policy and Administration and Social Work.

John Barkham

Alexa Christou

Susan Hill

30 November 2004