Conclusions
In the early review period, the reviewers expressed concern over institutional arrangements for ensuring the quality of provision. In some examples they noted that clear and comprehensive quality frameworks did not exist, and, in others, that they were not fully developed or implemented. There was frequent reference to an over-reliance on informal quality procedures that lacked rigour and were insufficiently robust. In subsequent periods, the reviewers saw clear evidence of development, with most institutions establishing effective quality management systems. However, even in the final round of visits, the reviewers saw examples where comprehensive, rigorous and robust institutional quality management systems existed but were not being applied.
Throughout the review cycle, the reviewers saw excellent examples of the annual monitoring of courses, leading to detailed annual reports drawing on a wide range of evidence. Such good practice was far from universal and, in the most recent round of visits, the reviewers noted instances where annual monitoring was not undertaken and reports not produced. Where annual monitoring was conducted, the reviewers identified widespread deficiencies in the evidence used, or in failure at institutional level to take appropriate action on recommendations made in reports.
A strong feature of quality management systems was in eliciting the views of students. This was done through questionnaire surveys and student participation in a number of committees. The reviewers usually commended the way staff respond to the concerns of students, and noted that an increasing number of institutions provided training for student representatives.
In every subject overview report, concern over the operation of the external examiner system was expressed. The reviewers saw evidence of excellent practice, but also significant deficiencies in the examining process. There were concerns about the briefing of external examiners, wide variations in the content of their reports, of reports not submitted, of external examiners not involved in the assessment process and the failure to take action following recommendations made by external examiners.
The reviewers saw good examples of the involvement of professional bodies and employers in the provision of courses. In the best examples, representatives of these groups met regularly in liaison committees with detailed records of the discussion and decisions. However, in many reports there was evidence that many subject teams did not consult with the professions or employers.
With respect to staff development activities, there was clear evidence of substantial development. Staff development strategies became commonplace, whereas at the commencement of the review period they were rare. Similarly, staff appraisal and peer observation of teaching were rare, but became more widespread across the sector. In early reports it was noted that little effort was made to disseminate good teaching/learning practice. The use of peer observation, teaching forums and the work of learning and teaching committees led to wider dissemination of good practice.
Throughout the review process, the reviewers generally commended the arrangements for the induction, mentoring and training of new full-time staff. However, they also noted that these arrangements were rarely extended to part-time staff or research and postgraduate students engaged in teaching. This was less often the case in visits conducted at the end of the review cycle.
Conclusion
Subject review in England and Northern Ireland between 1993 and 2001 was the largest and most comprehensive exercise of its kind ever undertaken. Some 2,904 subject review reports and 62 subject overview reports were published.
In 1994, after reviewing the initial period of operation, HEFCE restated the purpose of subject review. This chapter considers the extent to which these purposes were achieved and in doing so it highlights some of the positive attributes of HE in England and Northern Ireland. It goes on to consider the changes to the design and delivery of HE programmes, and then outlines some of the challenges remaining. It concludes by noting how the sector has responded to the issues identified in subject review reports.
The aims of subject review
In 1994, following the first round of subject review (1993-94), HEFCE reconsidered the operation of the process in the light of extensive consultation with the sector and through internal evaluation. As a result, the method was modified restating the purpose of subject review as being: to ensure accountability of public funds; to provide a link with funding to enhance quality; to provide accessible public information; and to share and publicise best practice.
In terms of the first stated aim of ensuring accountability of public funds, the cycle of subject review confirmed that the overwhelming majority of institutions were providing students with programmes that met expectations for quality and standards.
The second aim set for subject review, to provide a link between funding and the enhancement of quality, was met to some extent. Throughout the cycle, HEFCE retained the sanction to remove funding from providers where they were not achieving their own aims or objectives. There were 16 reviews where, on the first visit, reviewers found that providers were not meeting their own aims or objectives.
However, on re-review, the vast majority of major shortcomings had been rectified and the provision was labelled approved. In only one case did a re-review lead to a second non-approved judgement. Neither HEFCE nor the Department for Education and Learning and its predecessors in Northern Ireland ever used subject review outcomes to direct extra funding specifically to high-scoring providers. Instead, they established a Fund for the Development of Teaching and Learning where institutions were able to bid for project support. These projects, usually of two years' duration and monitored by external advisers, were intended to give a boost to the dissemination of good practice in teaching and learning. They were superseded in 1998, by the setting up of 24 Learning and Teaching Subject Networks that were intended to provide long-term curriculum leadership in the subject areas.
The third main purpose of subject review was to provide accessible public information. This purpose has been achieved with the publication of all subject review reports since 1995. These are now available to prospective students and other stakeholders. In addition, 62 subject overview reports have been published. This achievement was recognised by Government in March 2001, when the Secretary for Education, David Blunkett, confirmed the end of universal subject review. In doing so, he noted at HEFCE and the Agency 'have now published a great deal of valuable information' which 'has rightly attracted international respect'. Reports have been made available on the Agency's website (www.qaa.ac.uk).
The final aim of subject review, the sharing of best practice, has been achieved in three principal ways. The first of these focused on the training of the individuals who undertook the reviews on behalf of HEFCE and the Agency. By the end of the cycle of subject review in 2001, 98 review chairs and more than 5,700 subject specialists had been trained in the subject review methodology. The majority of those trained worked in institutions that were HE providers in England and Northern Ireland. The training ensured widespread knowledge of existing good practice among the reviewers. Evaluations completed by subject reviewers after each review repeatedly attested to the fact that the experience of undertaking a subject review enabled them to reflect in great depth on their own and their colleagues' practices. As such, subject review contributed to the spreading of a great deal of good practice across the sector.
Secondly, institutions were informed of sector-wide best practice by their own staff acting as subject specialists visiting other institutions, and also participating in subject reviews themselves. An early indication of the impact of the cycle of subject review on institutions was provided in a 1997 HEFCE report, which concluded that 'two-thirds of assessors' recommendations had been acted upon'. These actions included the 'formalisation of procedures and improved documentation, reviews and revisions to curricula and to teaching and assessment methods, staff development and resource allocation'.
Finally, best practice has also been shared through the development and publication of subject review reports and subject overview reports. The main body of these reports highlights the major trends in HE provision during this period. These are briefly considered here with reference to the design and delivery of HE programmes.
The design and delivery of HE programmes
HE provision in England and Northern Ireland remains diverse in content and style of delivery, and is focused on student learning. However, students entering HE programmes today will have a very different experience from that of their predecessors in 1993. Information about programmes is more comprehensive and understandable, as it is produced with students more in mind. Programmes are more focused on meeting students' needs. Students are expected to take a greater role in their own education, as the emphasis has shifted from teaching to learning. Also, students now have a greater role in quality management and enhancement.
Changes to the context of HE in England and Northern Ireland during the cycle of subject review - including a rapid increase in the number of students enrolling for HE programmes, increased emphasis on transferable and cognitive skills and, at the same time, a real decline in the unit of resources per student provided by Government - has meant that providers have had to take a more systematic approach to the design and delivery of HE programmes. This has manifested itself in a number of ways in the design and delivery of curricula, and has centred on student achievement of the intended learning outcomes set for a programme.
In terms of curricula, the majority of subject review reports demonstrate that they are carefully designed to be internally coherent. Clearly defined and regularly reviewed aims and learning outcomes for a programme are a key point in this process. As the review cycle progressed, it became apparent that generic and transferable skills were becoming an increasingly important aspect of the curricula, alongside student's attainment of subject-specific knowledge.
In line with this more systematic approach to the design and delivery of HE programmes, there is greater emphasis on the match between teaching and learning, and student achievement of learning outcomes. Departmental teaching and learning committees frequently oversee strategic development to this end. As a result, there has been considerable innovation in teaching and learning. While lectures remain the key mechanism for the delivery of subject material, small-group teaching and IT resources are increasingly employed to ensure the attainment of relevant transferable and cognitive skills. Nowadays, moreover, students are encouraged to take greater responsibility for their own learning, leading to greater emphasis on independent learning.
Similarly, there are many instances of considerable development in the area of assessment. Innovation has occurred in terms of the range of assessment methods employed. For example, examinations are no longer the dominant form of summative assessment, and continuous assessment has become increasingly common. Both formative and summative assessment often include the assessment of transferable and cognitive skills as well as subject knowledge. Information with regard to assessment criteria is now much more transparent to all involved in the marking process, including subject staff, external examiners and, most importantly, students. Finally, helpful and timely feedback on both formative and summative work is increasingly the norm.
HE providers in England and Northern Ireland have long been regarded as particularly effective in the provision of student support and guidance, especially for undergraduates. From the outset of subject review, the vast majority of subject review reports and subject overview reports have confirmed this reputation. Nonetheless, subject providers have sought to further develop their systems. Admission and induction arrangements for all, but especially mature, part-time and postgraduate students, together with those having non-traditional entry requirements, have been enhanced. Personal tutor systems operated by the subject provider now more often encompass students other than undergraduates and are more closely integrated with a wide range of centrally provided support services, such as welfare and careers. Personal tutors are often provided with specific training to undertake this role.
The accelerating cost of learning resources, including the rapid introduction of ICT to support learning, at a time of declining levels of resourcing, is a further area which has demanded a more systematic approach by providers. As a result, new acquisitions are more often predicated on the needs of taught students. Institutions have facilitated access to additional resources by negotiating arrangements with other libraries, museums or art galleries, and provided access to online journals and the internet. Access to existing resources has often been enhanced, for example, through extending opening hours, creating electronic (and, increasingly, online) library catalogues and making arrangements for short-term borrowing.
The approach to the design and delivery of HE programmes in England and Northern Ireland summarised above has been underpinned in part by a more systematic approach to quality management and enhancement. The subject review and overview reports, especially in the later rounds, note a range of positive features. Integrated systems for validation, annual monitoring and periodic review are much more widespread, often involving the participation of external subject specialists or practitioners. Reviews are based on a comprehensive evidence base, including analysis of student progression, performance and achievement statistics. Students' views on the quality of provision are elicited more extensively and routinely, particularly through module and programme evaluation questionnaires, staff-student liaison committees and student participation in programme, departmental and faculty quality assurance committees. Staff development, including the mentoring and training of new full and (increasingly) part-time staff, and peer observation of teaching and other mechanisms for sharing good-practice, also characterise effective quality management and enhancement systems.
Challenges remaining
In general, reports emanating from the cycle of subject review attest to the positive reputation of HE provision in England and Northern Ireland, and that the vast majority of students receive education of a good quality. However, the reports also point to a number of areas where concerns about the quality of provision remain. It needs to be stressed that, overall, these are not widespread problems, but they do remain an important issue for some providers.
Most providers have adopted a more systematic approach to the design and delivery of their provision. The focus is on student achievement of intended learning outcomes. However, some subject review reports, including some from the final rounds, demonstrate that this approach is not uniform. For example, in designing their programmes some providers continue to confuse aims with intended learning outcomes. In other instances, providers have failed to design the curriculum adequately to ensure opportunities for students to attain the transferable and cognitive skills relevant to their programme of study.
The assessment of students' attainment of intended learning outcomes also remains uneven. In some extreme instances, learning outcomes are not assessed. In other examples there is a lack of clearly communicated assessment criteria and inadequate feedback on assessed work. As such, assessment remains one of the aspects of HE provision in England and Northern Ireland in which there remains scope for improvement.
Although student support and guidance is one of the most prominent strengths of HE provision in England and Northern Ireland, some problems remain. These principally relate to the frequency of contact between students and their tutors, and the less structured support systems that operate for combined honours, part-time, distant and postgraduate students. As with student support and guidance, the reported strengths in the learning resources aspect heavily outweigh persisting concerns. The most common of these relates to teaching accommodation, which was quite often reported as overcrowded, unsuitable or poorly equipped, and to the limited access to library or ICT resources.
Quality management and enhancement at the subject level features as a concern in a wide number of reports. In some instances institutional procedures are not being scrupulously applied at the subject level. In other instances annual monitoring is not undertaken, or the evidence (including statistics relating to progression and achievement) on which it is undertaken is lacking or deficient. In other examples there are concerns about the operation of the external examiner system. In these examples, institutional management does not always fulfil its responsibilities, especially in FECs, where there is limited experience of external scrutiny of HE quality assurance procedures. Examples of these are ensuring that annual monitoring is conducted appropriately or ensuring that the external verifier or examiner system is operated in an acceptable way.
Finally, in some instances, mechanisms for enhancement are sometimes inadequately developed. Staff development, especially the induction, mentoring and training of part-time staff and teaching assistants is insufficient or not equal to that provided for full-time staff. Peer observation, although widespread, is not always used to ensure the dissemination of good practice in teaching.
Quality assurance and the future
The HE sector has not been complacent about the weaknesses that became evident during the cycle of subject review. Even before 2001 and the end of the cycle, the sector had instigated a new era of quality assurance in HE, which included for the first time a new focus on the standards set for and achieved by students. At the heart of this new era is the academic infrastructure. Developed largely in response to the recommendations of the Dearing Committee, the infrastructure consists of 42 Subject benchmark statements, The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ), the Code of practice for the assurance of academic quality and standards in higher education and Guidelines on programme specifications. Written by representatives of the HE sector working in conjunction with the Agency, the academic infrastructure is designed as a series of reference points for HE providers to use in the assurance of quality and standards.
New methods of external quality assurance have been developed. These are founded on the shared belief that responsibility for assuring quality and standards is best located at the level of the institution. The new methods focus on how institutions, in their management of quality and standards, are engaging with the academic infrastructure. Principal among these at the institutional level is institutional audit and, at the subject level, academic review.
In its design, academic review at the subject level differs significantly from subject review in that it reviews quality and standards. It also responds to the weaknesses emerging from subject review reports. Student achievement of intended learning outcomes is given greater prominence, as are assessment and the management and enhancement of quality and standards. The nature of the review process has also been altered. First, in line with the Dearing report's recommendations, academic standards set for and achieved by students now come within the remit of external review teams. Secondly, review visits are no longer concentrated into a few consecutive days activity, but involve a number of visits over a six-week period, so in total there are fewer days spent by review teams in an institution. Academic review is not universal, and there is no system of numerical grading. In the first instance, as designated by HEFCE, it has focused on those sectors (for example, FECs) and institutions which had relatively poor track records during the cycle of subject review or which had had little or no track record in subject review. Academic review has not been applied in Northern Ireland.
This report provides a summary of the trends and findings of the cycle of subject review undertaken in England and Northern Ireland between 1993 and 2001, and assesses the extent to which subject review achieved the purposes set for it by HEFCE in 1994. The 2,904 subject review reports and 62 subject overview reports emanating from subject review provide extensive evidence for the high quality of HE in these two constituent parts of the United Kingdom. It is from this sound base that a new era of external quality assurance has been developed, and which is now being implemented, more information on which can be found at www.qaa.ac.uk
Annex 1: The Grade Descriptors
Tests to be applied
To what extent do the student learning experience and student achievement, within this aspect of provision, contribute to meeting the objectives set by the subject provider?
Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?
Scale Points
1 The aims and/or objectives set by the subject provider are not met; there are major shortcomings that must be rectified.
2 This aspect makes an acceptable contribution to the attainment of the stated objectives, but significant improvement could be made. The aims set by the subject provider are broadly met.
3 This aspect makes a substantial contribution to the attainment of the stated objectives; however, there is scope for improvement. The aims set by the subject provider are substantially met.
4 This aspect makes a full contribution to the attainment of the stated objectives. The aims set by the subject provider are met.
Annex 2: Subjects reviewed by round, and publication number of subject overview reports
| Subjects | Reference no for subject overview report | |
| Round 1 April 93 - Jan 94 |
Law Chemistry History Mechanical Engineering |
QO 1/95 QO2/95 QO3/95 QO4/95 |
| Round 2 Jan 94 |
Applied Social Work Architecture Business and Management Computer Studies Social Policy and Administration Environmental Studies Geography English Anthropology Geology Music |
QO5/95 QO6/95 QO7/95 QO8/95 QO9/95 QO10/95 QO11/95 QO12/95 QO13/95 QO14/95 QO15/95 |
| Round 3 Jan 94 - Apr 95 |
Chemical Engineering French German and Related Languages Iberian Languages and Studies Italian Linguistics Russian and East European Languages and Studies Sociology |
QO1/96 QO2/96 QO3/96 QO4/96 QO5/96 QO6/96 QO7/96 QO8/96 |
| Round 4 Apr 95 - Sept 96 |
American Studies East and South Asian Studies Middle Eastern and African Studies Materials Technology Land and Property Management General Engineering Building Food Science |
QO1/98(PDF) QO2/98(PDF) QO3/98(PDF) QO4/98(PDF) QO5/98(PDF) QO6/98(PDF) QO7/98(PDF) QO8/98(PDF) |
| Round 5 Oct 96 - Sept 98 |
Agriculture, Forestry and Agricultural Sciences |
QO9/98(PDF) QO10/98(PDF) QO11/98(PDF) QO12/98(PDF) QO13/98(PDF) QO14/98(PDF) QO15/98(PDF) QO16/98(PDF) |
| Round 6 Oct 98 - Sept 00 |
Veterinary Medicine |
QO1/2000(PDF) QO2/2000(PDF) QO3/2000(PDF) QO4/2000(PDF) QO5/2000(PDF) QO6/2000(PDF) QO7/2000(PDF) QO8/2000(PDF) QO9/2000(PDF) QO10/2000(PDF) QO11/2000(PDF) QO12/2000(PDF) QO13/2000(PDF) |
| Round 7 Oct 00 - Dec 01 |
Economics Theology and Religious Studies Hospitality, Leisure, Recreation, Sport and Tourism Celtic Studies Business and Management Librarianship and Information Management Politics Archaeology Classics and Ancient History Education Philosophy |
QO1/2001 QO2/2001 QO3/2001 QO4/2001 QO5/2001 QO6/2001 QO7/2001 QO8/2001 QO9/2001 QO10/2001 QO11/2001 |
Glossary
Cognitive skills
The Dearing Report included 'an understanding of methodologies or ability in critical analysis.
The cycle of subject review
The cycle of subject review commenced in 1993 and was completed in 2001. It consisted of seven rounds of activity. During each round a group of subject areas were reviewed.
Dearing Report
See National Committee of Inquiry into Higher Education. (NCIHE)
EHE
Enterprise in Higher Education initiative
FEC
Further Education College
FEFC
Further Education Funding Council. Until 1999 the FEFC was responsible for funding the provision of higher national certificates (HNCs) in FECs. Subsequently, HEFCE has been responsible for funding all HE in FECs.
HE
Higher Education
HEFCE
Higher Education Funding Council for England. Originally commissioned, designed and managed the early rounds of subject review (1993-1997). For later rounds it commissioned the QAA to manage the process.
HEI
Higher Education Institution
HESDA
Higher Education Staff Development Agency (formerly UCOSDA). Reflecting the growing role of FECs in higher education UCOSDA was renamed in 2000 as HESDA.
Key skills
Often referred to as transferable skills, the Dearing Report listed these as 'communication, numeracy, the use of information technology and learning how to learn'.
National Committee of Inquiry into Higher Education. (NCIHE)
Often referred to as the Dearing Committee (after its Chairman), its terms of reference were to make recommendations on how the purposes, shape, structure, size and funding of higher education, including support for students, should develop to meet the needs of the United Kingdom over the next 20 years, recognising that higher education embraces teaching, learning, scholarship and research'. Its far-reaching and influential report Higher education in the learning society (often referred to as the Dearing Report) was published in 1997
PSRBs
Professional, Statutory and Regulatory Bodies
QAA
Quality Assurance Agency for Higher Education. Established in 1997, it was commissioned by HEFCE the design and management of later rounds of subject review.
Review chair
Independent or seconded HE experts who chaired review teams, and edited reviewer's contributions to produce a report
Subject review rounds
In order to ease the management of the cycle of subject review, there were seven rounds of subject review activity. In each round all the providers of a number of subject areas were reviewed. Typically, a subject review round lasted for 1 - 1.5 years.
Subject
Subject is used here as shorthand for unit of assessment. It is recognised that a subject may contain one or more courses or programmes of study, possibly at different levels (for example diploma, undergraduate, taught postgraduate).
Subject providers
The departments / schools / faculties which, on behalf of their institution, provide the education which is the object of the review.
Subject overview report
Once all the providers of a particular subject had been reviewed, a subject overview report was produced. The number of providers, and therefore the basis of the overview reports, ranged in size from 4 (Celtic Studies) to 162 (Business and Management). These can be found on the QAA website at www.qaa.ac.uk.
Subject review reports
For each subject review visit undertaken a subject review report was written. A total of 2904 reports were produced. These can be found on the QAA website at www.qaa.ac.uk.
Subject review facilitator (SRF)
SRFs were first established in 1998. They were intended to act in a liaison capacity between the review team and the subject providers. Importantly, they were also to act as a guardian of the subject review process ensuring that the review team adhered to the methodology set out in the Subject Review Handbook.
Subject specialist reviewer (SSR)
The vast majority of subject specialists were lecturers in higher education. The remainder were recruited from industry and the professions. Most review teams consisted of 3 SSRs and a review chair.
Transferable skills
See key skills
UCoSDA
Universities and Colleges Staff Development Agency (See HESDA above).
References
Improving the Quality of Education: The Impact of Quality Assessment on Institutions: 1997 John Brennan, Mark Frederiks and Tarla Shah
Assessment of the Quality of Education February 1993 3/93
The Quality Assessment Method from April 1995 (December 1994 39/94)
www.dfes.gov.uk/statistics/DB/SBU/b0285/162-t8.htm
Subject Review Handbooks (1998 and 2000)
ISBN 1 84482 006 8
Top