Learning from subject review 1993 - 2001

Chairman's Foreword

Between 1993 and 2001 a cycle of reviews of academic subjects was undertaken in higher education in England and Northern Ireland. This report examines the findings of the seven rounds of the subject review cycle, which were carried out by the Quality Assessment Division of the Higher Education Funding Council for England (HEFCE) until 1998 and thereafter by the Quality Assurance Agency for Higher Education (the Agency). During the cycle, 2,904 subject review visits were made, encompassing 62 subject areas. As each round came to an end, an overview report was published for each subject area.

The individual subject review reports placed in the public domain an unprecedented amount of information about the quality of higher education programmes in England and Northern Ireland. This report provides a synopsis of the major trends and findings that the subject review reports revealed, and attempts to assess the extent to which subject review achieved the purposes that were intended for it.

The report is concerned with the quality of higher education provision in England and Northern Ireland between 1993 and 2001 and a number of general observations can be made. The overwhelming majority of subject provision was approved. Very little had to be revisited. As the cycle unfolded it became apparent that providers were adopting a more systematic approach to the design and delivery of higher education programmes than in the past. This manifested itself in a number of ways, including approaches to the design of the curriculum, the delivery of teaching and learning, student support and guidance, the management of learning resources and the management and enhancement of quality. However, review teams also found opportunities for improvement. These were largely related to the assessment of students and the management and enhancement of quality.

Subject review was always contentious. This report, while recognising that fact, does not seek to reignite the debate as to the merits of the process. It is provided as a synopsis of the trends found in the 2,904 subject review reports and 62 subject overview reports. It highlights both the lessons learnt and the opportunities for the dissemination of good practice in learning and teaching between subjects and institutions. As such, I hope that it will be a useful source of information for higher education providers and policy makers, and those people engaged in the study of higher education.

Christopher Kenyon CBE

Chairman of the Agency, 1997-2004

Dedication

This publication records the history of one of the largest programmes of evaluation of the quality of teaching and learning support in higher education ever undertaken. Even for those who were heavily involved with it at one stage or another, this overview of 10 years of activity and critique serves as a reminder of the scale and the audacity of the venture. Given that many people had individual and idiosyncratic experiences of subject review, it also serves to focus the attention on the large-scale impact of the process and the results that it delivered for the sector as a whole.

The person who was most centrally involved in subject review from beginning to end was Dr Peter Milton, to whom this volume is dedicated. From my first appearance in the Quality Assessment Division of the Higher Education Funding Council for England (HEFCE), Peter was completely engaged with every aspect of the process, from leading teams of assessors to writing the Assessors' Handbook, the vade mecum of everyone involved with it. With the creation of the Quality Assurance Agency, Peter took over the direction of subject review and guided it to its completion in 2001. His commitment and energy made a substantial contribution to the enhancement of our understanding of best practice in teaching and learning support.

Speaking to Peter one week before he died, I learned of his mounting pleasure, and relief, that the process of writing this document was coming to a conclusion. It is greatly to be regretted that he did not live to see it published, as in large part the achievements of that process were also Peter's achievements as well, for which the higher education sector in England and Northern Ireland will, in the longer term, have much to be thankful.

Dr Paul M Clark

Director of Quality Assessment, HEFCE, 1993-96, Pro-Vice Chancellor (Learning and Teaching) The Open University

Acknowledgements

The compilation of this report has involved a substantial amount of work over a considerable length of time and has involved a number of people. The late Dr Peter Milton, to whom it is dedicated, undertook much of the early work. Peter's work was developed into this report principally by Adam Biscoe and Gerry Crawley. It was Gerry, though, who methodically and unstintingly undertook the reading and analysis of 193 subject review reports and 62 subject overview reports. He then crafted the findings into the seven chapters, which make up the main body of this report.

Adam Biscoe

Executive summary

The cycle of higher education (HE) subject reviews in England and Northern Ireland began in 1993 and was completed in 2001. Subject groups across 62 subject areas in all universities and colleges funded by the Higher Education Funding Council for England (HEFCE) and the comparable bodies for Northern Ireland were included. Some 2,904 subject review reports were produced. In addition, an overview report for each subject area was published.

This report is a synopsis of the major trends and findings of the cycle of subject review. The broad findings are:

HE provision in England and Northern Ireland was generally found to be of a very high quality: the overwhelming majority (99 per cent) of subject review visits resulted in the provision being approved in the first instance. This was true not only of provision in universities and HE colleges, but also in the further education colleges included in the subject review cycle after 2000, although only 97 per cent of the last were approved in the first instance.
The main aims set by the HEFCE in 1994 for subject review were to ensure accountability of public funds; provide public information on the quality of HE in England and Northern Ireland; and act as a mechanism for the sharing of good practice. These aims were broadly achieved.
More than 5,700 subject specialist reviewers and 98 review chairs were trained to undertake subject review visits in what was the largest ever single staff development exercise in UK higher education. The reviewers acted as an important conduit for the dissemination of good practice within their institution. The publication and wide circulation of subject review and subject overview reports contributed to the spread of good practice across the sector.
During the period 1993-2001 providers became more self-critical and adopted a more systematic and rigorous approach to the design and delivery of HE programmes. Their focus now includes more explicitly student achievement of relevant and clearly stated learning outcomes.
Concerns about the quality of some aspects of provision, particularly the assessment of students and quality maintenance and enhancement processes applied at the subject level, continued as a theme throughout the cycle.
HE provision in England and Northern Ireland managed to enhance considerably the quality of students' learning during the subject review cycle, despite considerable changes to its operating environment.

Subsequently, the HE sector in England and Northern Ireland has not been complacent about the quality of education. A new era of quality assurance, including a stronger emphasis on the maintenance and enhancement of academic standards, has already begun.

Guide to findings

The following is a summary of the findings in the main body of the report, grouped by chapter.

Aims and objectives

problems with the clarity of aims and objectives persisted across the review cycle;
a wide variety of aims across subjects and institutions reflecting diversity of provision in the variety of programmes delivered;
increasing prominence given to the aim of developing students' transferable skills, and developing students as independent learners.

Curriculum design, content and organisation

evidence of a more strategic approach to the planning of curricular content;
curricular content now better related to the aims and objectives of the provider;
greater attention paid to subject, generic and transferable skills in the curriculum;
increased flexibility within the curriculum;
transferable skills not always systematically mapped across modules in a consistent manner;
currency of curriculum could be strengthened in some cases by closer links with local/regional employers.

Teaching, learning and assessment

clear strategies for teaching and learning now more common place;
far greater clarity in defining learning outcomes and in ensuring that the teaching and learning methods employed enabled the students to acgieve the learning outcomes;
significant innovation in teaching and learning, including diversification of methods employed and extensive use of information technology (IT);
high-quality of teaching sustained over the period;
establishment of learning and teaching committees contributed to the wider dissemination of good teaching practice;
students encouraged to take greater responsibility for their learning;
wider range of assessment methods employed;
assessment of a broader range of skills;
students better informed about assessment criteria;
students more likely to receive sound feedback on their assessed work;
some providers do not have well defined teaching and learning strategies;
some times over reliance on teaching methods which did not encourage student involvement in their learning;
lack of coherence in their assessment strategies;
some evidence of assessment methods not well matched to learning outcomes;
some concerns about the quality of feedback provided on students' assessed work.

Student progression and achievement

improvements in management information systems so that the progression, performance and achievements of students were more accurately recorded, and could be fed into enhancing the student experience;
recognition and response to the needs of a far more diverse student population;
higher levels of attainment overall as reflected in class of awards;
students acquired a broader range of transferable skills;
students successful in gaining employment or in proceeding to further study;
lingering concerns about the reliability of statistical data, particularly in relation to career destination data.

Student support and guidance

excellent care for students delivered through effective systems, and served by dedicated and caring staff;
further education colleges (FECs), partly due to the size of HE provision, was particularly well equipped to achieve a close and caring community between staff and students;
clear evidence of innovation and enhancement in the support and guidance offered to students;
improvements in the admission and induction arrangements, particularly for mature students and those with non-traditional entry qualifications;
staff often specifically trained for personal tutoring role;
sound systems usually in place to support students on placement visits significant extension of study skills programmes, and support for students in need of remedial assistance;
part-time students and students on joint programmes did not always receive the same level of advice as single honours students;
less structured tutor support for postgraduate students.

Learning resources

a more strategic approach in making decisions of learning resources so that they were more closely integrated into programme planning;
increasing establishment of learning resource centres;
learning resources more readily accessible to students;
increasing use of IT resources;
evidence of enhancement in the provision of specialist equipment, collections and museums;
quality of accommodation remained a concern for reviewers, although substantial developments were evident later in the review cycle;
administrative, secretarial and technical staff undertake important functions, and often closely integrated into the provison.

Quality (assurance) management and enhancement

in some early reviews clear and comprehensive quality frameworks did not exist;
similarly, there was over-reliance on informal quality procedures that lacked rigour or were insufficiently robust;
examples of excellent annual monitoring were common place but not universal. In some instances annual monitoring did not occur, or were based on a deficient evidence base;
increasingly, quality management systems elicited the views of students;
concerns with regard to the operation of the external examiners system persisted throughout the cycle of review;
there were many examples of effective involvement of professional bodies and employers in the provision of courses;
staff development strategies became commonplace, including staff appraisal and peer observation of teaching, and the dissemination of good practice;
arrangements for the induction, mentoring and training of new full time staff were often commended, but less often for part-time staff or research and postgraduate students involved in teaching.

Background to Subject review

Introduction

The purpose of this report is to provide a synopsis of the major trends and findings of subject review in England and Northern Ireland during its nine years of operation (1993-2001) and to consider the extent to which subject review achieved the purposes set for it. In particular, it focuses on student learning and how some of the factors that affected this changed during the period. As such it highlights both good practice and areas where the opportunities for improvement were observed.

This report has been written for key stakeholders with a specific interest in HE. Thus, it is aimed principally at HE policy makers, academic practitioners, quality managers and researchers looking at pedagogical issues. It will also be of interest to overseas policy makers wanting to be informed about the experience and outcome of the largest ever exercise of external subject review yet undertaken anywhere. However, although it includes a summary of the changes that were observed to be influencing student learning between 1993 and 2001, it is not written principally with an audience of current or prospective students in mind.

In recent decades responsibility for HE has been increasingly devolved, especially in terms of funding, to the four constituent parts of the United Kingdom: England, Northern Ireland, Scotland and Wales. Although external review at the subject level operated during the 1990s in Wales and Scotland, it differed significantly from that in England and Northern Ireland. This report is focused solely on the cycle of subject review in England and Northern Ireland.

The report has three main sections. The introductory section sets out the statutory basis for subject review, the changing context in which it was undertaken, and briefly outlines the subject review method and the ways in which it was revised during the period. It also explains the method used for compiling this report. The second section is the main body of the report, and is divided into seven chapters. The first chapter analyses the trends reported in providers' efforts to set out their aims and objectives. The following chapters analyse in turn the reported trends for each aspect - curriculum development, content and organisation, teaching learning and assessment, student progression and achievement, student support and guidance, learning resources, quality (assessment) management and enhancement - that were the focus of detailed scrutiny in the reviews. Each chapter has four sub-sections: the first briefly outlines the scope of each aspect; the second provides a statistical analysis of the grades awarded between 1995 and 2001. The third sub-section provides a detailed analysis of the main findings. The final sub-section highlights some of the main conclusions to be drawn. The final section of the report, the conclusion, assesses the extent to which the main purposes set for subject review were achieved. In so doing it draws together the findings of the main body of the report.

Rationale for, and context of, subject review

This section describes the statutory basis for subject review and briefly outlines the dynamic context in which the cycle of subject review (1993-2001) was conducted. It also describes the key constant features of the subject review method, and notes the major changes that were introduced during the cycle.

The statutory basis for subject review

The statutory basis for subject review was the Further and Higher Education Act 1992 (Section 70). The Act required HEFCE to 'secure that provision is made for assessing the quality of education provided in institutions for whose activities they provide, or are considering providing, financial support'.

At the start of the work to meet this requirement through what initially became known as Teaching Quality Assessment (TQA), HEFCE identified three purposes of quality assessment:

i. to ensure that all education for which HEFCE provides funding was of satisfactory quality or better, and to ensure speedy rectification of unsatisfactory quality;

ii. to encourage improvements in the quality of education through the publication of assessment reports and an annual report;

iii. to inform funding and reward excellence.

In the light of experience gained from a series of pilot reviews undertaken in 1993, institutions' responses to HEFCE's plans for the development of teaching quality assessment, and further statements by Government, HEFCE, in 1994, restated the purposes of what was subsequently to become known as subject review. They were to ensure 'accountability of public funds, provide a link with funding to enhance quality, providing accessible public information, and sharing and publicising best practice'.

The changing context of subject review

The subject review cycle did not occur in isolation. The nine-year cycle, spanning much of the 1990s, was a period of significant development in the HE sector, and any synopsis of the major trends and findings of subject review need to be set within the context of these changes. The most important developments were:

1. Changes in Government Policy

A decision in the early 1990s rapidly to increase the proportion of 18-30 year olds experiencing some form of HE, from 10 per cent to 30 per cent. In 1997, following the election of a Labour Government, a commitment was made to increase this proportion further, to 50 per cent of 18-30 year olds.
The Research Assessment Exercise (RAE) was introduced in 1986. This procedure was designed to link research funding to research quality. A competitive exercise operated by HEFCE, the RAE is widely believed to have tilted academic activity more towards research, arguably at the expense of the focus on teaching.
In 1996 the government appointed the National Committee of Inquiry into Higher Education, under the chairmanship of Lord Dearing. Its terms of reference were 'To make recommendations on how the purposes, shape, structure, size and funding of higher education, including support for students, should develop to meet the needs of the United Kingdom over the next 20 years, recognising that higher education embraces teaching, learning, scholarship and research'. Its far-reaching and influential report Higher Education in the learning society (often referred to as the Dearing Report) was published in 1997. It recommended the replacement of universal teaching quality assessment with a 'lighter touch' approach that included a focus on academic standards. To underpin this methodology the report also recommended the creation of an 'academic infrastructure'.
In 1999 the responsibility for funding HNC programmes was transferred from the Further Education Funding Council (FEFC) to HEFCE. As a result, all HNC programmes came within the purview of subject review.
Between 1993 and 2000, unit funding by Government per full-time equivalent (FTE) student fell in real terms from £5,820 per annum to £4,900 per annum (www.dfes.gov.uk/statistics/DB/SBU/b0285/162-t8.htm).

2. Changes in the nature of provision

In recent decades there have been significant shifts in pedagogical thinking, which were in part encompassed in the Dearing Report. These included an emphasis on the student's concurrent acquisition of key or transferable skills (for example, communication, numeracy, the use of information technology (IT) and learning how to learn) and of cognitive skills (for example, an understanding of methodologies or ability in critical analysis) and subject knowledge. Also, there has been a shift of emphasis from teaching to learning (especially independent learning) and an increasing focus on intended learning outcomes.
The introduction of IT, including relatively inexpensive personal computers, email and public access to the internet, has impacted on HE students in a number of ways. These include:
i. employers have increasingly expected graduates to have a grounding in the use of IT;

ii. many providers have developed e-learning strategies and resources;

iii. providers are increasingly expected to make electronic learning resources available, for example, electronic library catalogues, access to electronic information and databases, access to email and the internet, and the provision of increasingly specialist hardware and software. This has placed a further major demand upon their resources.
Subject review began at a time when UK higher education was in the process of reviewing curriculum structures, especially in the light of the trend towards modularisation.

3. HE in the global context

The subject review cycle in England and Northern Ireland cannot be seen in isolation from wider developments in international HE, both elsewhere within the UK, and overseas. Overall, the United Kingdom has experienced a significant increase in the number of European Union and overseas students studying in the UK, and remains a net exporter of educational services. However, there is strong competition for students, especially from the United States of America and Australia, and from distance-learning providers. The quality and standards of HE are now an important consideration for students choosing a foreign country in which to study.

Changes to the method of subject review (1993-2001)

The subject review cycle began in 1993 and ended in 2001. During this period there was one major (in 1995) and two minor modifications to the method of review. Full descriptions of the various review methods are contained in HEFCE's Quality Assessment Handbooks (1993, 1995 and 1996) and the Agency's Subject Review Handbooks (1998 and 2000).

Key principles of subject review

The term 'subject' is used in this report as shorthand for the unit of activity that was reviewed. A subject might contain one or more courses or programmes of study, possibly at different levels (for example, diploma, undergraduate, taught postgraduate). For the purposes of subject review, HEFCE determined the broad definition of each subject area (see annex 2).

Normally, review teams comprised at least three subject specialists and a review chair. Where the subject under review was large or complex the number of subject specialists was increased proportionately in agreement with the institution.

Subject review was a system of external peer review. More than 5,700 subject specialists, who understood the particular subject under scrutiny and were familiar with HE teaching and learning processes, were recruited and trained as reviewers. The vast majority of subject specialists were teachers in HE. The remainder were recruited from industry and the professions. The training programmes were developed by a combination of staff from HEFCE, and, after 1997, the Agency and the Universities' and Colleges' Staff Development Agency (UCoSDA), later the Higher Education Staff Development Agency (HESDA). From 1995 onwards, review chairs also became involved in the planning, preparation and delivery of these intensive training sessions. In part, they took the form of a simulated visit. Every session was evaluated fully and evaluations indicated that these sessions were successful as staff development for trainees as well as training for subject review visits.

Some 98 review chairs were recruited and trained. The review chair was responsible for the whole process, from receipt of the self-assessment document (SAD) to production of the final report, and managed the review team's activities. Review teams were assisted by the Visit Support Team based initially at HEFCE, and from 1998 at the Agency.

Using the published review handbooks, review teams assessed the quality of educational provision in terms of the educational experience of students, judged against the subject providers' own stated aims and objectives. The aims and objectives were set out in the SAD, in which providers were also required to undertake their own evaluation of the provision and provide general data, such as progression statistics, that would be useful to review teams.

Review visits were necessarily of limited duration, usually no more than three and a half days in the institution. They normally included meetings with subject staff, students, graduates and employers. Reviewers also observed teaching and learning and reviewed samples of assessed student work. The visits resulted in evidence-based judgements for each aspect under review, an oral feedback meeting and, after 1995, a published report.

In order to ensure fairness, institutions' had a right to complain about the outcome of a subject review or, in the case of a failing (grade 1) judgement, to make a representation. This process was initially operated by HEFCE, and later by the Agency. Exceptionally, where there appeared to be a significant perversity in either the judgement or in the way in which the review process had been undertaken, HEFCE could, and did, initiate a full or partial re-review.

1993-1995

In the first round of subject review known as Quality Assessment (1993-1995), subject providers, within their SAD, were invited to judge themselves as either 'excellent' or 'satisfactory' in terms of the quality of their educational provision. All SADs were read by a review chair and a pro forma completed. In order to claim excellence, the SAD had to provide a specific section citing evidence for the claim. If the provider claimed excellence, two review chairs read the SAD. Where there was insufficient evidence to support the claim of excellence, the provision was judged to be satisfactory. Where there was sufficient supporting evidence, a review team was appointed to undertake a visit. If the team corroborated the claim of excellence, then the original claim stood. If not, they were judged to be satisfactory. The subject providers judging themselves satisfactory were visited by a team on a sample basis. All provision identified as potentially unsatisfactory was visited.

During a visit, reviewers were invited to confirm the providers' claim or to conclude otherwise. The judgement of unsatisfactory was also open to review teams. Such a judgement led to a re-assessment within one calendar year in order to assess the progress made by the provider in addressing the deficiencies identified by the original review team. A further judgement of unsatisfactory at this re-assessment could result in partial or complete withdrawal of funding by the funding body. Reports were published only where a visit had been undertaken. Four subject overview reports were produced at the end of the first cycle of subject review.

1995-2001

The most significant alteration to the method of subject review occurred in 1995. This was the result of internal reflection by HEFCE, external evaluation initiated by HEFCE, and extensive feedback from institutions on the operation of subject review to date. The following major changes were introduced:

a system of universal visiting was introduced;
subject providers were no longer required to judge the quality of their own provision;
six inter-related aspects of provision were established to provide a more coherent structure to review activity:
i. Curriculum design, content and organisation (CDCO)

ii. Teaching, learning and assessment (TLA)

iii. Student progression and achievement (SPA)

iv. Student support and guidance (SSG)

v. Learning resources (LR)

vi. Quality Assurance and Enhancement (QAE) pre-1998 and Quality Management and Enhancement (QME) post-1998;
each aspect was numerically graded 1-4, with 4 representing the highest grade (see annex 1)
where one or more grades 1 were given, the provision would be subject to a re-review within one year. A further grade 1 judgement could result in the partial or complete withdrawal of funding;
all reviews resulted in a published report;
after all subject providers within a particular subject area had been visited a subject overview report would be compiled to provide an overview of the quality of HE within the subject area.

Three further alterations to the method, introduced under the aegis of the Agency in 1998, are noteworthy:

institutions were invited to appoint an institutional facilitator (IF) to liaise between the review team and the subject providers. Importantly, along with all other participants in the process, they were expected to act as guardian of the subject review process, ensuring that the visit was conducted in accordance with the methodology set out in the relevant Subject Review Handbooks.
where a review team judged that three or more aspects made only an acceptable contribution to the providers' aims and objectives (grade 2), the institution was required to produce an action plan to demonstrate how it would address the identified shortcomings in the quality of education. This was followed-up with a one-day visit to the institution by an Agency officer to check the progress in implementing the action plan, and to check that the quality of students' education had been, or was in the process of being, enhanced.
the focus of the sixth aspect, 'Quality Assurance,' was altered to 'Quality Management and Enhancement'. In part this was intended to re-focus reviewers' enquiries away from institutional policies and procedures, which were dealt with by Agency institutional audits, towards their application by those at the subject level who were responsible for the local management of quality and standards.

It is also important to note that in 1999 HEFCE took over from FEFC responsibility for funding HNC programmes in further education colleges (FECs). Consequently, all FEC provision of HE funded by HEFCE came within the orbit of subject review. Some 280 subject reviews were carried out in FECs during the nine-year cycle, the vast majority of which were undertaken between 1999 and 2001 as a result of this change in funding.

The method of analysis

This report is based on the 62 subject overview reports, together with a sample of 193 subject review reports which comprise all the reports from seven institutions. This approach is intended to demonstrate changes to the learning experience of students across the period of subject review. The institutions were chosen on the basis that they covered a broad range of institutional types: civic and non-civic universities; pre and post-1992 universities; a HE college and a FEC. The selected institutions offered an extensive range of subjects, which allows this report to comment on all subject areas.

The format of each subject review report after 1994 included sections on: aims and objectives; curriculum design, content and organisation; teaching, learning and assessment; student progression and achievement; student support and guidance; learning resources; and quality (assurance) management and enhancement. To arrive at the findings in this report, the content of the sample of 193 subject review reports was analysed to identify any overall patterns or trends. These findings were then considered in the light of the outcomes of the 62 subject overview reports. The use of the subject overview reports as a control provided confidence that the sample of subject review reports chosen for the report was representative of the sector as a whole.

Next>> Section 2, Analysis and findings