Handbook for major review of healthcare programmes
QAA 034 05/2003

The Department of Health (England), in partnership with the Nursing and Midwifery Council, the Health Professions Council and the Workforce Development Confederations, has contracted with the Quality Assurance Agency for Higher Education (the Agency) to carry out reviews of all NHS-funded healthcare programmes in England during the period 2003-06. This Handbook has been agreed and endorsed by all partners.

The Agency was established with charitable status in 1997 to provide an integrated quality assurance service for UK higher education. It is an independent body whose mission is to promote public confidence in the quality of provision and standards of the awards. To this end, the Agency carries out major reviews, institutional audits, academic reviews and other review and enhancement activity.

Introduction

Background

1 This Handbook describes the method and procedures for carrying out major reviews in England, the academic and practitioner standards achieved, and the quality of learning opportunities provided. Major review is an activity undertaken by peer reviewers, resulting in judgements on the standards and quality of NHS-funded healthcare programmes. Major review examines the learning opportunities in theory and practice, however and wherever delivered. The method and procedures comprise an integrated approach focusing on the establishment, maintenance and enhancement of academic and practitioner standards. These are matters for which the responsibility lies with the higher education institutions (HEIs) in partnership with its practice placement providers.

2 The process of major review of healthcare programmes will require a high degree of openness, transparency and trust in the partnership between the Quality Assurance Agency for Higher Education (the Agency), each HEI, Workforce Development Confederation (WDC) and partner placement providers. The process of major review described in this Handbook has been developed from the Agency's academic review in partnership with the Department of Health (DH) in England, the WDCs, the Health Professions Council (HPC), the allied health professions bodies, the Nursing and Midwifery Council (NMC) as well as a large number of representative bodies, practitioners, and academics. All these organisations and people have worked together to minimise duplication, reduce overlap between the organisations and promote appropriate links with the different quality assurance activities. Prototype reviews carried out in six HEIs, two published reports evaluating the prototypes and wide ranging discussions among stakeholders helped to refine the process and inform revision of this Handbook.

3 This Handbook will assist the reviewers, HEIs, practice placement providers, professional statutory and regulatory bodies (PSRBs), employers, funding bodies and the general public, all of whom have an interest in the arrangements for the review of health-related provision in higher education and the outcomes of it. The subject groupings that will be reviewed, in the first instance, using this Handbook are set out in Annex A.

4 A key principle underpinning major reviews is that they should be conducted in a spirit of dialogue and cooperation between the HEIs, practice placement providers, their staff and the review teams. The process is one of peer review, it is carried out by specialist teams of professional peers. These subject specialists have roles in other education providers similar to those of the staff in the providers being visited for review. They are drawn both from academic staff and from among practitioners in appropriate practice areas. Peer review enables judgements to be made by those who understand the healthcare programmes under scrutiny and who are familiar with teaching and learning processes. It enables judgements to be credible to subject providers, and to command their respect. For a peer review process to have credibility with external stakeholders, such as PSRBs, NHS Trusts, WDCs, other health service providers and potential students, judgements must be made in a transparent manner, and reported publicly.

Key considerations underpinning major reviews of healthcare programmes

5 The DH in England provides funds for the provision of higher education programmes for many of the health professions. The DH has an interest in bringing key stakeholders together to provide assurance that programmes produce practitioners who are safe and competent to practice and who are equipped to work in a patient-centred NHS. The quality assurance arrangements should build on the internal quality assurance of healthcare education providers and make best use of existing documentation and data.

6 In addition, a number of features particular to healthcare education need to be taken into account and fully reflected in the review process to be used. Healthcare programmes are composed of two complementary and interrelated elements - theory and practice. Competence and safe practice, the pre-requisites for registration for a licence to practice, must be developed and demonstrated on the pre-registration programmes. Students should be prepared for careers as competent healthcare professionals and equipped for self-critical lifelong learning.

7 Practice learning is an integral and vital component of all health profession programmes. Learning in practice placements is an important feature, contributing to the attainment of aims and outcomes. Hence, it must be reviewed as a part of any overall review of a healthcare programme. Therefore, when the standards achieved by students and the quality of the learning opportunities available to them on a particular programme are reviewed, the process must address both the theoretical and the practice components, however and wherever delivered.

8 Work has been undertaken collaboratively by 11 different groups of health professionals to produce benchmark statements for their subjects. These statements describe the nature and standards of programmes of study that lead to awards made by HEIs in the UK. As work progressed on them, considerable overlap among the subject statements became increasingly apparent. Accordingly, a common health professions framework began to emerge. This provides, on the one hand, the shared context upon which the education and training of healthcare professionals rests and, on the other, the uniquely profession-specific context within which programmes are organised. This emerging framework helps to define the competencies/proficiencies, approved by the regulatory bodies, to be expected and reflected in the learning outcomes of programmes that provide the focus for major review. Further work is being undertaken on developing an overarching health professions framework and benchmark statements for more health professions.

9 The development of interprofessional learning is an important principle developing from the emerging healthcare professions framework and in line with DH policy. The need increasingly to deliver patient-centred care has helped to bring about a greater emphasis on interprofessional education and training involving joint learning experiences for students training for different healthcare professions. The emphasis on patient-centred care enables advantage to be taken of the opportunities for shared learning across professional boundaries. This represents a significant and ongoing cultural development in healthcare education. It is a requirement of major review that review teams will comment on the breadth and quality of interprofessional learning that is provided by the HEIs and partner placement providers they visit. This will involve direct reference to the emerging health professions framework and in future the overarching health professions framework which is currently being developed.

10 Major review is concerned with higher education programmes in the healthcare professions from pre-registration to post-registration provision. Provision therefore includes undergraduate certificate, diploma and degree, and taught (ie not through programmes of research) postgraduate diploma and master's programmes, as well as the range of short courses that form part of post-registration continuing education. Through the provision of practice placements, NHS Trusts are co-providers of healthcare programmes of higher education.

11 The need to put the service user/patient at the centre of the student's learning experience is widely acknowledged. Major review is a collaborative initiative to assure the quality of healthcare programmes in terms of standards and the professional requirements of a health service designed around the patient. This collaboration results in an integrated quality assurance process involving three stakeholders:

those professional and regulatory bodies that have a statutory responsibility for ensuring that programmes are adequate to prepare newly qualified practitioners as fit for practice. In addition for some professions, the professional and statutory regulatory bodies have a responsibility for post qualifying courses, for example recordable qualifications;
the WDCs who are responsible for judging whether programmes are suitable preparation for staff to be fit for purpose;
HEIs with degree-awarding powers who are responsible for ensuring that programmes produce graduates (or diplomates) who are fit for award.

What are the aims and outcomes of major review of healthcare programmes?

12 The aims of major review are:

to encourage improvements and facilitate enhancement in the quality of education provided;
to contribute towards statutory bodies fulfilling requirements for the protection of the public;
to provide effective and accessible public information on the quality of higher education in the healthcare professions.

13 The outcomes of major review are:

confirmation that publicly-funded education is of approved quality;
speedy identification of major shortcomings in the quality of education where they occur, so that rectification can take place;
judgements about the standards and quality of education that inform funding;
sharing of good educational practice to encourage improvement in provision;
published reports that provide accessible public information about the quality of higher education in the healthcare professions;
used to inform and be informed by on-going quality monitoring and enhancement, and approval, see Figure 1;
it is anticipated that the Commission for Health Improvement (CHI) review processes will also inform and be informed by major review.

Figure 1, Five Elements of Quality Assurance Framework

Figure 1 Five Elements of Quality Assurance Framework (Department of Health 2003)

What does the major review process involve?

14 The DH with its partners is contracting with the Agency to review healthcare programmes. The Agency and the DH work closely with the PSRBs, the WDCs, the practice placement providers and the HEIs to ensure that the approach to review facilitates the integration of the quality assurance processes promoted by each of these key stakeholders. The process focuses on the establishment, maintenance and enhancement of academic and practitioner standards and the quality of learning opportunities for students. Reviews are undertaken by teams of reviewers that include appropriately qualified academics and practitioners (Annex B).

15 This Handbook sets out the detail of the review process. The process accommodates the review of practice as well as the theoretical aspects of the programme. It therefore fully recognises the key importance of the learning and teaching that take place within practice settings as well as within campus settings.

16 The major review of healthcare programmes takes account of the Agency's institutional audit process being applied to all HEIs. This process combines scrutiny of internal quality assurance systems at an institutional level with investigation of how those systems operate at the level of the discipline. Whereas major review of healthcare programmes may use the evidence provided by institutional audit reports, it will report independently upon institutional management of standards and quality specifically in relation to the subjects being reviewed. Its reports will be used in annual quality monitoring and enhancement by the principal stakeholder bodies at three levels - the institutional level of management and its impact on the subject; the management of standards and quality in relation to the WDCs; and how the HEI, in partnership with placement providers, meet the requirements of PSRBs.

17 The major review process:

accommodates a wide diversity of institutional mission and approaches to subjects;
reflects the core processes of design, delivery, support, assessment and review of programmes of study;
articulates with an HEI's and partner placement providers' internal processes for the regulation of academic standards and quality;
articulates with an HEI's and their partner placement providers' processes for meeting the statutory and professional requirements within the healthcare programmes;
seeks to identify good practice in learning and teaching in the health professions programmes, for dissemination throughout the HE and health services sectors.

18 In preparing their reports and making their judgements, review teams will use evidence provided by the HEIs and partner placement providers. Where possible, that evidence will be produced as part of their internal quality assurance arrangements. The Agency will encourage stakeholders to recognise the mutual value of this evidence base.

19 In the first cycle of major review, there will be a common intensity used across all healthcare programmes. However, where the provision is very extensive and covers a wide range of professional requirements, the review will require greater resources to ensure that evidence-based judgements are securely made. Nevertheless, there is a proper expectation that any system of quality assurance will be as efficient as possible and will consume no more overall resources than necessary. To this end, the method used by the Agency:

provides transparency of process through the use of qualifications frameworks, subject benchmark statements, the emerging health professions framework, regulatory requirements, programme specifications and the Code of practice;
involves exchange of information between major review and institutional audit processes;
allows HEIs and their partner WDCs and the PSRBs to discuss the timing of major reviews in relation to their various shared requirements. This makes it possible to align external review with both internal reviews, re-validation/approval and PSRB timetables;
facilitates alignment of major review with internal processes by spreading the review over a period rather than imposing a 'snapshot' style review visit. Thus evidence from internal processes can be made available to reviewers on request, so that the need for the preparation and assembly of large amounts of documentation in advance of a visit is removed;
ensures that the amount of time taken to conduct a major review is the minimum necessary to enable reliable judgements to be made;
draws on evidence provided by audits of the healthcare environment that are concerned with the quality of practice placements.

20 Public confidence in the quality and standards of healthcare education depends on the availability of objective and independent public information about these two interdependent areas:

reporting on academic and practitioner standards of programme outcomes is concerned with the appropriateness of the intended learning outcomes set by the subject providers (in relation to relevant programme requirements/specifications, subject benchmark statements, qualification levels and the overall aims of the provision); the effectiveness of curricular content and assessment arrangements (in relation to the intended learning outcomes); and the achievements of students;
reporting on the quality of learning opportunities is concerned with the effectiveness of learning and teaching activities, student progression, the availability and use of learning resources, and staff support for student learning and achievement in campus and practice environments across the programmes in the subject area.

21 The method of major review is centred on the self-evaluation document (SED) produced by the HEI and its partner placement providers. The first task for reviewers is to test, by means of their own observations and analyses, the statements made by the HEI and the partner placement providers in the SED. Secondly, as a team, they will make judgements on the appropriateness and effectiveness of the provision, as outlined above.

Who has an interest in healthcare programmes?

22 DH and WDCs will wish to be assured that the programmes they fund are of approved quality: in terms of academic and practitioner standards and quality of learning opportunities. Judgements made will inform future decisions about funding arrangements and will provide external assurance that provision is fit for purpose, fit for practice, and fit for award.

23 The evidence produced by the review, and used to inform it, will assist the PSRBs to fulfil their statutory duties and responsibilities for assuring the fitness for practice of successful students. The evidence will be in the form of the judgements and commentary contained in the reports. The judgements will address the academic and practitioner standards being applied to the provision being reviewed, and the quality of the learning opportunities available to the students on the programmes. PSRBs will nominate individuals to act as reviewers and so contribute to team judgements.

24 HEIs will be able to demonstrate the effectiveness of their internal arrangements for assuring the quality and standards of health profession programmes. They will be able to do so in a way that is familiar to the HE sector as a whole, using the preparation of the SED as a focus for internal review and to facilitate the enhancement of their provision. HEIs, too, will nominate individuals to act as reviewers and so contribute to team judgements.

25 The WDCs, including their constituent NHS Trusts and other health service providers, will be able to use the information that results from the reviews to identify areas where the Trusts have demonstrated a strong contribution to the learning opportunities available to students, and also where improvements are necessary. WDCs will nominate individuals to act as reviewers and so contribute to team judgements.

26 The NHS Trusts and other health service providers, through the provision of practice placements, are co-providers of healthcare programmes. They will be able to review the contributions they make to healthcare programmes through the availability and quality of effective practice placements. They, too, will nominate individuals to act as reviewers and so contribute to team judgements.

27 All reviewers will only be deployed in teams reviewing provision in which they do not have a direct interest.

28 The public as well as service users and carers have an interest in the independent review of healthcare programmes to provide them with assurance about the quality and standards of provision. Reports on the provision reviewed will be published and will provide information for the public and assurance to them. They will also provide information that encourages continuous enhancement, through the identification both of areas that would benefit from improvement and those that represent good practice. The providers' responses to the review will be disseminated through the publication of action plans.

Preparing for review

What will be reviewed and when?

29 This Handbook refers to both subjects and programmes. Within each subject, providers offer one or more of the programmes of study followed by students. Programmes may be offered at different levels (for example, pre-registration diploma, degree or postgraduate) within a single subject. The unit of review is the subject although, normally, a group of healthcare subjects will be reviewed together, particularly where they are linked by joint programmes or by interprofessional learning.

30 The HEIs and their partner WDC/s will be asked to complete a 'scope and activities' information form. This is designed to gather information about the range of subjects offered, the programmes to be included under each subject heading, the estimated numbers of student FTEs for each programme, the other quality assurance activities occurring in the review period and the HEI's and WDC's preferred timing for the review. This advance information provides the basis for further discussions to plan and agree the scope and timing of major reviews for the three-year period.

31 As far as possible, the Agency will seek to accommodate the preferences for the timing of reviews expressed by HEIs and their partners. However, the balance of the Agency's overall workload needs to be maintained across the three years of the programme of major reviews and the overall schedule must take into account the availability of reviewers with appropriate expertise. Where 'scope and activities' responses indicate that a preferred timing is to enable a review to coincide with a scheduled internal review, priority will be given to accommodating such preferences. Particular efforts will be made to accommodate the requirements/activities of the PSRBs and WDCs when decisions are made about the timing of reviews.

How do the HEI and its partner placement providers prepare?

32 An SED must be prepared for each major review. It is central to the process and the most important of the small number of documents made available to reviewers in advance of a review. Guidelines for producing SEDs appear at Annex D. These guidelines are intended to ensure that HEIs and partner placement providers address the relevant issues and include the material needed by reviewers to evaluate the provision accurately. The SED will be required by the Agency normally two months before the start of the review.

33 The SED allows the HEI and partner placement providers to evaluate what they do and why, and the methods they use to fulfil their aims. Reviewers analyse the SED, and then visit the HEI and practice placements to gather the evidence they need to make their judgements. The SED fulfils three main functions:

It should enable reviewers to see how the HEI and its partner placement providers meet statutory and regulatory requirements, and the extent to which the provision meets the need for an education that is fit for practice (the responsibility of PSRBs), for purpose (the responsibility of WDCs) and for award (the responsibility of HEIs).
It is intended to encourage the subject providers to evaluate the standards achieved by students and the quality of the learning opportunities offered to them. It provides an opportunity for the staff of the subject providers to reflect on 'what do we do?', 'why do we do it?', and 'why do we do it in the way that we do?'.
It provides a framework for a process of major review based on the testing and verification of statements made by subject providers. The document should reflect on current provision in a manner that evaluates both strengths and weaknesses, indicates the changes that have taken place since earlier external reviews, and considers what may be necessary to change in the future. It has to be endorsed by the lead WDC, the HEI and their partner placement providers.

34 Reviewers will expect to see evidence of careful self-analysis and the involvement of students and practice staff in the production of the SED. This should involve an evaluation of the perceived strengths of the provision, with reference to the evidence that justifies the statements made, and of weaknesses, where these are recognised. Where weaknesses are acknowledged, the subject providers are expected to discuss the issues and the steps being taken to bring about improvements.

35 Specifications for each of the programmes included must be annexed to the SED in order to make them readily available for reviewers. The Agency has prepared a published document, Guidelines for preparing programme specifications, to assist HEIs and partner placement providers. The specifications should provide the core factual information about the programmes, allowing the SED itself to provide a reflective analysis of the provision and its development.

What points of reference are there for the review of academic and practitioner standards and quality?

36 Reference points particular to the healthcare field will support education providers in maintaining the standards and quality of their provision. Taken together with other reference points, these will assist HEIs and partner WDCs to be explicit about the ways in which they assure the quality of the student learning experience and the standards of the qualifications they award. These include:

PSRB regulations/requirements;
the emerging shared health professions framework, developed by groups of appropriate specialists and illustrating the shared context upon which the education of healthcare professionals rests. This is being developed to become an overarching health professions framework;
NHS policies and protocols;
WDC guidelines;
National Service Frameworks DH.

37 The Agency, in partnership with the sector, has developed a number of reference points intended to assist education providers to assure themselves, and others, of the standards and quality of their provision. These are:

The framework for higher education qualifications in England, Wales and Northern Ireland;
a Code of practice for the assurance of academic quality and standards in higher education;
subject benchmark statements;
programme specifications.

The framework for higher education qualifications

38 The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ) sets out the structure in which higher education qualifications are organised. It applies to degrees, diplomas, certificates and other academic awards (except honorary degrees and higher doctorates) and qualifications granted by an HEI. It provides reference points to be used to determine whether the intended outcomes for programmes, and actual student achievement, are appropriate to the level of the qualification awarded. The FHEQ helps provide public assurance that qualifications bearing similar titles represent similar levels of achievement.

39 Implementation of the FHEQ will take place over the three-year cycle of major review. In making judgements about the provision, reviewers will consider how the HEIs and partner placement providers have all of their provision aligned with the FHEQ.

The Code of practice

40 The Agency's Code of practice for the assurance of academic quality and standards in higher education (the Code) addresses good practice in individual areas of academic management. Those sections relating directly to quality and standards provide HEIs and their partner placement providers, and reviewers, with a background against which judgements can be made. The Code is accessible at the Agency's web site and is published in the following sections:

postgraduate research programmes;
collaborative provision;
students with disabilities;
external examining;
academic appeals and student complaints on academic matters;
assessment of students;
programme approval, monitoring and review;
careers education, information and guidance;
placement learning;
recruitment and admissions.

In view of the importance of practice-based learning in health profession programmes, education providers and reviewers will be expected to pay particular attention to the way in which adherence to the placement learning section of the Code is demonstrated.

41 Reviewers will expect institutional systems to have at least an 'equivalent effect' to the precepts recorded in the Code. In addition, institutional systems will be expected to meet PSRB requirements. The precepts in the Code provide criteria against which reviewers can make their judgements on both subject provision and institutional management.

Subject benchmark statements

42 The benchmarking of standards in healthcare subjects was undertaken by groups of specialists drawn from service providers, HEIs and the PSRBs. Subject benchmark statements have been developed in collaboration with a number of healthcare professions and have so far been developed for 11 of the healthcare professions. Statements are cast within an emerging health professions framework that illustrates, on the one hand, the shared basis upon which the education and training of healthcare professionals rests and, on the other, the profession-specific context within which undergraduate programmes are organised. The format of the statements seeks to reflect these two dimensions. There is further work being undertaken to develop subject benchmark statements for more health professions and an overarching health professions framework.

43 The subject benchmark statements include expectations about the academic standards required for a higher education award and the practitioner standards required for an award that is fit for the purpose of preparing students for the start of their careers in practice. Programmes in healthcare integrate academic and practice elements. This integration will be considered throughout the review process, however and wherever the programme under review is delivered. The benchmark statements can be found on the Agency's web site.

44 It is essential that reviewers use both PSRB standards and benchmark statements as sources of guidance to inform the review process, these standards are encompassed within the benchmark statements. Statutory and professional body requirements should be referred to directly for the purposes of major review. Reviewers will use them as a means of determining whether the intended learning outcomes of individual programmes are appropriate. HEIs and partner placement providers should be able to demonstrate how subject benchmark statements have been used to inform decisions about the intended outcomes of programmes and in calibrating the overall demands of the assessment framework.

45 The Agency recognises that HEIs have their own cycles of programme review, and that it is through these that any changes, to reflect benchmark statements, will come. Reviewers will be sensitive to the challenges that institutions will face in working with benchmark statements for the first time. Nevertheless, they provide an immediate starting point for discussion and reflection within teaching teams, and between teaching teams and reviewers, about the appropriateness of the outcomes of their programmes. Benchmark statements are not set in stone and, over time, they will be revised in the light of feedback from subject communities.

46 For some programmes more than one benchmark statement may be relevant, while in some specialist, innovative or interdisciplinary fields there may not be any statement that is of direct relevance. In such cases, the level descriptors of the overall qualifications framework, and the guidance in the Code of practice section on programme approval, monitoring and review, will assist HEIs and their partners in ensuring that their provision meets generally accepted standards for a given level of award. In all cases the HEI remains responsible for identifying and assuring the standards of its awards and for ensuring that they reflect appropriate external indicators. However the partner placement providers of the HEIs also have responsibility for the standards and quality of the awards and qualifications through the practice placements.

Programme specifications

47 To be most effective, programme specifications should become part of an HEI's and partner providers curricular planning, approval and review processes. Preparation of them is an opportunity for teaching teams to reflect on the purposes and intended outcomes of their provision. Programme specifications will be the starting point for reviewers as they seek to understand the intended outcomes and the assessment methods for the programmes under review. The programme specification should make it clear whether a programme is accredited by a PSRB. Relevant PSRB requirements will then provide additional reference points that will be used during the review.

Principles for the conduct of major review

Peer review

48 The reviews will be undertaken by teams of academic and practitioner reviewers who are subject specialists, managed and led by a review co-ordinator. The main responsibility of reviewers is to read, analyse and test the self-evaluation produced by the HEI and partner placement providers, and to gather whatever further evidence they need during selective and focused lines of enquiry to make the judgements described later in this Handbook. Academic and practitioner reviewers are trained and briefed by the Agency before taking part in reviews. Further details of their function may be found in Annex B. For a note on the composition of teams, see Annex C.

49 Review co-ordinators are individuals with extensive experience of higher education programmes and quality assurance but are chosen not to be specialists in the subject under scrutiny. Each is provided with training specific to their co-ordinating function and following that, each must attend a one-day briefing on major review. Further details of their function are given in Annex B.

50 Particularly when a single member represents discipline expertise on a review team, there may arise the need for specialist consultation. Accordingly, provision is made for the review co-ordinator to call in an Agency specialist adviser who will give a second opinion on specific aspects of the provision and will advise the team accordingly (Annex F). Depending upon the programmes affected, the adviser may also need to be a Visitor for the relevant regulatory body.

Review against the broad aims of the providers

51 Subject providers should be able to set out clearly the broad aims of their programmes. Such statements should indicate, in general terms, what the subject providers are seeking to achieve, how these aims relate to external indicators such as subject benchmark statements, PSRB, or employer expectations, and the general attributes of its diplomates or graduates. These broad aims will provide the context in which the major review takes place. Accordingly, the Agency will expect the aims of the provision to be stated at the beginning of any SED and will publish the aims in the major review report. The statement must be sufficiently clear to allow intended learning outcomes to be set that ensure the aims are achieved, and to provide a sound basis for the major review to be planned and carried out. Any PSRB accreditation should be made clear in the SED and programme specifications (see Annex D).

Review over an extended period

52 The available reviewer days are spread over a period of normally six weeks to allow reviewers to gain a better understanding of the subject provision than is possible in a concentrated, 'snapshot' use of the same number of days. This reduces the need to prepare large amounts of documentation in preparation for a single visit. Normally, there will be five visit days in total (common intensity), spread over two or three periods of one and two days. Visits to practice placements will normally be the sole activity on the second and/or third day of the review.

Production of a published report

53 The reports produced from the major review process are public reports of 6,000 - 8,000 words that describe the findings of the team of reviewers. They are the principal documented outcome of the review process and provide the main feedback to the HEI and practice placement providers. Normally, one major review report is published for each major review of nursing, midwifery and health visiting and the allied health professions.

54 A report includes:

a brief description of the review method;
the subject provision and overall aims;
an evaluation of the academic and practitioner standards achieved;
an evaluation of the quality of the learning opportunities provided;
maintenance and enhancement of standards and quality;
a summary of the main review outcomes.

55 A report provides:

information for the public, who can use it to inform choices about where to study and to have confidence in the quality and standards of the provision. Potential students, employers and other members of the public require clear and concise information about a subject provision that allows them to distinguish between different providers of similar programmes. For this purpose, each major review report contains a summary of the findings and judgements made on both standards and quality;
information for the PSRBs who can use it to inform the decisions they must take about student fitness for practice and in validating programmes for professional purposes. The report may meet PSRB needs for quality assurance of the provision in the year in which the review takes place;
information for the WDCs who need to be assured about the quality and standards of the provision being commissioned and its fitness for purpose. WDCs will use the published report as an equivalent to that of a fundamental review or an annual contract review;
information for the providers, both HEIs and NHS Trusts, who can use it as a basis for enhancing the quality of their provision, ensuring that standards are maintained, and ensuring the fitness for award. The report will meet HEI needs for annual monitoring of the provision in the year in which the review takes place;
information for the DH which needs to be assured that the provision the NHS is funding is fit for purpose and to demonstrate that those in receipt of public money are accountable for that funding;
information for prospective partners who may be considering establishing new courses in the health profession field;

56 The publication of a review report is an important stage in the review process, but it is not the final stage. The providers, HEIs and partner placement providers, will prepare an action plan setting out the actions they will take to build on the strengths identified by the review team and to address any aspects in need of improvement. This action plan will provide the focus for on-going quality assurance activities in which the HEIs, WDCs, PSRBs and the DH will have an interest.

How the process works

Major review teams

57 The Agency invites nominations of academic and practitioner reviewers from various stakeholders. These nominees are trained by the Agency and reviewers' names are listed on the Agency's web site. The Agency selects an appropriate review team from this list. The number of reviewers in each team will reflect the size, range and complexity of the education provided. There will normally be up to a maximum of eight reviewers in a team. Review teams will normally include at least one member who is a registered practitioner for each of the subject areas to be reviewed. Where applicable, there will be due regard to Parts of a Register.

58 As far as possible, within the resources available, the Agency will match the expertise of the team with the broad specialisms of the subject provision under scrutiny. The criteria for team composition are given in Annex C. The role of reviewers is set out in Annex B.

Preparation for major review

Liaison between the HEI, partner placement providers and the review team

59 HEIs may nominate a facilitator for each review, the major review facilitator (MRF). The lead WDC will nominate a person performing the strategic leadership role in the WDC for major review, a practice review facilitator (PRF) (see Annex E). The MRF role is liaison between the review co-ordinator, the PRF, the HEI and the practice placement providers. The MRF and PRF will ensure that the team obtains accurate and comprehensive information about the subject provision and its context. The Agency offers briefing sessions for MRFs and PRFs. Further details of the roles are given in Annex E. To ensure that practice placement providers in the partnership are fully prepared for the review, the MRF will liaise closely with the PRF.

Advance planning and the preparatory meeting

60 Advance planning begins with the return by HEIs and partner WDCs of a completed scoping information form sent to them before the start of the review cycle. The Agency will seek mutual agreement for a six-week period within which the review will take place. Once the review period has been agreed, the HEI and WDC partners will know the exact dates of the five days (normally maximum) that will be used for the review visit. These dates are fixed and cannot be changed. The reviewers and review co-ordinator (CR) will be recruited for those dates. Although the Agency will ask the HEI/WDCs to identify a preferred period for their review it may not be possible to accommodate their preference. If this is the case, the matter will then be discussed with the HEI and partner WDCs with a view to reaching agreement on another period. In the absence of agreement, the Agency will notify the providers of the period to be adopted. Providers with large student FTE numbers or a multiplicity of healthcare programmes may require two reviews. This division will be based on discipline/programme groupings and will be subject to negotiation and agreement with the providers on the exact timing. In the unlikely event of agreement not being reached, the Agency will notify providers of the periods to be adopted.

61 A common approach to planning the major reviews will be adopted. The review co-ordinator will initiate a preparatory meeting with the MRF, subject staff (leaders/managers) and the PRF. An Agency officer may be present. The meeting should take place after submission of the SED, and before the start of the agreed review period. The purposes of this meeting are to establish effective relationships between all parties involved in the review; to agree protocols and responsibilities; and to agree the timetable of events that will make up the review.

Documentary evidence available to the reviewers initially to inform the review

62 Although reviews will take place using a common intensity, past evidence of good or less good practice in the subject provision will be available to the review team (Annex D, Annex G). Such evidence might include, where available:

data and evidence regarding partnership working;
reports from PSRBs, such as previous annual monitoring reports;
subject review reports;
clinical governance and audit reports.

63 This evidence will assist the review teams to conduct the review efficiently and effectively, targeting their enquiries appropriately. Indicators of a need for improvement, drawn from previous subject review reports, might include:

recommendations for action;
issues highlighted for attention;
grade(s) 2 or below (or equivalent) in subject review graded profile(s);
evidence that the management of standards and quality is not in line with the expectations of good practice expressed in the Code;
suggestions of differential performance between levels and modes.

Analysis of the SED

64 The review team will use the SED prepared by the HEI and partner placement providers to help to set priorities for and to plan the review. Reviewers will consider whether the broad aims of the provision are clear and whether the intended learning outcomes allow the aims to be achieved. The reviewers will also consider whether:

the aims are an adequate expression of the broad educational purposes of the provision;
the aims reflect appropriately any relevant outcomes/competencies/proficiency statements of approved programmes;
the aims reflect appropriately any relevant subject benchmarks statements;
there is a clear relationship between the broad aims and the intended learning outcomes.

If the aims, or their relationship with the intended learning outcomes, are unclear, the providers will be asked for clarification.

65 The SED will be checked to see that all sections required (Annex D) have been included. The reviewers evaluate the SED from their specialist perspective. They produce separate written commentaries to provide an analysis of the strengths of the subject that they need to confirm, the initial questions they need to ask, and the points that require further clarification. These commentaries are distributed to all reviewers prior to Day One of the review.

Conducting the review

Testing the self-evaluation and gathering evidence

66 Reviewers assume a collective responsibility for gathering, verifying and sharing evidence that enables them to test statements made in the SED and to inform robust judgements on academic and practitioner standards and the quality of learning opportunities.

67 In general, reviewers will seek to establish that:

intended learning outcomes are clearly expressed, and that they appropriately meet regulatory and professional requirements, reflect relevant subject benchmark statements and the overall aims of the programme;
curricular content supports the intended outcomes, and that assessments effectively measure their achievement;
there is effective communication with students and academic and practice staff, so that learners, practitioners and teachers know what is expected of them;
there is effective partnership working with practice placement providers, WDCs, and PSRBs.

68 The review method addresses:

subject provision and aims;
academic and practitioner standards:

a learning outcomes;

b curricula;

c assessment;
d student achievement;
quality of learning opportunities;

a learning and teaching, including strategies for establishing, supporting and monitoring appropriate practice-based learning environments;

b student progression;
c learning resources and their effective utilisation;
maintenance and enhancement of standards and quality.

69 Reviewers gather evidence through a combination of scrutiny of documentary evidence, direct observation, and meetings with key stakeholders. The review co-ordinator is responsible for ensuring that the review team meets sufficiently often to consider the accumulating evidence and the team's findings. The MRF will normally attend these meetings. However, the MRF may not attend team meetings or parts of meetings at which direct discussion of judgements takes place.

70 Documentary evidence includes items listed in Annex G. Reviewers will consider external examiners' reports from the three years prior to the review, and will themselves sample student work. The balance between reliance upon the reports of external examiners and direct sampling of student work will depend on the confidence that reviewers have in the external examining arrangements of the HEI. The section of the Code on external examining will provide a point of reference.

71 Review teams will visit a number of practice placements (Annex H) in order to enable them to make judgements about the overall quality of the learning and teaching environment. The practice settings visited and questions raised will reflect strengths and weaknesses highlighted in the SED. The review co-ordinator will ensure that, for each profession, placements deemed suitable by the reviewers are visited.

72 Although all reviews will involve visits to practice placements and other learning environments, reviewers may not need to make direct observations of campus and practice based teaching where subject providers can provide evidence of good quality delivery. Such evidence is likely to come from internal peer review; from student questionnaires and other arrangements for gathering feedback; from the deployment of learning resources; and from student performance in assessments. Direct observation of teaching will be required if:

there are issues that reviewers feel would be best addressed by such observation;
there is insufficient other evidence that effective delivery is being achieved; or
there are indications that the learning opportunities for students are less than satisfactory.

A guidance note on the observation of teaching is at Annex I.

73 Each review includes a number of meetings between members of the HEI and partner providers and reviewers to consider the various aspects of provision related to standards and quality. The views of current and recent past students and employers (WDCs, NHS Trusts, and other partner placement providers) are important. These are recorded in meetings to enable reviewers to establish:

the students' understanding of the overall aims and intended learning outcomes;
their responses to the teaching and learning experiences they have experienced;
their and the employers' views about the appropriateness of the curricula;
views on academic and practice support and the resources available;
feedback about the courses and the learning opportunities in practice;
how the views of students and employers are represented and responded to.

The MRF and the PRF do not attend meetings with students and employers but may be consulted about the issues raised by the students/employers. Arrangements for meetings with current and past students and employers are set out in Annex J. The reviewers may also seek the views of former students, their employers, and representatives from relevant organisations or professions.

74 All reviewers are expected to identify, share, consider and evaluate evidence related to the programmes under scrutiny. Reviewers should keep notes of all meetings with staff and students, of their observations, and of comments on the quality of students' work and its assessment. Notes should be analytical rather than merely descriptive, should summarise strengths and weakness and refer to sources of information. Notes are circulated within the review team and collated by the review co-ordinator. These notes will form part of the collective evidence base on which judgements are based. Circulation of the notes is via the Agency's confidential web folders.

75 Team meetings are used to review the evidence gathered, form preliminary judgements, identify good practice, and determine which issues require further exploration. Discussion of the emerging judgements must involve the whole review team. Discussion also takes place through the web folder - each major review has a specific folder on the Agency's communications system for the use of the review team.

76 At the end of the review period, a final judgement meeting of the review team will be held, led by the review co-ordinator. The MRF will not attend this meeting. Following the meeting, the HEI and partner placement providers will be given informal feedback about the reviewers' findings and the judgements reached. A letter to the HEI and lead WDC will confirm these judgements within two weeks of the final judgement meeting. Judgements should remain confidential to those parties involved until the review report is published.

What judgements do review teams make?

Judgements on academic and practitioner standards

77 Reviewers will make a judgement about the academic and practitioner standards set and demonstrated in the programmes for each of the health professions offered by a provider. The issues that reviewers must address in order to make these judgements are described in greater detail in Annex D. Subject benchmark statements represent general expectations about standards in a discipline, particularly in relation to intellectual demand and challenge. Issues of interprofessional learning are also addressed in the benchmark statements and the health professions framework. The FHEQ sets expectations for awards at a given level more generally. Reference points are provided to assist reviewers in determining whether the provision is meeting the standards expected by the subject and professional communities, for awards of a particular type and level.

78 The reviewers will consider:

whether there are clear aims and learning outcomes that have been set appropriately in relation to the FHEQ, regulatory requirements, the relevant subject benchmark statements, the health professions framework and the relevant professional requirements;
whether the curriculum is designed to enable the intended outcomes to be achieved;
whether assessment is rigorous, equitable and consistent in measuring achievement of the outcomes; and
whether student achievement matches the intended outcomes and the level of the qualification.

79 In the light of this, review teams will state whether they have:

confidence in academic and practitioner standards: a judgement that is made if reviewers are satisfied with current standards and with the prospect of those standards being maintained into the future. A judgement of limited confidence is made if standards are being achieved but the reviewers have doubts about the ability of the HEI and partner placement providers to maintain them into the future; or
no confidence in academic and practitioner standards: a judgement that is made if arrangements are inadequate to enable standards to be achieved or demonstrated. If a failure to achieve standards has occurred in specific programme/s and/or mode/s and/or level/s only, and there is confidence in standards at other levels, the failing programme/s mode/s level/s will be identified separately.

80 If the intended learning outcomes are found not to match expectations, it is unlikely that reviewers will have confidence in the standards of the provision. An example of potential failure is when a postgraduate programme has learning outcomes set at an undergraduate level only. If a significant number of the intended learning outcomes are found not to be supported by the curriculum, it is unlikely that reviewers could have confidence in the standards of the provision. If significant intended learning outcomes appear not to be assessed, or if there are serious doubts about the integrity of the assessment procedures, it is unlikely that reviewers could have confidence in the standards of the provision. Reviewers will take account of the section of the Code on assessment of students.

81 Where an expression of 'limited confidence' in academic and practitioner standards is made, reviewers must identify in what respect the HEI and partner placement providers needs to take action to safeguard standards. The HEI in partnership with the placement providers will then be asked to prepare immediately a short-term plan, for implementation over the following three months. It will be used and monitored by the NMC and/or HPC, WDCs and the HEI, as well as by the Agency.

82 If a judgement is made that standards are not being achieved, there will normally be a further major review by the Agency within one calendar year. If standards continue not to be achieved, funding is potentially at risk. Decisions about funding are not within the remit of the Agency. Where regulatory requirements are identified as not being met, mechanisms are in place to promptly inform the relevant statutory body.

Judgements on the quality of learning opportunities

83 Judgements about the quality of learning opportunities offered to students are made against the broad aims of the provision and the intended learning outcomes of the programmes. This part of the major review process focuses on the learning opportunities on campus and in the practice placements that enable students to achieve the academic and practitioner standards. Normally, all provision will be covered by one set of composite judgements about the quality of learning opportunities. However, where the quality of provision differs across programme, mode or level, then separate judgements will be reported. This is designed to ensure that there is no averaging of overall performance that could conceal good practice deserving praise or shortcomings that require attention.

84 Reviewers consider:

the effectiveness of learning and teaching, wherever and however it takes place - in relation to curriculum content and programme aims;
student progression - the effectiveness of recruitment, and of support for students during their learning, and the extent to which they progress successfully to complete their programme;
learning resources and their effective utilisation - the adequacy of the library, equipment, accommodation, placement facilities, skills laboratories, and the effectiveness of their use by students; the qualifications and adequacy of the academic and practitioner staff resource.

85 Each of these three categories is judged as either:

commendable - the provision contributes substantially to the achievement of the intended outcomes, with most elements demonstrating good practice; or
approved - the provision enables the intended outcomes to be achieved, but improvement is needed to overcome weaknesses. The summary report will normally include a statement containing the phrase 'approved, but...', which will set out the areas where improvement is needed; or
failing - the provision makes a less than adequate contribution to the achievement of the intended outcomes; significant improvement is required urgently if the provision is to become at least adequate.

86 Where the review covers provision in two or more professional areas, separate judgements will be made on the academic and practitioner standards for each of those benchmarked areas. If the reviewers have no confidence in the standards achieved, or if they find that any element of the quality of learning opportunities on campus or in practice placements is failing, then the provision in that professional area will normally be subject to a further formal review within one year. It follows that all provision that is not failing is approved. The report of the review will state whether or not provision in each professional area is approved. Where regulatory requirements are identified as not being met, mechanisms are in place to promptly inform the relevant statutory body.

Good practice in teaching and learning in healthcare programmes

87 Review teams have the opportunity to identify good practice in any of the settings under scrutiny. The good practice:

should be directly linked to the teaching and learning supporting programmes under scrutiny;
could be of an innovative nature but it also could be ordinary practice that is undertaken very well;
should be commented upon and noted within the relevant section/s of the academic and practitioner standards and/or quality of learning opportunities.

88 Review co-ordinators need to be kept informed by the reviewers and to ensure the review team has robust evidence for the identification of good practice. Good practice will be commented upon in the summary of the report.

89 Good practice will be monitored in a number of ways. Firstly, as the review reports are drafted the frequency and nature of the good practice will be recorded. Secondly, during visit support any reference, discussion and agreement on good practice by the review team will be noted. Thirdly, an analysis of the action plans produced after a review will be undertaken to identify how the good practice is used, further developed and disseminated.

90 A formal analysis of the types, number and characteristics of good practice will be undertaken as part of the annual review trends report.

The Major Review Report

91 In all cases, major review reports will contain a narrative commentary on the strengths and weaknesses that reviewers have identified in the provision. Sections of the draft report will be written by the reviewers and edited by the CR. There will be sections devoted to each element of the judgements made about standards and quality of learning opportunities, together with a brief summary of the evidence upon which these judgements are based (see Annex D).

92 Reporting on standards takes account of the education in both theory and practice with reference where appropriate to the relevant regulatory and professional requirements, sections of the FHEQ, the Code, the health professions framework and subject benchmark statements. The narrative may identify good practice wherever it occurs, matters for particular commendation and matters of concern, together with the ability of the HEI and partner placement providers to maintain standards into the future. Good practice may include both innovative features and ordinary ones that are being done particularly well. Reporting on the quality of learning opportunities in the campus and practice environment takes the form of a narrative commentary evaluating each of the three elements of provision in both campus and practice placement settings, learning and teaching, student progression and learning resources and their effective utilisation. The commentaries will identify particular strengths, weaknesses, and areas of good practice, and will conclude with the reviewers' judgements.

93 The section on the maintenance and enhancement of standards and quality will be a narrative commentary, considering the effectiveness of the partnership between the HEI and practice placement providers; fitness for purpose of the quality assurance processes at the subject level; the partnerships between the HEI and their WDC/s and finally, the effectiveness of the relationship between HEI and their partner placement providers with the PSRBs.

94 The HEI/lead WDC receives a copy of the draft report. The HEI is responsible for appropriate liaison with placement providers for comment on matters of factual accuracy. Feedback about the review is provided to the HEI, the WDC/s and their partner placement providers through the draft report. The Agency aims to publish reports, on its web site, within 20 weeks from the end of a review.

95 Prior to the publication of reports, the HEIs and WDCs will prepare an action plan setting out the actions they will take to build on the strengths identified in the report and to address any aspects in need of improvement. The action plan will be published as part of the report, it is the responsibility of the MRF and PRF to ensure all partners have agreed and signed off the action plan. Where appropriate, the PSRBs should be involved in the action plan. The plan should be reviewed internally following normal quality assurance procedures of the HEIs. The PSRBs, the WDCs, the partner placement providers and the HEIs will make use of the action plan as well as the review report in their ongoing monitoring. The Agency will draw on both documents for the institutional audit and they may be used by other organisations such as the Commission for Health Improvement (CHI). DH will examine the reports to identify the themes which will inform the annual review trends reports.

Annual review trends report

96 At the completion of each annual schedule of reviews, an annual report of emerging trends will be produced and published. This report is designed to record the findings of the review teams and to promote best practice. It will focus on:

learning gained about standards and learning opportunities across the sector during the conduct of major reviews;
subject-specific issues arising in each of the healthcare professional areas;
good practice identified and of value for dissemination through the sectors.

In the light of these reports, the subject communities will consider any amendment they may deem necessary to the subject benchmark statements.

ISBN 1 85824 900 7

Handbook for major review of healthcare programmes QAA 034 05/2003

Contents