Final review trends report 2003-06

RG 311 03/07

in partnership with

   

PDF version (400KB)

Contents

• Overview
• Executive summary
• Chapter 1  Introduction
• Chapter 2  Healthcare organisational structures underpinning Major review
• Chapter 3  Processes
• Chapter 4  Outcomes
• Chapter 5  A summary of key learning points from Major review 
• Chapter 6  Conclusions
• Appendix 1 Variations to contract
• Appendix 2  Analysis of judgments by type of provision
• Appendix 3  Timeline for Major review
• Appendix 4  Detailed analysis of themes identified in the bullet points
• Appendix 5a List of Major reviews (2003-06) by HEI
• Appendix 5b List of Major reviews (2003-06) by lead SHA

Overview

The Quality Assurance Agency for Higher Education (QAA) was awarded the contract by the Department of Health (DH) and its partners, to develop, implement and manage Major review. A quality assurance review method that looked at all National Health Service (NHS)-funded healthcare education in England from 2003 to 2006.

Major review was developed in partnership with the DH, the Nursing and Midwifery Council (NMC), the Health Professions Council (HPC) and Strategic Health Authorities (SHAs)/Workforce Development Confederations (WDCs) with input from higher education institutions (HEIs), NHS healthcare trusts and the voluntary and independent health sectors.

This overview highlights the key findings from the analysis of outcomes from each Major review and the evaluation of the process itself. Further details are captured in the Executive summary and the report that follows.

Outcomes of Major review

• Evaluation of the reviews showed that 95 per cent of all those involved in Major review were satisfied that the reviews were conducted well and according to due protocols (section 3.2.5).
• Major review teams had confidence in the academic and practitioner standards achieved across all 90 reviews. Only one programme received a judgement of no confidence, and one programme received a judgement of limited confidence (section 4.1).
• The reviewers found that the quality of learning opportunities was commendable in more than 90 per cent of the provision. The quality of the remaining programmes was approved. No programmes were found to be failing (section 4.1).
• Major review confirmed that students¹ who successfully completed programmes were fit for practice, purpose and award (section 4.2.1.4).

Key strengths found in the provision reviewed

• HEIs and their partners work effectively together to plan, develop and implement the curriculum (section 4.2.1.2).
• Graduates and diplomates achieve their learning outcomes, are fit for purpose and are well prepared for employment in the NHS (section 4.2.1.4).
• Teaching and assessment methods are effective in promoting the integration of theory and practice using some innovative methods (section 4.2.2.1).
• Interprofessional learning is well supported in practice (sections 4.2.1.2 and 4.2.2.1).
• There is a vast array of high-quality resources to support learning and teaching (section 4.2.2.3).

Some aspects of good practice

• Post-registration and CPD curricula are well designed and effectively incorporate work-based learning (section 4.2.1.2).
• Practice placement facilitators² perform a significant role in supporting mentors, practice educators and assessors in placement areas (Sections 4.2.1.1 - 4.2.1.3, and 4.2.2.1 - 4.2.2.3.
• There is widespread use of problem-based and enquiry-based teaching methods to develop students' and trainees' critical and analytical skills (sections 4.2.1.2 and 4.2.2.1).
• Some providers offer unusual or innovative placement opportunities on campus or in non-traditional settings (sections 4.2.2.1 and 4.2.2.3).

Frequently occurring weaknesses

• There are limited opportunities in some disciplines for service user and carer involvement in curriculum development and/or delivery (section 4.2.1.2).
• There is a lack of sufficient mentor or practice assessor training, updating or support in some provision (section 4.2.2.2).
• Feedback to students on assessment is not always timely, consistent nor useful (section 4.2.1.3).
• Strategies for programme monitoring and placement audit are not always thorough nor consistently applied (section 4.2.3)
• There are high attrition rates on some programmes and strategies to improve retention are not always effective (section 4.2.2.2).

Positive features of Major review

• Major review was a success: it achieved its aims fully and numerous areas of strength and good practice have been identified (chapters 1, 4, and 5).
• Major review promoted and strengthened partnership working between HEIs, SHAs and placement providers (sections 3.2.5.3/ 3.2.5.9/ 3.2.5.11/ 4.2).
• Major review stood the test of time and was able to adapt to the changing environment, while ensuring consistency across all reviews (chapter 2, section 3.2.5.3, chapter 5).
• Major review recognised that 50 per cent of learning takes place in practice, and raised the profile of quality assurance of education in placement areas (sections 3.2.5.9 and 3.2.5.11)
• The ability to differentiate judgements ensured that not all the provision was penalised if, for example, there was a difficulty in one programme, mode of delivery or level of award (section 3.2.5.10 and 4.1).
• Bullet points enabled strengths and good practice to be celebrated as well as to identify weaknesses (section 4.1).
• Some streamlining was possible: incorporating NMC annual monitoring; sharing evidence with the HPC and British Psychological Society in relation to their monitoring and approval processes; and coordinating schedules with other QAA review methods (section 2.1).
• The roles of the Review Coordinator, Major Review Facilitator and Practice Review Facilitator were pivotal in managing the reviews (section 3.2.5).
• Review teams consistently adopted an open, friendly and professional approach to the reviews (section 3.2.5).
• Major review encouraged interprofessional team working in conducting the review and in writing the report. The teams found it highly beneficial to learn from other professions in this way (sections 3.2.5.9/3.3.1).

Challenges in Major review

• Five days of review spread across a six-week period was demanding both in terms of travel and balancing review work with the day job (section 3.2.5.12 and 3.2.5.13).
• The review model had limited ability to take account of differences in size and complexity - all providers reviewed with common intensity (section 3.2.5.12 and 3.2.5.13).
• Providers made a large amount of evidence available, which was not always well signposted, focused or targeted on the claims made in the self-evaluation document (sections 3.1.7/3.2.3/3.2.5.5).
• The lack of common definitions and presentation of data on student progression, achievement and employment (sections 3.1.7/ 3.2.5.8/ 3.2.5.12/ 3.2.5.13).
• Post-registration and continuing professional development programmes were often less visible in the review reports (section 4.4).
• The differences in terminology between the professions reviewed created some difficulties in writing the reports (section 3.2.5.13).
• The rapidly changing healthcare context in which Major review took place (chapter 2).

Thoughts for the future (chapters 5 and 6)³

• It is important to build on the success of Major review, to retain effective quality assurance processes that include an appropriate amount of externality (paragraphs 393, 395, 413, 430).
• Major review has helped to establish strong partnership working between providers which should be supported by the new quality assurance processes (paragraph 418).
• There is now an opportunity to develop a review process that is proportionate to the size of the provision, level of risk and previous performance identified in Major review reports and from other sources (paragraph 431).
• Any new review process should ensure that academic and practice elements continue to be reflected equally, and recognise interprofessional programmes which span a number of different subjects (paragraphs 401, 422)
• Judgements need to be expressed in language which is more commonly used and understood (paragraphs 406, 407)
• Major review has trained a large number of reviewers and facilitators from HEIs, SHAs and practice areas who now have considerable expertise which should not be lost (paragraphs 397, 400, 402, 425, 428).
• There should be continuing work with professional statutory and regulatory bodies, SHAs and SfH to strive towards a more streamlined quality assurance process (paragraph 431).

Chapter outlines

Executive summary

The executive summary provides a detailed synopsis of the annual trends report and is designed to give readers a complete overview of Major review before they consider the detailed sections of the main report.

Chapter 1  Introduction

This chapter provides some background and context to the Major review of healthcare programmes in England . It outlines the purpose and scope of Major review, and charts its development.

Chapter 2 Healthcare organisational structures underpinning Major review

Chapter 2 describes how QAA has worked with partners and stakeholders to streamline quality assurance of healthcare education within the changing landscape of the NHS and national policy.

Chapter 3  Processes

Chapter three outlines how Major review was planned, implemented and evaluated. Each step of the review process is considered, from the scoping and scheduling of the reviews through to the publication of the reports. It highlights the particular strengths and challenges for the review method and suggests some improvements for the future.

Chapter 4  Outcomes

This chapter details the judgements arising from Major review before discussing the strengths, good practice and weaknesses that were identified by the review teams. It also considers how providers have responded to these in their action plans. The final part of the chapter includes a statistical analysis of the student achievement, progression, employment data recorded in Major review reports.

Chapter 5  A summary of key learning points from Major review

Drawing on the earlier sections of the report, this chapter lists the learning points identified from the implementation and evaluation of the reviews.

Chapter 6  Conclusions

The final chapter provides a summary of the key outcomes from Major review, and points to note in relation to the development of any future quality assurance of healthcare education.

---

Executive summary

Introduction

This report covers all the Major reviews of healthcare education undertaken by the Quality Assurance Agency for Higher Education (QAA) during the period 2003 to 2006 and follows the two annual trends reports for 2003-04 and 2004-05. QAA has been contracted by the Department for Health ( England ) (DH) and, latterly, by Skills for Health (SfH)⁴, to review all National Health Service (NHS)-funded healthcare education programmes in England . Major review was developed in partnership with the DH, Nursing and Midwifery Council (NMC), Health Professions Council (HPC), Strategic Health Authorities (SHAs) (formerly Workforce Development Confederations), and with input from higher education institutions (HEIs), healthcare Trusts and the voluntary and independent health sectors. To reduce the quality assurance burden placed on HEIs the method incorporated NMC annual monitoring for the year that Major review took place with the inclusion of a member of the review team who was also an NMC Visitor (see sections 2.1, 3.1.2, 3.1.3). QAA also ensured that those undergoing Major review would not also be subject to a discipline audit trail in NHS-funded healthcare programmes during any institutional audit that took place during the period 2003 to 2006 (see sections 2.1; 3.1.1).

The purpose of Major review was to provide the public with the assurance and confidence that the students who successfully complete healthcare programmes are competent and safe practitioners, who are fit for purpose. The review method and subsequent reports consider, with equal emphasis, practice and campus-based learning. This was a significant development from previous review methodologies, which only considered campus-based activity, and did not recognise that 50 per cent of the students' learning takes place in practice.

The main purposes of the annual trends reports are to record the findings of review teams; promote good practice, focusing on learning gained about academic and practitioner standards, and the quality of learning opportunities; log the developments in the review process and procedures, and the changing context in which the review method operates; and highlight learning points that could helpfully feed into other future methodologies (see section 1.1).

The judgements

Of the 90 reviews undertaken, the majority of the judgements in academic and practitioner standards were 'confidence', with only two programmes receiving different judgements - one of 'limited confidence' and one of 'no confidence'. In the quality of learning opportunities, there were 87 commendable judgements in learning and teaching, 89 in student progression and 86 in learning resources and their effective utilisation. The largest number of approved judgements relating to the totality of the provision reviewed was in learning resources and their effective utilisation, although this was still a small proportion of the judgements made (see section 4.1).

The process

Evidence confirms that Major review has worked well, particularly considering the complexity and breadth of the provision being reviewed with some 15 disciplines and a broad range of awards. It met fully its stated aims and outcomes as listed in the Handbook for major review of healthcare programmes (the Handbook). It is now a tried, tested and refined review methodology. This is in no small part due to the reviewers and facilitators

The process of Major review and other variations to contract work related to the partnership quality assurance framework was managed by a dedicated Health Team at QAA. Contract reviewers (CRs), more commonly known as Review coordinators, managed the review teams and coordinated the writing of the reports.

Reviewers, who were specialists in one of the disciplines under review, were drawn from both academic and practice backgrounds. The reviews were facilitated on behalf of the providers by a nominated Major Review Facilitator (MRF) from the HEI and a Practice Review Facilitator (PRF) from the lead SHA. All of these were fully trained and briefed by the QAA Health Team before taking part in the process.The process and criteria for the selection of individual reviewers were agreed in advance by all stakeholders. A total of 438 nominations were received. Equal opportunities were an important part of the reviewer nomination and selection process, and the profile of reviewers matches that within the NHS. Normally, the review team consisted of a pair of reviewers from each discipline under review, one drawn from an academic post and one from practice. After the experience of a very large and complex review early in the cycle, the number of reviewers was limited to 10 for a six-discipline review, and reviews with five disciplines were limited to eight reviewers. Four disciplines or less were allocated two reviewers for each discipline (see section 3.1.2).

From the 373 reviewers who were trained between 2003 and 2005, 269 (72 per cent) have been used as reviewers on a Major review. Each intensive three-day reviewer training event followed the broad pattern of the Major review visit. The provider's self-evaluation document (SED) was the cornerstone of each visit. Having evaluated this, reviewers sought evidence from documentation, meetings with academic and practice staff and students to verify the claims made in each SED. The reviewer training materials developed by QAA were designed with this process in mind. Twenty-four training events were conducted. All were very positively received by reviewers, with many comments on the high quality of tuition, the effective structure of the training, and the success of these events in preparing for the reviews. The role of the NMC Visitor was included from the start of the training (see section 3.1.3).

Evaluation

Major review was evaluated with care and consistency to determine how well it worked. Following each of the 90 Major reviews, evaluation questionnaires were disseminated to the review teams, the CRs, MRFs, PRFs and the relevant NHS placements and other independent placement providers, through the PRFs. These evaluations focused on the stages of the Major review process. Fourteen focus groups, carried out through the period 2004 to 2006, were attended by 238 people, including reviewers, CRs, MRFs, PRFs, subject and practice staff (see section 3.2.5.1).

Overall, these evaluations show that responses throughout the cycle were consistently positive, with an average of 95 per cent satisfaction rate across all participants. Communication between all those involved in the reviews was predominantly seen as effective. From the perspective of providers, one of the benefits of Major review was considered to be the opportunity it afforded them to ensure that appropriate processes and procedures were in place for quality assurance, while formalising and embedding partnership working, which was also seen as a very positive output.

Preparatory meetings were useful for both the CR and providers in preparing for the review. The majority of PRFs involved were also very satisfied with the visits. Placement staff involved in visits to practice also found the process beneficial. Placement respondents welcomed the opportunity to strengthen the relationship between themselves and their partner HEIs. Evaluations also indicated that the role of the CRs was one of the most positive features of Major review. The CR was regarded as pivotal to managing the process successfully. They demonstrated three key skills - organisation, facilitation and communication. The review teams were noted as a positive feature as well, with judgements made by them found to be consistent with the dialogue during the review (see sections 3.2.5.3 to 3.2.5.11).

A fundamental underpinning of Major review was the need to acknowledge at every level that the education provided was shared equally between academic and practice learning.For logistical reasons, the number of placement areas visited was in most cases still small in relation to the total number used by the educational providers. Nevertheless they provided a legitimate cross-section that contributed significantly to the evidence base. The geographical spread of practice locations, frequently far removed from the HEI, provided serious logistical challenges. Reviewers visited each placement area in pairs. They found that visiting placements was a very beneficial aspect of the process and greatly valued the experience. CRs also commented positively about visits to practice (see section 3.2.5.9).

From the qualitative evaluations it can be seen that less positive aspects of the reviews tend to be related to process, with the most frequent references being to the timescale of reviews, the quality of the SED prepared by the institutions, and the student work and statistical data provided as evidence. Across all responses to the evaluation questionnaires, the most frequent area of dissatisfaction was in relation to the adequacy of student information and reviewers' access to it. Where issues arose, the QAA Health Team was able to respond to and resolve them, and offer further guidance, briefings or discussion forums where appropriate (see section 3.2.5.12).

Major review encouraged review teams to work interprofessionally in conducting the reviews and in writing the reports. Therefore, in each of the main sections, the reviewer responsible needed to write, on behalf of the team, about all the professions represented in the review. In turn, this meant that throughout the review visit there was a premium on interprofessional dialogue within the team. It was often challenging for the reviewers to write across disciplines, but the benefits in terms of interprofessional working far outweighed the difficulties. The rigour and accuracy of the review report are paramount, as the reports are public documents (see section 3.3.1).

Strengths, weaknesses and good practice

Each published Major review report includes key bullet points about the strengths, good practice and weaknesses of the provision as identified by the review team. Taking the 90 reports together, there were 2,686⁵ bullet points. These were analysed to identify key themes arising across the whole provision, and any trends across the reviews through the review cycle or within disciplines (see section 4.2).

Key strengths indicated by the summary bullet points in the Major review reports include the following: intended learning outcomes (ILOs) are communicated effectively to academic and practice staff and students, emphasising the strength of partnerships between the HEIs and placement areas. Partnerships work together with stakeholders to develop the curricula, demonstrating effective planning, design and approval processes. The security, integrity and consistency of assessment procedures, in setting, marking and moderating, are also common areas of strength. Graduates and diplomates are achieving their ILOs, are fit for purpose and are well prepared for employment. The most frequently occurring area of strength in learning and teaching is the effectiveness of teaching methods in promoting the integration of theory and practice. Most frequently identified strengths in student progression are recruitment, admission and induction processes, closely followed by processes for student support. In learning resources and their effective utilisation, the most common area of strength is in the quality of material resources. The effectiveness of partnership arrangements in all aspects of programme planning, delivery and monitoring are notable in the maintenance and enhancement of standards and quality (see sections 4.2.1.1 to 4.2.3).

Good practice is demonstrated across the sector and follows similar themes to those in strengths. These include the effective communication of ILOs, stakeholder involvement in the development of ILOs; partnerships, curriculum design and interprofessional learning. Good practice in assessment focuses on the range and appropriateness of methods, integrity and security of procedures, practitioner involvement and mentor support. In learning and teaching the effectiveness of learning opportunities in placements, of teaching, and of the use of learning resources is noted. The majority of good practice in student progression centres on the theme of student support. In learning resources, good practice follows similar themes to strengths, with the most frequently occurring areas being the quality of material resources, the quality of access to resources, teaching staff and partnership working. In the maintenance and enhancement of standards and quality, the operations of committees or groups involved in quality monitoring, including stakeholder representation through a student council or placement learning unit, are highlighted. Several of these bullet points focus on practice areas (see sections 4.2.1.1 to 4.2.3).

Weaknesses identified in ILOs, curricula and assessment of some provision are often in the same areas identified as strengths in other reviews. For example, in curricula, the lack of interprofessional learning becomes less evident later in the cycle of reviews. Limited opportunities for service-user and carer involvement in curriculum development were noted in some disciplines while, in assessment, the need to strengthen the consistency of feedback given to students is a common theme. There were no consistent weaknesses in student achievement. Weaknesses in learning and teaching are almost identical to areas of strength and good practice, and cover learning opportunities on placements, interprofessional learning opportunities, learning and teaching methods and the effective management of learning and teaching. In student progression, the majority of weaknesses relate to attrition rates and work undertaken to reduce them. Weaknesses identified in the learning resources of some providers include the resource of academic and practice staff, material resources, and placement provision. Strategies and processes for monitoring and placement audit are not thorough in the maintenance and enhancement of standards and quality of some provision (see sections 4.2.1.1 - 4.2.3).

Student data

Three standard data tables were included in all Major review reports: achievement, employment and progression statistics for the last three completing cohorts. These provided some useful trends although were challenging to construct. Three-quarters of all students were enrolled on pre-registration programmes. Forty-three per cent of all students were studying on diploma programmes: the majority of these were nursing students, with the remainder taking programmes in midwifery, operating department practice (ODP) and radiography. Within allied health profession (AHP) disciplines, the highest enrolment rates are in physiotherapy. Completion and achievement statistics show that, in general, the pass rate across all disciplines of pre-registration programmes was similar to post-registration programmes (96.9 and 97.5 per cent respectively). The average degree classification profile for all disciplines was similar to that for all higher education degree students studying at HEIs in England . Of the nursing, midwifery and health visiting disciplines, the achievement level is highest in health visiting. Among the larger AHP disciplines, the achievement rate is good, with 98.8 per cent achievement in physiotherapy and 97.5 per cent in occupational therapy (see section 4.4.1).

The largest proportion (22,974 students, 56.6 per cent) of students who completed programmes successfully, across all discipline areas, were employed (within six months of leaving) by employers local to the providers. The employment data revealed emerging patterns of mobility after graduation in relation to both discipline and level. It is probable that these are related to the local economic conditions and the demand for skills or employees within specific sectors at a certain skill level, or to the spatial mobility of individual students. The overall average for unemployment within six months of graduation across all disciplines during the period was 4.4 per cent (see section 4.4.2).

The average withdrawal rate across all programmes within the scope of the review was 10.2 per cent of the initial recruitment. There are six disciplines with a higher than average withdrawal rate: midwifery, ODP, podiatry, prosthetics and orthotics, radiography and orthoptics. Physiotherapy and health visiting have notably lower than average levels of withdrawal. Considered by level of award, withdrawal rates were highest for diploma programmes (11.2 per cent), compared to 9.9 per cent for bachelor's degrees (see section 4.4.3).

Action planning

The identification in the Major review report, in bullet-point form, of key strengths, weaknesses and good practice within the provision forms the basis of an action plan which is completed by the HEI/SHA and published as part of the report. The action plan addresses all the summary bullet points. Action plans have been a significant and important part of Major review and raised a number of learning points. The action plan should be an active, useable document. SMART (specific, measurable, agreed, realistic and time-bound) responses to the bullet points are vital. Some providers were unsure how good practice should be actioned where it is included in the action plan. Focus groups reported that the process of completing the action plans has continued to enhance partnership working. However, responsibility for undertaking the actions continues to lie predominantly with the HEI. The kinds of action taken can broadly be categorised as follows: production of new or enhancement of existing documents and strategies; specifically designed events, either one-off or more regularly instituted; use of information technology to enhance communication or share information; use of committees, working groups or liaison and collaboration between different groups or organisations; dissemination, particularly of strengths and good practice, across a range of groups and bodies; staff development activities; resources; scoping and evaluation (see sections 4.3.1 to 4.3.8).

Streamlining the quality assurance of healthcare education at a national level has been an underlying aim throughout Major review. There has been a series of developments enabling more sharing of evidence across review processes, working closely with the NMC, HPC and British Psychological Society (BPS). A significant advance in streamlining, noted in the DH 'streamlining' document⁶, was the production of the healthcare benchmark statements, providing a uniform expectation of standards to be reached for each discipline. The emerging health professions framework which led to the publication of the statement of common purpose for health and social care is a further streamlining development (see section 2.1).

Partnership working

The QAA Health Team has developed close working partnerships with stakeholders involved in Major review, and with other organisations that may use the outcomes of Major review as evidence for their quality assurance processes. Regular meetings were held with other stakeholders, including joint meetings with SfH and the BPS, the Commission for Health Improvement, which later became the Commission for Healthcare Audit and Inspection (more commonly known as the Healthcare Commission). The process of Major review itself has been able to adapt throughout the review cycle to respond to changes and developments in the healthcare environment, while still working within the protocols agreed by all partners at the start, to ensure consistency across all reviews. The number of students commissioned by SHAs has varied through the cycle due to workforce demand, leading to difficulties for HEIs running programmes with either very few or very large numbers of students. This has had an impact on the availability and suitability of placements, and placed strain on mentors and practice assessors dealing with larger numbers of students. Major review has also responded to national policies and initiatives that have been introduced during the cycle. QAA has ensured that reviewers understood and were up to date with these developments (see sections 2.2 to 2.4).

The trends report highlights a wealth of invaluable learning points, of which the Executive summary provides a flavour. It is hoped that the experience of, and lessons learned from Major review will contribute significantly to any future quality assurance processes, and that the Major review reports, including the action plans, provide a robust baseline on which a risk-based and proportionate quality assurance approach can be developed.

List of acronyms

AHP   allied health profession

APEL  accredited prior experiential learning

ARCS  QAA Academic Reviewer Communications Service

BPS   British Psychological Society

CHAI  Commission for Healthcare Audit and Inspection

CHI   Commission for Health Improvement

CPD   continuing professional development

CR    review coordinator (contract reviewer)

DH    Department of Health ( England )

ECR   editing contract reviewer

FEC   Further education college

FHEQ  The framework for higher education qualifications in England , Wales and Northern Ireland

HCC   Healthcare Commission

HEI   higher education institution

HESA  Higher Education Statistics Agency

HERRG Higher Education Regulation Review Group

HPC   Health Professions Council

ILO   intended learning outcome

IPL   interprofessional learning

IT    information technology

KSF   Knowledge and Skills Framework

MESQ  maintenance and enhancement of standards and quality

MPET  multiprofessional education and training

MRF   Major Review Facilitator

NHS   National Health Service

NMC   Nursing and Midwifery Council

NMH   Nursing, midwifery and health visiting

ODP   operating department practice

OQME  ongoing quality monitoring and enhancement

OSCE  objective structured clinical examination

PDP   personal development planning

PQAF  Partnership Quality Assurance Framework for Healthcare Education in England

PPF   Practice Placement Facilitator

PRF   Practice Review Facilitator

PSRB  professional and statutory regulatory body

QA    quality assurance

QAA   Quality Assurance Agency for Higher Education

SED   self-evaluation document

SfH   Skills for Health

SHA   Strategic Health Authority

SLT   speech and language therapy

VLE   virtual learning environment

WDC   workforce development confederation

Chapter 1 Introduction

1.1   Purpose and scope

1     This is the third and final trends report for the Major review of healthcare education. It covers all the reviews undertaken in the period 2003 to 2006⁷ as part of the contract between the Quality Assurance Agency for Higher Education (QAA) and the Department of Health (England) (DH), later passed to Skills for Health (SfH)⁸, to review all National Health Service (NHS) funded healthcare education programmes in England.

2     The main purposes of the annual trends reports are to:

• record the findings of the reviewers
• promote good practice, focusing on learning gained about academic and practitioner standards, and the quality of learning opportunities
• log the developments in the review process and procedures and the changing context in which the review method has operated
• highlight learning points that could helpfully feed into any future quality assurance methodology.

3     This final report presents the key themes and lessons learned during whole cycle of Major review.

4     A range of data sources has been used in producing this report:

• the 90 Major review reports and action plans (including the prototype reviews) published between December 2003 and December 2006
• the analysis of evaluation questionnaires sent to the participants in each of the Major reviews: review coordinators (CR), reviewers, subject staff (both academic and clinical) and Strategic Health Authority (SHA) staff. The evaluations of the latter two groups are coordinated by the major review facilitators (MRFs) and the practice review facilitators (PRFs) respectively
• the 14 focus groups held during 2003 to 2006, attended by 238 participants from each of the constituencies involved in the reviews
• the evaluations for all of the reviewer training events, facilitators briefings and self-evaluation document (SED) workshops
• the reflections from the QAA officers recorded through 113 days of visit support for preparatory meetings (40 days), judgement meetings (55 days) and 18 'call outs' for additional support.

5     Major review was developed in partnership with the DH, the Nursing and Midwifery Council (NMC), the Health Professions Council (HPC), SHAs (formerly workforce development confederations (WDCs)), with input from higher education institutions (HEIs), NHS healthcare Trusts, and the voluntary and independent health sectors. The review methodology was piloted in 2001-02 by six providers and evaluated by QAA and an external evaluator commissioned by the DH. The method was refined in the light of these experiences. The six Major review prototypes were converted to the agreed Major review report format in autumn 2003⁹. The first non-pilot review took place in January-February 2004 and the last review was completed in November 2006, with the final reports published in March 2007.

6     The purpose of Major review was to provide the public with the assurance and confidence that the students and trainees who successfully complete healthcare programmes are competent and safe practitioners. The aims of Major review were:

• to promote continuous improvement and to facilitate enhancement of the quality and standards of the education provided
• to test accountability, through demonstrating that the needs of the key stakeholders were being met, including contributing to fulfilling the requirements of the professional statutory regulatory bodies (PSRBs)
• to provide clear, effective and accessible public information on the quality of higher education in the healthcare professions.

7     The method was also designed to review the theoretical and practice elements of healthcare education and the integration of these two aspects in order to:

• meet the requirements of commissioners in ensuring students were fit for purpose
• meet the requirements of PSRBs in ensuring registrants were fit for practice
• meet the requirements of the HEIs in ensuring diplomates and graduates were fit for award.

8     The outcomes of Major review were expected to confirm the standard and quality of the provision, identify shortcomings and inform funding decisions through the judgements made on the provision; to share good practice across disciplines and across the sector; to provide public information; to inform ongoing quality monitoring processes; and to inform the review process of the Healthcare Commission (HCC). All of these outcomes have been achieved through the production of 90 reports on all of the NHS-funded healthcare provision in England and the positive judgements that have been made on the programmes reviewed. Good practice has been identified in these reports and has been disseminated across the sector through evaluation forums, conferences and the quarterly newsletter produced by QAA on behalf of SfH for the duration of the Major review cycle.

9     The review and subsequent reports consider, with equal emphasis, practice and campus-based learning. Major review initially included 11 disciplines, with a further four added early in the cycle. The 15 disciplines were:

• Audiology
• Clinical psychology
• Dietetics
• Health visiting
• Midwifery
• Nursing
• Occupational therapy
• Operating department practice (ODP)
• Orthoptics
• Paramedic science
• Physiotherapy
• Podiatry
• Prosthetics and orthotics
• Radiography (diagnostic and therapeutic)
• Speech and language therapy (SLT).

The scope of each review was confirmed with the providers prior to the start of the review. In the light of an early review experience, for the most part, reviews were limited to six disciplines. Where the number of disciplines offered was greater, two reviews took place.

10    Major review considered mainly NHS-funded programmes at a range of levels from certificate of higher education to professional doctorate, and included those regulated by PSRBs. Continuing professional development (CPD) modules and programmes also formed a significant part of Major review.

11    Major review did not take place in isolation. It was an integral part of the Partnership Quality Assurance Framework for Healthcare Education in England (PQAF) that has been developing since 2003 led initially by DH and latterly SfH. Although Major review was the first part of this framework to come to fruition, prototyping of two other elements of the PQAF; OQME and approval; took place in 2004-05¹⁰ and was managed by QAA. Following evaluation of the prototypes¹¹ by QAA and HSHS [Homerton School of Health Studies],who were contracted by SfH to provide an external evaluation), stakeholder forums and reference groups were convened to discuss the recommendations raised. Interim standards were subsequently published in May 2006 for use in the quality assurance of healthcare education in England on a voluntary basis, while the final framework is currently envisaged to be agreed and implemented during the academic year 2008-09.

1.2   The development of Major review

12    The Major Review Working Group, chaired by QAA, was established to oversee the initial development of the Major review process, including agreeing the methodology and the Handbook for major review of healthcare programmes (the Handbook). It included representation from the partners (DH, WDCs/SHAs, NMC and HPC) and HEI, Trust and voluntary and independent sector representatives as key stakeholders. This group met to discuss a wide range of operational issues, including potential no confidence or failing outcomes and the follow-up responses to these, and the then imminent publication of the national minimum dataset (finally published in April 2006 as part of the Multi-professional education and training (MPET) national standard contract). Following the implementation of the Major review pilots, the working group was re-formed as the Major Review Steering Group, which included the previous members together with further representation from HPC, the British Psychological Society (BPS) and from SHAs and partner HEIs with specific experience of implementing the prototype reviews.

13    The working group and subsequently the Steering Group provided excellent help and guidance to the Health Team at QAA who led the management and implementation of the Major reviews. The group provided a constructive forum to discuss any issues arising from the reviews and proposals for varying the contract to undertake additional work (see section 1.3). The Group was also instrumental in promoting Major review and encouraging nominations from the professions to become reviewers. It also agreed the format for the first annual review trends report. The last Steering Group meeting was held in November 2004, following the move of the DH quality assurance (QA) team to SfH. Following the transfer of responsibilities from DH, SfH intended to establish a Quality Assurance Key Stakeholder Advisory Forum to provide stakeholder engagement and to advise SfH about future policy direction and priorities. Following a review of steering and advisory groups, this group was established later as a Partnership Summit, the first meeting of which was held in February 2006 to bring a wider range of potential partners into the process and to establish a formal partnership agreement on the quality assurance of healthcare education. No formal partnership has been established, but the group continues to meet as a Partners' Forum to discuss and comment from their different perspectives on quality assurance in healthcare education, including developments towards the new framework and the agreement of shared principles.

14    The Major Review Steering Group included an ongoing quality monitoring and enhancement (OQME) and Approval Sub-group, established to advise on the development and implementation of the pilots for these two elements of the proposed framework. These groups were reformed into an integrated Quality Assurance Framework Management Group that first met in June 2005, chaired by SfH. This group was to provide advice on the implementation of the proposed Partnership Quality Assurance Framework for Healthcare Education in England (PQAF) elements, and formulate proposals and recommendations for change. To clarify its scope it was renamed at the second meeting in September 2005 as the QA Stakeholder Development Group.

1.3   Variation to contract

15    Variations to the contract were introduced in February 2004 in the light of a lower than expected number of reviews due to take place over the whole cycle. These variations included a regular newsletter, annual trends reports, consultations, roadshows, conferences, evaluation activities and the management, delivery and evaluation of OQME and Approval pilots. Details of variations to the contract between the DH/SfH and QAA are given in Appendix 1.

Chapter 2 Healthcare organisational structures underpinning Major review

2.1   Streamlining

16    In 2003, the DH produced a document entitled Streamlining quality assurance in healthcare education: purpose and action¹², which set out the 'context for, and Departmental approach to streamlining quality assurance of NHS-funded programmes of professional education and further development' while further developing the PQAF. The starting point for this work was noting that the three groups of key stakeholders, WDCs, PSRBs and education providers, employed different approaches to quality assuring programmes. Different evidence was used and different definitions of the same evidence, for example of attrition, were employed across the disciplines, with diverse QA approaches of the various stakeholders.

17    Major review has been successful in contributing to streamlining through a number of mechanisms, including building on existing HEI and SHA internal QA processes, making use of existing documents and data wherever possible in order to reduce the burden on providers, ongoing dialogues with partners and stakeholders - for example, the invaluable contribution of NMC and HPC on the Major Review Steering Group, sharing review scheduling information, sharing evidence and integrating processes where possible. A key example of this is the inclusion of reviewers on teams who are trained for Major review but are also qualified as NMC Visitors who, once the Major review visits are completed, use the same evidence base to produce the annual monitoring report for the NMC. In addition, in the year that Major review took place the provider's SED was accepted by NMC as the provider's annual monitoring report. A copy of the SED was forwarded by QAA to the NMC for its records.

18    QAA is also an associate signatory to the healthcare Concordat, which is designed to bring about further streamlining in the audit, inspection and review of healthcare practice.

19    Another advance in streamlining was the production of the NHS-funded healthcare benchmarks statements¹³ (DH 2003)¹⁴, which provide a uniform expectation of standards to be reached for each discipline. Prior to the publication of the benchmark statements, there were no shared overall outcome standards for healthcare programmes. Eleven benchmark statements were produced in 2001, prior to the start of the cycle, with a further five published during the cycle to cover all disciplines that fell within the scope of Major review. QAA commits to review all of its benchmark statements after five years. The DH-contracted benchmark statements for the health professions, in the context of Major review, are to be evaluated by SfH shortly, with consideration given as to whether they will need to be revised. As most of the health professions statements were published in 2001, it is timely to consider whether they should be reviewed and updated in order to maintain their currency and usefulness in providing a baseline expectation of standards to be achieved in each discipline as part of the student learning experience.

20    A related development is the emerging health professions framework, published in each health professions benchmark statement, which led to the statement of common purpose for health and social care. As benchmark statements were published, considerable overlap between disciplines was noted, from which the common framework began to emerge. This provides the shared context in which the different professions operate and helps to define the competencies and proficiencies, as defined by the PSRBs, expected and reflected in the intended learning outcomes (ILOs) of the programmes. The statement of common purpose was developed to emphasise the shared values and principles that underpin the wide spectrum of health and social care practice, and QAA was commissioned by the DH to work with stakeholders to further the development of the framework and, in turn, the statement.

21    The DH's National standard framework contract, introduced in April 2006, provides generic guidance about a national dataset and will incorporate a standard framework for QA to be used by education commissioners. The detail of this framework is still to be worked out. However, there is room within it for local variation in order to suit the needs of particular commissioners, education providers and professional requirements. SfH are currently developing quality arrangements which may be adopted nationally, when finalised.

2.2   Working with partners and stakeholders

22    In implementing the Major review, QAA has worked closely with partners and stakeholders and a variety of representatives through regular contract monitoring meetings, working groups, steering groups, advisory and development groups. QAA has also contributed to statutory regulatory body and professional body events. QAA has worked closely with the HPC, NMC, DH and SfH in managing all aspects of Major review and associated work, and has taken account of its own schedules for audit and review to ensure a coordinated and integrated approach between QAA methods and other inspection and review processes. Major review and all other PQAF-related activities have been led by the Health Team at QAA, working with support teams within the QAA Reviews Group and, for the analysis of evaluation activities, the QAA Information Unit. The Health Team has managed all aspects of the reviews.

23    A key principle underpinning QAA reviews is that they should be conducted in a spirit of dialogue and cooperation between the HEIs, practice placement providers, their staff and the review teams. The process is one of peer review and is carried out by specialist teams of peers, drawn from both academia and practice. Peer review enables judgements to be made by those who understand the healthcare programmes under scrutiny and who are familiar with teaching and learning processes. It enables judgements to be credible to subject providers, and to command their respect. For a peer review process to have credibility with external stakeholders, such as PSRBs, NHS Trusts, SHAs, other health service providers and potential students, judgements must be made in a rigorous and transparent manner and reported publicly.

24    Meetings between the QAA, DH, NMC and HPC in summer 2004 considered Approval and OQME prototypes and the involvement of the PSRBs. Further consideration was also given to how the HPC might link more closely with Major review. A briefing session was arranged for HPC and NMC Visitors involved in the Approval and OQME prototypes. Regular meetings were also held with other stakeholders, including joint meetings with SfH and the British Psychological Society (BPS), the Commission for Health Improvement (CHI), which later became the Commission for Healthcare Audit and Inspection (CHAI) (known now as the Healthcare Commission). Work continued on developing and enhancing communications between the DH, SfH and QAA.

25    During the Major review cycle significant changes and developments have taken place for the partners including the move of responsibility for the development of the quality assurance framework for healthcare programmes, of which Major review is one element, from DH to SfH under service-level agreements. This saw the QA team responsible within DH also move across to SfH. HPC has rolled out, after consultation, its QA processes and related documentation including its approval process, and annual monitoring process. NMC has also undergone considerable change for example the change to the Register (on 1 April 2004) from its previous 15 parts to three: nursing, midwifery and specialist community public health nursing, the implications of which will be discussed later in this report in sections 2.4.1 and 3.17.

26    Significant developments have also taken place in the WDCs/SHAs to which Major review has had to respond. In 2004, the WDCs merged with the SHAs, forming 28 regional bodies. Some WDCs retained their identity within the SHA, and others were subsumed into directorates or departments. This saw a significant change in personnel and meant that a number of additional briefing events for PRFs were needed often at short notice. In July 2006, the SHAs were further reconfigured and reduced to 10 organisations. This created some difficulties for publishing the reports in the 2005-06 part of the cycle, for example, in identifying the appropriate contacts to continue developing the action plans and to sign off the final published version. The destination of the responsibility for the quality assurance of healthcare education within the new SHA structures is as yet unknown but a notable reduction in resources available to support Major review and its related activities has been seen. An example of this is the PRF network; this network was established in September 2004 and met every three months across England to share experience and good practice gained from PRF involvement in the Major reviews and other elements of the PQAF. Its future is in doubt because of the movement of personnel within the SHA, the loss of the existing PRFs and funding to host the meetings.

27    Throughout the cycle the Health Team also liaised with professional bodies such as the British Paramedic Association, Association of Operating Department Practitioners and the British Academy of Audiology to attract further reviewer nominations from these disciplines and to provide information on and updates about Major review and Approval/OQME prototype processes. A member of the Health Team also attends as an observer at HPC Education and Training committee meetings and at NMC Council meetings in order to maintain currency of knowledge about the latest developments.

2.3   Healthcare education commissioning

28    A number of different patterns of commissioning have been observed through Major review. In some regions, SHAs worked together to commission by discipline, for example, in London. Rather than each SHA commissioning every discipline required, they would specialise in commissioning a few disciplines each and spot purchasing extra provision where necessary. Many HEIs developed partnerships with a number of different SHAs, which led to variations in contract monitoring as each SHA adopted a slightly different approach. Cross-commissioning was also observed, where one SHA commissions provision usually from a local provider on behalf of another SHA. Where this took place, it was included within the lead SHA for the review. In one instance an HEI did not have a 'traditional' contract with an SHA, rather it had spot purchasing arrangements with a number of SHAs. This had implications for the PRF role.

29    The number of students commissioned by SHAs has varied through the cycle due to workforce demand, leading to difficulties for HEIs running programmes with either very few or very large numbers of students. This has had an impact on the availability and suitability of placements, and placed strain on mentors and practice assessors dealing with more and more students. In the current financial climate, following a period of significant investment by HEIs in new buildings and equipment to train healthcare professionals, considerable cut backs have been made in the number of students commissioned, especially now that the MPET funding for pre-registration training is no longer ring-fenced and centrally held by the DH, but has been distributed to the SHAs.

2.4   Responding to national policy

30    Major review has also responded to national policies and initiatives which have been introduced during the cycle. The Knowledge and Skills Framework (KSF), which forms part of the Agenda for Change, has had an impact on the way healthcare professional roles are defined and developed. In turn, this has had impacted on curricula, with new expectations for competencies and skills to be gained by newly-qualified staff. National service frameworks and national occupational standards have also influenced the development of curricula. QAA worked to ensure that reviewers understood and were up to date with these developments and could identify how HEIs are able to respond to the rapidly-changing health environment. The Skills Escalator and, more recently, policy on Commissioning a Patient-led NHS have also impacted on training and development for both pre and post-registration students, with the emergence of new roles which cross traditional boundaries between disciplines, and new career development pathways, particularly for practitioners in primary care. Again the Health Team has ensured that the reviewers were aware of these developments and their relevance to the Major review process.

Chapter 3 Processes

31    The 2004-05 review trends report discussed each stage of the review process from early planning and preparation for review, to the production of the report and the action plan. This report also noted changes to the process during the cycle in response to modifications in scope or to evaluation feedback.

32    This 2003 to 2006 report examines in more detail the processes for nominating, selecting and training reviewers, noting issues of diversity and equality of opportunity, and the challenges associated with maintaining a sufficient pool of reviewers from a wide variety of backgrounds and disciplines to make up the review teams. It also considers the training and briefing workshops provided by QAA to prepare reviewers, review facilitators and education providers for Major review, drawing on the data from the evaluation questionnaires given out at each training or briefing event. This is followed by a section on evaluation of Major review from the early planning stage, through to the final day of the review (normally day 5), when the reviewers reached and delivered their judgements on the provision.

33    QAA evaluates all its review methodologies according to its evaluation strategy (see paragraph 90). This strategy utilises a combination of questionnaires and focus groups. Questionnaires are returned by the CRs, review facilitators, placement areas and reviewers. They are analysed along with feedback from focus groups, which test the questionnaire data and to provide further information and discussion about the issues raised, including areas of good practice and recommendations for the development of the process. These have been used to inform the final two chapters of this report. This chapter concludes with a discussion of the action planning process, using evidence from the four discussion forums that were held in 2005-06 to give education providers, SHAs and placement areas the opportunity to feed back on their experience of action planning and the opportunities and challenges associated with its implementation.

3.1   Pre-review

3.1.1 Scoping and scheduling

34    WDCs/SHAs that commissioned programmes within the scope of Major review and their partner HEIs were sent a scoping and scheduling form in March 2003, asking for a joint response. Following clarification of the programmes in scope, criteria were developed by the QAA Health Team for scheduling the reviews, which gave the providers two preferences of term and academic year. These also took account of NMC and HPC requirements for annual monitoring or approval, where known, and previous QAA activity with the HEI.

35    During the Major review cycle, the scope of provision at a number of institutions needed to be amended as programmes were closed, transferred to other providers or new programmes started. This affected reviewer recruitment and team composition as well as the schedule of reviews. An initial communication problem with WDCs arose in cases where the letter did not reach the appropriate person in the organisation. Following feedback from the WDCs themselves and assistance from the DH in identifying the appropriate contacts, this issue was resolved. A number of providers queried the dates they were offered, which led to negotiations with the QAA Health Team to amend the schedule. This resulted in considerable extra work. The scope of provision also needed to be verified by the WDC to confirm that all commissioned programmes were included in the reviews.

36    In August 2003, after the publication of the Handbook, containing the list of disciplines to be reviewed (Annex A), four further disciplines were added to scope by the DH, which created further challenges for scoping and scheduling. The review schedule was finalised in December 2003 with the number of reviewers, year and term agreed. Specific dates were agreed for each review between six months to one year in advance. Further scheduling revision was required due to two HEIs taking on additional provision, and a few reviews being postponed to later terms for a variety of reasons.

37    Normally, the review team consisted of a pair of reviewers from each discipline under review, one drawn from an academic post and one from practice. However, it was not always possible to achieve this balance. One review that took place in autumn 2004 covered seven disciplines and involved two CRs and a team of 13 reviewers; in the light of this experience and after discussion with the Steering Group it was agreed that the number of disciplines for each review would be limited to six and the team size limited proportionately. This resulted in some providers needing two reviews, creating further scheduling issues.

38    The high number of reviews in the autumn term 2005 led to a minor revision of the protocol for agreeing review dates, with the partners being offered only one set of dates instead of two. However, the partners were still able to negotiate jointly for alternative dates if necessary, and many took the opportunity to do so.

39    QAA internally undertook to coordinate Major review activities with institutional audit. This ensured that no HEI undergoing Major review would also be subject to a discipline audit trail in any NHS-funded healthcare programme, for the duration of the Major review cycle. QAA also ensured that audits referred to Major review reports and vice versa, to share the evidence gained from each process, and reduce the burden on the HEIs¹⁵ or the potential to duplicate evidence.

3.1.2       Reviewer nominations

40    Evidence confirms that Major review has worked well, particularly considering the complexity and breadth of the provision being reviewed. It is now a tried, tested and refined review methodology. This is in no small part due to the reviewers. The quality and standard of their work has ensured the successful completion of 90 reviews over the three-year cycle. A considerable amount of work has ensured that the process of nomination, selection and training of reviewers has been fair and monitored closely. This was not only to ensure that the methodology was being systematically followed but also that the necessary number of reviewers, against predicted requirements, were trained for each of the disciplines, thus ensuring the review teams reflected the size, range and complexity of the provision being reviewed.

41    Major review is a peer review process. The nomination, selection and training of reviewers from both academic and practice settings were an important part of ensuring that the peer process works. Review teams which include peers from HEIs, SHAs and practice settings were a central tenet of Major review. The process and criteria for the selection of individual reviewers were agreed in advance by all stakeholders through the Major Review Steering Group.

42    The reviewer nominations were initially acquired through the work of the Major Review Working Group, the DH QA reference groups and the QAA Health Team. Further reviewers were attracted by advertisements placed in professional journals in the first year. A high level of interest in becoming a reviewer was noted at the start of the process in May 2003.

43    A large amount of additional work needed to be undertaken to attract reviewers from some smaller disciplines. The WDCs had a key role in promoting nominations. The advertisement on the QAA website was adapted to reflect exact requirements. NHS computer firewalls prevented a number of targeted emails from being received. This problem was solved with the help of DH and WDC colleagues in further disseminating information. A second round of advertising was conducted in November 2003 as the number of applications dropped. Targets were reached quickly for adult nursing, and recruitment to this discipline was closed in November 2003. Occupational therapy and podiatry numbers were reached by June 2004. Recruitment for prosthetics and diagnostic radiography also closed in November 2004.

44    In the majority of disciplines, more academics than practitioners were trained. However, this was not the case in three of the allied health professions (audiology, orthoptics and paramedic science). Although more academic nominations were received in most disciplines, therefore resulting in more academics trained, the work to boost the practice figures did reduce the gap significantly, particularly in the second year of the Major review cycle (2004-05). Disciplines in which it remained difficult to recruit sufficient reviewers included audiology, clinical psychology, podiatry, occupational therapy (practitioner reviewers only), ODP, paramedic science and SLT. The relevant professional bodies assisted in attracting more nominations, along with continued work by the QAA Health Team, DH, SfH, WDCs and SHAs. Further calls for reviewers were also placed in the QA newsletter and on the DH website.

45    Good progress was made in attracting sufficient nominations from all disciplines, with the only areas of concern for the 2005-06 year being audiology, clinical psychology and ODP. Although a sufficient number of reviewers had been trained, availability of reviewers to take part in reviews was particularly difficult in these and other small disciplines. However, further nominees from these disciplines were trained at the final training event in November 2005.

46    A total of 438 nominations were received. QAA is committed to the principle of equal opportunities in its approach to selection. Therefore, equal opportunities were an important part of the reviewer nomination and selection process. Great care was taken in designing the nomination form to be user friendly and ensuring fair and equal selection criteria for all. As part of the monitoring process, nominees were asked to complete an equal opportunities form.

47    All nomination forms were screened anonymously by one QAA officer, thereby ensuring that the screening protocol was applied consistently. Prior to the screening and as part of the selection process, it was necessary to check that the nominees' registration with their PSRB was current and relevant.

48    The proportions of gender and ethnicity of reviewers were in line with national healthcare workforce statistics for the professions. The majority (74 per cent) of reviewers were white females across a wide age range. Approximately 7 per cent of NHS-employed, non-medical healthcare practitioners are from ethnic minority groups. This closely matched the nominations received for Major review in which 91 per cent were white and 8.7 per cent were from ethnic minority groups.

49    Allocation of reviewers to teams posed some problems in 2005-06 due to the large number of reviews taking place in that year, and with reviewers, particularly from practice, unable to make themselves available. However, all teams were allocated according to the agreed protocols in the Handbook.

50    From the 373 reviewers who were trained in the period 2003-05, 269 have been used as reviewers on a Major review, across 90 reviews. In percentage terms, 72 per cent of trained reviewers have undertaken a Major review (see Tables A and B below). It was the intention to try and utilise as many of the reviewers as possible. The majority took part in more than one review; the average number undertaken by each reviewer was two reviews.

51    Of the 269 reviewers who have undertaken a Major review, 162 (60 per cent) were from an academic background and 107 (40 per cent) were from a practice background. Even though these percentages show a higher number of academic reviewers, there are a number of disciplines where more practice staff have undertaken reviews. Some of the smaller professions show this to be the case (for example, audiology, ODP and health visiting).

52    The aim was always to ensure that the number of reviewers on each team reflected the size, range and complexity of the provision being reviewed, normally with up to a maximum of eight reviewers in a team. Review teams normally included at least one member who was a registered practitioner for each of the subject areas to be reviewed. As far as possible, within the resources available, QAA has matched the expertise of the team with the broad specialisms of the subject provision being reviewed. The precise number of reviewers was dependent on the number of professions represented in the provision under review, and the breadth and complexity of programmes offered. Reviewers were required to declare any conflicts of interest with providers, and HEIs/SHAs were also able to consider this before the team was confirmed.

53    The number of reviewers in a team was limited to 10 for a six-discipline review, and those with five disciplines were usually limited to eight reviewers. Four disciplines or less were normally allocated two reviewers for each discipline. At times, it was not always possible to appoint an academic and practice pair of reviewers for each discipline, and there were instances when two academic or two practice-based reviewers for a discipline were allocated to a team. However, all teams were appointed according to the protocols agreed by the Steering Group. Major review also made provision for a specialist adviser¹⁶, drawn from the pool of trained reviewers, who could, if required, provide additional specialist advice to the team on a specific matter, but they did not become a full member of the team, and withdrew after advice was given. This facility was only used in a very small number of cases across the review cycle.

54    Of the 114 adult nurses trained, 52 were used on a review (Table A). This figure is low in comparison with other disciplines, due to the significant number of nominations received and the need to use reviewers who were also trained NMC Visitors in order to accommodate the NMC annual monitoring requirements within a review team. The number of mental health nurses used on review is also low in comparison, and again this can be attributed to the need to use those who were also NMC Visitors. The limited availability of these reviewers was also a contributing factor to the comparatively low number used.

Table A: Comparison of nursing, midwifery and specialist community public health nursing reviewers trained and used

Discipline	Trained	Used on reviews
		Number	%
Adult nurse	114	52	45.6
Children's nurse	20	19	95
Learning disabilities nurse	11	8	72.7
Mental health nurse	30	17	56.6
Midwifery	30	27	90
Specialist community public health nursing (health visiting)	22	21	95.4
Totals	227	144
Average			63.4

55    A total of 146 reviewers were trained in the AHPs. This may seem small in comparison to the nursing, midwifery and health visiting numbers. However, of these, 125 reviewers have been used on reviews (85.6 per cent) (Table B).

Table B: Comparison of allied health profession reviewers trained and used

Discipline	Trained	Used on reviews
		Number	%
Audiology	5	5	100
Clinical psychology	19	18	94.7
Dietetics	5	5	100
Occupational therapy	13	13	100
ODP	12	12	100
Orthoptics	5	4	80
Paramedic science	7	3	42.8
Physiotherapy	26	22	84.6
Podiatry	13	10	76.9
Prosthetics and orthotics	1	1	100
SLT	12	10	83.3
Radiography	28	22	78.5
Totals	146	125
Average			85.6

3.1.3       Reviewer training

56    If accepted, nominees from both academic and practice backgrounds were invited to attend a three-day reviewer training event. In 2003-04, 245 reviewers were trained, with a further 128 trained in 2004-05. The first reviewer training event took place in September 2003, with a further 23 events following, the last of which took place in November 2005. During 2004-05, there was an increase in the number of last-minute trainee withdrawals due to work pressure or personal circumstances. Efforts were made to fill all available places on training sessions, and withdrawals were monitored to determine whether nominees were repeatedly failing to attend training.

57    There were set criteria for the allocation of reviewers to training: a maximum of 18 trainees at each event, with a mix of disciplines and a mix of academics and practitioners. The event was designed to simulate a review visit, with up to six participants allocated to one of three teams, each led by a CR. The training covered all aspects of the review process, from preparatory work to report writing.

58    In Major review, the SED is the foundation of the review activity. Having evaluated the SED, the reviewers seek evidence from documentation, meetings with academic and practice staff and students, and visits to practice placements to verify the claims made in it. The training materials developed by QAA were designed with this process in mind. Feedback from the first training session in June 2003, for the reviewers on the prototype reviews, informed and helped to finalise the materials. An SED, other relevant documents such as external examiners' reports, completed student assessed work notes and meetings notes, were created for programmes at the fictional University of Beeston by a group of CRs, with guidance from the QAA Health Team and the Major Review Working Group.

59    Each training event followed the broad pattern of the Major review visit up to and including the judgement meeting, at which the review team determines the extent of its confidence in academic standards and whether the components of the quality of learning opportunities are commendable, approved or failing. The training was intensive, like a review visit, with simulated meetings with subject staff and students and preparation for them, briefing presentations from QAA officers and team meetings. There was a premium placed on evening written work to provide first-draft Major review report sections. This gave the reviewers a realistic experience of reviews and feedback on their commentaries.

60    Each training event was delivered by members of the Health Team and three CRs. While those presenting from QAA remained constant, with one of two leading each event, all the CRs involved in Major review participated; therefore, comprehensive tutor notes were also developed to ensure consistency between training sessions.

61    Thirteen training events were conducted in 2003-04, followed by nine in 2004-05 and two final sessions in 2005-06. All were very positively received by the reviewers, with many comments on the high quality of tuition, the effective structure of the training and the success of these events in preparing for review. Key messages from trainees were that the training was intense and hard work but invaluable preparation for reviews, and that the quality of learning resources was excellent. Key highlights from training were the structure, knowledge and friendliness of tutors, and the varied professional composition of trainee groups, in line with the commitment to promote interprofessional learning (IPL).

62    Although the training materials were found to be appropriate, following the first three training sessions, the opportunity was taken to add a briefing to outline in more detail what was required from the overnight work to enable trainees to understand fully their written tasks. A need to monitor the use of the practice placement form on training was identified, as some trainees found it confusing. This form, along with some other training materials, was revised in summer 2004. The practice placement form was maintained in the training session and examples of completed forms were included in the training pack. The reviewers found it helpful to see examples of well and inadequately completed forms and how this affected the evidence base of a review. The materials were also amended as the context in which Major review was operating changed.

63    The role of the NMC Visitor was included from the start of the training. Normally, one Major review team member, for reviews which contained NMC-approved programmes was an NMC Visitor. This reviewer drew the material for the separate NMC monitoring report from the Major review report. Where the NMC monitoring sample covered more than one part of the register, further NMC Visitors needed to be included, which occasionally led to more than two nursing reviewers on a team.

64    At the start of Major review it was anticipated that there might have been an opportunity for some HPC QA activities to be incorporated into Major review. In the event, this did not take place because of the different time frames for the development of Major review and HPC's approval and monitoring processes. However, once confirmed, the HPC processes used Major review reports as a verified source of evidence.

65    Training sessions were observed by QAA staff involved in supporting Major review, by the DH QA team and by NMC observers to enable them to familiarise themselves further with the process. HPC observers were also invited to attend but were unable to do so.

3.1.4       Review coordinator briefings

66    The CRs are contracted by QAA for the purpose of leading each of the Major reviews. Their role is to be the independent chair of the review team, to manage each review from its start, with a preparatory meeting at the HEI, through to its completion with the editing and eventual publication of the report. They also contribute to reviewer training and other activities related to Major review. CRs are mainly independent consultants, while a small number are seconded from HEIs or further education colleges (FECs). They gave consistent and unqualified support throughout the Major review process.

67    The first CR briefing was held in May 2003 for 12 CRs who would coordinate Major reviews in the first year of the cycle. The evaluation showed that CRs considered that the training fulfilled the stated purposes in briefing them both on managing the reviews and acting as tutors for reviewer training events. CRs requested further briefing on training reviewers, and two more workshops were organised for July 2003 to meet this request. These two workshops were well received and enabled the CRs to contribute fully to the content of the tutor notes as well as being able to see the training materials in full.

68    An operational pack was introduced for CRs and QAA staff in February 2004 to support the review processes. This pack included forms, guidance and protocols. All of the forms for a review were also available to the CRs and the reviewers through a web-based platform, the QAA Academic Reviewer Communications Service (ARCS).

69    Two CR workshops were held in 2004 for new CRs, and annual CPD sessions were held in 2005 and 2006, providing updates to the operational pack and giving CRs the opportunity to share good practice of reviews and to discuss how best to allocate elements of report writing to the reviewers, or approaches to formulating good practice bullet points in the provision reviewed.

3.1.5       WDC workshops

70    It was agreed between the DH and QAA that it would be beneficial to DH, QAA and the WDCs to hold workshops specifically to brief commissioners. These workshops were designed to brief them on the Major review process and give them the opportunities for further clarification and to raise any issues specific to the partners, and were held during July and August 2003. There were 76 attendees at these workshops in total. The workshops were deemed to provide a useful overview of the Major review process and were positively evaluated, with the majority of participants welcoming the opportunity to: raise and clarify issues in an open forum; gain a good overview of the process and its partners, and network with colleagues from the other WDCs, QAA and the DH. Participants also appreciated the materials that were circulated, both before and during the workshops, particularly the opportunity to read through the Handbook.

71    In April 2004 the WDCs merged into 28 SHAs, and all Major review documentation was updated. A number of personnel changes at this point led to further briefings for new PRFs as part of the review facilitator workshops.

3.1.6       Review facilitator workshops

72    Workshops were held for MRFs and PRFs to brief them on their roles in the Major review process. Five took place in 2003-04. The first two were aimed at those completing a review in the first year, but high demand led to inclusion of those with reviews in the second and third years. The first event raised areas for consideration by QAA, which were subsequently acted on, with the success being reflected in feedback from the second event. The revised programme included interactive group work in both the morning and afternoon sessions. Key positive features of the workshops recognised by MRFs and PRFs included the opportunity to work together and to clarify issues with QAA staff. They valued being able to discuss the two roles and to plan their implementation jointly. Feedback from these events stated that they had provided the opportunity to clarify any areas of confusion, reinforce the methodology, raise awareness of the amount of work to expect and the timeframe for the various stages of the process, and provided an invaluable chance to meet colleagues from other HEIs, WDCs/SHAs and to network. The briefing sessions were found to be both useful and informative, with valuable handouts providing a clear step-by-step guide through the method. Some PRFs repeatedly attended these briefings with different HEI partners and found that they benefited, despite some repetition, and commented that the briefings were well organised and appropriate.

73    Further briefing events were held during the academic years 2004-05 and 2005-06. The later events were organised to train replacement facilitators, reflecting the turnover of staff, particularly in SHAs, following the reconfiguration of the WDCs. All attendees said that their expectations were met and they were better informed following the workshop. A final review facilitator briefing was combined with an SED-writing workshop in October 2005, due to insufficient numbers to run two separate events. This worked well and delegates found it useful to hear about the processes they were not directly involved in. All MRFs and PRFs who took part in a Major review were briefed by QAA before they took on the role.

3.1.7       Self-evaluation document workshops

74    The first SED workshop took place in July 2003. This provided a valuable opportunity for HEI and WDC colleagues to work together to consider the logistics involved. Five further workshops took place in 2003-04. They continued to be valued by all delegates attending. Increased attendance at later events was considered to be due to the delegates' desire to benefit from the experience of the QAA Health Team and CRs up to this point. In February 2004, further guidance on the quantitative data required as an annex to the SED was produced and disseminated following feedback from providers. Participants in the evaluative focus groups acknowledged that, although there was some guidance provided, there were no nationally agreed definitions and providers often used their own interpretation of the categories when completing the tables, making further analysis problematic.

75    During 2004-05, the SED workshops continued to be evaluated positively, and delegates found it helpful to consider issues around ownership of the action plan and the relationship of this to OQME. Other factors which contributed to the success of SED workshops, identified from the evaluation questionnaires, were the helpful and supportive atmosphere, provision of real-life examples, the skills and personalities of the QAA tutors, interactive sessions and networking opportunities.

76    Despite representatives from all HEIs and SHAs attending the SED workshops, in 2004-05 some SEDs were submitted that did not follow the structure required, as outlined in the Handbook. This created further challenges for the review teams. Feedback from providers at focus groups held in 2004-06 suggested that writing the SED had been beneficial for those involved, and provided the opportunity to review processes across disciplines. While the time and staff resource required to produce the SED had been considerable, the benefits from interprofessional working were considerable.

77    Additional guidelines on the structure of the SED were issued during the summer of 2005 in response to this variability. It was also made clear that any SED received in an inappropriate format would be returned to the HEI/SHA for amendment. However, following this guidance, no SED needed to be returned and only one provider was asked to provide additional material.

78    A difficult challenge for both providers and CRs leading reviews was identifying an accurate list of programmes using the correct terminology. In Major review reports, the listed provision is divided between pre-registration and post-registration or post-qualifying programmes. In relation to post-qualifying and post-registration, the NMC uses the latter term while AHPs use the former. Return to Practice, an NMC-approved programme, is a pre-registration one, despite the insistence of some institutions that they designated it post-registration. In addition, it needed to be made clear which programmes were also to form the NMC annual monitoring of 20 per cent of the provision as part of Major review, and which programmes were approved by the HPC or BPS. The changes to the NMC register and the development of interprofessional programmes also posed challenges for providers in determining under which discipline a particular programme should be cited. For example, health visiting needed to be separated from the rest of the specialist community public health nursing provision. To ensure greater consistency across the reports, including the list of programmes, all reports were edited by the Assistant Director leading Major review and an editing review coordinator (ECR).

3.2   The reviews

3.2.1       Prototype Major reviews

79    Six HEIs/WDCs volunteered to pilot the Major review process in 2002-03. The payback for this commitment was that each review report remained confidential until the Major review methodology was finalised, after which those in the pilots had the option to be reviewed again if they wished or if the methodology had changed significantly. None of the providers elected to be reviewed again and, as only minor amendments to the methodology were required before roll-out, the pilot review reports were 'converted' to full Major review reports in December 2003.

80    The prototypes allowed all aspects of the Major review process to be carefully evaluated by QAA and an external consultant commissioned by the DH¹⁷. Most importantly, they provided 'road tests' for the draft handbook, for the pattern of visit days, for protocols around review team composition and management, and for a developing report format. The outcomes of both evaluation reports were positive and provided helpful suggestions in relation to the operation of the methodology. This led to the publication of the Handbook in November 2003 and the rolling out of the reviews from January 2004.

3.2.2       The timeline for a Major review

81    The majority of the Major reviews have been conducted using the same pattern of review teams visiting the providers for five days. In all but three of the reviews, a 2+2+1 model was used, where two consecutive days are spent on site, followed by a break of usually two weeks before the reviewers return for a further two consecutive days, followed by a further break of one or two weeks before they return for the final day (see Table C). Days 1 and 4 were usually spent at the HEI, meeting staff and students, and looking at documentation and learning resources. Days 2 and 3 were spent visiting a range of practice placements, and day 5 was spent with the reviewers discussing and deciding on overall judgements, and discussing an early draft of the review report (see appendix 3 for a more detailed timeline for Major review).

Table C: Pattern of visits for Major review

Day 1	Day 2	Interim of 10-14 days	Day 3	Day 4	Interim of 10-14 days	Day 5
Review team based on campus Meetings with staff and students Scrutiny of documentation	Visits to practice		Visits to practice	Review team based on campus Meetings with staff and students Scrutiny of documentation		Clarification of any matters outstanding Team meet to discuss draft report and to reach judgements

82    Early in the review cycle, one review followed a 2+3 visit model and two small reviews followed a 2+2 model. In each of these instances there was no period of reflection before the final day and the judgements. The CRs and reviewers deemed that this lack of reflection was unhelpful and the model was time constrained, a view consistent with the first annual review trends report (2004). The remainder of the reviews have followed the 2+2+1 model. On the whole, the standard model was welcomed by the HEIs, SHAs, Trusts and the reviewers. For the HEI/SHA, although it could provide some logistical challenges in relation to the setting up of the room(s) where the reviewers were located, the pattern of days facilitated reflection and enabled preparation before a visit, including responding to the reviewers' queries. What was crucial, however, was the continued communication between the CR and the facilitators. For some reviewers, these benefits have to be set against the additional travelling and time away from work and home commitments that the model requires. For many reviewers, the key to maintaining engagement with the review during the periods away from the providers was continued communication with the CR and other team members through ARCS.

83    A consistent comment through all the evaluation was that the time available for Major review can be pressured, particularly given the complex structure of healthcare education provision. QAA wishes to acknowledge the level of commitment and workload that the reviewers, CRs and facilitators have undertaken in preparing for, during and following the reviews.

3.2.3       The SED

84    The SED provided the basis of the review on which reviewers based their initial lines of enquiry. Reviewers were encouraged to comment on the quality and usefulness of the SED in preparing for the review, in the maintenance and enhancement of standards and quality (MESQ) section of the report. Almost half the reports have only positive comments to make on the SED. Nine reports lack any comment on the SED, four of which were of prototype reviews. Only four reports had no positive comments to make.

85    More than half of the SEDs were considered to be evaluative, with one-fifth substantially lacking in evaluative comment, tending towards description for the large part. Similarly, around one-third of SEDs were noted to be honest, open and self-critical, identifying weaknesses in the provision and action taken to remedy them.

86    Eleven SEDs were considered to be detailed and informative, and an equal number were noted for their lack of detail, either in part or in whole. Six SEDs were noted for their well-organised structure, whereas 11 were noted for not conforming to QAA guidelines, with some disregarding the recommended structure throughout and some in part. Where SEDs did not conform to the appropriate format or were too generic throughout, the reviewers needed to request substantially more documentary evidence to inform their judgements, than on reviews where SEDs were detailed and well organised. Twenty-four reviews noted a well-referenced SED, and three commented on poor references.

87    Over half of SEDs showed that they were produced collaboratively, with only five lacking any evidence or detail of collaboration. The remaining reports made no comment as to who had contributed its production.

3.2.4       Visit support

88    Visit support for Major review served a number of purposes, including allowing the tracking of preparations and progress of a review. A QAA officer normally supported each review. One purpose was to monitor the process, by observing the ways in which the published review method was being implemented, giving advice and consistent interpretation of the method to the review team and subject providers, testing the evidence base, conclusions and judgements of the team during the final judgement meeting, and to provide specific support when requested. The CR could request the attendance of a QAA officer if there are any emerging concerns, for example, a potential 'no confidence' or 'failing' judgement. The QAA officer met the CR, review team, MRF and PRF, and could observe their meetings with subject staff, students and employers. On a few occasions, where a review had been deemed sensitive, the QAA officer met with the providers before the start of the review to discuss any concerns. The QAA officer was able to access supporting papers and the ARCS folder used by the review team. This additional support was welcomed by the providers and reassured them that due process was being followed in all cases.

89    The QAA officer completed a support log for each visit. The purpose of the visit largely defined the information noted, depending on whether it was a preparatory meeting, during the review itself, or at the judgement meeting. One hundred and thirteen support logs were completed for Major review: 19 for the prototypes and 94 for the reviews following roll out. Data from these logs are included in the evaluation analysis below.

3.2.5       Evaluation: how well did Major review work?

3.2.5.1     Collecting the data

90    The evaluation of the Major reviews followed QAA's evaluation policy¹⁸ and took the form of evaluation questionnaires complemented by focus groups. Following each of the 84¹⁹ Major reviews between May 2004 and November 2006, evaluation questionnaires were disseminated to the review teams, the CRs, MRFs, PRFs and the relevant NHS placements and other independent placement providers through the PRFs. The primary aim of the questionnaires was to evaluate perceptions of the Major review process from the experience of the institutions, the partner SHAs, NHS placements, the review teams and the CRs.

91    During the period between November 2004 and November 2006, 14 focus groups were conducted. Each was structured in order to include participants across all roles and included reviewers, CRs, PRFs, MRFs, and academic and practice staff. Across all focus groups, a total of 238 attendees took part. For each role there were:

• 79 reviewers
• 21 CRs
• 31 PRFs
• 32 MRFs
• 40 academic staff
• 35 practice staff.

92    Questionnaires were analysed on a termly basis. The focus groups were used to explore and verify the themes and issues identified though the evaluation questionnaires. Participants were also given the opportunity to raise other areas of interest or concern.

3.2.5.2     Analysis of evaluation data

93    The evaluation questionnaire (see appendix 5a) largely focused on the stages of the Major review process: the initial contact and ongoing support from QAA, including briefings and workshops; the review team; early review activity, including the preparatory meeting; the review period; communication between the team, the CR, the review facilitators and all subject, practice staff and students involved in meetings, and the judgements given in oral feedback at the end of day 5 of the review. All questionnaires contained three qualitative questions at the end, which asked respondents for the most positive and least positive aspects of the review and any suggestions for how the process could be improved. The most frequent themes arising are summarised in sections 3.2.5.3 to 11 below. Some questions are only completed by a particular type of respondent; for example, some questions are addressed only to CRs or only to review facilitators. Placement providers received a separate questionnaire containing six qualitative questions (see appendix 5b).

94    Over the two-year period, the evaluation questionnaire was disseminated to 516 team members, 84 MRFs, 83 PRFs at the SHAs (and through them to a wider number of placement areas) and 85 CRs20 (see Table D below). Most PRFs collated the responses from practice placement areas and, therefore, the number responding overall was higher than the number of questionnaires returned. Review teams make up the largest respondent group, with a minimum of two and a maximum of 13 reviewers on each team. The questionnaires received from them make up 66 per cent of all questionnaires returned. Of those who specified their current background on the questionnaire, 40 per cent were practitioner reviewers and 60 per cent academic reviewers. There are no notable differences in response between the reviewers from each category. The response rate was the highest of all QAA review methods during the last three years, due in no small part to the work of the QAA Information Unit in chasing responses and the work of the PRFs in encouraging more responses from placement areas.

Table D: Overall response rates to questionnaire survey by respondent group

Respondent group	Sent	Received	Response rate %
Team	516	503	97
CR	85*	82	96
MRF	84	71	85
PRF	83**	62	75
Placements	84	58	69
Total	852	776
Average			91

*For one review there were 2 CRs, due to the large size of the provision

** For one review there was no PRF

95    A total of 40,860 responses from the questionnaires returned have been recorded. A breakdown of the number of these responses is given below.

Total valid responses

Figure 1    Total number of responses to the Major review questionnaire by type of response

96    Across all questionnaires, 95 per cent of responses were 'agree' or 'strongly agree', indicating a high level of satisfaction among all participant groups throughout the review cycle.

3.2.5.3     The process overall

97    Overall, HEIs, represented by the MRFs, were very satisfied with the Major review process. There have been few responses of 'strongly disagree' and 95 per cent of the responses from this group were 'agree' or 'strongly agree'. The level of satisfaction felt by the HEIs involved has risen by 5 per cent since the start of the process. The highest level of satisfaction was found to be in relation to the reviews that took place in summer 2005, when a total 98 per cent of responses from HEIs were 'agree' or 'strongly agree'.

98    The majority of PRFs involved in Major reviews were very satisfied with the visits, with a 'disagree' response level of only 5 per cent. PRFs who have been involved in more than one review do not tend to show higher levels of satisfaction as the number of reviews increases.

99    The most positive element of QAA involvement was the briefing for the MRF and PRF. All but two MRFs (97.2 per cent) agreed that the briefing proved helpful during the review. The Handbook was also considered by most respondents (91 per cent) to provide clear guidance about the information to be included in the SED. Support for the reviews from the QAA office was viewed as effective by all but three of the MRFs (4 per cent).

100   Interactions with QAA were viewed positively by the PRFs, with all but one of the respondents agreeing that QAA supported the review in an effective way. The Handbook was said to have provided clear guidance about information to be included in the SED by all but eight of the respondents (87 per cent). The support logs have noted that the MRF and PRF also find the reviews to be open and transparent and, if concerns or complaints have been raised, they have been resolved promptly and efficiently.

101   The placement areas involved in visits by the reviewers to practice found the process beneficial. In general, visits to practice placements were seen as 'positive', 'well received' and 'very useful', comments that have been made right from the start of the process. One respondent was pleased that 'the reviewers were able to ask pertinent questions and had obviously tuned into issues which were also of concern to the practice educators'. One respondent from a later review stated 'I think visits to placement areas are crucial as these areas are central to the education of healthcare professionals'.

102   A large number of placement respondents highlighted the fact that being involved in a review has benefits for them, through the reflection involved and the highlighting and sharing of good practice. They also welcomed the chance to 'demonstrate the quality learning environments that they have developed'. One placement respondent was pleased that the reviewers 'were mindful and considerate of the needs of the clients and that the work needs to continue'. Other placements commented that a positive part of the review for them was the 'acknowledgement of the partnership between HEI and placement provider, that we play an important role in the education of students', 'recognition of joint responsibility, practice and HEI' and that it 'highlighted the importance of partnership working between HEIs and practice'.

103   Many placements also mentioned that a positive aspect of the reviews was the 'feeling that a lot of what we do is OK and to have it evaluated formally by external bodies ratifies what we do'. Other comments included the 'reassurance that we are doing a good job', the 'opportunity to review practice in line with NMC and QAA requirements', the chance for the HEI to 'view practice placements and receive feedback from QAA', and that 'the positive feedback received confirming that current practices and education provision within the workplace were very good'.

104   Placement respondents welcomed the opportunity to strengthen the relationship between themselves and the HEIs. Positive elements given by respondents include 'the opportunity to contribute along with other stakeholders to the SED', 'partnership working between all placements and university staff, with involvement of clinicians at all levels', 'partnership working between placements, WDC and working more closely with the University' and 'university and practice staff working together to promote excellent learning environments for students'.

105   From the perspective of providers in focus groups, one of the benefits of Major review was considered to be the opportunity it afforded them to ensure appropriate processes and procedures were in place for QA. It was also noted that, through the Major review process, recognition of partnership working was a very positive output, and the preparation required prior to the review visits helped build and strengthen partnership relationships. The involvement of clinical staff in the process was also considered to be very beneficial. Through the interactions that took place as part of the process, awareness was raised among all partners of the work that was carried out in other areas. This also allowed for greater understanding between the HEI, the placement providers and the SHA. However, it was noted that tensions existed where professional bodies worked together with regulators.

106   It was stated by some members of focus groups that the Major review process did not serve post-registration provision well. Some considered that the process was not sufficiently flexible to cope with the reality of provision delivery, particularly with respect to placement visits.

3.2.5.4     The work of the CRs

107   The MRFs expressed satisfaction with the work of the CRs, with 89 per cent of MRFs agreeing that CRs had demonstrated facilitation, communication and organisational skills. Many MRFs commented that the CRs were one of the most positive features of the reviews for them. Comments about the most positive aspects of the review include: 'facilitation provided by the CR', 'the responsiveness of the CR and panel to the particular circumstances of the University's (untypical) healthcare provision' and the 'professionalism and courtesy of the CR'. Another MRF noted that 'the CR was very effective in his role and established a constructive atmosphere in the meetings with staff'.

108   Equally, one area of the reviews that the PRFs were most content with was the CRs. This is shown by comments such as 'the CR was very organised and structured but also very caring of his team and all the stakeholders', 'CR was meticulous in his approach to detail, ensuring a fair review was undertaken' and 'excellent chair, thorough but well planned and organised approach'. Ninety per cent of PRFs were highly positive about the CRs' skills, with 73 per cent of the 'agree' responses being 'strongly agree'. Comments from PRFs about the CR include: 'the CR organised the review in such an effective way that there was no uncertainty for HEI and practice staff', and 'the excellent facilitation and negotiation skills of the CR who offered support and encouragement to all involved throughout the review'. The PRFs agreed strongly that 'effective communication was maintained between the CR and the PRF'.

109   Like the MRFs and PRFs, the reviewers considered the CRs a positive aspect of reviews. The majority of team members agreed that for their visits the CRs demonstrated facilitation, communication and organisational skills. Communication between the CR and the reviewers was perceived by most of the team members to be satisfactory, with a very small number of respondents (8 per cent) disagreeing.

110   It was evident from the support logs that the CR was pivotal to the process. The CR provided a consistent channel of communication between the different parties - HEI, SHA, review team and QAA. The CR also provided essential guidance to the reviewers throughout the process, keeping them focused and clear of the team's objectives. It was clear that the CR's meetings with the MRF and PRF at the end of each review day, to inform them of any lines of enquiry and those that have been closed down or make further requests for documentation or clarification, is essential. They also discussed concerns from the subject providers, for example, the progression of the review. This enabled the process to be open and transparent. 

3.2.5.5     The work of the review teams

111   MRFs were also satisfied with the review teams, mostly indicating that they demonstrated a clear understanding of the review method, applied skills and techniques appropriate to verifying the evidence base, and adopted an open and flexible approach to interactions. Across all institutions, negative responses to the five statements regarding the review team amounted to 7.6 per cent, from 22 per cent of MRFs. The largest number of 'disagree' responses regarding the teams came in relation to the range of expertise within the review team and whether this reflected the practice and academic context of the provision being reviewed.

112   The PRFs felt that the review teams were a positive feature of the reviews. Most respondents agreed that the teams demonstrated clear understanding of the method and of the significance of developing a dialogue with the HEI and the SHAs. It was also widely noted that the reviewers applied skills and techniques appropriate to verifying the evidence base. The main area of dissatisfaction among PRFs was the same as for MRFs, with 13 per cent of MRFs and 28 per cent of PRFs questioning the range of expertise and whether this reflected both the practice and academic context. However, most respondents were satisfied with the composition of the review teams, stating that 'the reviewers were knowledgeable in their subject area and reflected this in the professional manner in which they conducted the review'.

113   Sixteen per cent of PRFs did not feel that lines of inves