section photograph

QAA report on the evaluation of Prototypes 2004-05

QAA 085 09/05

Contents

Executive Summary

The Partnership Quality Assurance Framework (PQAF) proposes that there are three inter-related quality assurance processes - major review, approval, and ongoing quality monitoring and enhancement (OQME) - supported by a shared evidence base, and benchmark and quality standards. Major review was tested through prototypes in 2002 and is in the second year of a three-year cycle, due for completion in December 2006. From August 2004 to May 2005, seven prototype sites tested the proposed approval and OQME processes, the evidence base and the standards template. Two sites prototyped the approval process, four sites the OQME process, and one site both approval and OQME. Each site consisted of one or more Strategic Health Authorities and a higher education institution in partnership with NHS Trusts and other practice placement providers. The PQAF and its constituent parts will be the subject of further review and debate following the evaluation of the prototypes.

This report outlines the findings from the evaluation activities for each of the four elements of the PQAF tested through the prototypes in both campus and practice-based learning settings. Issues are highlighted and recommendations made. The prototypes tested the relevant PQAF elements with the following disciplines: nursing (all branches); midwifery; specialist community public health nursing; physiotherapy; occupational therapy; dietetics; operating department practice; radiography; clinical psychology; and podiatry. As the PQAF was initiated and developed in England, the prototypes did not explore placements outside England. Disciplines that were not included were prosthetics and orthotics, orthoptics, speech and language therapy, and paramedic science.

The report identifies the issues that arose during the prototype process, the elements of the PQAF that worked, the strengths expressed by participants and the recommendations for change. The conclusions and recommendations are detailed and we urge interested parties to consider Chapter 6 fully. However the following points encapsulate the key messages arising from the QAA evaluation.

1 Practice placement staff and their organisations were enthusiastic about their active involvement in the self-evaluation of standards. They saw that practice had a formal, meaningful partnership in the consideration of placement learning and standards of healthcare education. They welcomed the responsibility and the ownership.

2 There was a perceived benefit of enhanced partnership relations and more interprofessional collaboration facilitated through completing the standards template and enabling a broader range of those in practice to be involved in quality assurance and enhancement.

3 The number of standards needs to be reduced and the quality assurance processes to be less cumbersome and resource intensive.

4 Approval needs to fully recognise and take on board the HEIs own approval/validation processes, while making the outcomes of self-evaluation of standards meaningful to approval panels.

5 OQME needs to be less burdensome and resource intensive.

6 Students and service user representatives need to be specifically invited and involved in agenda setting and decision-making in both processes.

7 The principle of using exception reporting was welcomed. However, much greater clarity and guidance is needed to make the process more consistent, particularly in the meaning of good practice.

8 More explicit guidance is needed on the type/sources of evidence required to support the self-evaluation, the level of detail required, and what constitutes sufficient evidence.

9 Whilst there was considerable reassurance about the rigour of the self-evaluation processes at local level, there remains a lack of explicit, visible evidence and detail for external stakeholders. If integration and streamlining is to be achieved, external stakeholders need reassurance as to the robust evidence base and less generalisation.

10 Achieving self evaluation in 100 per cent of placements cannot, and indeed should not, be the aim in any of the processes. Further consideration needs to be given to targeting of placements, with a fresh perspective on assurance of standards of learning in practice. This perspective would move away from an HEI-centric and discipline-specific approach to placements to one of partnership between placement providers and all relevant HEIs.

11 Terminology needs to be clarified and be more user-friendly. For example, there is considerable reluctance, particularly at approval, to state that standards are not met or at risk.

Chapter one

Introduction

1 Partnerships were established at both strategic and operational levels between the Department of Health England (DH), the Nursing and Midwifery Council (NMC), the Health Professions Council (HPC) and its predecessor body, higher education institutions (HEIs), the National Health Service (NHS) and other stakeholders to develop a more streamlined and integrated quality assurance system for healthcare education in England. This work continued with Skills for Health, a sector skills council, under a service-level agreement from the DH, working in close partnership with the Strategic Health Authorities (SHAs)/Workforce Development Confederations (WDCs), NMC, HPC, HEIs, Trusts and other stakeholders to develop the PQAF. The PQAF establishes that there are three quality assurance processes - major review, approval, OQME - supported by a shared evidence base, and benchmark and quality standards. Major review was tested through prototypes in 2002 and is in the second year of a three-year cycle, due to complete in December 2006. From August 2004 to May 2005, seven prototype sites tested the proposed approval and OQME processes, the evidence base and the standards template.

2 The five elements of the PQAF for healthcare education are:

  • approval
  • OQME
  • major review
  • benchmark and quality standards
  • evidence base.

Figure 1 shows how it is proposed that the elements fit together within the PQAF to ensure continuous quality improvement of NHS-funded healthcare education in England.

Continuous quality improvement of healthcare education

The Partnership Quality Assuracnce Framework

Figure 1 The Partnership Quality Assurance Framework

3 After significant collaborative work, with input from a wide range of stakeholders, a proposed framework, the PQAF, was developed. QAA was contracted to organise the process of consultation on the PQAF. This consultation took place between March and June 2004. After a number of amendments in response to the feedback received, a Prototype Document (see www.qaa.ac.uk/health/framework/default.asp) was prepared and published, setting out the processes and principles to be tested. This Prototype Document contains a glossary of terms agreed by the working groups with representation from partners and stakeholders, and these terms are used within this report. QAA was contracted to coordinate the prototypes of two of the processes of the PQAF: OQME and approval, in the academic year 2004-05, as well as the evidence base and the standards. Seven sites were selected as prototypes, each involving the relevant SHA(s) and HEI. Two sites prototyped the approval process, four sites the OQME process, and one site both approval and OQME. Prototype activity took place between September 2004 and May 2005. The prototypes tested the relevant PQAF elements with nursing (all branches); midwifery; specialist community public health nursing; physiotherapy; occupational therapy; dietetics, operating department practice; radiography clinical psychology; and podiatry.

Role of the QAA

4 QAA's role related to the preparation, implementation, coordination and evaluation of the two processes, the evidence base and the standards (including the templates). To help with the implementation of the processes, QAA facilitators were allocated to each site. Each prototype could draw upon one facilitator to provide support for the HEI, the SHA/WDC and the practice placement providers. Ten facilitator days were available for the OQME prototypes and five days for the approval prototypes. How the facilitator time was used was directed and discussed by the HEI and SHA/WDCs.

5 QAA has undertaken a significant amount of work in coordinating, preparing and supporting the prototypes. Activities included:

  • finalising all documentation for publication (in a range of formats) and use by the prototype sites (documentation developed and agreed by the OQME and approval working groups)
  • undertaking initial scoping of the provision to be included, with related practice areas, in August 2004
  • subsequently holding detailed planning meetings with stakeholders at each site in September and October 2004
  • preparing the communication protocols and agreed channels
  • organising and chairing the NMC/HPC Operational Group meetings, attended by Skills for Health
  • organising and delivering the one-day briefing event for NMC and HPC Visitors in November 2004
  • organising, in response to requests from the prototype sites, a one-day forum to enable the sites to meet and share experiences in December 2004
  • responding to many queries from the prototypes
  • attending and providing input to a variety of prototype stakeholder events
  • monitoring prototype activity against timescales and goals and undertaking appropriate action in order to ensure that targets were achieved.

Evaluation methodology

6 QAA is committed to reflecting on all its activities. Therefore, it has undertaken a formal evaluation of the prototypes. The aim of this evaluation is to provide systematic analysis of the prototypes to examine the approval and OQME processes, the usefulness of the standards template and the value and appropriateness of the evidence base in supporting the two processes to determine if they:

  • provide an effective standard mechanism for assessing quality and promoting enhancement
  • streamline quality assurance requirements, thereby reducing the burden for stakeholders
  • provide outcomes that meet the needs of both stakeholders and partners
  • engage HEI and practice placement staff in quality assurance processes effectively
  • make a positive impact on the monitoring and enhancement of quality assurance of healthcare provision
  • provide an opportunity to highlight good practice to share with the sectors.

7 The evaluation was undertaken in accordance with QAA's evaluation framework (see Appendix 1). The framework ensures a consistent and robust methodology in accordance with QAA's standards. A two-stage evaluation methodology was used, examining the preparation, facilitation and delivery of the two processes, the evidence base and the standards template. The data collection strategies have been designed to ensure that meaningful qualitative and quantitative data were collected and analysed. Sources of data include:

  • documentation - including communications, preparatory material, reflections from facilitators and QAA officers, documentation prepared and used by the prototype sites, for example, a Prototype Document, an overview of practice-based learning and the PQAF document
  • observation of aspects of the process (in essence this was participant observation by QAA officers and QAA facilitators)
  • QAA postal questionnaire to all prototypes
  • QAA postal questionnaire to NMC and HPC Visitors
  • QAA focus groups - participants from Trusts and other practice placement providers, SHA/WDC staff, HEI staff, QAA facilitators, NMC, HPC, the British Psychological Society (BPS - the professional regulator for psychology) and Skills for Health senior quality assurance coordinators.

Appendix 2 outlines in more detail the evaluation techniques/approaches that were identified at the beginning of the prototypes.

8 The first stage of the evaluation was to gather and analyse data from the observations, communication and documentation. In the second stage, postal questionnaires were developed, one for those involved in the prototypes and one for NMC/HPC Visitors. These were distributed through the main HEI and SHA contacts for each of the prototypes and sent to each of the Visitors directly. Thirty-three questionnaires were returned (35.8 per cent return rate overall, 35.7 per cent return rate from Visitors). This stage then explored all analysed data further through focus groups involving each of the groups of participants in the prototypes: HEIs, SHAs/WDCs, Trusts and other placement providers, and the QAA facilitators. Two focus groups were held on 18 and 24 May 2005 in London and Leeds respectively, with a total of 53 participants (see Appendix 3).

9 In total, 86 contributions were made either through the questionnaires (see Appendix 4) or attendance at the focus groups. Of these, five contributed to both fora and all prototype sites were represented at HEI, SHA and Trust level, with the exception of one HEI which neither replied nor attended.

10 This report outlines the findings from the evaluation activities for each of the four elements of the PQAF tested through the prototypes: the evidence base; the standards and the template; the approval process; and the OQME process. Issues are highlighted and recommendations are made.

11 Skills for Health and their partners also commissioned an external evaluation, the contract being awarded to Homerton School of Health Studies, Cambridge. The external evaluation report is available on www.health-homerton.ac.uk/research/pqaf

12 The NMC considered that it would be informative to undertake an NMC Visitor verification exercise. This consisted of the Visitors spending one day meeting with clinical staff who either had undertaken, or were in the process of undertaking, the self-evaluation against practice standards. The NMC Visitors were given written guidance for this exercise and the reports were all anonymous and non-attributable. This enabled the NMC to explore the process from a regulator's perspective and to gain feedback. The activity was restricted to one day, mostly in December 2004. The report was structured around the relevant standards statements and the visits were in placements relevant to the Part of the NMC Register and the provision under scrutiny, either through approval or OQME.

The prototypes

Background

13 The eight prototypes were undertaken by seven HEIs and their partner SHAs, a lead SHA being identified for each prototype. One HEI/SHA undertook both an approval and OQME prototype.

Table 1 Scope of the prototypes and dates of main process events

Prototype Process Provision - disciplines Annual Review Meeting or Approval Event
Prototype 1 Approval Graduate entry to nursing - adult, mental health and childrens branches. 10 March 2005
Prototype 2 Approval Nursing, midwifery, physiotherapy, dietetics, occupational therapy, and operating department practice. w/c 9 May 2005
Prototype 3 Approval Midwifery, nursing, occupational therapy, physiotherapy, radiography, and specialist community public health nursing. 7 December 2004 - Allied Health Professions
8 March 2005 - Return to Practice Nursing, and RTP Midwifery, Adaptation programme for overseas nurses
22 March 2005 - Promoting Practice Effectiveness (community pathways)
Prototype 4 OQME Nursing and specialist community public health nursing 15 March 2005
Prototype 5 OQME Midwifery, nursing, and nutrition and dietetics. 16 March 2005
Prototype 6 OQME Midwifery, nursing, and specialist communitypublic health nursing. 23 March 2005
Prototype 7 OQME Nursing, midwifery, occupational therapy, and podiatry. 15 March 2005
Prototype 8 OQME Clinical Psychology, midwifery, nursing, occupational therapy, operating department practice, physiotherapy, radiography, and specialist community public health nursing. 17 March 2005

HEIs and their partner SHAs were invited to volunteer to be prototype sites, and a group of partners then agreed which sites would go ahead against an agreed set of criteria. The choice of sites ensured that rural and urban areas were included, with a wide variety of student and service-user populations. Placements available included Acute Trusts, Primary Care Trusts (PCTs), the independent sector, the social services sector and the prison services. Placements were distributed across England, the majority relatively local to the HEI and some at a considerable distance. The prototypes did not explore placements outside England. Disciplines that were not included were prosthetics and orthotics, orthoptics, speech and language therapy, and paramedic science.

Timescales

14 The prototypes ran from September 2004 to May 2005. Initial meetings to establish the scope of each prototype were undertaken by QAA in July and August 2004, followed by formal detailed planning meetings in September and October 2005. The prototypes were conducted in a shorter timeframe than would normally be used in the two processes. Therefore, it has been important in this evaluation to identify when an issue arose from the timeframe of the prototype, when it might not have arisen within the time available for the planned, normal process. The timescales were challenging and influenced the approach taken by the prototypes, determining for example, which placement areas were sampled for the self-evaluation process and the amount of evidence that was available.

Selection of placement areas

15 In all prototypes, there was a varied approach to the selection of placement areas chosen to undertake the self-evaluation of standards. The discussion on which placements were to be included commenced in the planning meetings. As the timescales for the prototypes were set by the external requirements, prototype sites were encouraged to consider a choice of placements relevant to the disciplines and quality assurance activity. In the Prototype Document (section 3.3) it is expected that all placements undertake self-evaluation 'at unit level'. Considerable debate started at the planning meetings about the meaning of unit and the feasibility of self-evaluating across all placements.

Approval prototypes

16 The Prototype Document (section 2.1) states that there are three reasons for approval:

i the development of new provision

ii a significant change in existing provision which partners agree requires approval

iii a result of OQME and/or major review processes.

The provision considered in the three approval prototypes was eligible for the first two reasons.

Prototype 1

The provision considered in this prototype was existing provision that had been revised significantly in the light of changes to NMC regulatory requirements for 'postgraduate shortened pre-registration' programmes. The prototype also included new provision, as two additional branches had been added to the programme. The approval event was conducted over a single day.

Prototype 2

This prototype included four allied health professions (AHP) and nursing and midwifery pre-registration programmes. The AHP programmes were all existing programmes that had been revised substantially. The HPC considered, in line with its own processes, that the revisions were 'significant' enough to require the presence of HPC Visitors at the approval event. Five HPC Visitors attended, accompanied by an HPC Officer. The HEI/SHA, at their own discretion, also invited representatives from the relevant allied health professional bodies.

The nursing and midwifery programmes included new (a learning disabilities branch of nursing pathway) as well as existing provision that had undergone significant revision. NMC Visitors attended, each representing one Part of the Register.

The approval event was conducted over a period of one week. The curriculum development included a significant element of interprofessional learning and represented a considerable conceptual and cultural change for the HEI and its practice partner providers. The sheer number of programmes for approval meant that the event would have been complex and logistically demanding, even without the approval prototype.

Prototype 3

Three separate (single-day) approval events were undertaken. The first event was held in December 2004 and was for AHP provision, occupational therapy, physiotherapy and radiography. After some consideration, HPC stated that, under its own approval process, there had not been sufficient significant changes to the provision to require approval by HPC, and therefore no HPC Visitor attended the event. An HPC officer and an HPC Visitor did attend as observers. The HEI/SHA, at their own discretion, invited representatives from the relevant allied health professional bodies.

The second one-day event, held in early March 2005, considered two nursing and one midwifery programmes. Two NMC Visitors from appropriate Parts of the Register attended.

The third one-day event, held in late March 2005, considered one specialist community public health nursing programme with a number of different pathways. This programme was existing provision but with sufficient revisions to warrant participation in the approval event. Two NMC Visitors from the appropriate Part of the Register attended.

OQME prototypes

17 Two of the OQME prototypes contained provision in the nursing and midwifery areas only, while the remaining three included both nursing and midwifery and the AHPs. The programmes considered were both at pre-registration and post-registration levels, and the range of awards was from certificate of higher education to doctoral level. The provision was delivered on campus and/or at a distance and/or work-based activity. All of the provision was to be included in the OQME process. The NMC made the decision that all NMC-approved programmes would be scrutinised by the NMC Visitors through the OQME process (NMC annual monitoring presently is 100 per cent with 20 per cent sampling for in-depth scrutiny once a year over a five-year cycle). The NMC agreed to receive the initial action report as the annual report from the programme provider, the NMC encourages providers to use their own internal reporting mechanisms to inform this report. NMC Visitors, where appropriate, were part of the Annual Review Meeting (ARM), the third stage of the OQME process where the initial action report produced through self-evaluation is verified. However, as the HPC does not have a separate annual monitoring process, preferring to link with HEIs' existing internal quality assurance procedures, there were no HPC Visitors present at the ARM. The HPC had agreed that it would receive, and if necessary act upon, the documentary outcomes of the ARM.

Chapter two

Evidence base

18 The Prototype Document (section 4.1) states that the purpose of the shared evidence base is to:

a provide a set of shared, transparent, measurable data, acceptable to all stakeholders that can be used to:

  • give validity to statements and thus invite confidence in claims
  • allow consistent judgements to be made against quality standards
  • allow improvement and enhancement of quality standards to be measured

b suggest appropriate sources of qualitative evidence that might be used to support/verify claims

c ensure commonality/consistency of the evidence base for the PQAF.

19 The evidence base being tested includes all information and data offered by the HEI and practice placements in support of their self-evaluation and, in the case of approvals, for use by the approval panel in order to verify the quality of provision. The evidence base proposal is that it is made up of core evidence (mainly quantitative data) and additional evidence. The evidence should always be existing information or data, not especially prepared for these processes.

20 One issue that was consistently raised in all the evaluation activities was that of providers (HEI and their placement partners) requiring additional guidance on:

  • what constitutes sufficient evidence for the standards
  • the level of detail required
  • potential sources of evidence mapped against the standards
  • how and when evidence should be available during the two processes, particularly at the ARM and the approval event
  • how external bodies, including Visitors, should access the evidence.

21 The prototype sites largely failed to include core evidence in the Initial Action Reports (IARs) or approval events. This may have resulted from the apparent lack of understanding about the evidence base by the prototypes. Different parts of the Prototype Document have conflicting statements about evidence, that core evidence is both essential and available on request only. There are also differing claims for the purpose of the evidence base in the sections on evidence and on OQME. The written guidance on the evidence base needs to be clearer and more precise.

22 Annex 5 of the Prototype Document was noted as having shortcomings in relation to the particular kinds of evidence that are relevant to providers' circumstances. Hence, one SHA produced its own list to assist staff completing the templates. The more general point is that, should the PQAF be rolled out, providers would need more help identifying relevant evidence than is currently given in Annex 5 of the Prototype Document.

23 It was apparent from the qualitative data received, and the observations undertaken that there was a perceived lack of evidence available for the ARM and approval events, and this raised some questions about the rigour of the processes. However, Trust staff noted that often a large amount of data/evidence had been gathered prior to the ARM and approval events but this was not explored or requested explicitly by panel members during the process. A minority expressed concerns that aspects of the evidence base for smaller (student numbers) disciplines could be lost during the preparation of the IAR. A small number of the Visitors and observers did note that discussion of the evidence base had taken place during the meetings, but in relation to oral evidence rather than specific documentation. Observers also noted variable linkage, by the prototypes, to evidence available from other external quality assurance processes such as major review reports and action plans. Active mapping and checking against standards is necessary in order to ensure there is continuity and no conflict. Documentation from other processes such as major review should be the full published documents and not selected excerpts.

24 Visitors noted that it was difficult, without visiting practice placement areas or talking to providers and students, to confirm the evidence base provided by the HEIs and their partner placement providers. At the Visitor briefing day, it was emphasised that the NMC verification exercise was not to be used in the OQME or approval processes as a direct form of evidence on standards or quality. However, in practice, most Visitors did directly draw upon this experience to inform their professional judgements as Visitors. In the NMC reports from practice, it is clear that evidence to support the self-evaluation of standards was available and this appeared to reassure the Visitors about the rigour of the process. Interestingly, the verification activity confirmed the consistently expressed need for Visitors to continue to have direct access to placement areas, and preferably students, as key sources of evidence.

25 The provision of quantitative data as part of the core evidence was deemed to be unclear and still awaits guidance from the DH about the standard model contract. However, there were contrasting views from two SHAs in relation to the inclusion of financial data, particularly in the OQME process. One SHA stated that the consideration of financial data should be separate from the OQME process, whereas the other SHA thought it should be an integral part of the process.

26 It was noted that a benefit of the approval and OQME processes is that they enable the structured gathering of evidence at Trust and unit level and also give the opportunity for providers to consider a wider range of evidence gathered by all the areas (disciplines) working together.

Chapter three

The standards template

27 The standards template provided the prototypes with a standardised format for the self-evaluation against the standards, within the 10 aspects (see Appendix 5). The template included colour coding to identify, within each of the 10 aspects, the locus of responsibility for each standard - HEI, placement providers or both. Feedback during the prototypes and at the focus groups indicated that some of those involved in the prototypes found the template cumbersome and would have preferred the document to be split into three smaller documents for each of the loci of responsibility. Unsurprisingly, the prototypes illustrated the need for some changes, particularly in relation to the exception reporting and the identification of standards at risk and good practice. However, overall the template was deemed workable, if cumbersome. A large majority returning questionnaires found the preparatory documentation (see Appendix 7) helpful.

28 The template was to be signed off at local level, that is unit or programme(s), and then at organisational level. The template needs to be structured to reflect this two-tier process of signing off. The present structure for signatures means that externals/members of the ARM or approval event cannot be clear as to the nature of 'the unit', the person responsible and where to ask for the evidence if wanted. Signatures at organisational level worked well in the Trusts, particularly where there was a clearly identified lead such as a clinical practice facilitator (CPF) or practice placement facilitator (PPF). The Trust board or equivalent body was rarely able to receive the organisation's self-evaluation document for formal signature. However, participants are very clear that this principle needs to be maintained and that this needs to be more explicitly built into the timescales for the PQAF processes.

29 A recommendation from participants is that suggested evidence should be referred to within the standards template so that the links are more explicit. The danger is for the evidence to become formulaic and a gathering of policy statements, rather than evidence that the policies are implemented with effect. During the prototypes, both commentary and some of the documentation reflected this concern; opportunities were not taken to test out the application of policies claimed as evidence.

30 The present layout/presentation of the self-evaluations is also a constraint. The task could be streamlined and given a clearer focus, and help discourage repetition, if exception reporting is linked to clusters or groups of standards, rather than the overall report approach. Some participants would find 'visual triggers' helpful in stimulating thought around the wide range of possibilities to consider as evidence for the self-evaluations.

31 There was some variability in the approach to completing the standards template at the practice level. Some practice areas responded in detail, with considerable critical analysis, whilst others used a tick-box approach. There is also some confusion regarding definitions of 'at risk' or 'good practice', and the need for greater guidance on terminology and completing the standards template in order to ensure consistency of approach was a key theme from the focus groups. It was suggested that it would be helpful to have 'exemplars' of the completed standards template to share with HEIs, SHAs, Trusts and other practice placement providers.

The standards

32 The standards were developed to integrate learning and quality assurance in both HEIs and practice settings and to address all key stakeholders' requirements. A specific locus of responsibility was indicated, while demonstrating the partnership between HEIs, SHAs, regulators and individual placement providers. The standards are meant to enable stakeholders' requirements for monitoring to be met but also to provide a focus on enhancement. Each standard applies to pre-registration and learning beyond registration provision.

33 In total, there are 104 standards listed in the standards template (see Appendix 6), of which 13 are evidenced through QAA institutional audit. These would not routinely be considered in the approval or OQME processes by the HEI, if that organisation was deemed successful in QAA institutional audit (Prototype Document section 5.1).

Number of standards

34 Of the 29 respondents to the questionnaire, 12 thought that there were too many standards, 17 that the number of standards was about right, and no one thought that there were too few standards. There was a consensus in the focus groups, discussions at the prototype sites and through observation, that there were too many standards. Merging of standards that were repetitive could help to rectify this. For example, standards 5.11 and 5.15 need to be combined in order to avoid suggesting that there should be two policies, one for the HEI and one for the Trust. There should be an agreed, shared policy to address students' poor performance. Other suggested standards that should be merged include 1.13 and 1.14, 5.5 and 5.17, 10.5 and 10.7.

35 In terms of the number of aspects, the consensus of the questionnaire data indicates that all the aspects should be kept. However, this consensus is not reflected in any other fora. One of the focus groups suggested that aspects 1 (Management and organisation) and 10 (Values, equalities and diversity) could be merged; another that the number of aspects should be reduced to six. One participant suggested adding an eleventh aspect, attrition and student mix, across pre-registration programmes, including ethnicity and diversity of recruitment 'to ensure rigour and to meet commissioning requirements'. In approval, many of the standards statements in Aspect 1 were dealt with long before the approval event. Table 2 in the Prototype Document shows the relationship between the 10 aspects and the eight elements in major review. If the number of aspects could be reduced, and achieve greater consistency between all three processes, then eight aspects could be identified: combining management and organisation with quality enhancement and maintenance (replacing maintenance and enhancement of standards and quality in major review), and integrate student support as part of student selection, progression and achievement. Keeping values, equalities and diversity as an aspect/element for all three quality assurance processes in the PQAF would reflect wider health policy.

36 There was also some suggestion in the focus groups that the locus of responsibility for certain standards could be modified. For example, standards 1.14, 3.7, 7.8 and 9.8 should all change from being the responsibility of placement providers to the responsibility of the HEI. One focus group suggested that all the standards should have joint responsibility. It was suggested that, if the approach of identifying the locus of responsibility was maintained, it would be easier operationally if the standards template were split into three separate documents based on the locus of responsibility; one for HEI standards, one for placement provider standards, and one for standards that have joint responsibility.

37 Three Visitors suggested that the standards should reflect regulatory requirements more explicitly or that a separate standard should be included to ensure that regulatory requirements are met. Another Visitor considered that the standards should be completely re-evaluated and significantly reduced for them to be of any worth. Although both regulators' requirements had been mapped against the standards, this lack of an explicit link did cause concern to many stakeholders. This was particularly the case for the HPC. The requirements of the allied health professional bodies were not part of this mapping process, this will need further consideration. This is needed particularly where the allied health programmes seeking approval do not include awards leading to eligibility to register with the HPC.

38 It is evident from the questionnaires, focus groups and observation that there is significant concern about the amount of paperwork that the self-evaluation against the standards generates, the resource required to undertake this part of the process and the consequent cost.

Language

39 All the data gathered, highlighted the need to review the style of language used in the standards. The consensus was that the language:

  • did not fully reflect post-registration provision or AHP provision, the terminology being too nursing-orientated
  • was too jargonistic and the statements too long.

40 A consequence of the latter was that the standards were open to interpretation which did not help consistency within the process. This was further exacerbated by the perceived generality of some statements. Examples of standards to be reviewed include:

  • 2.8 which 'caused some difficulty' and 'some confusion' in two prototypes. While this does not suggest an argument for removing it, it may suggest strengthening the wording
  • the definition of standard 6.4 is unclear because of the lack of definition of timing
  • 8.9 - one provider considered that mentoring abilities should feature within individual appraisal as part of mentor updating.

Exception reporting

41 Exception reporting is an important part of the self-evaluation process. It is a supportive commentary included at the end of each cluster of standards with a particular locus of responsibility in each aspect. It provides additional details where standards have yet to be met fully, or where attainment of the standard is at risk, or where there is good practice that should be recorded for further dissemination. However, it was apparent from the evaluation data that not all those involved in the prototypes were clear about the purpose of exception reporting and there was a request for greater guidance on this area, particularly in relation to what was deemed good practice. A varied approach to exception reporting was seen in the prototypes with variable levels of critical self-evaluation (see Table 2). Exception reporting was extensive in practice settings but far less so in the campus settings. This difference reflects a larger and more fundamental issue that (with one prototype site exception) the campus-based activity/programmes were not visible in the self-evaluation/IAR/approval documentation. Exceptions were not identified as either at risk or in good practice for programmes in documentation that went to ARMs and/or approval events.

42 While all the prototypes undertook some self-evaluation, one prototype in approval did not undertake any exception reporting, noting that the Prototype Document did not make it clear that this should be done. Where exception reports occurred, there was a positive and constructive discussion and an identification of possible solutions in the majority of cases. There was evidence that, where exception reports highlighted difficulties, these were often resolved at the local level very promptly. However, the exception reports were not always visible during the approval events or ARMs. Indeed, there were clear examples of where the exception reporting was ignored in these activities. HEIs need to consider how they might demonstrate the rigour of their self-evaluation processes and their willingness to make a critical self-evaluation of standards available for external scrutiny. Placements generally grasped this opportunity with enthusiasm and their willingness to be reflective and critical should be recognised and built upon.

Table 2 Amount of exception reporting by the prototypes

Exception Reporting in Approval Prototypes

Approval Site Standards not met Standards at risk/difficulty Good practice
1 0 15 31
2 0 31 23
3 Nil return Nil return Nil return

Exception Reporting in OQME Prototypes

OQME Site Standards not met Standards at risk/difficulty Good practice
4 0 2 22
5 2 19 50
6 0 7 6
7 0 13 7
8 2 15 41

43 Table 2 outlines the number of exceptions reports made against standards that were not met, standards that were at risk/difficulty and good practice. The small number of items in the 'not met' column probably demonstrates an unwillingness to record an unmet standard. Such exposure could be regarded as risky. However, it may also reflect that action has immediately been taken, as it should be, in response to a 'not met' judgement. Proper transparency should allow a record of such action, which shows rapid and effective response to perceived shortcomings. Positive and constructive discussion and identification of possible solutions occurred in the majority of cases. A number of the participants indicated that they would have been more comfortable with exception reporting against 'emerging issues' rather than standards at risk. This understandable concern, about stating a standard is not met, must be recognised and other categories need to be identified but without a proliferation of options.

44 The wide variation in the number of exceptions in the good practice column indicates there was uncertainty about the definition of good practice as evidenced through the observations and focus groups. There were consistent comments across all sites, from the meetings at prototype sites, by Trust staff and by external representatives, that there was a lack of shared understanding of the term 'good practice'. The glossary of terms in the Prototype Document identifies that good practice 'relates to strengths, good ideas and innovations...' (Page ii). It also says 'usually, good practice means that one or more standards have been met at a higher level'; however, no indication is given about the meaning of 'higher level'. Some participants observed that a considerable amount of filtering was necessary to remove what would be better defined either as 'standard, sound practice' or 'strength'. Whilst such items are always worth recording locally, they may not be worthy of specific dissemination.

45 Another issue that arose from the data is the organisational level at which particular standards need to be considered. For instance, it would be helpful to differentiate between those that need to be addressed at local or programme level and those that require a higher-level Trust or HEI response.

46 Some standards need re-writing to clarify whether they relate to students or staff or both.

Next >> Chapter four

Acknowledgements

The Quality Assurance Agency (QAA) is extremely grateful to all the staff, students and service users at the prototype sites, including the practice placement areas connected with these sites.

Approval

City University/North East London SHA

Coventry University/West Midlands South SHA

University of Teesside/County Durham & Tees Valley SHA

OQME

Buckinghamshire Chilterns University College/North West London SHA

University College Chester/Cheshire and Merseyside SHA

Middlesex University/North Central London SHA

University College Northampton/Leicestershire, Northamptonshire and Rutland SHA, Bedfordshire and Hertfordshire WDC, Thames Valley SHA

University of Teesside/County Durham & Tees Valley SHA

All participants have actively and positively contributed to the considerable amount of information and feedback that we have gained during the prototype experience. We hope that colleagues can recognise some of their experiences of the last year and their significant contribution.

Skills for Health are working in partnership with the Nursing and Midwifery Council, the Health Professions Council, Strategic Health Authority Workforce Directorates, higher education providers and other stakeholders to streamline and make more effective quality assurance arrangements for professional healthcare education. The Quality Assurance Agency is working with Skills for Health and their partners under contract.

ISBN 1 84482 373 3

TopTop