Introduction
Background
1 The mission of the Quality Assurance Agency (the Agency) is to safeguard the public interest in sound standards of higher education qualifications and to encourage continuous improvement in the management of the quality of higher education. To this end, the Agency carries out reviews and audits of the academic performance of institutions. This Handbook describes the Agency's method and procedures for undertaking reviews of higher education institutions (HEIs) in Wales. The arrangements for auditing and reviewing institutions in England, Northern Ireland and Scotland are described in separate documents.
2 The process of institutional review described in this Handbook has been developed by the Agency in partnership with the Higher Education Funding Council for Wales (HEFCW). For Welsh HEIs, it replaces the previous processes of continuation audit at institutional level, undertaken by the Agency at the request of Universities UK and the Standing Conference of Principals, and assessments and engagements relating to the quality and standards of provision at subject level. The former were undertaken by HEFCW and the latter were undertaken by the Agency on behalf of HEFCW as part of HEFCW's statutory responsibility for assessing the quality of education that it funds.
3 Institutional review is an evidence-based process carried out through peer review. It is part of a wider quality assurance and standards framework for Wales, developed by the HEFCW Quality Working Group, which has representatives from across the higher education sector in Wales. The framework has been developed in the context of previous assessments of the quality of subject provision in Wales having found all provision to be at least satisfactory, and more than a third of all provision to be excellent. All engagements at subject level have found grounds for confidence in the academic standards of the subjects reviewed, and have judged the quality of the students' learning experience to be either commendable or approved. These outcomes provide the basis for an approach to external quality assurance that is designed to ensure that high quality and standards are maintained and allows for swift action to address any identified weaknesses.
4 Within this Handbook, the word discipline is used to describe defined areas of academic study and the word programme is used to describe the full diet of modules, options, and other structured learning opportunities, individual research study, and associated learner support, which together comprise a pathway that leads to an award. To avoid confusion with the Agency's subject review process, the use of the word subject is limited to references to the 42 subject groupings used by the Agency in relation to both subject review and the Subject benchmark statements developed by academic communities under the aegis of the Agency.
Purpose, core principles and requirements
5 The purpose of institutional review is to meet the public interest in knowing that institutions in Wales are providing higher education awards and qualifications of an acceptable quality and appropriate academic standard. Institutional review addresses the ultimate responsibility for the management of the quality of provision and the academic standards of awards that rests with an institution as a whole. It is concerned, particularly, with the way in which an institution exercises its powers to grant degrees and/or other awards in its own name or that of an awarding body.
6 Institutional review is based on a number of core principles. The process seeks:
- to provide robust assurance of the effectiveness of quality assurance and standards mechanisms embedded across HEIs in Wales;
- to make available to a wide range of stakeholder groups accurate and timely data and reliable information about the quality of the learning opportunities and academic standards across Wales;
- to provide clear statements and evidence of continuous quality enhancement and improvement activities being undertaken within HEIs in Wales;
- to provide an efficient and cost effective process for HEFCW to operate and institutions to work within.
- continuing commitment by institutions to an external element in quality assurance mechanisms, exemplified through involvement in external examining, assessment, curriculum design, course and programme validation, feedback processes and student complaints procedures;
- ownership of quality and standards residing with institutions with a recognition of the need for this to be widespread, particularly in the context of promoting quality improvement and enhancement across the sector;
- an emphasis on 'proportionality' - that intervention should be in proportion to risk, but recognising that where problems are identified firm action will be taken by HEFCW;
- comparability of judgements with other countries of the UK, although not necessarily based on identical processes and procedures;
- recognition of the context of quality assurance in Wales including, in particular, the requirements of the Welsh Assembly Government.
8 Institutional review applies to all higher education provision in Wales.
The review cycle
9 Reviews will take place on a six-year cycle beginning during the academic year 2003-04. Three years after each review, the institution concerned will be required to submit a report to the Agency commenting on its progress since the review and on other relevant developments, and will receive a short visit from the Agency to discuss these matters (see Annex I). Institutions with reviews in the latter half of the first six-year cycle (ie in 2006-07, 2007-08 or 2008-09) will also be required to submit a report, three years before the review, on the progress made in addressing the findings of previous audit and engagement reports by the Agency.
10 The institutional review process includes consideration of each institution's approach to the information requirements developed by the HEFCW Quality Working Group as part of the quality assurance and standards framework for Wales (see Annex D). Institutions in Wales are expected to meet the information requirements described in Annex D by the end of the calendar year 2004, subject to developments elsewhere in the UK. The Agency is aware that institutions visited early in the review cycle may not have complete information sets available for consideration, and will provide advice to review teams as appropriate.
The Agency's operational principles and process standards
11 The process of institutional review requires a high degree of openness, transparency and trust in the partnership between the Agency and each HEI. To ensure that the process is robust, impartial and deserving of that trust, the Agency's work is underpinned by a set of general principles and the adoption of explicit process standards. Further details are provided in Annex J.
12 Reviews are conducted in accordance with the terms of the Agency's approved Welsh Language Scheme (the Scheme), the full details of which are published on the Agency's web site. The specific terms of the Scheme relating to the process of institutional review are provided in Annex K.
The institutional review process in summary
Scope
13 Institutional reviews examine two main areas:
- the methods by which an institution secures the quality of its programmes and the academic standards of its awards, and the effectiveness of its internal quality assurance structures and mechanisms. This provides public information on an institution's soundness as a provider of higher education qualifications of national and international standing;
- the accuracy, completeness and reliability of the information that an institution publishes about the quality of its programmes and the academic standards of its awards. This provides information on the trust that can be placed in an institution's own published descriptions of the quality and standards of its provision; it also makes that description more useful to students and other interested parties.
14 In examining these areas, review teams give consideration to:
- publicly available information about the quality of programmes and the standards of awards;
- internal reviews of academic provision, and their outcomes;
- the ways in which the institution monitors its provision through the use of external evaluation;
- the use made of external reference points, including the Agency's Code of practice for the assurance of academic quality and standards in higher education (the Code of practice), The framework for higher education qualifications in England, Wales and Northern Ireland (the FHEQ), Subject benchmark statements, and the Credit and Qualification Framework for Wales;
- the development and use of programme specifications;
- the ways in which the academic standards expected of students are articulated through programme specifications;
- the experience of students as learners;
- the engagement of students in the evaluation and review of programmes;
- procedures for student complaints and academic appeals;
- the means by which the quality of teaching staff is assured, including appointment criteria and the ways in which teaching effectiveness is appraised, improved and rewarded.
15 As part of the process, review teams consider the ways in which an institution's quality assurance structures and mechanisms operate in practice. Teams carry out this task by selecting a range of thematic trails for pursuit during the review visit. The trails are concerned with testing how well institutional procedures work and how effective they are in practice.
Collaborative provision
16 The review process applies to all higher education provision in Wales. It includes higher education programmes provided by further education institutions (FEIs). Such provision is assessed through the appropriate HE partner institution and, where relevant, representatives from partner FEIs are asked to participate in reviews. However, where an institution's collaborative provision is too large or complex for a reliable scrutiny to be undertaken, it is not included in the review. Instead, the Agency undertakes a separate collaborative audit, using a separate process, of the way in which the provision is managed by the institution concerned. The Agency also expects to continue with its programme of audits of specific partnerships between UK institutions and providers overseas.
17 Reviews of institutions that are members of, or have another form of relationship with, the federal University of Wales include consideration of the interaction with the University of Wales, particularly in respect of quality assurance structures and mechanisms, and academic standards. The review cycle also includes a review of the University of Wales in its own right, as a separate entity.
Information and evidence
18 To enable them to form their judgements, review teams have available a variety of information sources about an institution, including:
- the information required as part of the quality assurance and standards framework for Wales. The Agency is aware that institutions visited early in the review cycle may not have complete information sets available for consideration, and will provide advice to review teams as appropriate;
- a self-evaluation document (SED);
- the mid-cycle progress report produced by the institution;
- information submitted by representatives of students of the institution;
- reports on the institution by the Agency and other relevant bodies within the six years preceding the review, including the Agency's brief report on the mid-cycle review;
- information (written or oral) acquired during and after the briefing visit, and during the review visit. This is likely to include contextual information relating to the information sets, such as commitee papers.
19 In 2002 the Agency undertook a number of subject-level engagements in HEIs in Wales. The reports of the engagements were not published and remain confidential to HEFCW and the HEI concerned. As a result, the Agency informs review teams of the engagements that have taken place but does not itself provide the reports for the teams. However, it strongly encourages institutions to make the reports available to teams, to provide as full a picture as possible of external review activity and findings in the period preceding the institutional review.
Judgements and reports
20 Each institutional review results in a report published by the Agency. The report sets out the review team's judgements on:
- the confidence that can reasonably be placed in the soundness of the institution's present and likely future management of the quality of its programmes and of the academic standards of its awards. This judgement provides one of three expressions of confidence - 'confidence', 'limited confidence' or 'no confidence'; and
- the reliance that can reasonably be placed on the accuracy, integrity, completeness and frankness of the information that an institution publishes about the quality of its programmes and the standards of its awards.
21 In making these judgements, review teams give particular attention to the Agency's expectations in two key areas. The first expectation is that institutions are making strong and scrupulous use of independent external examiners in summative assessment procedures. The second is that a similar use is made of independent external persons in the internal periodic review of disciplines or programmes. Teams are unable to make a judgement of confidence in an institution if either of these elements is seriously deficient.
22 Institutional review reports also provide comment on other matters, including the characteristics, strengths and limitations of the institution's internal quality assurance methods. The reports highlight features of good practice and make recommendations for further consideration by the institution.
Students
23 Students are central both to the principal focuses of review and to the review process itself. Review teams scrutinise a range of matters directly relevant to students, including the accuracy of the information provided for them, the ways in which their learning is facilitated and supported, the means by which they can give feedback on the quality of provision, the means by which they can make a complaint or an academic appeal, and their involvement in internal reviews. In each review, students are invited to participate in the key stages of the process. Their representative body - normally the Students' Union, or equivalent - has the opportunity to participate in the preliminary meeting between the Agency and the institution and may make a written submission to the team in advance of the review visit. Officers of the representative body and other students are invited to participate in specified meetings during the briefing and review visits, and have the opportunity to ensure that the team is aware of matters of primary interest or concern to them. Further information about the involvement of students is provided in Annex C.
Review personnel
24 Review teams comprise three reviewers and a review secretary, all of whom have expertise and recent experience relevant to their roles. Reviewers and review secretaries are selected by the Agency, generally from nominations made by institutions, on the basis of published selection criteria. All teams are expected to include at least one member with recent and substantial experience of the higher education sector in Wales, and at least one member from a HEI in England, Scotland or Northern Ireland.
25 All review team members are provided with training to ensure that they are familiar with the purpose, core principles and requirements of the review process, and their own roles and tasks within it. The training includes the provision of specific information about the context of higher education in Wales and the role of the federal University of Wales. Further information about the Agency's arrangements for selecting and training review teams is provided in Annex E.
26 Institutions are invited to nominate an institutional facilitator to liaise between the review team and the institution and to provide the team with advice and guidance on institutional structures, policies, priorities and procedures. Further details about the role of the facilitator are provided in Annex F.
27 Each review is coordinated by an assistant director of the Agency. In the period preceding the review visit, the assistant director provides advice to the institution on its preparations for the review. He or she accompanies the team during the briefing visit and for part of the review visit, providing advice as appropriate. It is the responsibility of the assistant director to test that the team's findings are supported by adequate and identifiable evidence, and that the review report provides information in a succinct and readily accessible form.
How the process works
Preparation
28 An outline of the institutional review process is provided in Annex A. A preliminary meeting between the institution and the Agency takes place around nine months before the review visit. The purpose of the meeting is to clarify the scope of the review; to discuss the interactions between the institution, the Agency and the review team; to ensure that the SED will be well-matched to the process of review; and to discuss the information requirements. The meeting also includes an opportunity for discussion between the Agency and officers of the student representative body about the student contribution to the review. Thereafter, until the submission of the SED, the Agency offers additional advice and guidance on the process at the request of the institution.
29 The review team is appointed by the Agency no later than 12 weeks before the review visit. The institution is notified of the names of the members of the team as soon as the full membership has been confirmed.
Documentation
30 The institution is required to submit its initial documentation for the review no later than 12 weeks before the review visit. The initial documentation comprises the SED and other documents that the institution wishes to provide for the review team in advance of the briefing visit. If representatives of students within the institution wish to make a separate written submission to the team, it is also sent to the Agency at this stage. Guidance on preparing the SED and the student submission is provided in Annexes B and C. On receipt, the documentation submitted by the institution and its students is distributed by the Agency to the team.
The briefing visit
31 The visit to the institution has two parts. The first part, the briefing visit, is held around five weeks before the review visit and lasts for three days, of which two days are spent at the institution. The purposes of the briefing visit are to permit the review team to gather any additional (written or oral) information that it requires to clarify what it has already received; to consider its detailed lines of enquiry for the review visit, including thematic trails (see below, paragraphs 37-38); to propose a programme for that visit; and to allocate particular responsibilities to individual team members. The assistant director accompanies the team throughout the visit.
32 The briefing visit is focused at the level of institutional management. It has a standard structure and includes meetings with representatives of the institution's staff (normally those who are involved in quality management at a senior level) and its students. The meetings with staff offer the institution an opportunity to bring the review team up to date on institutional developments and changes since the SED was submitted. The meeting with students offers an opportunity for student representatives to offer their perspective on the SED, and a further opportunity to draw the team's attention to matters of interest to the student body.
33 Following the briefing visit, the assistant director writes to the institution to confirm the programme for the review visit (including the thematic trails to be followed) and the illustrative documentation that the review team would wish to be made available in advance of, or at the start of, the visit. The documentation is limited to no more than what is necessary to inform the team's proposed enquiries.
The review visit
34 For most institutions, the review visit extends over five working days, Monday to Friday. The detailed programme for each visit, based around meetings with staff and students, is decided by the review team. Most visits include:
- opportunities for the team to read the documentation provided to support the review, including external examiners' reports, documentation relating to internal reviews, and reports from professional, statutory and regulatory bodies;
- discussions with staff and students of the institution and, where appropriate, its collaborative partner institutions;
- pursuit of the selected thematic trails;
- during the closing stages, a meeting with senior staff to discuss aspects of the review, and any matters that are outstanding or that require further clarification;
and exploration of:
- the institution's approach to quality assurance;
- the relationship between institutional procedures and their operation at the level of the programme, discipline or academic department, giving particular attention to the effectiveness of internal reviews of programmes and awards;
- where appropriate, the institution's interaction with the University of Wales, particularly in respect of quality assurance structures and mechanisms, and academic standards;
- where appropriate, the role of collaborative partner institutions in quality assurance processes;
- the ways in which the institution is using the FHEQ, the Code of practice, Subject benchmark statements and the Credit and Qualification Framework for Wales;
- procedures for student complaints and academic appeals, with particular reference to the relevant section of the Code of practice;
- the accuracy, completeness and reliability of the information published for students, potential students and others;
- the ways in which students are supported and their opportunities to learn are optimised;
- the ways in which the quality of teaching staff is assured, including appointment criteria, and the appraisal, improvement and reward of teaching effectiveness.
35 On the final day of the review visit, the review team considers its overall conclusions. It formulates its judgements, identifies features of good practice, and agrees its recommendations. The assistant director joins the team for this final part of the process.
36 There is no oral report to the institution at the end of the visit, but a letter is sent to the head of the institution within two weeks, outlining the main findings of the review and the likely recommendations in the draft report.
The selection of thematic trails
37 The review team's selection of thematic trails is made at the briefing visit and is communicated formally to the institution at the end of that visit. The trails are concerned with testing how well institutional processes work and how effective they are in practice, at local level and across the institution as a whole. They enable the team to gather information in relation to the institutional processes on which it is required to report (see Annex H). In undertaking its trails, the team may gather information at the level of individual disciplines, programmes, and/or academic departments.
38 In making its selection of trails, the review team takes account of:
- indications in the SED or other documentation of potential strengths or possible weaknesses in institutional quality assurance arrangements, which might be best explored by the team through testing how those arrangements operate at local level, or across the institution as a whole;
- lack of clarity in the SED about particular aspects of institutional quality assurance arrangements, which might be better illustrated for the team through examination of how those aspects operate at local level, or across the institution as a whole;
- the desirability of selecting a range of trails that, when taken together, provide a good representative sample of procedures in operation at local level and across the institution as a whole.
39 The institution is not required to produce additional SEDs to support the trails.
40 In the event that any emerging or unforeseen areas of concern come to light during the review visit, the review team may diverge from its previously identified areas of discussion to address those areas.
Use of reference points
41 When considering the institution's management of quality and standards, the review team draws upon a range of external reference points, including the FHEQ, Subject benchmark statements, the Code of practice and the Credit and Qualification Framework for Wales. In so doing, it is not seeking evidence of compliance, but rather for evidence that the institution has considered the purpose of the reference points, has reflected on its own practices in the relevant areas, and has taken, or is taking, any necessary steps to ensure that appropriate changes are being introduced:
- in respect of the FHEQ, the team considers the institution's procedures for relating its awards to the appropriate level of the FHEQ, where relevant using the thematic trails to gain further insight;
- in respect of the Code of practice, the team does not seek information about adherence on a precept-by-precept basis. It expects to see a statement in the SED about how the intentions of the precepts have been addressed, and to discuss during its visits any key changes that the institution has made to its practices and any areas that have caused particular difficulty, where relevant using the thematic trails to gain further insight. The team looks in particular at how the institution has addressed the additional Guidelines for providers of higher education programmes in Wales for effective practice in assessing and examining in a language other than the language of tuition, a supplement to the section of the Code of practice relating to the assessment of students;
- in respect of Subject benchmark statements, the team enquires into the way in which the statements have been taken into account when establishing and/or reviewing programmes and awards, as illustrated through programme specifications. It may request evidence of practice during the thematic trails. The Agency views the statements as authoritative reference points, but not as definitive regulatory criteria for individual programmes or awards.Judgements and reports
Judgements and reports
42 The review results in a report published by the Agency. The report sets out the review team's judgement on:
- the confidence that can reasonably be placed in the soundness of the institution's present and likely future management of the quality of its programmes and of the academic standards of its awards.
- The judgement is based on a number of factors, including the extent and degree to which the team concludes that quality and standards are managed successfully, with reference to the institution's individual situation, context and mission, as well as to external reference points.
43 The judgement provides one of three expressions of confidence - 'confidence', 'limited confidence' or 'no confidence' - the detailed criteria for which are set out in Annex G. The statement of confidence is, in essence, a judgement of probability: it cannot be unconditional. In general terms, where the review team judges that the institution is managing quality and standards soundly and effectively and that its future capacity for maintaining quality and standards appears good, confidence is expressed. Where the team has doubts, either about the current assurance of quality and standards, or about the institution's capacity to maintain quality and standards in the future, it expresses limited confidence. Very occasionally, a team may make a judgement of no confidence in an institution. The team is required to indicate clearly the areas of concern that have given rise to any limitation of confidence and the reasons for its judgement.
44 The report also sets out the review team's judgement on:
- the reliance that can reasonably be placed on the accuracy, integrity, completeness and frankness of the information that the institution publishes about the quality of its programmes and the standards of its awards.
This judgement contributes to the confidence judgement described in paragraph 42.
45 There are no separate judgements on the thematic trails, or on individual disciplines, programmes or academic departments.
46 In making its two judgements, the review team gives particular attention to the Agency's expectations in two key areas. The first expectation is that the institution is making strong and scrupulous use of independent external examiners in summative assessment procedures. The second is that a similar use is made of independent external persons in the internal periodic review of disciplines or programmes. The team is unable to make a judgement of confidence in an institution if either of these elements is seriously deficient.
47 The two judgements are accompanied by recommendations for consideration by the institution, categorised in order of priority:
- 'essential' recommendations refer to important matters that the review team believes are currently putting quality and/or standards at risk and which require urgent corrective action;
- 'advisable' recommendations refer to matters that the team believes have the potential to put quality and/or standards at risk and require preventive, or less urgent, corrective action;
- 'desirable' recommendations refer to matters that the team believes have the potential to enhance quality and/or further secure standards.
48 The report provides comment on other matters, including the characteristics, strengths and limitations of the institution's internal quality assurance arrangements. It highlights features of good practice and indicates any area in which the team considers that an action plan should be produced and implemented by the institution.
49 The draft report is prepared and submitted to the institution as soon as possible following the review visit, normally within eight weeks. The assistant director coordinates its production and the format and contents follow a standard structure (see Annex H). The institution is asked to provide the Agency, within four weeks of receipt of the draft report, with corrections of errors of fact. The final report is prepared in light of the institution's response.
50 As the published report is intended to provide information of use to both lay and professional readers, it includes a summary intended primarily for the public, especially potential students, which is available separately from the rest of the report. The summary is submitted to the institution in draft form as part of the report text. In addition, the institution is invited to provide a brief statement to be published as an appendix to the report. The statement provides an opportunity for the institution to report on developments since the review visit, particularly in respect of actions taken or proposed to address the recommendations of the review team.
51 The normal expectation is that the report is published within 20 weeks of the review visit.
Sign-off and follow-up
52 The review is completed when it is formally signed off. Where the report makes a statement of confidence, the review is signed off on report publication. A brief enquiry is made by the Agency through correspondence with the institution after one year on the way in which the institution has responded to the report.
53 Where the report makes a statement of limited confidence, the report is published, but there is a programme of follow-up action. The Agency consults with HEFCW and requires, within three months of the report's publication, an action plan from the institution and, subsequently, a progress report on how the action plan has been implemented. The review is not formally signed off until the Agency is satisfied that the plan has been implemented successfully, with a maximum time limit of 18 months. If at that point concerns remain about the effectiveness of the remedial action, the Agency conducts a further visit.
54 Where the report makes a statement of no confidence, the report is published, but a further review of the institution is required by HEFCW. The review is initiated no later than 12 months, and normally no earlier than six months, following publication of the report. The nature, scope and timing of the review is determined in accordance with the principle of proportionality, and through a dialogue between the institution, the Agency and HEFCW.
55 A summary of the relationship between the review team's judgements and recommendations, and the follow-up action required, is provided in Annex G.
56 Three years after the review, the institution is required to submit a report to the Agency commenting on its progress since the review and outlining its intentions in respect of managing quality and standards over the three years until the next review. The Agency makes a short visit to the institution to discuss the progress report and provides feedback to the institution on its perceptions of the progress that has been made and of any strengths and weaknesses in the institution's current and future plans (see Annex I).
Review administration and institutional contacts
57 Responsibility for the coordination of the review rests with the assistant director, but the review team makes the judgements and recommendations resulting from the review. However, it is the responsibility of the assistant director to test that the team's findings are supported by adequate and identifiable evidence, and that the review report provides information in a succinct and readily accessible form. To this end the Agency retains editorial responsibility for the final text of the report.
58 The Agency endeavours to protect the quality of the review process through the adoption of explicit operational principles and process standards (see Annex J) and quality assurance mechanisms. The latter include the opportunity for participants in the process, including students, to provide structured feedback on their experiences.
59 The Agency operates a institutional liaison scheme through which each institution is invited to nominate a correspondent to liaise with designated staff of the Agency on a continuing basis. The liaison scheme is separate from the process of review management and is conducted by a different member of the Agency's staff.
Complaints and representations
60 Complaints about the conduct of the review and representations against the judgements made by the review team are considered by the Agency in accordance with the formal procedures published on its web site.
Top