Handbook for enhancement-led institutional review: Scotland

QAA 033 04/2003

Preface

The publication of this Handbook marks an important stage in the development of our new approach to quality in Scottish Higher Education. Over the period of 2001-03, the Scottish Higher Education Funding Council, Universities Scotland, the Quality Assurance Agency for Higher Education, and representatives of the student body have worked closely together on the development of this new approach through our joint Quality Working Group.

The new strategy has five main elements:

a comprehensive programme of subject reviews, to be run by institutions themselves;
institution-level review, which is the subject of this Handbook, and which will involve all Scottish HE institutions over a four-year cycle;
improved forms of public information about quality, based on addressing the different needs of a range of stakeholders including students and employers;
a greater voice for student representatives in institutional quality systems, supported by a new national development service;
a national programme of enhancement themes, aimed at developing and sharing good practice in learning and teaching in higher education.

We believe that this collaborative approach to quality is unique in many respects - in its balance between quality assurance and enhancement; in the emphasis which it places on the student experience; in its focus on learning and not solely on teaching; and (perhaps most importantly) in the spirit of cooperation and partnership which has underpinned all these developments. We are confident that this new approach places Scotland at the forefront of international good practice in this area.

The new approach will be fully implemented from 2003-04. This will raise a new set of challenges. Not least, we need to ensure that our new approach is perceived by staff and students to be relevant to their needs, and that it supports continuous quality enhancement of learning and teaching in the Scottish HE sector. The Handbook for enhancement-led institutional review: Scotland provides strong evidence of our commitment to these objectives, and we commend it to you.

April 2003

Section 1 Background and context

1 The process of enhancement-led institutional review has been designed by the Quality Assurance Agency for Higher Education (the Agency) in collaboration and consultation with Universities Scotland (US) and its member higher education institutions, the student bodies in Scotland, and the Scottish Higher Education Funding Council (SHEFC). It has been designed as an integral element of the new approach to the enhancement of quality and maintenance of standards in Scottish higher education.

2 Over the past decade the higher education sector in Scotland has participated in a range of initiatives in the external assurance of quality and standards. These initiatives have included:

a complete round of teaching quality assessments at the subject level;
75 academic reviews at the subject level;
a complete round of audits of institutional management of quality;
13 second round, continuation audits.

The outcomes of these activities have demonstrated that, in general, Scottish higher education institutions had in place effective quality management systems relating to the experience of students and the standards of their awards, and that the subject provision experienced by students was highly satisfactory or better.

3 Building on these foundations, national and institutional activities in relation to managing quality and standards have been changing to focus more explicitly on the enhancement of the learning experience of students. As part of this process, in 2000-01, SHEFC conducted an extensive consultation on the characteristics of a high quality higher education sector (HEC07/00). This debate involved significant contributions from individual higher education institutions as well as from the Universities Scotland Learning & Teaching Committee and the SHEFC Learning and Teaching Committee and student bodies. The debate concluded (SHEFC HE11/2001) that the key characteristics of a high quality sector are:

a sector which is flexible, accessible, and responsive to the needs of learners, the economy and society;
a sector which encourages and stimulates learners to participate in higher education and to achieve their full potential;
a sector where learning and teaching promotes the employability of students;
a sector where learning and teaching is highly regarded and appropriately resourced;
a sector where there is a culture of continuous enhancement of quality, which is informed by and contributes to international developments.

4 This shared thinking on the key aspects of a high quality sector informed the subsequent debate on the design of the new national strategy in Scotland for the continuing assurance and enhancement of quality and standards in higher education. In this debate, a strong consensus emerged among the student bodies, the higher education institutions, US, SHEFC, and other key stakeholders that:

institutions in Scotland manifestly take a responsible approach to the maintenance of quality and standards;
ownership of quality and standards issues rests with the institutions and not with SHEFC or the Agency;
institutions are committed to the principle of continuous quality enhancement;
students should have a major involvement in internal and external quality processes;
students and other stakeholders should have access to relevant public information about the nature and quality of provision.

5 This shared understanding, together with the agreed statement of the key characteristics of a high quality higher education sector, form the foundations on which the new approach has been constructed.

The elements of the new enhancement model

6 The new enhancement model consists of the following five inter-related elements:

A comprehensive framework of internal review at the subject level within the higher education institutions. The nature of internal reviews will be decided by individual institutions but will share certain agreed features including the use of trained reviewers and also externality within review teams. The SHEFC Guidelines on internal review at the subject level are included in this Handbook as Annex 5.

An agreed set of public information provided by the institutions.
The effective involvement of students in quality management through a variety of mechanisms including:

a the involvement of student members in review teams within the new institutional review process;

b the systematic representation of students at all levels within institutions;

c the effective training and support for student representatives through both internal mechanisms, existing external structures and through a new national Student Development Service; and,

d better information on the student experience through national surveys both of the student experience within institutions and also longitudinal surveys of student and graduate cohorts.

Quality enhancement engagements. These take the form of a structured programme of engagements each year which will involve the sector in a series of developmental activities on themes selected by the sector. These themes may be drawn from the outcomes of internal and external reviews and the outcomes of these engagements are likely to impact on the reflections of institutions as they consider their own quality enhancement strategies.
The institutional review process. This is an enhancement-led peer review process which, while providing information on the security of the institution's management of quality and standards, is focused on the institution's strategic management of quality enhancement.

7 It is important to note that all these elements are closely inter-related. For example, the quality enhancement engagements will include areas highlighted through internal reviews at subject level and the institutional review processes. Both review processes will use the enhancement engagements as one set of reference points. The effective involvement of students in the internal management of quality will play an important part in both review processes. The review processes will both contribute to, and underpin the authenticity of the information provided for stakeholders.

The required characteristics of institutional review within the new model

8 In order to meet the agreed requirements of SHEFC, US and the student bodies, the design of the institutional review methodology has embraced a focus on the strategic management of enhancement; a focus on the effectiveness of student learning; the use of a range of reference points including the Scottish Credit and Qualifications Framework (SCQF), the Agency's Code of practice and subject benchmark information; appropriate reference to employer and international perspectives; the inclusion of a student (or representative of students) within the audit team; and, a published report which includes commentary on:

the ability of the institution's internal review systems to monitor and maintain quality and standards at the level of the programme or award. This commentary leads to a judgement on the level of confidence which may be placed in the institution's management of quality and standards;
the institution's arrangements for ensuring that the information it publishes about the quality of its provision is complete, accurate and fair;
the effectiveness of the institution's approach to promoting an effective learning experience for students;
the combined effect of the institution's policies and practices for ensuring improvement in the quality of teaching and learning;
the effectiveness of the institution's implementation of its strategy for quality enhancement.

9 In line with the above requirements, it is important to emphasise five important themes that have underpinned the development of the new approach to review.

Enhancement includes assurance

10 The overarching theme of institutional review is the strategic management of standards and the enhancement of the quality of the student learning experience. Section 2 of this Handbook outlines how enhancement is interpreted in the review context. However, it is important to emphasise at the outset that a key element of an effective enhancement strategy involves knowing where one is starting from, ie how does the institution assure itself that standards and quality are being appropriately maintained (see below, paragraph 30). This can then be linked to the related element of the management of enhancement - improving the effectiveness of student learning, seeking to learn from current activities, reference points and good practice, and to make the most effective use of resources to support learning.

Looking forward

11 The focus within the review process will be on how an institution learns from the past in order to inform the future.

Enhancement and risk

12 Enhancement is the result of change which may include innovation. Change and innovation will frequently involve risk. Institutions are required to manage this risk in a way that provides reasonable safeguards for current students. The review process will recognise and support effective risk management and adopt a supportive and not punitive role in this context.

Supporting diversity

13 The review process will support the rich diversity of higher education institutions in Scotland. While there are commonalities of purpose, each higher education institution in Scotland has its own unique mission and will seek to meet the needs of its own particular students in its own particular ways. The enhancement strategies of individual institutions will therefore have their own particular characteristics and the review process will consequently engage with the enhancement of the particular learning experiences of students in the context of the particular institution.

The UK and international context

14 Institutions in Scotland operate and compete in a global environment. The review process will support institutions in this context. Key outcomes from the review process will be directly comparable with outcomes from related processes elsewhere in the UK. Some of the key reference points for quality and standards used by institutions and the review process will be common across the UK. In general, institutional enhancement strategies, the review process and related elements in the overall enhancement model, will draw on good practice, not only across the UK, but internationally. In addition, the review process and its outcomes will be used proactively to promote the high standing of Scottish higher education internationally.

15 In the light of the above factors, the new review process is referred to as enhancement-led institutional review (ELIR).

Section 2 Interpreting the enhancement focus within enhancement-led institutional review

16 For the purposes of ELIR, enhancement is defined as taking deliberate steps to bring about continuous improvement in the effectiveness of the learning experience of students. The deliberate steps at an institutional level will be strategic and will manage enhancement in a planned way. Enhancement does not necessarily imply the application of additional resources. Enhancement strategies will seek to optimise the deployment of resources in the development of effective student learning. The commonly used phrase, continuous improvement, refers to the ongoing nature of enhancement strategies, ie the notion that no matter where one is starting from, it is always possible to seek improvement. In operational terms, the process cannot literally be continuous, but will involve a cycle of planned phases including reflection, planning, implementation and evaluation. The emphasis on the effectiveness of the student learning experience reflects one of the important characteristics required of ELIR referred to above (paragraph 8). This is discussed more fully below.

Taking deliberate steps to bring about continuous improvement

17 In order to take deliberate steps, an institution (and its constituent departments, faculties, schools etc) will ask itself:

Where are we now? How effective is the current learning experience of our students?
Where do we want to be in the future? What are the patterns and mechanisms of supporting learning which the institution wishes to develop in order to enhance the learning experience of its students?
How are we going to get there? How are we as an institution going strategically to manage the processes of enhancement that will allow us to move towards meeting our aspirations?

18 In addressing these questions the institution will make use of a wide variety of reference points. Some of these reference points, such as the Agency's quality framework, will be common to all institutions while others may be limited to particular institutions or groups of institutions. Some reference points will be largely determined externally (eg professional body guidelines) while others will be internally defined (eg institutional strategic plans). Some reference points will be national while others will be international (eg from international learned societies). It is likely, therefore, that while there will be very significant elements of commonality, the particular combination of reference points used by individual institutions may vary widely. There is also likely to be significant variation in the way reference points are used both within and between institutions.

19 Some of the most common reference points are likely to include:

base-line reference points that tell the institution about the current learning experience of students - eg outcomes from internal reviews at the subject level; reports from Professional and Statutory Bodies (PSBs); external examiner reports; information from student feedback mechanisms; and, information from employer feedback mechanisms;
the Agency's quality framework - SCQF; subject benchmark information; and the Code of practice;
guidelines and reports from PSBs;
experience gained through participation in Scottish quality enhancement engagements (see above, paragraph 6);
reports and publications from the Learning and Teaching Support Network (LTSN), the Institute for Learning and Teaching in Higher Education (ILTHE), and other similar bodies;
publications and guidelines from learned societies and subject associations - both UK and overseas;
reports from employers - employment sector specific and general, national and international;
relevant government and EC publications;
guidelines, frameworks and legislation for promoting equal opportunities;
institutional policies and strategies;
international reports and guidelines on good practice.

20 One of the reference points used by institutions (and its component parts) will be the vision agreed by the sector and other stakeholders in Scotland of the meaning of a high quality higher education sector referred to above (paragraph 3), namely:

a sector which is flexible, accessible and responsive to the needs of learners, the economy and society;
a sector which encourages and stimulates learners to participate in higher education and to achieve their full potential;
a sector where learning and teaching promotes the employability of students;
a sector where learning and teaching are highly regarded and appropriately resourced;
a sector where there is a culture of continuous enhancement of quality, which is informed by and contributes to international developments.

Aspects of the learning experience of students

21 The learning experience of students is an extremely broad concept. The focus within ELIR will be on those aspects of the student learning experience for which the institution bears a direct responsibility. It is recognised that there will be significant variation in both the actual and desired learning experiences of students both within and between institutions and between students in different types of programme eg taught and research.

22 The aspects of the student learning experience to which ELIR will relate will normally include the inter-related elements of:

the curriculum - structure, aims, intended learning outcomes and assessment;
promoting and supporting effective learning.

23 In relation to the curriculum, enhancement refers to the processes used to sustain and develop the currency of the curriculum outcomes in the light of developments in knowledge and understanding, professional practice, employer and other stakeholder expectations and other appropriate reference points. In addition to subject-related outcomes, this will also relate to more general outcomes expected from higher education related, for example, to employability and life-long learning. Enhancement in this context also refers to the improvements in assessment policy and practice and the alignment of appropriate assessment tools and approaches with learning strategies and intended learning outcomes.

24 In relation to promoting and supporting effective teaching and learning, enhancement embraces all the structures and processes used by an institution to support effective student learning in all the contexts in which learning takes place. This would include, for example, support through lectures, tutorials, research supervision work-based learning, independent learning, e-learning and distance learning. It would also relate to the role of 'support' services in supporting effective learning. For example, the way in which institutional careers services impact on the effective learning experience of students through influencing curriculum selection and the development of employment related skills. It would also relate to the support of effective learning through counselling and other support services of the institution and to the provision of libraries and information services and the institution's IT infrastructure. In general, supporting effective learning relates to the way in which an institution identifies and addresses the learning support needs of its particular students.

25 A key aspect influencing the learning experience of students is the role played by students themselves. Students are active partners with shared responsibilities for their own learning and achievement. Indeed, one of the defining characteristics of higher education is the extent to which it relies on this active participation in, and student ownership of, the learning process. The ELIR process will wish to engage with how this partnership is defined, managed and reflected on by institutions and their students. It will also wish to engage with how, in general, students both exercise their responsibilities for effective learning as well as how they contribute to the monitoring and enhancement of their learning experience.

Enhancing quality and maintaining standards

26 As discussed above, changes in the curriculum will reflect developments in knowledge, professional practice, and general employer and stakeholder expectations. Within this dynamic context, institutions will maintain the standards of their awards through their processes for defining awards, validating programmes, and assessing learning outcomes achieved. The ELIR process will engage with the effectiveness of these institutional processes for maintaining standards of awards taking account of appropriate reference points.

27 It is important not to confuse the maintenance of standards of awards with the standards of outcomes achieved by students. Improving the student learning experience will, potentially, improve the standards of outcomes achieved by students and result in an increase in the number of students progressing and achieving awards or achieving higher grades of awards.

Enhancement, complexity and risk

28 It follows from the above discussions, that enhancement is a complex process involving significant elements of risk and uncertainty. It is also the case that the strategic management of enhancement in the sense used by ELIR is a relatively new approach for many institutions. The general expectations within ELIR will therefore be of a growing maturity of approach over the first cycle of its operation. Indeed, it is hoped that one of the outcomes of the operation of ELIR will be to support institutions in this new context. It is recognised that both the ELIR process and the development and operation of institutional strategies for enhancement will mature during the cycle. This will be recognised, particularly in the context of institutions involved in the early years of the cycle (see, for example, paragraph 36 below). The Agency therefore recognises the challenge facing institutions involved in reviews early in the first ELIR cycle, and will provide appropriate guidance and support to institutions and briefing to review teams.

29 It is also important in this context to emphasise that enhancement is the result of change, often involving innovation. Some forms of innovation will be relatively straightforward and risk free. However, other forms of innovation are likely to involve some element of risk. Institutions manage this risk in a way that will provide reasonable safeguards for current students. The ELIR process and reporting will recognise and support effective risk management in relation to change and innovation and will adopt a supportive and not punitive role in this context. It is inevitable that some changes will be more successful than others and often more can be learned in the long run from analysing the reasons for less successful outcomes.

Enhancement and the assurance of quality and standards

30 It follows from the above discussions that the base line for enhancement is systematic awareness of the current learning experience of students, the standards of their achievements and the standards of the institution's awards. The outcomes from the institution's internal reviews at the subject level, utilising appropriate sets of reference points, will be fundamentally important in providing the institution with this information. Through ELIR's engagement with the efficacy of these and related processes, it will be able to provide a well-founded view on how effectively the institution assures itself that standards and quality are at least being maintained in line with national expectations. Within the ELIR approach, this base-line assurance represents only the initial phase of engaging with the enhancement processes and will not form the main focus of the ELIR activities or its reporting.

Section 3 The enhancement-led institutional review process

Scope of ELIR

31 The scope of ELIR includes the mechanisms to support all award bearing provision within the institution. An award in this context includes the award of credit. ELIR therefore relates to the learning experience of all students on credit bearing provision: undergraduate and post-graduate students; taught and research students; full-time and part-time students; initial higher education students and those involved in credit-bearing continuing professional development; campus-based, work-based, distance-learning students (including those supported wholly or in part using information technology); students entering HEIs from school, through wider access initiatives from FE colleges and from the work-place; and students who are publicly funded and students who are privately funded.

32 ELIR will include within its processes provision leading to the award of the institution (including credit) but delivered elsewhere, such as through a college of further education or employer organisation. In cases where the delivering institution is itself a Scottish higher education institution, the delivering institution will receive an ELIR review in its own right. Responsibility for the standards of awards offered through such arrangements, however, remains unambiguously with the awarding institution. Where provision is made in conjunction with an overseas partner, ELIR will relate to the arrangements in place in the Scottish institution for managing the quality of the learning experience and standards of the awards. The security of the Scottish institution's arrangements as they operate in the location of the overseas partner will continue to be audited through separate processes. The outcomes from any such overseas audit reports should form useful reference points.

The ELIR Process

33 One of the hallmarks of ELIR is that it is designed to support institutions in reflecting on the past in order to improve the future. The underlying ELIR inquiry is therefore to ask the institution: 'how do you learn from your current and past activities in order to improve the learning experience of your students?'

In this context the ELIR process will seek to explore with the institution:
In what ways does the institution seek to understand the quality of the current learning experience of its students?
What reference points does the institution use to manage its quality enhancement strategy and how does it use these reference points?
How does the institution ensure that its quality enhancement strategy continues to be informed by experience?

34 In seeking to explore these matters, the ELIR process will seek to:

be open and transparent;
support the sector, individual institutions and their staff and students in enhancing effective learning;
be forward looking;
relate to the wider model of quality enhancement in Scotland;
be conducted in a collaborative spirit, avoiding surprises.

35 The ELIR process consists of four integrated elements: an annual discussion with each institution; once in the cycle, the submission of a Reflective Analysis (RA); a review visit following the submission of the RA; and, each year, sector-wide feedback on the learning points from ELIR activity across the sector.

Annual ELIR discussion

36 In order to facilitate the review process, an annual meeting will be held between a member of staff of the Agency and a small group which is likely to comprise senior colleagues of the institution and a representative of the student body. The meeting will provide an important opportunity for the sharing of information between the Agency and the institution. Annual meetings will not result in any judgements or any public reporting. It is anticipated that, particularly in the early years of the cycle, the annual meeting will play an important role in supporting institutions in their engagement with the ELIR process.

37 A particular focus of the annual meetings will be discussion of the institution's approach to internal review at the subject level and what the institution is learning from the outcomes of its internal review in that year. There may be other activities which the institution would wish to share in the context of these annual discussions eg the outcomes of visits by professional or statutory bodies; significant developments in the institution's approach to quality enhancement; the institution's experience of quality enhancement engagements.

38 The recorded outcome of the annual discussion will be an institutional profile which will be agreed each year with the institution. The profile will contain key descriptive information about the institution and its approach to internal review at the subject level together with a summary of outcomes of review activity. There will also be a section of the profile in which the institution can provide any information it wishes in relation to developments in that year. The profile will be updated and agreed each year with the institution. It will not be published by the Agency but will remain confidential to the Agency and the ELIR team appointed to work with the institution. The profile, along with the RA will form the basis for briefing the ELIR team on the institution's approach to internal reviews at the subject level and the use of the outcomes of these, and related processes.

39 The annual discussion should not require the preparation of any additional material by the institution. To support the annual discussion, institutions will be requested to submit each year:

copies of the reports of internal reviews at the subject level for the previous 12 months (with any sections confidential to the institution removed, for example, where an institution includes financial matters in such reviews and reports). By submitting these reports annually, the institution will not require to re-submit them in association with their ELIR visit;
a copy of the definitive internal document(s) describing the approach of the institution to review at subject level;
a copy of any additional existing documents relating to changes in the institution's approach to quality enhancement that the institution would wish to share.

Preparation and submission of a Reflective Analysis

40 The RA will be submitted once in the cycle, in advance of the ELIR visit. The RA will discuss the institution's reflections on its management of quality enhancement through addressing the matters identified in paragraph 33 above and discussed more fully in Section 2. Essentially, the RA will be a reflection by the institution on what processes and reference points it is using to manage enhancement (including the reference points shared by all the institutions in Scotland); what the use of these processes and reference points indicates; and how the institution is using the outcomes from these processes to further enhance the effectiveness of the learning experience of its students. The most important aspect of the RA is the reflection on experience. What has worked well? What has not worked so well? How can we spread good practice? The RA will indicate the sources of evidence on which the institution has based its analysis.

41 The RA will also include an annex including one or more case studies of effective enhancement of learning which the institution views as being of particular significance in illustrating its strategic management of enhancement and which it is willing to share more widely. This will be valuable material for the review team illustrating the operation of the institution's enhancement strategy. The case study will outline the good practice itself (which might be generic or subject-related) together with the ways in which the example illustrates the institution's approach to supporting enhancement. These case studies will inform both discussion during the ELIR visit and, potentially, future Scotland-wide quality enhancement engagements and other activities (see paragraph 51 below).

42 The expectation is that the process of compiling the RA will be inclusive of all appropriate interests within the institution. In particular, institutions will be asked to indicate how student views have been incorporated within the process.

43 Annex 1 provides a suggested outline for the RA.

The ELIR visit and report

44 Aspects of the potential agenda for the ELIR visit are shaped initially during the annual discussion between the Agency and the institution as reflected in the institutional profile (see paragraph 36 above). This is described as influencing the 'potential agenda' because these discussions are held with officers of the Agency and the peer ELIR team must ultimately have the freedom to determine its own agenda.

45 This is followed by a part 1 visit to the institution by the ELIR team, normally lasting two days. The purpose of the part 1 visit is to:

ensure that the ELIR team has a sound understanding of the institution and its approach to the strategic management of enhancement as laid out in the RA;
identify the ELIR themes which are of particular interest to the team;
share with the institution the ELIR themes;
identify the most appropriate programme of activities for part 2 of the visit;
identify additional evidence, referred to in the RA, to which the team would wish access.

46 In order to fulfill these purposes, the part 1 visit will normally start with up to a half day of meetings, discussions and/or presentations with agendas and personnel determined by the institution but discussed in advance with the Agency. The objective of this half day will be to facilitate the understanding of the ELIR team as to the nature of the institution and its strategies for managing quality enhancement. This will be followed by a series of meetings to follow agendas developed by the team in the light of the RA and the initial meetings/presentations, normally, over a period of 1.5 days. These meetings will include a meeting with senior staff, a meeting with student representatives, and meeting(s) with a sample of staff and students associated with the institution's internal review activity at the subject level. The part 1 visit will conclude with a meeting between the ELIR team and the senior member of institutional staff responsible for the management of quality to agree the programme and themes for part 2 of the visit.

47 The length of the part 2 visit will vary between three and five days, depending largely on the size and complexity of the institution. The part 2 visit will follow the programme agreed at the conclusion of the part 1 visit. The team will wish to meet with a broad range of staff, students and, potentially, associated employers. The focus of these discussions will be on what the institution in different ways and at different levels is learning through the operation of its enhancement strategy including its use of reference points; how it is maintaining quality and standards; how it understands the effectiveness of the current learning experience of students; how good practice is identified and shared; how less effective practice is identified and improved; how staff are supported in their exploration of new strategies for developing effective learning etc. Discussions will also refer to the ways in which the case study material presented in the RA illustrates the embedding of institutional mechanisms for promoting and supporting effective learning.

48 A letter will be sent to the institution following the conclusion of the part 2 visit which will summarise the provisional key themes of the ELIR teams report.

49 A draft of the report will be sent to the institution for comment within an agreed period of the conclusion of part 2 of the visit. The institution will be asked to provide comment following which the Agency will finalise and publish the final reports (see also Section 4).

50 Annex 2 provides an outline of the main report sections.

Sector-wide feedback and workshops

51 The Agency will conduct an annual analysis of the main themes emerging from the ELIR activities across the sector. This analysis, together with material drawn from the case studies of good practice in the strategic management of enhancement will form the basis of publications, workshops and other activity. Publications associated with these activities may be distributed nationally and internationally. These activities will also integrate with the other enhancement engagements in Scotland.

Next > Section 4 - The review report