Enhancement-led institutional review handbook: Scotland

(Second edition)

QAA 247 06/08

PDF version

Contents

• Preface to the second edition
• Section 1 - Interpreting the enhancement focus within Enhancement-Led Institutional Review (ELIR)
• Section 2 - The ELIR approach
• Section 3 - The ELIR reports
• Section 4 - The ELIR team
• Section 5 - Monitoring and evaluation

Annexes

• Annex 1 - Main ELIR report structure
• Annex 2 - Guidance for institutions on a reflective analysis (RA) structure
• Annex 3 - Criteria for the selection of reviewers
• Annex 4 - SFC Guidance to higher education institutions on quality issues
• Annex 5 - European Association for Quality Assuraance in Higher Education (ENQA) 2007
• Annex 6 - QAA’s values and standards

---

Preface to the second edition

Introduction

1 Enhancement-led institutional review (ELIR) is one component of the Quality Enhancement Framework (QEF), a radical approach to quality assurance and enhancement in higher education introduced in Scotland in 2003. The QEF emerged out of the experiences of more than a decade of audit and review. According to the final report of the external evaluation¹, the QEF was 'based on the pooling of expertise and knowledge of literatures on teaching, learning, change and quality from a wide range of sources, all shot through with a commitment to enhancing students' experiences as learners'. In other words, the QEF brought right to the fore the simple and powerful idea that the purpose of quality systems in higher education is to improve student experiences and, consequently, their learning. In the second edition, the ELIR Handbook remains firmly focused on this target.

2 In the first edition of the Handbook, it was stressed that this powerful new focus on the enhancement of the student learning experience was not at the expense of the assurance of quality and the standards of awards. Rather, it was emphasised that enhancement built on, reinforced and made more powerful the basic processes of assurance: enhancement included assurance. The evidence of the reviews carried out in the previous version of the ELIR method clearly supports the robustness of this approach, which is taken forward in the second edition of the Handbook.

3 The second edition integrates ELIR more fully with other elements of the QEF, which are also progressively becoming integrated more substantively into the fabric of Scottish higher education. The main elements of the QEF, in addition to ELIR, are:

• the Enhancement Themes
• institution-led quality review
• the engagement of students in quality management, including the support provided through the national independent development service, student participation in quality scotland (sparqs)
• the provision by institutions of an agreed set of public information.

4 The Enhancement Themes were a novel development within the QEF. They have now become well established under the direction of the Scottish Higher Education Enhancement Committee (SHEEC). The remit of SHEEC relates both to the strategic management of the rolling plan for the Enhancement Themes, and also to supporting the development of institutional quality cultures. The outcomes from the work of SHEEC now represent a valuable resource for the sector in the management of quality, drawing widely as they do on both national and international practice, evidence and experience. There was, and is, no expectation that the outcomes of the Enhancement Themes will lead to compliance with specific approaches. Rather, the outcomes now provide a rich repertoire of reference points for institutions and their students to draw on in developing their own policies and practice within the context of their own timeframes and priorities. The Enhancement Themes, therefore, now increasingly provide an important context for discussion within ELIR, a feature that will be reflected in ELIR reviewer training and continuing development (see below, paragraph 29). The Enhancement Themes are also increasingly referred to within institution-led quality reviews.

5 In general, ELIR and the wider QEF represented a radical departure from previous approaches. The individual elements of the QEF have matured over the first version of the ELIR method, and interrelationships strengthened. The preparation of the second edition of the ELIR handbook provides the opportunity to build further on this solid foundation.

Continuity and change: continuity

6 The second edition of the ELIR Handbook reflects a clear development in approach from the first edition, refining areas where the need for change has been demonstrated, and sharpening the focus on the enhancement of the student learning experience. The final report of the external evaluation of the QEF opens with the statement, 'the approach to quality that we review here is ambitious, distinctive and, so far, successful'. The revised approach to ELIR described in this second edition of the Handbook represents continuity, building on that success and developing from the experience.

7 The final external evaluation report identifies the key drivers of the success of the QEF.

'Distinctively, the quality enhancement framework which we review here, is a commitment to:

• Students and the continuing enhancement of their learning in higher education.
• Partnerships between agencies...higher education institutions...and other stakeholders, most distinctively seen in the active involvement of students and student bodies in the QEF.
• A theory of educational change that placed far more weight on consensual approaches than on the more coercive stances embedded in some quality assurance regimes. The approach emerged from serious discussion and thinking.
• A culture shift - away from top-down compliance-inducing processes to participative and critical supported self-evaluation; away from audit and towards improvement; away from ruffling the surface of higher education practices and towards permeating the system with practices compatible with the QEF; away from mechanistic models based solely on inputs and outcomes and towards more sensitive other forms of evidence of cultural change, while maintaining rigour and challenge.
• Reflexivity, in the sense of exposing QEF itself to evaluation from the very beginning. Evaluation was valued for the contribution it could make to the enhancement of the QEF itself, even as the QEF was working out what it would mean to have a higher education system committed to quality enhancement.'

Importantly, the final evaluation report also stresses that a key success factor is commitment to the long term, recognising that the development and nourishment of quality cultures does not happen overnight: the dynamic of real enhancement depends to a significant extent on operating within a relatively stable, predictable policy environment.

8 At the heart of the QEF to date has been the successful operation of professional partnerships with Universities Scotland, with the individual autonomous institutions, with the Scottish Funding Council and with students and their associated bodies, importantly NUS (Scotland) and sparqs. One vital piece of evidence on the importance of building further on the partnership model comes again from the final external evaluation report which indicates:

'Not surprisingly, we have found evidence of the persistence of behaviours redolent of the displaced quality assurance regime. We have also noticed, though, that when it comes to enhancement-led institutional review (ELIR), institutions are increasingly willing to lay out areas of imperfect practice and publicly consider ways in which they could improve on them in coming years. Perhaps the shift from the concealing behaviours associated with the previous quality assurance regimes operating in UK HE in the 1990s towards - and let it be clear that this is a direction of travel - disclosure of areas for improvement is the biggest cultural shift in thinking and the most distinctive feature of Scotland's fresh thinking about quality.'

9 In general, the evidence confirms that the founding principles of the enhancement-led approach remain fit for purpose and that the first edition of the ELIR Handbook provides a sound basis from which to develop. This evidence is drawn from a number of sources including: the outcomes of the external evaluation referred to above; the outcomes of QAA Scotland's own monitoring and evaluation of each element of ELIR; and the outcomes of the evaluation carried out by the Scottish Funding Council (SFC) during 2006-07 (involving extensive consultations across the sector).

10 The second edition of the ELIR Handbook thus takes forward five key themes from the first edition.

• Enhancement includes assurance. The overarching theme of ELIR continues to be the strategic management of academic standards and the enhancement of the quality of the student learning experience. Section 1 of this Handbook outlines how enhancement continues to be interpreted in the review context. However, it is important to emphasise at the outset that a key element of an effective enhancement strategy involves knowing where one is starting from, ie how does the institution assure itself that standards and quality are being appropriately maintained. This can then be linked to the related element of the management of enhancement; improving the effectiveness of student learning; seeking to learn from current activities, reference points and good practice; and to make the most effective use of resources to support engagement and high-quality learning. Assurance and enhancement are inextricably linked within the quality cultures of institutions.
• Looking forward. The focus within the ELIR process will continue to be on how an institution learns from the past in order to inform the future.
• Enhancement and risk. Enhancement is the result of change and innovation that will frequently involve risk. Institutions are expected to manage this risk in a way that provides reasonable safeguards for current students. The review process will continue to recognise and support effective risk management and adopt a supportive and not punitive role in this context.
• Supporting diversity. The review process will continue to support the rich diversity of higher education institutions in Scotland. While there are commonalities of purpose, each higher education institution in Scotland has its own unique mission and will seek to meet the needs of its own particular students in its own particular ways. The quality culture of each institution will therefore have its own individual characteristics, and the ELIR process will consequently engage with the enhancement of the learning experiences of students in the context of their own institution.
• The UK and international context. Institutions in Scotland operate and compete in a global environment. The ELIR process will support institutions in this context. Key outcomes from ELIR will be directly comparable with outcomes from related processes elsewhere in the UK. Some of the reference points for quality and standards used by institutions and ELIR will be common across the UK, for example the Academic Infrastructure and the relevant sections of the Standards and Guidelines for Quality Assurance in the European Higher Education Area. In general, institutional enhancement strategies, ELIR and related elements in the overall enhancement model will draw on good practice, not only across the UK, but internationally. In addition, ELIR and its outcomes will be used proactively to promote the high standing of Scottish higher education internationally.

Continuity and change: change

11 While this edition of the Handbook does not embody revolutionary change, it does nonetheless represent a significant evolution from the first edition, reflecting, to a large extent, refinements rather than fundamental change. In some areas this reflects changes in the external environment. Two of the key publications to which the second edition has responded are the Report of the Joint Quality Review Group (SFC 2007)² and the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ENQA 2007)³ adopted by European ministers as the definitive document on quality assurance within the Bologna process (see Annex 5). The development of the second edition has also been informed by the International Network of Quality Assurance Agencies (INQAAHE) Guidelines of Good Practice (INQAAHE 2007)⁴, Learning to improve: quality approaches for lifelong learning (Scottish Executive 2005)⁵, and the principles embodied in the Crerar Review (Scottish Government 2007)⁶. In addition to responding to these reports, as other elements of the QEF have matured new opportunities to take forward further interrelationships have developed. As indicated above, the outcomes from the Enhancement Themes are now providing a rich repertoire of international practice, evidence and experience for dialogue in the context of both ELIR and institution-led quality reviews. There is also the potential for the SHEEC project on Indicators of Enhancement being undertaken in 2007-08 to provide additional constructive reference points for use in relation to ELIR. The outcomes from this project may also have a useful role to play in the context of institution-led quality reviews which, in themselves, have an important interplay with ELIR. Sparqs is also proving to be an increasingly important focus for joint work with students, including the training of students who are reviewers.

The report of the Joint Quality Review Group (JQRG)

12 The second edition of the Handbook reflects the outcomes of the JQRG report for the higher education context. Many elements highlighted in the JQRG report were already well established within the ELIR philosophy and practice. The JQRG report reinforces the characteristics of a high-quality sector explicit in the first edition of the ELIR Handbook. This is retained with some modification (see below, paragraph 30). The JQRG report identifies three fundamental principles that 'should underpin every aspect of our quality systems, namely: high quality learning across all provision; student engagement; and, a culture of quality and continuous improvement'. These principles are embodied in this edition of the Handbook.

13 In general, the principle of quality culture has become the overarching principle underpinning not only ELIR but other elements within the QEF. The approach of ELIR from the outset has been designed to support and facilitate the development and nourishment of quality cultures in Scotland. Indeed, the final report from the external evaluators indicated early evidence of the emergence of the 'green shoots' of quality cultures. In preparing the second edition of the Handbook, we have been conscious of the need to support the continuing development of reflective institutions (and faculties, schools and departments) on which quality cultures depend. The processes around the development and use of the Reflective Analysis; the sharing of case-study material across the sector; the highlighting of the role of the Enhancement Themes; the annual discussions; and further development of the interactive approach to review visits, are all ways in which the second edition of the Handbook reflects this important, overarching principle.

14 The second edition of the Handbook also takes forward the key concepts of student engagement and high-quality learning. Student engagement is developed in two related contexts: engagement with quality processes, and students' engagement with their learning. Student engagement in the former sense has, from the outset, been central to ELIR and the QEF, and developments in this area were very prominent in the previous version of the ELIR method. Fundamental to this has been the role of student reviewers within ELIR teams, as well as effective student engagement with internal quality and representative structures and processes. This will continue. It is, however, increasingly important that ELIR should support student engagement in the latter sense: students' engagement with their learning. The second edition of the Handbook, therefore, contains more explicit reference to student engagement with their learning as outlined below.

15 Similarly, the JQRG emphasis on high-quality learning is taken forward within the concept of quality culture. High-quality learning in higher education results, in large part, from effectively engaging learners in the process of learning, and we have therefore taken this element forward jointly with consideration of student engagement. In the preparation of the second edition of the Handbook, the following aspects of institutional quality cultures have been particularly emphasised as contributing to effective student engagement and high-quality learning:

• the support of students as active participants, engaged in the creation of their knowledge and understanding, and taking responsibility for their own learning
• learners being engaged through effective guidance and support mechanisms in making informed choices in relation to their learning, curricula and future career and personal development paths
• learners being engaged in decision-making processes relating to the curricula, and learning and support strategies and provision
• learners being engaged in the provision of feedback, perspectives and insights.

These aspects have impacted, in particular, on the development of the Reflective Analysis; the training of reviewers; the ELIR visit; and the ELIR report. Indeed, the second edition of the Handbook indicates that the first substantive section of the ELIR report will address the effectiveness of student engagement.

16 In addition, high-quality learning is also defined in part by the achievement of appropriate outcomes; appropriate in relation to the academic standards of awards and to students' future employment and their participation in civic life, not only following course completion, but as a preparation for lifelong learning and career development. The second edition of the Handbook maintains a clear and explicit focus on the outcomes achieved and, in particular, on the academic standards of awards.

17 A key element, fundamental to the support of high-quality learning and effective student engagement, is the professionalism of staff responsible for designing and supporting the student learning experience. The second edition of the Handbook therefore identifies a more explicit focus on staff support and development in the context of institutional management of the student learning experience.

18 The second edition continues the focus, emphasised in the JQRG report, on the quality of the experience of all students irrespective of race, gender, sexuality, faith, age or disability. ELIR highlights the importance of the experience of all students, both on and off-campus, in an environment that supports and promotes equality and diversity and the fruitful interaction of all students, both domestic and international.

19 Other aspects of the JQRG report which have been taken forward explicitly in the second edition of the Handbook include:

• reporting: The JQRG report acknowledges the limited role that external review plays in learner choice, and that clarity on this should result in reports being designed to meet the needs of key audiences, the institutions, SFC, and public stakeholders. There will in future be two reports: the definitive report written largely for the institution (with little description and largely analytical) and only available electronically; and a short summary report written in a style appropriate for the lay person and available both in print and electronically. The institution's 'year-on' response to its ELIR report will be published electronically by QAA alongside the original report
• judgements: The JQRG report concluded that reports (in both the college and university sectors) should conclude with judgements 'summarised using a three point confidence rating scale'. This was the general approach adopted in the first edition of the ELIR Handbook and is continued. The wording of the overarching judgements will undergo a marginal change from the first edition, in part to maintain the same wording as the review reports relating to higher education institutions elsewhere in the UK, and in part reflecting feedback received during the previous version of the ELIR method
• collaborative activity including international dimensions: In general, all the matters raised in the JQRG report with regard to international matters have been responded to and are addressed more fully below. In addition, there is a more explicit focus within ELIR for dealing with domestic, as well as international, collaborative activity.

International dimensions

20 The second edition of the Handbook provides a focus on all students, including a focus on the experience of international students studying on-campus in Scotland, and the nature of the interaction of overseas with domestic students.

21 The higher education qualifications framework in Scotland (part of the SCQF), one of the important reference points used by institutions and ELIR in maintaining academic standards, has now been formally recognised within the Bologna process. In general, the ELIR method has been revised in line with international good practice in the management of quality in higher education. The 'Standards and Guidelines for Quality Assurance in the European Higher Education Area', central to the Bologna process, have been fully embraced (see Annex 5) as have the International Network of Quality Assurance Agencies in Higher Education (INQAAHE) 'Guidelines of Good Practice'.

22 The international dimension is extended by including for the first time, in each ELIR team, a reviewer drawn from outside the UK. By adding an international reviewer to all ELIR teams, the range of experience and expertise is extended, and both institutions and the ELIR process will benefit from this wider global perspective. The international reviewer will bring international perspectives on quality assurance and enhancement and will generally be in a position to draw the enhancement and assurance discussions in ELIR on a wider canvas. In addition to the direct benefits, international reviewers will facilitate dissemination abroad of Scottish higher education quality and standards. In addition, international reviewers may be in a position to contribute to wider elements within the QEF, in particular the Enhancement Themes. To get maximum benefit, it is important that international reviewers are trained, and drawn from institutions, organisations and countries appropriate to the diverse higher education institutions in Scotland. As detailed in section 4, institutions will therefore have an important role in nominating reviewers to the international reviewer pool.

ELIR coordinating reviewer

23 The role of the review secretary has been replaced by the new role of ELIR coordinating reviewer (ECR). Monitoring and evaluation during the previous version of the ELIR method highlighted the importance of the role played by the review secretaries. It has also been apparent that the potential contribution of these senior academic administrators to the process has been artificially constrained by the current definition of the role. In addition, the monitoring and evaluation of the report production processes, together with changes being made to the reports themselves, have highlighted the need for a different contribution to reports from the ELIR team in general, and from the review secretary in particular.

The Quality Enhancement Framework

24 A fundamental backdrop to the revision of ELIR has been the developments in the other elements of the QEF and their interrelationships with ELIR. The Enhancement Themes, which continue to mature and reflect significant developments, have an important relationship with ELIR. As indicated above, the Enhancement Themes now provide a rich repertoire of reference points for institutions to draw on in developing their own policies and practice. Institution-led quality reviews also have a vital role to play in relation to the ELIR process, and continue to form an important focus for the annual discussions. ELIR will continue to depend on active engagement with students and its continued effective delivery will depend to a very significant extent on close working with NUS (Scotland), sparqs and other student bodies.

Summary

25 The revised ELIR method has been prepared through wide consultative processes to practise what ELIR preaches: to learn from the evidence of the past to improve the future, informed by national and international benchmarks. The evidence is that, in general, the introduction of the QEF has been successful. The second edition of the ELIR Handbook is intended to build on that success and learn from its relative weaknesses. The changes foreshadowed above and detailed in subsequent sections are the result of shared reflection on the evidence and related material, and a commitment to maintaining ELIR and the QEF more generally as an effective framework for supporting the quality culture of Scottish higher education and its national and international reputation.

Section 1- Interpreting the enhancement focus within Enhancement-Led Institutional Review (ELIR)

Defining enhancement within ELIR

26 For the purposes of ELIR, enhancement is defined as taking deliberate steps to bring about improvement in the effectiveness of the learning experiences of students.

• The deliberate steps at an institutional level will be strategic, designed to manage enhancement in a planned way. Enhancement does not necessarily imply the application of additional resources. Enhancement strategies will seek to optimise the deployment of resources in the development of effective student learning.
• Improvement. The first edition of the Handbook employed the commonly used phrase continuous improvement, referring to the ongoing nature of enhancement strategies, ie the notion that no matter where one is starting from it is always possible to seek improvement. Although, in general terms, this remains the case, the experience of the previous ELIR method was that the emphasis on 'continuous' improvement could artificially provide too strong a focus on the ongoing, relatively marginal changes in day-to-day practice. While in a quality culture it is clearly important that this continues, an important focus of ELIR is on the strategic management of more step-change elements of enhancement and the associated cycles of planned phases including reflection, planning, implementation and evaluation.
• The emphasis on the effectiveness of the student learning experience reflects the focus on student engagement and high-quality learning referred to in the preface above and is articulated more fully below.

Taking deliberate steps to bring about continuous improvement

27 In order to take deliberate steps, an institution (and its constituent departments, faculties, schools etc) will ask itself:

• where are we now? For example: Who are our students? What are the characteristics and learning needs of our students? How effective is the current learning experience of our students? Are some groups of students more successful learners than others? Are some groups of students better prepared for post-graduation life than others? What evidence can we draw on? How robust is the evidence? What is the evidence telling us?
• where do we want to be in the future? For example: What are the patterns and mechanisms of supporting learning which the institution wishes to develop in order to support student engagement and high quality learning?
• how are we going to get there? For example: How are we as an institution going to manage strategically the processes of enhancement that will allow us to move towards meeting our aspirations?
• how will we know when we get there? For example: What monitoring and evaluation processes do we have in place? How will the outcomes be analysed? How, and to whom, will the outcomes be disseminated?

28 In addressing these questions, the institution will make use of a wide variety of reference points. Some of these reference points, such as the Academic Infrastructure or the UK Professional Standards Framework for teaching and supporting learning in higher education, will be common to all institutions, while others may be limited to particular institutions or groups of institutions. Some reference points will be determined largely externally (eg Professional, Statutory or Regulatory body guidelines) while others will be internally defined (eg institutional strategic plans, quality enhancement plans etc). Some reference points will be national while others will be international (eg from international learned societies, international groupings of institutions, Bologna guidelines etc). It is likely, therefore, that while there will be significant elements of commonality, the particular combination of reference points used by individual institutions may vary widely. There is also likely to be significant variation in the way reference points are used both within and between institutions.

29 Within this variation, and in common with previous practice, all Scottish higher education institutions are expected to adhere to the Academic Infrastructure, including the higher education qualifications framework within the SCQF and have regard to Part 1 of the Standards and Guidelines for Quality Assurance in the European Higher Education Area. Adherence to the Academic Infrastructure involves institutions being able to demonstrate that the elements of the Infrastructure are addressed through the design and operation of the various institutional policies and processes for securing academic standards and assuring quality, but it does not involve narrow compliance. The outcomes of the Enhancement Themes provide interesting and potentially valuable reference points. Institutions are expected to engage with the Themes and their outcomes in the manner appropriate to the institution and its strategic approach. There is no expectation that institutions should comply with particular outcomes or approaches. Nonetheless, over time, certain practices may become common across the sector and it will be entirely appropriate for an ELIR team to explore with an institution why it has chosen to adopt a particular approach. Equally, it will be emphasised, for example in ELIR training, that diversity of approach is to be supported.

30 One of the reference points used by institutions (and their component parts) will be the vision agreed by the sector and other stakeholders in Scotland of the meaning of a high quality higher education sector referred to above, namely:

• a sector that is flexible, accessible and responsive to the needs of learners, the economy and society
• a sector that encourages and stimulates learners to participate in higher education and to achieve their full potential
• a sector where learning and teaching promotes personal and intellectual development and⁷ the employability of students
• a sector where learning and teaching are highly regarded and appropriately resourced
• a sector where there is a culture of continuous enhancement of quality, which is informed by, and contributes to, international developments.

Aspects of the student learning experience

31 The student learning experience is an extremely broad concept. The focus within ELIR will be on those aspects of the student learning experience for which the institution bears a direct responsibility. It is recognised that between institutions, and indeed within institutions, there will be variations in what is considered the optimum framework to support student learning. It is not a 'one size fits all'. Different students in different institutions on different programmes may well have different needs.

32 The aspects of the student learning experience to which ELIR will relate will include the institution's arrangements for managing:

• the curricula - structure, aims, intended learning outcomes and assessment
• the promotion and support of effective student engagement and high-quality learning for all students, including the role of all staff with responsibilities in these areas.

33 In relation to the curricula, enhancement refers to the processes used to sustain and develop the currency of the curricula and its outcomes in the light of developments in knowledge and understanding, professional practice, employer and other stakeholder expectations and other appropriate reference points. In addition to subject-related outcomes, enhancement will also relate to more general outcomes expected from higher education, for example those associated with employability and preparations for continuing lifelong learning. Enhancement in this context also refers to the improvements in assessment policy and practice, and the alignment of appropriate assessment and learning and teaching strategies with intended learning outcomes.

34 In relation to promoting and supporting effective student engagement with their learning, enhancement embraces all the structures and processes used by an institution to support effective student learning in all the contexts in which learning takes place. This would include, for example, support through curricular design and pedagogical approach, lectures, tutorials, research supervision, work-based learning supervision, independent learning support, e-learning and distance learning. It would also relate to the role of 'support' services in contributing to the student learning experience; for example, the way in which institutions evaluate the impact of careers services on the effective learning experience of students through influencing curriculum selection and the development of employment related skills. It would also include the support of high-quality learning through counselling and other support services of the institution and to the provision of library and information services and the institution's information technology infrastructure. In general, supporting student engagement with their learning relates to the way in which an institution identifies, addresses and evaluates the learning and support needs of its particular students. These will include the support services identified by institutions in their own institution-led quality reviews (see SFC Guidance, Annex 4).

35 ELIR recognises the role played by students themselves in influencing the nature and extent of their engagement with their learning. Students are active partners with shared responsibilities for their own learning and achievement. Indeed, one of the defining characteristics of higher education is the extent to which it relies on this active participation in, and student ownership of, the learning process. ELIR will engage with how this partnership is defined, managed and reflected on by institutions and their students. It will also wish to engage with how, in general, students are supported in becoming effective higher education learners and how they both exercise their responsibilities for engaging with their learning and contribute to the monitoring and evaluation processes.

Enhancing quality and maintaining standards

36 As discussed above, changes in the curricula will reflect developments in knowledge, professional practice, and general employer and stakeholder expectations. Within this dynamic context, institutions will maintain the academic standards of their awards through their processes for defining awards, validating and reviewing programmes, and assessing learning outcomes achieved. The ELIR process will engage with the effectiveness of these institutional processes for maintaining the academic standards of awards, taking account of appropriate reference points.

37 It is important not to confuse the maintenance of academic standards of awards with the standards of the outcomes achieved by students. Improving the student learning experience will, potentially, improve the levels of outcomes achieved by students and result in an improvement in student performance, and so an increase in the number of students progressing and achieving awards or achieving higher grades of awards.

Enhancement and the assurance of quality and standards

38 It follows from the above that the baseline for enhancement is systematic awareness of the current learning experience of students, the outcomes achieved by students, and the standards of the institution's awards. The outcomes from the institution-led quality reviews, utilising appropriate sets of reference points, will be fundamentally important in providing the institution with this information. Through ELIR's engagement with the efficacy of these and related processes, it will be able to provide a well-founded view on how effectively the institution assures itself that academic standards and quality are at least being maintained in line with national expectations. Within the ELIR approach, this baseline assurance, although clearly important, represents only the initial phase of engaging with the enhancement processes and will not form the main focus of the ELIR activities or its reporting.

Enhancement, complexity and risk

39 It is important to emphasise that enhancement is the result of change, often involving innovation. Some forms of innovation will be relatively straightforward and risk-free. However, other forms of innovation are likely to involve some element of risk. Institutions manage this risk in a way that will provide reasonable safeguards for current students. The ELIR process and reporting will recognise and support effective risk management in relation to change and innovation, and will adopt a supportive and not punitive role in this context. It is inevitable that some changes will be more successful than others and often more can be learned in the medium to long run from analysing the reasons for less successful outcomes.

Summary

40 It is clear that the creation of effective and penetrating quality cultures poses significant challenges. The second edition of the ELIR Handbook continues to recognise these challenges. The general expectations embodied within the revised version of ELIR are of a growing maturity of approach, building further on the 'green shoots' identified in the previous version of the method. Indeed, it is hoped that one of the outcomes of the continued operation of ELIR will be the further support of institutions in the development of their individual quality cultures, and collectively in the quality culture of Scottish higher education.

Section 2 - The ELIR approach

Scope of ELIR

41 The scope of ELIR includes the mechanisms to support all credit-bearing provision within the institution. ELIR therefore relates to the learning experience of all students on credit-bearing provision: undergraduate and postgraduate students; taught and research students; full-time and part-time students, including those involved in credit-bearing continuing professional development; campus-based, work-based, distance-learning students (including those supported wholly or in part using information technology); students entering higher education institutions from school, through wider access initiatives from further education colleges and from the workplace; students who are publicly funded and students who are privately funded; and international students whatever their location of study. ELIR will also embrace collaborative provision wherever and however it is delivered.

42 ELIR will include within its scope provision leading to the award of the institution (including credit) that is delivered elsewhere, such as through a further education college or employer organisation. In cases where the delivering institution is itself a Scottish higher education institution, the delivering institution will receive an ELIR review in its own right. Responsibility for the academic standards of awards offered through such arrangements, however, remains unambiguously with the awarding institution. Where provision is made in conjunction with an overseas partner, ELIR will relate to the arrangements in place in the Scottish institution for managing the quality of the learning experience and the academic standards of the awards. The Scottish higher education institutions will continue to participate in overseas audits and the outcomes from those audit reports will form useful reference points in ELIR.

43 ELIR will include consideration of an institution's approach to the management of the learning experience of all students on credit-bearing provision, and how all these students' needs are considered, whatever their race, gender, sexuality, faith, age or whether they have a disability.

Nature of ELIR

44 The ELIR process is conceived and designed to support institutions' self-evaluation and reflection. Central to the ELIR method, therefore, is the institution's Reflective Analysis (RA), which will highlight the main and the distinctive features of the institution's arrangements for enhancing the student learning experience and securing academic standards. Crucially, the RA will set out the institution's reflections on the effectiveness of its approach in those areas, citing the evidence on which these reflections are based.

45 ELIR is concerned with strategic management and the effectiveness of its implementation in order to secure academic standards and improve the student learning experience. A core ELIR inquiry is, therefore, to ask the institution how it learns from current and past activities in order to inform its strategic approach to quality enhancement and how, in turn, that approach improves the learning experience of its students. The institution is asked to identify the reference points it uses in reflecting both on its strategic approach and on the student learning experience it offers. These reference points may include the institution's own activity, or experience identified within the sector or internationally (see above, paragraphs 28 and 29).

46 The ELIR method seeks to:

• be open and transparent, forward-looking, and conducted in a collaborative spirit, avoiding surprises
• support the sector, individual institutions and their staff and students in securing academic standards and enhancing effective learning
• relate to the wider quality enhancement framework in Scotland.

UK-wide and international comparability

47 In addition to the added-value of the ELIR approach, comparability of reporting with audit and review methods across the UK is provided through:

• the use of key UK-wide reference points both by the institutions and the ELIR method (these include the Code of practice, published by QAA, subject benchmark information, and the aligned qualifications frameworks)
• reporting the outcomes of ELIR in the form of a confidence statement using levels of confidence that are equivalent to those in other parts of the UK
• the use of ELIR teams that include members drawn from across the UK.
• 48 In the revised ELIR method, greater emphasis has been placed on international comparability through:
• encouraging institutions to evaluate their use of external reference points and practice, including the Standards and Guidelines for Quality Assurance in the European Higher Education Area
• including an international member on each ELIR team
• disseminating internationally the practice and outcomes of ELIR and the related programme of Enhancement Themes.

ELIR method

49 The ELIR method consists of four integrated elements: an annual discussion with each institution; the production of an RA; a review visit following the submission of the RA; and sector-wide feedback on the learning points from ELIR activity.

Annual ELIR discussion

50 Annual discussions facilitate the review process and provide an important opportunity for information sharing between QAA Scotland and the institution. These annual meetings will be held between a member of QAA Scotland staff and a small group from the institution, which is likely to comprise senior colleagues and a representative of the student body. As was previously the case in ELIR, the annual discussions will not result in any formal judgements or any public reporting. Following the meeting, the QAA officer will write to the institution to confirm any action points agreed and/or to outline any key topics explored.

51 As was previously the case in ELIR, a particular focus of the annual meetings will continue to be discussion of the institution's approach to institution-led quality reviews, and what the institution is learning from the outcomes of the reviews that have been held in the preceding year. Institutions may wish to share other information and activities at these meetings, for example the outcomes of visits by professional, statutory and regulatory bodies; the institution's experiences of working with elements of the Academic Infrastructure and of engaging with sector initiatives; or significant developments in the institution's approach to quality enhancement.

52 The meetings also provide an opportunity for discussing preparations for, and the outcomes of, ELIR. The annual meeting held closest to the anniversary of the publication of the ELIR report will provide a key opportunity to discuss the year-on response to ELIR, which the institution will be asked to produce (see below, paragraph 81).

53 Other than the year-on response to ELIR, institutions will not be expected to prepare bespoke material for the annual discussions; instead it is anticipated that the meetings will be supported by a set of existing material, such as:

• a copy of the definitive internal document(s) describing the approach taken to institution-led quality review, including information about the forward timetable of reviews
• copies of the institution-led quality review reports for the previous 12 months
• a copy of any internal documents analysing the outcomes from institution-led quality reviews or evaluating the review method(s) used
• a copy of any existing documents relating to changes in the institution's approach to quality enhancement
• a copy of the most recent annual institutional return to the Scottish Funding Council relating to institution-led quality reviews.

54 As was previously the case in ELIR, institutions will be encouraged to provide other material originally prepared for internal purposes (such as awaydays or other strategic meetings), which the institution considers is likely to inform the annual ELIR discussions.

Institutional profile

55 In the year in which the institution is preparing its RA, the QAA officer managing the ELIR will support the institution in producing an institutional profile, which will largely be formed from the series of annual returns made to the Scottish Funding Council relating to institution-led quality reviews over a period agreed between the institution and the officer. The institutional profile is a descriptive document providing outline information for the ELIR team about the institution's method of institution-led quality reviews and identifying summary information about the key outcomes of the reviews held over the preceding period.

Reflective analysis

56 In advance of the ELIR visit, the institution will be asked to submit an RA. The RA should act as a demonstration of the institution's capacity for self-reflection and critical evaluation in relation to the matters within the scope of ELIR. The evaluation will be evidence-based and the RA should include the evidence, or clear reference to the evidence, on which any analysis is based.

57 There will be a close link between the RA and the ELIR report; institutions will be asked to address the overviews set out within the ELIR report structure, providing a clear indication of the institution's approach to these matters, and providing an analysis of the effectiveness of the institution's approach alongside references to the evidence on which that analysis is based. Normally, the institution will use the outcomes of the previous ELIR, or institutional practice at that time, as the starting point for the analysis. The RA should contain the minimum descriptive text necessary consistent with setting out the nature of the institution's approach and in order to elucidate the evaluative material. Where any further explanation of the institution's processes or systems is thought to be desirable, the institution should provide existing documents or handbooks alongside or subsequent to submission of the RA.

Case-studies

58 Case-studies are an important part of the revised method, supporting the information base for the review. They provide institutions with an opportunity to identify for ELIR teams self-contained examples of the institution's strategic approach in action. The focus of the case-studies should be on the way the institution is managing particular activity, rather than on individual examples of good practice in themselves. In this way, the text of the case-studies should illustrate the linkage between the institution's strategic approach and its operational management.

59 Case-studies may relate to any aspect within the scope of ELIR; that is institutional approaches to managing academic standards, the student learning experience, or quality enhancement. Institutions should submit one or more case-studies with their RA. There should be explicit links between the RA and the case-studies. The case-study material should set out a clear rationale for its inclusion with the RA; an indication of the institution's process of selecting the case-study; and a clear statement of the aspect(s) of the institution's strategic approach that the case-study is intended to illustrate. In common with the rest of the RA, the case-study text should be reflective and evidence-based. An evaluative case-study should seek to address what the institution was seeking to achieve; why the institution selected the management approach it adopted; how the institution has (or will) evaluate its approach, including an indication of the impact of the institution's actions, insofar as it is possible to do so; and how the institution intends to adjust its approach in future. If an activity has not been in operation long enough to permit impact to be evaluated, the institution could usefully include an indication of the anticipated impact.

Student involvement

60 As the first edition of the Handbook stressed, it is anticipated that the process of producing the RA will be inclusive of all appropriate interests within the institution. In particular, institutions are expected to involve students in the process of creating the RA. The RA should indicate how student views have been elicited and incorporated, and should offer a view on the impact of that student engagement.

The ELIR visit

61 The operation of ELIR introduced a two-part visit structure, with a five-week interval between the two parts. Following careful consideration and discussion, this has been retained.

62 The Part 1 visit involves the ELIR team visiting the institution for two days. The purpose of Part 1 is to:

• ensure that the ELIR team has a sound understanding of the institution and its approach to the matters within the scope of ELIR as laid out in the RA
• allow the team to identify the ELIR themes to be explored during the Part 2 visit and to share these with the institution
• allow the team time to agree and share with the institution an appropriate programme of activities for the Part 2 visit
• identify particular documentation to facilitate the team's exploration of ELIR themes in the Part 2 visit.

63 The programme of activities during the first half-day of the Part 1 visit is determined by the institution. The institution has full flexibility in providing a programme of activities which it believes will most effectively assist the ELIR team in understanding the institution's approach to managing and enhancing the student learning experience. Evaluations of ELIR have demonstrated that this half-day is highly beneficial in establishing a positive relationship between the ELIR team and the institution and, therefore, in setting the tone for the remainder of the review. In order to further facilitate the establishment of a collaborative relationship, QAA Scotland will liaise with the institution in advance of the Part 1 visit, sharing the ELIR team's early identification of themes arising from the RA. The institution, in turn, will be asked to share at an early stage the outline of its programme for the first half-day with the team.

64 The remainder of the Part 1 visit will involve a series of meetings to explore agendas developed by the ELIR team following its consideration of the RA and the interaction of the first half-day. There will be meetings with a group of senior staff; a group of student representatives; and a group of staff who have experience of the institution-led quality review activity. At the end of the Part 1 visit, the ELIR team will share with the institution the themes to be explored during the Part 2 visit, together with a draft programme for that visit.

65 The Part 2 visit will last between three and five days depending on the size and complexity of the institution, and on the themes emerging from the RA and the Part 1 visit. In exploring these themes during Part 2, the ELIR team will wish to consider a specified set of the institution's documents and to meet with a range of staff and students. Discussions during these meetings will cover the effectiveness of the institution's strategic approach to enhancement; the effectiveness of its management of the student learning experience; and the effectiveness of its arrangements for institution-led monitoring and review. The Part 2 visit will include opportunities for the ELIR team and the institution to clarify any matters as appropriate. On the final day of the Part 2 visit, the ELIR team will meet with the assistant director to agree its conclusions and to compile an outline of the draft ELIR report.

66 One week after the end of the Part 2 visit, the assistant director will send a 'key themes' letter to the institution summarising the provisional conclusions of the ELIR. A draft of the main and summary ELIR reports will be sent to the institution for comment within an agreed period of the conclusion of the Part 2 visit (see below, paragraph 75). Following receipt of the institution's comments, QAA Scotland will finalise and publish the ELIR reports.

Disseminating ELIR outcomes

67 QAA Scotland, working with the sector, will play an active role in disseminating ELIR outcomes and good practice identified through the ELIR process. All ELIR reports are available on the QAA website and are, therefore, readily accessible in their full and summary forms. QAA Scotland will undertake regular analysis of the main themes emerging from ELIR activities across the sector. This analysis, together with examples of institutional practice drawn from the ELIR process more generally, will form the basis of publications, workshops/conferences and other activity. QAA Scotland will liaise closely with the institutions in selecting these examples and in identifying the most effective dissemination mechanism. As part of its regular reporting to the Scottish Funding Council (SFC), QAA Scotland will provide SFC with annual reports on the outcomes of ELIR activity.

68 In addition, QAA Scotland will continue to publish the series of 'Learning from ELIR' reports which address different aspects of the enhancement-led approach to institutional management in the Scottish sector. That series, and other ELIR reports and events, will provide evidence of the impact of the enhancement-led approach, will inform national and international debate and, in particular, will guide SHEEC as it takes forward its strategy to support the implementation and embedding of a quality culture in general and the Enhancement Themes in particular.

Complaints and representations

69 The procedures for making complaints on any aspect of the operation of the ELIR process, or representations in respect of reported judgements, are available on the QAA website.

Section 3- The ELIR reports

Audiences and reporting

70 The purposes of reporting on the outcomes of ELIR are to provide informed peer feedback to individual institutions and to provide public information on quality and academic standards within and beyond Scotland. Public reporting also has an important role to play in the wider promotion and support of quality enhancement. The purposes of ELIR reports are:

• to provide public information about the security of academic standards and the management of the student learning experience
• to support the development of quality cultures and the strategic management of quality enhancement in individual institutions and the Scottish higher education sector as a whole
• to promote the positive reputation of Scottish higher education institutions within the UK and internationally in relation to good practice in academic standards and quality management
• to provide a formal record of the individual review and its outcomes.

71 ELIR outcomes attract interest from a wide audience, for example students and student bodies; public bodies with an interest in quality management including SFC, Universities Scotland, HEA, and public, statutory and regulatory bodies; and employers. The full ELIR reports also attract a specialist audience including key members of staff from the institution being reviewed, quality managers at other institutions, and officers from QEF partner bodies, notably QAA Scotland and SFC officers.

72 In recognition of the different needs and interests of these audience groups, there will be two reports on ELIR outcomes for individual institutions: the main report and a summary report specifically aimed at a wider, less specialist audience. Both reports will be published on the QAA website; the summary reports will also be available in print format.

Main report

73 The main ELIR report will provide information about the effectiveness of the institution's approach in three broad areas:

• management of the student learning experience
• institution-led monitoring and review of quality and academic standards
• strategic approach to quality enhancement.

74 Further detail on the structure and indicative content of the main ELIR report is provided in Annex 1.

75 The main and the summary reports will both be sent to the institution in draft form for comment within eight weeks of the end of the Part 2 visit and will be published on the QAA website within 20 weeks of the end of the Part 2 visit.

Summary report

76 The summary report will be a short document setting out key information about the method and outcomes of the review aimed at an informed lay audience. It will be drafted by QAA Scotland and agreed by the ELIR team, before being sent to the institution for comment at the same time as the main ELIR report. The summary report will include:

• brief introductory material about the institution being reviewed, emphasising the context of the review
• a short summary of key matters raised in the review, emphasising the approach of the institution
• an indication of the commentaries
• the confidence judgement which will be set out in full with brief explanatory text where this is thought to be necessary in order to make the text fully accessible to a wider audience.

Commentaries and judgement

77 The three main sections of the ELIR report will each lead to a commentary in which the ELIR team will set out its views of the effectiveness of the institution's approach. The commentaries will draw on the material included within the overviews of the relevant report section and there will be no prescribed formulae for expressing the team's views. Each of the commentaries will include the team's views on the institution's management of collaborative activity and the international dimensions referred to earlier. Since these areas will be fully integrated, there will not be separate commentaries on those aspects.

78 The commentaries will highlight:

• the effectiveness of the institution's management of the student learning experience
• the effectiveness of the institution's arrangements for institution-led monitoring and review of quality and academic standards of awards, however and wherever delivered
• the effectiveness of the institution's implementation of its strategic approach to quality enhancement.

79 These commentaries will lead to a single overarching judgement which will be expressed in the form of a confidence statement. The confidence statement is intended to provide an holistic view of the effectiveness of the institution's management of academic standards, and of the assurance and enhancement of the student learning experience, both currently and in the future. It will be possible, where the ELIR team considers it necessary, to separate the elements of this judgement to express different levels of confidence in the different aspects of the institution's management, or if the team has a different level of confidence in the current and likely future management of activity. It will also be possible to add caveats to the confidence judgement as appropriate to the findings of the review and in order to draw attention to the management of specific aspects of activity, for example collaborative activity, or to specific groups of students.

80 ELIR teams will express their level of confidence in one of three standard forms: confidence, limited confidence, or no confidence. The confidence statement is a judgement of probability; it cannot be unconditional. Where the team finds that the institution is managing the security of academic standards effectively, and where this looks likely to continue in the future, the team will express 'confidence'. Similarly, where the team finds that the institution is managing the quality of the student learning experience effectively, including taking appropriate steps to enhance that experience, and is likely to continue to do so in the future, the team will express 'confidence'. If the ELIR team has substantial doubts about the current, or likely future, management of the security of academic standards and/or the effectiveness of the student learning experience, the team will express 'limited confidence' and will indicate clearly the areas of concern that have given rise to the limitation of confidence. A judgement of 'limited confidence' will indicate that there is evidence that the institution's capacity to manage effectively the academic standards of its awards and/or the student learning experience is limited or is likely to become limited in the future. It is not a judgement of failure but it does indicate that improvements need to be made. If the ELIR team had serious concerns about the current, or likely future, management of the security of academic standards and/or the quality of provision, it would express 'no confidence' and would indicate clearly the significant areas of concern that had given rise to the judgement. A judgement of 'no confidence' indicates that there is substantial evidence of serious and fundamental weaknesses in the institution's capacity to secure the academic standards of its awards and/or to maintain an appropriate quality of educational provision.

Follow-up to the report

81 One year after publication of the ELIR report, QAA Scotland will ask the institution to produce a year-on response to ELIR. This will focus on the action taken following the review and will include consideration of the effectiveness of that action insofar as it is reasonable for the institution to take such a view in the time that has elapsed. The year-on response will be a focus for the annual ELIR discussion held closest to the anniversary of the publication of the ELIR report.

82 In the year-on ELIR response, institutions will be asked to pay particular attention to setting out the action that has been taken in relation to any recommendations that are associated with the confidence statement. Institutions will continue to have flexibility in the length and style of the year-on responses they submit to QAA.

83 After the annual discussion at which the year-on ELIR response has been discussed, the QAA Scotland officer will write to the institution to confirm whether the action the institution has taken is broadly in line with the outcomes of the ELIR report. QAA Scotland is asked to report to SFC on the extent to which institutions are responding appropriately to the outcomes contained within the ELIR reports. The year-on responses to ELIR will form a key information source for these regular reports.

84 The final version of the institution's year-on response to ELIR will be published on the QAA website alongside the ELIR report to which the response refers. For this version, institutions will be asked to provide brief information about the areas in which they have taken action since publication of the ELIR report, focusing particularly on any areas of action that relate to the confidence judgement. Institutions will be able to write these reports in their own words and there will be some flexibility in the structure used to fit the nature of the outcomes of the ELIR report in question, but there will be a number of standard topics to be addressed:

• identifying the main areas in which action has been taken by the institution since the ELIR
• how action has been taken forward, or is planned to be taken forward, and how its effectiveness will be evaluated
• identifying any matters that have not been addressed, with a rationale for this or with an outline of future plans
• identifying any other matters the institution wishes to highlight.

85 SFC will be provided with copies of the full and summary ELIR reports. As a matter of course, SFC may wish to provide a commentary to institutions following receipt of reports. Where the ELIR report indicates that there can only be limited or no confidence in the institution's ability to manage quality and/or academic standards, SFC will require the institution to undertake follow-up action in specified areas. Institutions will have the opportunity to rectify limited or no confidence judgements, normally through a follow-up review. The precise methodology and timescale for the follow-up review will be agreed by QAA Scotland and SFC in consultation with the institution, having regard to the reasons for the limited or no confidence judgement and the nature of the remedial action the institution is required to take.

Section 4 - The ELIR team

The ELIR team

86 ELIR teams will comprise six peer reviewers: a student reviewer, four reviewers who are senior academic managers (including an international reviewer), and an ELIR coordinating reviewer. While some reviewers may have specific responsibilities, the ELIR team will act together during the review visits and decisions relating to the commentaries and conclusions within the ELIR report will be taken collectively by the team. All reviewers are full members of the ELIR team, with equal status.

87 All reviewers will have responsibility for:

• reading and analysing the RA and any other documentation provided by the institution
• participating in the review visits
• reaching conclusions on the basis of the information gained during the review
• contributing to and commenting on the review report.

88 The international reviewer will bring an added perspective to the ELIR team's consideration of the institution's approach to quality assurance and the enhancement of the student learning experience. The international reviewer will have a range of knowledge and experience that will benefit the review process, the institution, and the wider Scottish higher education sector. International reviewers will be senior peers, selected from appropriate institutions or organisations, and will be required to undertake full training for their role.

89 The student reviewer brings a learner perspective to the review. Their responsibilities will focus on lines of enquiry relating to the institution's management of the student learning experience and the effectiveness of the institution's approach to engaging students.

90 The ELIR coordinating reviewer (ECR) will, usually, be a senior administrator and will have responsibility for maintaining an overview of the review progress and its outcomes. The ECR will have particular, although not necessarily exclusive, responsibility for:

• ensuring the ELIR team has access to appropriate documentation during the review
• maintaining a record of discussions held with staff and students
• facilitating the team's identification and evaluation of the key themes to be pursued during the review, ensuring alignment of these themes with the overviews and commentaries on which the team must report
• supporting the team in identifying the evidence on which its conclusions and commentaries are based.

91 In undertaking these responsibilities, the ECR will maintain an ongoing record of the team's emerging conclusions and supporting evidence. At the end of the review visit, the ECR will use the ongoing record to support the team and the QAA assistant director in producing an outline report, identifying the key themes for inclusion within each of the report commentaries. Other members of the ELIR team will have responsibility for supporting the ECR in preparing the outline report and will have responsibility for drafting report text. Along with other members of the ELIR team, the ECR will support the assistant director in editing the review report, providing additional information and evidence as necessary.

92 Each review will be managed by a QAA assistant director. From early in the process, the assistant director will provide advice to the institution on its preparations for the review, and will work with the ELIR team on the initial analysis of documentation. The assistant director will accompany the team during the Part 1 visit and for elements of the Part 2 visit, notably the final day, providing advice as appropriate. The assistant director, supported by the ECR, is responsible for testing that the team's commentaries are based on adequate and identifiable evidence, and for editing the ELIR report.

Selection criteria for reviewers

93 All members of ELIR teams will be selected by QAA Scotland according to the criteria identified in Annex 3 and having regard to the timetable for reviews in Scotland.

94 Student reviewers will be recruited annually. QAA Scotland will seek nominations from student representative bodies and Scottish higher education institutions. Student reviewers will be eligible to undertake reviews for as long as they continue to meet the selection criteria, in particular provided it is not more than three years since they undertook study in a Scottish higher education institution.

95 International reviewers will be drawn from outside the UK. On an annual basis, Scottish higher education institutions will be invited to nominate one or more international reviewers to the reviewer pool. In addition, QAA Scotland will seek nominations through its contact with institutions and relevant organisations in other countries.

96 QAA Scotland will actively seek nominations from all UK higher education institutions for senior academic reviewers and ELIR coordinating reviewers. Every Scottish institution will be encouraged to nominate at least one candidate for each role. Individuals who are existing reviewers, auditors or secretaries with QAA may be eligible to become ELIR reviewers and can apply using a streamlined process.

Allocating reviewers to teams

97 QAA Scotland will allocate reviewers to ELIR teams. Reviewers will not be allocated to the ELIR team for their own institution. Institutions will have an opportunity to comment on the composition of their ELIR team, and every effort will be made to select an appropriate team for the institution being reviewed.

ELIR reviewer training

98 All ELIR reviewers, including those trained in other QAA methods, are required to undertake ELIR training. Reviewers are also expected to participate in continuing development and information seminars. Where appropriate, training and continuing development may be targeted to specific groups of reviewers, such as students or ELIR coordinating reviewers. Training in the ELIR method will be provided each year in order to enhance effectiveness by minimising the gap between training and reviews taking place.

99 To support international reviewers, introductory materials (for example, on the characteristics of the Scottish Higher Education sector) will be sent to international reviewers in advance of training. In addition, there will be an induction meeting of international reviewers, along with QAA Scotland officers. International reviewers will be expected to attend the full training programme alongside UK-based reviewers.

100 ELIR training will be designed and managed by QAA Scotland, although delivery may be shared with external facilitators, in particular experienced reviewers drawn from the sector. Training will cover the philosophy and methodology of ELIR as well as skills-related activities.

Continuing development and information sessions

101 In order to share experience of the ELIR method and to maintain the knowledge of experienced reviewers, annual information sessions will be held. All reviewers who have participated in an ELIR team during the cycle, together with those allocated to teams for the following year, will be invited to attend. These information sessions may be combined with events for representatives of the Scottish higher education sector.

Section 5 - Monitoring and evaluation

Purpose and principles of monitoring and evaluation

102 QAA Scotland will monitor, on an ongoing basis, the operation of ELIR, undertake regular evaluation of the effectiveness of ELIR, and also contribute to the broader external evaluation of the arrangements for quality assurance and enhancement in Scotland. The overall purpose of monitoring and evaluation is to:

• support QAA Scotland in effectively delivering the agreed ELIR methodology
• inform the ongoing development of the ELIR method, in the context of the Quality Enhancement Framework.

103 The underlying requirements of monitoring and evaluation are that the processes should:

• be regular and timely
• ensure higher education institutions and reviewers can provide structured feedback
• support the ongoing training and continuing development of reviewers
• support ongoing improvement of QAA Scotland's performance
• encourage active reflection and dialogue on the design and improvement of the ELIR approach to ensure it continues to be fit for purpose.

Monitoring

104 Monitoring activities will encompass all stages of the ELIR process as follows:

• annual discussions
• ongoing QAA Scotland support for institutions, and the sector, in their preparation for ELIR
• parts 1 and 2 of the ELIR visit
• notifications of ELIR outcomes and arrangements for report drafting and finalisation.

105 All those engaging in ELIR will be involved in the monitoring process: the institution, reviewers, and the assistant director responsible for managing the review. The monitoring process will involve both formal and informal feedback.

106 Formal feedback will be sought through monitoring questionnaires, which all participants in ELIR will be asked to complete. The questionnaires will seek comment both on operational aspects of the review, and broader questions relating to the effectiveness of ELIR in achieving its aims. The latter will allow ongoing accumulation of information and evidence to inform the wider process of evaluation.

107 Informal feedback will also be encouraged. The assistant director responsible for a review will be the main focus for communication of feedback from institutions and reviewers, and will also have their own observations to contribute. Informal feedback will continue to be sought from groups with which QAA Scotland liaises, including the Universities Scotland Teaching Quality Forum and sparqs.

Evaluation

108 Building on the feedback from monitoring activity, QAA Scotland will undertake ongoing evaluation of the effectiveness of ELIR in achieving its objectives. The overall purpose of evaluation is to 'stand back' and take stock of the effectiveness of the ELIR method. This will pose macro questions such as: Does ELIR meet its objectives? What aspects work well? Which elements could be improved and in what ways?

109 When a number of reviews have been completed and the feedback from them has been analysed, evaluation events will be held involving reviewers and institutional representatives. The purpose of these events will be to encourage reflection on the effectiveness of the ELIR approach, and to promote dissemination of key messages.

110 The findings from monitoring and evaluation will help to inform the ongoing training and development of ELIR reviewers, to ensure that they are effectively prepared and supported in fulfilling their roles.

External evaluation of the quality enhancement framework

111 SFC will commission an external evaluation of the arrangements for quality assurance and enhancement in Scottish universities and colleges. The evaluation team is to be appointed by the start of the review cycle in 2008-09. QAA Scotland will fully participate in this external evaluation and use the outcomes generated as appropriate.

Annex 1 - Main ELIR report structure

Notes on the report structure

1 The primary audience for the main ELIR report is intended to be staff in the institution being reviewed, quality managers from other institutions and officers from bodies such as SFC and QAA. This has a number of impacts on the style and content of the text. In particular, there will be significantly less descriptive text overall as compared to ELIR reports in the previous version of the method.

2 The text of the main ELIR report will be organised around a series of overviews contained within the three main report sections. Each overview will include brief text to indicate the key nature of that topic at the institution being reviewed, followed by text setting out the ELIR team's view of the effectiveness of the institution's approach, with an indication of the basis for that view. The final section of the report will set out the three commentaries and will conclude with the confidence statement. This structure is intended to ensure that the reports continue to address the particular approach of the institution being reviewed. It is also hoped that the structure will be helpful to the reader as there will be less repetition of text, and material on particular topics should be easier to identify than was sometimes the case with reports in the previous version of the method.

3 Each of the three main sections of the report will include text relating to the effectiveness of the institution's management of collaborative activity. The ELIR team's view will be set out in the relevant commentaries; there will not be a separate commentary relating to collaborative provision.

4 As was previously the case in ELIR, there will be a strong link between the report and the institution's RA. There will also be a strong link between the outcomes of the previous ELIR and the starting point for ELIR in the revised method. Institutions will be asked to bear this in mind in drafting their RA.

5 Institutions should submit one or more case-studies with their RA. These will be listed in the report introduction, and evaluative text relating to the case-studies will also appear within the report section and overview(s) to which they relate (this will be dependent on the management process that the submitting institution intends the case-study to illustrate).

6 Equality of opportunity and diversity of student populations will be key themes in the revised ELIR method and will underpin a number of the lines of enquiry for ELIR teams. Matters relating to these areas may be covered under a number of the report overviews, and are explicitly highlighted in relation to the section on the Management of the Student Learning Experience.

Report section 1: Introduction

7 The text of the introduction will be brief, covering the following material:

• standard summary information about the ELIR method
• brief information about the ELIR method as it related to this institution, including key features of the conduct of the review and the members of the team
• summary information about the institution including size, broad breakdown of student population and location of students if significant numbers are studying on collaborative programmes, and the key institutional structure
• summary information about collaborative activity
• short statement of contextual factors at the time of the review including:
  • key developments within the institution, this might include, but is not limited to: structural changes, merger, relocation
  • key changes in areas relevant to ELIR; a summary outline should be provided in the introduction with more detail provided in the relevant substantive section(s) of the report
• the institution's statement of its strategic approach to quality enhancement.

Report section 2: Management of the student learning experience

8 The overviews in the student learning experience section will cover:

• key features of the student population and the effectiveness of the institution's approach to managing information about its student population
• the effectiveness of the institution's approach to engaging and supporting students in their learning
• the effectiveness of the institution's approach to promoting the development of graduate attributes, including those relating to employability, in all of its students
• the effectiveness of the institution's approach to managing the learning environment
• the effectiveness of the institution's approach to promoting equality of opportunity and effective learning for all of its students
• the effectiveness of the institution's approach to supporting and developing staff to promote effective learning for their students.

9 In relation to collaborative activity, this section will cover:

• the effectiveness of the institution's management of the student learning experience on collaborative programmes with regard to the topics set out in the overviews above.

Report section 3: Institution-led monitoring and review of quality and standards

10 The overviews in the institution-led monitoring and review section will cover:

• key features of institution-led monitoring and review at the institution, and the extent to which these arrangements meet sector-wide expectations
• the extent to which the institution's monitoring and review arrangements include consideration of all students
• the effectiveness of the institution's approach to self-evaluation including the use made of external reference points
• the effectiveness of the institution's approach to the management of information to inform the operation and evaluation of its monitoring and review activity
• the effectiveness of the institution's approach to setting and maintaining academic standards, including the management of assessment
• the effectiveness of the institution's approach to managing public information about quality and academic standards, including the linkage with the institution's monitoring and review arrangements
• the effectiveness of the institution's approach to linking its monitoring and review processes to its enhancement arrangements.

11 In relation to collaborative activity, this section will cover:

• the effectiveness of the institution's approach to monitoring and reviewing collaborative activity with regard to the topics set out in the overviews of this section.

Report section 4: Strategic approach to quality enhancement

12 The overviews in the strategic approach to quality enhancement section will cover:

• key features of the institution's strategic approach to quality enhancement
• the effectiveness of the institution's implementation of its strategies and policies for promoting quality enhancement across the institution
• the effectiveness of the institution's use of external reference points in its approach to quality enhancement, including the extent to which the institution's approach is informed by national and international practice
• the effectiveness of the institution's approach to identifying, disseminating and implementing good practice in the context of its strategic approach to enhancement

13 In relation to collaborative activity, this section will cover:

• the effectiveness of the institution's approach to enhancing collaborative provision.

Report section 5: Conclusion

14 This section will set out the report commentaries, followed by the confidence statement covering:

• the effectiveness of the institution's management of the student learning experience
• the effectiveness of the institution's arrangements for institution-led monitoring and review of quality, and academic standards of awards, however and wherever delivered
• the effectiveness of the institution's implementation of its strategic approach to quality enhancement.

15 The commentaries will lead to the overarching confidence judgement. The core confidence statement will be expressed as follows:

The findings of the ELIR indicate that there can be confidence/limited confidence/no confidence in the institution's current and likely future management of the academic standards of the awards it offers, and the quality of the student learning experience it provides.

Note: As is currently the case, different levels of confidence may be expressed in different aspects of the judgement if the ELIR team considers this is necessary. The confidence judgement may also be qualified with the addition of caveats. In addition, the confidence judgement may be contextualised to emphasise the key elements of the commentaries on which it is based.

Annex 2 - Guidance for institutions on a reflective analysis (RA) structure

Notes on the reflective analysis

1 The text of the RA should predominantly be evaluative with specific references to supporting material or further reading. Evaluation of the ELIRs carried out between 2003 and 2007 has demonstrated that there is considerable benefit in institutions following the structure of the ELIR report when producing their RA. For the revised ELIR method, institutions may choose to present their analyses in different structures but, however the text is organised, institutions are asked to address the overviews and commentaries set out in the ELIR report. In relation to each overview, institutions are asked to set out their view of their own effectiveness and, in each case, institutions should make clear the basis for that view, including specific references to supporting material as appropriate.

2 Similar to the case in the previous ELIR method, there will be clear links between the RA and the ELIR report. There will also be a strong link between the outcomes of the previous ELIR and the starting point for the current ELIR. In many cases it is expected that institutions would wish to use the timing of their first ELIR as the starting point for evaluation in the RA.

3 Institutions should submit one or more case-studies with their RA to illustrate for the ELIR team the linkages between the institution's strategic approach and its operational management. It should be clear from the text of the RA which feature(s) of the institution's strategic management each case-study is intended to illuminate. The text of the case-study material itself should be evaluative and evidence-based, it should not simply describe individual instances of good practice. The ELIR Handbook (see above, paragraph 59) gives an indication of the form the case-study text should take. Institutions have flexibility in deciding on the method of presenting the case-study material, for example it may be appended to the RA or located within the structure of the RA itself.

4 Equality of opportunity and diversity of student populations will be key themes in the revised ELIR method and will underpin a number of the lines of enquiry for ELIR teams. These matters will be included in the ELIR report section on the Management of the Student Learning Experience but, if it is appropriate to the institution's approach, may also be addressed elsewhere in the ELIR report.

5 It is open to each institution to include with its RA background or other information about the institution and its approach in the format that it considers will be most helpful for the ELIR team.

RA section 1: Introduction

6 Institutions are asked to provide the following information:

• a statement of contextual factors at the time of the review, including:
  • key developments within the institution; this might include, but is not limited to, structural changes, merger, relocation
  • key changes in areas relevant to ELIR; a summary outline could be provided in the introduction with more detail provided in the relevant substantive section(s) of the RA
• brief information on the method used to produce the RA, including the evaluative processes employed and the extent to which the views of all staff and students have been incorporated within the evaluation. The RA should offer a view on the impact of the engagement of students in this process. This might be covered in greater detail in later sections of the RA
• any other information the institution considers would be helpful to the ELIR team to support the team's understanding of the institution's context and priorities.

RA section 2: Management of the student learning experience

7 The overviews of the student learning experience section are set out here. In each case the institution is asked to identify the particular features of its approach and provide a view on the effectiveness of that approach, making clear the basis for its view:

• key features of the student population and the effectiveness of the institution's approach to managing information about its student population
• the effectiveness of the institution's approach to engaging and supporting students in their learning
• the effectiveness of the institution's approach to promoting the development of graduate attributes, including those relating to employability, in all of its students
• the effectiveness of the institution's approach to managing the learning environment
• the effectiveness of the institution's approach to promoting equality of opportunity and effective learning for all of its students
• the effectiveness of the institution's approach to supporting and developing staff to promote effective learning for their students.

8 In relation to collaborative activity, the institution is asked to:

• identify its approach to managing the student learning experience on collaborative programmes with regard to the topics set out in the overviews above, and provide the institution's view of the effectiveness of that approach.

RA section 3: Institution-led monitoring and review of quality and standards

9 The overviews of the institution-led monitoring and review section are set out here. In each case the institution is asked to identify the particular features of its approach and provide a view on the effectiveness of that approach, making clear the basis for its view:

• key features of institution-led monitoring and review at the institution, and the extent to which these arrangements meet sector-wide expectations
• the extent to which the institution's monitoring and review arrangements include consideration of all students
• the effectiveness of the institution's approach to self-evaluation, including the use made of external reference points
• the effectiveness of the institution's approach to the management of information to inform the operation and evaluation of its monitoring and review activity
• the effectiveness of the institution's approach to setting and maintaining academic standards including the management of assessment
• the effectiveness of the institution's approach to managing public information about quality and academic standards, including the linkage with the institution's monitoring and review arrangements
• the effectiveness of the institution's approach to linking its monitoring and review processes to its enhancement arrangements.

10 In relation to collaborative activity, the institution is asked to:

• identify its approach to monitoring and reviewing collaborative activity with regard to the topics set out in the overviews of this section, and provide the institution's view on the effectiveness of its approach.

11 In order to support the evaluative material in this section, the institution should include an analysis of the outcomes of the institution-led quality reviews it has carried out, together with information about the action taken to address the matters raised by those reviews.

RA section 4: Strategic approach to quality enhancement

12 The overviews of the strategic approach to quality enhancement section are set out here. In each case the institution is asked to identify the particular features of its approach and provide a view on the effectiveness of that approach, making clear the basis for its view:

• key features of the institution's strategic approach to quality enhancement
• the effectiveness of the institution's implementation of its strategies and policies for promoting quality enhancement across the institution
• the effectiveness of the institution's use of external reference points in its approach to quality enhancement, including the extent to which the institution's approach is informed by national and international practice
• the effectiveness of the institution's approach to identifying, disseminating and implementing good practice in the context of its strategic approach to enhancement.

13 In relation to collaborative activity, the institution is asked to:

• identify its approach to enhancing collaborative provision and provide the institution's view of its effectiveness in this regard.

RA section 5: Conclusion

14 The institution is asked to draw together the material set out earlier in the RA by evaluating the effectiveness of its approach in the three areas covered by the ELIR commentaries. In each case, the institution is asked to highlight what it considers are the key features of its approach and provide a view of its effectiveness, setting out the basis for its view:

• the effectiveness of the institution's management of the student learning experience
• the effectiveness of the institution's arrangements for institution-led monitoring and review of quality and academic standards of awards, however and wherever delivered
• the effectiveness of the institution's implementation of its strategic approach to quality enhancement.

Annexes to the RA

15 The following information should be included as annexes to the RA:

• an index to supporting evidence and further reading; there should be clear cross-references to this material throughout the RA
• summary information about the institution, including the organisational structure (schools/faculties/etc) and a breakdown of staff and student numbers
• information about collaborative activity, including the student numbers by collaborative partner, location and programme
• case-study material can be appended or embedded within the structure of the RA.

Annex 3 - Criteria for the selection of reviewers

Introduction

1 All ELIR reviewers will be selected by QAA Scotland on the basis of the criteria set out below. Nominations will be invited from institutions across the UK, with every Scottish institution encouraged to make at least one nomination to each reviewer role. Student reviewers may be nominated by Scottish student representative bodies or Scottish higher education institutions. International reviewers will be selected from Scottish higher education institution nominations and from QAA Scotland's contacts with relevant institutions and organisations in other countries.

2 The qualities required in ELIR reviewers are detailed below. Student reviewers are required to have current or recent direct experience of study at a Scottish higher education institution. International reviewers will be recruited for the broader perspective they are able to bring to the areas of enquiry within the scope of ELIR. ELIR coordinating reviewers and UK-based senior academic reviewers will be drawn from across the UK. Every attempt will be made to ensure that the total pool of ELIR reviewers reflects sectoral, geographical, and equal opportunity dimensions.

3 All reviewers will be provided with training by QAA Scotland to ensure that they are familiar with the ELIR approach. The purposes of ELIR training are to ensure that all reviewers:

• understand the aims and objectives of the ELIR process as well as the key elements of the method
• understand their own roles and tasks, the importance of team coherence, and the collaborative style ELIR aims to achieve
• have an opportunity to explore and practise review skills, including information assimilation and analysis; the development of programmes for visits; achieving a constructive dialogue; the construction and testing of hypotheses; the formation of evidence-based conclusions and statements of confidence; and the preparation of reports.

Qualities required in all reviewers

4 All reviewers will be able to demonstrate the ability to:

• understand a range of perspectives; relate to a range of individuals including students and senior managers; lead discussions about strategic and operational approaches to the management of quality and standards in general, and the enhancement of the student learning experience in particular
• assimilate a large amount of disparate information and analyse it to form reliable, evidence-based conclusions
• communicate clearly, orally and in writing
• work productively and cooperatively in small teams delivering to tight deadlines
• maintain the confidentiality of sensitive matters.

Additional qualities required in UK-based senior academic reviewers

5 In addition to the qualities required in all reviewers, UK-based senior academic reviewers will be able to demonstrate:

• current or recent (within three years) wide experience of academic management, preferably relating to quality assurance and enhancement, at the institutional level in the UK
• personal and professional credibility with staff, including senior managers, heads of institutions, and staff currently engaged in learning and teaching
• knowledge and understanding of the QAA Academic Infrastructure and other key reference points, including the Scottish Credit and Qualifications Framework
• awareness of the distinctive features of the Scottish higher education system in general, and the enhancement-led approach in particular (ELIR training will seek to emphasise this but some initial awareness is highly desirable).

6 QAA Scotland will also be interested in reviewers identifying other relevant experience they may have, for example experience of good practice in quality assurance and enhancement in an international setting.

Additional qualities required in international reviewers

7 In addition to the qualities required in all reviewers, international reviewers should be able to demonstrate a number of the following attributes:

• current or recent (within three years) wide experience of academic management, preferably relating to quality assurance and enhancement, at the institutional level outside the UK
• current or recent (within three years) experience of external review of higher education institutions outside the UK, either as a panel member or through senior involvement with a quality assurance or enhancement organisation
• peer-acknowledged expertise in the development of good practice in learning and teaching
• knowledge and experience of practice in more than one country in addition to the UK (it will be highly desirable to have international comparative knowledge and experience)
• an awareness of the distinctive features of the Scottish higher education system in general, and the enhancement-led approach in particular (ELIR training will emphasise this but some initial awareness is highly desirable).

Additional qualities required in student reviewers

8 QAA Scotland will actively encourage applications from students with all backgrounds and with experience of any mode or level of study.

9 In addition to the qualities required in all reviewers, student reviewers will be able to demonstrate:

• current or recent (within three years) experience of study at a Scottish higher education institution, equivalent to a minimum of one year's full-time education
• experience of representing students' interests at institutional (including faculty or school) level
• general awareness of the diversity of the Scottish higher education sector beyond their 'home' institution, and awareness of the arrangements for quality assurance and enhancement in Scotland (ELIR training will provide further information on this and QAA Scotland is looking for applicants who have the ability to build on their existing experience).

10 QAA Scotland will also be interested in student reviewers identifying their other relevant experience.

Additional qualities required in ELIR coordinating reviewers

11 In addition to the qualities required in all reviewers, ELIR coordinating reviewers will be able to demonstrate:

• current or recent (within three years) experience of senior academic administration at institutional (including faculty or school) level in UK higher education
• wide experience of working with senior committees in UK higher education
• awareness of the distinctive features of the Scottish higher education system in general, and the enhancement-led approach in particular (ELIR training will seek to emphasise this but some initial awareness is highly desirable)
• ability to keep a reliable record of discussions, summarise the key outcomes and produce coherent text in a specified format to tight deadlines
• experience of drafting, collating and editing complex reports
• ability to retain an effective overview of complex tasks, and to support and manage a small team in achieving those tasks.

Annex 4 - SFC Guidance to higher education institutions on quality issues

This annex sets out the SFC guidance document.

Introduction to the guidance

1 It is institutions, and not the Council, which bear primary responsibility for, and ownership of, the quality of educational provision. But the Council has responsibilities to ensure that quality is being maintained and enhanced across the sector, and that good practice is being shared. Since 2003, the Council has addressed these responsibilities through periodic external review by the Quality Assurance Agency (QAA), and through providing guidance to institutions on quality issues. The Council also funds a range of enhancement-related activities, including the national enhancement themes, the Higher Education Academy, the Joint Information Systems Committee, and the sparqs service. The Council will continue to adopt these approaches in the next quality cycle beginning in AY 2008-09.

2 An important mechanism by which institutions assure and enhance the quality of provision is through processes of institution-led review. For universities, these processes are usually referred to as 'internal review' or 'subject review'. The Council issued generic guidance on the operation of internal review in 2003. Institutions have considerable flexibility in how they organise and run these processes but there are clear expectations that all aspects of provision will be reviewed systematically over a defined schedule.

3 Internal review processes are themselves subject to scrutiny through periodic Enhancement-Led Institutional Reviews (ELIR) conducted by QAA. There is clear evidence from the outcomes of ELIR (and indeed from previous rounds of QAA reviews) that quality and standards of provision are secure, that internal review is robust and effective, and that institutions use these processes effectively to manage and enhance the quality of provision. We therefore start from a position of considerable strength and success.

4 The purpose of revising the guidance at this time is not to propose any major change to the principles or operation of the current approach, but to further enhance the benefits of the current model.

Main areas for change

5 The Council's key source of strategic information on quality arrangements in the university sector derives from QAA's reports on the outcomes of ELIR. We will now further enhance QAA's strategic input to the Council by inviting the Director of QAA Scotland to provide an annual briefing to the Council on outcomes and issues arising from the previous year's ELIR activities.

6 The Council also currently receives a short annual report from each institution which summarises the outcomes of internal reviews conducted in the previous session. This information is useful but in the light of experience we now feel it would be helpful to update and clarify the guidance on the content and scope of these reports.

7 Over the last few years, ministerial guidance and parliamentary scrutiny have placed increasing emphasis on governance issues and on the role of governing bodies in providing public accountability for all aspects of institutional activities, including quality assurance and enhancement. The Council has decided that it would be helpful to gain a clearer formal understanding of the arrangements by which the governing body of each HEI ensures accountability for quality. We are therefore asking each institution to produce a short but definitive 'baseline statement' describing these arrangements.

8 The final area of change arises from our increasingly close work with the student body and with the sparqs service. This experience has emphasised the important role of support services (guidance, learning resources, ICT, recruitment, student finance and so on) in determining the overall quality of the student learning experience. There is also increasing focus on issues of diversity and equalities, to ensure that the needs of all students whatever their race, gender, sexuality, faith, age or whether they are disabled should be fully and explicitly taken into account in quality systems. This also applies to international students whether studying here or overseas. There is also a need to give further consideration to the role of public information in supporting informed choice by applicants, and in preparing students to engage as effectively as possible with their learning.

9 In order to address the above issues, the Council is now issuing new or revised guidance in each of the following areas:

A: Institutional reporting to the Council

B: The nature and scope of institution-led quality review

C: Involvement of learners in quality processes

D: Public information about quality.

Further information

10 Further information about this guidance can be obtained from Dr Bill Harvey, Deputy Director, Learning and Teaching, tel: 0131 313 6513, email: bharvey@sfc.ac.uk or Lesley Sutherland, Assistant Director, Learning and Teaching, tel: 0131 313 6681, email: lsutherland@sfc.ac.uk

A: Guidance on institutional reporting

A1: Baseline report on quality arrangements

Introduction

11 The Council recognises and endorses the important role of the governing body of each institution in quality issues, both in terms of statutory responsibilities and in relation to Council's accountability requirements. However, the Council currently has no documentation which describes for each institution the procedures by which the governing body engages with quality issues and satisfies itself that the institution is meeting its responsibilities for delivering a high quality learning experience to its students.

12 It is a matter for each institution to decide for itself how the governing body discharges its responsibilities, and there is considerable diversity in Scotland as to how this is done. In some institutions, there will be formal (or indeed statutory) divisions of responsibility of relevant matters between bodies such as Court, Senate, and/or Academic Boards. In other cases, there may be formal or de facto processes for delegation of authority either to members of staff, particularly the Principal, and/or to sub-committees of the governing body. The Council wishes to better understand, but in no way to determine, the manner in which each institution operates such arrangements.

13 The Council therefore now asks each HE institution to provide, by 31 December 2008, a summary statement which explains, in the institution's own words, how the governing body discharges its strategic responsibilities for quality - that is, how the governing body satisfies itself that there are appropriate processes in place with regard to quality assurance and enhancement of educational provision. In preparing this statement, it may be helpful to note that the approach to quality which has been developed in Scottish HE over the last few years is based on a multi-dimensional model of quality. These dimensions include internal review, external review, quality enhancement, student representation and engagement, and public information about quality.

14 Production of this summary statement is a 'one-off' activity to describe current arrangements. To minimise the need to create any new documentation, we would encourage institutions to refer where possible to relevant existing institutional documentation which should accompany the statement. As an indicative guide, we suggest that the summary statement would be a few pages in length.

A2: Annual reports from institutions

Introduction

15 Since AY 2003-04, HE institutions have provided the Council with an annual summary of internal and external reviews and this will continue in the cycle beginning in AY 2008-09. This information is very helpful not only as a formal assurance that quality is being maintained, but also as a source of information about sector-wide development issues. The current guidance, which was issued on 28 June 2005 as Annex B of circular HE/26/05, continues to be valid. However, in the light of the Council's recent review of quality, it is appropriate to make modest extensions to the guidance to take account of current thinking.

16 An analysis of these annual summaries will be considered by the relevant Council committee and we will ask QAA to draw on these statements as part of the evidence base for their annual report to Council. With the exception of institutional reports on the outcomes of QAA ELIR review (see section A3 below), the Council does not intend to publish these statements, although under the terms of the Freedom of Information (Scotland) Act 2002 we may be obliged to release information in response to a valid request.

17 The changes will apply to the summary due in September 2009 and all subsequent summaries.

Guidance on the provision of the annual institutional statement of internal review activity

18 The following guidance applies for the provision of statements for AY 2008-09. We envisage that the progress report will be no more than three or four pages long. It should cover all learning provision for the subjects included and should include:

• a comprehensive list of subject areas which were the subject of internal reviews during AY 2008-09
• a comprehensive list of subject areas which were reviewed by other bodies, for example, by professional or statutory bodies (PSBs), during AY 2008-09
• indicate the ways in which support services (such as libraries, IT services and guidance services) were included in review processes, for example by consideration of their impact on the student experience in particular subjects, and/or by separate processes by which support services were reviewed
• the outcomes of subject review, whether carried out internally or by PSB, indicating whether provision was approved and, if not, what actions are being taken to address outstanding issues
• any significant issues relating either to development needs or to the identification of good practice which the institution has identified as a result of these review processes
• the role and nature of student involvement in these internal review processes
• a brief reflective overview which highlights key findings from reviews conducted in AY 2008-09, including comments on areas of strength and issues for further development
• a forward plan or calendar of future reviews for AY 2009-10.

19 The first such statement based on this revised guidance should be sent to the Council by 30 September 2009, and thereafter by 30 September annually. The statement should indicate either that it has been endorsed by the governing body of the institution, or else how it will be so endorsed at a future date.

20 In addition to providing us with this report, we would encourage institutions to consider ways in which they could share information about current issues not only in the annual statement, but also, where appropriate, through ad hoc briefings on a 'no surprises' basis. This might be particularly helpful where there is follow-up action to address any issues arising from an internal/PSB review, but might also deal with other issues which may emerge from time to time.

A3: Reporting on the institution's response to outcomes of QAA ELIR review

21 As in previous years, each institution will engage with QAA one year after publication of the institution's ELIR review report, to consider actions taken to address issues raised by the review. Institutions have an opportunity to prepare a written account of these actions, and as part of the design of the next round of the ELIR process, this account will now be published on QAA's website. The ELIR handbook explains in more detail how the content of this account will be agreed and published.

22 The Council would also wish to receive a report from each institution on its response to the ELIR review, endorsed by the governing body. To minimise the burden on institutions, we propose that the report prepared for publication on QAA's website should also be used for this purpose.

23 Exceptionally, where external review identifies issues of significant concern, the Council may wish, as at present, to require institutions to take urgent action and/or to prepare a detailed action plan to address deficiencies. Given the importance of governance and accountability in such cases, any such action plan should in future include commentary on how the governing body will be involved in implementation and monitoring of the plan.

B: Revised guidance on the nature and scope of institution-led quality review

Introduction

24 Institution-led internal review is an important mechanism in assuring and enhancing the quality of students' educational experience in Scottish HE institutions. This is wholly consistent with the Council's recognition that quality is owned by institutions. It is a matter for each institution to determine how precisely it organises its internal processes for reviewing provision. However, the Council has responsibilities for ensuring that institution-led review is comprehensive and rigorous, and considers the full range of issues which may have a bearing on quality. It is therefore appropriate that the Council should issue broad generic guidance on how such processes should be conducted.

25 The Council has already issued guidance on the characteristics of internal review (Annex B of circular HE/04/03, issued on 23 January 2003). This guidance remains broadly valid and none of the characteristics of internal review described in that guidance will be significantly changed by what is now presented. The revised guidance basically amplifies and extends the previous commentary in some important areas.

26 The main areas of change are as follows:

l some minor updating to take account of more recent developments in areas such as the formation of SFC, the creation of the SCQF partnership, the role of SHEEC and its revised approach to national enhancement themes, and the creation and role of sparqs

• increasing recognition of the role of support services in contributing to the quality of the student experience
• a more central role for student engagement in internal review processes.

Guidance on the characteristics of institution-led quality review

27 Institution-led quality reviews should include the following characteristics.

All provision should be reviewed on a cycle of not more than six years.

28 It is a matter for each institution to decide how to schedule and aggregate its provision. However, good practice would be to ensure that programmes and subjects are aggregated in ways which provide coherence (for example reviewing all programmes in a subject at the same time; reviewing all programmes within a department at the same time). Excessive aggregation would mean that the process cannot examine the 'fine structure' of provision and may not be able to identify specific issues affecting a small number of programmes; large groupings may also become unwieldy if they involve too many members of staff. We suggest that the typical level of aggregation should be at Department level.

29 Although the primary focus of internal review is likely to be on undergraduate degree programmes, we expect institutions to include within the review programme all credit-bearing provision in the subject area(s) being reviewed, including postgraduate awards, CPD, collaborative and overseas provision, supervision of research students, online and distance learning and provision (such as extra-mural courses) which provide only small amounts of credit. It will be a matter for institutions to decide how to aggregate such provision (for example by subject, mode of delivery, or level).

30 There is increasing recognition of the important role of support services (guidance, learning resources, ICT, recruitment, student finance and so on) in determining the overall quality of the student learning experience. Institutions should satisfy themselves that there are appropriate mechanisms in place to facilitate periodic review of the strategic and operational role of support services in relation to their impact on the student experience. It is a matter for each institution to determine how this should be done. Whatever the approach taken, the evidence should be such as to allow the institution to reflect on the contribution of support services to the 'quality culture' within the institution, the ways in which the services engage with students to monitor and improve the quality of services, and the ways in which the services promote high quality learning and continuous quality enhancement.

31 The timescale of six years was intended to provide continuity with the schedule which had been set in 1999. However, institutions may now wish to take advantage of the greater flexibility now available for aggregation and this may lead to a truncation of the cycle length. It would not be good practice to compress reviews into (say) a period of three years, followed by three further years of inactivity. We expect that internal review will inevitably identify a range of developmental issues and there is benefit to the institution from generating such insights on an ongoing basis. We therefore propose that, however the timetable is constructed, there should normally be some form of internal review activity taking place within each academic session.

Reviews should take full account of benchmarks and the code of practice.

32 Benchmark statements provide a useful guide to national expectations about the characteristics of programmes in different subjects. We accept that benchmarks are less helpful in considering curricula and learning outcomes in interdisciplinary programmes and in modular structures which offer wide choice between options. Internal review processes should be designed in such a way as to establish that providers have engaged with relevant benchmarks and are able to demonstrate that programme design and learning outcomes are consistent with relevant benchmarks.

33 QAA's Code of Practice contains a section on programme approval, monitoring and review. ELIR reviewers will look for evidence that this code has been embedded in institutional systems. More generally, the Code contains helpful guidance on a wide range of institutional functions, and internal review processes should be designed in such a way as to effectively monitor the implementation of the Code at the programme or subject level. Significant amounts of provision in Scottish HEIs are accredited by professional and statutory bodies (PSBs). We would encourage institutions to engage with PSBs to explore appropriate ways in which the burden of audit might be further reduced, for example through the use of common documentation, or through joint processes which meet the needs of both internal review and external accreditation. We would also look to institutions to reflect on the outcomes of relevant PSB accreditations within internal reviews. At a wider level, we would encourage institutions to take full advantage of activities (such as the national enhancement themes, the work of SHEEC, the work of the HE Academy and its subject centres, and advice from sparqs) which provide opportunities for reflection on their performance and how it might be usefully compared with that of others.

Reviews should take full account of the Scottish Credit and Qualifications Framework.

34 The Scottish Credit and Qualifications Framework (SCQF) has been identified as a key strategic element in Scotland's education system, and the HE sector has been highly influential in its development. We look to institutions to actively engage with the framework and with the work of the SCQF Partnership.

35 Internal reviews should be designed in order to promote scrutiny and discussion of the provider's approach to SCQF, with an expectation that providers will have adopted a proactive approach aimed at exploiting the flexibility which SCQF provides. This should include consideration of:

• strategy for recognition of prior learning, for example through articulation arrangements with college providers and/or general statements about criteria for entry with advanced standing
• consistency in the allocation of credit and level
• approach to credit-rating of non-standard learning components, for example work placement
• flexible pathways to awards.

Reviews should provide an objective review of provision based on an understanding of national and international good practice. Each review team should include at least one external member with a relevant background.

36 It will be a matter for institutions to determine the composition of review teams and hence to select appropriate external member(s). Institutions will wish to select members who are able to appreciate the specific context in which programmes are presented. However, good practice would be to ensure that review teams are able to bring a range of experience to the process and hence are able to act as 'critical friends' to the institution. Team size and composition must also take account of the range and volume of provision to be reviewed. We suggest that institutions should consider more extensive use of external members, for example one from another Scottish institution and at least one from outside Scotland.

37 We do not expect internal review teams to routinely include members from outside the UK (although, where this can be achieved, it may be valuable and we encourage institutions to actively consider the scope for this option). However, review processes should be designed to include some element of reflection on international good practice, such as a reflective statement from the provider on how their provision compares with similar practice in some other countries. Institutions may wish to consider how they can support such informal 'benchmarking' at a central level.

Internal processes should take full account of student feedback, and include procedures to obtain student views of the provision being reviewed.

38 We expect all institutions to have ongoing processes of obtaining student feedback; the Council is actively considering ways in which it can support institutions to develop good practice in this area. Institutional processes for student feedback will be explicitly considered as part of institutional audit. Internal review should be designed in order to explore the ways in which providers have generated, considered and acted on feedback from their students in the design and operation of their programmes and the organisation of students' learning environment. We also propose, as a separate measure, that each internal review process should gather additional specific information from students as part of the evidence base for reviews. An appropriate methodology would be one which:

• generated holistic evidence about student views of provision and of their learning experience
• differentiated between the views of different categories of students where these are likely to be significant (for example part-time and full-time, junior and senior, entrants from school and entrants from FE, etc.)
• allowed identification of distinctive characteristics of major subsets of provision
• took account of the views of graduates on the relevance of provision for their careers.

39 Each ELIR review team has since 2004 included a student or someone nominated by the student body and this has been a successful and distinctive element of the Scottish quality framework. We now expect institutions to develop and deploy mechanisms to directly involve students in processes of institution-led quality review (that is, student representatives should contribute directly to the review of evidence and to the deliberation of internal review teams) and the revised guidance will now look to institutions to devise appropriate mechanisms to achieve this.

40 Institution-led quality processes should also take account of the increasing diversity of the student body, in terms of race, gender, sexuality, faith, age or whether they are disabled, and of the range of modes and location of delivery. This does not mean that multiple parallel processes of review need be carried out, each addressing distinct groups of students. Rather, it is intended to ensure primarily that internal review processes do not exclude or impede categories of students from engaging with review because of the way in which review is defined or operated. More positively, there is scope to reflect on the opportunities which diversity provides for enriching the learning experience. It is up to each institution to decide how to address these issues, taking account of the specific demographics and characteristics of its own provision.

Internal reviews should consider the effectiveness of annual monitoring arrangements and follow-up action for programmes covered by the review.

41 We expect each institution to operate systems of annual monitoring across the full range of provision; this is likely to include not only student survey data, but also performance data on recruitment, progression and achievement. Good practice in such monitoring would be to include mechanisms which allow some benchmarking of provision against other areas of the institution's activities, as well as equivalent provision elsewhere. Institutions should also make appropriate use of external reference points including external examiners' reports. A key element of quality assurance and enhancement is the extent to which the outcomes of such monitoring are scrutinised and acted on in order to address shortcomings and spread good practice. Internal review processes should be designed in a way which allows reflection on the effectiveness of monitoring and follow-up, for example by following audit trails of previous monitoring outcomes and considering processes in place to reflect and act on feedback and performance data.

42 At the institutional level, the Council expects institutions to reflect on strategic issues arising from regular monitoring, and to make use of this information in its strategy for continuous quality improvement. This will be explicitly considered during institutional audit; however, institutions may wish to consider how they can best design internal review processes in order to facilitate institution-level reflection on the outcomes of monitoring. Good practice would be to have reporting procedures at the programme, subject or department level which passed on relevant issues for consideration at institutional level. Internal review processes should be designed to allow constructive reflection on the effectiveness of these procedures.

Assurance and enhancement

43 The primary purpose of internal reviews is to provide assurances about the quality and standards of provision. Since there will no longer be a programme of external subject reviews, it is vital that internal reviews provide robust, comprehensive and credible evidence that standards and quality of provision in Scottish HEIs are being maintained. However, the Council's new approach gives a central role to quality enhancement, and we would therefore encourage institutions to develop internal review processes which also:

• promote dialogue on areas in which quality might be improved
• identify good practice for dissemination within the institution and beyond (such as quality enhancement engagements)
• encourage and support providers' efforts to reflect critically on their practice.

C: Involvement of learners in institutional quality processes

Introduction

44 The Council issued guidance to the university sector in 2003 on student representation in quality processes (see Annex C of circular HE/04/03, issued on 23 January 2003 - www.sfc.ac.uk/information/info_circulars/shefc/2003/he0403/ he0403.html). This set expectations that students would be involved in all processes relating to quality assurance and enhancement, and that students should be represented as widely as possible in each institution's consultative and decision-making forums.

Next steps

45 The 2003 guidance remains valid, but it is now clear that it needs further development. For example, the 2003 guidance does not address contemporary expectations on diversity and equalities across the student body. Also, the sparqs organisation was not in existence when the 2003 guidance was issued, and therefore its distinctive role was not recognised in the guidance. It is also helpful to acknowledge current thinking about the meaning of 'student engagement', which now extends well beyond the specific issue of engagement with quality processes, into broader discussion about students' engagement with their own learning.

46 As a result of these trends, we think the emphasis in future will not be on the particular mechanisms by which students interact with their institution, but on the quality and effectiveness of these interactions, and the potential to amplify the 'student voice'. Although there are good signs of progress across the sector, more work is needed to flesh out concepts such as 'engaging with learning' and to clarify their implications. The HE QWG and its partner organisations will take forward this work.

47 These are areas where there is a great deal of current development which we are confident will lead to positive outcomes, but it is too soon to draw conclusions about their implications. The HE QWG has therefore concluded that it would be premature to issue additional or revised guidance on student representation and engagement at this stage. However, we encourage institutions to reflect on the implications of diversity, on the opportunities presented by sparqs, and on the development of broader concepts of student engagement with learning, as means of continuing to enhance their current approaches.

D: Guidance on public information about quality

Introduction

48 The Council issued guidance to the university sector in 2003 and 2004 on public information about quality, see - www.sfc.ac.uk/information/info_circulars/shefc/2003/he1903/he1903.html and www.sfc.ac.uk/information/info_circulars/shefc/2004/he1804/he1804.html

This guidance invited institutions to reflect on the information needs of different user groups, and stressed the importance of supporting learners in making informed choices.

Next steps

49 The 2003 and 2004 guidance remains valid, but it is now clear that it needs further development. For example, the current guidance does not take account of more recent thinking on learner choice, the evidence available from the On Track longitudinal surveys, and policy developments including the introduction of the National Student Survey in England. The emphasis within the JQRG report on student engagement also highlighted further issues relating to public information, particularly in relation to preparing potential learners to engage more effectively in their intended learning experience.

50 Further work is needed to consider the implications of these developments and this will be taken forward by the HE QWG. The HE QWG has therefore concluded that it would be premature to issue additional or revised guidance on public information at this stage. However, we encourage institutions to continue to reflect on the ways in which they present information about provision to students, employers and other stakeholders, and on the scope for enhancing the effectiveness of such information.

Annex 5 - European Association for Quality Assuraance in Higher Education (ENQA) 2007

Standards and Guidelines for Quality Assurance in the European Higher Education Area, 2nd edition

The standards and guidelines for quality assurance in the European higher education area comprise three parts, of which Parts 1 and 2 are the most relevant to the ELIR process:

• Part 1: European standards and guidelines for internal quality assurance within higher education institutions
• Part 2: European standards and guidelines for the external quality assurance of higher education.

The following text is an extract from the 2007 edition of the Standards and Guidelines, and the numbering of the headings is as contained in the original document.

2.3 Part 1: European standards and guidelines for internal quality assurance within higher education institutions

2.3.1 Policy and procedures for quality assurance

Standard:

Institutions should have a policy and associated procedures for the assurance of the quality and standards of their programmes and awards. They should also commit themselves explicitly to the development of a culture which recognises the importance of quality, and quality assurance, in their work. To achieve this, institutions should develop and implement a strategy for the continuous enhancement of quality.

The strategy, policy and procedures should have a formal status and be publicly available. They should also include a role for students and other stakeholders.

Guidelines:

Formal policies and procedures provide a framework within which higher education institutions can develop and monitor the effectiveness of their quality assurance systems. They also help to provide public confidence in institutional autonomy. Policies contain the statements of intentions and the principal means by which these will be achieved. Procedural guidance can give more detailed information about the ways in which the policy is implemented and provides a useful reference point for those who need to know about the practical aspects of carrying out the procedures.

The policy statement is expected to include:

• the relationship between teaching and research in the institution
• the institution's strategy for quality and standards
• the organisation of the quality assurance system
• the responsibilities of departments, schools, faculties and other organisational units and individuals for the assurance of quality
• the involvement of students in quality assurance
• the ways in which the policy is implemented, monitored and revised.

The realisation of the EHEA depends crucially on a commitment at all levels of an institution to ensuring that its programmes have clear and explicit intended outcomes; that its staff are ready, willing and able to provide teaching and learner support that will help its students achieve those outcomes; and that there is full, timely and tangible recognition of the contribution to its work by those of its staff who demonstrate particular excellence, expertise and dedication. All higher education institutions should aspire to improve and enhance the education they offer their students.

2.3.2 Approval, monitoring and periodic review of programmes and awards

Standard:

Institutions should have formal mechanisms for the approval, periodic review and monitoring of their programmes and awards.

Guidelines:

The confidence of students and other stakeholders in higher education is more likely to be established and maintained through effective quality assurance activities which ensure that programmes are well-designed, regularly monitored and periodically reviewed, thereby securing their continuing relevance and currency. The quality assurance of programmes and awards are expected to include:

• development and publication of explicit intended learning outcomes
• careful attention to curriculum and programme design and content
• specific needs of different modes of delivery (eg full-time, part-time, distance-learning, e-learning) and types of higher education (eg academic, vocational, professional)
• availability of appropriate learning resources
• formal programme approval procedures by a body other than that teaching the programme
• monitoring of the progress and achievements of students
• regular periodic reviews of programmes (including external panel members)
• regular feedback from employers, labour market representatives and other relevant organisations
• participation of students in quality assurance activities.

2.3.3 Assessment of students

Standard:

Students should be assessed using published criteria, regulations and procedures which are applied consistently.

Guidelines:

The assessment of students is one of the most important elements of higher education. The outcomes of assessment have a profound effect on students' future careers. It is therefore important that assessment is carried out professionally at all times and takes into account the extensive knowledge which exists about testing and examination processes. Assessment also provides valuable information for institutions about the effectiveness of teaching and learners' support.

Student assessment procedures are expected to:

• be designed to measure the achievement of the intended learning outcomes and other programme objectives
• be appropriate for their purpose, whether diagnostic, formative or summative
• have clear and published criteria for marking
• be undertaken by people who understand the role of assessment in the progression of students towards the achievement of the knowledge and skills associated with their intended qualification
• where possible, not rely on the judgements of single examiners
• take account of all the possible consequences of examination regulations
• have clear regulations covering student absence, illness and other mitigating circumstances
• ensure that assessments are conducted securely in accordance with the institution's stated procedures
• be subject to administrative verification checks to ensure the accuracy of the procedures.

In addition, students should be clearly informed about the assessment strategy being used for their programme, what examinations or other assessment methods they will be subject to, what will be expected of them, and the criteria that will be applied to the assessment of their performance.

2.3.4 Quality assurance of teaching staff

Standard:

Institutions should have ways of satisfying themselves that staff involved with the teaching of students are qualified and competent to do so. They should be available to those undertaking external reviews, and commented upon in reports.

Guidelines:

Teachers are the single most important learning resource available to most students. It is important that those who teach have a full knowledge and understanding of the subject they are teaching, have the necessary skills and experience to transmit their knowledge and understanding effectively to students in a range of teaching contexts, and can access feedback on their own performance. Institutions should ensure that their staff recruitment and appointment procedures include a means of making certain that all new staff have at least the minimum necessary level of competence. Teaching staff should be given opportunities to develop and extend their teaching capacity and should be encouraged to value their skills. Institutions should provide poor teachers with opportunities to improve their skills to an acceptable level and should have the means to remove them from their teaching duties if they continue to be demonstrably ineffective.

2.3.5 Learning resources and student support

Standard:

Institutions should ensure that the resources available for the support of student learning are adequate and appropriate for each programme offered.

Guidelines:

In addition to their teachers, students rely on a range of resources to assist their learning. These vary from physical resources such as libraries or computing facilities to human support in the form of tutors, counsellors, and other advisers. Learning resources and other support mechanisms should be readily accessible to students, designed with their needs in mind and responsive to feedback from those who use the services provided. Institutions should routinely monitor, review and improve the effectiveness of the support services available to their students.

2.3.6 Information systems

Standard:

Institutions should ensure that they collect, analyse and use relevant information for the effective management of their programmes of study and other activities.

Guidelines:

Institutional self-knowledge is the starting point for effective quality assurance. It is important that institutions have the means of collecting and analysing information about their own activities. Without this, they will not know what is working well and what needs attention, or the results of innovatory practices.

The quality-related information systems required by individual institutions will depend to some extent on local circumstances, but it is at least expected to cover:

• student progression and success rates
• employability of graduates
• students' satisfaction with their programmes
• effectiveness of teachers
• profile of the student population
• learning resources available and their costs
• the institution's own key performance indicators.

There is also value in institutions comparing themselves with other similar organisations within the EHEA and beyond. This allows them to extend the range of their self-knowledge and to access possible ways of improving their own performance.

2.3.7 Public information

Standard:

Institutions should regularly publish up-to-date, impartial and objective information, both quantitative and qualitative, about the programmes and awards they are offering.

Guidelines:

In fulfilment of their public role, higher education institutions have a responsibility to provide information about the programmes they are offering, the intended learning outcomes of these, the qualifications they award, the teaching, learning and assessment procedures used, and the learning opportunities available to their students. Published information might also include the views and employment destinations of past students and the profile of the current student population. This information should be accurate, impartial, objective and readily accessible and should not be used simply as a marketing opportunity. The institution should verify that it meets its own expectations in respect of impartiality and objectivity.

2.4 Part 2: European standards and guidelines for the external quality assurance of higher education

2.4.1 Use of internal quality assurance procedures

Standard:

External quality assurance procedures should take into account the effectiveness of the internal quality assurance processes described in Part 1 of the European Standards and Guidelines.

Guidelines:

The standards for internal quality assurance contained in Part 1 provide a valuable basis for the external quality assessment process. It is important that the institutions' own internal policies and procedures are carefully evaluated in the course of external procedures, to determine the extent to which the standards are being met.

If higher education institutions are to be able to demonstrate the effectiveness of their own internal quality assurance processes, and if those processes properly assure quality and standards, then external processes might be less intensive than otherwise.

2.4.2 Development of external quality assurance processes

Standard:

The aims and objectives of quality assurance processes should be determined, before the processes themselves are developed, by all those responsible (including higher education institutions), and should be published with a description of the procedures to be used.

Guidelines:

In order to ensure clarity of purpose and transparency of procedures, external quality assurance methods should be designed and developed through a process involving key stakeholders, including higher education institutions. The procedures that are finally agreed should be published and should contain explicit statements of the aims and objectives of the processes as well as a description of the procedures to be used. As external quality assurance makes demands on the institutions involved, a preliminary impact assessment should be undertaken to ensure that the procedures to be adopted are appropriate and do not interfere more than necessary with the normal work of higher education institutions.

2.4.3 Criteria for decisions

Standard:

Any formal decisions made as a result of an external quality assurance activity should be based on explicit published criteria that are applied consistently.

Guidelines:

Formal decisions made by quality assurance agencies have a significant impact on the institutions and programmes that are judged. In the interests of equity and reliability, decisions should be based on published criteria and interpreted in a consistent manner. Conclusions should be based on recorded evidence and agencies should have in place ways of moderating conclusions, if necessary.

2.4.4 Processes fit for purpose

Standard:

All external quality assurance processes should be designed specifically to ensure their fitness to achieve the aims and objectives set for them.

Guidelines:

Quality assurance agencies within the EHEA undertake different external processes for different purposes and in different ways. It is of the first importance that agencies should operate procedures which are fit for their own defined and published purposes. Experience has shown, however, that there are some widely-used elements of external review processes which not only help to ensure their validity, reliability and usefulness, but also provide a basis for the European dimension to quality assurance.

Amongst these elements the following are particularly noteworthy:

• insistence that the experts undertaking the external quality assurance activity have appropriate skills and are competent to perform their task
• the exercise of care in the selection of experts
• the provision of appropriate briefing or training for experts
• the use of international experts
• participation of students
• ensuring that the review procedures used are sufficient to provide adequate evidence to support the findings and conclusions reached
• the use of the self-evaluation/site visit/draft report/published report/follow-up model of review
• recognition of the importance of institutional improvement and enhancement policies as a fundamental element in the assurance of quality.

2.4.5 Reporting

Standard:

Reports should be published and should be written in a style which is clear and readily accessible to its intended readership. Any decisions, commendations or recommendations contained in reports should be easy for a reader to find.

Guidelines:

In order to ensure maximum benefit from external quality assurance processes, it is important that reports should meet the identified needs of the intended readership. Reports are sometimes intended for different readership groups and this will require careful attention to structure, content, style and tone. In general, reports should be structured to cover description, analysis (including relevant evidence), conclusions, commendations, and recommendations. There should be sufficient preliminary explanation to enable a lay reader to understand the purposes of the review, its form, and the criteria used in making decisions. Key findings, conclusions and recommendations should be easily locatable by readers. Reports should be published in a readily accessible form and there should be opportunities for readers and users of the reports (both within the relevant institution and outside it) to comment on their usefulness.

2.4.6 Follow-up procedures

Standard:

Quality assurance processes which contain recommendations for action or which require a subsequent action plan, should have a predetermined follow-up procedure which is implemented consistently.

Guidelines:

Quality assurance is not principally about individual external scrutiny events: it should be about continuously trying to do a better job. External quality assurance does not end with the publication of the report and should include a structured follow-up procedure to ensure that recommendations are dealt with appropriately and any required action plans drawn up and implemented. This may involve further meetings with institutional or programme representatives. The objective is to ensure that areas identified for improvement are dealt with speedily and that further enhancement is encouraged.

2.4.7 Periodic reviews

Standard:

External quality assurance of institutions and/or programmes should be undertaken on a cyclical basis. The length of the cycle and the review procedures to be used should be clearly defined and published in advance.

Guidelines:

Quality assurance is not a static but a dynamic process. It should be continuous and not 'once in a lifetime'. It does not end with the first review or with the completion of the formal follow-up procedure. It has to be periodically renewed. Subsequent external reviews should take into account progress that has been made since the previous event. The process to be used in all external reviews should be clearly defined by the external quality assurance agency and its demands on institutions should not be greater than are necessary for the achievement of its objectives.

2.4.8 System-wide analyses

Standard:

Quality assurance agencies should produce from time to time summary reports describing and analysing the general findings of their reviews, evaluations, assessments etc.

Guidelines:

All external quality assurance agencies collect a wealth of information about individual programmes and/or institutions and this provides material for structured analyses across whole higher education systems. Such analyses can provide very useful information about developments, trends, emerging good practice and areas of persistent difficulty or weakness and can become useful tools for policy development and quality enhancement. Agencies should consider including a research and development function within their activities, to help them extract maximum benefit from their work.

Annex 6 - QAA’s values and standards

Values

1 QAA's work is underpinned by core values.

The importance of higher education

2 QAA acknowledges the academic vocation and the importance of higher education in the lives of citizens. We respect the diversity and autonomy of higher education providers, and value the high regard in which UK higher education is held internationally.

The entitlements of learners

3 Students are entitled to a higher education that has value, with academic standards that reflect national expectations and awards that meet published specifications, and to fair and equitable treatment in all aspects of their studies. QAA values the participation of students in the assurance of academic standards and quality.

The significance of the responsibilities of the providers of higher education

4 The providers of higher education have the primary responsibility for protecting academic standards and quality; QAA works with them to meet that responsibility. We depend on help from many colleagues in higher education and the professions who work with us, and we value their contribution to our peer processes.

The validity of the public interest in higher education

5 The public invests a lot in higher education. It has a legitimate expectation that the standards of higher education qualifications are maintained and that the quality of provision supports learners to achieve the necessary standards.

Standards

6 QAA sets itself high standards in all its work.

Integrity

7 We aim to show impartiality, fairness, independence and honesty in our work and to base our judgements on evidence.

Professionalism

8 We aim to achieve high professional standards and provide a cost-effective service. We aim to get it right first time. Where we get something wrong, we will acknowledge it, correct it and learn from it.

Accountability

9 QAA is accountable to its subscribers - the universities and higher education colleges - and to a wide range of other stakeholders. We aim to demonstrate that we use our resources to good effect.

Openness

10 We aim to be open and approachable, and to be transparent in our work and methods. We aim to communicate in a clear, consistent and accessible way.

---

¹ Enhancing the quality of teaching and learning in Scottish Universities 2007. Final report of the external evaluation quoted in SFC circular SFC/11/2007 'Evaluation of the higher education quality enhancement framework: final report'.

² Final report from the Joint Quality Review Group to Council, SFC August 2007.

³ Standards and Guidelines for Quality Assurance in the European Higher Education Area, ENQA, Finland, 2007.

⁴ Guidelines of Good Practice 2007, INQAAHE, Dublin 2007.

⁵ Learning to improve: quality approaches for lifelong learning, Scottish Executive, Edinburgh 2005.

⁶ The Crerar Review: The report of the independent review of regulation, audit, inspection and complaints handling of public services in Scotland, Scottish Government, Edinburgh 2007.

⁷ Additions suggested during initial discussions with SHEEC, ELIR Steering Committee and US TQF during 2007-08. The vision originated from a national debate about the key characteristics of a high quality sector which took place in 2000-01.

---

ISBN 978 1 84482 825 8