section photograph

Handbook for enhancement-led institutional review: Scotland
(Second edition)

Draft handbook for consultation
March 2008

Preface to the second edition

Introduction

1 Enhancement-led institutional review is one component of the Quality Enhancement Framework (QEF), a radical approach to quality assurance and enhancement in higher education introduced in Scotland in 2003. The QEF emerged out of the experiences of more than a decade of audit and review. According to the final report of the external evaluation1, the QEF was ‘based on the pooling of expertise and knowledge of literatures on teaching, learning, change and quality from a wide range of sources, all shot through with a commitment to enhancing students’ experiences as learners’. In other words, the QEF brought right to the fore the simple and powerful idea that the purpose of quality systems in higher education is to improve student experiences and, consequently, their learning. In the second edition, the ELIR Handbook remains firmly focused on this target.

2 In the first edition of the Handbook, it was stressed that this powerful new focus on the enhancement of the student learning experience was not at the expense of the assurance of quality and the standards of awards. Rather, it was emphasised that enhancement built on, reinforced and made more powerful the basic processes of assurance: enhancement included assurance. The evidence of the reviews carried out in the previous version of the ELIR method clearly supports the robustness of this approach which is taken forward in the second edition of the Handbook.

3 The second edition integrates ELIR more fully with other elements of the QEF which are also progressively becoming integrated more substantively into the fabric of Scottish higher education. The main elements of the QEF in addition to ELIR are:

  • the Enhancement Themes
  • institution-led review at the subject level
  • the engagement of students in quality management including the support provided through the national independent development service, Student Participation in Quality Scotland (sparqs)
  • the provision by institutions of an agreed set of public information.

4 The Enhancement Themes were a novel development within the QEF. They have now become well established under the direction of the Scottish Higher Education Enhancement Committee (SHEEC). The remit of SHEEC relates both to the strategic management of the rolling plan for the Enhancement Themes, and also to supporting the development of institutional quality cultures. The outcomes from the work of SHEEC now represent a valuable resource for the sector in the management of quality drawing widely as they do on both national and international practice, evidence and experience. There was, and is, no expectation that the outcomes of the Enhancement Themes will lead to compliance with specific approaches. Rather, the outcomes now provide a rich repertoire of reference points for institutions and their students to draw on in developing their own policies and practice within the context of their own timeframes and priorities. The Enhancement Themes, therefore, now increasingly provide an important context for discussion within ELIR, a feature that will be reflected in ELIR reviewer training and continuing development (see below, paragraph 29). The Enhancement Themes are also increasingly referred to within institution-led reviews.

5 In general, ELIR and the wider QEF represented a radical departure from previous approaches. The individual elements of the QEF have matured over the first version of the ELIR method, and inter-relationships strengthened. The preparation of the second edition of the ELIR handbook provides the opportunity to build further on this solid foundation.

Continuity and change: continuity

6 The second edition of the ELIR Handbook reflects a clear development in approach from the first edition, building on its strengths, refining areas where the need for change has been demonstrated, and sharpening the focus on the enhancement of the student learning experience. The final report of the external evaluation of the QEF (op cit) opens with the statement, ‘the approach to quality that we review here is ambitious, distinctive and, so far, successful’. The revised approach to ELIR described in this second edition of the Handbook represents continuity, building on that success and developing from the experience.

7 The final external evaluation report (op cit) identifies the key drivers of the success of the QEF.

‘Distinctively, the quality enhancement framework which we review here, is a commitment to:

  • Students and the continuing enhancement of their learning in higher education.
  • Partnerships between agencies…higher education institutions…and other stakeholders, most distinctively seen in the active involvement of students and student bodies in the QEF.
  • A theory of educational change that placed far more weight on consensual approaches than on the more coercive stances embedded in some quality assurance regimes. The approach emerged from serious discussion and thinking.
  • A culture shift – away from top-down compliance-inducing processes to participative and critical supported self-evaluation; away from audit and towards improvement; away form ruffling the surface of higher education practices and towards permeating the system with practices compatible with the QEF; away from mechanistic models based solely on inputs and outcomes and towards more sensitive other forms of evidence of cultural change, while maintaining rigour and challenge.
  • Reflexivity, in the sense of exposing QEF itself to evaluation from the very beginning. Evaluation was valued for the contribution it could make to the enhancement of the QEF itself, even as the QEF was working out what it would mean to have a higher education system committed to quality enhancement.’

Importantly, the final evaluation report also stresses that a key success factor is commitment to the long term, recognising that the development and nourishment of quality cultures does not happen overnight: the dynamic of real enhancement depends to a significant extent on operating within a relatively stable, predictable policy environment.  

8 At the heart of the QEF, to date, has been the successful operation of professional partnerships with Universities Scotland, with the individual autonomous institutions, with the Scottish Funding Council (SFC) and with students and their associated bodies, importantly NUS (Scotland) and sparqs. One vital piece of evidence on the importance of building further on the partnership model comes again from the final external evaluation report (op cit) which indicates:

‘not surprisingly, we have found evidence of the persistence of behaviours redolent of the displaced quality assurance regime. We have also noticed, though, that when it comes to enhancement-led institutional review (ELIR), institutions are increasingly willing to lay out areas of imperfect practice and publicly consider ways in which they could improve on them in coming years. Perhaps the shift from the concealing behaviours associated with the previous quality assurance regimes operating in UK HE in the 1990’s towards – and let it be clear that this is a direction of travel – disclosure of areas for improvement is the biggest culture shift in thinking and the most distinctive feature of Scotland’s fresh thinking about quality.’

9 In general the evidence confirms that the founding principles of the enhancement-led approach remain fit for purpose and that the first edition of the ELIR Handbook provides a sound basis from which to develop. This evidence is drawn from a number of sources including: the outcomes of the external evaluation referred to above; the outcomes of QAA Scotland’s own monitoring and evaluation of each element of ELIR; and the outcomes of the evaluation carried out by the SFC during 2006-07 (involving extensive consultations across the sector).

10 The second edition of the Handbook takes forward five key themes from the first edition:

  • Enhancement includes assurance. The overarching theme of ELIR continues to be the strategic management of academic standards and the enhancement of the quality of the student learning experience. Section 1 of this Handbook outlines how enhancement continues to be interpreted in the review context. However, it is important to emphasise at the outset that a key element of an effective enhancement strategy involves knowing where one is starting from, ie how does the institution assure itself that standards and quality are being appropriately maintained. This can then be linked to the related element of the management of enhancement – improving the effectiveness of student learning, seeking to learn from current activities, reference points and good practice, and to make the most effective use of resources to support engagement and high quality learning. Assurance and enhancement are inextricably linked within the quality cultures of institutions.
  • Looking forward. The focus within the ELIR process will continue to be on how an institution learns from the past in order to inform the future.
  • Enhancement and risk. Enhancement is the result of change and innovation which will frequently involve risk. Institutions are expected to manage this risk in a way that provides reasonable safeguards for current students. The review process will continue to recognise and support effective risk management and adopt a supportive and not punitive role in this context.
  • Supporting diversity. The review process will continue to support the rich diversity of higher education institutions in Scotland. While there are commonalities of purpose, each higher education institution in Scotland has its own unique mission and will seek to meet the needs of its own particular students in its own particular ways. The quality cultures of each institution will therefore have its own individual characteristics, and the ELIR process will consequently engage with the enhancement of the learning experiences of students in the context of their own institution.
  • The UK and international context. Institutions in Scotland operate and compete in a global environment. The ELIR process will support institutions in this context. Key outcomes from ELIR will be directly comparable with outcomes from related processes elsewhere in the UK. Some of the reference points for quality and standards used by institutions and ELIR will be common across the UK for example the Academic Infrastructure. In general, institutional enhancement strategies, ELIR and related elements in the overall enhancement model will draw on good practice, not only across the UK, but internationally. In addition, ELIR and its outcomes will be used proactively to promote the high standing of Scottish higher education internationally.

Continuity and change: change

11 While this edition of the Handbook does not embody revolutionary change, it does nonetheless represent a significant evolution from the first edition, reflecting, to a large extent, refinements rather than fundamental change. In some areas this reflects changes in the external environment. Two of the key publications to which the second edition has responded are the Report of the Joint Quality Review Group (SFC 2007)2 and the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ENQA 2007)3 adopted by European ministers as the definitive document on quality assurance within the Bologna Process. The development of the second edition has also been informed by the International Network of Quality Assurance Agencies (INQAAHE) Guidelines of Good Practice (INQAAHE 2007)4, Learning to improve: quality approaches to lifelong learning (Scottish Executive 2005)5, and the principles embodied in the Crerar Review (Scottish Government 2007)6. In addition to responding to these reports, as other elements of the QEF have matured new opportunities to take forward further inter-relationships have developed. As indicated above, the outcomes from the Enhancement Themes are now providing a rich repertoire of international practice, evidence and experience for dialogue in the context of both ELIR and institution-led reviews. There is also the potential for the SHEEC project on Indicators of Enhancement being undertaken in 2007-08 to provide additional useful reference points for use in relation to ELIR. The outcomes from this project may also have a useful role to play in the context of institution-led reviews which in themselves have an important interplay with ELIR. Sparqs is also proving to be an increasingly important focus for joint work with students, including the training of students who are reviewers.

The report of the Joint Quality Review Group (JQRG)

12 The second edition of the Handbook reflects the outcomes of the JQRG report for the higher education context. Many elements highlighted in the JQRG report were already well established within the ELIR philosophy and practice outlined above. The JQRG report reinforces the characteristics of a high quality sector explicit in the first edition of the Handbook. This is retained with some modification (see paragraph 30). The JQRG report identifies three fundamental principles that ‘should underpin every aspect of our quality systems, namely: high quality learning across all provision; student engagement; and, a culture of quality and continuous improvement’. These principles are embodied in this edition of the Handbook.

13 In general, the principle of quality culture has become the overarching principle underpinning not only ELIR but other elements within the QEF. The approach of ELIR from the outset has been designed to support and facilitate the development and nourishment of quality cultures in Scotland. Indeed, the final report from the external evaluators indicated early evidence of the emergence of the ‘green shoots’ of quality cultures. In preparing the second edition of the Handbook we have been conscious of the need to support the continuing development of reflective institutions (and faculties, schools and departments) on which quality cultures depend. The processes around the development and use of the Reflective Analysis; the sharing of case study material across the sector; the highlighting of the role of the Enhancement Themes; the annual discussions; and further development of the interactive approach to review visits, are all ways in which the second edition of the Handbook reflects this important, overarching principle.

14 The second edition of the Handbook also takes forward the key concepts of student engagement and high-quality learning. Student engagement is developed in two related contexts: engagement with quality processes and students’ engagement with their learning. Student engagement in the former sense has, from the outset, been central to ELIR and the QEF, and developments in this area were very prominent in the previous version of the ELIR method. Fundamental to this has been the role of student reviewers within ELIR teams as well as effective student engagement with internal quality and representative structures and processes. This will continue. It is, however, increasingly important that ELIR should support student engagement in the latter sense: students’ engagement with their learning. The second edition of the Handbook, therefore, contains more explicit reference to student engagement with their learning as outlined below.

15 Similarly, the JQRG emphasis on high-quality learning is taken forward within the concept of quality culture. High-quality learning in higher education results, in large part, from learning that effectively engages learners in the process of learning, and we have therefore taken this element forward jointly with consideration of student engagement. In the preparation of the second edition of the Handbook, the following aspects of institutional quality cultures have been particularly emphasised as contributing to effective student engagement and high quality learning in higher education:

  • the support of students as active participants, engaged in the creation of their knowledge and understanding, and taking responsibility for their own learning
  • learners being engaged through effective guidance and support mechanisms in making informed choices in relation to their learning, curriculum and future career and personal development paths
  • learners being engaged in decision-making processes relating to the curriculum and learning and support strategies and provision
  • learners being engaged in the provision of feedback, perspectives and insights. 

These aspects have impacted, in particular, on: the development of the Reflective Analysis; the training of reviewers; the ELIR visit; and the ELIR report. Indeed, the second edition of the Handbook indicates that the first substantive section of the ELIR report will address the effectiveness of student engagement.

16 In addition, high-quality learning is also defined in part by the achievement of appropriate outcomes: appropriate in relation to the academic standards of awards and to students’ future employment and their participation in civic life,  not only following course completion, but as a preparation for lifelong learning and career development. The second edition of the Handbook maintains a clear and explicit focus on the outcomes achieved and, in particular, on the academic standards of awards.

17 One of the elements fundamental to the support of high-quality learning and effective student engagement is the professionalism of staff responsible for designing and supporting the student learning experience. The second edition of the Handbook therefore identifies a more explicit focus on staff support and development in the context of institutional management of the student learning experience.

18 The second edition continues the focus, emphasised in the JQRG report, on the quality of the experience of all students irrespective of race, gender, sexuality, faith, age or disability. ELIR highlights the importance of the experience of all students, both on and off-campus,­­­ in an environment which supports and promotes equality and diversity and the fruitful interaction of all students, both domestic and international.

19 Other aspects of the JQRG report which have been taken forward explicitly in the second edition of the Handbook include:

  • Reporting: The JQRG report acknowledges the limited role that external review plays in learner choice, and that clarity on this should result in reports being designed to meet the needs of key audiences – the institutions, SFC, and public stakeholders. There will in future be two reports: the definitive report written largely for the institution (with little description and largely analytical) and only available electronically; and a short summary report written in a style appropriate for the lay person and available both in print and electronically. The institution’s ‘year-on’ response to its ELIR report will be published electronically by QAA alongside the original report.
  • Judgements: The JQRG report concluded that reports (in both the college and university sectors) should conclude with judgements ‘summarised using a three point confidence rating scale’. This was the general approach adopted in the first edition of the ELIR Handbook and is continued. The wording of the overarching judgements will undergo a marginal change from the first edition, in part to maintain the same wording as the review reports relating to higher education institutions elsewhere in the UK, and in part reflecting feedback received during the previous version of the ELIR method.
  • Collaborative activity including international dimensions: In general,  all the matters raised in the JQRG report with regard to international matters have been responded to and are addressed more fully below. In addition, there is a more explicit focus within ELIR for dealing with domestic as well as international collaborative activity.

International dimensions

20 As indicated above, the second edition of the Handbook provides a focus on all students, including a focus on the experience of international students studying on-campus in Scotland, and the nature of the interaction of overseas with domestic students.

21 The higher education qualifications framework in Scotland (part of the SCQF), one of the important reference points used by institutions and ELIR in maintaining academic standards, has now been formally recognised within the Bologna Process. In general, the ELIR method has been revised in line with international good practice in the management of quality in higher education. The ‘Standards and Guidelines for Quality Assurance in the European Higher Education Area’, central to the Bologna process, have been fully embraced as have the International Network of Quality Assurance Agencies in Higher Education (INQAAHE) ‘Guidelines of Good Practice’.

22 The international dimension is extended by including for the first time, in each ELIR team, a reviewer drawn from outside the UK. By adding an international reviewer to all ELIR teams, the range of experience and expertise is extended, and both institutions and the ELIR process will benefit from this wider global perspective. The international reviewer will bring international perspectives on quality assurance and enhancement, and will generally be in a position to draw the enhancement and assurance discussions in ELIR on a wider canvas. In addition to the direct benefits, international reviewers will facilitate dissemination abroad of Scottish higher education quality and standards. In addition, international reviewers may be in a position to contribute to wider elements within the QEF, in particular the Enhancement Themes. To get maximum benefit, it is important that international reviewers are trained, and drawn from institutions/organisations and countries appropriate to the diverse higher education institutions in Scotland. As detailed in later sections, institutions will therefore have significant influence in nominating the international reviewers.

ELIR Coordinating Reviewer

23 The role of the review secretary has been replaced by the new role of ELIR Coordinating Reviewer (ECR). Monitoring and evaluation during the previous version of the ELIR method highlighted the importance of the role played by the review secretaries. It has also been apparent that the potential contribution of these senior academic administrators to the process has been artificially constrained by the current definition of the role. In addition, the monitoring and evaluation of the report production processes, together with changes being made to the reports themselves, have highlighted the need for a different contribution to reports from the ELIR team in general, and from the review secretary in particular.

The QEF

24 A fundamental backdrop to the revision of ELIR has been the developments in the other elements of the QEF and their inter-relationships with ELIR. The Enhancement Themes, which continue to mature and reflect significant developments, have an important relationship with ELIR. As indicated above, the Enhancement Themes now provide a rich repertoire of reference points for institutions to draw on in developing their own policies and practice. Institution-led reviews also have a vital role to play in relation to the ELIR process, and continue to form an important focus for the annual discussions. ELIR will continue to depend on active engagement with students and its continued effective delivery will depend to a very significant extent on close working with NUS (Scotland), sparqs and other student bodies.

Summary

25 The revised ELIR method has been prepared through wide consultative processes to practise what ELIR preaches: to learn from the evidence of the past to improve the future, informed by national and international benchmarks. The evidence is that, in general, the introduction of the QEF has been successful. The second edition of the Handbook is intended to build on that success and learn from its relative failures. The changes foreshadowed above and detailed in subsequent sections are the result of shared reflection on the evidence and related material, and a commitment to maintaining ELIR and the QEF more generally as an effective framework for supporting the quality culture of Scottish higher education and its national and international reputation.

Section 1: Interpreting the enhancement focus enhancement-led institutional review

Defining enhancement within ELIR

26 For the purposes of ELIR, enhancement is defined as taking deliberate steps to bring about improvement in the effectiveness of the learning experiences of students.

  • The deliberate steps at an institutional level will be strategic, designed to manage enhancement in a planned way. Enhancement does not necessarily imply the application of additional resources. Enhancement strategies will seek to optimise the deployment of resources in the development of effective student learning.
  • Improvement. The first edition of the Handbook employed the commonly used phrase continuous improvement, referring to the ongoing nature of enhancement strategies, ie the notion that no matter where one is starting from, it is always possible to seek improvement. Although, in general terms, this remains the case, the experience of the previous ELIR method was that the emphasis on ‘continuous’ improvement could artificially provide too strong a focus on the ongoing, relatively marginal changes in day-to-day practice. While in a quality culture it is clearly important that this continues, an important focus of ELIR is on the strategic management of more step-change elements of enhancement and the associated cycles of planned phases including reflection, planning, implementation and evaluation.
  • The emphasis on the effectiveness of the student learning experience reflects the focus on student engagement and high-quality learning referred to in the preface above and is articulated more fully below.

Taking deliberate steps to bring about continuous improvement

27  In order to take deliberate steps, an institution (and its constituent departments, faculties, schools etc) will ask itself:

  • Where are we now? For example: Who are our students? What are the characteristics and learning needs of our students? How effective is the current learning experience of our students? Are some groups of students more successful learners than others? Are some groups of students better prepared for post-graduation life than others? What evidence can we draw on? How robust is the evidence? What is the evidence telling us?
  • Where do we want to be in the future? For example: What are the patterns and mechanisms of supporting learning which the institution wishes to develop in order to support student engagement and high-quality learning?
  • How are we going to get there? For example: How are we as an institution going strategically to manage the processes of enhancement that will allow us to move towards meeting our aspirations?
  • How will we know when we get there? For example: What monitoring and evaluation processes do we have in place? How will the outcomes be analysed? How, and to whom, will the outcomes be disseminated?

28  In addressing these questions the institution will make use of a wide variety of reference points. Some of these reference points, such as the Academic Infrastructure, will be common to all institutions while others may be limited to particular institutions or groups of institutions. Some reference points will be determined largely externally (eg, professional, statutory or regulatory body guidelines) while others will be internally defined (eg, institutional strategic plans, quality enhancement plans etc). Some reference points will be national while others will be international (eg, from international learned societies, international groupings of institutions, Bologna guidelines etc). It is likely, therefore, that while there will be significant elements of commonality, the particular combination of reference points used by individual institutions may vary widely. There is also likely to be significant variation in the way reference points are used both within and between institutions.

29  Within this variation, there is an expectation that all Scottish higher education institutions will adhere to the Academic Infrastructure, including the higher education qualifications framework within the SCQF. In general, it is expected that institutions will design and review their policies and approaches with reference to the Academic Infrastructure, but the relationship is not one of narrow compliance. The outcomes of the Enhancement Themes also provide interesting and potentially valuable reference points. It is expected that institutions will engage with the Enhancement Themes and their outcomes. The precise nature of that engagement will be determined by institutions individually.

30  One of the reference points used by institutions (and their component parts) will be the vision agreed by the sector and other stakeholders in Scotland of the meaning of a high quality higher education sector referred to above, namely:

  • a sector which is flexible, accessible and responsive to the needs of learners, the economy and society
  • a sector which encourages and stimulates learners to participate in higher education and to achieve their full potential
  • a sector where learning and teaching promotes personal and intellectual development and* the employability of students
  • a sector where learning and teaching are highly regarded and appropriately resourced
  • a sector where there is a culture of continuous enhancement of quality, which is informed by and contributes to international developments.

Aspects of the student learning experience

31  The student learning experience is an extremely broad concept. The focus within ELIR will be on those aspects of the student learning experience for which the institution bears a direct responsibility. It is recognised that between institutions, and indeed within institutions, there will be variation in what is considered the optimum framework to support student learning. It is not a ‘one size fits all’. Different students in different institutions on different programmes may well have different needs.

32  The aspects of the student learning experience to which ELIR will relate will normally include the management of the inter-related elements of:

  • the curriculum – structure, aims, intended learning outcomes and assessment
  • promoting and supporting, for all students, effective student engagement and high quality learning, including the role of all staff with responsibilities for student engagement and high-quality learning.

33  In relation to the curriculum, enhancement refers to the processes used to sustain and develop the currency of the curriculum and its outcomes in the light of developments in knowledge and understanding, professional practice, employer and other stakeholder expectations and other appropriate reference points. In addition to subject-related outcomes, enhancement will also relate to more general outcomes expected from higher education, for example those associated with employability, and preparations for continuing lifelong learning. Enhancement in this context also refers to the improvements in assessment policy and practice, and the alignment of appropriate assessment and learning and teaching strategies with intended learning outcomes.

34  In relation to promoting and supporting effective student engagement with their learning, enhancement embraces all the structures and processes used by an institution to support effective student learning in all the contexts in which learning takes place. This would include, for example, support through curriculum design and pedagogical approach, lectures, tutorials, research supervision, work-based learning supervision, independent learning support, e-learning and distance learning. It would also relate to the role of 'support' services in contributing to the student learning experience. For example, the way in which institutional careers services impact on the effective learning experience of students through influencing curriculum selection and the development of employment related skills. It would also include the support of high-quality learning through counselling and other support services of the institution and to the provision of library and information services and the institution's IT infrastructure. In general, supporting student engagement with their learning relates to the way in which an institution identifies and addresses the learning and support needs of its particular students. These will include the support services identified by institutions in their own institution-led reviews.

35  A key aspect influencing the effective engagement of students in their learning is the role played by students themselves. Students are active partners with shared responsibilities for their own learning and achievement. Indeed, one of the defining characteristics of higher education is the extent to which it relies on this active participation in, and student ownership of, the learning process. The ELIR process will engage with how this partnership is defined, managed and reflected on by institutions and their students. It will also wish to engage with how, in general, students are supported in becoming effective higher education learners and how they both exercise their responsibilities for engaging with their learning and contribute to the monitoring and evaluation processes.

Enhancing quality and maintaining standards

36  As discussed above, changes in the curriculum will reflect developments in knowledge, professional practice, and general employer and stakeholder expectations. Within this dynamic context, institutions will maintain the academic standards of their awards through their processes for defining awards, validating and reviewing programmes, and assessing learning outcomes achieved. The ELIR process will engage with the effectiveness of these institutional processes for maintaining the academic standards of awards taking account of appropriate reference points.

37  It is important not to confuse the maintenance of academic standards of awards with the standards of the outcomes achieved by students. Improving the student learning experience will, potentially, improve the levels of outcomes achieved by students and result in an improvement in student performance, and so an increase in the number of students progressing and achieving awards or achieving higher grades of awards.

Enhancement and the assurance of quality and standards

38  It follows from the above that the baseline for enhancement is systematic awareness of the current learning experience of students, the outcomes achieved by students and the standards of the institution's awards. The outcomes from the institution-led reviews utilising appropriate sets of reference points will be fundamentally important in providing the institution with this information. Through ELIR's engagement with the efficacy of these and related processes, it will be able to provide a well-founded view on how effectively the institution assures itself that academic standards and quality are at least being maintained in line with national expectations. Within the ELIR approach, this baseline assurance, although clearly important, represents only the initial phase of engaging with the enhancement processes and will not form the main focus of the ELIR activities or its reporting.

Enhancement, complexity and risk

39  It is important to emphasise that enhancement is the result of change, often involving innovation. Some forms of innovation will be relatively straightforward and risk free. However, other forms of innovation are likely to involve some element of risk. Institutions manage this risk in a way that will provide reasonable safeguards for current students. The ELIR process and reporting will recognise and support effective risk management in relation to change and innovation, and will adopt a supportive and not punitive role in this context. It is inevitable that some changes will be more successful than others and often more can be learned in the medium to long run from analysing the reasons for less successful outcomes.

Summary

40  From the above material relating to the definitions of terms, the complexity of quality cultures, and assurance and enhancement processes, it is clear that the creation of effective and penetrating quality cultures poses significant challenges. The second edition of the Handbook continues to recognise these challenges. The general expectations embodied within the revised version of ELIR are of a growing maturity of approach, building further on the ‘green shoots’ identified in the previous version of the method. Indeed, it is hoped that one of the outcomes of the continued operation of ELIR will be the further support of institutions in the development of their individual quality cultures, and collectively in the quality culture of Scottish higher education.

Section 2: The enhancement-led institutional review approach

Scope of ELIR

41  The scope of ELIR includes the mechanisms to support all award-bearing provision within the institution. An award in this context includes the award of credit. ELIR therefore relates to the learning experience of all students on    credit-bearing provision: undergraduate and postgraduate students; taught and research students; full and part-time students, including those involved in    credit-bearing continuing professional development; campus-based, work-based, distance-learning students (including those supported wholly or in part using information technology); students entering higher education institutions from school, through wider access initiatives from further education colleges and from the workplace; students who are publicly funded and students who are privately funded; and international students whatever their location of study. ELIR will also embrace collaborative provision wherever and however it is delivered.

42  ELIR will include within its processes provision leading to the award of the institution (including credit) that is delivered elsewhere, such as through a further education college or employer organisation. In cases where the delivering institution is itself a Scottish higher education institution, the delivering institution will receive an ELIR review in its own right. Responsibility for the academic standards of awards offered through such arrangements, however, remains unambiguously with the awarding institution. Where provision is made in conjunction with an overseas partner, ELIR will relate to the arrangements in place in the Scottish institution for managing the quality of the learning experience and the academic standards of the awards. The Scottish higher education institutions will continue to participate in overseas audits and the outcomes from those audit reports will form useful reference points in ELIR.

43  ELIR will include consideration of an institution’s approach to the management of the learning experience of all students, and how all students’ needs are considered, whatever their race, gender, sexuality, faith, age or whether they have a disability.

Nature of ELIR

44  The ELIR process is conceived and designed to support institutions’ self-evaluation and reflection. Central to the ELIR method, therefore, is the institution’s Reflective Analysis (RA) which will highlight the distinctive features of the institution’s arrangements for enhancing the student learning experience and securing academic standards. Crucially, the RA will set out the institution’s reflections on the effectiveness of its approach in those areas, citing the evidence on which these reflections are based.

45  ELIR is concerned with strategic management and the effectiveness of its implementation in order to improve the student learning experience. A core ELIR inquiry is, therefore, to ask the institution how it learns from current and past activities in order to inform its strategic approach to quality enhancement and how, in turn, that approach improves the learning experience of its students. The institution is asked to identify the reference points it uses in reflecting both on its strategic approach and on the student learning experience it offers. These reference points may include the institution’s own activity, or experience identified within the sector or internationally (see paragraphs 28 and 29).

46  The ELIR method seeks to:

  • be open and transparent, forward-looking and conducted in a collaborative spirit, avoiding surprises
  • support the sector, individual institutions and their staff and students in enhancing effective learning
  • relate to the wider quality enhancement framework in Scotland.

ELIR method

47  The ELIR method consists of four integrated elements: an annual discussion with each institution; the production of an RA; a review visit following the submission of the RA; and sector-wide feedback on the learning points from ELIR activity.

Annual ELIR discussion

48  Annual discussions facilitate the review process and provide an important opportunity for information sharing between QAA Scotland and the institution. These annual meetings will be held between a member of QAA Scotland staff and a small group from the institution which is likely to comprise senior colleagues and a representative of the student body. As was previously the case in ELIR, the annual discussions will not result in any judgements or any public reporting.

49  As was previously the case in ELIR, a particular focus of the annual meetings will continue to be discussion of the institution’s approach to institution-led reviews, and what the institution is learning from the outcomes of the reviews that have been held in the preceding year. Institutions may wish to share other information and activities at these meetings, for example: the outcomes of visits by professional, statutory and regulatory bodies; the institution’s experiences of working with elements of the Academic Infrastructure and of engaging with sector initiatives; or significant developments in the institution’s approach to quality enhancement.

50  The meetings also provide an opportunity for discussing preparations for, and the outcomes of, ELIR. The annual meeting held closest to the anniversary of the publication of the ELIR report will provide a key opportunity to discuss the     year-on response to ELIR which the institution will be asked to produce (see paragraph 79).

51  Other than the year-on response to ELIR, institutions will not be expected to prepare bespoke material for the annual discussions; instead it is anticipated that the meetings will be supported by a set of existing material, such as:

  • a copy of the definitive internal document(s) describing the approach taken to institution-led review including information about the forward timetable of reviews
  • copies of the institution-led review reports for the previous 12 months
  • a copy of any internal documents analysing the outcomes from institution-led reviews or evaluating the review method(s) used
  • a copy of any existing documents relating to changes in the institution’s approach to quality enhancement
  • a copy of the most recent annual institutional return to the SFC relating to institution-led reviews.

52  As was previously the case in ELIR, institutions will be encouraged to provide other material originally prepared for internal purposes (such as awaydays or other strategic meetings) which the institution considers is likely to inform the annual ELIR discussions.

53  There will be no formal report of the annual discussions. However, following the meeting, the QAA officer will write to the institution to confirm any action points agreed and/or to outline any key topics explored.

Institutional profile

54  In the year in which the institution is preparing its RA, the QAA officer managing the ELIR will support the institution in producing an institutional profile which will be formed from the series of annual returns made to the SFC relating to institution-led reviews over a period agreed between the institution and the officer. The institutional profile is a descriptive document providing outline information for the ELIR team about the institution’s method of institution-led reviews and identifying summary information about the key outcomes of the reviews held over the preceding period.

Reflective analysis

55  In advance of the ELIR visit, the institution will be asked to submit an RA. The  RA should act as a demonstration of the institution’s capacity for self-reflection and critical evaluation in relation to the matters within the scope of ELIR. The evaluation will be evidence-based and the RA should include the evidence, or clear reference to the evidence, on which any analysis is based.

56  There will be a close link between the RA and the ELIR report; institutions will be asked to address the overviews set out within the ELIR report structure, providing a clear indication of the institution’s approach to these matters, and providing an analysis of the effectiveness of the institution’s approach alongside references to the evidence on which that analysis is based. Normally, the institution will use the outcomes of the previous ELIR, or institutional practice at that time, as the starting point for the analysis. The RA should contain the minimum descriptive text necessary consistent with setting out the nature of the institution’s approach and in order to elucidate the evaluative material. Where any further explanation of the institution’s processes or systems is thought to be desirable, the institution should provide existing documents or handbooks alongside or subsequent to submission of the RA.

Case studies

57  Previously in ELIR, institutions were invited to submit case-studies with their RA to illustrate the institution’s strategic management of enhancement. All institutions undertook significant work to select and write up the case-studies but the      case-study material only played a central role in some ELIR visits and reports. For the revised ELIR method it is intended that the case-study material will form a significant part of the evidence base for all reviews and that there will be a more explicit and systematic approach to disseminating the case-study material within and beyond the sector. In order to ensure that the case-studies do fulfil a more central role, there will be a number of implications for their focus and content. Recognising a shift from description towards analytical and clearly evidence-based text in the RA, the case-studies provide institutions with an opportunity to identify for ELIR teams self-contained examples of the institution’s strategic approach to quality enhancement or to the management of academic standards.

58  Institutions should submit one or more case studies with the RA. The case-studies should illustrate for the ELIR team the linkages between the institution’s strategic approach and its operational management, rather than describing individual instances of good practice. The case-study material should set out: a clear rationale for its inclusion with the RA; the institution’s process of selecting the case-study; information about the management process relating to the practice set out in the case-study, for example which aspect(s) of the institution’s strategic approach is the case-study intended to illustrate and what arrangements has the institution put in place to evaluate the effectiveness of the practice. In common with the rest of the RA, the case-study text should be reflective and evidence-based.

Student involvement

59  As the first edition of the Handbook stressed, it is anticipated that the process of producing the RA will be inclusive of all appropriate interests within the institution. In particular, institutions are expected to involve students in the process of creating the RA. The RA should indicate how student views have been elicited and incorporated, and should offer a view on the impact of that student engagement.

The ELIR visit

60  The operation of ELIR introduced a two-part visit structure, with a five-week interval between the two parts. Following careful consideration and discussion, this has been retained.

61  The Part 1 visit involves the ELIR team visiting the institution for two days. The purpose of Part 1 is to:

  • ensure that the ELIR team has a sound understanding of the institution and its approach to the matters within the scope of ELIR as laid out in the RA
  • allow the team to identify the ELIR themes to be explored during the Part 2 visit and to share these with the institution
  • allow the team time to agree and share with the institution an appropriate programme of activities for the Part 2 visit 
  • identify particular documentation to facilitate the team’s exploration of ELIR themes in the Part 2 visit. 

62  The programme of activities during the first half-day of the Part 1 visit is determined by the institution. The institution has full flexibility in providing a programme of activities which it believes will most effectively assist the ELIR team in understanding the institution’s approach to managing and enhancing the student learning experience. Evaluations of ELIR have demonstrated that this half-day is highly beneficial in establishing a positive relationship between the ELIR team and the institution and, therefore, in setting the tone for the remainder of the review. In order to further facilitate the establishment of a collaborative relationship, QAA Scotland will liaise with the institution in advance of the Part 1 visit, sharing the ELIR team’s early identification of themes arising from the RA. The institution, in turn, will be asked to share the outline of its programme for the first half-day with the team.

63  The remainder of the Part 1 visit will involve a series of meetings to explore agendas developed by the ELIR team following its consideration of the RA and the interaction of the first half-day. There will be meetings with: a group of senior staff; a group of student representatives; and a group of staff who have experience of the institution-led review activity. At the end of the Part 1 visit, the ELIR team will share with the institution the themes to be explored during the Part 2 visit, together with a draft programme for that visit.

64  The Part 2 visit will last between three and five days depending on the size and complexity of the institution, and on the themes emerging from the RA and the Part 1 visit. In exploring these themes during Part 2, the ELIR team will wish to consider a specified set of the institution’s documents and to meet with a range of staff and students. Discussions during these meetings will cover: the effectiveness of the institution’s strategic approach to enhancement; the effectiveness of its management of the student learning experience; and the effectiveness of its arrangements for institution-led monitoring and review. The Part 2 visit will include opportunities for the ELIR team and the institution to clarify any matters as appropriate. On the final day of the Part 2 visit, the ELIR team will meet, with the assistant director, to agree its conclusions and to compile an outline of the draft ELIR report.

65  One week after the end of the Part 2 visit, the QAA assistant director will send a ‘key themes’ letter to the institution summarising the provisional conclusions of the ELIR. A draft of the main and summary ELIR reports will be sent to the institution for comment within an agreed period of the conclusion of the Part 2 visit (see paragraph 73). Following receipt of the institution’s comments, QAA Scotland will finalise and publish the ELIR reports.

ELIR outcomes and sector-wide feedback

66  QAA Scotland will undertake regular analysis of the main themes emerging from ELIR activities across the sector. This analysis, together with the material drawn from the case studies, will form the basis of publications, workshops/conferences and other activity. As part of its regular reporting to the SFC, QAA Scotland will provide SFC with annual reports on the outcomes of ELIR activity. QAA Scotland will continue to publish the series of Learning from ELIR reports which address different aspects of the enhancement-led approach to quality management in the Scottish sector. That series, and other ELIR reports and events, will provide evidence of the impact of the enhancement-led approach, will inform national and international debate and, in particular, will guide the SHEEC as it takes forward its strategy to support the implementation and embedding of a quality culture in general and the Enhancement Themes in particular.

Complaints and representations

67 The procedures for making complaints on any aspect of the operation of the ELIR process, or representations in respect of reported judgements, are available on the QAA website.

Section 3: The enhancement-led institutional review reports

Audiences and reporting

68 The purposes of reporting on the outcomes of ELIR are to provide informed peer feedback to individual institutions and to provide public information on quality and academic standards within and beyond Scotland. Public reporting also has an important role to play in the wider promotion and support of quality enhancement. The purposes of ELIR reports are:

  • to provide public information about the security of academic standards and the management of the student learning experience
  • to support the development of quality cultures and the strategic management of quality enhancement in individual institutions and the Scottish higher education sector as a whole
  • to promote the positive reputation of Scottish higher education institutions within the UK and internationally in relation to good practice in academic standards and quality management
  • to provide a formal record of the individual review and its outcomes.

69 ELIR outcomes attract interest from a wide audience, for example: students and student bodies; public bodies with an interest in quality management including SFC, Universities Scotland, the Higher Education Academy, and public, statutory and regulatory bodies; and employers. The full ELIR reports also attract a specialist audience including key members of staff from the institution being reviewed, quality managers at other institutions, and officers from QEF partner bodies, notably QAA Scotland and SFC officers.

70 In recognition of the different needs and interests of these audience groups, there will be two reports on ELIR outcomes for individual institutions: the main report and a summary report specifically aimed at a wider, less specialist audience. Both reports will be published on the QAA website; the summary reports will also be available in print format.

Main report

71 The main ELIR report will provide information about the effectiveness of the institution’s approach in three broad areas:

  • management of the student learning experience
  • institution-led monitoring and review of quality and academic standards
  • strategic approach to quality enhancement.

72 Further detail on the structure and indicative content of the main ELIR report is provided in Annex 1.

73 The main and the summary reports will both be sent to the institution in draft form for comment within eight weeks of the end of the Part 2 visit and will be published on the QAA website within 20 weeks of the end of the Part 2 visit.

Summary report

74 The summary report will be a short document setting out key information about the method and outcomes of the review aimed at an informed lay audience. It will be drafted by QAA Scotland and agreed by the ELIR team, before being sent to the institution for comment at the same time as the main ELIR report. The summary report will include:

  • brief introductory material about the institution being reviewed, emphasising the context of the review
  • a short summary of key matters raised in the review, emphasising the approach of the institution
  • an indication of the commentaries
  • the confidence judgement will be set out in full with brief explanatory text where this is thought to be necessary in order to make the text fully accessible to a wider audience.

Commentaries and judgement

75 The three main sections of the ELIR report will each lead to a commentary in which the ELIR team will set out its views of the effectiveness of the institution’s approach. The commentaries will draw on the material included within the overviews of the relevant report section and there will be no prescribed formulae for expressing the team’s views. Each of the commentaries will include the team’s views on the institution’s management of collaborative activity and the international dimensions referred to earlier. Since these areas will be fully integrated, there will not be separate commentaries on those aspects.

76 The commentaries will be:

  • the effectiveness of the institution’s management of the student learning experience
  • the effectiveness of the institution’s arrangements for institution-led monitoring and review of quality and academic standards of awards however and wherever delivered
  • the effectiveness of the institution’s implementation of its strategic approach to quality enhancement.

77 These commentaries will lead to a single overarching judgement which will be expressed in the form of a confidence statement. The confidence statement is intended to provide a holistic view of the effectiveness of the institution’s management of academic standards, and of the assurance and enhancement of the student learning experience, both currently and in the future. It will be possible, where the ELIR team considers it necessary, to separate the elements of this judgement to express different levels of confidence in the different aspects of the institution’s management, or if the team has a different level of confidence in the current and likely future management of activity. It will also be possible to add caveats to the confidence judgement as appropriate to the findings of the review and in order to draw attention to the management of specific aspects of activity, for example collaborative activity, or to specific groups of students.

78 ELIR teams will express their level of confidence in one of three standard forms: confidence, limited confidence or no confidence. The confidence statement is a judgement of probability; it cannot be unconditional. Where the team finds that the institution is managing the security of academic standards effectively, and where this looks likely to continue in the future, the team will express ‘confidence’. Similarly, where the team finds that the institution is managing the quality of the student learning experience effectively, including taking appropriate steps to enhance that experience, and is likely to continue to do so in the future, the team will express ‘confidence’. If the ELIR team has substantial doubts about the current, or likely future, management of the security of academic standards and/or the effectiveness of the student learning experience, the team will express ‘limited confidence’ and will indicate clearly the areas of concern that have given rise to the limitation of confidence. A judgement of ‘limited confidence’ will indicate that there is evidence that the institution’s capacity to manage effectively the academic standards of its awards and/or the student learning experience is limited or is likely to become limited in the future. It is not a judgement of failure but it does indicate that improvements need to be made. If the ELIR team has serious concerns about the current, or likely future, management of the security of academic standards and/or the quality of provision, it will express ‘no confidence’ and will indicate clearly the significant areas of concern that have given rise to the judgement. A judgement of 'no confidence' indicates that there is substantial evidence of serious and fundamental weaknesses in the institution's capacity to secure the academic standards of its awards and/or to maintain an appropriate quality of educational provision.

Follow-up to the report

79 One year after publication of the ELIR report, QAA Scotland will ask the institution to produce a year-on response to ELIR. This will focus on the action taken following the review and will include consideration of the effectiveness of that action insofar as it is reasonable for the institution to take such a view in the time that has elapsed. The year-on response will be a focus for the annual ELIR discussion held closest to the anniversary of the publication of the ELIR report.

80 In the year-on ELIR response, institutions will be asked to pay particular attention to setting out the action that has been taken in relation to any recommendations that are associated with the confidence statement. Institutions will continue to have flexibility in the length and style of the year-on responses they submit to QAA.

81 After the annual discussion at which the year-on ELIR response has been discussed, the QAA Scotland officer will write to the institution to confirm whether the action the institution has taken is broadly in line with the outcomes of the ELIR report. QAA Scotland is asked to report to SFC on the extent to which institutions are responding appropriately to the outcomes contained within the ELIR reports. The year-on responses to ELIR will form a key information source for these regular reports.

82 The final version of the institution’s year-on response to ELIR will be published on the QAA website alongside the ELIR report to which the response refers. For this version, institutions will be asked to provide brief information about the areas in which they have taken action since publication of the ELIR report, focusing particularly on any areas of action which relate to the confidence judgement. Institutions will be able to write these reports in their own words and there will be some flexibility in the structure used to fit the nature of the outcomes of the ELIR report in question, but there will be a number of standard topics to be addressed:

  • identifying the main areas in which action has been taken by the institution since the ELIR
  • how action has been taken forward, or is planned to be taken forward, and how its effectiveness will be evaluated
  • identifying any matters that have not been addressed with a rationale for this or with an outline of future plans
  • identifying any other matters the institution wishes to highlight.

83 SFC will be provided with copies of the full and summary ELIR reports. As a matter of course, SFC may wish to provide a commentary to institutions following receipt of reports. Where the ELIR report indicates that there can only be limited or no confidence in the institution’s ability to manage quality and/or academic standards, SFC will require the institution to undertake follow-up action in specified areas.

Comparability with the rest of the UK

84 In addition to the added-value of the ELIR approach, comparability of reporting on baseline quality and academic standards across the UK is provided through:

  • the use of key UK-wide reference points both by the institutions and the ELIR method (these include the Code of practice for the assurance of academic quality and standards in higher education, subject benchmark information, and the aligned qualifications frameworks, all published by QAA)
  • reporting the outcomes of ELIR in the form of a confidence statement using levels of confidence that are equivalent to those in other parts of the UK
  • the use of ELIR teams that include members drawn from across the UK.

International dimension

85 In the revised ELIR method, greater emphasis has been placed on international comparability through:

  • encouraging institutions to evaluate their use of external reference points and practice
  • including an international member on each ELIR team
  • disseminating internationally the practice and outcomes of ELIR and the related programme of Enhancement Themes.

Section 4: The enhancement-led institutional review team

The ELIR team

86 ELIR teams will comprise six peer reviewers: a student reviewer, four reviewers who are senior academic managers (including an international reviewer), and an ELIR coordinating reviewer (ECR). While some reviewers may have specific responsibilities, the ELIR team will act together during the review visits and decisions relating to the commentaries and conclusions within the ELIR report will be taken collectively by the team. All reviewers are full members of the ELIR team, with equal status. 

87 All reviewers will have responsibility for:

  • reading and analysing the RA and any other documentation provided by the institution
  • participating in the review visits
  • reaching conclusions on the basis of the information gained during the review
  • contributing to and commenting on the review report.

88 The international reviewer will bring an added perspective to the ELIR team’s consideration of the institution’s approach to quality assurance and the enhancement of the student learning experience. The international reviewer will have a range of knowledge and experience which will benefit the review process, the institution and the wider Scottish higher education sector. International reviewers will be senior peers, selected from appropriate institutions or organisations, and will be required to undertake full training for their role.  

89 The student reviewer brings a learner perspective to the review. Their responsibilities will focus on lines of enquiry relating to the institution’s management of the student learning experience and the effectiveness of the institution’s approach to engaging students.

90 The ECR will, usually, be a senior administrator and will have responsibility for maintaining an overview of the review progress and its outcomes. The ECR will have particular, although not necessarily exclusive, responsibility for:

  • ensuring the ELIR team has access to appropriate documentation during the review
  • maintaining a record of discussions held with staff and students
  • facilitating the team’s identification and evaluation of the key themes to be pursued during the review, ensuring alignment of these themes with the overviews and commentaries on which the team must report
  • supporting the team in identifying the evidence on which its conclusions and commentaries are based.

91 In undertaking these responsibilities, the ECR will maintain an ongoing record of the team’s emerging conclusions and supporting evidence. At the end of the review visit, the ECR will use the ongoing record to support the team and the QAA assistant director in producing an outline report, identifying the key themes for inclusion within each of the report commentaries. Other members of the ELIR team will have responsibility for supporting the ECR in preparing the outline report and will have responsibility for drafting report text. Along with other members of the ELIR team, the ECR will support the QAA assistant director in editing the review report, providing additional information and evidence as necessary.

92 Each review will be managed by a QAA assistant director. From early in the process, the assistant director will provide advice to the institution on its preparations for the review, and will work with the ELIR team on the initial analysis of documentation. The assistant director will accompany the team during the Part 1 visit and for elements of the Part 2 visit, notably the final day, providing advice as appropriate. The assistant director, supported by the ECR, is responsible for testing that the team's commentaries are based on adequate and identifiable evidence, and for editing the ELIR report.

Selection criteria for reviewers and review secretaries

93 All members of ELIR teams will be selected by QAA Scotland according to the criteria identified in Annex 3 and having regard to the timetable for reviews in Scotland.

94 Student reviewers will be recruited annually. QAA Scotland will seek nominations from student representative bodies and Scottish higher education institutions. Student reviewers will be eligible to undertake reviews for as long as they continue to meet the selection criteria, provided it is not more than three years since they undertook study in a Scottish higher education institution.

95 International reviewers will be drawn from outside the UK. On an annual basis, Scottish higher education institutions will be invited to nominate one or more international reviewers. In addition, QAA Scotland will seek nominations through its contact with institutions and relevant organisations in other countries.   

96 QAA Scotland will actively seek nominations from all UK higher education institutions for senior academic reviewers and ECRs. Every Scottish institution will be encouraged to nominate at least one candidate for each role. Individuals who are existing reviewers, auditors or secretaries with QAA may be eligible to become ELIR reviewers and can apply using a streamlined process.

Allocating reviewers to teams

97 QAA Scotland will allocate reviewers to ELIR teams. Reviewers will not be allocated to the ELIR team for their own institution. International reviewers may be allocated to the ELIR team for the institution which nominated them provided there are no direct conflicts of interest. 

ELIR reviewer training

98 All ELIR reviewers, including those trained in other QAA methods, are required to undertake ELIR training. Reviewers are also expected to participate in continuing development and information seminars. Where appropriate, training and continuing development may be targeted to specific groups of reviewers, such as students or ECRs. Training in the ELIR method will be provided each year in order to enhance effectiveness by minimising the gap between training and reviews taking place.

99 ELIR training will be designed and managed by QAA Scotland, although delivery may be shared with external facilitators, in particular experienced reviewers drawn from the sector. Training will cover the philosophy and methodology of ELIR as well as skills-related activities.

Continuing development and information sessions

100 In order to share experience of the ELIR method and to maintain the knowledge of experienced reviewers, annual information sessions will be held. All reviewers who have participated in an ELIR team during the cycle, together with those allocated to teams for the following year, will be invited to attend. These information sessions may be combined with events for representatives of the Scottish higher education sector.

Section 5: Monitoring and evaluation

Purpose and principles of monitoring and evaluation

101 QAA Scotland will monitor, on an ongoing basis, the operation of ELIR, undertake regular evaluation of the effectiveness of ELIR, and also contribute to the broader external evaluation of the arrangements for quality assurance and enhancement in Scotland. The overall purpose of monitoring and evaluation is to:

  • to support QAA Scotland in effectively delivering the agreed ELIR methodology
  • to inform the ongoing development of the ELIR method, in the context of the Quality Enhancement Framework.

102 The underlying requirements of monitoring and evaluation are that the processes should:

  • be regular and timely
  • ensure higher education institutions and reviewers can provide structured feedback
  • support the ongoing training and continuing development of reviewers
  • support ongoing improvement of QAA Scotland's performance
  • encourage active reflection and dialogue on the design and improvement of the ELIR approach to ensure it continues to be fit for purpose.

Monitoring

103 Monitoring activities will encompass all stages of the ELIR process:

  • annual discussions
  • ongoing QAA Scotland support for institutions, and the sector, in their preparation for ELIR 
  • parts 1 and 2 of the ELIR visit
  • notifications of ELIR outcomes and arrangements for report drafting and finalisation.

104 All those engaging in ELIR will be involved in the monitoring process: the institution, reviewers and the assistant director responsible for managing the review. The monitoring process will involve both formal and informal feedback.

105 Formal feedback will be sought through monitoring questionnaires, which all participants in ELIR will be asked to complete. The questionnaires will seek comment both on operational aspects of the review, and broader questions relating to the effectiveness of ELIR in achieving its aims. The latter will allow ongoing accumulation of information and evidence to inform the wider process of evaluation.

106 Informal feedback will also be encouraged. The assistant director responsible for a review will be the main focus for communication of feedback from institutions and reviewers and will also have their own observations to contribute. Informal feedback will continue to be sought from groups with which QAA Scotland liaises, including the Universities Scotland Teaching Quality Forum and sparqs.

Evaluation

107 Building on the feedback from monitoring activity, QAA Scotland will undertake ongoing evaluation of the effectiveness of ELIR in achieving its objectives. The overall purpose of evaluation is to 'stand back' and take stock of the effectiveness of the ELIR method. This will pose macro questions such as: Does ELIR meet its objectives? What aspects work well? Which elements could be improved and in what ways?

108 When a number of reviews have been completed and the feedback from them has been analysed, evaluation events will be held involving reviewers and institutional representatives. The purpose of these events will be to encourage reflection on the effectiveness of the ELIR approach, and to promote dissemination of key messages.

109 The findings from monitoring and evaluation will help to inform the ongoing training and development of ELIR reviewers, to ensure that they are effectively prepared and supported in fulfilling their roles.

External evaluation of the quality enhancement framework

110 SFC will commission an external evaluation of the arrangements for quality assurance and enhancement in Scottish universities and colleges.  The evaluation team is to be appointed by the start of the review cycle in 2008-09. QAA Scotland will fully participate in this external evaluation and use the outcomes generated as appropriate.

Annex 1: Main enhancement-led institutional review report structure

Notes on the report structure

The primary audience for the main ELIR report is intended to be staff in the institution being reviewed, quality managers from other institutions and officers from bodies such as SFC and QAA. This has a number of impacts on the style and content of the text. In particular, there will be significantly less descriptive text overall as compared to ELIR reports in the previous version of the method. 

The text of the main ELIR report will be organised around a series of overviews contained within the three main report sections. Each overview will include brief text to indicate the key nature of that topic at the institution being reviewed, followed by text setting out the ELIR team’s view of the effectiveness of the institution’s approach, with an indication of the basis for that view. The final section of the report will set out the three commentaries and will conclude with the confidence statement. This structure is intended to ensure that the reports continue to address the particular approach of the institution being reviewed. It is also hoped that the structure will be helpful to the reader as there will be less repetition of text, and material on particular topics should be easier to identify than was sometimes the case with reports in the previous version of the method.

Each of the three main sections of the report will include text relating to the effectiveness of the institution’s management of collaborative activity. The ELIR team’s view will be set out in the relevant commentaries; there will not be a separate commentary relating to collaborative provision.

As was previously the case in ELIR, there will be a strong link between the report and the institution’s RA. There will also be a strong link between the outcomes of the previous ELIR and the starting point for ELIR in the revised method. Institutions will be asked to bear this in mind in drafting their RA.

Institutions are invited to submit one or more case-studies with their RA. These will be listed in the report introduction, and evaluative text relating to the case-studies will also appear within the report section and overview(s) to which they relate (this will be dependent on the management process that the submitting institution intends the case-study to illustrate).

Equality of opportunity and diversity of student populations will be key themes in the revised ELIR method and will underpin a number of the lines of enquiry for ELIR teams. Matters relating to these areas may be covered under a number of the report overviews, and are explicitly highlighted in relation to the section on the Management of the Student Learning Experience.

Report section 1: Introduction

1  The text of the introduction will be brief, covering the following material:

  • standard, summary information about the ELIR method
  • brief information about the ELIR method as it related to this institution, including key features of the conduct of the review and the members of the team
  • summary information about the institution including size, broad breakdown of student population and location of students if significant numbers are studying on collaborative programmes, and the key institutional structure
  • summary information about collaborative activity should be summarised here. Note: this section will not include information about the historical origins of the institution and the total text will not usually extend beyond two paragraphs.
  • short statement of contextual factors at the time of the review including:
    • key developments within the institution. This might include, but is not limited to: structural changes, merger, relocation
    • key changes in areas relevant to ELIR – a summary outline should be provided in the introduction with more detail provided in the relevant substantive section(s) of the report
  • the institution’s statement of its strategic approach to quality enhancement.

Report section 2: Management of the student learning experience

2  The overviews in the student learning experience section will be:

  • key features of the student population and the effectiveness of the institution’s approach to managing information about its student population
  • the effectiveness of the institution’s approach to engaging and supporting students in their learning
  • the effectiveness of the institution’s approach to promoting the development of graduate attributes, including those relating to employability, in all of its students.
  • the effectiveness of the institution’s approach to managing the learning environment
  • the effectiveness of the institution’s approach to promoting equality of opportunity and effective learning for all of its students
  • the effectiveness of the institution’s approach to supporting and developing staff to promote effective learning for their students.

3  In relation to collaborative activity, this section will cover:

  • the effectiveness of the institution’s management of the student learning experience on collaborative programmes with regard to the topics set out in the overviews above.

Report section 3: Institution-led monitoring and review of quality and standards

4  The overviews in the institution-led monitoring and review section will be:

  • key features of institution-led monitoring and review at the institution, and the extent to which these arrangements meet sector-wide expectations
  • the extent to which the institution’s monitoring and review arrangements include consideration of all students
  • the effectiveness of the institution’s approach to self-evaluation including the use made of external reference points
  • the effectiveness of the institution’s approach to the management of information to inform the operation and evaluation of its monitoring and review activity
  • the effectiveness of the institution’s approach to setting and maintaining academic standards including the management of assessment
  • the effectiveness of the institution’s approach to managing public information about quality and academic standards, including the linkage with the institution’s monitoring and review arrangements
  • the effectiveness of the institution’s approach to linking its monitoring and review processes to its enhancement arrangements.

5  In relation to collaborative activity, this section will cover:

  • the effectiveness of the institution’s approach to monitoring and reviewing collaborative activity with regard to the topics set out in the overviews of this section.

Report section 4: Strategic approach to quality enhancement

6  The overviews in the strategic approach to quality enhancement section will be:

  • key features of the institution’s strategic approach to quality enhancement
  • the effectiveness of the institution’s implementation of its strategies and policies for promoting quality enhancement across the institution
  • the effectiveness of the institution’s use of external reference points in its approach to quality enhancement, including the extent to which the institution’s approach is informed by national and international practice
  • the effectiveness of the institution’s approach to identifying, disseminating and implementing good practice in the context of its strategic approach to enhancement.

7  In relation to collaborative activity, this section will cover: the effectiveness of the institution’s approach to enhancing collaborative provision.

Report section 5: Conclusion

8  This section will set out the report commentaries, followed by the confidence statement:

  • the effectiveness of the institution’s management of the student learning experience
  • the effectiveness of the institution’s arrangements for institution-led monitoring and review of quality and academic standards of awards however and wherever delivered
  • the effectiveness of the institution’s implementation of its strategic approach to quality enhancement.

9  The commentaries will lead to the overarching confidence judgement. The core confidence statement will be expressed as follows:

The findings of the ELIR indicate that there can be confidence/limited confidence/no confidence in the institution’s current and likely future management of the academic standards of the awards it offers and the quality of the student learning experience it provides.

Note: As is currently the case, different levels of confidence may be expressed in different aspects of the judgement if the ELIR team considers this is necessary. The confidence judgement may also be qualified with the addition of caveats. In addition, the confidence judgement may be contextualised to emphasise the key elements of the commentaries on which it is based.


Annex 2: Guidance for institutions on a Reflective Analysis structure

Notes on the Reflective Analysis:

The text of the Reflective Analysis (RA) should predominantly be evaluative with specific references to supporting material or further reading. Evaluation of the ELIRs carried out between 2003 and 2007 has demonstrated that there is considerable benefit in institutions following the structure of the ELIR report when producing their RA. For the revised ELIR method, institutions may choose to present their analyses in different structures but, however the text is organised, institutions are asked to address the overviews and commentaries set out in the ELIR report. In relation to each overview, institutions are asked to set out their view of their own effectiveness and, in each case, institutions should make clear the basis for that view including specific references to supporting material as appropriate.

Similar to the case in the previous ELIR method, there will be clear links between the RA and the ELIR report. There will also be a strong link between the outcomes of the previous ELIR and the starting point for the current ELIR. In many cases it is expected that institutions would wish to use the timing of their first ELIR as the starting point for evaluation in the RA.

Institutions are invited to submit one or more case-studies with their RA to illustrate for the ELIR team the linkages between the institution’s strategic approach and its operational management. It should be clear from the text of the RA which feature(s) of the institution’s strategic management each case-study is intended to illuminate. The text of the case-study material itself should be evaluative and evidence-based, it should not simply describe individual instances of good practice. Case-study material should set out: a clear rationale for its inclusion with the RA; the institution’s process of selecting the case-study; and information about the management process relating to the practice set out in the case-study, for example which aspect(s) of the institution’s strategic approach is the case-study intended to illustrate and what arrangements has the institution put in place to evaluate the effectiveness of the practice.

Equality of opportunity and diversity of student populations will be key themes in the revised ELIR method and will underpin a number of the lines of enquiry for ELIR teams. These matters will be included in the ELIR report section on the Management of the Student Learning Experience but, if it is appropriate to the institution’s approach, may also be addressed elsewhere in the ELIR report.

It is open to each institution to include with its RA background or other information about the institution and its approach in the format that it considers will be most helpful for the ELIR team. 

RA section 1: Introduction

1  Institutions are asked provide information on:

  • a statement of contextual factors at the time of the review including:
  • key developments within the institution. This might include, but is not limited to: structural changes, merger, relocation
  • key changes in areas relevant to ELIR. A summary outline could be provided in the introduction with more detail provided in the relevant substantive section(s) of the RA.
  • brief information on the method used to produce the RA, including the evaluative processes employed and the extent to which the views of all staff and students have been incorporated within the evaluation. The RA should offer a view on the impact of the engagement of students in this process. This might be covered in greater detail in later sections of the RA
  • any other information the institution considers would be helpful to the ELIR team.

RA section 2: Management of the student learning experience

1  The overviews of the student learning experience section are set out here. In each case the institution is asked to identify the particular features of its approach and provide a view on the effectiveness of that approach, making clear the basis for its view:

  • key features of the student population and the effectiveness of the institution’s approach to managing information about its student population
  • the effectiveness of the institution’s approach to engaging and supporting students in their learning
  • the effectiveness of the institution’s approach to promoting the development of graduate attributes, including those relating to employability, in all of its students
  • the effectiveness of the institution’s approach to managing the learning environment
  • the effectiveness of the institution’s approach to promoting equality of opportunity and effective learning for all of its students
  • the effectiveness of the institution’s approach to supporting and developing staff to promote effective learning for their students.

2  In relation to collaborative activity, the institution is asked to:

  • identify its approach to managing the student learning experience on collaborative programmes with regard to the topics set out in the overviews above, and provide the institution’s view of the effectiveness of that approach.

RA section 3: Institution-led monitoring and review of quality and standards

3  The overviews of the institution-led monitoring and review section are set out here, in each case the institution is asked to identify the particular features of its approach and provide a view on the effectiveness of that approach, making clear the basis for its view:

  • key features of institution-led monitoring and review at the institution, and the extent to which these arrangements meet sector-wide expectations
  • the extent to which the institution’s monitoring and review arrangements include consideration of all students
  • the effectiveness of the institution’s approach to self-evaluation including the use made of external reference points
  • the effectiveness of the institution’s approach to the management of information to inform the operation and evaluation of its monitoring and review activity
  • the effectiveness of the institution’s approach to setting and maintaining academic standards including the management of assessment
  • the effectiveness of the institution’s approach to managing public information about quality and academic standards, including the linkage with the institution’s monitoring and review arrangements
  • the effectiveness of the institution’s approach to linking its monitoring and review processes to its enhancement arrangements.

4  In relation to collaborative activity, the institution is asked to:

  • identify its approach to monitoring and reviewing collaborative activity with regard to the topics set out in the overviews of this section, and provide the institution’s view on the effectiveness of its approach.

5  In order to support the evaluative material in this section, the institution should include an analysis of the outcomes of the institution-led reviews it has carried out, together with information about the action taken to address the matters raised by those reviews.

RA section 4: Strategic approach to quality enhancement

6  The overviews of the strategic approach to quality enhancement section are set out here. In each case the institution is asked to identify the particular features of its approach and provide a view on the effectiveness of that approach, making clear the basis for its view:

  • key features of the institution’s strategic approach to quality enhancement
  • the effectiveness of the institution’s implementation of its strategies and policies for promoting quality enhancement across the institution
  • the effectiveness of the institution’s use of external reference points in its approach to quality enhancement, including the extent to which the institution’s approach is informed by national and international practice
  • the effectiveness of the institution’s approach to identifying, disseminating and implementing good practice in the context of its strategic approach to enhancement.

7  In relation to collaborative activity, the institution is asked to:

  • identify its approach to enhancing collaborative provision and provide the institution’s view of its effectiveness in this regard.

RA section 5: Conclusion

8  The institution is asked to draw together the material set out earlier in the RA by evaluating the effectiveness of its approach in the three areas covered by the ELIR commentaries. In each case the institution is asked to highlight what it considers are the key features of its approach and provide a view of its effectiveness, setting out the basis for its view.

  • the effectiveness of the institution’s management of the student learning experience
  • the effectiveness of the institution’s arrangements for institution-led monitoring and review of quality and academic standards of awards however and wherever delivered
  • the effectiveness of the institution’s implementation of its strategic approach to quality enhancement.

Annexes to the RA

9  The following information should be included as annexes to the RA:

  • an index to supporting evidence and further reading – there should be clear cross-references to this material throughout the RA
  • summary information about the institution including the organisational structure (schools/faculties/etc) and a breakdown of staff and student numbers
  • information about collaborative activity including the student numbers by collaborative partner, location and programme
  • case-studies.

Annex 3:  Criteria for the selection of reviewers

Introduction

1  All ELIR reviewers will be selected by QAA Scotland on the basis of the criteria set out below. In general, nominations will be invited from institutions across the UK, with every Scottish institution encouraged to make at least one nomination to each reviewer role. Student reviewers may be nominated by Scottish student representative bodies or Scottish higher education institutions. International reviewers will be selected from Scottish higher education institution nominations and from QAA Scotland’s contacts with relevant institutions and organisations in other countries. 

2  The qualities required in ELIR reviewers are detailed below. Student reviewers are required to have current or recent direct experience of study at a Scottish higher education institution. International reviewers will be recruited for the broader perspective they are able to bring to the areas of enquiry within the scope of ELIR. ELIR coordinating reviewers and UK-based senior academic reviewers will be drawn from across the UK. Every attempt will be made to ensure that the total pool of ELIR reviewers reflects sectoral, geographical and equal opportunity dimensions. 

3  All reviewers will be provided with training by QAA Scotland to ensure that they are familiar with the ELIR approach. The purposes of ELIR training are to ensure that all reviewers: 

  • understand the aims and objectives of the ELIR process as well as the key elements of the method
  • understand their own roles and tasks, the importance of team coherence, and the collaborative style ELIR aims to achieve 
  • have an opportunity to explore and practise review skills including information assimilation and analysis; the development of programmes for visits; achieving a constructive dialogue; the construction and testing of hypotheses; the formation of evidence-based conclusions and statements of confidence; and the preparation of reports.

Qualities required in all reviewers

4  All reviewers will be able to demonstrate the ability to:

  • understand a range of perspectives, relate to a range of individuals including students and senior managers, and lead discussions about strategic and operational approaches to the management of quality and standards in general, and the enhancement of the student learning experience in particular
  • assimilate a large amount of disparate information and analyse it to form reliable, evidence-based conclusions 
  • communicate clearly, orally and in writing
  • work productively and cooperatively in small teams delivering to tight deadlines
  • maintain the confidentiality of sensitive matters.

Additional qualities required in UK-based senior academic reviewers

5  In addition to the qualities required in all reviewers, UK-based senior academic reviewers will be able to demonstrate:

  • current or recent (within three years) wide experience of academic management, preferably relating to quality assurance and enhancement, at institutional level in the UK
  • personal and professional credibility with staff including senior managers and heads of institutions
  • knowledge and understanding of the QAA Academic Infrastructure and other key reference points including the Scottish Credit and Qualifications Framework
  • awareness of the distinctive features of the Scottish higher education system in general and the enhancement-led approach in particular (ELIR training will seek to emphasise this but some initial awareness is highly desirable). 

6  QAA Scotland will also be interested in reviewers identifying other relevant experience they may have, for example, experience of good practice in quality assurance and enhancement in an international setting.

Additional qualities required in international reviewers  

7  In addition to the qualities required in all reviewers, international reviewers should be able to demonstrate a number of the following attributes:

  • current or recent (within three years) wide experience of academic management, preferably relating to quality assurance and enhancement, at the institutional level outside the UK
  • current or recent (within three years) experience of external review of higher education institutions outside the UK, either as a panel member or through senior involvement with a quality assurance or enhancement organisation 
  • peer-acknowledged expertise in the development of good practice in learning and teaching
  • knowledge and experience of practice in more than one country in addition to the UK; it is highly desirable to have international comparative knowledge and experience
  • an awareness of the distinctive features of the Scottish higher education system in general and the enhancement-led approach in particular (ELIR training will emphasise this but some initial awareness is highly desirable). 

Additional qualities required in student reviewers

8  QAA Scotland will actively encourage applications from students with all backgrounds and with experience of any mode or level of study.

9  In addition to the qualities required in all reviewers, student reviewers will be able to demonstrate:

  • current or recent (within three years) experience of study at a Scottish higher education institution, equivalent to a minimum of one year's full time education.
  • experience of representing students' interests at institutional (including faculty or school) level
  • general awareness of the diversity of the Scottish higher education sector beyond their 'home' institution, and awareness of the arrangements for quality assurance and enhancement in Scotland (ELIR training will provide further information on this and QAA Scotland is looking for applicants who have the ability to build on their existing experience).

10 QAA Scotland will also be interested in student reviewers identifying their other relevant experience.

Additional qualities required in ELIR coordinating reviewers

11 In addition to the qualities required in all reviewers, ELIR coordinating reviewers will be able to demonstrate:

  • current or recent (within three years) experience of senior academic administration at institutional (including faculty or school) level in UK higher education
  • wide experience of working with senior committees in UK higher education
  • awareness of the distinctive features of the Scottish higher education system in general and the enhancement-led approach in particular (ELIR training will seek to emphasise this but some initial awareness is highly desirable)
  • ability to keep a reliable record of discussions, summarise the key outcomes and produce coherent text in a specified format to tight deadlines
  • experience of drafting, collating and editing complex reports
  • ability to retain an effective overview of complex tasks, and to support and manage a small team in achieving those tasks.

1 Enhancing the quality of teaching and learning in Scottish Universities 2007. Final report of the external evaluation quoted in SFC circular SFC/11/2007 ‘Evaluation of the higher education quality enhancement framework: final report’.

2 Final report from the Joint Quality Review Group to Council, SFC August 2007

3 Standards and Guidelines for Quality Assurance in the European Higher Education Area, ENQA, Finland, 2007

4 Guidelines of Good Practice 2007, INQAAHE, Dublin 2007

5 Learning to Improve: quality approaches for lifelong learning, Scottish Executive, Edinburgh 2005

6 The Crerar Review : The report of the independent review of regulation, audit, inspection