section photograph

Handbook for academic review

Introduction

Background

1 The mission of the Agency is to promote public confidence that quality of provision and standards of awards in higher education are being safeguarded and enhanced. To this end, the Agency carries out academic reviews of the performance of subjects and institutions. This Handbook describes the method and procedures for carrying out academic reviews, in respect of both programmes (single subject or inter-disciplinary) and institutions, the quality of learning opportunities provided, the academic standards achieved, and the quality assurance systems operating in higher education (HE) institutions in the UK.

2 The method and procedures comprise an integrated approach focused on the establishment, maintenance and enhancement of academic standards. These are matters for which the primary responsibility lies with the HE institution. However, the funding councils have a statutory responsibility to secure that provision is made for assessing the quality of education they fund. The Agency acts on behalf of the funding bodies in this respect.

3 The main purposes of this statutory assessment are:

  • to secure value from public investment, by ensuring that all education for which funding is provided is of approved quality, and by encouraging speedy rectification of major shortcomings in the quality of education, and to enable judgements to inform funding should the funding council so decide;
  • to encourage improvements in the quality of education through the publication of subject review reports and subject overview reports, and through the sharing of best practice;
  • to provide, through the publication of reports, effective and accessible public information on the quality of HE.

4 Public confidence in the quality and standards of higher education depends on the availability of public information that is objective and independent. The Agency provides this by addressing three inter-dependent areas:

  • reporting on programme outcome standards is concerned with the appropriateness of the intended learning outcomes set by the subject provider (in relation to relevant subject benchmark statements, qualification levels and the overall aims of the provision), the effectiveness of curricular content and assessment arrangements (in relation to the intended learning outcomes), and the achievements of students;
  • reporting on the quality of learning opportunities in a subject is concerned with the effectiveness of the teaching, the learning resources and the academic support in promoting student learning and achievement across the various programmes in the subject area;
  • reporting on institutional management of standards and quality is concerned with the robustness and security of institutional systems relating to the awarding function. This involves, in particular, arrangements for dealing with approval and review of programmes, the management of institutional credit and qualification arrangements, and the management of assessment procedures.

5 The overall process of gathering evidence about standards and quality must be effective and efficient and avoid, as far as possible, duplication of effort. Reports on programme outcome standards and the quality of learning opportunities will be outputs of a single process of review of subjects, which forms the focus of Part 1 of this Handbook. These reports feed information into the process of scrutiny of institutions, thereby providing material for reports on institutional management of standards and quality. This process is dealt with in Part 2 of this Handbook.

6 Conceptually, academic review may be seen as a dynamic engagement with the internal processes of an institution over a six-year cycle. Subject review will update continuously the picture that the Agency has of an institution, and will provide audit trails to inform judgements about overall institutional systems. For review of institutional systems, it should not be necessary to collect significant quantities of additional information, except in relation to the most senior layers of the institutional structure.

7 Institutional review is in this sense a continuous process, with each subject review contributing information to the overall picture. That picture will help determine the intensity of scrutiny needed to report reliably in subsequent subject reviews, and to make the summative judgements on overall institutional systems once in each cycle.

8 For both subjects and institutions, the method of academic review will be centred on self-evaluation documents produced by the institution. Guidance on the preparation of self-evaluation documents is provided in the annexes to this Handbook. The first task for academic reviewers is to test, by means of their own observations and analyses, the statements made by institutions in their self-evaluation documents. Secondly, they will make judgements on the appropriateness and effectiveness of the provision, as outlined above.


Code of practice

9 The Agency publishes a Code of practice for the assurance of academic quality and standards in higher education, with sections addressing good practice in individual areas of academic management. Those sections relating directly to quality and standards provide both institutions and reviewers with a background against which judgements can be made. Reviewers will expect institutional systems to have at least an 'equivalent effect' to the precepts of the Code.

10 Academic reviewers should ensure that they are familiar with the Code. In particular they should be aware of the precepts in the following sections which deal directly with quality
and standards:

  • programme approval, monitoring and review;
  • assessment of students;
  • external examining; and
  • collaborative provision.

The precepts in these sections provide criteria against which academic reviewers can make their judgements on both subject provision and institutional management.



Intensity of scrutiny

11 There is a proper expectation that any system of quality assurance will be as efficient as possible and will consume no more overall resource than is necessary. To this end, the method used by the Agency:

  • provides transparency of process through the use of qualifications frameworks, subject benchmark statements, programme specifications and the Code of practice;
  • involves exchange of information between the subject and institutional review processes, thereby reducing duplication to a minimum;
  • allows institutions to negotiate the timing and aggregation of subject reviews. This enables external review to be aligned with internal review, re-validation or professional and statutory body timetables, should an institution so wish;
  • facilitates alignment of subject review with internal processes by spreading reviews over a period rather than imposing a 'snapshot' style review visit. Thus evidence from internal processes can be made available to reviewers on request, so that the need for the preparation and assembly of large amounts of documentation in advance of a visit is removed;
  • ensures that the amount of time taken to conduct a subject review is the minimum necessary to enable reliable judgements to be made.

12 Any process of scrutiny should evolve from one of universal intensity to one in which intervention is in inverse proportion to success. Where the Agency has confidence in an institution's ability to assure quality and standards, a lower intensity of review (a 'lighter touch') may be expected, but where there is no convincing evidence of robust and effective systems, greater intensity may be necessary. However, in the case of subject review, the Agency reserves the right to conduct a limited sample of reviews at a standard intensity, even in those institutions with the best record, as a means of refreshing the evidence base.

 

Part 1 - Subject Review

The review method

13 Part 1 of this Handbook describes the methods used for the review of subject quality and standards. Programmes taught at sub-degree, degree and postgraduate levels are reviewed, including programmes franchised to other UK institutions.


Understanding the process

Subjects and programmes

14 The Handbook refers to both subjects and programmes. The Agency reports on academic provision in 42 broad subject areas (listed in Annex K). For each subject, benchmark information (see paragraphs 34 to 36) is produced.

15 'Programme' refers to the programmes of study followed by students. Programmes may be offered at different levels (eg HND, honours degree or taught master's degree) within a single subject. A programme may be multi-disciplinary, for example a joint honours degree. The term 'programme' may refer also to the main pathways through a modular scheme, which itself may include several subjects.

16 The unit of review is the subject. Institutions may ask for a group of subjects to be reviewed together, particularly where they are linked by joint programmes or by modular pathways.

Main features of the method

17 This section of the Handbook describes the main features of the method used to conduct subject reviews. The features are grouped under the following headings:

  • Preparing for review;
  • Points of reference for review;
  • Conducting the review;
  • Judgements and reports.


Preparing for review

Scope and preference surveys

18 The 2000-2006 cycle of subject review is divided into two three-year periods. Approximately one year before the commencement of each period, every institution will be asked to complete a 'scope and preference' information form. This is designed to gather information about the range of subjects offered, the programmes to be included under each subject heading, the estimated numbers of student FTEs for each programme, the institution's preferred timing for the review of each subject and any preferences for subjects to be reviewed together. This advance information provides a basis for further discussions with each institution to plan and agree the scope and timing of academic reviews for the three-year period.

19 As far as possible, the Agency will seek to accommodate the preferences for the timing of reviews expressed by institutions. However, the balance of the Agency's overall workload needs to be maintained across the three years of the review programme, and the overall schedule must take into account the availability of academic reviewers with appropriate expertise. Where 'scope and preference' responses indicate that a preferred timing is to enable a review to coincide with a scheduled internal review, or with an accreditation visit by a professional or statutory body with an interest in the same provision, priority will be given to accommodating such preferences.


Initial profiles

20 The integrated method described in this Handbook requires mutual exchange of information between subject and institutional reviews. Reports of institutional reviews will inform the approach to the review of individual subjects by providing an indication of the degree of confidence that can be placed in the effectiveness of institutional quality assurance processes. This will help determine the intensity of scrutiny that is appropriate for subject review. In the early stages of the 2000­2006 review cycle, few institutional reviews will have been conducted with the aim of providing such information. Accordingly, the Agency will use instead specially prepared initial profiles to commence a dialogue with each institution about the overall approach to and intensity of review.

21 The Agency has considerable information about individual institutions, derived from previous quality assessments and quality audits. This information will be used to prepare a profile of each institution, which will be shared with the institution concerned, but not published. Each profile will summarise evidence related to the broad aims of the institution, the appropriateness of the curricula to deliver intended learning outcomes, the quality of curricular delivery and student learning, student support and progression, learning resources and student achievement. Further details are provided in Annex B.

22 On the basis of the profile, the Agency will discuss with each institution the approach to subject reviews for the subsequent three-year period. For each subject, the initial profile, previous subject report(s), and other relevant and available information (eg accreditation reports by a professional body) will be used to reach a provisional view on the amount of reviewer activity needed to allow reliable judgements to be made. Subsequently, analysis of the self-evaluation document by the team of academic reviewers will be used to confirm, or vary, the likely intensity of the review. Finally, once the review has commenced, it is open to reviewers to indicate if they believe that the review requires more or less reviewer time than agreed. In that event, the Agency will contact the institution in order to agree any change to the pattern of review activity.

23 Initial profiles and reviews will not be used to band institutions, with differing intensities of scrutiny applying to each band. Higher education institutions are large, complex organisations with balances of strengths and weaknesses. They cannot be differentiated in such a simplistic way. Individual judgements will be made that will take into account also the perceptions of the institution about its own strengths and weaknesses, its effectiveness in identifying and addressing any weaknesses, and any trends identifiable from the overall track record of results of subject reviews.


Self-evaluation documents and programme specifications (Annexes C, D)

24 The self-evaluation document is central to the process of subject review, and fulfils two main functions. First, it is intended to encourage the subject provider to evaluate the quality of the learning opportunities offered to students and the standards achieved by them. It provides an opportunity for the staff of the subject provider to reflect on 'what do we do?', 'why we do it', and 'why do we do it in the way that we do?'. Academic reviewers will expect to see evidence of careful self-analysis. This should involve an evaluation of the perceived strengths of the provision, with reference to the evidence which justifies the statements made, and of weaknesses, where these are recognised. Where weaknesses are acknowledged, the subject provider is encouraged to discuss the issues and the steps being taken to bring about improvements.

25 Second, the document provides a framework for a process of academic review based on the testing and verification of statements made by subject providers. The document should reflect on current provision in a manner that evaluates both strengths and weaknesses, indicates the changes that have taken place since earlier external reviews, and considers what may be necessary to change in the future. It is the most important of the small number of documents made available to reviewers in advance of a review.

26 Guidelines for producing self-evaluation documents appear at Annex C. These guidelines are intended to ensure that institutions address the relevant issues, and include the material needed by academic reviewers prior to a review in order to obtain an accurate picture of the provision. A document prepared for internal review processes may be able to meet the guidelines, thus removing the need to prepare a separate self-evaluation purely for external review.

27 Institutions may wish to aggregate subjects for the purposes of review. This will be acceptable where the grouping is of cognate subjects, and can be justified by reference to the ways in which combinations of subjects are offered to students, eg through a modular scheme. Subjects proposed for aggregation must be proposed for review in the same year. Self-evaluation documents must cover all provision that is included within any such grouping, either as a single document, or as a coherently related set of documents. If reviewers find that programmes within a grouping, of a particular kind, or at a particular level, are performing significantly differently from the generality of programmes in that grouping, separate judgements will be made and reported on those programmes.

28 Specifications for each of the programmes included must be annexed to the self-evaluation in order to make them readily available for academic reviewers. The specifications should provide the core factual information about the programmes, allowing the self-evaluation itself to provide a reflective analysis of the provision and its development.

29 Programme specifications should make explicit the intended outcomes in terms of knowledge and understanding, and skills and other attributes. They should help students to understand what is expected of them. They should enable teaching teams to articulate the teaching and learning methods that enable students to achieve the outcomes; the assessment methods that enable achievement to be demonstrated; and the relationship of the programme and its study elements to the qualifications framework and to any subsequent professional qualification or career path.

30 To be most effective, programme specifications should become part of an institution's curricular planning, approval and review processes. Preparation of them is an opportunity for teaching teams to reflect on the purposes and intended outcomes of their provision. Programme specifications will be the starting point for academic reviewers as they seek to understand the intended outcomes and the assessment methods for the programmes under review. The Agency has provided information to assist in the preparation of these documents (Annex D).

31 The self-evaluation will be required approximately one month before commencement of the period (usually the academic year) in which the subject review will take place. If a review period commences later in the academic year (for example, to coincide with an internal review), a submission date will be agreed with the institution.


Points of reference for review of standards

Qualifications framework

32 The framework of higher education qualifications provides reference points to be used to determine whether the intended outcomes for programmes, and actual student achievement are appropriate to the level of the qualification awarded. The framework helps provide public assurance that qualifications bearing similar titles represent similar levels of achievement. There are two qualifications frameworks, one for England, Wales and Northern Ireland, and one for Scotland. There are points of alignment between the two frameworks, particularly at postgraduate level. This Handbook refers to the qualifications framework in the singular. This is to avoid confusion as, in general, only one framework will apply to any one HE institution.

33 Implementation of the framework will take place over the first six-year cycle of academic review. In making judgements about the provision under review, academic reviewers will recognise that institutions may need time to ensure that all of their provision is aligned with the framework.


Subject benchmark statements

34 Subject benchmarks are statements which represent general expectations about standards for the award of qualifications at a given level in a particular subject area. Benchmarking is not about listing specific knowledge; that is a matter for institutions in designing individual programmes. It is about the conceptual framework that gives a discipline its coherence and identity; about the intellectual capability and understanding that should be developed through the study of the discipline to the level in question; the techniques and skills which are associated with developing understanding in the discipline; and the intellectual demand and challenge appropriate to study of the discipline to the level in question.

35 Reviewers will use relevant benchmark statements as a means of determining whether the intended learning outcomes of individual programmes are appropriate. Institutions should be able to demonstrate how subject benchmark statements have been used to inform decisions about the intended outcomes of programmes, and in calibrating the overall demands of the assessment framework. The Agency recognises that institutions have their own cycles of programme review, and that it is through these that any changes, to reflect benchmark statements, will come. Reviewers will be sensitive to the challenges that institutions will face in working with benchmark statements for the first time. Nevertheless, benchmark statements provide an immediate starting point for discussion and reflection within teaching teams, and between teaching teams and academic reviewers, about the appropriateness of the outcomes of their programmes. Over time, benchmark statements will be revised in the light of feedback from subject communities.

36 For some programmes more than one benchmark statement may be relevant, while in some specialist, innovative or inter-disciplinary fields there may not be any statement that is of direct relevance. In such cases, the level descriptors of the overall qualifications framework, and the guidance in the Code of practice section on programme approval, monitoring and review, will assist institutions in ensuring that their provision meets generally accepted standards for a given level of award. In all cases the institution remains responsible for identifying and assuring the standards of its awards and for ensuring that they reflect appropriate external indicators.



Conducting the review

Peer review

37 Peer review enables judgements to be made by those who understand the subject under scrutiny and who are familiar with teaching and learning processes. It enables judgements to be credible to, and to command the respect of, subject providers. For a peer review process to have credibility with external stakeholders, such as employers and potential students, judgements must be made in a transparent manner, and reported publicly.

38 Subject review is carried out by a team of subject specialists, whose main responsibility is to read, analyse and test the self-evaluation produced by the institution, and to gather whatever further evidence they need to make the judgements described later in this Handbook. Subject specialist reviewers are drawn mainly from the higher education sector, although nominations and applications from industry, commerce and the professions are actively sought by the Agency. Subject specialists are trained and briefed by the Agency before taking part in reviews. Further details of their function may be found in Annex A. For a note on the composition of teams, see Annex G.

39 Management of each review and liaison with the institution is the responsibility of a review coordinator. Coordinators are not specialists in the subject under scrutiny, but individuals with extensive experience of HE and quality assurance. Each is provided with training specific to their coordinating function and each attends at least one specialist reviewer training course. Further details of their function are given in Annex A.


Review against the broad aims of the provider

40 Subject providers should be able to set out clearly the broad aims of their programmes. Such statements should indicate, in general terms, what the subject provider is seeking to achieve, how these aims relate to external indicators (such as subject benchmark statements, professional body requirements, or employer expectations), and the general attributes of its graduates. These broad aims will provide the context in which the review takes place. Accordingly, the Agency will expect the aims of the provision to be stated at the beginning of any self-evaluation and will publish the aims in the subject review report. The statement must be sufficiently clear to allow intended learning outcomes to be set which ensure that the aims are achieved, and to provide a sound basis for the subject review to be planned and carried out.


Review over an extended period

41 Spreading the available reviewer days over a period should allow reviewers to gain a better understanding of the subject provision than is possible in a concentrated, 'snapshot' use of the same number of days. It reduces the need to prepare large amounts of documentation in preparation for a single visit. Spreading review activity over a period will enable academic reviewers to observe, when appropriate and by arrangement with the subject provider, internal quality assurance 'events', such as programme committee meetings, programme approval events, and examination and assessment boards, and thus to use documentation already prepared for internal quality assurance purposes.

42 In addition, academic reviewers will be able to:

  • visit the subject provider at mutually convenient times over a period;
  • attend relevant internal events as observers by invitation;
  • carry out off-site desk-based analyses of information and data;
  • vary the pattern of visiting according to the confidence placed in the quality assurance systems operated by the subject provider.



Judgements and reports

Judgements on academic standards

43 Judgements are made on the academic standards in each subject under scrutiny. Where a group of subjects is aggregated so as to be reviewed together, a separate judgement is made on each to enable strengths and weaknesses in individual subjects to be identified. These judgements focus on whether intended learning outcomes are appropriate and whether the outcomes achieved are consistent with the intentions. The judgements are not graded; either the intended outcomes are appropriate and are achieved, or they are not. Reviewers will make their judgement accordingly, that there can be confidence, or not, in the standards of the provision. If standards are being achieved, but reviewers have concerns about the ability of the institution to maintain them into the future, a judgement of 'limited confidence' may be made. If a failure to achieve standards has occurred in programmes at one level only, and there is confidence in standards at other levels, the failing level will be identified separately. The issues which academic reviewers must address in order to make these judgements are described in greater detail in later sections of this Handbook and in the aide-mémoire for academic review (Annex E).

44 Where an expression of 'limited confidence' in academic standards is made, academic reviewers must identify areas where improvement is needed. The subject provider may then be asked to prepare an improvement strategy, implementation of which is monitored by the Agency. If a judgement is made that standards are not being achieved, there will be a further, formal review by the Agency within one calendar year. If standards continue not to be achieved, funding is potentially at risk.


Judgements on quality of learning opportunities

45 For each subject, and through the same process of scrutiny, the quality of the provision is reviewed. Judgements are made about the extent to which the three aspects of provision outlined below contribute to the achievement of the intended learning outcomes in the subject area under review:

  • teaching and learning;
  • student progression;
  • effective utilisation of learning resources.

This part of the academic review process focuses on the learning opportunities that enable students to achieve the academic standards established by the subject provider.

Judgements on groups of programmes and individual programmes

46 Each review of a subject will usually cover a number of programmes, sometimes at different levels. Subjects may also be aggregated for the purposes of review. In most cases, a judgement on standards or quality may be made with confidence for all programmes in that subject area, or grouping. However, if a particular programme, subject or level is performing significantly better or worse than the generality of provision within the unit of review, it will be commented on separately in the review report. This is designed to ensure that there is no averaging of overall performance that could conceal good practice deserving praise or shortcomings that require attention.


Subject review reports

47 At the end of each subject review, a report of approximately 4,000 words is published which describes the findings of the team of reviewers. It is the main documented outcome of the review process and provides the main feedback to the institution and its subject provider. The report will include:

  • a brief description of the review method;
  • the overall aims of the subject provider;
  • an evaluation of the quality of the learning opportunities provided and the academic standards achieved;
  • the conclusions reached and the judgements made;
  • a one-page summary of the main conclusions.

48 Reporting on the quality of learning opportunities takes the form of a narrative commentary on each of the three aspects of provision. The commentaries will identify particular strengths and weaknesses and will place each aspect into one of three categories: 'failing', 'approved' or 'commendable'. Within the 'commendable' category, exemplary features may also be identified. Further details of the categories and the criteria for judgements are given in paragraphs 88 to 94 of this Handbook.

49 Reporting on standards takes the form of a narrative commentary which addresses strengths and weaknesses by reference (where appropriate) to the relevant sections of the Code of practice, subject benchmark statements, and the qualifications framework, and leads to the overall judgement. The narrative may identify matters for particular commendation or matters of concern, including the ability of the institution to maintain standards into the future. While a point of concern need not mean that standards are not being achieved, several points of concern may mean that there is limited confidence in the ability of the subject provider to maintain standards. Where weaknesses exist which, if uncorrected, could imperil standards, it is important that there is a mechanism to ensure that appropriate corrective action is taken.

50 Potential students, employers and other members of the public require clear and concise information about the provision within a subject that allows them to distinguish between different providers of similar programmes. For this purpose, each subject report will contain a one-page summary of the findings and judgements made on both quality and standards.

51 Draft reports are sent to the institution for comment on matters of factual accuracy. Feedback on the review is provided to the institution through the draft report. As the review may take place over a period, and include both off-site consideration of written material and on-site observation and meetings, there is not a formal concluding session at which oral feedback is given to the institution. The Agency aims to publish reports, on its web site, within 20 weeks from the end of a review.

52 On completion of the schedule of the reviews in a particular subject area, a subject overview report is published. In the light of this report the subject community will be invited to consider any adjustment to the subject benchmark statement as may be necessary. The overview reports are designed to record the findings of the review teams and to promote best practice.

How the process works

Preparation for subject review

Liaison between the institution and the review team

53 Institutions may nominate a subject review facilitator for each review to facilitate liaison between the team of reviewers and the institution, and to ensure that the team obtains accurate and comprehensive information about the subject provision and its institutional context. Institutions may find it convenient to nominate the same facilitator for a number of reviews. The Agency offers briefing sessions for facilitators.

Further details of the role of a subject review facilitator are given in Annex F.

Advance planning and preparatory meetings

54 Advance planning begins with the return by institutions of replies to the 'scope and preference' enquiry made before the start of the first or second half of the review cycle. If the Agency is able to agree the proposed distribution of subjects for review across the three year period, the institution will be notified of this. If there are operational reasons why the preferred pattern cannot be accommodated, the matter will be discussed with the institution, with a view to reaching agreement on a revised pattern. In the absence of agreement, the Agency will notify the institution of the distribution of subjects that will be adopted over the period.

55 Some six months before the start of each academic year, the Agency will initiate a discussion with each institution about the pattern of review that appears appropriate for each subject due to be reviewed in that year. These discussions will take as a starting point the initial profile, based on historical evidence about the institution as a whole; evidence about performance in the subject(s) in question, from the previous subject review(s); and any available accreditation reports by professional or statutory bodies. The institution will be invited to suggest the most appropriate pattern of review activity, having regard for the timing of any relevant internal events or accreditation visits. The Agency will then seek to reach agreement with the institution on the pattern, timing and intensity of review activity for each subject, or group of subjects. In the absence of agreement, the decision of the Agency will be final.

56 The Agency will consider the intensity of scrutiny likely to be necessary to enable valid and reliable judgements to be made. On the one hand, it will seek to verify continuity of a record of good practice, which might indicate that a relatively low intensity is needed. On the other hand, where there has been less good or unsatisfactory practice, a relatively higher intensity is likely to be needed, to enable the reviewers to establish whether improvements have been made. Statements from academic audit and institutional review reports about the level of confidence that may be placed in institutional management of quality and standards will be taken into account.

57 Indicators of good practice, drawn from institutional and subject review reports, might include:

  • positive commendations;
  • few negative statements;
  • few recommendations for attention;
  • no 'essential' recommendations from institutional review reports;
  • evidence of management of quality and standards in line with the expectations of good practice expressed in the Code of practice;
  • no suggestions of differential performance by level or mode.

58 Indicators of a need for improvement, drawn from institutional and subject review reports might include:

  • recommendations for action;
  • issues highlighted for attention;
  • grade(s) 2 or below (or equivalent) in subject review graded profile(s);
  • 'essential' recommendations from institutional review reports;
  • evidence that the management of quality and standards is not in line with the expectations of good practice expressed in the Code of practice;
  • suggestions of differential performance between levels and modes.


Academic review teams

59 The number of academic reviewers in each team will reflect the size, range and complexity of the education provided. As far as possible, within the resources available, the Agency will match the expertise of the team of reviewers with the broad specialisms of the subject provision under scrutiny. The criteria for team composition are in Annex G. The role of reviewers is set out in Annex A.

60 A register of reviewers is published by the Agency.

61 The proposed composition of a review team will be notified to the institution. Any concerns about the composition of a team, for example because of a conflict of interest of which the Agency is unaware, must be notified to the Agency within four weeks. If a review is to be combined with accreditation by a professional or statutory body, composition of the team will be discussed with that body.


Analysis of the self-evaluation

62 The review team will use the self-evaluation prepared by the institution to help to set priorities for and to plan the review. Reviewers will consider whether the broad aims of the provision are clear; and whether the intended learning outcomes allow the aims to be achieved. The reviewers will consider whether:

  • the aims are an adequate expression of the broad educational purposes of the provision;
  • the aims reflect appropriately any relevant subject benchmarks;
  • there is a clear relationship between the broad aims and the intended learning outcomes.

If the aims, or their relationship with the intended learning outcomes, are unclear, the provider will be asked for clarification before the review proceeds.

63 Each section of the self-evaluation will be assessed to ensure that it is evaluative, rather than merely descriptive. An institution may be asked to revise a document if it fails to provide a suitable basis for the review.

 

Conducting the review

General approach

64 Reviews are intended to be conducted in a spirit of dialogue and cooperation between the institutions, their subject staff, and the review teams. Reviewers must be able to gather sufficient evidence on the subject provision to allow them to test statements made in the self-evaluation, and to form robust judgements on the quality and standards of the provision.

65 At its first meeting, the review team will consider the:

  • self-evaluation and any other documentation supplied by the institution prior to the review;
  • scope and nature of the provision;
  • main matters for review and judgement;
  • role of the facilitator in relation to the conduct of the review;
  • allocation of individual responsibilities amongst the members of the team;
  • programme activities, both on- and off-site, required for the review;
  • pattern and timing of visits to the subject provider.

66 The review team will then hold an initial meeting with the subject provider. The provider may wish to make a brief presentation to introduce the provision to be reviewed, and to describe any developments since the self-evaluation was prepared. The review coordinator will remind both the team and the institutional representatives of the method and protocols of review. Reviewers will agree an outline programme for the review and will establish the:

  • range of student work which can be made available for scrutiny, and the extent to which this constitutes a representative sample of student achievement in the subject;
  • nature of relevant documentation held by the institution and its availability for scrutiny by reviewers;
  • range and timing of internal quality assurance 'events', such as programme committees, faculty boards (or equivalent) or examination boards, which might provide documentary evidence and/or be attended (by agreement with the subject provider) by reviewers;
  • timing of any related visits by professional or statutory bodies;
  • probable agenda and timing of meetings with academic staff, students and former students;
  • other practical arrangements for the review.

67 The review team will not normally ask for specially-prepared documentation, other than the self-evaluation. It will endeavour to make use of existing documentation used for internal processes related to quality and standards. The pattern of review activity over a period will enable material to be requested well in advance of any visit to the institution. Annex H summarises the range of documentation to which reviewers expect to have access and also provides guidance on the student work to be made available. In most cases, subject providers will be able to identify appropriate samples from work completed by students in the current academic year or from materials kept routinely for examination purposes.


Testing the self-evaluation and gathering evidence

68 The review method provides a structure for the self-evaluation, and for the visits, judgements and reports made by reviewers. It involves addressing:

  • subject provision and aims;
  • learning outcomes;
  • curricula and assessment;
  • quality of learning opportunities;
  • student achievement;
  • maintenance and enhancement of quality and standards.

As may be seen in Annexes C and E, the self-evaluation is written to this framework and the aide-mémoire for subject review is structured similarly.

69 Assessment of the quality of subject provision and of academic standards achieved is through a combination of direct observation and scrutiny of documentary evidence. Documentary evidence includes internal reports from committees, boards and individual staff with relevant responsibilities; and external reports from examiners, employers, validating and accrediting bodies. Emerging judgements are refined and tested against as wide a range of evidence as possible; for example, the views expressed in meetings by staff or by students are tested against the documentation provided.

70 Academic reviewers may not need to make direct observations of teaching where a subject provider can provide evidence of good quality delivery. Such evidence is likely to come from internal peer review; from student questionnaires and other arrangements for gathering feedback; from the deployment of learning resources; and from student performance in assessments. Direct observation of teaching will be required if:

  • there are issues that reviewers feel would be best addressed by such observation;
  • observation might help confirm a judgement about exemplary provision;
  • there is insufficient other evidence that effective delivery is being achieved; or
  • there are indications that the learning opportunities for students are less than satisfactory.

A note on observation of teaching is at Annex I.

71 Each review includes a number of meetings between members of the institution and reviewers to consider the various aspects of provision related to quality and standards. The review coordinator is responsible for ensuring that the review team meets sufficiently often to consider the accumulating evidence and the team's findings. If such meetings take place at the institution, the team may find it helpful to include the facilitator, who can provide factual information relevant to the team's discussions. However, the facilitator may not attend team meetings or parts of meetings at which direct discussion of judgements takes place.

72 The views of students are important, but should be treated as one source of evidence among several. Meetings with students enable reviewers to establish the students':

  • understanding of the overall aims and intended learning outcomes;
  • responses to the teaching;
  • learning experiences;
  • views on academic support and the resources available;
  • feedback and representation arrangements.

The facilitator does not attend these meetings but may be consulted about the issues raised by the students. Arrangements for meetings with students are set out in Annex J.

The reviewers may also seek the views of former students, their employers, and representatives from relevant industries or professions.

73 All reviewers are expected to identify, share, consider and evaluate evidence related to the programmes under scrutiny. Reviewers should keep notes of all meetings with staff and students, of their observations, and of comments on the quality of students' work and its assessment. Notes should be analytical rather than merely descriptive, and should refer to sources of information as well as to direct observations. Strengths and weaknesses should be summarised. Circulation of notes within the review team, and collation by the review coordinator, will assist in developing a collective evidence base on which judgements can be made.

74 Team meetings are used to review the evidence gathered, form preliminary judgements, and determine which issues require further exploration. Reviewers are expected to evaluate how the evidence gathered compares with the self-evaluation prepared by the subject provider and to test the strength of the evidence adduced to support the judgements. Discussion of the emerging judgements must involve the whole review team.

 

Making judgements

Evidence

75 The subject review aide-mémoire at Annex E provides guidance on the questions likely to lead to the evidence necessary to make judgements about academic standards and the quality of learning opportunities.

76 The collective judgements of the review team should be informed by the totality of evidence accumulated. Accordingly, all team members are expected to share information they have gathered which appears relevant to any matter on which judgement is to be made.

77 Judgements about academic standards will be made on the appropriateness of the intended learning outcomes set by the subject provider in relation to subject benchmark statements, qualification levels and the overall aims of the provision; on the effectiveness of curricular content and assessment arrangements in relation to the intended learning outcomes; and on actual student achievement.

78 Judgements about the quality of learning opportunities will be made on the effectiveness of teaching and the learning opportunities provided; on the effectiveness of the use of learning resources (including human resources); and on the effectiveness of the support provided to students to enable them to progress within the programme.

79 Reviewers will seek to establish that:

  • intended learning outcomes are clearly expressed, and reflect appropriately relevant subject benchmark statements and the overall aims of the programme;
  • curricular content supports the intended outcomes, and that assessments measure appropriately their achievement;
  • there is effective communication to staff and students, so that learners and teachers know what is expected of them.

80 Reviewers will seek to establish, by reference to subject benchmark statements where appropriate, that the design of the curriculum facilitates:

  • acquisition of knowledge and understanding;
  • acquisition of cognitive skills;
  • acquisition of subject-specific skills, including practical and professional skills;
  • acquisition of transferable skills;
  • progression to employment and/or further study.

Judgements on academic standards

81 A single, threshold judgement is made about academic standards. Having regard to all the matters listed below, reviewers will decide whether they have confidence in the academic standards of the provision under review. A 'confidence' judgement will be made if reviewers are satisfied both with current standards, and with the prospect of those standards being maintained into the future. If standards are being achieved, but there is doubt about the ability of the institution to maintain them into the future, reviewers will make a judgement of 'limited confidence'. If, in relation to any of the matters listed below, reviewers feel that arrangements are inadequate to enable standards to be achieved or demonstrated, then their overall judgement will be that they do not have confidence in the academic standards of the provision under review.

82 Reviewers will assess, for each programme, whether there are clear learning outcomes which appropriately reflect applicable subject benchmark statements and the level of the award. Subject benchmark statements represent general expectations about standards in an academic discipline, particularly in relation to intellectual demand and challenge. The qualifications framework sets expectations for awards at a given level more generally. Reference points are thereby provided to assist reviewers in determining whether provision is meeting the standards expected by the academic community generally, for awards of a particular type and level. If the intended learning outcomes are found not to match those expectations, it would be unlikely that reviewers could have confidence in the standards of the provision. An example of potential failure would be if a postgraduate programme has learning outcomes set at an undergraduate level only.

83 Reviewers will assess whether the content and design of the curriculum are effective in achieving the intended programme outcomes. It is the curriculum that ensures that students are able to meet the intended outcomes of the programme. Providers should be able to demonstrate how each outcome is supported by the curriculum. 'Curriculum' for this purpose includes both the content necessary to develop understanding and the acquisition of knowledge, and the opportunities to develop practical skills and abilities where these are stated as intended outcomes. If significant intended learning outcomes are found to be unsupported by the curriculum, it would be unlikely that reviewers could have confidence in the standards of the provision.

84 Reviewers will assess whether the curriculum content is appropriate to each stage of the programme, and to the level of the award. Providers should be able to demonstrate how the design of the curriculum secures academic and intellectual progression by imposing increasing demands on the learner, over time, in terms of the acquisition of knowledge and skills, the capacity for conceptualisation, and increasing autonomy in learning. Reviewers will have regard to the guidance on programme design in the section of the Code of practice on programme approval, monitoring and review.

85 Reviewers will assess whether assessment is designed appropriately to measure achievement of the intended outcomes. Providers should be able to demonstrate that achievement of intended outcomes is assessed, and that, in each case, the assessment method selected is appropriate to the nature of the intended outcome. There must also be confidence in the security and integrity of the assessment process, with appropriate involvement of external examiners. An assessment strategy should also have a formative function, providing students with prompt feedback, and assisting them in the development of their intellectual skills. There should be clear and appropriate criteria for different classes of performance, and these criteria should be communicated effectively to students. If significant intended learning outcomes appear not to be assessed, or if there are serious doubts about the integrity of the assessment procedures, it would be unlikely that reviewers could have confidence in the standards of the provision. Reviewers will have regard to the section of the Code of practice on assessment of students.

86 Reviewers will assess whether student achievement matches the intended outcomes and level of the award. Reviewers will consider external examiners' reports from the three years prior to the review, and will themselves sample student work. The balance between reliance upon the reports of external examiners and direct sampling of student work will depend on the confidence that reviewers have in the external examining arrangements of the institution. Regard will be had to the section of the Code of practice on external examining.

87 Where a review covers a number of subjects, separate judgements on standards will be made in respect of each subject. Where programmes are offered at more than one level, separate judgements will be made in respect of each level, if there are significant differences between them. In all cases, reports will contain a narrative commentary on strengths and weaknesses in relation to each aspect of the standards judgement.


Judgements on quality of learning opportunities

88 Judgements about the quality of the learning opportunities offered to students will be made against the broad aims of the provision and the intended learning outcomes of the programmes. Judgements will normally cover all provision within the scope of the review. However, if performance is significantly different in a subject area, or at a particular level, separate judgements will be made.

89 Reviewers will assess the effectiveness of teaching and learning, in relation to curriculum content and programme aims. They will consider large and small group teaching, practical sessions, directed individual learning, the integration of skills within curricula, and distance learning. Reviewers will evaluate the breadth, depth, pace and challenge of teaching; whether there is a suitable variety of teaching methods; the effectiveness of the teaching of subject knowledge; and subject-specific, transferable and practical skills.

90 Reviewers will evaluate student progression by considering recruitment, academic support, and progression within the programme. They will assess whether there is appropriate matching of the abilities of students recruited to the demands of programmes; and whether there are appropriate arrangements for induction and identification of any special learning needs. They will assess the effectiveness of academic support to individuals, including tutorial arrangements and feedback on progress. They will consider general progression within programmes, as well as non-completion rates.

91 In making judgements about learning resources, reviewers will assess whether the minimum resource necessary to deliver each programme is available, and will then consider how effectively resources are utilised in support of the intended learning outcomes of the programmes under review. Consideration will be given to the use of equipment (including IT), accommodation (including laboratories), and the library (including electronic resources). Reviewers will look for a strategic approach to the linkage of resources to programme objectives. Effective utilisation of academic, technical and administrative staff will be considered, as will the matching of the qualifications, experience and expertise of teaching staff to the requirements of the programmes.

92 Reporting on the quality of learning opportunities will place each of the three aspects of provision into one of three categories, 'failing', 'approved' or 'commendable', and will be made on the following basis:

  • provision makes a less than adequate contribution to the achievement of the intended outcomes. Significant improvement is required urgently if the provision is to become at least adequate. In the summary report, this judgement will be referred to as 'failing';
  • provision enables the intended outcomes to be achieved, but improvement is needed to overcome weaknesses. In the summary report, this judgement will be referred to as 'approved'. The summary will normally include a statement containing the phrase 'approved, but...', which will set out the areas where improvement is needed;
  • provision contributes substantially to the achievement of the intended outcomes, with most elements demonstrating good practice. In the summary report, this judgement will be referred to as 'commendable'.

93 Within the 'commendable' category, reviewers will identify any specific features of the aspect of provision that are exemplary. To be deemed 'exemplary', a feature must:

  • represent sector-leading best practice; and
  • be worthy of dissemination to, and emulation by, other providers of comparable programmes; and
  • make a significant contribution to the success of the provision being assessed. Incidental or marginal features do not qualify for designation.

The characteristics of exemplary features will, by their nature, vary between institutions and programmes. The criteria listed above will ensure that features identified as 'exemplary' will be broadly comparable in weight and significance.

94 If provision is found to be failing in any aspect of quality, or if reviewers have no confidence in the standards achieved, the provision will be regarded, overall, as 'failing'. It follows that all provision that is not failing is 'approved'. The report of the review will state whether or not provision is approved.


Maintenance and enhancement of standards and quality

95 Institution-wide systems for the management and enhancement of standards and quality are addressed through institutional review. Subject reviewers will gather evidence, not least from their discussions with staff and students and their scrutiny of external examiners' reports, on the operation of institutional systems in each subject area. The final section of the subject report will express their confidence, or otherwise, in the ability of the institution to maintain and enhance quality and standards in the particular subject. These views will inform the subsequent institutional review.

Next > Part 2 - Institutional review

TopTop