Handbook for academic review: England, 2004
For review of directly funded higher education in further education colleges

Part one

Introduction

1 This is the second edition of the Handbook for academic review (the Handbook), which will be used for reviews starting after October 2004. It is for use only for reviews of directly funded higher education (HE) in further education colleges (FECs) in England. The basic principles and processes of academic review remain unchanged and continue to comprise an approach focused on the establishment, maintenance and enhancement of academic standards and the quality of learning opportunities. The Quality Assurance Agency for Higher Education (the Agency) sought comment as to how the first edition of the Handbook for academic review, 2000 could be improved and focused amendments on the needs of FECs, Review Coordinators, specialist reviewers, subject review facilitators, academic staff and students. This second edition of the Handbook takes account of the suggestions for improvements. It is intended for use throughout the review period 2004 to 2006.

Background

2 The mission of the Agency is to safeguard the public interest in sound standards of HE qualifications and to encourage continuous improvement in the management of the quality of HE. To this end, the Agency carries out reviews of directly funded subjects in FECs (Annex A provides a list of the subjects/units of review designated by the Higher Education Funding Council for England (HEFCE) and institutional audits of higher education institutions (HEIs)). This Handbook describes the method and procedures for carrying out the review of directly funded HE provision in FECs in England.

3 HEFCE has a statutory responsibility to ensure that provision is made for assessing the quality of the education it funds and has a responsibility for reporting to government on standards and quality. All HE provision funded directly to FECs by HEFCE is subject to review, whether validated by an HEI or other awarding body. When a FEC receives funding from an HEI to deliver programmes, they are considered within the scope of the HEI's institutional or collaborative audit. The Agency is contracted by HEFCE to carry out review work but acts independently of them. The Agency's role is to review and report its finding to HEFCE. The Agency has no statutory powers to effect change, it is for HEFCE to act, or otherwise, on the findings of the review.

4 The main purposes of the reviews are:

to secure value from public investment through ensuring that all education for which funding is provided is of approved quality, and, where identified, to encourage speedy rectification of major shortcomings in the quality of education, and to enable judgements to inform HEFCE funding decisions;
to encourage improvements in the quality of education through the publication of review reports and overview reports, and through the sharing of best practice; and
to provide, through the publication of reports, effective and accessible public information on the quality of HE.

5 The Agency provides public and objective information through review reports which address two interdependent areas:

reporting on academic standards in a subject is concerned with the appropriateness of the intended learning outcomes set by the subject provider, in relation to relevant subject benchmark statements, qualification levels and the overall aims of the provision, the effectiveness of curricular content and assessment arrangements, in relation to the intended learning outcomes, and the achievement of these outcomes by students;
reporting on the quality of learning opportunities in a subject is concerned with the effectiveness of the teaching, the academic support and the learning resources in promoting student learning, achievement and progression across the various programmes involved in the review.

6 The Agency carries out academic reviews of taught HE programmes only. Research degrees are not included in these reviews. Reviews are structured in one of three ways.

7 These are:

single subject review, in which one review includes all of the directly funded provision at HE level within a subject, as defined by the HEFCE units of review, for example, computing or art and design;
parallel reviews, in which up to three subjects/units of review may be considered. For each subject/unit of review there will be a separate self-evaluation, a separate review team, judgements and a separate published review report. Reviewers from each team may meet together with the college to share evidence. Colleges may wish to request parallel reviews to reduce the number of meetings staff may need to attend if the three reviews are scheduled to take place separately. Examples include:

Building and surveying parallel to psychology;
Mathematics, statistics and operational research parallel to music;

aggregate reviews, in which normally no more than three related subjects/units of review may be reviewed together within one review. In this case, there is one self-evaluation that covers all of the subjects/units of review, one review team with specialists for each of the units of review, separate standards judgements for each subject reviewed and, usually, one judgement for each of the aspects of the quality of learning opportunities. There is one published report. Examples of aggregate reviews include:

Art and design, with dance, drama and performance arts, and with communications, media, film and television studies;
Computing with engineering.

Features of academic review

8 Academic review is based on self-evaluation. The task for reviewers is to test, by means of their own observations and analyses of the evidence provided by the college, the statements made in a self-evaluation. The review aide-mémoire in Annex B provides guidance on the questions likely to lead to the evidence necessary to make judgements about academic standards and the quality of learning opportunities. The reviewers will make judgements on the appropriateness and effectiveness of the provision in academic standards and quality of learning opportunities, as outlined above. Additionally, reviewers will report on the maintenance and enhancement of standards and quality. To assist them, reviewers will make reference to the Academic Infrastructure (see part two of this Handbook) developed by the Agency in consultation with providers of HE.

9 The key features of academic review include:

a focus on the students' learning experience;
peer review: teams include current lecturers from HEIs and FECs and, whenever possible, for vocational subjects, reviewers from the relevant employment sector (Annex C provides details of the role and responsibilities of reviewers);
flexibility of process to minimise disruption to the college; there is negotiation between the Agency and the college about the timings of the review;
a process conducted in an atmosphere of mutual trust; the reviewers do not normally expect to find areas for improvement that the college has not identified in the self-evaluation;
an emphasis on the maintenance and enhancement of academic standards and the engagement with the Academic Infrastructure;
use of the self-evaluation as the key document; this should have a reflective and evaluative focus;
an onus on the college to provide all relevant information; any material identified in the self-evaluation should be readily available to reviewers; and
evidence-based judgements.

10 There is a proper expectation that any system of quality assurance will be as efficient as possible and will consume no more overall resource than is necessary. To this end, academic review:

makes use of existing information and documented evidence, and prefers internal review documents to be submitted as the self-evaluation. Reviewers will use college information and evidence that is available electronically, for example, on an intranet site;
ensures that the amount of time taken to conduct a review is the minimum necessary to enable reviewers to make judgements;
provides transparency of process through the use of the Academic Infrastructure;
includes the role of the subject review facilitator, who is a member of the college staff, to assist effective communication between the reviewers and the college; and
is monitored closely by Agency officers.

11 The Agency makes great efforts to assure the quality of the review process. An Agency officer is assigned to each review for all of the review process. This involves such matters as approving the carefully selected review team before it is proposed to the college, confirming that the self-evaluation forms a basis for the review to proceed, monitoring the progress of the review by means of the electronic folder set up by the Agency to support the review. Sometimes, as part of the Agency's normal quality assurance activities, the officer may observe the review. Officers also follow the progress of the judgement meeting and the full process of producing the report.

The review process

12 The review process comprises three stages:

preparation for the review;
the review period;
the reporting stage.

Annex D provides a timeline for the review stages and a flowchart of relevant documents.

Stage 1: Preparation for review

13 The process begins with the Scope and Preference Survey with colleges supplying the Agency with information about their programmes. This information is used by the Agency to establish a programme of reviews and to select the proposed review teams. Annex E sets out the criteria for review team composition. On completion of any further enquiries, the Agency opens a dialogue with each college, involving such matters as the appropriate timing of the review.

14 The college submits its self-evaluation, including programme specifications, to the Agency no later than two months before the agreed initial meeting date. The self-evaluation should be submitted to the Agency with a standard cover sheet (Annex B, Stage 1). Once the Agency has checked the cover sheet, the self-evaluation is sent to the Review Coordinator (the review team leader) who analyses it against a standard template to ensure that it forms an appropriate basis for the review to proceed (Annex B, Stage 2). The Agency officer with responsibility for the proposed review will check the self-evaluation to ensure that it has the required contents and that a review can proceed (Annex B, Stage 3). When the Agency has notified the college that it accepts the self-evaluation as the basis for review, the college sends copies of the self-evaluation to the review team members. The specialist reviewers read and comment upon the self-evaluation; the Review Coordinator uses their comments to help plan and set priorities for the review.

15 A preliminary meeting takes place between the Review Coordinator and relevant staff identified by the college, normally no later than four weeks before the start of the first visit of the whole review team to the college, the Initial meeting. At the preliminary meeting, the college may be invited to send further documents to the reviewers before the Initial meeting. The standard agenda for the preliminary meeting is included at Annex F. It is important that the college prepares for this meeting and is ready to discuss each item on the agenda by, for example, ensuring that they have up to date student data and timetables available at the meeting. The preliminary meeting provides college staff with a valuable opportunity to clarify their understanding of the review method. The Review Coordinator will ensure that a record of the agreements made at the preliminary meeting is shared with the college and the reviewers.

16 Annex G summarises the range of documents that reviewers may typically expect to have access to and also provides guidance on the student work to be made available.

Stage 2: The review period

17 The main period of review activity normally lasts six weeks from the Initial meeting of the reviewers and the college staff to the judgement meeting. Reviewers will spend some of this time in the college. It is essential that reviewers are able to gather sufficient evidence to allow them to test statements made in the self-evaluation, and to form robust judgements on the quality and standards of the provision. Reviewers reach collective judgements at the judgement meeting at the end of this period.

18 A number of key meetings are held during the review period. These will be with subject and other staff from the college, current and former students, and, where appropriate, employers. Annex H provides an indicative agenda for meetings with current and former students, and Annex I sets out a draft agenda for meetings with employers. The subject review facilitator, appointed by the college, will be invited to attend all reviewers' meetings except those with current and former students, employers and meetings where judgements are discussed. Further details of the role of a facilitator are given in Annex J.

First review team meeting

19 The reviewers, together with the subject review facilitator, will meet before the Initial meeting to discuss the review schedule and to share their early perceptions. The Review Coordinator will ensure that there is a shared understanding of the nature and purpose of the review. The reviewers will agree key questions for discussion with staff. The agenda for this meeting will:

clarify and confirm the scope and nature of the provision;
reiterate the key features of the process of review;
restate the required outcomes from the review process: judgements and a published report;
clarify the role of the subject review facilitator in relation to the conduct of the review;
confirm the reviewers' understanding of, and comments on, the self-evaluation and any other documents supplied by the college before the review;
identify key questions for discussion with the college;
agree the programme of activities, including their timing and location, for the review.

Initial meeting with the college

20 With the agreement of the reviewers, the subject provider may make a brief presentation, typically no more than 10 minutes, to introduce the provision to be reviewed and to describe any developments since the self-evaluation was prepared. The Review Coordinator will remind both the reviewers and the college representatives of the method and protocols of review and the schedule agreed so far. The college and reviewers will wish to confirm:

the nature and range of student work available for scrutiny and the extent to which this constitutes a representative sample of student achievement in the subject;
the availability of relevant documents held by the college;
the range and timing of internal quality assurance events, such as programme committees, faculty boards or examination boards, which might provide evidence. The reviewers will need to decide whether attendance at such activity makes effective use of their time, or whether relevant evidence may be gathered by other means. If reviewers wish to attend an event, this must be with the agreement of the college;
the agenda and timing and composition of meetings with college staff, current and former students, and, if applicable, employers; and
other practical arrangements for the review.

Other meetings

21 Other meetings will be arranged with staff to discuss academic standards, the quality of learning opportunities and the maintenance and enhancement of academic standards and quality. There is no fixed pattern of meetings, as the reviewers and the college will need to agree a plan for each review which enables the reviewers to gain the evidence they need to make judgements with minimal disruption to the college. The college may wish to consider who is appropriate to attend these meetings. The review may also include meetings with employers and work placement providers.

22 This flexibility in the application of the academic review method means that reviews within the same college may follow different patterns.

23 The programme for the review may change, with agreement between the Review Coordinator and the subject review facilitator, depending on the review's progress. Critically, the Review Coordinator and the subject review facilitator need to maintain regular communication throughout the process to ensure a clear understanding about the need for particular arrangements.

24 In summary, meetings that will always take place are:

the preliminary meeting;
the Initial meeting;
meeting(s) with staff;
meeting with current students;
meeting with former students;
the reviewers' team meetings.

Testing the self-evaluation and gathering evidence

25 The reviewers have a collective responsibility for gathering, verifying and sharing evidence so that they are able to test statements made in the self-evaluation and develop judgements on quality and standards. The reviewers will share and discuss the evidence gathered, will check their understanding and interpretation of data and will triangulate different sources to arrive at collective conclusions. They will gather evidence on the elements of the review set out in the aide-mémoire (Annex B).

26 The reviewers are selective in their lines of enquiry and focus on their need to arrive at judgements against clearly stated criteria. They refine emerging judgements against as wide a range of evidence as possible. For example, the views expressed in meetings by students or by staff are tested against other sources of evidence.

27 Documents are important sources of evidence that assist the reviewers to evaluate the quality of learning opportunities and academic standards achieved. Documentary evidence includes student work, internal reports from committees, boards and individual staff with relevant responsibilities; and external reports from examiners, verifiers, employers, validating and accrediting bodies. Reviewers also gain evidence from observing some elements directly to evaluate their quality, for example, learning resources.

28 Meetings with students are strictly confidential between the students attending and the reviewers; no comments are attributed to individuals. The subject review facilitator does not attend meetings with students but may be consulted about matters raised by them.

29 The reviewers will not normally need to observe teaching if a college can provide evidence of internal procedures for assuring good quality delivery of the programmes under review. Such evidence is likely to come from a college's scheme for peer observation of teaching; from the analysis of student questionnaires and other arrangements for gathering feedback; from the deployment of learning resources; and from student performance in assessments. Reviewers will only observe teaching if:

there are questions that the reviewers feel would be best addressed by such observation;
observation might help confirm a judgement about exemplary provision;
the college does not provide evidence that teaching of the programmes subject to review is of an appropriate quality; or
there are indications that the learning opportunities for students are not satisfactory.

A note on the observation of teaching is provided in Annex K.

30 Review meetings are used to evaluate the evidence gathered, to form preliminary judgements and to determine which questions require further exploration. The reviewers are required to evaluate how the evidence gathered compares with the college's self-evaluation and to test the strength of the evidence supplied to support the judgements. This includes taking a view of the extent to which the college has achieved consistency of effect with the Academic Infrastructure. It is essential that discussion of the emerging judgements involves the whole review team, both on the college site and through discussion using the electronic folder for the review. If concerns emerge at this point, the college will be given every opportunity, within an agreed timescale, to provide alternative and current evidence to address the reviewers' concerns.

31 All reviewers are expected to identify, share, consider and evaluate evidence related to the programmes under review. The reviewers keep notes of all meetings with staff and students, of their observations, and of comments on the quality of student work and its assessment. These should be analytical rather than descriptive, and refer to sources of information as well as to direct observations. Strengths and area for improvement are summarised. Circulation of notes between the reviewers, and collation of notes by the Review Coordinator, will assist the reviewers to develop a collective evidence base on which judgements can be made.

Making judgements

32 The reviewers will arrange to meet in order to arrive at their final judgements; this is usually around two weeks after their last visit to the college. The reviewers will share and consider all forms of evidence gained during the review to enable them to arrive at accurate and robust collective judgements.

33 Judgements about academic standards are made on the appropriateness of the intended learning outcomes set by the subject provider in relation to subject benchmark statements, qualification levels and the overall aims of the provision; on the effectiveness of curricular content and assessment arrangements in relation to the intended learning outcomes; and on actual student achievement.

34 Judgements about the quality of learning opportunities are made on the effectiveness of teaching and the learning opportunities; on the effectiveness of learning resources, including staff, and of the academic support provided to students to enable them to progress in their studies.

Judgements on academic standards

35 Reviewers make a single, threshold judgement about academic standards. They take into account the points set out below to decide whether they have confidence or not in the academic standards of the provision under review:

a 'confidence' judgement will be made if the reviewers are satisfied both with current standards and with the prospect of those standards being maintained into the future;
a 'limited confidence' judgement will be made if standards are being achieved, but there is doubt about the ability of the college to maintain them into the future;
a 'no confidence' judgement will be made in relation to any of the matters listed below, in the academic standards of the provision under review if the reviewers feel that arrangements are inadequate to enable standards to be achieved or demonstrated.

36 The reviewers will assess, for each programme, whether there are clear intended learning outcomes that appropriately reflect a range of reference points: subject benchmark statements and the level of the award as set out in The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ). More detail of these reference points is provided in Part 2 of this Handbook. The reference points are provided to assist reviewers in determining whether provision is meeting the standards expected by the academic community generally, for awards of a particular type and level. If the reviewers find that the intended learning outcomes do not to match those expectations, it is unlikely that they could have confidence in the standards of the provision. For example, if an intermediate-level programme has intended learning outcomes set at certificate level only, the reviewers would have no confidence in the standard of the programme.

37 The reviewers will assess whether the content and design of the curriculum are effective in enabling students to achieve the intended learning outcomes for the programme. Colleges should be able to demonstrate how each outcome is supported by the curriculum. For this purpose, the term 'curriculum' includes both the content necessary to develop students' understanding and the acquisition of knowledge, and their opportunities to develop practical skills and abilities where these are stated as intended learning outcomes. Within the academic review process, the term 'curriculum' also refers to the range of awards in the subject. If students cannot develop significant intended learning outcomes through the curriculum, it would be unlikely that the reviewers could have confidence in the standards of the provision.

38 The reviewers will assess whether the curriculum content is appropriate to each stage of the programme, and to the level of the award. Colleges should be able to demonstrate how the design of the curriculum secures academic and intellectual progression by imposing increasing demands on the learner, over time, in terms of the acquisition of knowledge and skills, the capacity for conceptualisation, and increasing autonomy in learning. The reviewers will refer to the guidance on programme design in the Code of practice for the assurance of academic quality and standards in higher education (Code of practice), Section 7: Programme approval, monitoring and review.

39 The reviewers will evaluate whether assessment is designed appropriately to measure student achievement of the intended learning outcomes. Colleges should be able to demonstrate how student achievement of intended learning outcomes is assessed, and that, in each case, the assessment method selected is appropriate to the nature of the intended learning outcomes. Confidence in the security and integrity of the assessment process, with appropriate involvement of external examiners, is essential. The range of assessments planned should include some that have a formative function and provide students with prompt feedback to help them to progress in their studies, and assisting them in the development of their intellectual skills. There should be clear and appropriate criteria for different classes or levels of performance, and these criteria should be communicated effectively to students. If significant intended learning outcomes are not assessed, or if the reviewers have serious doubts about the integrity of the assessment procedures, it would be unlikely that they could have confidence in the standards of the provision. The reviewers will refer to the Code of practice, Section 6: Assessment of students.

40 The reviewers will assess whether student achievement matches the intended learning outcomes and level of the award. The reviewers will consider external examiners'/verifiers' reports from the three years prior to the review, and will also sample student work. The balance between reliance upon the reports of external examiners'/verifiers' and direct sampling of student work will depend on the confidence that the reviewers have in the internal examining and verification arrangements of the college. Reviewers will refer to the Code of practice, Section 4: External examining.

41 Where a review covers a number of subjects, reviewers will make separate judgements on standards for each subject. Where programmes are offered at more than one level of the FHEQ (ie at award level, not a stage within an award) reviewers may make separate judgements for each level, or for each FHEQ award, if there are significant differences between them. Review reports will include comments on strengths and areas for improvement for each element of the standards judgement.

Judgements on quality of learning opportunities

42 Reviewers will make judgements about the quality of the learning opportunities offered to students against the broad aims of the provision and the intended learning outcomes of the programmes. Each judgement will normally cover all provision within the scope of the review. However, if performance is significantly different in a subject area, or for a particular award, separate judgements will be made.

43 The reviewers will rely usually on secondary evidence rather than direct observation of teaching and learning, wherever possible. The reviewers will assess the effectiveness of teaching and learning in relation to curriculum content and programme aims.

44 The reviewers will evaluate student progression by considering their recruitment, academic support and progression within the programme.

45 In making judgements about learning resources, the reviewers will assess whether the minimum resource necessary to deliver each programme is available, and will then consider how effectively resources are used in support of the intended learning outcomes of the programmes under review. The reviewers will look for a strategic approach to the linkage of resources to intended learning outcomes for each programme.

46 When reporting on the quality of learning opportunities reviewers will place each of the three aspects of provision into one of three categories; 'commendable', 'approved' or 'failing' and judgements will be made on the following basis:

the provision contributes substantially to the achievement of the intended learning outcomes, with most elements demonstrating good practice. In the report, this judgement will be referred to as 'commendable';
the provision enables the intended learning outcomes to be achieved, but improvement is needed to overcome weaknesses. In the report, this judgement will be referred to as 'approved'. The report will normally include a statement containing the phrase 'approved, but...' which will set out the areas where improvement is needed;
the provision makes a less than adequate contribution to the achievement of the intended learning outcomes. Significant improvement is required urgently if the provision is to become at least adequate. In the report, this judgement will be referred to as 'failing'.

47 Within the 'commendable' category, the reviewers will identify any specific features of the aspect of provision that may be exemplary. To be judged 'exemplary', a feature is expected to:

represent sector-leading best practice;
be worthy of dissemination to, and emulation by, other providers of comparable programmes; and
make a significant contribution to the success of the provision being assessed.

48 Incidental or marginal features do not qualify for the designation of 'exemplary'. The characteristics of exemplary features will, by their nature, vary between colleges and programmes. The criteria listed above will ensure that features identified as 'exemplary' will be broadly comparable in weight and significance.

49 If the reviewers have no confidence in the standards achieved, or if the provision is found to be failing in any aspect of quality, the provision will be regarded overall as 'failing'. It follows that all provision that is not failing is 'approved'. In the event of a 'no confidence' or 'failing' judgement, a re-review of the programme will take place, usually 12 months after the first review.

Maintenance and enhancement of standards and quality

50 College-wide systems for the maintenance and enhancement of standards and quality are addressed through the college's arrangements for programme review and evaluations. The reviewers will gather evidence, not least from their discussions with staff and students and their scrutiny of external examiners' reports, on the operation of college systems in each subject under review. The final section of the review report will express their evaluation of the ability of the college to maintain and enhance standards and quality in the particular subject.

51 The Agency requires the Review Coordinator to notify the Agency if there is the possibility of a 'no confidence' or 'failing' judgement. In this case, an Agency officer will attend review meetings and/or the judgement meeting to ensure that the published review method is followed. The officer will not contribute to the process of making judgements.

Liaison between the college and the reviewers

52 Liaison between the reviewers and the college is vital to ensure that the reviewers obtain accurate and comprehensive information about the subject under review and also to ensure that the college is clear about the areas on which reviewers require further information. All communications concerning the conduct of the review should normally be through the subject review facilitator and the Review Coordinator.

Stage 3: The reporting stage

53 At the end of the review period, the Review Coordinator will usually notify the college by letter of the judgements reached. This is normally two weeks after the judgement meeting. During this period the Review Coordinator will prepare a draft report drawing upon the contributions of the specialist reviewers. The draft report will be edited before it is sent to the college, normally within eight weeks of the end of the review period. The college is asked to comment upon matters of factual accuracy and return these to the Review Coordinator within three weeks of receipt. Following the receipt of these comments, a further draft is prepared and forwarded to the Agency. The draft is checked to ensure that the reviewers have considered properly the college's comments. Reports are normally published within 20 weeks of the end of the review period. The report structure is set out in Annex L.

54 The Agency has published a statement of its standards in its Strategic plan 2003-05. These can be found at www.qaa.ac.uk

55 While the Agency makes every effort to assure quality of process and to meet its published standards, occasionally a college may feel it has cause to express dissatisfaction with a review. If a college is dissatisfied with the conduct of a review, it may wish to make a complaint. Details of the complaints procedure can be found at the Agency's web site.

56 If a college wishes to appeal against a failing or no confidence judgement, then it may make a representation to the Agency. Details of the representation procedures can be found at the Agency's web site.

Part two

Reference points for review

The Academic Infrastructure

57 In common with other methods of review carried out by the Agency, academic review uses a set of nationally agreed reference points, known as the Academic Infrastructure, to consider the standards and quality of HE awards. These reference points were developed by the Agency in consultation with providers of HE. The purpose of the Academic Infrastructure is to help colleges to set standards and have guidance on good practice. The components of the Academic Infrastructure published by the Agency are intended for consultation collectively rather than in isolation. They are:

the FHEQ and the Framework for qualifications of higher education institutions in Scotland which include descriptions of different HE qualifications. Full details are available on the Agency's web site;
subject benchmark statements, which describe the characteristics of degrees in different subjects;
Guidelines for preparing programme specifications, which describe what is on offer to students in individual programmes of study. Full details are available on the Agency's web site;
the Code of practice. All sections of the Code of practice are operational. Full details are available on the Agency's web site;
the Foundation Degree qualification benchmark. Full details are available on the Agency's web site.

58 The Agency is continuing to work with the Qualifications and Curriculum Authority (QCA) to align the QCA's National Qualification's Framework and the FHEQ. The Agency helps to define clear and explicit standards, for public information and as nationally agreed reference points. To this end it has worked with the HE sector and other stakeholders on the following initiatives.

The frameworks for higher education qualifications

59 These are designed to make it easier to understand HE qualifications. They promote a clearer understanding of the achievements and attributes represented by the main titles such as bachelors degree with honours, master's degree and doctorate. By setting out the attributes and abilities that can be expected of the holder of a qualification, the frameworks help students and employers understand the meaning and level of qualifications. They also provide public assurance that qualifications bearing similar titles represent similar levels of achievement.

60 There is a qualifications framework for England, Wales and Northern Ireland, and one for Scotland, which is part of a wider Scottish Credit and Qualifications Framework.

Subject benchmark statements

61 Subject benchmark statements set out expectations about standards of degrees in a range of subject areas. They describe the conceptual framework that gives a discipline its coherence and identity, and define what can be expected of a graduate in terms of the techniques and skills needed to develop understanding in the subject. They also identify the level of intellectual demand and challenge represented by an honours degree in subject areas, and help HEIs when they design and approve programmes.

Programme specifications

62 Programme specifications are the sets of information that each institution provides about its programmes. Each specification clarifies what knowledge, understanding, skills and other attributes a student will have developed on successfully completing a specific programme. It also provides details of teaching and learning methods, assessment and subsequent career opportunities, and sets out how the programme relates to the qualifications framework.

63 This information allows prospective students to make comparisons and informed choices about the programmes they wish to study and provides useful guidance for recruiters of graduates.

The Code of practice for the assurance of academic quality and standards in higher education

64 The Code of practice sets out guidelines on good practice relating to the management of academic standards and quality. Each section of the Code of practice has precepts or principles that institutions should satisfy, with guidance on how they might meet these precepts. The Code of practice has 10 sections:

Postgraduate research programmes
Collaborative provision
Students with disabilities
External examining
Academic appeals and student complaints on academic matters
Assessment of students
Programme approval, monitoring and review
Career education, information and guidance
Placement learning
Student recruitment and admissions

Teaching Quality Information

65 At present, FECs are not included in the requirement in HEFCE's document, Information on quality and standards in higher education: Final guidance (HEFCE 03/51). Currently, colleges do not have to publish information on the Higher Education and Research Opportunities in the United Kingdom (HERO)/Teaching Quality Information (TQI) web site. If a college has HE provision that is indirectly funded through a HEI, it is the HEI that will be responsible for reporting data to the Higher Education Statistics Agency (HESA) for inclusion on the TQI site, and for producing the qualitative reports that are submitted directly to the HERO/TQI site www.tqi.ac.uk The information will refer to the college.

66 If a FEC has HE provision directly funded by HEFCE, this provision is subject to review, and academic quality and standards will be reported on through the publication of the report. There is no further requirement for TQI of this provision.

67 The sponsoring bodies, HEFCE, Universities UK (UUK) and the Standing Conference of Principals (SCOP) recognise that more comprehensive information will be available about HE provision in HEIs than in FECs. They are investigating the potential for providing information on the quality and standards of HE in FECs, especially where this can be generated without additional burden to FECs. Initially, the sponsor will consider providing links to the Agency review reports and exploring whether Learning and Skills Council data can be presented in a format that is comparable to HESA-derived data for HEIs. For an update on these developments, visit the HERO TQI web site at www.tqi.ac.uk or HERO's web site at www.hero.ac.uk/

68 During the normal course of the review, reviewers may make reference to the TQI web site.

Annex A

HEFCE schedule for subject review 2000-06

This Handbook will be used for reviews for 2003-2006 as set out below

Subject

Period during which subject will be reviewed

Librarianship & Information Management*
Economics*
Politics & International Relations*
Classics & Ancient History*
Archaeology*
Philosophy*
Theology & Religious Studies*
Hospitality, Leisure, Sport & Tourism*
Business & Management Studies*
Education Studies*
Social Policy & Administration and Social Work
Sociology & Anthropology
English
Engineering
Geography
Earth, Environmental Sciences & Environmental Studies
Architecture, Architectural Technology & Landscape Architecture
Computing
Law
Accountancy
History
Medicine
Dentistry
Veterinary Medicine
Biosciences
Subjects Allied to Medicine**
Nursing & Midwifery**
Physics & Astronomy
Chemistry
Psychology
Agriculture, Forestry, Agricultural & Food Sciences
Materials
Building & Surveying
Town & Country Planning
Mathematics, Statistics & Operational Research
Linguistics
Area Studies
Languages & Related Studies
Communications, Media, Film & Television Studies
Art & Design
Dance, Drama & Performance Arts
Music

2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2000-03
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06
2003-06

* To be reviewed in England and Northern Ireland during 2000-2001 using the former subject review method.

** Schedule subject to agreement with the NHS and other funding bodies. These subjects may be reviewed earlier in England.

Annex B

Aide-mémoire for academic review

1 This aide-mémoire is intended to support:

colleges' preparation of self-evaluations;
reviewers' analysis of the self-evaluation before visiting the college; and
reviewers' collection of evidence during the review.

2 The aide-mémoire covers the main features of the review process, but it is not intended to be prescriptive or exhaustive. The college's self-evaluation, the statement of aims, and the intended learning outcomes of programmes may all raise questions particular to the programme under scrutiny. The reviewers will use self-evaluations in any appropriate form, provided they contain the information that the reviewers need to plan and conduct the review.

Guidance for colleges when preparing self-evaluations

3 The range of activities and documents reviewed will vary from one review to another. Colleges should take care to ensure that their self-evaluation meets the needs of the review and answers the questions the reviewers are likely to pursue. In the interests of keeping preparation to a minimum, the Agency encourages each college to send a collection of current review and evaluation documents as the self-evaluation rather than a purpose-written document. Colleges will find it helpful to refer to the prompts and questions in the aide-mémoire below. Colleges should also refer to the components of the Academic Infrastructure such as the Code of practice, the FHEQ and relevant subject benchmark statements, where appropriate.

4 A self-evaluation should discuss both the strengths and the areas for improvement the college has identified in the subject to be reviewed. It provides an opportunity for the college to demonstrate how previously identified strengths have been built upon and how any identified areas for improvement have been addressed. Where areas for improvement remain, existing plans for addressing these should be discussed. The reviewers will give credit where appropriate remedial plans are in place to address acknowledged areas for improvement, provided students are experiencing a programme of the adequate standard and of, at least, approved quality.

5 Effective self-critical evaluation makes reference to the sources of evidence, the analysis of which has informed reflection. Sources of evidence should be cited and could usefully be included as a list of references. These references will help the reviewers in gathering evidence, and avoid the need for colleges to include merely descriptive material in the evaluation. The references quoted in the self-evaluation should be made available to the reviewers during the review period so that they may triangulate evidence gathered with evidence cited.

6 Colleges are asked to take a flexible approach in preparing and presenting self-evaluations to accommodate the range and potential complexity of each review. For example, some reviews may be aggregate reviews which cover more than one subject. The self-evaluation should reflect the internal organisation and delivery. For example, where a subject contains more than one subject/unit of review, it may be appropriate to evaluate discrete programmes, or groups of related programmes separately, introduced by a short overview dealing with the college's approach to matters common to all disciplines, such as the maintenance and enhancement of standards and quality.

7 The process of review focuses on the establishment and maintenance of academic standards by the college, their achievement by students, and the quality of the learning opportunities the college offers. Neither 'standards' nor 'quality' can be reviewed in isolation. They are interrelated and must be reviewed as such.

8 The reviewers will need factual information about the programme under review. There must be a clear statement of the overall aims of the subject programme that should not exceed 500 words in length. The reviewers will use this to assess whether the programme achieves its broad aims. The statement of aims will be reproduced at the start of the academic review report. Overall aims will reflect the distinctive mission of the college and the college's purpose in offering HE level programmes in the subject. They place the study of a discipline in contexts by:

enabling students to develop their capacity to learn;
meeting local, regional, national or international needs;
preparing students for employment and/or further study;
widening access to HE.

9 Statements of aims should be succinct but convey clearly the parameters of the subject programme. They may be presented as narrative statements, bullet points or a mixture of the two. Precise, definitive programme titles should be stated.

10 The evaluation of the programmes in the subject/unit of review should not normally exceed 6,000 words in length. Programme specifications form part of the self-evaluation and should be appended to provide the detail of programme aims, intended learning outcomes and the means by which they are achieved. Factual material provided in the programme specifications need not be repeated elsewhere. Where the programmes in the subject/unit of review are organised and delivered through different departments or schools, clarification of the academic structures and scope of responsibility for the programmes will be helpful to the reviewers. This should include information about relevant modular structures, internal and external collaborative arrangements and relationships with local, regional or national employers.

11 The reviewers need statistical data on student achievement and progression for the last three student intakes that have completed the programme(s). This requires more than three years of entry data so that the reviewers can see the data for the students who have completed the programme(s). For example, for an HND programme:

	2000-01	2001-02	2002-03	2003-04
Year one	Intake W	Intake X	Intake Y	Intake Z
Year two		Intake W	Intake X	Intake Y

For a three-year degree programme:

	1999- 2000	2000-01	2001-02	2002-03	2003-04	2004-05
Year one	Intake A	Intake B	Intake C	Intake D	Intake E	Intake F
Year two		Intake A	Intake B	Intake C	Intake D	Intake E
Year three			Intake A	Intake B	Intake C	Intake D

12 Statistical indicators should include data on recruitment and admissions; entry profile (qualifications, age, gender, ethnicity - profile data relevant to mission and aims); progression rates and completion rates; student achievement in summative assessment; and progression of completing students to employment and further study. There is a need to distinguish between students who complete and gain the award and students who complete without gaining the award. Reviewers will need to know the number of students who enrol for the first year of each programme and progress to complete the programme and gain the award.

13 The data should distinguish clearly between those students in the first-year entry cohort, those joining directly at subsequent stages, withdrawals (including reasons for withdrawal), referrals (showing those subsequently failing and those passing), outright failures and those achieving the award. It is important that such data are presented in a suitably disaggregated form to clearly inform the reviewers as to the levels of achievement, such as the number of Distinctions, Merits, Passes: Referrals and Failures should also be included. The reviewers will be interested to know how the college uses such data to evaluate, manage, plan and enhance academic quality and standards.

Guidance for review teams

Aims and outcomes

14 The college's self-evaluation should address the appropriateness of the intended learning outcomes in relation to the overall aims of the provision, relevant subject benchmark statements and the FHEQ. The self-evaluation should discuss the effectiveness of measures to ensure that staff and students have a clear understanding of the aims and intended learning outcomes of the programmes.

15 The reviewers will consider:

how well the intended learning outcomes relate to the overall aims of the programme and whether they enable the aims to be met;
the extent to which they are aligned with external reference points, including the FHEQ, to provide an appropriate level of challenge to students;
the extent to which they are aligned with, and informed by, relevant subject benchmark statements, where appropriate, and any professional or statutory body requirements;
how well the intended learning outcomes of a programme and its constituent parts are communicated to staff, students and external examiners/verifiers.

16 Programme specifications will provide detail of the overall aims and intended learning outcomes for each award. Module or unit descriptors will provide a further level of detail. Review activities may include analysis of module descriptors and level descriptors, consideration of the balance and range of outcomes relating to subject, professional, practical and transferable skills and the relationship between the intended learning outcomes for each award and its title. The reviewers will engage in discussions with staff and students and review the reports of external examiners/verifiers. Student handbooks and curricula documents will provide factual information and will illustrate how aims and outcomes are communicated to staff and students.

17 The reviewers should be able to judge whether intended learning outcomes are clearly stated and are appropriate to the level of the award, meeting the requirements of relevant external reference points.

Curricula

18 The college's self-evaluation should review the effectiveness of curriculum design and content in enabling the intended learning outcomes to be achieved.

19 The reviewers will consider:

how the college plans the curriculum design and content and how decisions about contributing modules and their sequencing are made;
whether the design and content of the curricula encourage achievement of the intended learning outcomes in terms of knowledge and understanding, cognitive skills, subject specific skills (including practical/professional skills), transferable skills, progression to employment and/or further study, and personal development;
the extent to which curricular content and design are informed by recent developments in techniques of teaching and learning, current research, scholarship or consultancy and by any changes in relevant occupational or professional requirements;
how the college ensures that the design and organisation of the curriculum provides appropriate academic and intellectual progression and is effective in promoting student learning and achievement of the intended learning outcomes.

20 The design and content of each programme will be evaluated in relation to its potential for enabling students to achieve the intended learning outcomes. Sources of information may include curricular documents, review reports, reports from professional bodies, placement reports from employers, course and student handbooks and module descriptions. The reviewers will need to know how appropriate sections of the Code of practice have been considered, for example, the approval, monitoring and review of programmes and placement learning, where appropriate.

21 The reviewers will be able to judge whether the curriculum is appropriate to each stage of the programme and to the level of the award.

Assessment

22 The college's self-evaluation should review the effectiveness of student assessment in measuring the achievement of the intended learning outcomes of programmes.

23 The reviewers will consider:

the extent to which the overall assessment strategy has an adequate formative function in developing student abilities, assists them in the development of their intellectual skills and enables them to demonstrate achievement of the intended learning outcomes, in both campus and placement settings;
the assessment methods selected and their appropriateness to the nature of the intended learning outcomes and to the type and level of work;
the criteria used to enable internal and external examiners/verifiers to distinguish between different categories of achievement, and the way in which criteria are communicated to students;
the security, integrity and consistency of the assessment procedures, the setting, marking and moderation of work in both campus and placement settings, and the return of student work with feedback;
how employers and other professionals contribute to the development of assessment strategies, where appropriate.

24 The reviewers will seek sources of evidence to help them evaluate whether the overall assessment process and the particular assessment instruments chosen are appropriate and effective and will seek evidence of clear guidance about the assessment arrangements. Where appropriate, the reviewers will evaluate the contribution of assessment of placement learning to the final award and the responsibilities of the parties involved. The review of samples of marked student work, annual review reports, external examiners'/verifiers' reports, statistical data and discussions with staff and students will help the reviewers to evaluate the contribution of assessment to the overall standards judgement. The reviewers will evaluate the appropriateness of assessment tools, the competence of those involved with the assessment arrangements and whether the assessment strategy reflects progression, integration and coherence. The sections of the Code of practice dealing with assessment of students (section 6) and external examining (section 4) will be important points of reference.

25 As a result of these activities, the reviewers should be able to judge whether assessment processes securely measure the achievement of the intended programme outcomes.

Achievement

26 The college's self-evaluation should review evidence of the extent to which students achieve the learning outcomes set.

27 The reviewers will consider:

the evidence that students' assessed work demonstrates their achievement of the intended learning outcomes;
the evidence that standards achieved by learners meet the minimum expectations for the award as measured against the FHEQ and relevant subject benchmark statements, if appropriate;
whether students are prepared effectively for their subsequent employment roles;
the levels of achievement indicated by the statistical data, whether there are any significant variations between modules or awards and the successful progression to employment;
how the college promotes student retention and achievement.

28 The reviewers will then evaluate whether student achievement meets such expectations. Sources of information will include external examiners'/verifiers' reports, placement supervisors' reports, assessment board minutes, samples of student work, and statistical data on achievement and career destinations. Review activities will include discussions with teaching teams, internal examiners/verifiers and students. Relevant subject benchmark statements and the level descriptors of the FHEQ will be important points of reference.

29 As a result of these activities, the reviewers should be able to judge whether appropriate standards are being achieved.

Teaching and learning

30 The self-evaluation should review the effectiveness of teaching and learning, in relation to programme aims, the intended learning outcomes and curriculum content.

31 The reviewers will consider:

the range and appropriateness of teaching methods employed in relation to curriculum content and programme aims;
how staff draw upon their research, scholarship, consultancy or professional activity to inform their teaching;
the ways in which participation by students is encouraged and how learning is facilitated;
how the materials provided support learning and how students' independent learning is encouraged;
student workloads;
how quality of teaching is maintained and enhanced through staff development, peer review of teaching, integration of part-time and visiting staff, effective team teaching and induction and mentoring of new staff.

32 The reviewers will evaluate the overall effectiveness of the teaching and learning activities on the programmes under review; in particular: the breadth, depth, pace and challenge of teaching; the effectiveness of the teaching of subject knowledge; and the effectiveness of the teaching of subject-specific, transferable and practical skills. Sources of evidence may include student evaluation of their learning experience, internal review documents, staff development documents, course and student handbooks and discussions with staff and students. Review activities will not normally include direct observation of teaching, but reviewers may request teaching observation where they judge this to enable them to reach their judgements. A note on the observation of teaching and how it is recorded is in Annex K.

33 As a result of these activities, the reviewers will be able to judge the quality of learning opportunities offered to students through the programme of teaching and learning and its contribution to achievement of the overall aims of the programme.

Student progression

34 The college should evaluate the effectiveness of strategies for recruitment, admission and academic support and guidance to facilitate students' progression and completion of the programme.

35 The reviewers will consider:

the effectiveness of arrangements for recruitment, admission and induction and whether these are generally understood by staff and applicants;
the overall strategy for academic support and its relationship to the student profile and the overall aims of the programme;
how learning is facilitated by academic guidance, feedback and supervisory arrangements;
the arrangements for academic tutorial support, their clarity and their communication to staff and students, and how staff are enabled to provide the necessary support to students;
the quality of written guidance;
the extent to which arrangements are in place and effective in facilitating student progression towards successful completion of their programmes.

36 The reviewers will evaluate the appropriateness of the overall strategy for academic support, including written guidance, and the extent to which it is consistent with the student profile and the overall aims of the programme. They will evaluate whether there is appropriate matching of the abilities of students recruited to the demands of the programme and whether there are appropriate arrangements for academic guidance and support to facilitate progression, completion and non-completion. The reviewers will consider progression within programmes as well as non-completion rates. Sources of evidence may include statistical data on applications, admission, progression and completion, policy statements on admission and learning support, course and student handbooks, student evaluation of admission, induction and tutorial support and discussions with staff and students. Sections of the Code of practice on recruitment and admissions, students with disabilities, placement learning, appeals and complaints, and career education, information and guidance are particularly relevant.

37 As a result of these activities, the reviewers will be able to make judgements about the quality of learning opportunities in support of student progression.

Learning resources

38 The college's self-evaluation should review the adequacy of human and physical learning resources and the effectiveness of their utilisation. In particular, the evaluation should demonstrate a strategic approach to linking resources to intended learning outcomes at programme level.

39 The reviewers will consider:

staffing levels and the suitability of staff qualifications and experience, including teaching and non-teaching staff;
professional updating to keep abreast of emerging, relevant subject knowledge and technologies;
staff development opportunities, including induction and mentoring for new staff, and whether opportunities are taken;
journal and electronic media;
access times and arrangements, and induction and user support provision;
computing hardware, and both general and subject-specific software availability, and currency;
accessibility, including times of opening and opportunities for remote access, and induction and user-support provision;
specialist accommodation, equipment and consumables;
adequacy, accessibility, induction, user-support and maintenance;
suitability of staff and teaching accommodation in relation to the teaching and learning strategy and the provision of support for students.

40 The reviewers will evaluate how effectively learning is facilitated through the overall deployment of resources, including whether appropriate technical and administrative support is made available, and the appropriateness of staff development strategy and practice. Sources of information will include direct observation of physical resources, internal review documents and minutes of meetings, equipment lists, library stocks, staff curricula vitae, external examiners'/verifiers' reports and staff development documents. The reviewers will meet staff and students.

41 As a result of these activities, the reviewers will be able to judge whether the learning resources available successfully underpin the programmes and whether there are appropriately qualified staff who are contributing effectively to the achievement of the intended learning outcomes.

Maintenance and enhancement of standards and quality

42 The college should evaluate the effectiveness of measures taken to maintain and enhance academic standards and the quality of learning opportunities provided for the programmes under review.

43 The reviewers will consider:

the college's approach to the quality assurance of its HE provision and the effectiveness of this approach for the programmes under review;
the use made of quantitative data and qualitative feedback from students, external examiners/verifiers and other stakeholders in a strategy of enhancement and continuous improvement;
the college's responsiveness to internal and external review and assurance processes;
the accuracy of the self-evaluation.

44 The reviewers will evaluate how well the internal mechanisms for assuring academic standards and quality are working. Sources of evidence will include student and staff feedback, external examiners'/verifiers' reports, quantitative data, employers' views, previously published subject review reports, other reports if available, and internal review reports.

45 The reviewers will not make a formal judgement on this element of the review. In the final section of the report, they will summarise their evaluation of the ability of the college to maintain and enhance quality and standards in the programmes under review.

Submission of the self-evaluation for analysis by reviewers

Introduction

1 This section sets out the procedures for handling the self-evaluation prior to the start of an academic review. This detailed annex has been included in the Handbook to help colleges understand the processes their self-evaluations pass through before the reviewers visit colleges. This may help colleges may seek to reassure themselves of the completeness of their self-evaluation.

Initial scrutiny of the self-evaluation

2 Colleges are asked to send three printed copies or one electronic copy of the self-evaluation to the Agency's Gloucester office, with the form below completed and attached to the front of the self-evaluation. The first part of this form allows the college to state clearly the documentation being submitted as the self-evaluation.

3 The self-evaluation is scrutinised as follows:

i the Agency will check the word length of the self-evaluation and, if it exceeds the length set out in the Handbook, (500 words for the statement of the overall aims: 6000 words for evaluation of the subject programme) will return it to the college for editing. In this case, the Agency will inform the Review Coordinator. Sometimes, however, there will be instances where some flexibility regarding word length is required. If a college would like the Agency to consider an extension, the college should write to request this. The Agency will reply in writing and notify the Review Coordinator. For example, if the provision to be reviewed is particularly complex or several subjects/units of review have been aggregated, this will be taken into account;

ii the Agency will check that programme specifications are attached as annexes to the self-evaluation. If any are missing, the Agency will inform the Review Coordinator and ask the college to supply the missing material. The Agency will also check that statistical data are included. These checks are for completeness of the self-evaluation and not an evaluation of its reliability as the basis for review.

Stage 1: Submission of the self-evaluation

4 On receipt of the self-evaluation, it is checked using the form that the college has attached to the front (see below). The form is emailed to the college during the period of arranging the review.

5 The form is based on simple tables for specific data when this needs to be assembled. Colleges complete the form as a checklist before submitting their self-evaluation, to confirm that all relevant material is included. The Agency uses it to cross check that the submitted self-evaluation contains the range of material needed to support the review. The self-evaluation is then forwarded to the Review Coordinator for analysis (see stage 2).

6 A model e-form is provided below. Where sections are in italics these are examples and should be overwritten. The form will expand as text is entered. Rows may be added to the tables to accommodate the information to be provided.

Example of submission for an aggregated review in Art and design/Communications, media, film and television studies

Please complete this form electronically and return it to the Logistics and Deployment (L&D) Team: L&D@qaa ac.uk with each self-evaluation that you submit.

College
QA contact name
Designation
Telephone number
Email address

Please list the subject and programmes included in this self-evaluation

Subject: Art and design

HND Fine art

HND Design studies

For office use only
Programme Specification appended? Yes/No	Statistical data included? Yes/No	Comment

Subject: Communications,
media, film and television
studies

HND Multimedia

HND Digital imaging

For office use only
Programme Specification appended? Yes/No	Statistical data included? Yes/No	Comment

The document(s) submitted include the following:
For completion by the college

Section	Confirm
Aims (maximum 500 words)	495 words
Evaluation of the provision
(maximum of 6,000 words)	5,095 words
Aims and outcomes	Yes
Curricula	Yes
Assessment	Yes
Achievement	Yes
Teaching and learning	Yes
Student progression	Yes
Learning resources	Yes
Maintenance and enhancement of standards and quality	Yes
Information re modular structures/ collaborative arrangements	No

For L&D use only
tick	Comment













	Page count =

Stage 2: Analysis of the self-evaluation

Scrutiny by the Review Coordinator

7 When stage 1 is successfully completed, the self-evaluation and the completed form, are passed to the Review Coordinator to ensure that the self-evaluation contains sufficient information to plan and conduct the review. The Review Coordinator will check that the self-evaluation broadly follows the guidelines in the Handbook and will notify the Agency of this by completing the form set out below. Colleges are advised to address the form themselves to test whether their self-evaluation is fit for purpose. The Review Coordinator will return this form electronically to the L&D officer and copy it to the Agency officer responsible for the review (see stage 3 below), within seven days of receipt of the self-evaluation.

8 If it becomes apparent on completion of the form that the self-evaluation does not contain the information needed to plan and conduct the review, or if there are significant uncertainties about the quality of the self-evaluation, the Review Coordinator and the Agency officer will discuss this and decide upon an appropriate course of action. The review should normally proceed using the self-evaluation as originally submitted. Questions of content or quality of the self-evaluation should normally contribute to the dialogue between the college and the reviewers and should influence the reviewers' lines of enquiry. Occasionally, it may be necessary for the Agency to ask a college to add to, or rewrite, the self-evaluation. Responsibility for this final decision rests with the Agency officer (see stage 3 below). The preliminary meeting of reviewers and the institution should not take place until it has been agreed that the self-evaluation provides a satisfactory basis for the conduct of the review.

9 Colleges will appreciate the demanding timescales above and will need to consider the security of their self-evaluation.

Review Coordinator's analysis of the self-evaluation

Review Coordinator:

Date self-evaluation and submission and analysis form forwarded to Review Coordinator:

(date to be entered by the L&D officer)

1 Are the programmes to be reviewed clearly identified within the self-evaluation?

Yes/No

Notes

2 Are the overall aims clear and do they provide a reasonable basis for the planning and conduct of the review?

Yes/No

Notes

3 Does the self-evaluation address academic standards, and in particular:

the appropriateness of the academic standards set for the programmes?

Yes/No

Notes
the effectiveness of the curriculum in delivering the intended outcomes of the programmes?

Yes/No

Notes
the effectiveness of student assessment in measuring attainment of the intended outcomes?

Yes/No

Notes
the extent to which students achieve the intended standards and outcomes?

Yes/No

Notes

4 Does the self-evaluation address the quality of learning opportunities and, in particular:

the effectiveness of teaching and learning?

Yes/No

Notes
student progression and, in particular, the effectiveness of strategies of academic support?

Yes/No

Notes
the adequacy of learning resources and the effectiveness of their utilisation?

Yes/No

Notes

5 Does the self-evaluation address the maintenance and enhancement of standards and quality in the subject?

Yes/No

Does the self-evaluation address both strengths and areas of importance of the programme(s) under review?

Yes/No

Notes
Is there evidence for the strengths and an action plan for areas of importance?

Yes/No

Notes

6 Are any programme specifications missing?

Yes/No

Notes

7 Do all programme specifications contain learning outcomes?

Yes/No

Notes

General observations

8 Please comment here on the quality of the self-evaluation eg is it evaluative? Is it helpfully structured? Is any essential information missing?

9 Do you accept that the review can proceed on the basis of this self-evaluation?

Yes/No

If no, please summarise reasons

10 Proposed date for the preliminary meeting with the institution: (insert date)

Date:

Date form returned to L&D:

Please email this completed form to the L&D officer and Agency subject coordinator within seven days of receiving the self-evaluation.

Stage 3: Confirmation that the review may proceed

10 When the Review Coordinator agrees that the self-evaluation forms the basis for the review to proceed, the Agency officer, with responsibility for the review, considers the documentation and decides if the review can proceed. The form below is used for this purpose.

For completion by Agency officer responsible for the review

Date received:

Should this self-evaluation be returned to the institution for amendment? Yes/No

Notes

Does the initial scrutiny of the self-evaluation suggest that visit support Yes/No should be provided?

Notes

Signed:
Date:

Once completed, this form is retained by the Agency officer with responsibility for the review and sent to the L&D Team and to the ARCS electronic folder for the review.

Annex C

Reviewers

Introduction

1 The Agency operates an equal opportunities policy. All applicants will be considered on the basis of their ability to meet the specifications outlined below. The Agency evaluates the performance of all reviewers, using feedback from review visits.

2 There are two types of academic reviewer used by the Agency:

specialist reviewers, with current teaching experience in the discipline concerned, or experience of relevant professional or occupational practice;
Review Coordinators, who lead academic reviews and have extensive experience of quality assurance and programme approval of HE programmes, usually gained by working with such procedures in more than one discipline.

Qualities required in all reviewers

3 Effective reviewers will possess the following qualities:

demonstrable commitment to the principles of quality assurance of HE provision;
an enquiring and sceptical disposition;
powers of analysis and sound judgement;
personal authority and presence coupled with the ability to act as an effective team member;
good time-management skills;
experience of chairing meetings;
the ability to make appropriate judgements in the context of the college or university being reviewed and recognising that it is different from their own place of work;
experience of organisation and management, particularly in relation to teaching and learning matters;
high standard of oral and written communication, preferably with experience of writing formal reports for publication to deadlines.

4 In addition, reviewers are expected to have a clear knowledge and understanding of the Agency's academic review process and the Academic Infrastructure. The Agency expects that reviewers will be familiar with the Code of practice and are aware of the precepts in the sections relevant to the provision under review. All sections of the Code of practice are operational.

Recruitment, training and role of specialist reviewers

5 Specialist reviewers are recruited by the Agency from individuals nominated by colleges or other organisations and from individuals who reply to advertisements. Specialist reviewers are recruited and trained to ensure that they are capable of carrying out their duties effectively. In particular, specialist reviewers who undertake reviews are expected to:

possess the knowledge and skills set out in detail below;
have completed successfully the Agency's training programme;
ensure that they are available for the whole period of a review for which they have been selected and have a commitment to complete all processes of a review once they have embarked upon it;
normally be available for up to three reviews each year.

6 Initial training of reviewers is carried out by the Agency by means of a three-day residential course. In recruiting reviewers, the Agency seeks to ensure that their collective expertise reflects the need to complete the full programme of HE in FEC reviews.

7 The Agency maintains a database of specialist reviewers. The primary purpose of the database is to show, for each reviewer, the main areas of teaching and learning that s/he is qualified to review. For this purpose, the Agency uses the Joint Academic Coding System (JACS), established by HESA and UCAS for use by both organisations from 2002.

8 The key purpose of acting as a specialist reviewer is to contribute to the maintenance and enhancement of standards in HE by reporting to the Agency on the standards and quality of the academic programmes scrutinised during academic reviews. Specialist reviewers are expected to agree individual timetables of activity with the Review Coordinator, with a view to making the most effective contribution to the review. The responsibilities of reviewers include:

reading and analysing the self-evaluation submitted by the college and any other documents sent in advance of a review;
adhering to the review schedule agreed between the college and the Review Coordinator;
participating in visits to the college in order to gather, share, test and verify evidence;
making judgements on the academic standards achieved and the quality of the learning opportunities provided;
contributing to and commenting on the compilation of the report of the review to agreed schedules and deadlines.

9 The Agency not only tries to ensure that the particular experience of individual reviewers is relevant to the reviews they undertake, but that, over time, each reviewer works in a variety of teams scrutinising programmes in their specialism in a range of colleges. There are significant benefits, both to the individuals and their colleges of becoming a specialist reviewer. Participants evaluate the initial reviewer training as an excellent opportunity for professional development. Current and former reviewers have commented upon the opportunity to learn from review visits and peer reviewers as invaluable to their own practice. They have also identified the sharing of dialogue with peers from other colleges as being of great benefit.

10 Specialist reviewers analyse and evaluate the self-evaluation provided, with particular emphasis on curricular content and its suitability for achieving the programme outcomes. They review and evaluate the assessment processes designed for the programmes and determine whether they are suitable to assess programme outcomes as stated in the programme specifications.

11 Specialist reviewers consider and evaluate overall student achievement, including progression to employment; the contribution made to student achievement by the quality of teaching; opportunities for learning; academic support intended to ensure effective progression of students; and learning resources and their deployment (including staffing).

12 Specialist reviewers judge the overall standards for subjects and the procedures associated with their maintenance and enhancement.

13 Finally, specialist reviewers contribute to the compilation of a report to the Agency. Each specialist reviewer will be expected to prepare material for the various sections of the report and relevant draft sections of the report, with specific references to the sources of evidence considered.

Knowledge and skills required of specialist reviewers

14 To carry out the role outlined above, for each review, specialist reviewers will need to demonstrate:

experience, knowledge and understanding of HE provision;
at least five years' experience of providing HE-level teaching and learning in delivered in FECs or HEIs: in the case of industrially or professionally-based reviewers, familiarity with HE teaching and learning;
familiarity with academic support strategies and the functions of academic tutorials;
experience of examining and/or verification (and preferably external examining or verification);
knowledge of the quality assurance processes employed by colleges providing HE.

Knowledge and understanding within the subject area

knowledge of, and familiarity with, the subject benchmark information produced for programmes within the subject area being reviewed;
familiarity with the subject matter of the self-evaluation and the programme specifications written for the subject area;
familiarity with comparable programmes and standards of awards in other colleges;
understanding of external examiners' reports and internal documents such as internal verification, second-marking and second reading;
understanding of programme entry requirements and ability to interpret progression statistics for each stage of the programmes, including withdrawal, transfer and failure rates;
understanding of programme learning objectives;
familiarity with destinations data and employment statistics.

Skills

ability to conduct meetings and interviews with staff;
ability to conduct meetings with a range of current and former groups of students;
ability to write succinctly and coherently;
ability to meet exacting timescales and deadlines;
ability to work effectively as a member of a team;
ability to communicate electronically, including emails, attachments and use of webmail.

Recruitment, training and role of Review Coordinators

15 Review Coordinators are also recruited from individuals nominated by colleges, universities or other organisations, and from individuals who reply to advertisements. They may be seconded or independent consultants. It is expected that they possess extensive experience of HE and of the assurance of standards and quality. They will be expected to perform a number of duties, of which managing reviews and writing reports are the major responsibilities. Opportunities to contribute to other activities such as editing reports, training specialist reviewers and drafting overview reports may also be available.

16 Because of the relative complexity of the Review Coordinator role, the individuals recruited will undergo a longer induction and training process than that provided for specialist reviewers. Induction into the review method will include attendance at, and participation in, at least one specialist reviewer training course, as well as attendance at workshops and conferences arranged by the Agency.

17 Reviews take place throughout the academic year and are variable in length. Review Coordinators will need to organise their time, and reach agreement with the colleges and their teams of reviewers about the pattern of review activities in such a way as to ensure effective use of the time available.

18 All academic reviews consist of four main activities:

preparation for review;
visits to the subject provider;
analysis of documentary evidence;
report writing.

Each Review Coordinator is responsible for maintaining an overview of the range and balance of these activities, and for helping the specialist reviewers to apportion their time effectively. The achievement of an appropriate balance between the various activities requires planning in advance of, and coordination throughout, the review. Above all, it is essential that it enables the reviewers to develop a robust evidence base on which to make judgements.

19 Should the Agency seek to recruit additional Review Coordinators during the current round of reviews, the following criteria for selection will apply.

Knowledge and skills required of Review Coordinators

20 In order to carry out their role, Review Coordinators will need to demonstrate:

Knowledge and understanding of HE provision

recent knowledge and understanding of current issues;
awareness of current teaching methods and curricula;
knowledge and understanding of the assurance of standards and quality;
experience of liaison with senior management and staff at other levels;

Skills

ability to manage small teams (with experience either in HE, FE or in industry);
ability to work within tight timescales and to strict deadlines;
ability to lead a team of experts;
ability to communicate effectively in face-to-face interaction;
ability to produce clear and succinct reports to time;
experience of word processing;
ability to communicate electronically, including emails, attachments and use of webmail.

The essential qualities outlined above might be reinforced by experience of a wide range of teaching at HE level and by experience of programme accreditation by professional, regulatory or statutory bodies, programme approval or validation events, quality audits, quality assessment/academic review or educational inspection.

Annex Di

Timeline of review schedule

The Initial meeting (IM) is the first visit of the whole review team to the provider. The day will include meeting with the course team(s), and may include other activities including meeting with current and former students, meeting employers, scrutiny of documents and student work and visits to sites of work-based learning.

This schedule of activity is set out in relation to the IM up to the judgement meeting (JM). Events after the judgement meeting are calculated from that.

IM minus 5 to minus 3 months	The Agency contacts the provider to agree a date for the IM.

IM minus 8 weeks	Submission by the college of the self-evaluation.

IM minus at least 4 weeks	Preliminary meeting between the Review Coordinator and the provider.

During the month before the IM	Submission by the college of any extra documents arising from the preliminary meeting; preparation for visit, including the collection of the sample of student work.

Initial meeting	Meeting of review team with the course team(s); may include activities such as meetings with current students, former students, employers, scrutiny of documents and student work, and visits to sites of work-based learning.

IM plus up to 5 weeks	Up to two more days of visits by the review team that may involve any of the activities listed above, or elsewhere in this Handbook.

IM plus 6 weeks = judgement meeting (JM)	Judgement meeting, not necessarily on the provider's premises.

JM plus 3 weeks	Notification of judgements to the provider.

JM plus 6 weeks	Provider receives a copy of the draft report to check for factual accuracy.

JM plus 20 weeks	Publication of the report on the Agency's web site.

This timeline is indicative. Individual events may be varied to accommodate specific circumstances such as Christmas, vacations or examinations. However, the Agency will endeavour to complete the publication of reports within 20 weeks of the judgement meeting.

Annex Dii

Documentation flow chart (represented as HTML)

Team composition signed off by Agency officer
Self-evaluation and programme specifications received by Agency
Self-evaluation analysis received from Review Coordinator by Agency officer
Agency officer signs off self-evaluation analysis
Review Coordinator attends preliminary meeting with institution
Preliminary meeting debrief received by Agency officer and any concerns are addressed
Agency officer reviews initial commentaries in folder & takes appropriate action if necessary
Review Team visits institution Visit support by Agency Officer
Use of ARCS monitored by Agency subject coordinator	Agency officer reviewsdraft report and editing reviewer's checklist

Annex E

Academic review teams

Team composition

1 Each review team reflects the nature and scope of the provision in the subject(s)/units to be reviewed. The number of reviewers in each team reflects the size, range and complexity of the education provided. As far as possible, the Agency matches the collective expertise of the team with the broad specialisms of the programmes to be reviewed. Using its database of reviewers and the criteria for composing teams outlined below, the Agency will propose a review team to a college eight weeks before the review starts. The Agency will take account of conflicts of interest declared by specialist reviewers.

2 Colleges are expected to comment on the composition of teams in writing to the Agency. Occasionally, a college will be unable to accept the proposed team. It is essential that any concerns, for example, about a conflict of interest or the team's match to the programmes to be reviewed, are made in writing and discussed with Agency officers as soon as possible after notification and no later than two weeks from notification. Delay in the above may result in undue delay for the preliminary meeting.

3 The main criterion used by the Agency for determining the number of reviewers required for a particular team is the size of the provision described in the Scope and Preference returns provided by colleges. Accordingly, as an approximate guide, review teams will comprise:

three specialist reviewers and one Review Coordinator for all provision up to 250 full-time equivalent (FTE) students;
four specialist reviewers and one Review Coordinator for 250 to 500 FTE students;
five specialist reviewers and one Review Coordinator for 500 to 1,000 FTE students;
six specialist reviewers and one Review Coordinator for 1,000+ FTE students.

4 Teams will not normally have fewer than three specialist reviewers. Very large and/or complex reviews may require more reviewers than shown above. For example, aggregate reviews may need four reviewers even if the FTEs are low.

Team function for academic review

General matters

5 Specialist reviewers focus their attention on the subject(s)/units of review and only address college management matters when they have a direct bearing on the student learning process. For example, academic reviewers might collect information relevant to college practices on external examining/verification when considering assessment practices in relation to a subject.

6 Specialist reviewers assume a collective responsibility for gathering and verifying evidence in relation to academic standards and the quality of learning opportunities. The Review Coordinator will deploy specialist reviewers in the most appropriate way to complete the review activities in the time available. Sometimes the reviewers will work together as a team, for example, for the Initial meeting. At other times, they may work in pairs or, for direct observation of teaching and learning, singly. The Review Coordinator will ask each specialist reviewer to write a brief commentary based on the self-evaluation and the evidence gathered during the review. These commentaries will make full reference to the aims of the subject provider and identify matters for which additional evidence is required. They will inform the reviewers' priorities, the balance of activities undertaken and the collective judgements made.

Review visits

7 The Review Coordinator confirms the intended pattern of the review with the Agency, so that the Agency can monitor the resources required to review the provision. It is usual for reviewers to spend two days in the colleges, although these may not necessarily be consecutive days.

8 The reviewers may visit the college at any time during the period of review, but always by mutual agreement with the subject provider and within the overall number of reviewer days allocated by the Agency. This may involve the whole team visiting together, as for the Initial meeting, or it may involve two or more reviewers visiting for specific observations, meetings or review of documents. Individual specialist reviewers will not normally visit subject providers alone, or hold meetings with staff and students on their own.

9 Each review will include a number of meetings between college staff and reviewers to consider the various questions related to standards and quality. The Review Coordinator is responsible for ensuring that the reviewers consider the accumulating evidence and reach judgements. The Review Coordinator will deploy specialist reviewers in the most appropriate way to complete the review activities in the time available.

10 All reviewers will be expected to identify, share, consider and evaluate evidence related to the programmes under review. The reviewers will keep notes of all meetings with staff and students, their observations, and comments on student work and its assessment, until the report is published. Circulation of these confidential notes within the review team will help develop a collective evidence base on which the judgements can be made. The reviewers communicate with each other and share their analyses and notes through the electronic folder system (ARCS) developed by the Agency to support academic review. The subject review facilitator is able to post electronic material to the review folder by agreement with the Review Coordinator, but is not able to access documents posted there by the reviewers. The reviewers will be expected to evaluate how the accumulating evidence compares with that provided by the college in the self-evaluation, and to test the strength of the evidence adduced to support the judgements. It is essential that discussion of the emerging judgements involve the whole review team.

11 Draft summaries written by the reviewers during the course of a review will focus on the evaluation of evidence related to their particular responsibilities, as agreed by the Review Coordinator. Summaries are analytical rather than descriptive and refer to sources of information as well as to direct observations. Any written evaluation will summarise the relevant strengths and areas for improvement of the provision and, overall, underpin the judgements made. A final meeting of the reviewers will be used to review any additional evidence, to agree the particular strengths and areas for improvement in relation to both standards and quality, to determine precisely what is to be reported and to finalise the judgements.

12 Occasionally, a review team considers it needs additional time to complete a review. The Review Coordinator and the subject review facilitator discuss this proposal. If they agree that extra time is needed, the Review Coordinator applies for this to the Agency officer responsible for the review. The Agency only grants extra time for a review exceptionally.

Reports

13 The Review Coordinator prepares the first draft of the report from the specialist reviewers' summaries immediately after completion of the last visit to the college. Specialist reviewers check and comment on this draft. Following the judgement meeting, the reviewers check the draft report before the Review Coordinator sends it to the college. As the reports provide the main feedback about reviews to colleges, it is particularly important that reviewers check their accuracy carefully.

14 The published reports are the main documented outcomes of the academic review process. Publication usually takes place within 20 weeks from the date the reviewers reach their judgements. Reports should, therefore, be characterised by succinct, accurate writing and a clear, consistent style with judgements clearly related to evidence.

Practical arrangements for reviewers

15 Practical arrangements made by the Agency for reviewers include:

hotel accommodation, where this is required;
travel and subsistence reimbursement;
administrative support.

16 The reviewers will need to have access to word processing facilities. Reviewers compile and transfer written summaries electronically. The reviewers must conform to the Agency's procedures described in the information technology guidelines for use of electronic communication, as these are designed to protect against damage and computer viruses.

Annex F

Preliminary meeting agenda

1 It is important that colleges prepare to discuss each item on this agenda by, for example, assuring that they have up to date student data and timetables available at the meeting.

2 The preliminary meeting provides college staff with a valuable opportunity to clarify their understanding of the review method.

3 The agenda below is indicative and the Agency considers it the minimum necessary to enable the college and the Review Coordinator to establish the requirements of the review. The Review Coordinator and the college may feel it appropriate to include additional items.

Agenda

Introductions
Purpose of preliminary meeting
Clarification of method of review
Clarification of the scope and nature of the provision
Questions arising from initial analysis of the self-evaluation
Confirmation that the statistical data are correct and accurate
The reviewers' requests for information to date
Establishing the programme of review activities during the Initial meeting, including the agenda
Clarification of the availability of documents, including student work
Questions from college staff
'Housekeeping' arrangements
Next steps

Annex G

Documents for academic review, including student work

College documents

1 The reviewers will not normally expect colleges to prepare documents especially for review. Colleges should direct the reviewers in the self-evaluation to the availability and relevance of documents that might assist them to test and verify the statements made, or which are relevant to the judgements they will make.

2 The reviewers will require the following documents in advance of the review:

the self-evaluation, including the programme specifications as annexes;
relevant prospectuses; and
a location map.

3 The Review Coordinator and the college will discuss the availability and relevance of further documents at both the preliminary and the Initial meeting with the college. As the review progresses, the reviewers may ask for further documents at any time during the review. Typically, the following documents will be relevant to the review:

subject or programme handbooks;
curricular documents, module or unit guides;
subject or programme annual monitoring reports, including those from external sources such as professional and/or statutory bodies, if these are available;
summaries and/or analyses of student questionnaire data and follow-up actions;
external examiners'/verifiers' reports for the previous three years;
student intake and progression data for the previous three years.

4 The following documents may also be relevant, but this list is neither prescriptive nor exhaustive:

minutes of relevant meetings, including examination/verification or assessment boards;
summary of equipment and other resources;
practical or work placement handbooks;
programme approval, validation and revalidation documents;
student destinations (further study and employment statistics);
subject staff list and short profiles (indicating main teaching and research interests and any administrative responsibilities of lecturers, and responsibilities of other staff).

5 The reviewers will not necessarily ask for copies of documents in advance. They may prefer to read the documents during the course of a visit. Documents can be provided in electronic form by mutual agreement between the college and the reviewers. The subject review facilitator may post some electronic documents to the ARCS electronic folder by agreement with the Review Coordinator.

6 There is no requirement or expectation that documents will be assembled in a 'base room' for the use of the reviewers. If the reviewers wish to see a document, they will ask for it. Because review takes place over an extended period, it will not be necessary for the college to make every document available immediately.

Student work

7 The reviewers will need to see a sample of student work. The range and nature of student work to be made available to the reviewers will be discussed at the preliminary meeting and confirmed at the Initial meeting. The reviewers will look at student work to evaluate whether:

student achievement matches the intended learning outcomes of the programme(s);
assessment is designed appropriately to measure achievement of the intended learning outcomes;
the assessments set provide an adequate basis for discriminating between different categories of attainment;
the actual outcomes of programmes meet the minimum expectations for the award.

8 The reviewers will not duplicate or 'second-guess' the work of external examiners or verifiers. As such, the reviewers will not normally expect to see work that is currently under consideration by external examiners or verifiers. Colleges are advised to retain an appropriate sample of student work that has been considered fully by the assessment process. The college should provide t

Handbook for academic review: England, 2004 For review of directly funded higher education in further education colleges

Contents

Part one

Introduction

Background

Features of academic review

The review process

Stage 1: Preparation for review

Stage 2: The review period

Stage 3: The reporting stage

Part two

Reference points for review

Teaching Quality Information

Annex A

Annex B

Aide-mémoire for academic review

Annex C

Reviewers

Annex Di

Timeline of review schedule

Annex Dii

Annex E

Academic review teams

Annex F

Preliminary meeting agenda

Annex G

Documents for academic review, including student work

Handbook for academic review: England, 2004
For review of directly funded higher education in further education colleges