Skip navigation

Handbook for academic review

Handbook for academic review (QAA 2000): users guide to the academic review of subjects in higher education institutions in the transitional period 2002-2005

Introduction

1. This Guide, to be read in conjunction with the Handbook for academic review (the Handbook), is intended to help higher education institutions (HEIs) in England in their preparations for subject-level review in the transitional period, 2002-2005. It draws on the experience of two years' operation of the method in HEIs in Scotland and of six months operation in further education colleges (FECs) in England. The Guide follows the structure and sub-headings of the Handbook and provides clear references to paragraphs in the Handbook.

2. In addition to using the Guide to help you to prepare for the academic review of subjects, we are aware that you will be looking beyond the transitional arrangements to your first experience of the new institutional audit process with its discipline audit trails. In this Guide we have suggested some ways of using subject-level review(s) as a preparation for institutional audit, which you may wish to consider.

3. Academic review was originally conceived as a 'dynamic engagement with the internal processes of an institution over a six-year cycle'. It was envisaged that the review of subjects would continuously update the picture and provide audit trails to inform judgements about institutional systems (paragraph 6 refers). The notion of published reports about the quality of programmes and standards of awards as 'outputs' of a single process of external review of subjects no longer applies. However, the underlying principle remains intact.

4. Review at the discipline or programme level will nevertheless generate considerable evidence about the ways in which institutional systems are working in practice. Reports of internal quality assurance reviews at the level of the discipline and/or programme will feed information into the institutional audit process, thereby providing material for audit reports on institutional management of quality and standards. Summary statements of the results and resultant actions of these internal reviews are part of the information set on the quality of programmes and standards of awards to be made publicly available ( Information on quality and standards in higher education: Final report of the Task Group. HEFCE 02/15).

5. In the transitional period, the academic review of subjects offers you the opportunity to test, in cooperation with the Agency, the strength of your internal review processes at the programme or discipline level. It provides a means of checking the robustness of the evidence you use in these processes against the experience of students and others involved at the programme or discipline level. Reports on subject-level reviews will be used as part of the evidence base that informs institutional audit. They will be published on the Agency's website.

The review method

6. The main source of reference for academic review is the Handbook ( http://www.qaa.ac.uk/public/acrevhbook/intro.htm). The academic review of subjects is described in Part 1 and the annexes A, C, D, E, F, G, H, I, J and K. Part 2 and the annexes B, L and M are concerned with institutional review and are not relevant to subject-level review. Institutional audit has now replaced institutional review as described in Part 2 of the Handbook. Further details of the method of institutional audit are provided in the Handbook for institutional audit: England (QAA 2002).

7. Most subject providers have found their experience of academic review positive, seeing benefits over the previous method, notably the 'lighter touch', which the shift in approach towards more selective lines of enquiry encourages.

Understanding the process

8. The inherent flexibility of the academic review method means that it can be readily adapted to the context of the transitional arrangements. For your ease of reference, the modifications we have introduced are grouped under the same headings as the main features of the method in Part 1 of the Handbook (paragraph 17 refers):

  • Preparing for review.
  • Points of reference for the review.
  • Conducting the review.
  • Judgements and reports.

Preparing for review

Scope and preference surveys and initial profiles (paragraphs 18-23)

9. The transitional period, 2002-2005, replaces the two three-year periods of the six-year cycle. The 42 subjects in Annex K of the Handbook represent the units of review. Where only some components of a unit of review meet the criteria for subject-level review, we would like to discuss with you the inclusion of only those components in the review. For example, if you offer programmes in a wide range of Engineering fields and two of these are eligible for review, we would expect to focus on these fields rather than the full range of engineering provision. We will need to gather some information about the programmes to be included in your review(s), but there will be no further scope and preference survey on the scale implied in paragraphs 18-19.

10. There will be no initial profile. Annex B of the Handbook, Constructing a profile, should be disregarded. By extension, such a profile can have no bearing on the likely intensity of the review(s).

11. The self-evaluation document (SED) significantly influences the intensity of review. A genuinely evaluative and self-critical SED is the main factor in enabling a 'lighter touch' in the review process. In planning the pattern of review activity, reviewers will take into account your perceptions about strengths and weaknesses of the current provision, your effectiveness in recognising and addressing areas for improvement, and any positive trends identifiable from recent subject and institution level reports. Guidelines for producing SEDs are in Annex C of the Handbook.

Self-evaluation documents and programme specifications (paragraphs 24-31)

12. Given the centrality of the SED to the review process, we have prepared a separate guidance note, based on what reviewers look for in the SED. This is at Annex 1. It supplements this section of the Guide.

13. The review process focuses on the setting of academic standards by the subject provider, their achievement by students and the quality of the learning opportunities offered to students. It recognises that standards and quality are interdependent and that the relationship between them is dynamic. Reviewers are concerned with the ways the subject provider assures and takes forward standards through promoting student learning and achievement. They are seeking to verify continuity of good practice and establish whether issues identified for improvement from previous internal and external reviews have been addressed.

14. Reviewers are looking for evidence of continuing improvements in teaching, learning resources and academic support; of systematic development of programme learning outcomes for continued relevance; and of regular review of curricula content, assessment arrangements and student achievement in relation to the intended learning outcomes of the programmes. The aide-mémoire in Annex E of the Handbook consists of questions and prompts to assist reviewers to analyse the SED, collect evidence during the review, and prepare the report.

15. Experience suggests that subject providers find it helpful in constructing SEDs to follow the headings and prompts of the aide-mémoire. These closely mirror the section headings of the report. However, we draw to your attention 'Student achievement'. In the aide-mémoire, 'Student achievement' comes under the major heading of 'Assessment', whereas it is actually a section of the report in its own right, underlining the significance of student achievement in outcomes-based learning. You may prefer to follow the report structure and make student achievement a discrete section of the SED.

16. The aide-mémoire prompts reviewers to evaluate whether the standards students achieve meet the minimum expectations for the award; the associated question is about the evidence base (Annex E, paragraph 16 refers). Reviewers draw on primary evidence of student achievement provided by a sample of students' work. They triangulate their observations with secondary evidence, such as external examiners' reports and examination board minutes. Annex H 'Documentation' includes advice on the sample of student work. Student progression and achievement statistics are also relevant. Analysis is typically based on data for the previous three years.

17. The transitional arrangements give institutions the chance to demonstrate enhancement through the subject-level review process. The report also includes a commentary on the 'Maintenance and enhancement of quality and standards' to which no judgement is attached. This provides an opportunity for reviewers to identify good practice, or less effective practice, and ways in which the subject provider may wish to build on strengths or bring about improvements. Again, you may prefer to follow the report structure and develop your approach to enhancement in a discrete section of the SED.

18. Indicators of good practice or a need for improvement, such as those in paragraphs 57 and 58 of the Handbook, are significant in this context. Where good practice is concerned they reinforce track record; where issues have been addressed, this supports enhancement. Instances of changes in progress or at the planning stage are also relevant, particularly if it can be shown how their effectiveness will be evaluated. From our experience of SEDs, subject providers tend not to give concrete examples of how they build on strengths and make improvements. Coverage of maintenance and enhancement of quality and standards often falls short of showing the effects of institutional management of quality and standards at programme level.

19. If you wish, you may use a document prepared for internal review purposes as the SED, provided it meets the guidelines in Annex C of the Handbook, thus removing the need to prepare a separate SED purely for external review (paragraph 26 refers).

20. Programme specifications are the starting point for reviewers as they seek to understand the intended learning outcomes and assessment methods for the programmes under review. The Agency publishes Guidelines for preparing programme specifications (www.qaa.ac.uk/crntwork/progspec/contents.htm), which supplement Annex D of the Handbook. The relevant programme specifications should be annexed to the SED.

21. We will normally require the SED two months before the start of the review period. This arrangement, which supersedes paragraph 31, takes into account that most institutions find it useful to have a preliminary meeting with the review coordinator (CR) to clarify protocols for the review. We strongly recommend this, especially for your first review. We have brought forward the due date for the SED to allow the review team to carry out an initial analysis of the SED, before the preliminary meeting. Where the review coincides with an internal review, a mutually acceptable date for receipt of the SED will be agreed. No SED will be required before 29 November 2002.

Points of reference for review of standards

Framework for higher education qualifications and subject benchmark statements (paragraphs 32-36)

22. When making judgements about the provision under review, the reviewers draw on external reference points, including the Code of practice for the assurance of academic quality and standards in higher education, The Framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ), and any Subject benchmark statements, identified as relevant by the subject provider. Here we emphasise that reviewers are not seeking evidence of compliance, rather evidence that the subject provider has considered the purpose of the reference points, has reflected on its own practices in relevant areas in the light of them and is taking steps to make appropriate changes.

Conducting the review

Peer review against the broad aims of the provider (paragraphs 37-40)

23. Peer review against the broad aims of the provision and the intended learning outcomes continues to be fundamental. Details of the subject specialist reviewer (SSR) function can be found in Annex A of the Handbook, which specifies the role in terms of the qualities required and the experience, knowledge and skills SSRs should demonstrate. It also outlines the arrangements for recruitment and training of SSRs.

Review over an extended period (paragraphs 41-42)

24. The review period is unlikely to exceed six weeks. It starts with the initial meeting of the full review team with the subject provider and ends with the final review team meeting at which the judgements are made. The six-week timeframe has evolved as optimal to deliver the intended benefits of review over an extended period (paragraph 41 refers). It also allows the division of the academic year into set review periods, which has certain practical advantages. It helps us with planning the reviews and enables you to know the start and target completion date(s) for your review(s) well in advance, so you can prepare accordingly.

Judgements and reports

25. These features are described in paragraphs 43-52 of the Handbook, with further details of the categories and criteria for judgements given in paragraphs 81-94. We draw to your attention first, the 'limited confidence' judgement (paragraphs 43 and 81 refer) and second, 'exemplary features' (paragraphs 48 and 93 refer). The 'limited confidence' judgement is a sub-set of the 'confidence' jugement and is rarely made. Its use would be appropriate if the reviewers were satisfied with current standards, but did not have confidence that the standards could be maintained into the future.

26. Exemplary is not a category of judgement above commendable. It signifies excellence in a feature that can be emulated elsewhere and used to promote enhancement. Reviewers will need to draw on their collective expertise to judge whether a feature represents sector-leading best practice. It will be a matter of both fact and judgement as to whether a feature makes a 'significant contribution' to the success of the provision, and is both worthy of dissemination and transferable to other situations.

27. The CR will send formal notification of the judgements to the Vice-chancellor or head of institution within two weeks of the final review team meeting. You will know the date of this meeting in advance. There is no formal session concluding the review at which you are given oral feedback. Feedback on the review is provided through the draft report. You will be asked to comment on the factual accuracy of the report.

28. There are now a significant number of published reports (www.qaa.ac.uk/revreps/reviewreports.htm) which give an indication of format, style and tone. We have withdrawn the paper 'Making Judgements' , formerly used as guidance in reviewer training. We do not intend to produce overview reports on completion of the schedule of subject-level reviews in the HEIs in the transitional period (paragraph 52 refers).

How the process works - preparing for review

Liaison, Advance Planning and preparatory meetings (paragraphs 53-58)

29. For scheduling purposes, we now work on the basis of fixed six-week review periods, with due dates for submission of SEDs two months before the start of review periods. Review periods for the remainder of the 2002-03 academic year and corresponding due dates for SEDs, are set out below. The arrangements supersede paragraphs 53-55. We will notify you by January 2003 of the due dates for submitting SEDs for reviews due to take place in 2003-2004.

Review period

Due date for SED

03 February 2003 - 14 March 2003

29 November 2002

17 March 2003 - 10 May 2003

13 January 2003

13 May 2003 - 21 June 2003

17 March 2003

30. Once your review(s) have been allocated to period(s), you will be asked to propose, for each review, specific dates within the first two-weeks of the respective period when the initial meeting of the review team with the subject provider can take place. We shall confirm the date, after establishing the reviewers' availability.

31. SEDs may be presented either electronically, which is the Agency's preferred means, or as hard copy (three copies). On receipt of a SED, we make routine administrative checks before sending it to the CR. At this stage, we will ask you to send SEDs to the SSRs, along with any other advance documentation. Reviewers should receive the SED at least four weeks before the initial meeting. Annex H of the Handbook lists minimum requirements for advance documentation.

32. In the lead-up to the initial meeting, the CR will contact you to explore whether a preliminary meeting would be beneficial or whether telephone or email communication is preferred. In our experience, institutions find face-to-face discussions with the CR helpful, particularly for first reviews. In particular, institutions have told us that they value the opportunities which these meetings provide to clarify subject staff's understanding of the review process and to arrive at a shared understanding with the CR of the detailed plan for the first stage of review. We are working on the likelihood that most HEIs will opt for a preliminary meeting, in our planning for the reviews in the transitional period.

Academic review teams (paragraphs 59-61)

33. We will send you brief details of the team(s) of SSRs some 8-12 weeks before the start of the review period. As far as possible, within available resources, we will match team size and SSRs' expertise with the broad areas of the provision. You will have the opportunity to discuss any concerns about the extent to which the specialisms of the review team match those of the provision, as well as other concerns, such as a conflict of interest of which we are unaware. The criteria for team composition and team size are in Annex G of the Handbook.

Analysis of the self-evaluation (paragraphs 62-63)

34. The reviewers carry out an initial analysis of the SED to ensure that it is evaluative, rather than merely descriptive (paragraph 63 refers). An institution may be asked to revise a document if it fails to provide a suitable basis for the review. To stress, an evaluative SED that directs reviewers to the evidence for statements made is the key to a less intensive review.

35. Here we would like to point to the value the subject review facilitator (SRF) can add over and above providing effective liaison between the team of reviewers and subject staff. An SRF can also help subject staff prepare for review by disseminating good practice within the institution and highlighting areas for improvement, identified by previous review activity. Details of the role, which is optional at the choice of the institution, are given in Annex F of the Handbook. The chosen SRF needs to participate in a briefing session provided by the Agency before facilitating a review.

Conducting the review

36. This is given very full treatment in paragraphs 64-74 of the Handbook. There is additional guidance in Annex E 'Aide-mémoire', Annex I, 'Observation of Teaching'and Annex J 'Agenda for meeting with students'. To reinforce, a meeting with current students is a mandatory part of every subject-level review and it generally follows the published agenda. So far there has been no observation of teaching in any subject-level review. Take-up of the facility for reviewers to attend relevant internal quality assurance events, by arrangement with the subject provider, has been low. This is because institutions have usually found that they can provide answers to reviewers' enquiries in ways which are less intrusive and less demanding of reviewers' time.

37. Good practice in conducting reviews requires regular feedback to the subject staff on progress. Organisation and management of the review is the responsibility of the CR and this extends to ensuring subject staff are aware of the issues being addressed, the evidence needed to clarify them and crucially, when lines of enquiry have been closed. As well as formal meetings between the reviewers and subject staff to investigate matters specific to standards and quality, regular progress meetings are held between the CR and subject staff. This is in addition to the liaison function of the SRF.

Annex 1

Guidance note on the self-evaluation document

1. This guidance note approaches the self-evaluation document (SED) from the standpoint of what the reviewers look for in a SED.

2. The academic review of subjects is centred on the SED produced by the subject provider. The reviewers'first task is to test by means of their own observations and analyses the statements made by the subject provider in the SED. Secondly, they will make judgements on the appropriateness and effectiveness of the provision (paragraph 8, the Handbook).

3. As stated in paragraph 13 of this Guide, reviewers are concerned with the ways the subject provider assures and takes forward standards through promoting student learning and achievement. They are seeking to verify continuity of good practice and establish whether issues identified for improvement from previous internal and external reviews have been addressed.

4. Paragraph 14 of this Guide reminds readers that reviewers are looking for evidence of continuing improvements in teaching, learning resources and academic support; of systematic development of programme learning outcomes for continued relevance; and of regular review of curricula content, assessment arrangements and student achievement in relation to the intended learning outcomes of the programmes. The aide-mémoire in Annex E of the Handbook consists of questions and prompts to assist reviewers to analyse the SED, collect evidence during the review, and prepare the report.

5. The subject provider is encouraged to use the SED to demonstrate on-going rigorous analysis and self-reflection. The SED should set out clearly what the issues have been, how they have been addressed and what remains to be done; it should identify strengths and how these are being built on.

6. During the transitional period, reviewers will be guided to focus on three related functions that the process of self-evaluation should perform. These are to:

  • encourage staff engaged in the programmes to evaluate the standards achieved by students and the quality of learning opportunities offered to them;
  • provide a basis for the review;
  • offer an opportunity for the HEI to prepare for institutional audit and learn from the experience of the subject-level review.

7. The first two functions are rooted in the academic review method of subject-level review. The third addresses the transition to institutional audit. Therefore the prompts for reviewers also draw on the paper Arrangements during the transitional period 2002-2005 for Higher Education Institutions in England and Annex C of the Handbook for institutional audit 'Guidelines for producing self-evaluation documents for discipline audit trails'.

8. Reviewers will explore whether the SED:

  • illustrates how internal processes play a full part in assuring academic standards and supporting continuing improvement;
  • supports a clear statement of overall educational aims is in place and informs quality assurance processes;
  • shows that the subject provider and the institution address academic standards and the quality of learning opportunities through the internal review process;
  • gives a self-critical account of known strengths and perceived opportunities for enhancement;
  • identifies the issues associated with any weaknesses, and shows that the steps being taken to bring about improvements are openly discussed and reported;
  • demonstrates the benefit to current provision from changes that have taken place and considers changes that may be necessary in future ;
  • confirms supporting information, both quantitative and qualitative, is already available or steps are being taken to develop a comprehensive evidence base in the near future. 1

9. Reviewers will expect to see examples, from the past two to three years, of how internal processes have resulted in enhancement. The SED might give at least one example under each of the headings of 'standards' and 'quality' in the aide-mémoire or alternatively focus on say four or five specific examples. The SED might also identify needs for improvement that the subject provider intends to address in the near future, outline how internal procedures will be used to bring about changes and how the effectiveness of any changes made will be evaluated.

10. Subject providers will wish to choose their own examples, but possible areas from which examples might be drawn are as follows. Narrative should show evaluation and point to the evidence for statements made.

  • How periodic/major internal review provides a distinctive focus for standards and quality over and above the annual monitoring process.
  • How the outcomes of either periodic/major internal review or annual monitoring have been translated into action to bring about enhancement.
  • How external reference points have been used at the subject level - to include an overview/critique of any central guidance provided by the institution, for example, on presenting programme specifications, implementing sections of the Code of practice, using Subject benchmark statements or the Framework for higher education qualifications (FHEQ).
  • How the subject provider is coming to terms with outcomes-based learning, formally linking assessment to intended learning outcomes, developing explicit grading criteria matched to outcomes, using these in assigning grades, giving formative feedback to students. This would include a review or critical commentary on central guidance provided within the institution.
  • How the subject provider takes into account national developments, for example through the Learning and Teaching Support Network (LTSN) subject centres.
  • How the subject provider ensures it is getting good value from its external examiners - to include a view on guidelines given to external examiners, often issued centrally. For example, do external examiners comment on 'level' of module learning outcomes in terms of Subject benchmark statements; on student achievement against intended learning outcomes or the level of the final qualification in terms of its location in the FHEQ?
  • How the subject provider interprets and uses statistical indicators/management information from central systems - to include a view on whether institutional targets are 'right'. For example, what is an 'acceptable' cohort completion rate in terms of the subject provider's aims?
  • How the subject provider has responded to previous subject review reports, including subject overview reports and relevant issues from institutional audits.

11. Discussion in the SED about maintenance and enhancement of quality and standards should not focus too narrowly on institutional level and formal process, but also show what happens at programme level. For example, it might include consideration of the strategy for, and implementation of, staff development, or the use made of advice from external examiners/advisers. Narrative could be freed from a mere description of institutional structures and procedures by the provision of a diagram.

12. Programme specification(s) are the principal annex of the SED. Here reviewers will concentrate on whether programme specification(s) provide user-friendly, accurate information that is consistent with other sources, such as the SED, website, promotional literature, student/programme handbooks.

1 Subject providers may wish to consider supporting information in the context of HEFCE 02/15. Reviewers will not take a checklist approach to the information sets.