The Subject Review Method
1. The purpose of this Subject Review Handbook is to describe the method and procedures for carrying out subject reviews in England and Northern Ireland during 2000 and 2001. It describes for reviewers and institutions the protocols, procedures and practical organisation of subject reviews from advance planning through to the publication of reports. Other publications such as the 'Higher Quality' bulletin can be found on the Quality Assurance Agency for Higher Education (QAA) web site at www.qaa.ac.uk.
2. The Higher Education Funding Council for England (HEFCE) has the statutory responsibility under the terms of the 1992 Further and Higher Education (FHE) Act to assess the quality of higher education. The HEFCE has signed a service level agreement with the QAA, which commissions the Agency, to conduct subject reviews in England and Northern Ireland in 2000-01. This programme of subject reviews completes the cycle of review, begun in 1993, of all subject areas in higher education. The 11 subjects to be reviewed during 2000-01 are:
- Hospitality, Leisure, Recreation, Sport and Tourism
- Librarianship and Information Management
- Business and Management
- Economics
- Politics
- Classics and Ancient History
- Celtic Studies
- Archaeology
- Philosophy
- Theology and Religious Studies
- Education.
This Subject Review Handbook has been updated and contains minor amendments and improvements. The subject review process remains essentially the same as that used during 1998 to 2000.
3. In accordance with the requirements of section 70 of the FHE Act of 1992, the HEFCE's purposes in contracting with the QAA for the review of the quality of higher education (HE) are:
a. To secure value from public investment:
i. by ensuring that all education for which the HEFCE provides funding is of approved quality, and by encouraging speedy rectification of major shortcomings in the quality of education;ii. by using subject review judgements to inform funding.
b. To encourage improvements in the quality of education through the publication of subject review reports and subject overview reports, and through the sharing of best practice.
c. To provide, through the publication of reports, effective and accessible public information on the quality of higher education.
4. The 2000-01 subject reviews will be carried out broadly along the lines already established. As far as possible, and within available resources, the QAA will seek to review all HE provision, including provision in higher education and further education institutions that is not funded by the HEFCE or the Department of Education for Northern Ireland.
5. A new process of academic review will begin in Scotland and Wales in October 2000 and in England and Northern Ireland in January 2002. The QAA is preparing for academic review and during 2000-01 will continue to produce subject benchmark information and codes of practice. These documents will provide background information only and will not be a part of the subject review process covered by this Handbook unless institutions include specific references to this material within their self-assessment documents.
6. The QAA attaches great importance to working collaboratively with professional and statutory bodies (PSBs) where possible. The HEFCE Circular 3/97 identified four models for collaboration:
a. A PSB suspends its own accreditation visits programme and uses the evidence from assessments (reviews) to determine whether an accreditation visit is necessary.
b. A single combined panel of reviewers and accreditors carries out assessment (review) and accreditation in one visit with one shared set of documentation.
c. Two panels carry out assessment (review) and accreditation separately, but at the same time, with some sharing of documentation, and possibly some sharing of personnel.
d. The panels carry out assessment (review) and accreditation separately, at different times, but with some common documentation.
7. The QAA seeks to adopt a flexible approach to collaborative arrangements. It will inform institutions of agreements reached with PSBs in relation to the subject reviews in 2000-01 and, where collaboration involves combined visits, may provide supplementary written guidance.
8. A number of developments introduced for 1998 to 2000 have proved effective and will continue in 2000-01. These developments included:
a. An increased emphasis on the structured review of student work.
b. A more systematic use of the reports of external examiners.
c. Publication of a register of reviewers.
d. The use of institutional facilitators.
Briefings for newly appointed institutional facilitators will take place during 2000 and nominations should be sent to Helen Markham at QAA, Southgate House, Southgate Street, Gloucester, GL1 1UB. Further details are set out in Annex N.
9. Taken as a whole, these changes are moving subject review in the direction of a more outcomes-based approach, as recommended by Dearing. At the same time, they are giving the institution a greater voice in key aspects of the process, through the revised arrangements for team composition and the option of appointing a facilitator.
10. Academic review is now at an advanced stage of planning. The new process incorporates innovative features but also draws heavily on what has been learnt from quality assessment and subject review. The value of the work conducted by the QAA and its predecessors between 1993 and 1999 should not be underestimated. I hope you find this Handbook interesting and useful. For those of you who will be taking part in the training programme, it is a key document. If you would like to make any comments on it or on any other matters related to the process of review, please contact me at the address given below:
Dr Peter Milton
Director of Programme Review
QAA
Southgate House
Southgate Street
Gloucester GL1 1UB
Introduction
11. The subject review method evaluates the quality of educational provision within a subject area as defined by a unit of review. It is focused, at the level of the subject, on the quality of the student learning experience and student achievement. Taught programmes of study at all levels are reviewed.
12. The following are the main features of the method:
- peer review
- self-assessment
- review against the subject provider's aims and objectives
- aspects of provision
- review visit
- graded profile
- overall summative judgement
- subject review report.
Peer Review
13. Reviews are carried out by a team of registered subject specialist reviewers, led by a review chair. It is the subject specialist reviewers' main responsibility to gather evidence and to make judgements on the quality of education provided. Subject specialist reviewers are drawn mainly from the further and higher education sector, although nominations and applications from industry, commerce and the professions are actively sought. Subject specialist reviewers are trained by the QAA in conjunction with the Universities and Colleges Staff Development Agency (UCoSDA) before their first review visit. Each newly appointed review chair is provided with training in managing visits and other relevant skills, and attends a subject specialist reviewer training course.
Self-Assessment
14. The self-assessment presents the aims and objectives of the provision and provides an evaluation of the student learning experience and student achievement. The evaluation is organised within the structure given by the six aspects of provision and should be supported by evidence. A self-assessment should discuss both strengths and weaknesses in the provision. Where weaknesses are acknowledged, the subject provider is encouraged to discuss the issues and the steps taken to improve quality. Guidance on the preparation of self-assessments, including the presentation of statistical data on student progression and achievement, is set out in Annex B.
Review against the Subject Provider's Aims and Objectives
15. The statement of the subject provider's aims and objectives included in the self-assessment provides a reference point for the review and is published in the subject review report. The statement of aims and objectives should be sufficiently clear to enable a review visit to be planned and undertaken. Reviewers evaluate the extent to which the student learning experience and student achievement in each of the aspects of provision contribute to meeting the objectives set by the subject provider. Reviewers also evaluate whether the objectives set, and the level of attainment of those objectives allow the aims set by the subject provider to be met.
Aspects of Provision
16. The aspects of provision provide a common structure for each stage of the subject review process: the self-assessment, the subject review visit, the judgements and the subject review report. The six aspects of provision are:
Curriculum Design, Content and Organisation
Teaching, Learning and Assessment
Student Progression and Achievement
Student Support and Guidance
Learning Resources
Quality Management and Enhancement.
It is intended that the aspects of provision should be sufficiently broad and inclusive to enable providers and reviewers to give full weight to the particular features of the provision in different contexts. Broad indications of what might be addressed within each aspect are listed in the Subject Review Aide Memoire (Annex A), which differentiates between the key features of each aspect and matters which interrelate and influence more than one aspect of provision. Subject providers may address matters in addition to those listed in the Subject Review Aide Memoire in order to provide an evaluation that is appropriate to the specific provision.
The Subject Review Visit
17. The purpose of the subject review visit is to gather, consider and test the evidence of the quality of education, in light of the subject provider's aims and objectives, and to establish a graded profile and an overall judgement on the quality of that provision.
The Graded Profile
18. The graded profile shows the extent to which the student learning experience and student achievement demonstrate that the aims and objectives set by the subject provider are being met. The profile is created by applying a grade to each of the six aspects of provision. The assignment of grades is a matter for the professional judgement of the review team, drawing on evidence from both the self-assessment and the review visit. The grade descriptors are set out in Annex L.
19. There are four numerical grades on the scale: 1, 2, 3, 4 in ascending order of merit. A grade of 2 or better in an aspect means that the aspect makes at least an acceptable contribution to the attainment of the stated objectives and that the aims are at least broadly met. A grade of 1 means either that the aspect does not make an acceptable contribution to the attainment of the stated objectives, or that the level of attainment of the objectives does not allow the aims set by the subject provider to be met.
The Overall Summative Judgement
20. The overall summative judgement is derived from the profile. Each aspect has equal weight. A profile with all aspects graded 2 or better will be reported as 'quality approved'. A profile with three or more grade 2s will result in a request by the QAA for the institution to provide an improvement plan. A profile with one or more aspects of provision graded 1 will be 'subject to further review within a year'. If, after the further review visit, the profile contains one or more aspects of provision graded 1, then the provision will be reported as 'quality not approved'. The judgements are given to the institution in an oral feedback by the review team at the end of the review visit.
The Subject Review Report
21. A subject review report is published after each review visit and is the main documented outcome of the review process.
Monitoring and Evaluation
22. Continuous monitoring and evaluation of the review process are integral to the QAA's activities. A sample of subject review visits are supported and monitored by QAA officers. Each review is also monitored through questionnaires completed by institutions, subject specialist reviewers and review chairs.
Preparation for Subject Review
Introduction
23. The preparation for review may be divided broadly into five stages: advance planning and self-assessment by the institution; analysis of the self-assessment; composition of the review team; the preparatory meeting; and preparation by the review team. The preparations for review are most effective when they are undertaken with full co-operation between the institution, the officers of the QAA, the review chair and the subject specialist reviewers.
Advance Planning
24. Advance planning is undertaken by the QAA in conjunction with the institutions. The precise timing of review visits is a matter for discussion and agreement between the institutions and the QAA's officers. Each institution completes an information form for each unit of review in which it has provision, to include information on the size, scope and nature of the provision to be reviewed, and the institution's preferred dates for the review visit. This advance information provides a basis for further discussions with institutions to plan and agree the timing of individual visits, and informs the composition of the review team. The scheduling of review visits may also include liaison with PSBs.
25. Institutions are asked to prepare their self-assessment in time to be submitted to the QAA up to six months before the date of the visit. Guidance on the self-assessment is provided in Annex B. The dates for the submission of self-assessments are also included in Annex B.
26. Institutions may make proposals for a joint or parallel review of provision. In joint reviews, one self-assessment and one review visit cover more than one unit of review. There is one review team and the visit results in one graded profile and one published report covering all the units reviewed jointly. In parallel reviews, up to three review visits take place in one institution at the same time. There are separate self-assessments and separate review teams, but some shared meetings. Each parallel review visit leads to a separate graded profile and published report.
27. Where part or all of the provision is organised in collaboration with other educational institutions in England or Northern Ireland, arrangements for sampling the range of centres will be discussed in advance of the review visit.
Analysis of the Self-Assessment
28. The self-assessment provides the basis for reviewers to set the priorities and the programme for the review visit. The Agency will check the self-assessment. The main functions are to ensure that the aims and objectives are clear and the document complies with the specified length and structure.
29. The test of the clarity of the aims and objectives is made by asking the following questions:
- do the aims express broad educational purposes for the provision in the subject?
- do the objectives set out the intended learning outcomes for each programme of study in the subject?
- are the aims and objectives expressed with sufficient clarity to develop an agenda for the review visit?
- are the aims and objectives consistent with those of the institution, where explicit reference is made to the latter?
- is there a clear relationship between the stated aims and the stated objectives?
30. If necessary, the subject provider may be asked to re-submit the statement of aims and objectives. Once clarification has been achieved, the preparatory meeting is held and the arrangements for the review visit proceed.
31. The review chair analyses the evaluation section of the self-assessment, using the Subject Review Aide Memoire (Annex A), before the preparatory meeting with the subject provider. At the preparatory meeting, the review chair may wish to clarify her/his understanding of the self-assessment, in order to achieve the best possible preparation for the review visit.
The Review Team
32. A review chair leads each review team and is responsible for co-ordinating and managing the review visit, and for ensuring that it is conducted within the QAA's guidelines. This involves: preparing for the review visit in liaison with the institution; co-ordinating the work of the subject specialist reviewers on the review team; ensuring that evidence is gathered and tested and that the judgements reached are robust; providing oral feedback to the institution at the end of the visit; and preparing the subject review report after the visit. The review chair is not normally a specialist in the subject being reviewed.
33. The responsibilities of subject specialist reviewers include: analysing the self-assessment and other documentation provided in advance of the visit; participating in the review visit to gather, share and test evidence; making judgements on the quality of the education; and commenting on the draft subject review report after the visit.
34. The QAA publishes a register of subject specialist reviewers, which is circulated to all institutions. The register shows, for each reviewer, the unit(s) of review and the main areas of teaching and learning that s/he is qualified to review. The experience and expertise of the subject specialist reviewers reflect, as far as is practicable, the range of provision in the subject across the sector.
35. Subject specialist reviewers may be asked to undertake up to four review visits over the review period. Reviewers are asked to complete an availability form on which they declare any conflicts of interest with institutions.
36. Using the register and criteria for review team composition, the QAA proposes a subject review team to the institution, normally not less than three months before the review visit. The number of reviewers in each team reflects the size and range of the education provided, taking due account of complexities such as operations on more than one site. The minimum number of subject specialist reviewers for each review team is three, although this may be reduced to two for visits to small higher education provision in further education colleges. The criteria for review team composition are set out in Annex C. For review visits that are combined with professional regulatory visits, the proposed team may be discussed with the relevant PSB. The institution is invited to comment on the composition of the team in relation to the published criteria. The institution will alert the QAA of any concerns about the suitability of the team as soon as possible after receipt of the team details. Where the institution considers, following discussion with the QAA subject officer, that the published criteria have not been met and the team is unsuitable, these concerns should be put in writing to the QAA.
The Preparatory Meeting
37. The review chair arranges a preparatory meeting through the nominated central institutional contact. The meeting is normally held a minimum of two months before the review visit. The main purpose of the preparatory meeting is to discuss:
- any necessary clarification of the self-assessment
- the means of securing a representative sample of provision, particularly in complex reviews
- sampling franchise provision, where appropriate
- the scope and nature of the provision to be reviewed
- the role of the institutional facilitator, where relevant
- the range of student work available for scrutiny and the extent to which this constitutes a representative sample of student achievement in the subject
- the range of teaching and learning sessions available for observation during the visit, and the extent to which this will enable an appropriate sample to be covered
- the documentation to be sent to reviewers prior to the review visit, the timetable for its receipt, and the documentation to be made available during the visit
- meetings to be arranged in advance
- the practical arrangements for the visit.
The agenda for the preparatory meeting is set out in Annex D. Following the preparatory meeting, the review chair writes to the institution (with a copy to the subject officer in the QAA), in order to confirm the arrangements agreed for the review visit.
38. The review process requires the institution to make available accurate, representative and accessible information and evidence. Annex E summarises the range of documentation and information which is usually provided prior to and during a review visit, and provides guidance on the sample of student work to be provided. The overall volume of material provided in the base room should be manageable within the scope of a three-and-a-half day visit. The student work provided should be sufficient to secure a reasonably representative sample, as indicated in Annex E. In most cases, institutions will be able to identify appropriate samples from the materials that are kept routinely for examination purposes. However, where relevant, samples of recently marked work that do not contribute to final assessment may be requested.
39. The institution sends the self-assessment and other advance documentation to the reviewers not less than three weeks before the review visit (see Annex E). Any other communication between the subject specialist reviewers and the institution, before or after the review visit, should be channelled through the review chair.
Preparation by the Review Team
40. The review chair contacts the subject specialist reviewers to discuss arrangements for the visit and to agree the responsibilities within the team. Subject specialist reviewers normally assume responsibility for co-ordinating and sharing evidence relating to one or more aspects of provision. They may also maintain oversight of one or more programmes of study. The review chair provides a written briefing for all team members at least three weeks before the visit. The reviewers receive information on the administrative arrangements for the visit directly from the QAA.
41. Prior to the visit, subject specialist reviewers are asked to produce a brief written commentary on assigned aspects, based on the self-assessment and the other advance documentation, including external examiners' reports. In producing this commentary, reviewers should refer to the questions in the Subject Review Aide Memoire (Annex A). The commentary should make full reference to the aims and objectives of the subject provider and identify matters on which further evidence is required. This should include comments on any internal arrangements for peer review of teaching that are described in the self-assessment. Reviewers' commentaries are discussed at the first review-team meeting; they inform the team's priorities and the balance of activities undertaken during the review visit.
Practical Arrangements for Review Visits
42. Three weeks in advance of the visit, the QAA confirms the arrangements for:
- hotel accommodation
- claiming travel and subsistence expenses
- secretarial support
- a laptop computer and printer support.
43. Subject specialist reviewers who bring laptop computers that are compatible with the QAA's equipment may compile and transfer written summaries electronically. However, reviewers are expected to conform to procedures designed to protect against damage and computer viruses, which are outlined in the information technology (IT) guidelines supplied to all reviewers and regularly updated.
The Subject Review Visit
Introduction
44. The review visit is conducted in a spirit of dialogue and co-operation between the institution, the subject staff, institutional facilitator and the review team. The main purpose of a review visit is to enable reviewers to gather sufficient evidence on the provision in a subject to allow them to form a collective judgement on the quality of education and student achievement, measured against the aims and objectives of the subject provider. A review visit normally lasts three-and-a-half days. A common range of activities is undertaken during visits, although the precise order and balance may be adapted to reflect the nature of the provision.
The Programme for a Review Visit
45. An outline programme is agreed at the preparatory meeting and included in the advance documentation the institution sends to the reviewers. A typical schedule for a programme is set out in Annex F.
46. The activities carried out by a review team during the visit include the following:
- scrutiny of institutional and course documents, reviews and reports, including external examiners' reports
- sampling of student work (examination scripts, coursework, projects, artefacts and dissertations)
- observation of the various forms of teaching and learning being carried out during the review visit (including direct observation of classes, seminars, workshops, tutorials, practicals and placements as appropriate)
- meetings with academic and administrative/support staff
- meetings with students, former students and, where appropriate, employers
- consideration of the learning resources
- meeting of the review team to consider the evidence, share information and form judgements.
47. The review chair maintains an overview of the range and balance of review activities, and guides subject specialist reviewers in apportioning their time. The achievement of an appropriate balance between sampling student work, observing teaching and learning, reading documentation, meeting staff, students and other stakeholders, producing written summaries and arriving at a collective judgement requires planning and co-ordination. It is essential that the overall range and balance of activities undertaken enable the team to develop a robust evidence base for the judgements to be made in relation to each of the six aspects of provision. Subject specialist reviewers are expected to prepare and agree individual timetables with a view to achieving this evidence base. In line with increasing emphasis on learning outcomes, reviewers should allocate sufficient time for reading student work.
48. On arrival at the institution, the review team meets before being introduced to the head of the institution and/or other senior representatives. At this meeting, it can be helpful if the institutional facilitator informs reviewers of any matters, from the institutional perspective, which may be important to their understanding of the subject being reviewed.
49. An initial meeting with available subject staff on the first afternoon of the visit allows an opportunity for the staff representatives to make a brief presentation on the provision to be reviewed, and to inform reviewers of any developments since the self-assessment was undertaken. The meeting with subject staff also provides an opportunity for the review chair to remind staff, with team members present, of the subject review method and its protocols. Student representatives may participate in the initial meeting with subject staff.
50. The review visit programme includes a range of meetings between members of the institution and the reviewers to consider the various aspects of the provision. Throughout the visit, the review team meets daily to discuss its programme and its findings. The institutional facilitator is an observer at most meetings and may provide factual information relevant to the team's discussions. However, the institutional facilitator may not attend team meetings or parts of meetings which involve direct discussion of grades, such as the final meeting.
Gathering Evidence
General Considerations
51. The review is focused on the extent to which the student learning experience and student achievement demonstrate that the aims and objectives set by the subject provider are being met. In gathering evidence, reviewers should refer to the Subject Review Aide Memoire (Annex A). All reviewers are expected to identify, share, consider and evaluate evidence related to all aspects of provision.
52. Reviewers make notes of meetings with staff and students, of teaching and learning observations and of comments on the assessment and quality of student work. The collation and circulation of notes within the review team assist in developing a collective evidence base for the judgements made. Reviewers refer to these notes in preparing written evaluations of the quality of provision; the notes are retained by the review chair at the end of the visit.
Sampling Student Work
53. The sampling of student work is an important dimension of the activities undertaken during a review visit. Subject review requires reviewers to:
- make judgements about the student achievement of the learning outcomes identified as objectives by the subject provider
- evaluate the contribution that assessment makes to student learning
- sample student work to evaluate the quality of assessment design and feedback to students.
Details of the sample of student work required for the purposes of review are provided in Annex E.
54. Judgements about the quality of student achievement are mainly based on the sampling of student work, external examiners' reports and observations of teaching and learning. Reviewers also consider the statistical data provided. The reviewers record their comments on the Student Work and Assessment Note (Annex I) and the Observation Note (Annex H).
55. Where the arrangements made by the subject provider include oral feedback to students instead of, or in addition to, a written commentary, reviewers will take this into consideration. Reviewers record their comments on assessment design and feedback in the Student Work and Assessment Note (Annex I).
Observation of Teaching and Learning
56. The arrangements for the review of teaching and learning during the visit should reflect the nature of the provision and normally include observations of a sample of scheduled sessions. The aim is to obtain a collective view of the quality of the teaching and learning, and to add to the overall understanding of the quality of the student learning experience. Reviewers should try to observe a reasonably representative range of teaching and learning sessions, covering each programme of study, and each level and mode of study. The institution may make arrangements for the observation of placements and other off-site teaching in advance of the review visit. In order to achieve an effective sample, the review chair prepares a preliminary schedule of observations for discussion with the review team.
57. The reviewer should meet the member of staff responsible for a teaching and learning session before it commences in order to introduce herself or himself, to discuss the overall objectives of the activity and to determine how students are intended to learn from it. Understanding the overall purpose of the teaching/learning session is highly important. For example, a lecture delivered for the express purpose of transmitting information will be structured differently from one designed to elicit student participation or stimulate extensive further reading. Reviewers will not make comments during a lecture, seminar or tutorial and should not be intrusive or engage directly in the teaching or learning. In the case of long sessions (more than an hour), an appropriate period or periods of observation should be agreed, if possible, at the preparatory meeting.
58. On occasion, students engaged in learning activities in practical sessions and independent learning may be asked by reviewers to talk about their learning experiences and how the activity being observed fits within their wider programme of study. As with other observations, reviewers should endeavour to meet the appropriate member of staff to ascertain the overall objectives of the session, and provide feedback wherever possible.
59. A standard Observation Note (Annex H) is completed for each teaching/learning session observed. In making their judgements about an individual session, reviewers evaluate whether the teaching and learning approaches and the materials used are effective in achieving the intended learning outcomes. This includes consideration of any relevant written or computer-based guidance for students and samples of student work where these are available.
60. After attending a session, the reviewer offers a brief oral feedback to the member of staff, even if this includes a later appointment being made to provide the feedback. This oral feedback is confidential to the member of staff, and is always given privately. The purpose of the feedback is to offer constructive comment on the observations made rather than to prescribe preferred practice. Reviewers must preserve the anonymity of staff in all written reports and in discussions with other members of the institution.
Learning Resources
61. Reviewers also gather evidence through direct examination of the student learning resources. Reviewers normally visit the facilities made available to the subject, and may observe students and staff using specialist IT or other equipment in the course of normal teaching and learning activities. In looking at library provision, reviewers may undertake catalogue searches or request access to on-line facilities. In evaluating the quality of learning resources, reviewers' direct observations of facilities are considered alongside evidence from student work, written documentation, meetings with relevant staff, and meetings with students. The emphasis is on access and use of facilities by students in the subject. The review of learning resources may inform judgements in relation to other aspects (such as: curriculum design, content and organisation; and teaching, learning and assessment), which are affected by the quality of available resources.
Meetings with Staff
62. As a general principle, reviewers focus their attention at the level of the subject. The review team addresses matters at a central institutional level only when necessary because of their impact on the learning experience of students in the subject. It may be helpful to structure meetings with staff around individual aspects of provision. For example, reviewers might meet jointly with academic counsellors, personal tutors and representatives of the student welfare services to gain an overview of student support and guidance. Alternatively, they might meet jointly with a departmental library representative, subject librarian and representative of the central computer services and/or other relevant staff to discuss learning resources in the subject. In developing specific agendas for these meetings, reviewers refer to the Subject Review Aide Memoire (Annex A) and keep a written record of proceedings.
63. Daily meetings between the review chair, the institutional facilitator and the subject contact provide an opportunity to highlight particular issues which the reviewers wish to explore, and for the institutional facilitator and the subject contact to raise any matters concerning the conduct of the review. Individual reviewers are encouraged to make any request for additional documentation or further meetings through the review chair. The review chair will liaise with the institutional facilitator and the subject contact in making these arrangements.
64. A clarification meeting between the review team and subject staff during the latter part of the visit (normally in the afternoon of day three) provides reviewers with an opportunity to identify, as far as possible, all key matters likely to affect the graded profile and to seek clarification of points. It also enables staff to engage in a dialogue about matters arising during the review visit. This meeting is not for the interim feedback of conclusions, nor is it the final opportunity for dialogue between the review team and staff.
Meetings with Students, Former Students and Employers
65. The views of students about the quality of their learning experience and achievements are important evidence. Whenever possible, all of the review team should attend the meetings with students, which typically last an hour. The review chair generally chairs these meetings. At the beginning of the meeting, the review chair introduces the review team and provides a brief outline of the subject review method. It should be emphasised that the views of students are valued, as subject review focuses on the student learning experience and student achievement. Those attending should be assured that the contributions made will not be attributed to individuals. Reviewers should also state clearly that their final evaluation of the quality of provision will be based on a wide range of evidence, of which comments made by students form one part. One team member or the review chair or the team secretary should take notes and ensure that a summary of the main points is circulated among the team. This summary should not attribute comments to individuals. The institutional facilitator does not attend these meetings, but may assist the team in identifying further information in order to clarify any issues students raise.
66. Discussion should be structured and all present should be encouraged to contribute. If a meeting has been arranged over a buffet lunch, informal discussion should be kept relatively brief and the meeting should then be convened formally to ensure proper coverage of key areas. Reviewers are seeking the views of students on the six aspects of provision and should ensure that a clear agenda is outlined. An aide memoire for meeting with student groups is provided in Annex G; this can be adapted to take account of the particular group and the subject provider's aims and objectives. Reviewers will be keen to gauge: students' awareness of the subject provider's aims and objectives; their experiences as learners in the subject; their responses to the teaching provided; and their views on academic and personal support, learning resources, and the arrangements made for student feedback and representation (including whether or not they were involved in the preparation of the self-assessment). Reviewers should allow time for students to raise points.
67. The review team may also take into account the views of recent former students who are able to give either an informed overview of the provision or of the quality of the graduates or diplomates, and employers or other representatives from the relevant industry or profession.
Scrutiny of Documentation
68. Reviewers also gather evidence through scrutiny of documentation. The evidence base drawn on includes reports and advice from external sources such as external examiners, employers and validating and accrediting bodies, as well as internal documents and reports. Emerging judgements are refined and tested against as wide a range of evidence as possible; for example, views expressed in meetings by staff or students will be checked and tested against the documentation provided.
Making the Judgements
General Considerations
69. The reviewers' collective judgements on the quality of education in relation to each aspect of provision are informed by all the evidence gathered and considered by the whole review team. In each case, judgements are based on the principle of fitness for purpose in relation to the stated aims and objectives and the extent to which these are being met. Grades are assigned by the whole review team, on the basis of the evidence gathered in relation to each aspect of provision and using the criteria for assigning grades (Annex L). All team members are expected to share information gathered which is relevant to any aspect of provision, but individual reviewers may co-ordinate the evidence related to particular aspects of provision. Written summaries of the information and evidence gathered should be distributed continuously among the team, including notes of meetings (Annex J), and completed observation and student work/assessment notes (Annexes H and I).
Review Team Meetings
70. The review team meets daily to discuss its findings, normally at the end of the afternoon or in the early evening. Team meetings are used to review the evidence gathered in relation to each aspect of provision, to form preliminary judgements and to determine which issues require further exploration. Reviewers are expected to evaluate how the evidence gathered compares with the self-assessment and the subject aims and objectives, and to test the strength of evidence adduced to support judgements. The review team meeting on the penultimate day should allow sufficient time for a discussion of the evidence gathered in relation to each aspect of provision and a preliminary discussion of gradings. Discussion of the information gathered and emerging judgements should involve the whole team. Therefore, where there are other timetabled evening activities, it may be necessary to hold this meeting in the afternoon.
71. The reviewers hold a final meeting early in the afternoon on the last day to review any additional evidence, to agree the outline of strengths and weaknesses in each aspect of provision, to finalise the grading for each aspect of provision and to determine the overall conclusions to be reported to the institution. The institutional facilitator does not attend this meeting.
Written Summaries
72. The draft summaries written by reviewers during the visit focus on the evaluation of the evidence gathered in relation to each aspect of provision. Summaries should be analytical rather than descriptive and make direct reference to relevant aims and objectives and to sources of information such as meetings, external examiners' reports, documents, reviews of the sample of student work and observations of teaching and learning. The written evaluation should summarise the relevant strengths and issues relating to each aspect of provision and, overall, should underpin the grade assigned.
Grading the Aspects of Provision
73. Reviewers are required to make judgements on, and agree a grade for, each of the six aspects of provision. In assigning grades, reviewers judge both the extent to which the student learning experience and student achievement in each of the aspects of provision contribute to attaining the objectives set by the subject provider, and whether the objectives set and the level of attainment of those objectives allow the aims set by the subject provider to be met. The criteria for assigning grades are included in Annex L.
The Oral Feedback Meeting
74. The meeting with senior representatives of the institution at the end of the review visit is intended to provide oral feedback on the outcomes of the visit. The meeting is chaired by the review chair, who will convey the graded profile, the overall judgement and the main evidence that will be included in the published report. It is not a consultative meeting which might affect the judgements of the reviewers, although a limited amount of clarification by the reviewers and the institutional facilitator may take place. The agenda for the oral feedback meeting is set out in Annex M.
Next > The review reports
Top