Introduction
Background
1 The Quality Assurance Agency for Higher Education’s (QAA) mission is to safeguard the public interest in sound standards of higher education qualifications and to encourage continuous improvement in the management of the quality of higher education. QAA does this by working with higher education providers to define academic standards, quality, and enhancement, and by carrying out reviews of higher education programmes against these standards. The Department for Employment and Learning (DEL) in Northern Ireland has engaged QAA to conduct a review of higher education programmes in each further education college (college) in Northern Ireland, in 2008-09.
2 The review method, known as Developmental review, was devised by QAA for DEL. This draft for consultation provides the opportunity for colleges and their respective awarding bodies to comment on the proposals and to contribute to the design of the method to be published at the end of the 2008 calendar year. By awarding bodies, we mean validating higher education institutions (currently the University of Ulster, Queen’s University Belfast, and the Open University) and other validating bodies (currently Edexcel).
3 The main aims of Developmental review are:
- to provide accountability for publicly-funded higher education in colleges
- to provide information about the maintenance of standards and the quality assurance and enhancement of higher education in colleges in Northern Ireland
- to inform the development of the new review cycle of higher education in further education colleges, to operate from 2009-10 onwards.
4 Developmental review is an evidence-based process lasting 20 weeks carried out by peer review and including two consecutive days of meetings with staff and students at the college, and the scrutiny of college documents both at and outside the college. Developmental review also normally includes a written submission to the review team (the team) by student representatives. Drawing on the QAA Academic Infrastructure as a source of external reference points (see paragraphs 15 and 40 below), Developmental review is designed to assist the colleges in identifying good practice for wider dissemination. Review teams (the teams) will comment in their resulting Developmental review report (the report) on the effectiveness of the college’s processes for managing its responsibilities with regard to the academic standards, quality and enhancement of higher education programmes. These themes are referred to as the three key areas, and are defined as follows.
- ‘Academic standards’ refers to the level of achievement a student has to reach in order to achieve a particular award or qualification.
- ‘Quality of learning opportunities’ considers the effectiveness of everything that is done or provided by the college to ensure that its students have the best possible opportunity to meet the stated outcomes of their programmes and the academic standards of the awards that they are seeking.
- For the purposes of Developmental review, QAA uses the term ‘enhancement’ to mean the continuous improvement of a college's management of the student learning experience of higher education, for the benefit of students, and within the context of its agreements with awarding bodies.
5 The developmental aspects of the review process include the opportunity for the college, in preparing for its review, to test, develop and refine internal quality assurance and enhancement processes. Development is also fostered by the inclusion of members of the college’s own staff as full members (institutional reviewers) of the team carrying out the review (see paragraphs 17 and 18 below).
Purpose, scope and aims
6 Developmental review is concerned with the extent to which the college, working with its awarding body partner(s), exercises its responsibilities in maintaining academic standards, and in assuring and enhancing the quality of students’ learning opportunities in higher education programmes. Developmental review does not consider directly the responsibilities of awarding bodies. The responsibilities of the higher education institution, as an awarding body, are assessed separately through the Institutional audit process.
7 Developmental review will enable each college to demonstrate the effectiveness of the management of its responsibilities with regard to the academic standards, the quality and the enhancement of its higher education programmes, as set out in the partnership agreement(s) with higher education institutions and other awarding bodies and within the context of section two of the Code of practice for the assurance of academic quality and standards in higher education (Code of practice). Areas identified by the team for development will be covered within the report. The Developmental review report is produced by QAA, and made available to the college and its awarding bodies, the Education and Training Inspectorate (ETI) and DEL. As the purpose of reviews is developmental, the reports will not be published. DEL will be able to draw any general lessons or conclusions from the reports produced (see also paragraph 13 below).
The Northern Ireland context
8 In planning, conducting and reporting on the reviews in Northern Ireland, QAA is sensitive to the recent restructuring of the colleges and subsequent changes. In this review method, we use the term former collegeto refer to the whole of one of the constituent colleges, previously an independent college before recent amalgamations. We also use the term former-college siteor siteto denote one location or campus within a former college. The training of reviewers will include information and discussion on the structure and processes of higher education in Northern Ireland colleges and on the context in which it operates.
The Developmental review process in summary
Information and evidence
9 The team will have access to a variety of information sources, including:
- a reflective statement and a portfolio of evidence from the college, outlining its approach to its role in the management of academic standards and the quality of learning opportunities (see paragraphs 25-27 below)
- an optional student written submission submitted by the student representatives of the college
- reports by relevant bodies produced within the three years preceding the Developmental review (see Annex D, paragraph 6)
- a copy of any partnership agreements with awarding bodies relevant to the Developmental review
- information acquired during the review visit. This information will
be from meetings and the scrutiny of further documents provided by the
college beyond the portfolio of evidence. At a minimum, review visit
information is likely to include evidence from:
- meetings with staff and students
- programme handbooks and/or programme specifications
- student support materials
- student handbooks
- records of staff-student liaison committees or equivalent
- assessment criteria or guidance to markers; examination or assessment board minutes
- outcomes of consultations or other engagements with students
- recruitment and student progression data including entry into employment
- details of the college’s employer engagement (particularly important for vocational or Foundation Degree programmes of study)
- staff development documents
- professional, statutory and regulatory body reports
- external examiners’ reports
- college strategy documents
- annual programme monitoring reports.
Key areas and thematic trails
10 Developmental review is conducted through key areas and thematic trails. The team looks at the ways in which the college’s management processes and systems work in practice across three key areas. These key areas are the three themes of the review:
- academic standards
- quality of learning opportunities
- quality enhancement.
11 The three key areas or themes are scrutinised through enquiries conducted along ‘thematic trails’. Thematic trails are a method for collecting evidence and are not an end in themselves. The thematic trails follow practical activities of the college within the three key areas; one thematic trail may well involve elements of scrutiny in some aspects of two, or three, key areas. The team will normally use three thematic trails to gather evidence about college processes.
12 An example of a thematic trail may be how the college manages its student support in higher education. The team may follow examples of the route that students take from their first contact with the college through to gaining an award. Typically, this will involve scrutiny of:
- the accuracy of the information given to prospective students
- how learning support needs are identified
- how the effectiveness of learning support is assured
- support for students through the programme, for example, tutorial and pastoral support
- data on progression through the programme and on to higher studies
- how students’ views are gathered and fed back into programme reviews.
In following such a thematic trail, the team would typically consult college records, minutes of staff/student meetings, minutes of management meetings where students’ views are discussed and actions taken. The team would meet students to gather their views and perceptions. It would take into account such reference points as the Code’s Section 10: Admissions to higher education.
13 The college will identify two of the thematic trails, to be agreed by QAA, in order to demonstrate to the team the methods that the college uses to fulfil its responsibilities in maintaining academic standards and in assuring and enhancing the quality of learning opportunities. The team will follow the thematic trails and consider the effectiveness of the college’s processes. In addition to the two thematic trails proposed by the college and agreed by QAA, DEL and QAA have identified the third thematic trail which will be common to the Developmental review in each college. This third thematic trail is student assessment, and has been chosen because of the challenges that student assessment presents to all education providers, and because of its importance to academic standards. The choice of this trail will enable DEL to consider this aspect of the management of higher education across Northern Ireland colleges, and will help to inform development.
14 A typical example of an assessment trail may be the use that the college makes of external examiners’ reports. An important reference point for this would be the Code of practice, Section 4: External examining.The review team may select a sample of programmes and all documentation and activities relating to the assessment of these programmes, for example:
- the external examiners’ reports
- the dissemination of reports
- actions resulting from any comments made in the external examiners’ reports
- the evaluation of any action taken
- consideration of the external examiners’ reports at programme and/or college-level committees
- any other documentation relating to external examiners’ reports such as annual programme monitoring reports.
The Academic Infrastructure and other external reference points
15 The Academic Infrastructure is a set of reference points for higher education published by QAA. It includes the Code of practice,The framework for Higher Education Qualifications in England, Wales and Northern Ireland (FHEQ), subject benchmark statements, the Foundation degree qualification benchmark and programme specifications. The team will consider how the college has engaged with the Academic Infrastructure and the impact that this has had on its higher education programmes. For more details on the use of external reference points, see paragraph 40 below (see also www.qaa.ac.uk/academicinfrastructure/default.asp)
Students
16 Students have an important role in the Developmental review process. The students’ representative body, normally the Students’ Union or equivalent, has the opportunity to make a written submission to the team in advance of the review. The student written submission, where provided, helps to inform the focus of the review (for more information, see Annex A). The team will scrutinise a range of matters directly relevant to students, including the quality of the information provided for students, and the ways in which their learning is facilitated and supported. Students are invited to take part in meetings with the team during the review visit, and have the opportunity to make the team aware of matters of primary interest to them.
The Developmental review team
17 In Developmental review, the Review Coordinator works with two types of reviewer, the first external to the college being reviewed, the second internal, with both types being full members of the review team. QAA may draw external reviewers, referred to here simply as reviewers, from various sources, including from Northern Ireland colleges. Colleges may endorse the applications of their staff to become reviewers to review in colleges other than their own, and these college-endorsed applications are subject to QAA’s normal reviewer screening and selection processes. In the interests of identifying and disseminating good practice within Northern Ireland, QAA has agreed with DEL to select one reviewer from each college who will be invited to participate as a reviewer in the Developmental review of a Northern Ireland college other than his/her home college. Other reviewers in Developmental review are drawn from QAA’s pool of existing trained reviewers and auditors who have expertise in managing the collaborative provision of higher education in colleges.
18 Each college in Northern Ireland will also be asked to identify, at the rate of one member of staff for each former college offering its higher education, staff members who would participate as internal reviewers, known here as ‘institutional reviewers’, in the review of their own college. In addition to screening and selection, QAA will provide training for all institutional reviewers.
19 Newly established colleges are composed of two or three former colleges. For colleges where higher education is taught at two former colleges, the team will number five, and consist of two reviewers (external to the college) appointed by QAA, two members of the college’s staff, the institutional reviewers, nominated by the college, and a QAA Review Coordinator who manages the review on behalf of QAA and is present throughout the review visit. For colleges with higher education currently taught at three former colleges, the team will number seven, and consist of three reviewers, three institutional reviewers and the Review Coordinator.
20 The reviewers and institutional reviewers are chosen for their relevant experience in the delivery of higher education programmes, including knowledge and understanding of:
- higher education delivered in further education colleges
- quality assurance and management
- teaching, learning and assessment
- academic management
- staff development
- the Northern Ireland context.
For further details on the role of the review team (reviewers, institutional reviewers, Review Coordinator), see Annex B. In identifying its institutional reviewers, each college will wish to consider the pivotal role that the institutional reviewers will play in liaising between the rest of the team and staff, and the extent of their knowledge and experience of the college’s higher education policies and procedures, and sites. The institutional reviewer will need qualities and experience which will assist in handling effectively any sensitive matters which may arise during the course of the Developmental review. Also, the institutional reviewers should be chosen according to their management responsibilities such that once the review is completed they may take a significant role in implementing any changes based on the outcomes of the Developmental review.
21 QAA sends brief details of the proposed reviewers to the college before the review begins. Colleges should discuss with QAA any concerns about the suitability of the proposed team as soon as possible. These concerns may include matters of which QAA is not aware; therefore it is essential that any concerns, for example, about a conflict of interest or the team's match to the college to be reviewed, are made in writing and discussed with QAA no later than two weeks from notification. Delay in the above may result in undue delay in further review planning.Conclusion of the review and reports
22 The Review Coordinator, accompanied by the team, presents an oral summary of the conclusions and recommendations to representatives of the college at the end of the visit. Colleges are expected to invite their respective awarding bodies to the meeting. The main outcome of each Developmental review is a written report which identifies good practice and includes recommendations for action. The written report is not published but will be provided to the college and its awarding bodies, DEL and the ETI. The college is asked to produce an action plan for inclusion in the final report, in collaboration with its awarding body/bodies, approximately two months after the review, which outlines how it intends to address the recommendations in the report.
How the process works
Initial planning and contact with QAA
23 At the start of the round of Developmental review activity, QAA will contact the college about the proposed scheduling of the review and of the review visit component. QAA will then provide abriefing sessionfor the college and its awarding bodies to prepare them for the review. The ETI will also be invited to attend the briefing session. The briefing session will include discussion of the reflective statement and evidence portfolio that colleges are asked to supply a few weeks in advance of the review’s start. A general timeline for the reviews is provided in Annex C.
24 Normally, a little after the start of the review period, and approximately six weeks before the review visit, the Review Coordinator will meet college staff and students in aplanning meetingto agree arrangements for the review visit. This planning meeting will include discussion of the programme of activities and the proposed thematic trails. The discussions at the planning meeting will help the college to identify any additional documents that the team is likely to need to see as evidence during the review visit.
Reflective statement and documentation
25 Central to the process of Developmental review are the self-evaluative processes that demonstrate the college’s effectiveness in the management of academic standards, and the quality and enhancement of learning opportunities. To facilitate the review, each college is asked to provide a reflective statement and a portfolio of existing documentary evidence that addresses the thematic trails and each of the three key areas: academic standards; quality of learning opportunities; and enhancement. The college should seek to demonstrate its capacity for reflection and self-evaluation. Further information is provided in Annex D.
26 The college should send QAA the reflective statement, referenced to and accompanied by a portfolio of evidence, 10 weeks before the start of the visit. Documentation should be submitted in an electronic format with one hard copy. Colleges are asked to use as electronic format CD-ROM or data stick, with the reflective statement as a Word file.
27 QAA will acknowledge receipt of the reflective statement and portfolio. A copy will be sent to the Review Coordinator asking for confirmation of whether the reflective statement and portfolio of evidence form an appropriate basis for the review. Once this has been agreed, QAA will notify the college and ask for copies of the reflective statement and portfolio of evidence to be sent to the team. Colleges are asked to avoid sending large files and large numbers of files by email: some email systems restrict the number and quantity of attachments. A location map, joining details and any other relevant information should also be sent to team members no later than four weeks before the review visit.
The review visit
28 The review visit will take place over two consecutive days, during which normally all former colleges delivering higher education will be visited. Through the Review Coordinator, the detailed programme for the review visit will be agreed between the college and the review team at the time of the planning meeting.
29 The review team will visit one site of each former college at which higher education courses are delivered in order to gain evidence to inform their overall evaluation of the college. The team will divide so that at least two team members (usually one reviewer and one institutional reviewer) visit each former college, and then join the rest of the team at the close of the day to share findings.
30 The review visit will provide the team with opportunities to read further documentation and for discussions with staff and students. Within this further documentation, a range of students’ assessed work and examiners’ reports will be needed in order to assist the team to follow the thematic trail on student assessment. Student work should not be sent as part of the portfolio but be supplied only at the visit as part of the further documentation. Student work supplied to the team should have passed through all stages of marking, feedback to students, and moderation. Documentary evidence is important in helping the review team to evaluate the college’s management of the quality and enhancement of learning opportunities and its role in the maintenance of academic standards. The only material in the portfolio and further documentation required by the review team will be that which is already held by the college or its awarding bodies. The only freshly authored document required in Developmental review will normally be the reflective statement which introduces the portfolio.
31 The team will also usually carry out direct observation of some elements of provision as part of an identified thematic trail, for example, learning resources. Evidence of teaching quality will usually be obtained from a documentary study of procedures such as the college’s observation of higher education teaching, meetings with students, and the analysis of student questionnaires and other arrangements for gathering feedback. There will not be any direct observation of teaching.
32 The review visit will conclude with a brief oral summary of conclusions and recommendations; this will be made to the college by the Review Coordinator. Further information on the review visit including an indicative visit schedule and guidance on the conduct of meetings is provided in Annex Eand Annex F.
The selection of thematic trails within the key areas
33 The key areas are described in more detail below, and examples of thematic trails are included under each as an illustration.
Key area one: Academic standards
34 Does the college discharge effectively its responsibilities, as set out in its partnership agreement(s), for the management and delivery of the standards of the award(s) for the higher education that it offers on behalf of its awarding bodies?
35 Thematic trails may include, but are not limited to:
- assessment of students (DEL and QAA agreed mandatory trail)
- external examiners and their reports
- student progression, achievement and completion statistics
- use of the Academic Infrastructure
- monitoring of the academic standards of existing programmes
- college processes for the internal approval of a new programmes and participation in the validation arrangements of the awarding body;
- the college’s processes in the exercise of its responsibilities in the periodic review of the academic standards of programmes
Key area two: Quality of learning opportunities
36 Does the college discharge effectively its responsibilities, within the context of the Code of practice and as set out in its partnership agreements with higher education institutions and other awarding bodies, for the management and delivery of the quality of the learning opportunities in higher education?
37 Thematic trails may include, but are not limited to:
- academic guidance, support and supervision, and pastoral support and guidance
- student representation on programme committees and arrangements for feedback from students
- the accuracy, completeness and reliability of the information that the college provides for students about the quality of the programmes and the standards of the awards that it offers on behalf an awarding body
- learning support resources
- staffing and staff development, including teaching, scholarly activity, and professional practice and updating.
Key area three: Enhancement
38 Does the college take systematic steps to improve continuously the college's management of its higher education, for the benefit of students, and within the context of its agreements with awarding bodies?
39 Thematic trails may include, but are not limited to:
- the college’s strategy for existing higher education programmes
- the frameworks for managing its role in the enhancement of the quality of its higher education programmes
- the responsiveness to internal and external monitoring, review and quality assurance processes
- processes linked to the assessment of students (DEL and QAA agreed trail for each college).
Use of external reference points
40 When considering how each college fulfils its responsibilities within the management of academic standards and the assurance and enhancement of the quality of learning opportunities, the Developmental review team draws upon a range of external reference points contained within the Academic Infrastructure namely the Code of practice, the FHEQ, subject benchmark statements, the Foundation degree qualification benchmark and programme specifications. In so doing, the team is not normally seeking evidence of compliance, but, rather, seeking evidence that the college has considered the purpose of the reference points, has reflected on its own practices, and has taken, or is taking, steps to ensure that appropriate changes are being introduced. Colleges may familiarise themselves with all components of the Academic Infrastructure by consulting the relevant section of QAA’s website at www.qaa.ac.uk/academicinfrastructure/default.asp
- In respect of the FHEQ, the team considers the college's procedures for relating the awards that it delivers to the appropriate level of the framework for higher education qualifications, where relevant using the thematic trails to gain further insight.
- In respect of subject benchmark statements, the team enquires into the way in which the statements have been taken into account in the college’s establishment and review of programmes and awards, as illustrated through programme specifications. QAA views subject benchmark statements as authoritative reference points, but not as definitive regulatory criteria for individual programmes or awards.
- In respect of the Foundation Degree qualification benchmark, the team will consider how the college engages with the defining characteristics of Foundation Degrees. Foundation Degrees involve the integration of the following characteristics: employer involvement; accessibility; articulation and progression; flexibility; and partnership. While none of these variables is unique to Foundation Degrees, their clear and planned integration within a single award, underpinned by work-based learning, makes the award very distinctive.
- In respect of programme specifications, the team will consider how the curriculum, teaching and learning, and assessment methods relate to the intended learning outcomes. Programme specifications show how modules combine to make coherent programmes. However, the learning outcomes of a programme specification are not simply an aggregation of module outcomes; they should express aims and objectives for the programme as a whole.
- In respect of the Code of practice, the team does not seek evidence of compliance with every precept. It expects to see a statement in the reflective statement about how the college has addressed the intentions of the Code’sprecepts, and to discuss during the review visit any key changes that the college has made to its practices and any areas that have caused difficulty. It will use the thematic trails to gain further insights.
Reports and the college’s response
41 Immediately following the review visit, the Review Coordinator offers an oral summary of findings to the college and its awarding bodies. This will be confirmed in a letter to the college director. The main outcome of the review is a written report. A template of the Developmental review report is provided as Annex G.
42 The report sets out analysis, conclusions, good practice and recommendations for improvement for the college. Within the context of the Code of practice, the conclusions will express the team’s view of the extent to which the college discharges its responsibilities as set out in the partnership agreement for the management of the standards of the higher education awards, and the assurance and enhancement of the quality of learning opportunities.
43 The team will consider features of good practice. Developmental review reports will identify the features of good practicethat the team considers, in the context of the college, make a particularly positive contribution to the college's approach to the management of academic standards and to the assurance and enhancement of the quality of learning opportunities.
44 Recommendations made in the report will be at one of three levels.
- Essential recommendations refer to important matters which the team believes are currently putting quality and/or standards at risk and which require urgent corrective action (see also paragraph 47 and 48 below).
- Advisable recommendations refer to matters which the team believes have the potential to put quality and/or standards at risk, and require preventative corrective action.
- Desirable recommendations refer to matters which the team believes have the potential to enhance quality, build capacity and/or further secure standards.
45 The Review Coordinator will send copies of the draft of the Developmental review report to the college and its awarding bodies for comments about four weeks after the conclusion of the review visit. This gives the college and its awarding bodies the opportunity to draw the team's attention to any areas which they regard as inaccurate or incomplete. The college and its awarding bodies should make a single, consolidated and agreed reply from the college to the team through the Review Coordinator. The team is then responsible for considering these comments and any supporting evidence which the college may offer at this stage, and for agreeing whether or not any aspect of the report should be amended in response. Teams will only be able to consider supporting evidence that was available at the time of the review visit.
46 The revised draft report will be returned to the college. At this stage, the college is also asked to complete, on a template provided by QAA, an action plan in response to the conclusions of the report (see Annex H). The action plan is intended to support the college in the continuing development of its higher education programmes by facilitating its expression of how it intends to address the findings of the Developmental review. The college may submit its action plan in collaboration with its awarding bodies. Whether produced singly or collaboratively, the action plan should set out how the college plans to disseminate or sustain good practice, and how it plans to address the team’s recommendations.
47 QAA will incorporate the action plan into the Developmental review report. Once the Developmental review team has considered and responded to the college's comments, a final version of the report will be sent to the college and its awarding bodies, to the ETI and to DEL. This completed report containing the action plan will form part of the evidence base for future review activity of higher education at the college. In all cases, the action plan, its implementation and impact will inform future review activity.48 Should a college Developmental review result in one or more essential recommendations, DEL may ask QAA to ask the college to offer a progress report on how it has addressed the recommendations, and to provide evidence that the action has been effective. This progress report will normally be required within six months of the Developmental review visit.
49 Exceptionally, essential recommendations may lead to DEL commissioning a second Developmental review at the college, because the review team has recorded evidence that quality and/or standards are at risk. QAA will also monitor the implementation of the action plans of the Developmental reviews through the subsequent review cycle.
Evaluation of the process
50 QAA has committed itself to monitoring and evaluating its work, in an open and reflective manner. It does this within the context of an Evaluation Policy. The Policy is implemented through such procedures as:
- questionnaire surveys of colleges, review coordinators, students, reviewers, and institutional reviewers. QAA will encourage colleges, and their staff who have acted as reviewers or institutional reviewers to contribute to the evaluation of the Developmental review process by inviting comment on the reviews in which they have participated
- focus groups. For example, with representatives from different constituents of the higher education sector, and other stakeholders, to explore particular issues in greater detail.
51 The full text QAA’s Evaluation Policy is available on its website at www.qaa.ac.uk/aboutus/policy/evaluationPolicy.asp
Complaints
52 QAA will endeavour to conduct reviews in a way which is effective and satisfactory for each college. Should a college have a need to complain about the conduct of a Developmental review, the complaint will be considered by QAA under the formal procedures published on its website at www.qaa.ac.uk/aboutus/policy/complaints.asp
53 QAA is committed to working in an open and accountable way. This includes
responding positively to complaints from institutions by investigating
them thoroughly, putting mistakes right where we can, and learning the
lessons.
Annex A: Student involvement in Developmental review
1 QAA greatly values the involvement of students and the role that they may play in Developmental review. There are several ways in which students can contribute:
- by producing or contributing to a student written submission for the reviewers and institutional reviewers
- by coming along to a preparatory meeting between the college and the Review Coordinator
- by talking to the reviewers and institutional reviewers when they visit.
2 Perhaps the most important thing to say about the involvement of students is that it ought, as far as possible, to reflect the views of the majority of the higher education students rather than just a minority.
3 Student involvement in approval and review processes may provide evidence to enable the team to evaluate the level of influence exercised by students, as well as their contribution of students to self-evaluative processes. It is usual for colleges to seek the views of students during their normal processes of annual monitoring, programme approval and programme review.
4 The team will normally expect to see examples from colleges of how students’ views are sought, for instance, students’ representation on committees and their role and effectiveness within them. Evidence of the effectiveness of student-support mechanisms will be sought, as will evidence of where the views of students has influenced change.
5 The effectiveness of the colleges’ processes for securing the wider views of the higher education students will be analysed. During meetings with students, the team will seek students’ views of their input and impact on the college’s management of academic standards and quality.
Student written submission
6 QAA invites a representative body of the students to make a written submission of their collective views on their experience as learners and the opportunities that they have to participate in the management and quality of standards. To help student representative bodies to prepare a written submission for Developmental reviews, QAA has written separate briefing notes. The notes provide contact details for the QAA officers managing the review process, who will answer students’ queries about Developmental review.
7 QAA asks that the written submission does not include any information that student representatives would not be happy to discuss with the reviewers. While QAA will respect any wishes for confidentiality, QAA encourages the student body to share its submission with the college.
Guidance on the format, length and content of the student written submission
8 The written submission should:
- include a statement of how it has been compiled, who wrote it, and the extent to which its contents have been shared with, and endorsed by, higher education students
- aim for a maximum of 1000 words
- refer to sources of evidence that would aid the team, for example, minutes of meetings, copies of correspondence between the students and the college, or reference to questionnaires
- not identify any individual either directly or indirectly.
Student meetings
9 The students’ experience of their education is fundamental to the college and the awarding bodies, and informs the entire process of Developmental review. Meetings with students enable the team to establish student views on the key areas and/or thematic trails being considered, and inform the team on the quality of the student learning experience.
10 The team holds meetings with representative groups of current students. These meetings provide an opportunity to hear directly the views of those present and to establish more generally whether there are effective arrangements for student feedback and representation. The meetings are held in confidence and comments made at the meeting are not attributed to any individual.
Annex B: Role of the review team
1 The role of the review team at the various stages of Developmental review is as follows:
2 Preliminary (after receipt of college’s documentation)
- analyse the reflective statement and portfolio of evidence and identify possible areas for enquiry within the thematic trails
- communicate with the Review Coordinator on matters arising from this analysis
- post material to QAA’s Academic Reviewer Communication Service (ARCS), the secure electronic folder dedicated to the review.
3 Developmental review visit
- agree the main areas of enquiry for exploration during the visit, and select and study evidence to support that exploration
- participate in discussions with staff and students of the college on areas of enquiry as agreed by the team as a whole, with due attention to the protocols of discussion as a peer group
- draw conclusions from the areas of enquiry being explored during the visit, with reference to oral and documentary evidence to underpin those conclusions
- contribute to the formation of evaluation on the three key areas and the college’s use of the Academic Infrastructure
- keep sufficient notes to be able to respond subsequently to the college’s comments on matters of factual accuracy during the report production phase
- keep sufficient notes to be able to respond subsequently to requests from the college for clarity on the insights gained from the Developmental review
- post material to the ARCS folder.
4 After the visit
- prepare a section of the report for the Developmental review report, and submit this to the Review Coordinator within a specified deadline
- comment on the draft report prepared by the Review Coordinator, responding to any requests for extra text or reference to evidence to agreed deadlines
- post material to the ARCS folder.
Developmental review team role descriptions and person specifications
Developmental review Coordinator
Role
5 The Review Coordinator manages Developmental review in the college
6 Key responsibilities include:
- leading reviews for QAA
- providing clear briefings to a wide range of college participants on the Developmental review method and participants' respective responsibilities
- discussing and agreeing with the college the thematic trails and review agenda that form the basis of the review
- discussing and agreeing focused review activities with the college and the review team to ensure effective use of time
- organising and coordinating review activities to ensure that conclusions and recommendations are sound and evidence-based
- liaising effectively with the colleges and teams through face-to-face, telephone, email and written communications to ensure the smooth running of each review
- providing additional training for reviewers and institutional reviewers, if necessary
- making effective use of ARCS, QAA's secure electronic folder system, throughout the review to ensure that a full evidence base is available to the team and QAA staff in a timely manner and is archived promptly
- respecting protocols on confidentiality
- producing high-quality reports that usefully inform all stakeholders of conclusions and recommendations where appropriate.
Reviewer
Role
7 The reviewers contribute to evaluating academic standards and the quality of higher education provision through a peer-review process. They engage in a variety of activities designed to gather and analyse evidence so that they can arrive at considered conclusions, recommendations and judgements. These outcomes help the college being reviewed to prepare an action plan to further enhance higher education provision.
8 Key responsibilities include:
- reading, analysing and preparing written commentaries of the reflective statement submitted by the college and the portfolio of evidence sent in advance of a review
- adhering to the review schedule agreed between the college and the Review Coordinator
- participating in visits to the college in order to gather, share, test and verify evidence
- drawing conclusions, making recommendations on the academic standards achieved and the quality of the learning opportunities provided
- recording evidence gathered from a variety of review activities and submitting this to the ARCS secure folder in a timely fashion
- drafting sections of the report that are referenced to evidence gathered during the review
- respecting protocols on confidentiality
- contributing to and commenting on the compilation of the report of the review to agreed schedules and deadlines
- being available for the whole period of a review for which the reviewer has been selected and committing to complete all processes of a review embarked upon.
Institutional reviewer (a member of staff of the college reviewed)
Role
9 The role provides an opportunity for members of a college's staff to consider the quality and standards of higher education across the college and to be part of the team testing the effectiveness of the college's reflective processes. Where colleges are considering the revision of internal review processes, participation in a Developmental review team as an institutional reviewer may contribute to ways in which processes might be revised.
10 Key responsibilities include:
- reading, analysing and preparing written commentaries of the reflective statement submitted by the college and the portfolio of evidence sent in advance of a review
- adhering to the review schedule agreed between the college and the Review Coordinator
- participating in the review of the college in order to gather, share, test and verify evidence
- drawing conclusions and making recommendations on the academic standards achieved and the quality of the learning opportunities provided
- recording evidence gathered from a variety of review activities and submitting this to the QAA electronic review folder in a timely fashion
- drafting sections of the report that are referenced to sound evidence gathered during the review
- respecting protocols on confidentiality
- contributing to and commenting on the compilation of the report of the review to agreed schedules and deadlines
- helping the college to draw up its action plan for implementation of the team's recommendations
- playing a lead role in the implementation of the action plan within the college following the review
- being available for the whole period of a review for which the institutional
reviewer has been selected and committing to complete all processes of
a review once embarked upon.
Annex C: Timeline for the Developmental review process 2008-09
This schedule of activity is set out in relation to planning activities that began earlier in 2008.
Month 3 - 4 |
QAA consults on the draft review handbook |
Month 4 - 5 |
QAA’s Information Unit analyses responses to consultation. QAA revises draft handbook in response to the consultation and sends it to DEL for approval prior to publication |
Month 6 |
QAA holds briefing events in Northern Ireland for colleges and other stakeholders |
Month 7 |
QAA holds training events for review teams in Northern Ireland |
Month 7 – 8 |
The colleges send to QAA reflective statements and portfolios of evidence to support the review |
Month 8 – 9 |
QAA holds a planning meeting between the Review Coordinator, QAA officer and the college. |
Month 10 – 11 |
The Developmental review team works at the college for two days. This involves meetings, for example, with staff, students and employers, scrutiny of further documentary evidence, (including a sample of students’ work), and an oral summary to the college |
Month 11-12 May-June 2009 |
QAA sends draft written report to the college and its awarding bodies for comments. This gives the college and its awarding bodies the opportunity to draw the team's attention to any areas which they regard as inaccurate or incomplete. |
Month 12-13 June-July 2009 |
Review Coordinator amends report as appropriate and returns it to college, and requests that the college complete the action plan template. QAA incorporates the completed action plan into the report before distribution to the college, awarding bodies, DEL and ETI. |
This timeline is indicative. Individual events may be varied to accommodate
specific circumstances such as Christmas, vacations or examinations. However,
QAA will endeavour to complete the distribution of the review report within
10 weeks of the visit.
Annex D: Guidance on reflective statement and portfolio of evidence
1 The reflective statement is the starting point for the review and is centred on the three key areas. The task of the review team is to test and verify the claims made and the evidence cited by the college. These guidelines are intended to help the provider to prepare its reflective statement.
2 The reflective statement is likely to be no more than 10 sides of A4. It will be an evaluative account on the college’s approach to the management of academic standards and of the assurance and and enhancement of the quality for its higher education programmes, and how it satisfies itself about the effectiveness of its approach. The statement will be supported by a portfolio of existing documents to act as evidence to support the statement made. The portfolio should be cross-referenced to the reflective statement.
3 The central requirement for the reflective statement is that it contains reflective and evaluative sections, and that evidence for any claims or assertions made is provided within the portfolio of documents or, if not within the portfolio, is referenced and made available to the team.
4 The colleges are asked to provide in the portfolio documentation which is well-organised and evaluative, and that draws upon robust internal and external review and validation procedures. The supply of documentation which contains an accurate and comprehensive evaluative explanation is likely to limit the later burden on the college of further document supply.
5 The role of the institutional reviewers is critical for gathering information prior to and during the review. The institutional reviewers will be familiar with information and also be able to guide the team to information held by the college and which, for some reason, may not be available within the portfolio. For example, the institutional reviewer could guide the team through areas of the college’s intranet, or would be in a position to assemble small groups of relevant students or staff, should the team be following a particular theme.
6 The portfolio should comprise at least the following documents to support the thematic trails within the three key areas:
- partnership agreement(s) and/or samples of any reviews of these agreement(s) with the awarding bodies
- programme specifications
- sample of external examiners’ reports (previous three academic years)
- sample of validation documents (latest)
- sample of annual monitoring reports (previous three academic years)
- sample of evidence relating to student course evaluations
- extracts from quality assurance/enhancement handbook/manual
- other evidence that the college wishes to provide in support of the agreed thematic trails.
7 QAA will provide the team with:
- ETI reports (to provide contextual information)
- publicly available information
- relevant QAA reports
- a briefing paper from QAA’s Information unit.
Annex E: Indicative visit schedule
There are opportunities to tailor the programme to the college, its higher education provision and the number of former-college sites to be visited. This example is not intended to be prescriptive, but offers an approach to organising the activities.
Day 1 |
||
0830-0945 |
First (private) meetingof the team to confirm areas of enquiry. |
|
0945-1100 |
Scrutiny by team of selected supporting evidence. |
|
1100-1130 |
Team’s initial meeting with the collegeto summarise findings and confirm agenda for further meetings with staff and students. |
|
1130-1230 |
Team members travel to other former-college site(s) where higher education is delivered: one for each former college |
|
1300-1330 |
Lunch |
|
1330-1430 |
Meeting between team and key staff of the college. |
|
1445-1545 |
Meeting of the team with current students. |
|
1545-1700 |
Regrouping of the team for collective scrutiny of supporting evidence. |
|
1700-1800 |
Team meeting to confirm the record of the activities completed, the range of evidence, the tentative evaluations and the matters that remain outstanding to be completed on day 2. |
|
Day 2 |
||
0830-1000 |
Meeting of the team and college contacts to confirm the emerging evidence, the agenda for day two and the interim evaluations. |
|
1000-1100 |
Meetings with employers/awarding bodies, if appropriate. |
|
1100-1230 |
Further scrutiny of selected evidence and/or any additional meetings that require to be arranged during the visit. |
|
1230-1300 |
Lunch |
|
1300-1530 |
Final team meeting to reach conclusions on the college’s role in maintaining academic standards and the assurance and enhancement of the quality of learning opportunities, in relation to the thematic trails; to identify significant strengths; to identify any recommendations; and to finalise the key points for the oral summary. |
|
1530-1600 |
Oral summary to college staff presented by the Review Coordinator, accompanied by the review team. awarding bodies are invited to this meeting. |
|
Annex F: Conduct of meetings
Guidance on the conduct of meetings
1 During the reviews, colleges, their awarding bodies and members of the team may wish to consider the following illustrative approaches to the key meetings held. In applying the guidance to local circumstances, all participants are encouraged to bear in mind:
- the purpose and intended outcome of the review
- the importance that QAA attaches to open dialogue between peers
- the need for the team to reach conclusions and to prepare a report.
2 Team members should not normally attempt a comprehensive and exhaustive scrutiny, but use the reflective statement and documentary evidence to inform their scrutiny of the thematic trail(s) and use the meetings to confirm good practice or areas for recommendations; clarify any matters arising from the evidence, and seek examples of policies and procedures in practice.
Illustrative approach applicable to the range of meetings
3 In preparing for and conducting meetings, the team may wish to identify the areas of enquiry for discussion using the following prompts.
- What are the salient features of the reflective statement and portfolio of evidence that inform the discussion?
- In particular, what can be readily confirmed as verified evaluation and information?
- What can be clarified by reference to existing additional documentary evidence outside the meeting?
- What points arise that deserve a focus within the meeting?
Suggested protocol and guidance for meetings with students and staff
4 In optimising the value of the discussions in meetings, it is suggested that for each meeting the following arrangements are made:
- an agenda is shared with all attending
- the Review Coordinator or another member of the team chairs the meeting and takes a record of the meeting and shares it with those attending
- the Review Coordinator will chair the meeting and briefly introduce the team and other participants and summarise the purpose of the meeting, the opportunity it provides for the exchange of views within the chosen focus and remind the meeting of this protocol for meetings
- the dialogue with students will normally start with a question to establish the basis on which the students were selected to attend the meeting. If meeting former students, the team will wish to confirm the year in which the former students completed their studies
- at any meeting with former students, the reviewers' agenda will focus particularly on the relevance and usefulness of the knowledge and skills delivered by the college in relation to their careers. The reviewers will wish to learn the nature of any continuing relationship former students have with the college. Throughout the meeting, students will be given opportunities to raise points not covered by the reviewers' agenda
- all attending should feel encouraged to put forward their views in the confident knowledge that all perspectives and views are respected in an open discussion. It is important that the participants are assured that no comments are attributed to any one person
- the timing of the meeting should be controlled within the agreed agenda and timetable. In exceptional circumstances, if any of the participants or staff believe that additional attention and time are required to address adequately the agreed agenda or other matters arising, the chair may, with the agreement of the meeting, either extend the meeting for a short time (for example, 15 minutes) or take steps to arrange an additional meeting at a later stage in the visit programme
- in closing the meeting, the chair will summarise the key points addressed,
thank the participants and close the meeting.
Annex G: Indicative report structure
The indicative report structure is set out below.
Preface
Standard QAA text
Purpose of Developmental review
Standard QAA text here
Evidence
Standard QAA text here
Academic Infrastructure
Standard QAA text here
Developmental review
[Name of participating college]
A Conduct of the Developmental review
This report presents the findings of the Developmental review conducted at [participating college]. The Developmental review was carried out by [Developmental review team members] and coordinating the review on behalf of QAA was [Insert name, Review Coordinator].
The Developmental review was conducted by the team in negotiation with the college and followed the method set out in Thehandbook for Developmental review of directly funded higher education in further education colleges (Northern Ireland) 2008-09,published by QAA.
The team included two/three institutional reviewers drawn from the college’s staff, who was a full member of the team.Evidence in support of the Developmental review included [for example, documentation supplied by the college, meetings with staff, students, employers and partner colleges, and from inspections by the ETI] .The Developmental review also considered the college’s use of the Academic Infrastructure, developed by QAA on behalf of the higher education sector.
Students’ written submission (SWS)
Students undertaking higher education programmes at the college were invited to present a written submission to the team [outline whether SWS was submitted, any support or guidance provided by the college to the student representatives in the event]. No attempt is made to evaluate the students’ written submission.
Context of the Developmental review
[One or two paragraphs detailing facts about the college, including its history, mission, total student numbers, higher education student numbers and any relevant information on structure and/or relocation]
Higher education at the college
[This section outlines the range of programmes offered, in bullet list form against the respective awarding bodies]
B Outcomes of the Developmental review
[Para number] The following addresses key area one.
Thematic trails [Their theme title or descriptor]
The following addresses key area two.
Thematic trails [Their theme title or descriptor]
The following addresses key area three.
Thematic trails [Their theme title or descriptor]
C Conclusions
[Para number] Based upon discussion with staff and students and scrutiny of evidence provided by the college, the team identified a number of features of good practice in the college’s discharge of its responsibilities for the assurance and enhancement of learning opportunities and for the academic standards of the awards offered on behalf of its awarding bodies. The team also makes some recommendations for consideration by the college. The areas of good practice and recommendations are set out below.
[Para number] During the Developmental review, the team identified the following areas of good practice:
- [Referenced bullet list of findings reflecting the broader main narrative addressing the thematic trails]
[Para number] The team also agreed the following areas where it would be essential for the college to take action:
- [Referenced bullet list reflecting broader findings as detailed in the narrative]
[Para number] The team agreed upon a number of areas where the college is advised to take action:
- [Referenced bullet list reflecting the broader findings as detailed in the narrative]
[And/or…]
[Para number] The team also considers that it is desirablefor the college to:
- [Referenced bullet list reflecting broader findings as detailed in the narrative]
[Para number] The team agreed the following conclusions in respect of the three key areas:
[Para number] Based upon its analysis of the college’s reflective statement and portfolio of evidence, other documentary evidence and from its meetings with staff and students, the team concludes that, in the context of this Developmental review, the college [discharges/does not discharge]its responsibilities effectively, as set out in the partnership agreement, for the management of the standards of the awards for the higher education that it offers on behalf of its awarding bodies.
[Para number] Based upon its analysis of the college’s reflective statement and portfolio of evidence, other documentary evidence and from its meetings with staff and students, the team concludes that, in the context of this Developmental review, the college [discharges/does not discharge]its responsibilities effectively, as set out in the partnership agreement, for the management of the assurance of the quality of learning opportunities for higher education to enable students to achieve the intended learning outcomes.
[Para number] Based upon its analysis of the college’s reflective statement and portfolio of evidence, other documentary evidence and from its meetings with staff and students, the team concludes that, in the context of this Developmental review, the college [discharge/does not discharge]its responsibilities, as set out in its partnership agreement for the enhancement of the quality of learning opportunities of its higher education programmes.Annex H: Action plan templates
Name of College ……………………………………….
Action plan relating to Developmental review: Date |
||||||
Good practice |
Action to be taken |
Target date |
Action by |
Success indicators |
Reported to |
How evaluated |
In the course of the Developmental review the review team identified the following areas of good practice that are worthy of wider dissemination within the college: |
||||||
Name of College ……………………………………….
Action plan relating to Developmental review: Date |
||||||
Essential |
Action to be taken |
Target date |
Action by |
Success indicators |
Reported to |
How evaluated |
The review team agreed the following areas where it would be essential to take action: |
||||||
Name of College ……………………………………….
Action plan relating to Developmental review: Date |
||||||
Advisable |
Action to be taken |
Target date |
Action by |
Success indicators |
Reported to |
How evaluated |
The review team agreed the following areas where it is advisable to take action: |
||||||
Name of College ……………………………………….
Action plan relating to Developmental review: Date |
||||||
Desirable |
Action to be taken |
Target date |
Action by |
Success indicators |
Reported to |
How evaluated |
The review team agreed the following areas where it is desirable to take action: |
||||||
Annex I: Activity flowchart
Top