Learning from developmental engagements

PDF version

Contents

• Executive summary
• The purpose of developmental engagements
• The programme of developmental engagements
• Report methodology
• Part A: Evaluation of the developmental engagement methodology
• Part B: Learning from developmental engagements
• Conclusions
• Appendix 1: A list of institutions that experienced developmental engagements
• Appendix 2: A list of subject disciplines covered by developmental engagements
• Appendix 3: Developmental engagements: a list of reviewers
• References
• Glossary

---

Executive summary

The purpose of this report is to provide an overview of the findings arising from developmental engagement (DE) reports and the Quality Assurance Agency for Higher Education (QAA) evaluation of the DE methodology. Part A provides a summary of QAA's findings based on its evaluation of the methodology, including an assessment of the developmental nature of DEs. Part B outlines the findings contained in reports, that is, what has been learned about higher education institutions (HEIs) from DEs. The conclusions draw on both Part A and Part B.

• DEs formed part of the operational arrangements for the transitional period, 2002 to 2005, for England and Northern Ireland. A total of 174 DEs were completed between January 2003 and October 2004, involving 82 HEIs and 340 reviewers and covering 15 subject areas.
• In all DEs, teams returned confidence judgements in both academic standards and the quality of learning opportunities. This is not to suggest that teams found everything to be positive, but that, on balance, any concerns identified were not of a sufficient magnitude to threaten standards and quality overall.
• DE reports provided clear evidence of the strength of HEIs' internal review procedures at the level of the discipline and the robustness of the evidence used in the internal reviews. Good practice included use made of statistical information, external examiners' reports, student feedback and staff development activities. However, in a small number of DEs, teams identified scope for improvement in these areas as well as the need to increase students' involvement in quality assurance processes.
• There was extensive evidence that HEIs had approached the Academic Infrastructure in a positive and constructive manner.
• In general, HEIs were reported to have made good progress in their preparation for publishing information on quality and standards of teaching and learning in line with HEFCE's document, Information on quality and standards in higher education, final guidance (HEFCE 03/51).
• In many reports, DE teams concluded that the self-evaluation document (SED) provided was insufficiently evaluative and, in some cases, did not reflect the evaluative ethos of the subject provider.
• QAA's evaluation of DEs showed that the role of the institutional nominee (IN), the composition of teams (with a number of types of reviewers) and the use of students' written submissions (SWS) were all positive features.

Number of judgements January to June 2003 and October 2003 to October 2004

	January	October 2003
	- June 2003	- October 2004
Confidence judgement: Academic standards	112	62
Confidence judgement: Quality of learning opportunities	112	62

The purpose of developmental engagements

1 In 2002, at the request of the Government, the Higher Education Funding Council for England (HEFCE) together with Universities UK (UUK) and the Standing Conference of Principals (SCOP) agreed the operational arrangements for the transitional period, 2002 to 2005 for England. At the heart of the arrangements was the belief that HEIs were best placed to manage and enhance the standard of their awards and the quality of student learning opportunities. The role prescribed for QAA was to confirm that HEIs were fulfilling their responsibilities under these arrangements.

2 The main operational feature of the transitional period was that each HEI should undergo an institutional audit during the period 2002 to 2005 (see the Handbook for institutional audit: England for details). For those HEIs that were awaiting their audit in the second and third year of the transitional period (2003-04 and 2004-05) HEFCE required that they should have some interaction with QAA at the level of the discipline, taking the form of either academic reviews at the subject level, or DEs. The criteria for an HEI undergoing a DE as opposed to an academic review can be found in the Arrangements during the transitional period 2002-2005 for Higher Education Institutions in England. The arrangements for HEIs in Northern Ireland followed this pattern, following discussions with the Department for Employment and Learning.

3 DEs had two main purposes. To provide 'an opportunity for institutions to test, in cooperation with the Agency [QAA], the strength of their internal review procedures at the level of the discipline or programme, and the robustness of the evidence they use in those procedures'. It was intended that this would assist HEIs in their preparations for institutional audit and, specifically, discipline audit trails (DATs). With this latter aspect in mind, the DE methodology closely adhered to the principles and methods of DATs set out in the Handbook for institutional audit: England.

4 The second purpose was to provide evidence of the standards and quality at the level of the subject in the form of two threshold judgements. The first judgement focused on the standards set for, and achieved by, students. The second judgement related to the quality of learning opportunities provided for those students. It was also intended that DEs would report, where appropriate, on matters for further consideration by the HEI with regard to their procedures for managing and enhancing standards and quality.

The programme of developmental engagements

5 The programme of DEs started in January 2003. There was a total of 174 DEs over the next 21 months. Between January and June 2003 there were 112 visits, and 60 visits were conducted in the academic year 2003-04. With the agreement of HEFCE, two DE visits were postponed at the request of the participating HEI until October 2004. In all, 82 HEIs were involved in the programme (see Appendix 1) and DEs covered 15 subject areas (see Appendix 2). There were two instances where HEFCE agreed that two subject areas eligible for a DE should be considered in a single report.

6 The formal programme of activity for a particular DE commenced with the subject provider submitting an SED nine weeks before the DE visit. This was followed some five weeks later with an optional preliminary meeting between institutional representatives and the DE Review Coordinator. The DE team's visit to the HEI was prescribed to last no more than one and a half days, and ended with an oral report on the team's main findings. The DE culminated in a written report confidential to QAA, the HEI and HEFCE. See Developmental Engagements: Guidance note for a full description of the DE method.

7 Normally, each DE team comprised one Review Coordinator, one Auditor, one subject specialist reviewer and one IN. The exception to this was where two subject areas were considered together; then, the team contained two subject specialists. In all, 340 DE team members were involved in the programme of DEs (see Appendix 3).

Report methodology

8 Part A of this report is based on QAA's evaluation of DEs. Three linked forms of evaluation were undertaken. First, after completing their DE training all participants were requested to complete a post-training questionnaire. Second, all team members, along with HEIs undergoing the DE, were asked to complete an evaluation questionnaire after each DE. Finally, at the end of each academic year QAA undertook a number of evaluation seminars which were informed by quantitative analysis of the post-training and post-DE questionnaires. An internal evaluation of the DE methodology was also undertaken after the first 10 DE visits had been completed.

9 Part B is based on a qualitative analysis of all available draft and finalised DE reports. Four experienced DE Review Coordinators read reports and completed a grid to summarise and highlight the findings of the reports. Each Review Coordinator was then required to draft a summary of the findings of an approximately equal number of sections based on their analysis of the four grids. A workshop was held to discuss and redraft the section summaries before being consolidated into a first draft of the report.

10 The conclusions section draws on both Parts A and B of the report. This report is intended to provide an overview of the findings arising from DE reports and a summary evaluation of the DE methodology. As such, it contains no direct reference to specific DEs by HEI or by subject area.

Part A: Evaluation of the developmental engagement methodology

11 The most extensive evaluation of DEs was undertaken through post-DE questionnaires. QAA received 650 completed questionnaires. All statements were posed as positive questions, and resulted in 94.75 per cent agreement or strong agreement with the statements. Areas that achieved the highest percentages were the role and contribution of the IN; the developmental nature of the DE; the DE team; and the interaction with the HEI. The areas that received the lowest percentages were the quality and timing of documentation; initial training; modifications to the methodology once the process was up and running; the short amount of time allocated for each team member to undertake the DE; and the variability in which the DE was organised. However, it should be noted that even the lowest rated area received more than four out of five positive responses. Some of the more innovative features of DEs are reported in more detail below.

Higher education institution involvement in determining the nature of a developmental engagement

12 In the spirit of working in cooperation with the HEIs, a key feature of DEs was the role of QAA in negotiating with the HEI and finalising a number of key aspects of the reviews. HEIs were able to select the subject focus of the DE, determine some of the key operational matters and nominate a member of the team as the IN.

13 The arrangements for the transitional period left it to the HEI concerned to decide which of its eligible subject areas should be involved in a DE. Eligible subject areas were those not previously reviewed by HEFCE or QAA (from which at least one complete cohort had graduated) and, under the HEFCE quality assessment method applying prior to 1995, subjects that received a 'satisfactory' judgement from HEFCE on the basis of the self-assessment report, but were not visited.

14 The Developmental Engagements: Guidance note set out a number of ways in which an HEI could determine some key operational aspects of the DE. These included stating the start date of the DE visit and whether it should be conducted over two consecutive days or on two separate days over a one month period. In the vast majority of cases the DE visit commenced on the first preferred date of the HEI and, in all cases, QAA was able to meet the HEI's preferences that the DE visit should take place over two consecutive days.

15 HEIs were also free to choose the type of SED they would submit and whether to have a preliminary meeting with the DE Review Coordinator. The Developmental Engagements: Guidance note set out three types of SED:

• a recent internal review report
• a set of documents, including a covering explanatory note and an index
• a specially written SED.

The intention was to ensure that DEs did not significantly add to the burden of subject providers by expecting specifically written documents, but to leave it to the discretion of the HEI to decide which form of SED to submit. An internal review report was submitted by 13 per cent, 5 per cent submitted a set of documents including a covering explanatory note and index, and 82 per cent provided a bespoke documentation.

16 All HEIs opted to have a preliminary meeting with the DE Review Coordinator. This usually involved a combination of institutional and subject staff and, in many cases, the IN was also present. Discussions at the preliminary meetings tended to focus on the scope of the visit and the programme of meetings during the visit. HEIs and Review Coordinators reported that these meetings were usually highly constructive and provided a degree of reassurance to the subject and institutional staff.

Team composition

17 A specific feature of DEs was that DE teams comprised a number of different types of reviewer. There were both external and internal members of the DE team. Three members were nominated by QAA and one, the IN, was nominated by the HEI. Of the three external members there was always one Review Coordinator, one Auditor, and one subject specialist reviewer. All external DE team members were trained in other QAA review methodologies, and all members of a DE team, including the IN, were required to complete the one-day DE training session provided by QAA.

18 Regardless of their role, all DE team members were required to undertake a range of tasks including reading and commenting on the SED and, where appropriate, the SWS in advance of the DE visit. They also attended meetings with subject and institutional staff and meetings with students, and all team meetings, including those in which judgements were agreed. Each team member was required to draft sections of the report and comment on the draft DE report.

19 The rationale for a team composed of a number of types of reviewers was to ensure that a single team, to which each member brought various attributes and skills, could achieve the main purposes of a DE within the time frame. Teams were able to test the strength of the internal review procedures at the level of the discipline or programme, the robustness of the evidence used in those procedures, and arrive at judgements on the standards set for, and achieved by, students and the quality of learning opportunities which enabled them to achieve the standards set.

20 It was clear from the evaluations that a DE team composed of a number of types of reviewers was in fact a strength of the DE methodology. Initial apprehensions regarding an individual subject specialist arriving at a judgement on standards were unwarranted. Where possible, QAA attempted to ensure that the DE team contained an Auditor, a Review Coordinator or IN who was also from the relevant discipline area. In addition, the whole team shared in the task of analysing student work in order to consider in detail the subject specialist reviewers' considered opinion on standards. Evaluation also showed that collective benefit resulted from a team of different types of reviewers. It enabled the team to cover the full breadth of the scope of a DE, gain a clear insight into the workings of the internal review procedures and look at the subject within the context of the HEI. This gave the subject provider a range of insights on the provision.

Institutional nominees

21 As noted above, one of the more developmental features of DEs was the provision for HEIs to nominate a member of the team. QAA's only requirements were that the IN be employed by the HEI, and that the IN attend a DE training session. The Developmental Engagements: Guidance note and the DE training sessions emphasised that the IN was a full member of the team with equal rights and responsibilities. Thus, in common with other team members, the IN undertook all the tasks detailed above (see paragraph 19).

22 Initial apprehensions relating to the role of the IN were not confirmed by evaluation and largely evaporated over time as the DE process settled down. All HEIs exercised the right to appoint an IN. INs were drawn from a range of roles within HEIs, including pro vice-chancellors responsible for quality, central administrators, senior academics from the discipline undergoing the DE and academics from other subject departments.

23 Although many DE trainees, including some INs, reported some anxiety about the role, these concerns were not extensively repeated in later evaluations. In fact, evaluation questionnaires completed by both HEIs and DE teams emphasised the importance of the role. The INs were often noted for their helpfulness and knowledge of the HEI, and their understanding of the institutional context. In addition, many INs were applauded for playing a full role in the work of the DE team and for their professionalism.

24 It was recognised at an early stage that INs potentially could be liable to divided loyalties. This was likely to be particularly the case for INs who were working in the discipline area undergoing the DE. Overall, QAA's evaluation clearly indicated that INs were very adept at undertaking this sensitive role, and that most played a full part in the work of the DE team and remained objective throughout. However, there was some evidence to suggest that INs who were academics from the department concerned with the DE were likely to be most vulnerable to the problem of divided loyalties: to the DE team, to their subject colleagues and to their HEI. It must be noted that some INs from the subject area were able to fulfil the role without any apparent difficulty. While QAA continued to respect the right of the HEIs to appoint any individual to be an IN, it encouraged them to appoint individuals who would be best suited to ensuring that the HEI gained the maximum benefit from the DE.

Students' written submissions

25 A further innovative feature of DEs was that students studying on the programmes were invited to submit a SWS that evaluated the standards set for, and achieved by, them and the quality of learning opportunities that supported their studies. This was the first time SWSs have been included in QAA reviews at the subject level. In part, the SWSs were intended to prepare students for the opportunity offered them in institutional audit to submit a written submission. To enable students to produce an SWS, QAA published a separate briefing note (for details see www.qaa.ac.uk/students/guides/DevEng.asp) and consulted national student representative bodies. Students were encouraged to share their SWS with the subject provider, but if they preferred they could ask that the DE team (including the IN) keep the contents of the SWS confidential.

26 SWSs were provided for 124 DEs. They varied in length, from a single page to a survey analysed by the Students' Union. The DE teams considered most of the SWS provided a positive and valuable input into the review and contained a balanced account of strengths and areas for improvement. One instance, where the SWS was shared with the HEI, prompted the HEI to state that the 'comments from students have provided insights which it would be difficult for the institution to obtain without the involvement of external reviewers'.

Developmental versus judgemental

27 From the outset of development of the DEs, QAA was sensitive to, and sought to minimise, the tensions inherent in a process that was both judgemental and developmental. The developmental nature of DEs was a key feature which appeared in the qualitative section of the evaluation questionnaires and was followed up in the focus group sessions, particularly with HEIs and their INs. It was clear from these activities that the judgements prompted some increased levels of apprehension about DEs. However, HEIs did not always find this to be a negative feature, prompting some to note how the DE had caused subject teams in advance of the DE to 'pull together' and to make 'linkages between quality processes'. When evaluating the relationship with the DE team, HEIs frequently commented on the 'continuous dialogue', which contributed to the sense that this was a discussion among peers intended to encourage reflection on matters that closely affected the student experience. HEIs also highlighted that, for a number of reasons, the DE had helped prepare them for institutional audit, especially for DATs. Underlying many of the developmental aspects of DEs was the insight gained by the IN, as a full member of the team, into QAA review methods and practices. DEs also proved a suitable introduction to QAA processes for those subject areas that had not recently been involved in a programme level engagement with QAA. Moreover, they provided subject areas and HEIs alike with an opportunity to reflect on their engagement with the Academic Infrastructure, and for HEIs to disseminate, good practice identified by DE teams.

28 The judgemental aspect of DEs did not feature strongly in the evaluations completed by HEIs and INs. However, when it did, they welcomed the fact that judgements were threshold judgements. A typical comment, which reflected a number of others, was that 'perhaps there is scope for separating this out and dealing with it at a separate or earlier part of the process so that thereafter the focus of discussion could be more explicitly on developmental issues'.

Part B: Learning from developmental engagements

29 This part of the report summarises the main findings of the DE teams. DE reports contained three main sections, with the first two containing two threshold judgements: standards set for, and achieved by, students, and the quality of learning opportunities; the third section commented on the procedures for the maintenance and enhancement of standards and quality. The first two sections summarise the many positive features which underpinned the threshold judgements and highlight issues where DE teams identified areas of concern. However, none of these concerns were considered by DE teams to threaten standards or quality overall. The final section reports on the strength of the internal review mechanisms operating at the subject level.

Academic standards set for, and achieved by, students

30 DE teams were explicitly required to make a judgement of confidence or otherwise in the standards set for, and achieved by, students. The judgement focused on whether the intended learning outcomes (ILOs) of the programme(s) were appropriate in content and academic level for the named award(s), the design of the curriculum, the assessment of student achievement, and whether actual student achievement was generally consistent with the ILOs.

31 In all cases, DE teams had confidence in the academic standards set for, and achieved by, students. The main features underpinning the confidence judgements in standards follow.

32 In all but the most exceptional cases, relating principally to individual programmes or parts of programmes, the ILOs of programmes reviewed were appropriate. The relationship between the ILOs and the relevant subject benchmark statements were found to be explicit, and the ILOs were appropriate to the level of the qualification as set out in The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ). DE teams found that, where appropriate, ILOs met the requirements of the professional, statutory and regulatory bodies. They also reported that the development of programme specifications had been utilised to good effect, providing students and academics alike with a clear statement of the content of their programmes.

33 Shortcomings in the definition of the ILOs were found to be more common in postgraduate than undergraduate courses. In a small number of cases it was reported that there was incongruence between the ILOs, the subject benchmark statements and the FHEQ. This sometimes occurred when modules were shared between postgraduate and undergraduate programmes. In some other programmes, it was reported that more explicit reference could have been made to key skills.

34 Standards were generally underpinned by curricula which were well designed to enable students to achieve the ILOs. Curricula were well structured to give students the opportunity to progress and develop from one level to the next. The DE teams made particularly positive statements about the benefit to students of placement or fieldwork opportunities built into curricula. In a number of instances, the teams made reference to the efforts made to ensure that currency of the curricula was maintained. In a large number of instances the DE teams referred to the strong and beneficial relationship between the curricula and the research interests of staff. Curricula provided a solid foundation of subject knowledge, and in several instances it provided students with the opportunity to develop a range of generic skills. Only in a small number of reports was concern expressed about the challenge and depth of the curricula.

35 Curricula were considered to contain clear vocational emphases where relevant. In general, such curricula were commented on particularly favourably. An area of good practice noted by a significant number of DE teams concerned the close working relationship between academic departments and representatives of the relevant industry or profession. This strengthened the vocational orientation and enhances graduates' employability. Some examples of the way in which providers made their courses more relevant to the workplace included an innovative unit relating to employability and a careers mentoring scheme, whereby each student was assigned to a local practitioner. One MSc was specifically designed to meet a particular local market need and extensively researched before its establishment. Although the close involvement of practitioners and representatives from industry and the professions was a welcome feature of the design and delivery of many programmes, DE teams identified a number of examples of subject providers that did not have an appropriate level of external involvement in curriculum design and development.

36 In the great majority of cases, appropriate assessment strategies were in place which allowed students to demonstrate achievement of the ILOs. Many subject providers had mapped the ILOs on to the assessment scheme. One DE team reported that this resulted '...in a clear, well-thought-out rationale for the chosen method of assessment'. However, other DE teams reported that this was still an ongoing process for a number of programmes and required further work. Where there was a lack of integration between the ILOs and assessment, it was not always clear how assessment was being used to measure achievement of the ILOs.

37 Subject providers employed a wide range of appropriate assessment tools, and showed examples of innovative approaches. In a minority of cases, DE teams considered that a greater variety could have enhanced the assessment process, thus reducing the reliance on unseen examinations and essays, and introducing, for example, the testing of oral skills. Although the use made of formative assessment was reported as a strength in some programmes, little use was made of it in others. In some instances, the assessment regime was more restricted than that suggested in the subject benchmark statement. In a small number of instances, the DE teams considered the assessment load to be too high and in others there was insufficient clarity or consistency in the assessment process. Sometimes, DE teams noted a reluctance to use the full range of marks.

38 Reviewers noted that subject teams usually adopted appropriate moderation processes to ensure rigour and fairness in the assessment of summative work. These included double marking, providing students and staff with clear assessment criteria and adopting practices such as anonymous marking. Examples of good practice included the introduction of generic marking criteria, to encourage the use of the full range of marks, and the videoing of assessed presentations to enable moderation processes to be applied. Some subject providers were reported to have rigorous practices relating to the blind double marking of scripts and, in certain cases, of seminar or live presentations.

39 There were a number of areas related to assessment practices where DE teams concluded that practice could be improved. In some instances, grading criteria were insufficiently articulated for staff and students. In others, there was evidence of inconsistency in the application of marking criteria and examples of both over-harsh and over-generous marking. Also, moderation processes were not always operated consistently, were poorly evidenced or there was no indication that student work had been double marked. Marking sometimes lacked transparency so that students were not aware of procedures. In other cases, DE teams concluded that school or departmental policies or practice were not fully consistent with those prescribed by the HEI.

40 A key activity underpinning DE teams' judgements on standards was the review of a sample of student work. From this activity, DE teams concluded that a significant number of subject providers were giving their students constructive and detailed feedback on their written assignments. In many of these cases, DE teams considered that the feedback had a positive impact on student learning, promoted reflection and development. Peer review by students was sometimes successfully exploited as part of this process. However, a relatively high proportion of subject providers was criticised for the feedback they provided. In many instances, this opportunity to support students in their learning was not exploited to the full. In a significant number of cases, the return of work was not timely and was found not to conform to university requirements. This meant that students were unable to benefit from feedback in completing their next assignment. In other instances, the feedback provided was of variable quality and quantity, with some examples of minimal written feedback, so that the DE team judged certain students to be disadvantaged. Although shortcomings in written feedback were often compensated by the availability of oral feedback, not all students took full advantage of this. There were some examples of written feedback on student work that did not obviously relate to the assessment criteria or the ILOs, and feedback that was not constructive and did not enable students to easily identify areas for improvement. A disparity was occasionally noted between positive comments and low marks.

41 The samples of student work scrutinised by DE teams, in almost all cases, indicated that student achievement was at least appropriate to the level of the named awards and demonstrated achievement of the ILOs. In a number of instances, the DE team highlighted significant added-value in the standards achieved by students from diverse non-traditional educational backgrounds. Some DE teams also commented on the particularly high level of student achievement. Only in exceptional cases was concern expressed over the standard of student achievement by the DE team. For example, in one case, the student work sample seen by the DE team suggested that a significant minority of students did not demonstrate achievement of the full range of ILOs in one particular element of a course, yet still obtained the award. Areas for improvement noted, in a few DE reports, concerned the referencing of essays and projects, and, in some cases, the way in which students' research skills were developed.

Quality of learning opportunities

42 DE teams were explicitly required to make a threshold judgement of confidence or otherwise in the quality of learning opportunities that support students in achieving the academic standards of the award(s) to which their programme(s) lead. DE teams focused on a number of features including admission and induction arrangements, programme and study guides, the quality and variety of teaching and learning opportunities, the nature of academic support, the quality of learning resources and, where appropriate, fieldwork or placement opportunities. In all cases, DE teams had confidence in the quality of learning opportunities, although in some cases reports included recommendations for further action. Many of the features underpinning these confidence judgements are listed below.

43 DE teams reported that admissions and induction arrangements were generally of good quality. Some subject providers ran pre-entry study weekends for prospective students, while other DE teams reported some good use of diagnostic testing and study-skills training during induction. A common feature was the appreciation expressed by students of the well-established arrangements for admission and induction which were designed to provide a caring or community ethos.

44 A lack of parity in the treatment of students and the opportunities provided for them was sometimes problematic, in spite of equal opportunities policies. Students on one campus, for example, described themselves as 'second-class citizens'. While the subject provider disagreed with this judgement, the DE team noted that the provider was nevertheless giving it thorough attention. A different kind of example was the inequitable experience of full and part-time students. Induction arrangements for part-time students sometimes failed to fully meet their particular needs. One example indicated a lack of parity and workload between part-time graduate diploma distance-learning students and their full-time colleagues, the latter receiving guidance packs for projects and dissertations.

45 Induction was usually supported by useful handbooks and guides. DE teams reported some high quality, comprehensive module and programme handbooks, several of which were praised by DE teams for their clarity and coverage. The clarity of other supporting documentation, including handbooks on key skills and personal development, was also commonplace.

46 DE teams reported that once actually enrolled on a programme, students could expect a good variety of teaching and learning experiences, and generally teaching of a high quality. Almost all DE teams commented positively on the quality of teaching and learning. In particular, they noted that the variety of approaches was appropriate and in many cases was demonstrating good innovative practice. This was shown, for example, by the development of writing skills through project work and interactive question and answer sessions. The willingness of staff to bring their research to bear on their teaching was also reported as a strength in several cases. Student learning was enhanced further through the incorporation of practical activities such as placements and fieldwork. A work-based learning scheme and a criminal clinic to develop practical legal skills were also examples of good practice in this area.

47 DE teams noted that teaching and learning approaches supported student achievement of the ILOs and were generally well matched to student cohorts. In particular, they met the needs of part-time students, of those from non-traditional backgrounds and of those with identified specific deficiencies in key subject areas such as mathematics. A number of subject providers had been highly effective in widening participation and, in support of this, had developed teaching and learning, and support strategies relevant to the needs of students from diverse backgrounds. There were also good examples of the integration of key skills into programmes.

48 It was reported that subject providers often placed considerable emphasis on developing students' learning skills through, for example, the use of progress files that actively supported learning. In addition, there was a growing use of managed learning environments, along with the use of the intranet, for dissemination of teaching materials and internet-based tutorials. Several HEIs provided high-quality learning materials for students, including intranet-based material and resources to support distance learning. While some subject providers had gone a long way towards harnessing the full potential of managed learning environments, there were many cases where DE teams reported that these were not yet being used to their full potential. Some DE teams expressed concern about equality of opportunity for those students who did not have a computer at home.

49 A carefully considered teaching and learning strategy was not available for all HEIs. DE teams concluded that, in some cases, the lack of an explicit strategy made it difficult to coordinate teaching approaches and disseminate good practice, notably at subject level. Some DE teams concluded that the strategies used by a number of subject providers for developing independent learning skills could have been developed to include virtual learning environments.

50 The provision of academic support was a key aspect of students' learning opportunities. This was reported as being particularly good. Students appreciated the general accessibility, supportiveness and helpfulness of subject staff. Students were provided with appropriate advice when selecting their programmes and making option choices. Many HEIs operated tutorial systems that provided students with opportunities to receive support and guidance in areas of weakness, and most students had a personal tutor. However, in some instances, students did not make full use of these opportunities. Reviewers reported that, in several cases, the monitoring of student progress, and effective systems for identifying both students at risk and reasons for high failure rates, had led to improved progression and achievement. A number of DE teams reported good practice in the support of students with specific learning needs, including dyslexia.

51 Although academic support and guidance was reported, mostly positively, there were a number of instances in which DE teams indicated that providers might improve student support arrangements. Examples included inconsistency in the effectiveness of personal tutors, support for students preparing for, and undertaking, work placements (some demonstrated little reference to the practice identified in the Code of practice for the assurance of academic quality and standards in higher education (Code of practice), Section 9: Placement learning), student handbooks which were missing key information, and poor systems for diagnosing and supporting students with specific academic weaknesses.

52 While HEIs generally had formal support systems in place, DE teams noted that students particularly valued the less formal support that resulted from the accessibility of staff, a point often made by student groups in discussions with DE teams. In many instances, students felt able to approach the member of staff they perceived to be the most appropriate for their needs. In some reports, DE teams expressed concern about this reliance on informal arrangements. They considered that such approaches could be jeopardised as a result of the increased numbers of students.

53 Student progression rates were generally satisfactory and, in many cases, good or excellent. This was particularly the case for single honours students. DE teams also often noted staff commitment to improve retention rates, and the success of some subject providers where students often enrolled with non-traditional qualifications. However, some DE teams noted that progression rates had been problematic, although appropriate action had been taken to address the issue. In others, student withdrawal rates had been, and remained high. Overall satisfactory progression sometimes obscured specific problems relating to high initial failure rates in specific subjects. It was reported that such problems were not always investigated by the HEI in order for possible corrective action to be taken.

54 DE teams generally cited well-qualified, research-active staff as a particular strength of the provision. In other cases, part-time or visiting lecturers were reported to have made a valuable contribution because of their knowledge and experience of current practice. In a few cases, provision was adversely affected by inadequate numbers of academic staff or poor management of staff deployment. In one case, the DE team reported that staffing difficulties were placing a programme in jeopardy. In another case, extensive reliance on part-time lecturers was considered to be having a negative impact on the quality of learning opportunities.

55 Information and communication technology (ICT) and library provision was of a high quality in many HEIs, although, while access to these facilities was generally satisfactory, it was sometimes reported to be poorly linked to part-time student timetables. In several cases, students had access to some world-class learning resources and very high-quality specialist resources, including well-equipped laboratories. Students also benefited from some excellent support from library and ICT staff. Poor teaching accommodation and inadequate library and ICT facilities were identified by only a few DE teams. Other providers, it was concluded, needed to improve their management of resources, to ensure effective deployment and appropriate student access, notably for part-time students and students with disabilities. In addition, some libraries provided a poor learning environment because of high noise levels.

56 DEs also considered work placement arrangements and other examples of experiential learning, where appropriate. While a small number of providers needed to systematically address the quality of placement learning, most were found to be appropriate and adhere to the precepts of the Code of practice, Section 9: Placement learning. Placements were generally found to provide valuable learning experiences, particularly when they were well monitored. Overseas fieldwork and periods of study abroad provided a challenge for some subject providers. In one case, the opportunities were reported to be of outstanding quality. However, the financial cost was a source of concern to a minority of students. The provision of lower-cost fieldwork was an alternative and was achieved in one case through a carefully managed budget that ensured sufficient funds were available to subsidise fieldwork costs for students.

57 One of the ways in which students could be supported effectively when abroad, or on work placement, was through high-quality, distance-learning materials. Occasionally, these were outstanding. Good practice was noted in the case of a subject provider working closely with local service agencies to support and guide students on placement. Good relations with employers ensured that students found appropriate placements in which they were well supported and supervised. In such cases, the future employability of graduates was often enhanced.

Maintenance and enhancement of standards and quality

58 In this section, DE teams reported their findings on the maintenance and enhancement of standards and quality (MESQ). The MESQ section is concerned with the robustness and security of institutional systems relating to the awarding function. In particular, it focuses on the arrangements for dealing with the approval and review of programmes, the management of assessment procedures and mechanisms for gaining feedback from stakeholders, especially students and public, statutory and regulatory bodies, on standards and the quality of learning opportunities, and staff development. In addition, this section of the report contains a judgement on the subject team's ability to engage with the Academic Infrastructure, and the progress the HEI had made with fulfilling the expectations in HEFCE 03/51.

Programme review

59 Most DE teams identified successful systems of programme review that were variously characterised as clear, coherent, comprehensive and rigorous. They were reported to conform to institutional policies, to be in accordance with the Code of practice, Section 7: Programme approval, monitoring and review, and to be clearly fit for purpose.

60 Good practice in programme review systems included institutional handbooks that definitively enumerated procedures in readily intelligible terms. The review systems were clearly integrated within the HEI's committee structure. A notable example of this was captured in the report of one DE team: 'The University appears to have successfully balanced devolution of responsibilities to faculties and schools with a resulting empowerment of individuals, and an oversight of how these responsibilities and powers operate at the level of the discipline'. Issues arising from reviews were identified and appropriate action taken. Reviews were accompanied by clear action plans that were carefully monitored. There was appropriate external representation. Such systems were sometimes described as complex but also as rigorous or secure. They also encouraged reflective practice, careful consideration of the programmes under review and educational enhancement, including the dissemination of good practice. This was reflected in the fact that many of the issues listed as concerns by DE teams had already been identified by the internal review systems.

61 Programme review was typically divided into annual and periodic review, the latter being more extensive and usually occurring on a five-year cycle. Good examples of both types of review were reported as being supported by a clear presentation of key information (such as statistics on student progress) analyses of student questionnaires and reports from external examiners. These were often embodied in reports from individual programmes or groups of programmes. Issues were identified and informative and evaluative reports were made to senior committees, with appropriate action taken. There were clear lines of accountability, together with checks that actions proposed had in fact been taken. This was all clearly recorded, leading to a well-documented audit trail. Such systems were reported to be under constant review and had, in some cases, been modified in the light of the changing QAA processes of academic review and institutional audit. Notable developments were the introduction of thematic reviews and the division of the periodic review into two, with separate examination of the programmes and quality assurance processes in each subject. In some cases, DE teams emphasised that they were unable to fully evaluate review processes that had only recently been adopted and had not yet become completely operational.

62 DE teams made critical comments about the minority of less successful review systems. Some comments referred to isolated defects in systems that were otherwise successful. Examples of the latter were references to procedural failings, such as reports not being considered by bodies whose duty it was to do so, and the need for a more explicit evidence base on which judgements were made. In both cases, however, it was made clear that the review process as a whole was beneficial.

63 Examples of the more general issues raised by DE teams concerning less successful review systems included gaps and omissions in the quality review handbooks. In addition, overall structures were not fully effective. While there may have been firm central direction, it was not accompanied by a sense of ownership by staff at the subject level. One of the most serious issues was the number of times that the DE teams criticised a lack of clarity in ensuring that issues raised were actually addressed, exemplified by the lack of an action plan. An allied issue was the fact that agreed procedures were not being followed and that the error had not been identified within the HEI. The lack of external representatives, especially in the case of the approval of new programmes, was seen as a problem, as was the lack of clear minute taking of meetings.

64 DE teams were more critical of annual review. While there were many good examples of annual review, it was on occasions as variable in quality, not well documented, lacking in detail, lacking in data and analysis, and mechanistic rather than evaluative. It was summed up in one report as being 'too much of a tick-box process'.

65 Reviewers considered that subject teams divided roughly equally into those where the provision of statistical information was at least satisfactory and often much better, and those where there were significant failings. In the better provision, the statistics were typically characterised as reliable, comprehensive, easily interpreted and fit for purpose. They were not gathered for their own sake, but were actively analysed and used intelligently to inform strategies in such areas as student progression. In this way they contributed to the educational provision. They performed an essential function in underpinning the annual review process. Analysis at subject level was facilitated by the provision of centrally-compiled data, but this was not always the case, and reference was also made to separate data being generated in the faculty or the department.

66 An example of good practice was the careful consideration of mean module marks across programmes and of degree classifications across different years. This analysis contributed to the maintenance of standards across modules and influenced the design and delivery of first year modules, thus contributing to student retention.

67 In the less satisfactory provision, statistics were sometimes characterised as being inaccurate, inconsistent and not sufficiently comprehensive. The major criticism was that they were not fit for purpose and did not sufficiently support the quality assurance processes. This resulted in the production of monitoring reports supported by partial data only, the analysis of which was perfunctory. Problems were also caused by parallel databases held locally and centrally. There was also criticism of the failure to the make best use of the data available. One DE team explicitly stated that increased staff awareness and understanding of statistical information would be beneficial.

68 A frequent comment by subject providers was that these problems would be addressed by the imminent introduction of a new, improved management information system. However, such optimism needs to be tempered by the comment in one report that difficulties in the production of data were caused by the fact that the new management information system was not yet embedded.

External examining

69 The greater majority of DE teams identified successful external examining arrangements that were variously characterised as well-defined, effective, robust and reflecting the Code of practice, Section 4: External examining. The following statement of one DE team was indicative of an effective system: 'External examiners' reports are completed according to University format and submitted to the Pro Vice-chancellor and then considered by faculty and school boards. Responses to the comments are conscientiously addressed and balanced. The process ensures that the issues raised by external examiners are effectively addressed and the external examiners are informed of actions taken'.

70 There was clear evidence that external examiners confirmed whether the level of achievement of the students was satisfactory, in line with the ILOs, and were consistent with national standards. There was less explicit evidence from DEs that external examiners ensured that assessment procedures were appropriate and were fairly operated, but this did not give rise to problems in practice. The influence of external examiners on the enhancement of provision was clear and a number of examples were quoted of positive action being taken in response to their reports. This most often related to improvements in assessment procedures, but some extended to areas such as curriculum development. Typically, HEIs had rigorous systems to ensure that concerns raised by external examiners were identified, carefully considered, and, if appropriate, acted upon, and that they were kept informed of the responses to their comments. The DE teams reported examples of good practice to include special briefing for external examiners; independent cross-checking of reports outside the subject department; and a collation of reports with key issues highlighted. The latter was available on the staff intranet and provided useful and rapid access to key issues and action plans.

71 The main point of criticism noted by DE teams was a lack of clarity in procedures to ensure that concerns expressed by external examiners were identified and addressed. In some cases, the DE teams reported there was limited evidence that the concerns of external examiners had been taken into account. They also made reference to external examiners not being informed whether their recommendations had been acted upon, sometimes in contradiction to institutional requirements. The other significant criticism was of the variability in quality of the external examiners' reports, some being described as perfunctory. In some cases, while action by the HEI was generally careful and considered, the lack of a rigorous system of checking meant that isolated, but serious, mistakes could be made. For example, in one case the DE team noted that one of the external examiners had not submitted a report for three years.

Student involvement

72 DE teams reported a wide range of mechanisms used by subject providers to gain input from students into MESQ. Questionnaires and representation on staff-student liaison committees (SSLCs) and other committees were the most common, and were generally effective in practice.

73 Questionnaire surveys usually took place at one or more of three levels: initially, at module level; then at programme level; and finally in relation to the overall student experience in the HEI. Formal student feedback of this kind was usually collected efficiently, responded to, and appropriate action was taken. In some cases, however, there were student evaluation procedures that lacked objectivity and did not result in visible actions. Additional security was given by the independent processing of questionnaires. Good practice was noted in the form of end-of-module questionnaires being supplemented by interim module review discussions and student representation on programme boards. This allowed for the possibility of responding quickly to make a difference to the learning experience of the current cohort.

74 Other good practice noted by DE teams included the production by course leaders of a report in response to questionnaire feedback. SSLCs generally considered such reports before they were passed on to the programme board. The results from these deliberations were typically fed back to students through noticeboards or incorporated in a module log which, in turn, fed into a programme or subject log.

75 DE teams generally found effective engagement of students in the quality process, with representation on appropriate departmental and institutional committees. Training provided for student representatives, often by Students' Unions, was common and, in a few cases, was supplemented by a course representatives' handbook.

76 The positive engagement of students in MESQ of their courses of study was not universal. There were a few instances where students were not properly engaged in the process. The DE teams commented on the lack, or ineffectiveness, of feedback and representation mechanisms, the haphazard gathering of student views to overcome poor return rates and limited quantitative analysis of questionnaires.

77 Generally students felt that their views were treated with respect by department staff. SSLCs provided opportunities for students to raise issues and, in the best examples, their input contributed much to the supportive and collegiate atmosphere of the subject provider. In these circumstances, SSLCs were regarded by staff and students to be of major importance for raising suggestions for improvements, with students expressing satisfaction with arrangements and with the responses received to their input. However, while most subject providers were responsive to student concerns, resultant action was not always well documented. DE teams sometimes considered that SSLCs could be more influential and visible in order to overcome the problem of students having no clear idea of the identity or role of their representatives. Infrequent attendance by student representatives was sometimes a problem. In some DEs, it was also reported that better staff commitment to the involvement of students in agreed quality assurance processes was also needed.

78 In many instances, informal channels for student feedback were reported to supplement more formal mechanisms and generally operated effectively. This was found by students to be the most effective way of achieving decisive action in response to their concerns, particularly in small provision. However, sometimes this led to a lack of solid evidence of the operation of an effective process. While an immediate one-to-one response to individual student issues was admirable, reviewers considered that students might be excluded from more reflective programme development.

Staff development

79 A majority of reports referred to a comprehensive staff development structure being in place, often linked to the subject providers teaching and learning strategy. Many HEIs had a clear and extensive portfolio of staff development activities including induction for new members of staff, Institute of Learning and Teaching in Higher Education accredited courses for members of staff new to the profession, and support for staff in post. However, the extent to which individual staff availed themselves of these opportunities was much more variable. Clear linkage between staff development and key staff roles appeared to be fundamental in engaging staff successfully. One example of successful practice was noted where development was clearly linked to maintaining current and appropriate modules. Study leave was often a crucial element in the integration of teaching and research.

80 There was clear evidence that teaching staff kept their approaches to teaching and learning under review. In particular, they were responsive to student feedback and to comments from external examiners. However, DE teams conveyed an overall impression of the haphazard dissemination of good practice. Few HEIs systematically made this an explicit and pivotal part of their quality enhancement processes. In one instance, the subject teaching committee, charged with the role of quality enhancement, consisted largely of part-time staff and was poorly attended. Whereas staff development processes lead naturally in the direction of disseminating good practice, they did not necessarily, nor alone, provide an efficient vehicle for identifying it. An example of good practice reported by one DE team was that the faculty Head of Quality Assurance attended boards of study and reported on good practice in other departments. Another department had taken the initiative of setting up a Forum for the Enhancement of Learning and Teaching.

81 Generally, there was alignment of subject providers' practice with institutional policy, and staff appraisal informed staff development planning and effective systems were characterised by strategic and systematic staff development linked to appraisal. Good practice was noted where formal personal development review informed the strategic programme of staff development. In one case, staff appraisal and development arrangements had led to Investors in People accreditation.

82 Good practice in staff development and appraisal was by no means universal. Inadequacies reported included a lack of explicit structures and procedures, suspended or inactive appraisal systems, or appraisal not completed within the agreed timescale. A lack of sharing of good practice was also noted, together with a focus on subject-based research to the exclusion of pedagogical development. In other cases, there was limited evidence of staff engagement in scholarship.

83 Many subject providers included their part-time and sessional staff in the opportunities provided for staff, as well as having comprehensive programmes for the induction and training of new staff. However, in one case, specific training of postgraduate teaching assistants was on a voluntary basis only. A lack of development opportunities for part-time staff was more commonplace. In another case, new staff were unaware of the requirements of the probationary system and found themselves without formal mentoring.

84 Peer review of teaching afforded the opportunity for staff to reflect on their practice. An effective process successfully built on the expertise of recognised good teachers and led to documented improvements. In many cases, there was systematic peer review that included feedback from students. DE teams noted evidence of peer observation having a positive impact on learning and teaching and resulting in the dissemination of good practice, notably with the introduction of independent observers. However, peer review was by no means universally practised or fully embraced by staff. In one case, peer review was institutional policy but was not implemented at subject level.

85 While peer review was one way of disseminating good practice, DE teams also alluded to other important ways by which this objective was achieved. Regular departmental planning conferences created opportunities for this, along with annual away days for reflection. Some DE teams reported a lack of evidence of systems for the dissemination of good practice, particularly between a university and partner institutions.

Engagement with the Academic Infrastructure

86 The Academic Infrastructure, including the FHEQ, subject benchmark statements, the Code of practice and guidance on programme specifications, was developed by QAA on behalf of the HE sector in response to The Report of the National Committee of Inquiry into Higher Education (the Dearing Report). The following section considers the extent to which the subject providers involved in DEs have engaged with the Academic Infrastructure.

87 In a significant number of instances, HEIs had systems in place to enable them to engage constructively with the Academic Infrastructure, to review their own procedures in the light of the various sections of the Code of practice and to ensure that their programmes were consistent with the FHEQ. That this engagement had taken place was most obviously apparent from institutional quality assurance handbooks. In most HEIs, procedures demonstrably took full or significant account of these external reference points. Typical comments made by DE teams included, 'the institution since 2000 has progressively and systematically reviewed institutional policy, systems and practice in the light of the Code of practice', the institution was 'monitoring and addressing precepts in the Code of practice in exemplary fashion' and 'the University is to be commended on its adoption of a systematic approach to the integration of the Academic Infrastructure and guidance into its own policies and procedures within the...Academic Handbook'.

88 In most cases, the process of applying policies consistent with the Academic Infrastructure, at programme level, was also well developed. DE teams reported some exceptions. In some instances, more work needed to be done in relation to the FHEQ, particularly in relation to certain postgraduate programmes. Only exceptionally did DE teams report little evidence of engagement with the Academic Infrastructure. One example cited inadequate quality assurance documentation to provide guidance for staff and there were some examples of a lack of consistent engagement with the Code of practice.

89 Many HEIs had made changes to their quality assurance processes as a result of the publication of the relevant sections of the Code of practice. In a few cases, assessment practices had been changed to reflect precepts of the Code of practice, as shown, for example, by the development of an institutional code of practice for the assessment of students and the mapping of assessment practices against the Code of practice.

90 There was clear evidence of subject providers taking due note of subject benchmark statements and the FHEQ when revalidating programmes. For example, in one case postgraduate programmes were being realigned to reflect the FHEQ more closely. In another example, the University had modified degree regulations in response to the FHEQ and the Code of practice.

91 There was considerable evidence that programme specifications were commonplace, especially in programmes that had been recently established or had undergone revalidation. In many instances DE teams were provided with evidence of useful institutional guidance provided for the writing of programme specifications. In other cases, DE teams concluded that more work could be done on the development of programme specifications to ensure that they met the needs of targeted stakeholders.

Self-evaluation document

92 One of the purposes of DEs was to enable HEIs to prepare for their institutional audit. DE teams were required to comment on the evaluative nature of the subject providers' SED. In this regard, DE teams found that all the SEDs provided an adequate basis for the DE, but many lacked sufficient critical evaluation. Good SEDs were described as: 'clear, frank and readable'; 'fair and realistic'; 'open, honest, evaluative and helpful', and provided a useful starting point for the DE. However, there were at least as many reported as lacking an evaluative dimension. Sometimes, this lack of evaluation was at odds with the approach the DE team found when scrutinising other documentation or when discussing the provision with staff. Examples included the following: 'in a School where self-reflection and monitoring is practised with commendable rigour, it is disappointing that the SED is much less evaluative than the internal documents it depends on'; the SED 'did not do justice to the Department's willingness to be evaluative'. In a surprising number of cases, it was actually noted that this unwillingness to be evaluative in their SED resulted in the subject provider understating their strengths.

93 Other areas for improvement in SEDs included poor referencing and the assertion of good practice without accompanying evidence. In some cases, concerns about ownership of the SED were expressed because it appeared to be the work of a small minority of staff. In other examples, not only staff, but also students had been fully involved in an extended, constructive and valuable consultative process. Where the SED consisted of existing internal review documentation, this was found to provide a good basis for the review.

Teaching Quality Information: Responses to HEFCE 02/15 and 03/51

94 DEs were also intended to report on HEIs' preparedness for meeting the requirements set out in Information on Quality and Standards in Higher Education Final Report of the Task Group (HEFCE 02/15). Analysis of the extent to which HEIs were addressing the agenda for Teaching Quality Information (TQI) was limited in the earlier DE reports, because they took place before institutions were informed of the detailed expectations (see HEFCE 03/51). Later, DE teams were provided with evidence that nearly all HEIs were preparing themselves to address the requirements. In some reports, teams noted that this was confined to a brief reference to working towards HEFCE 03/51 specifications. Others referred positively to the establishment of task forces, carefully overseen by central committees, with a business-like approach and a confident and justified expectation that all the requirements would be implemented by the due dates.

95 A small minority of HEIs, however, faced serious problems. Typical comments were that the quality of the statistical data needed to be improved significantly and swiftly. This problem was generally understood by senior managers. For example, it was reported that an HEI was aware that at the time, it did not have adequate facilities for making available the data it held on students and student performance in a useful format for analysis. Rather more worrying were the comments in one report that there was no evidence of work in progress towards meeting the requirements for TQI, and in another that the lack of analysis of student achievement was a serious and pressing issue.

Conclusions

96 The operational arrangements for the transitional period, 2002 to 2005 for England and Northern Ireland, agreed by the Higher Education Funding Council for England (HEFCE), Department for Employment and Learning Ireland (DEL), Universities (UUK) and Standing Conference of Principles (SCOP) had at their heart the belief that HEIs were best placed to maintain and enhance the standard of awards and the quality of student learning opportunities. DEs were intended to provide a mechanism by which QAA reported on the extent to which those HEIs, awaiting their institutional audit in the latter part of the period, were fulfilling the responsibilities placed on them by these arrangements.

Accordingly, DEs had two stated purposes:

• to provide threshold judgements on standards and quality
• to test the strength of an HEI's internal review procedures operating at the subject level, and the robustness of the evidence used in those procedures.

97 In all DEs, the teams had confidence in both the standards of awards and quality of learning opportunities. This is not to suggest that teams found everything to be positive, but that, on balance, any concerns identified were not of a sufficient magnitude to threaten standards and quality overall.

The more common features underpinning the confidence judgements on standards included:

• student achievement appropriate to the level of the award
• ILOs that reflected relevant external reference points
• a strong and beneficial relationship between curricula and research interests of staff
• assessment strategies which enabled students to demonstrate achievement of ILOs
• sound moderation processes
• feedback on student work which is consistent and detailed.

Areas of concern relating to standards noted by DE teams included:

• ILOs that were inappropriate for the level of the award
• a limited variety of assessment methods
• insufficient clarity or consistency of assessment processes
• assessment processes that were not consistent with institutional policies
• feedback on student work which was not timely or was of variable quality.

The more common features underpinning the confidence judgements quality of learning opportunities included:

• high-quality information given to students
• high-quality teaching
• extensive use of the intranet and managed learning environments to aid learning
• formal and informal student support and guidance mechanisms
• provision of learning resources and ICT
• extensive use of fieldwork and placements to aid student learning.

Areas of concern relating to quality of learning opportunities identified by DE teams included:

• the occasional unequal treatment afforded to postgraduate or part-time students as opposed to full-time and undergraduate students
• inconsistent or ineffective support and guidance for part-time students and for students on placement
• staff numbers which were insufficient or where there was an over-reliance upon part-time or sessional staff
• the limited use of diagnostic testing to identify learning needs.

98 DE reports also contained compelling evidence of the in-depth strength of HEI's internal review procedures operating at the subject level, and the robustness of the evidence used in those procedures. DE reports highlighted the fact that each HEI has different arrangements pertaining to internal review. However, it was clear that their design and operation generally reflected the precepts of the Code of practice. It was also clear that the evidence base which underpins subject providers' internal review procedures was extensive and, in general, robust. DE teams reported in detail on the use made of statistical information, external examiners' reports, student feedback and staff development activities. They noted extensive good practice in all of these areas in many reports, although there were notable exceptions. Over half of the reports were positive about the use made of statistical information in the management and enhancement of quality and standards. In almost as many cases, however, DE teams reported that statistical information was incomplete, locally produced data were not consistent with centrally produced data, or the subject providers did not use the data to maintain or enhance standards and quality effectively.

99 By way of contrast, DE teams were often much more positive about the role of external examiners. Good practice reported included special briefing or training for external examiners, careful consideration of, and response to, their reports at the subject level (including through periodic and annual review) and the institutional monitoring of responses to external examiners including, in some cases, the production of an annual overview of matters arising from all external examiner reports. However, engagement with external examiners was not uniformly good. In a minority of cases, DE teams noted that arrangements for responding to external examiners' reports were not always clear, nor necessarily adhered to by the subject provider. In others, reviewers noted that external examiners were not always informed of how their concerns had been addressed. In a small minority of cases, teams expressed concern about the quality of the external examiners' reports, with some being described as perfunctory.

100 DE teams found that, in the large majority of cases, internal review procedures were extensively informed by student feedback. This was collected through module questionnaires and normally used as part of annual review, while programme-level questionnaires and other questionnaires focusing on the student experience were used to inform periodic review. Teams reported that usually feedback of this kind was collected efficiently, carefully considered by the subject provider, and that students were informed of the outcome of any deliberations. Procedures for representation were similarly widespread and effective. In most cases there existed a staff-student liaison committee and students informed DE teams that they felt their views were treated with respect and, in many cases, had resulted in action by the subject provider. Matters raised at such committees often fed into both annual and periodic review. Only in a minority of cases did teams report that formal mechanisms to gain student involvement were limited, or that the mechanisms were ineffective or insufficiently responsive, especially in terms of the analysis of questionnaire returns.

101 DE teams also noted that they were not always aided in their work by sufficiently evaluative SEDs. While most SEDs were at least fit for purpose they often lacked critical evaluation, sometimes to the point of obscuring the subject providers' strengths. Given that one of the specific purposes of DEs was to provide an opportunity for HEIs to gain some experience of being self-evaluative in preparation for institutional audit, this was perhaps the most limited area of development anticipated by DEs. Moreover, the limited evaluation contained in some SEDs was often out of kilter with what DE teams found on the ground; subject teams demonstrated greater capacity and practice for self-evaluation than was found in the SED.

102 The findings emanating from DEs generally confirm the picture of HE in England and Northern Ireland reported in other QAA publications. It is clear overall that HEIs operate rigorous internal review procedures which demonstrate that they are fulfilling their responsibilities for the maintenance and enhancement of standards and quality. DE teams found that the evidence base used for such reviews was extensive and robust, although they did reveal some areas of concern.

Developing external quality assurance

103 As with many external quality assurance processes provided by QAA, DEs were created to achieve a number of different objectives in a short space of time. The main stated purposes were to be both developmental and judgemental, and as part of these purposes, DE teams were expected to comment on the subject providers' engagement with the Academic Infrastructure and preparations for TQI. Composing DE teams of an Auditor, a subject specialist, a Review Coordinator and an IN, and the synergism that resulted, meant that most of these many purposes were achieved for each DE, while ensuring that the length of the visit to the HEI was no longer than one and a half days. In particular, the role of the IN was generally very well received by both HEIs and other team members. The IN not only provided the HEI with confidence about the detailed considerations of the team, but also provided an opportunity for the individual concerned to disseminate the experience of a DE internally, in advance of the institutional audit. From the DE teams' perspective, the IN not only provided a deep insight into the workings of the HEI but, in most cases, acted as a full and objective member of the team, involved in making judgements and drafting the final report.

104 In many DEs, teams were also considerably helped in their work by the provision of SWSs. These submissions often provided a means by which the team could initially evaluate the subject providers' own evaluation and, thus, provided a key starting point for the DE team to compose an agenda for their work.

Appendix 1: A list of institutions that experienced developmental engagements

Institution

Anglia Polytechnic University
The Arts Institute at Bournemouth
Aston University
University of Bath
Birkbeck College, University of London
Birmingham College of Food, Tourism and Creative Studies
Bishop Grosseteste College
Bolton Institute of Higher Education
Bournemouth University
University of Bradford
University of Brighton
Brunel University
Canterbury Christ Church University College
University of Central England in Birmingham
City University
Coventry University
Cranfield University
Dartington College of Arts
University of Derby
University of Durham
University of Essex
University of Exeter
University of Gloucestershire
Goldsmiths College
University of Greenwich
University of Hull
Imperial College of Science, Technology and Medicine
University of Keele
University of Kent
Kent Institute of Art and Design
King's College London
Kingston University
University of Central Lancashire
University of Lancaster
University of Leicester
University of Liverpool
Liverpool Hope University College
Liverpool John Moores University
University College London
The London Institute
London Metropolitan University
London School of Economics and Political Science
Loughborough University
University of Luton
University of Manchester
University of Manchester Institute of Science and Technology
The Manchester Metropolitan University
University of Newcastle upon Tyne
University College Northampton
University of Northumbria at Newcastle
The Nottingham Trent University
Open University
University of Oxford
Oxford Brookes University
University of Plymouth
University of Portsmouth
Queen Mary, University of London
The Queen's University of Belfast
University of Reading
Royal Agricultural College
Royal Holloway and Bedford New College
University of Salford
University of Sheffield
Sheffield Hallam University
University of Southampton
Southampton Institute
St Martin's College
St Mary's College
University of Surrey
University of Sussex
Thames Valley University
University of Ulster
University of Warwick
University of the West of England, Bristol
University of Westminster
Wimbledon School of Art
University of Wolverhampton
University College Worcester
Writtle College
University of York

Appendix 2: A list of subject disciplines covered by developmental engagements

Subject and JACS code

B Subjects Allied to Medicine
D Veterinary Science, Agriculture and related subjects
F Physical Sciences
G Mathematical and Computing Sciences
H Engineering
K Architecture, Building and Planning
L Social Studies
M Law
N Business and Administrative Studies
P Mass Communications and Documentation
Q Linguistics, Classics and related subjects
R European Languages, Literature and related subjects
V Historical and Philosophical Studies
W Creative Arts and Design
X Education

Appendix 3: Developmental engagements: a list of reviewers

QAA is very grateful to the following reviewers, including institutional nominees, for their involvement in the programme of developmental engagements.

Mrs Florence Ali Dr Christopher Amodio Mr Tim Appelbee Mrs Sue Applegarth Professor Allan Ashworth Mr Duncan Backhouse Dr Trevor Bailey Dr George Bainbridge Dr Bala Balachandran Dr Graham Baldwin Professor Richard Barker Dr John Barkham Mr Michael Bartlett Dr Christopher Batchelor Emeritus Prof John Beeby Professor Janet Beer Professor Clive Behagg Mr David Blaney Dr Christine Bleasdale Mr Derek Blease Mr Timothy Blinko Mrs Jennifer Blumhof Ms Anne Boddington Professor A Michael Bourn Dr Huw Bowen Mr John Bradbeer Ms Lynne Braham Professor Hugh Brayne Dr Elizabeth Briggs Ms Amanda Bright Dr John Broad Ms Claire Browning Mr A John Buchanan Dr Ian Buchanan Ms Susan Buckingham Mr Peter Bullman Professor Rodney Burgess Ms Sue Burkinshaw Dr Julia Bush Professor David Buss Professor Richard J Buswell Mr Jeffery H Butel Ms Sarah Butler Professor Jackie Campbell Dr Philip Cardew Dr Michael P Carter Dr Bernadette Casey Dr Neil Casey Professor Derek Cassidy Dr Stephen Chadd Eur Ing Alan Chantler Mr Peter Clarke Mr Jim Claydon Professor Andy Cobb Dr Henry Cohn Dr Paddy Coker Dr John Coleman Professor James Connelly Dr Nigel Copperthwaite Dr John Corbett Professor John Cowan Mr Charles Cowap Eur Ing Anthony Cowling Mr Gerry Crawley Professor Tony Cryer Professor John Cullen Professor Anne Curry Dr Edmund Cusick Mr Michael Cuthbert Dr Roger Dackombe Mr Alan Davidson Professor R Sam Davies Dr Robert Davison Dr Gail Day Professor Anthony Dean Ms Corine Delage Dr K Sara Delamont Dr Peter Denley Mr Patrick Devlin Dr Hazel Dewart Mr Alan Dordoy Mrs C Elisabeth Downes Dr Peter Dunleavy Mr Richard G Eales Professor Rae Earnshaw Mr Dai Edwards Professor Nathan Efron Mr Nicholas Ellison Dr Michael Emery Professor Eric Evans Professor Mark Everist Mr Marco Federighi Dr Ian C Fisher Professor Sue Frost Dr Colin Fryer Dr David Furneaux Professor Mads Gaardboe Professor Tony Gale Mr Roddy Gallacher Mr John Gannon Dr Paul Gardiner Dr Barry Garnham Ms Alex Geal Dr Roger Geary Dr Michael Gilmore Ms Judy Glasman Professor Sally Glen Ms Ruth Glenister Mrs Ruth Goodison Mr Colin Gordon Professor Barry Gower Dr Peter Grannell Mrs Amanda Greason Dr David Green Mr Paul Griffiths Mr Peter Griffiths Dr Joanna De Groot Dr Joan Halfpenny Mr Nigel Hall Professor Sue Hallam Professor Norman Hamilton Reverend Professor Robert Hannaford Dr Roger Hannam Dr E Lynne Hapgood Professor John Harper Mr Neil Harris Professor Robert Harris Dr Margaret Harrison Dr Wendy Cealey Harrison Mr David Hartley Dr Paul Hartley Ms Jenny Hawkes Mrs Jenny Hawkins Professor David Heeley Mr William Henry Professor Michael Hicks Mr Martin P Hill Dr Susan A Hill Dr Stephen Hill Mr Paul Hodges Mr Robert Hodgkinson Mr John Hodgson Mrs Mary Holmes Ms Sarah Hordern Mr A Grant Horsburgh Mr James Howell Dr Richard Howell Miss Carol Howells Professor Jim Hughes Mr Alan Hunt Mr Jim Hunter Professor Geoff Hurd Dr John Hurley Mr Stuart Hutchinson Mrs Stephanie Hutchison Dr David Hyde Mr Robert Ingram Dr Adrian Jackson Professor Howard Jackson Professor Adrian L James Professor Jennifer James Mr Michael Jefferson Ms Sara Jennett Dr David E Johnson Ms Hilary Johnson Professor Keith Johnson Professor John Jones Mr Robert H Jones Dr Trevor Joscelyne Dr Susan Kay Professor Terry Kemp Mr Anthony Kidd

Dr Stuart S Kidd Professor Graham King Mr Russell Kinman Professor David Kirk Mr Peter Klein Mr C Andy Knowles Ms Anna Kyprianou Dr Daniel Lamont Dr Mary Langan Dr Richard Latto Ms Mary Lawrence Dr Ian Lean Mr Adrian Lee Mr Patrick Lees Mrs Rosemary Leno Miss Siobhan Leonard Dr David H Lewis Dr Jenifer Lewis Ms Linda Lewis Dr Elisabeth Lillie Mr Philip Lloyd Professor Debbie Lockton Professor Jim Longhurst Professor Peter Lovell Professor James Low Ms Marion Lowe Mrs Patricia Lowrie Professor Robert Macredie Dr Paddy Maher Dr Ralph Manly Dr Frances Mannsåker Mr Philip Markey Ms Helen Marshall Professor Ian Marshall Dr Peter Marshall Dr John Martin Dr Peter Martin Dr Howard Maskill Dr Alan Maybury Professor James McAuley Dr Colin McClean Mr A C Paul McGrath Dr John McInnes Mr A Neil Mclaughlin-Cook Ms Marilyn McMenemy Mr Maurice Mealing Professor Madjid Merabti Professor David Miers Dr Anne Miller Mr Bob Millington Dr Fiona Montgomery Mr Clive Morphet Professor Clare Morris Dr Karen Moss Dr Harry T Mount Dr Chris Mulhearn Professor Robert Munn Ms Angela T Murphy Professor Ajit Narayanan Mr Jeffrey Naylor Prof Clive Neal-Sturgess Professor Jethro Newton Mr Alan Nisbett Mr Geoffrey Notcutt Mrs Annet Nottingham Ms Rona O'Brien Dr Pat Odber De Baubeta Dr Jon Owen Mrs Monica Owen Professor Michael Page Professor Chris Park Mrs Liz Parton Professor Dilipkumar Patel Miss Rowena Pelik Mrs Cheryl Penna Professor Paul Periton Mr Simon Persighetti Mrs Julie Phillips Mrs Clare Pickles Mr Andrew Pilkington Ms Hilary Placito Mrs Christine Plumbridge Ms Jacqueline Powell Mr Nick Pratt Dr Lynda Prescott Dr Shirley Price Ms Mary Prior Professor Lyn Pykett Mr John Pymm Mr Ken C Reid Mr Philip Richardson Dr Jim Rimmer Dr Larry Roberts Mr Brian Robinson Professor Geoffrey Robinson Ms Maureen Robinson Dr Richard Robinson Professor Anthony Rosie Professor David Ross Professor Rick Rylance Professor Jenny Saint Dr Keith Salmon Dr Robert E Schofield Dr Jon Scott Mr Martin Seath Mr Andrew Sedgwick Professor Marion Shaw Dr Sally Sheldon Professor Dominic Shellard Dr Valerie Shrimplin Professor John Simmons Professor John Simons Mr Paul Simpson Dr George Smith Professor Ruth Soetendorp Mr Garfield Southall Dr Philip Speare Dr Jon Spencer Dr Derek Spooner Mrs Agnes Stalker Dr Linda Stanier Professor Beryl Starr Dr Peter Steer Miss Marcia Stewart Professor Michael Stewart Mr Martin Stimson Dr Marie Stinson Professor Frank Stowell Dr Marie Stowell Professor Caroline Strange Professor Alice Sullivan Dr David Taylor Dr Neil Taylor Dr Stan Taylor Mrs Marion Temple Ms Helen Thomas Mr Michael Thomas Dr Anna Thomas-Betts Mr Alan Thomson Mrs Liz Thussu Professor Diana Tribe Dr Anthony Vickers Dr Carol A Vielba Mr John Wakefield Professor Peter Waldron Mr Lawrie Walker Dr Christopher Walsh Professor Doug Walton Dr Keith Walton Professor Malcolm Wanklyn Professor Richard Ward Mrs Elizabeth Warr Dr Sean Wellington Dr Roy Westbrook Professor Adam Wheeler Dr Richard Wheeler Mr Jeremy White Dr Margaret Whitelegg Professor Graeme Wilkinson Dr John Wolffe Ms Alison Wood Ms Amanda Wood Dr Jo Wood Professor Nigel Wood Mr David Woodhill Mr Andrew Wright

References

Arrangements during the transitional period 2002-2005 for Higher Education Institutions in England (2004)
http://www.qaa.ac.uk/reviews/institutionalAudit/archive/transitional/transitional.asp

Developmental Engagement: Guidance note (2004)
http://www.qaa.ac.uk/reviews/institutionalAudit/archive/transitional/dev_eng_guidance.asp

Guidance on producing an SWS (2004)
http://www.qaa.ac.uk/students/guides/DevEng.asp

Handbook for Institutional Audit: England (2002)
http://www.qaa.ac.uk/reviews/institutionalAudit

Glossary

Academic Infrastructure

The Academic Infrastructure, including the FHEQ, subject benchmark statements, the Code of practice and guidance on programme specifications, was developed by QAA on behalf of the HE sector as a whole in response to the Dearing Report.

Course

See programme

DATs

Discipline Audit Trails

Dearing Report

See National Committee of Inquiry into Higher Education (NCIHE)

DE(s)

Developmental engagement(s)

DEL

Department of Education and Learning in Northern Ireland

HEFCE

The Higher Education Funding Council for England

HEFCE 02/15

See HEFCE 03/51

HEFCE 03/51

This refers to the HEFCE document which provided final guidance (initial guidance was provided in HEFCE 02/15) on the provision of information on quality and standards of teaching and learning in higher education, to be published on a national Teaching Quality Information (TQI) website. This document was also sometimes called the Cooke Report. HEIs were able to begin loading their reports onto the TQI site from March 2004, and were expected to provide the first complete set of annual information by December 2004 (subsequently delayed until 2005).

HEIs

Higher education institutions

ILOs

Intended learning outcomes

ILTHE

Institute of Learning and Teaching in Higher Education

IN

Institutional nominee

National Committee of Inquiry into Higher Education (NCIHE)

Often referred to as the Dearing Committee (after its Chairman), its terms of reference were to make recommendations on how the purposes, shape, structure, size and funding of HE, including support for students, should develop to meet the needs of the United Kingdom over the next 20 years, recognising that HE embraces teaching, learning, scholarship and research. Its far-reaching and influential report Higher education in the learning society (often referred to as the Dearing Report) was published in 1997

Programme

It is recognised that HEIs use a variety of means to describe the main focus of a students study including course, pathway and programme. For reasons of consistency with other QAA reports the term programme is used here.

PSRBs

Professional, statutory and regulatory bodies

Quality

Academic quality is a way of describing how well the learning opportunities available to students are managed to help them to achieve their award. It is about making sure that appropriate and effective teaching, support, assessment and learning opportunities are provided for them.

SCOP

Standing Conference of Principals

SED

Self-evaluation document. Subject providers were invited to submit SEDs for DEs in one of three formats: a recent internal review report; or a set of documents, including a covering explanatory note and an index; or a specially written SED.

Standards

The words 'academic standards' are used to describe the level of achievement that a student has to reach to gain an academic award (eg, a degree). For similar awards, the threshold level of achievement should be the same across the UK.

Subject providers

The departments/schools/faculties which, on behalf of their institution, provide the education which is the object of the review.

Subjects

In this report, 'subjects' is used as shorthand for the academic areas of study at the focus of a DE. It is used without preference to include discipline.

SWS

Students' written submissions

UUK

Universities UK