Outcomes from institutional audit published papers

Programme monitoring arrangements
Second series

QAA 202 03/08

Summary

Consideration of the 59 institutional audit reports published between December 2004 and August 2006 indicates that, on balance, institutions had processes for monitoring academic programmes that provided an effective oversight of the quality of learning opportunities they provided and the academic standards of their awards. This supports the claim made by many institutions in their self-evaluation documents that programme monitoring was integral to their ability to monitor quality and standards.

Features of good practice relating to programme monitoring arrangements were identified in nine audit reports. They included: the effectiveness of the design and operation of monitoring at programme or field level; the ability to disseminate good practice or contribute to staff development; the approach taken to managing change; the inclusion of students on placements and professional training in programme monitoring; the use of professional advisers; and the generation and use of statistical data.

Recommendations concerning monitoring were made in 27 of the audit reports. A quarter of these concerned some aspect of variability or inconsistency of practice. Examples included: the information used to inform monitoring; the level of analysis applied to such information; the degree of reflection in monitoring reports; adherence to institutional policy with respect to content and/or timing; the ability to follow up matters identified in the monitoring reports for action or consideration; and the ability to disseminate good practice. The audit reports also made recommendations in other areas, including: ensuring that individual programmes were not obscured in reporting procedures; ensuring that all provision was covered by monitoring; obtaining and using student feedback consistently; and ensuring feedback from the institutional level back to the local level.

On the whole, the audit reports indicated that institutions provided clear guidance on the monitoring process. Some, however, indicated that the guidance provided may have been insufficient, or that there was a wide degree of latitude for interpretation at the local level. Some institutions used a standard form to reduce variability, while a number of others used staff independent of the programme to ensure consistency. A minority of reports identified institutions that actively involved students in the monitoring process.

It is clear from the audit reports that institutions kept their monitoring arrangements under review, so much so that almost half of the reports indicated that the system in place at the time of audit was new. There is evidence from the reports that some institutions saw monitoring as a key component of a developing approach to enhancement, although success in these endeavours at this time would appear to have been mixed. Some reports indicated that the introduction of a new monitoring process had led to confusion among staff which, in turn, contributed to variability of practice.

Very few audit reports indicated that the statistical data available for programme monitoring was inadequate, which would suggest that student information systems were improving. A number of institutions provided datasets centrally, which encouraged consistency, while some also provided statistical analysis for use by programme teams. However, some reports indicated that the information available was not always used to the best effect in programme monitoring.

Preface

An objective of institutional audit is 'to contribute, in conjunction with other mechanisms, to the promotion and enhancement of high quality in teaching and learning'. To provide institutions and other stakeholders with access to timely information on the findings of its institutional audits, the Quality Assurance Agency for Higher Education (QAA) produces short working papers that describe features of good practice and summarise recommendations from the audit reports. Since 2005 these have been published under the generic title Outcomes from institutional audit (hereafter, Outcomes...). The first series of these papers drew on the findings of the audit reports published between 2003 and November 2004. This paper is based on the findings of the institutional audit reports published between December 2004 and August 2006. It includes a brief section at the end of the paper comparing its key features with those of its predecessor in the first series of Outcomes... papers.

A feature of good practice in institutional audit is considered to be a process, a practice, or a way of handling matters which, in the context of the particular institution, is improving, or leading to the improvement of, the management of quality and/or academic standards, and learning and teaching. Outcomes... papers are intended to provide readers with pointers to where features of good practice relating to particular topics can be located in the published audit reports. Each Outcomes... paper therefore identifies the features of good practice in individual reports associated with the particular topic and their location in the main report. Although all features of good practice are listed, in the interests of brevity not all are discussed in this paper. In the initial listing in paragraph 5, the first reference is to the numbered or bulleted lists of features of good practice at the end of each institutional audit report, the second to the relevant paragraphs in Section 2 of the main report. Throughout the body of this paper, references to features of good practice in the institutional audit reports give the institution's name and the paragraph number from Section 2 of the main report.

It should be emphasised that the features of good practice mentioned in this paper should be considered in their proper institutional context, and that each is perhaps best viewed as a stimulus to reflection and further development rather than as a model for emulation. A note on the topics identified for this second series of Outcomes... papers can be found at Appendix 3 (page 21).

As noted above, this second series of Outcomes... papers is based on the 59 institutional audit reports published by August 2006, and the titles of papers are in most cases the same as their counterparts in the first series of Outcomes.... Like the first series of Outcomes... papers, those in the second series are perhaps best seen as 'work in progress'. Although QAA retains copyright in the contents of the Outcomes... papers, they can be freely downloaded from the QAA website and cited, with acknowledgement.

Introduction and general overview

1 This paper is based on a review of the outcomes of the 59 institutional audit reports published between December 2004 and August 2006 (see Appendix 1, page 17). A note on the methodology used to produce this and other papers in this second Outcomes... series can be found at Appendix 4 (page 23).

2 The Handbook for institutional audit: England (2002) described the process used for all the institutional audits under consideration in this paper. One of the three main areas to be investigated during the audit process was 'the effectiveness of an institution's internal quality assurance structures and mechanisms', taking into account QAA's Code of practice for the assurance of academic quality and standards in higher education (the Code of practice). Audits were also to investigate 'the way in which the quality of the institution's programmes and the standards of its awards are regularly reviewed and the resulting recommendations implemented' (Handbook, paragraph 11, page 3).

3 In 2000, QAA published advice on the conduct of programme monitoring in the Code of practice, Section 7: Programme approval, monitoring and review. This edition was current when all the audits covered by this paper took place. All references in this paper are to this edition, although a revised version was published in September 2006. Section 7 of the Code of practice acknowledged the relationship between monitoring and review in its advice that: 'Institutions should consider the appropriate balance between regular monitoring and periodic review of programmes'. It went on to state that: 'Monitoring should consider the effectiveness of the programme in achieving its stated aims, and the success of students in attaining the intended learning outcomes'. Monitoring was characterised as a process likely to be undertaken at the local level, while periodic review was described as normally being an institutional process.

4 Programme approval, monitoring and review almost invariably forms a section in the institutional audit reports. In the guidance to audit teams, auditors were asked to evaluate: the extent to which programme monitoring arrangements were appropriate to the institution's needs; how they were monitored; the extent to which loops were closed; how far arrangements were aligned with the Code of practice; how far consistency was maintained across schools or faculties; and the extent to which institutional oversight was maintained. The sub-section of the audit reports usually entitled 'Annual monitoring' has therefore provided a key source to inform this Outcomes... paper, as did the findings of the discipline audit trails.

Features of good practice

5 The 59 institutional audit reports published between December 2004 and August 2006 identified a number of features of good practice explicitly linked to programme monitoring arrangements. Where the link between the feature of good practice and programme monitoring is not immediately clear, some explanatory text has been included in square brackets. The features of good practice are as follows:

the well planned, resourced and supported placement system which adds significantly to the quality of the student experience within the University [in particular, the process for monitoring undergraduates' experience of training and placements, which included a survey of students and a survey of tutors to monitor adherence to University regulations and guidelines] [University of Surrey, paragraph 221, bullet 2; paragraphs 41-2, 62, 77-9, 113-4, 150 and 160]
the systematic approach, incorporating effective consultation, piloting, evaluation, project management and institutional oversight adopted for the introduction of strategic University developments [in particular, subject staff demonstrated 'strong ownership of, and commitment to, the processes of annual subject monitoring both in relation to continual improvement and the dissemination of best practice'] [University of Ulster, paragraph 215 ii; paragraph 47]
the joint venture with Kingston University which allows transfer of good practice in quality assurance through joint membership of committees and shared procedures and practice [in particular, the institution's positive use of annual monitoring to highlight good practice, its attempts to refine the monitoring process in the light of experience, and its plans for the development of the process] [St George's Hospital Medical School, paragraph 188 i; paragraphs 33, 35, 38, 43, 85 and 158]
the comprehensive design and thorough operation of the annual field review process [University of Gloucestershire, paragraph 316 iii; paragraph 75]
the robustness and apparent effectiveness of the design it has adopted for annual evaluation of pathways and programmes... [University of Huddersfield, paragraph 317 ii; paragraphs 61 and 70]
the implementation of a unitary model linking all levels of the University in a common quality management structure based on and promoting a close working partnership between academic and support staff [in particular, one element of this was the use of external 'rapporteurs', in the consideration of annual monitoring reports] [Staffordshire University, paragraph 251 ii; paragraphs 44 and 61]
the role and use of professional advisers, and links with employers and professional bodies [in particular, the institution's intention to include reports from professional advisers in annual monitoring reports in support of graduate employability] [Harper Adams University College, paragraph 189 iii; paragraphs 36, 56 and 61]
the effective links between programme monitoring reports and the identification of staff development needs [Heythrop College, paragraph 200 ii; paragraph 102]
the ability to generate and process reliable data relating to retention, progression and achievement for all students and the guidance provided for staff on the interpretation of those data [University of Bolton, paragraph 196, bullet 4; paragraphs 46 and 47].

6 Although not explicitly citing features of good practice in programme monitoring among their findings, a number of audit reports concluded that programme monitoring was effective and contributed to a clear overview of quality and standards. Among the reports offering such conclusions were:

Royal Holloway and Bedford New College [paragraphs 56 and 57]
Sheffield Hallam University [paragraph 44]
University of Northumbria at Newcastle [paragraph 50]
University College, Winchester [paragraphs 56 and 57]
University of Central England in Birmingham [paragraph 47].

Themes

7 Consideration of the features of good practice and recommendations in the audit reports relating to programme monitoring arrangements suggests that the following broad themes merit further discussion:

focus, scope and frequency of monitoring

- monitoring of postgraduate programmes

- monitoring of placements

- monitoring of collaborative provision

- monitoring of support services

- frequency of monitoring
design and operation of the monitoring process

- staff awareness and use of monitoring procedures

- monitoring reports: analysis and reflection

- sources that inform monitoring

- statistical data

- student involvement

- independent monitoring and internal audit

- the monitoring timetable

- follow up

- enhancement and the dissemination of good practice

reviewing the monitoring process.

Focus, scope and frequency of monitoring

8 Most of the audit reports indicated that the taught programme (or equivalent) was the focus of institutional monitoring processes. In one institution where the importance of programme monitoring had recently been reaffirmed, the report found that monitoring continued to operate at the field level and commented on the difficulty of identifying how monitoring worked at the programme level. The report went on to say that when taken together with changes to validation procedures and to the curriculum, the intention to present reports to the senior committee in groups of cognate fields could obscure particular programmes. It recommended that the institution consider 'whether annual course monitoring should be more focused on individual programmes' rather than on fields. (For another example of monitoring at the field level, see paragraph 17).

9 In two instances, not all undergraduate provision fell within the scope of the institution's programme monitoring process. These two audit reports made recommendations that such provision should be covered. One noted that there was no separate annual monitoring of joint programmes, and recommended that the institution introduce such monitoring and build on existing good practice to support the equitable treatment of students. The other identified a professionally accredited programme that, although subject to the procedures of the professional body concerned, was not encompassed by the institution's normal programme monitoring processes. It recommended that such provision be brought within the normal monitoring arrangements.

Monitoring of postgraduate programmes

10 Where taught postgraduate programmes were mentioned explicitly, the audit reports generally indicated that these were subject to broadly the same monitoring process as taught undergraduate programmes, though usually to a different timetable to accommodate the different pattern of delivery. However, in an institution where all postgraduate activity fell under the responsibility of its graduate schools, the report noted that taught postgraduate programmes had a completely separate monitoring system from that applied to undergraduate provision (see also paragraph 15).

11 Few audit reports explicitly mentioned the monitoring of postgraduate research provision. Where the monitoring of such provision was mentioned, all but one of the reports confirmed that annual monitoring took place. In the one exception, the report recommended that the institution concerned 'review the process of monitoring the progress of postgraduate research students individually and collectively'. Although a procedure was in place whereby both student and supervisor evaluated progress on a single form, its application was found to be incomplete. The report noted that the rate of return of progress reports varied among faculties, with the rate for one faculty being only 60 per cent. In addition, it questioned the suitability of an open form as a vehicle for students to raise concerns about their supervision.

Monitoring of placements

12 The explicit inclusion of placement students within the scope of programme monitoring was specifically noted in two audit reports. One of them identified this as a feature of good practice. It highlighted the institution's annual survey of undergraduates' experience of professional training and placement learning, complemented by a survey of tutors to monitor adherence to institutional guidelines; analyses of both were fed back to schools and reported on at institutional level [University of Surrey, paragraph 41].

Monitoring of collaborative provision

13 The audit reports indicated that monitoring of collaborative provision usually followed the same process as that for on-campus provision, sometimes with a few modifications to accommodate the additional complexity and risk involved, or to deal with cases where there was no 'home department' that would normally manage the monitoring of its own collaborative provision. One report expressed concern that, while the integration of collaborative provision into programme reports was logical, it had the potential to make it difficult to identify problems with a particular location or partner. The report therefore recommended that the institution ensure greater delineation of collaborative provision in its annual monitoring through reporting on such arrangements to an Annual Monitoring Event. This was in order to disseminate good practice and to identify and address common problems. Another report, for an institution with extensive collaborative provision overseas, noted that the institution had a separate process for the review of its overseas collaborative partners. The monitoring of such provision is more fully addressed in the Outcomes... paper on collaborative provision in this series.

Monitoring of support areas

14 While a number of audit reports commented that institutions monitored support services indirectly through student feedback, two noted that the programme monitoring process was augmented by reports produced by the support units themselves. One observed that the annual reports of all 'student facing' services were 'well formulated and identify and produce action plans for issues which need to be addressed'.

Frequency of monitoring

15 Overwhelmingly the audit reports noted that monitoring of undergraduate provision was undertaken annually. One institution was in the first year of annual monitoring at the time of audit, having just moved from a biennial process. Another, which operated a full annual review of each pathway, was in the first year of moving to triennial full review at the time of audit, with a 'lighter touch' review in intervening years in order to reduce the routine burden and provide greater time for in-depth analysis. Where the monitoring of taught postgraduate provision is discussed in the reports, this was also found to be conducted on an annual basis, except in the case of the institution mentioned in paragraph 10, where such provision was subject to a separate (biennial) process managed by the graduate schools.

Design and operation of the monitoring process

16 The audit reports revealed that programme monitoring processes in many institutions were broadly similar in design, in that they drew on a number of information sources to produce a report, with a view to identifying clearly any matters requiring action, as well as good practice to disseminate. These processes often comprised four or five stages, as monitoring proceeded from the module or unit, to the programme, through the department and/or faculty/school to the institutional level. In most cases, institutional oversight entailed the ability of a key institutional-level committee, the senior committee of the deliberative structure, to be adequately informed so that it could take appropriate action when necessary, and could identify good practice and ensure that it was shared appropriately. Two audit reports indicated that an annual report was also sent to the governing body of the institution, following consideration by the deliberative processes.

17 Most of the audit reports indicated that programme monitoring was largely effective, notwithstanding the room for improvement identified in almost all of them. Two reports cited the overall working of annual monitoring as a feature of good practice. In one case, the report noted the comprehensive features of annual monitoring, which included student involvement and independent scrutiny, as well as the thoroughness with which the process was conducted. It concluded that '...the design and implementation of the University's annual field review process constitute a feature of good practice' [University of Gloucestershire, paragraph 75]. The other report identified the design adopted by the institution for the annual evaluation of pathways and programmes as 'thorough and effective' and a feature of good practice, noting the time and resources that schools devoted to considering monitoring reports and the detailed feedback and advice offered where matters had not been dealt with effectively [University of Huddersfield, paragraph 61]. Other reports, while not specifically identifying good practice in programme monitoring, were complimentary about the overall operation of the process and its contribution to the management of quality and standards (as indicated in paragraph 6).

18 A few audit reports, however, noted a degree of systemic variability (which applied more widely than to programme monitoring) in some institutions' arrangements. Typically, this resulted from the devolution of responsibilities to the local level in a way that allowed considerable scope for interpretation and variation. For example, one report, while finding the institution's framework for managing quality and standards to be robust, identified variability of practice at school level in areas ranging from degree classification to peer observation. It recommended the establishment of 'more clearly defined methods to monitor variability with respect to the operation of quality assurance processes and the assessment of students across the [institution]'. Elsewhere, the report also identified some variability in the quality and detail of monitoring reports and welcomed the institution's intention to use standard forms and centrally provided datasets in order to reduce this.

19 Nearly half the audit reports made recommendations that specifically addressed programme monitoring. One report noted that, although monitoring at discipline level was found to be effective, it would be enhanced by the institution's plan to specify the minimum requirements and clarify responsibilities for quality management at departmental level. Consequently, the report recommended that the institution complete 'the regularisation of annual monitoring as expeditiously as possible, taking any steps necessary to ensure that it is implemented in a systematic and consistent way, and that procedures are in place to identify and act upon any consistent themes which emerge'. Another report identified examples of the non-submission of monitoring reports, the very late submission of reports, or the lack of scrutiny by the appropriate school committee. It recommended that the institution 'review the nature, consistency, and timeliness of the annual review process to ensure that it contributes fully to the [institution's] framework for quality management and assurance'.

20 In a 'single faculty' institution, where the monitoring of programmes was left to the discretion of departments, the audit report found a 'wide variation in the formality and comprehensiveness of activities at department level'. Not all the responses to this latitude were found to be 'fully fit for purpose'. Moreover, it was unclear how the Academic Board discharged its responsibility for academic standards, given that it did not receive reports on the results of programme monitoring. In this case, the audit report recommended that the institution 'reconsider its annual monitoring policy with the aim of producing consistent and appropriate arrangements that add value to programmes and provide effective assurance at [institutional] level regarding the quality and standards of its awards'.

Staff awareness and use of monitoring procedures

21 The audit reports commented on the extent to which staff were aware and made use of quality management principles and processes, including programme monitoring. One report noted unequivocally that 'staff whom the audit team met had a clear understanding of the [annual monitoring] process and the importance attached to it by the [institution]. The team found evidence that course teams and schools are engaging with the process, and that the process was being effectively monitored and led...'. According to the report, several factors contributed to this finding, but staff development was central. All course leaders had participated in a leadership programme focusing on annual monitoring. One outcome of this activity was the development of institutional guidelines for preparing annual monitoring reports, which were incorporated into the quality handbook.

22 A number of other audit reports, however, noted that the degree of staff awareness and usage contributed to variability of practice. It is not unusual to encounter references in the reports to different degrees of staff awareness at the local level, which could be exacerbated by recent change. One report found that staff were unfamiliar with an annual monitoring process that had not completed its first cycle. It encouraged the institution 'in its stated intention to review the programme monitoring and review system following its first year of operation, with particular reference to its embedding at both departmental and central levels'. Another report, for an institution where the recent change was more far-reaching, found that the multiple titles used as the heading for the standard review form were indicative of the confusion encountered at the local level during the audit. The report attributed much of this confusion to the recent transition from schools to faculties and the resulting changes in nomenclature, which had yet to become standardised across the institution. For a further institution, where only one full cycle of annual monitoring had been completed, the audit report stated that '...minutes read by the team indicated that this new process is taking time to become fully embedded'. The report also noted some variability in the information provided for annual monitoring, as well as in the depth of reflection on issues arising. In a final example, where the institution involved was refining its process to address concerns identified by a previous quality audit, the report found that staff across the institution were using a varied nomenclature and had different understandings of the process and its purpose. This led to the recommendation that the institution ensure that all members of staff share the same understanding of the purpose of programme monitoring reports and produce those reports in a timely fashion.

Monitoring reports: analysis and reflection

23 The audit reports frequently commented on the monitoring reports seen during the course of the audit. Many cited examples where the level of analysis and reflection was good, but very few found this to be consistently the case. One report, however, found evidence of more pervasive reflection and praised 'the effectiveness of the way in which the [institution] uses self-reflection to inform its development'. Specifically, it recorded that the audit team encountered many examples of initiatives being encouraged and adopted, and 'noted the enthusiasm with which enhancement issues were being addressed throughout the [institution], but particularly at school and faculty level'. The report also noted how a substantial improvement of both institution-wide and local processes had resulted from self-reflection at institutional level during the process of introducing a new system for internal review.

24 Several audit reports made recommendations about the level of analysis and reflection found in programme monitoring arrangements. One of them noted that omissions in reporting, coupled with the increasingly synoptic nature of the annual reports, compounded an impression that such reports were 'sometimes lacking in self-critical analysis' and not fully able to provide the institution with a sufficiently comprehensive picture of quality and academic standards across its provision. The report recommended that the institution introduce 'measures to secure a greater degree of critical analysis throughout its annual monitoring process, and more consistency in the annual monitoring reports from schools'. The rationale offered was that 'this would increase the [institution's] capacity for synthesis of key matters and thereby ensure that annual monitoring makes a more effective contribution to strategic decision-making and institution-wide implementation'.

25 Another audit report noted that, despite the introduction of a requirement for school monitoring reports to be submitted in a standard form, they varied significantly across the institution in terms of their layout, detail and degree of critical analysis. It saw the need for 'further progress in terms of achieving greater comparability of presentation and a more consistent degree of critical analysis' in internal review reports, which would 'allow the Academic Board...to take an appropriate overview of the [institution's] academic objectives and the contribution of each school to their achievement'. A further report recommended that the institution 'draw on existing good practice in departments to achieve consistency in the extent of the analysis and areas covered in the reports from departments', noting that a more evaluative approach would also contribute to the quality assurance and enhancement of the institution's provision.

Sources that inform monitoring

26 The guidance accompanying Precept 7 of Section 7 of the Code of practice noted that routine monitoring activity 'may consider, for example: external examiners' reports; any reports from accrediting or other external bodies; staff and student feedback; feedback from former students and their employers; [and] student progress information'.

27 The audit reports generally confirmed that external examiners' reports and the outcomes of student feedback were routinely used in programme monitoring. Student feedback might be drawn from a variety of sources including module questionnaires, programme questionnaires, the minutes of staff-student liaison meetings and, in one case, a survey of graduates. A small number of reports identified areas for improvement in the use of student feedback in programme monitoring processes. For example, one noted that while some modules had been evaluated effectively, others had not been evaluated at all, which contravened the policy of the institution concerned. Consequently, it recommended that consideration be given by the institution to 'ways of ensuring that its intention, that all modules are evaluated, is fulfilled'. Another report noted that module evaluations were confidential to the teacher concerned, although staff were encouraged to share the outcomes of such evaluations with colleagues, and relevant academic managers could ask to see them. It recommended that 'all module evaluations are disclosed, critically analysed and incorporated effectively into the annual monitoring process', to secure greater transparency and increase the opportunities for sharing good practice. Student representation and feedback is the focus of another paper in this series.

28 A number of audit reports referred explicitly to the use of reports from external bodies, typically accreditation reports from professional, statutory and regulatory bodies, as an additional source to inform programme monitoring. Several audit reports also identified programme monitoring processes that were informed by feedback from employers: in one case through industrial liaison committees, and in others by using whatever employer feedback was available. A further report identified as a feature of good practice the institution's links with employers and professional advisers, one aspect of which was a planned annual report from such advisers in support of graduate employability, to be included in annual monitoring [Harper Adams University College, paragraph 56].

Statistical data

29 The majority of the comments made in the audit reports about the information on which programme monitoring drew related to student progression and achievement data. Some of these comments concerned the availability and appropriateness of the data, while others related to its use. Several reports specifically mentioned the use of graduate employment data in programme monitoring. Institutions' use of progression and completion statistics is the focus of another paper in this series.

30 Several audit reports recorded that student information systems had a limited ability to produce adequate, reliable data to inform programme monitoring. One noted that 'the statistics available...are limited by the constraints of the current student record system', which made it impossible to analyse cohorts. The report recognised that the nature of this institution's provision made the accumulation of statistical data particularly difficult, and that a new student records system was planned. However, it nonetheless recommended that the institution establish a strategy to make more effective use of statistical data in the evaluation of academic standards. Other reports, however, while noting past difficulties, indicated significant improvements resulting from the development of the existing student information systems or the acquisition of new ones. One report recorded that the institution 'has invested significantly in the development of [its student record system] and is now able to provide improved data to support annual monitoring'. The system could now identify trends over several years in progression statistics and in tabulations of students' final award classifications by entry qualifications.

31 Around a third of the audit reports stated that standard datasets for programme monitoring were provided centrally by the institution. One report noted that the provision of standard datasets had enabled statistical information to be included in the annual review process. A few institutions also provided some analysis to accompany the raw data. In one instance, the relevant report noted that the production and processing of data based on both leaving and starting cohorts allowed the performance of part-time students to be monitored in a well-founded way. It went on to identify the provision and analysis of data as a feature of good practice, describing the flexibility and scope of the analysis that could be provided at the request of a department [University of Bolton, paragraphs 46 and 47].

32 Some audit reports found that the availability of standard datasets, even when analysis was provided centrally, did not necessarily mean that the data was used. One report encouraged the institution concerned 'to undertake further work on the use of data...', including the analysis of admission, progression and retention statistics. Another report found examples of high first-year withdrawal and/or failure rates that had been subject to little analysis in annual monitoring. It went on to suggest that the institution 'further consider how data on student achievement at module level may be incorporated' into annual monitoring, 'with the potential that modules causing particular concerns could be highlighted...'. The same report included a wider recommendation to 'develop strategies to ensure that management information systems are fully exploited in order to achieve oversight of issues related to the management of quality and standards at institutional level'.

Student involvement

33 More than 10 per cent of the audit reports described how students were actively involved in the programme monitoring process; reference has already been made to the involvement of students on placement and professional training schemes in programme monitoring (see paragraph 12). One report noted that the outcomes of annual monitoring of modules were taken to staff-student consultative committee meetings, so that students were involved in the final evaluation. Some reports indicated that students were represented on the committees which evaluated monitoring reports at some stage in the process. Two reports identified where institutional procedures mandated student involvement, but were unable to confirm that this happened in practice. In one of these instances, while action planning and action review linked to annual monitoring were clearly effective, it was less clear that the process involved feedback to students or their representatives as required. The report noted that students recognised their involvement in quality assurance processes as being generally low. It recommended the institution to 'ensure that the development and implementation of mechanisms for the collection, analysis and use of student feedback is addressed...'.

Independent monitoring and internal audit

34 As noted above (see paragraph 27), the audit reports showed that external examiners' views were routinely incorporated in programme monitoring. Other external sources commonly used were reports from professional, statutory and regulatory bodies and input from employers. One audit report described how an institution had appointed an external panel member, as well as an independent member of staff from within the institution, to a faculty panel convened to consider annual reports. Many others recorded ways in which institutions used internal staff, independent of the programme in question, as a means to ensure consistency in monitoring and to secure the wider dissemination of good practice. One report noted the use of two 'rapporteurs' from another faculty, who had been trained to identify strengths, weaknesses and good practice; they produced summary reports highlighting the issues for consideration by the senior committee at institutional level. It concluded that the annual monitoring of taught provision, including the role played by rapporteurs, was 'a commendable example of the aspects of good practice identified [elsewhere in the report] in the implementation of a unitary model linking all levels of the University in a common management structure' [Staffordshire University, paragraph 61].

35 A variety of other models of independent scrutiny were also evident from the audit reports. For example, one report recorded an institution's use of 'cross-readers', normally course leaders from another school, to read annual monitoring reports with a critical eye and to help in sharing good practice. As well as attending the course board that discussed annual monitoring reports, cross-readers attended a relevant school board, to which they submitted a report. Some institutions conducted internal audits of programme monitoring, over and above the use of independent monitors, either as a means of reassuring themselves that the process was working at the local level, or in order to balance rigour and effort. For example, one report described a system of random audit in which small teams scrutinised one unit from each school to ensure that the annual monitoring process complied with the institution's procedures.

36 Two audit reports exemplified different ways in which sampling was being applied in order to achieve a balance between effectiveness and workload. One recorded that a sample of programmes was scrutinised in detail as part of the upward reporting process. The other outlined how the institutional committee reviewed in detail a small number of areas where concerns or potential areas of good practice had been identified.

The monitoring timetable

37 Two audit reports commented specifically on the deadlines applied in monitoring procedures. One of them noted that annual monitoring was only partially embedded, which - when considered in conjunction with a schedule of comprehensive reviews about to be introduced at that time - risked placing considerable pressure on departments and the quality management framework, and raised concerns about the feasibility of delivery. It recommended that the institution consider 'requiring that its monitoring, reporting and action planning arrangements ensure that realistic deadlines are set and procedures completed in a timely manner'. The other report highlighted the problems inherent in an absence of deadlines, in an institution where the responsible committees received monitoring reports from programmes as and when they were submitted. It noted that this had inevitable consequences for the ability of monitoring committees to produce summative material. In some cases timeliness was not an issue confined to programme monitoring.

Follow up

38 Two audit reports highlighted concerns and made recommendations regarding the identification of clear points for action in monitoring arrangements. One noted that annual monitoring summaries did not routinely draw together the issues to be addressed and did not, therefore, constitute formal action plans. Consequently, it recommended that the institution consider 'how...it can best ensure that identified actions in [annual reports] are implemented'. The other report recommended that the institution establish 'a more explicit annual planning statement which sets and monitors operational priorities and targets'. It was considered that this would lead to a greater focus on reflection and analysis rather than on operational details, as well as eliminating the potential for the confusion of priorities identified elsewhere in the report.

39 Many audit reports indicated explicitly that monitoring reports and action plans from one year fed into monitoring in the following year. In one case, however, it was noted that a new procedure for monitoring the fulfilment of points for action was taking time to become embedded. The report also noted that the annual report template lacked a section for comment on action taken from the previous year's report.

40 With the proviso that only one cycle of the institution's handling of departmental reports had been seen, another audit report found limited evidence of the institutional committee reporting back to the departmental committee, or setting deadlines for addressing matters raised in programme monitoring. Instances were identified where departmental committees had not updated programme specifications or had not completed signing-off procedures on material that was subsequently made publicly available. This led to a recommendation to expedite a review of the effectiveness of the departmental committees.

Enhancement and the dissemination of good practice

41 A few audit reports commented positively on the dissemination of good practice through the programme monitoring process, with some of them linking this to enhancement. One report indicated that the institution concerned had moved to position annual monitoring as an enhancement-led process. In support of this, a standard form had been introduced to identify good practice and areas for development and to highlight issues for consideration and action at institutional level. Departmental annual reports were used at institutional level 'to identify good practice for wider dissemination, areas for development and recommendations for actions to improve procedures'.

42 'Enhancement' appeared in several other audit reports as a term used by the institution, introduced by the audit team, or both. In an example cited in paragraph 22, the institution's capacity for self-reflection was noted positively. The report commented on 'the enthusiasm with which enhancement issues were being addressed throughout the [institution]...'. The same report, while noting some areas for improvement, described the annual monitoring process as 'very well designed and thorough with effective mechanisms for identification of both weakness and innovation...'.

43 In several instances, audit reports noted that programme monitoring arrangements worked well at a local level, but with limited scope for disseminating good practice, thereby impairing institutions' abilities to realise their aspirations for enhancement. One such noted that: 'Annual review was described to the audit team as focussing on enhancement but, while it does encourage programme directors to reflect critically on major aspects of teaching provision, the team did not regard quality enhancement as central to the process', because of the lack of any institutional oversight of annual review action plans. Another report noted that 'the mechanism for annual oversight of programmes at faculty level was well designed, taken seriously and operating in an effective manner'. However, the same report encouraged the institution to 'consider ways in which good practice identified in the annual reports of course directors at faculty level, can be noted at institutional level to facilitate dissemination of good practice across the faculties'. A further report indicated that the annual monitoring reports, while good in themselves, were not being used to best advantage, across schools in particular. It recommended that the institution 'make more effective use of the annual review process and develop further its formal systems for the dissemination of good practice across the institution'.

Reviewing the monitoring process

44 Section 7 of the Code of practice advised that: 'Institutions should evaluate the effectiveness of programme approval, monitoring and review practices' [Precept 9]. Almost all the audit reports gave an indication, in one way or another, that institutions kept their monitoring processes under review.

45 One manifestation of this is that almost half of the 59 audit reports indicated that the monitoring system in place at the time of audit was new, sometimes in its first year of operation. While this made it difficult to assess fully the effectiveness of the new process, it demonstrated the institution's ability to review its processes. For example, one report noted that 'it was impossible as yet to assess the effectiveness of the system by reference to its outcomes', as the first cycle of a comprehensively changed system had not been completed at the time of audit. However, the report recorded the team's view that the new procedure was likely to be effective, and stated that: '[t]he introduction of the changes to annual monitoring is an example of the [institution's] ability to evaluate itself and to introduce improvement'. Another report, in supporting its identification of good practice, noted that 'the [institution] had adopted a careful and evolutionary approach to the development of annual monitoring...'. The report went on to identify 'this approach to the review and development of institutional processes, involving as it did effective consultation, piloting, project management, evaluation and institutional level oversight, to be highly effective' [University of Ulster, paragraph 47]. This systematic methodology was found to underpin many of the features of good practice identified in the report.

**The findings of this paper compared with its counterpart in the first series of Outcomes... papers**

46 The findings of this paper align well with those of its companion in Series 1. In general, programme monitoring continued to work well in providing institutions with an effective oversight of the quality of learning opportunities they provided and the academic standards of their awards. There is clear evidence that institutions were continuing to refine their processes for programme monitoring.

47 Proportionally fewer reports (15 per cent as opposed to 19 per cent) cited features of good practice linked to programme monitoring. Similarly, a smaller proportion (46 per cent as opposed to 57 per cent) made recommendations with respect to these arrangements.

48 Inconsistent and variable practice continued to be an issue for many institutions. The reports reflected the varying degrees of concern found and the variety of causes of this variability; they appear to be the same across both papers.

49 There is evidence to suggest that student record systems were better able to provide the data required for an effective programme monitoring process, although the effective use of that data continued to be variable.

Conclusions

50 Analysis of the 59 institutional audit reports published between December 2004 and August 2006 shows that, in general, institutions' arrangements for programme monitoring were working effectively. The time and resources devoted to these processes testified to the institutions' conviction that they represented an essential component in their ability to oversee standards and the quality of learning opportunities.

51 The reports indicated that Section 7 of the Code of practice had informed the development of programme monitoring processes, which were regularly reviewed.

52 Inconsistency and variability of practice continued to give rise to varying degrees of concern. Inconsistent practice was reported in areas such as: the information used to support monitoring; the level of analysis applied to such information; the degree of reflection in reports; adherence to institutional policy with respect to content and/or timing; the ability to follow up identified issues; and the ability to disseminate good practice.

53 There was evidence to suggest that some student record systems were improving and were better able to supply data in support of programme monitoring. More institutions had moved to the provision of standard datasets, sometimes accompanied by an analysis of the data.

54 The reports indicated that some institutions had begun to develop their approaches to quality enhancement, the success of which could depend on institutional commitment, the degree of institutional oversight and consistent practice at local level.

ISBN 978 1 84482 764 0