Outcomes from institutional audit

Programme monitoring arrangements

QAA 096 01/06

PDF version (149KB)

Summary

The published audit reports show that, on balance, institutions have developed systems for programme monitoring which provide an effective overview of quality and standards. In a number of institutions, however, monitoring schemes had been introduced or revised too recently for any judgement to be made of their effectiveness.

Features of good practice are cited in 13 reports. Strengths identified include the overall rigour of monitoring systems; their reflectiveness; their capacity to identify and address issues; their dissemination of good practice and quality enhancement; and their use of external input.

Recommendations are found in 40 reports. These reveal concerns about the institution's overview of quality, standards and the procedures for their assurance; variability in the content and quality of monitoring reports; the coordination and clarity of monitoring responsibilities; arrangements for action planning, monitoring and feedback; use of statistical information; staff and student involvement; monitoring schedules; and review of monitoring processes.

In general, institutions provide clear and full guidance on the requirements of the monitoring process and, specifically, on the content and purpose of monitoring reports. In some cases, however, the guidance supplied by the institution may not be explicit enough to ensure that the scope of the information it receives by way of reports is either consistent or adequate for its monitoring purposes. There is comment in some reports on the process of 'filtering' monitoring information at school or departmental level. This may have the effect of obscuring matters about which the centre may need to know in meeting its responsibility for oversight.

Many institutions have established monitoring systems that facilitate both reflection and action at the appropriate level, and clear central oversight of quality and standards. In some cases, the annual monitoring sequence culminates in an institutional event with a focus on quality enhancement. Several reports note central initiatives in the dissemination of good practice identified through annual monitoring. Reports also provide evidence of improvements in monitoring processes stimulated by review.

Other reports, however, observed a lack of clarity in action planning procedures and a lack of coordination between levels and also cycles. This might be associated with difficulties in maintaining the reporting schedule. In some institutions, feedback on monitoring from central bodies to departments and other interested parties such as students and external examiners was found not to be timely or not apparent.

The involvement of both students and staff in annual monitoring processes is variable. In some cases, programme monitoring may lack an effective student perspective because student feedback is confined to modules or departments. The degree of staff engagement with monitoring procedures may be influenced by such factors as the extent of participation asked for and the experience of central feedback.

A number of reports suggest that the use of statistical data requires development. This may be due to a lack of or inadequacy in central data provision, or because only limited use is made of the data available for analysis. Other reports, however, identified effective use of statistical data. Work being undertaken by institutions to improve management information systems was widely noted.

Preface

An objective of institutional audit is 'to contribute, in conjunction with other mechanisms, to the promotion and enhancement of high quality in teaching and learning'. One of the ways in which this can be accomplished is through identifying features of good practice across the reports and areas where reports have commonly offered recommendations for improvement.

In due course, QAA intends to produce an extended reflection on institutional audit in the Learning from audit series, but since the final institutional audit reports in the present audit cycle will not be published until spring 2006, Learning from institutional audit is unlikely to be published before late 2006. To give institutions and other stakeholders more timely information, QAA has therefore decided to produce a series of short working papers, describing features of good practice and summarising recommendations from the audit reports, to be published under the generic title Outcomes from institutional audit (hereafter, Outcomes...).

A feature of good practice in institutional audit is considered to be a process, a practice, or a way of handling matters which, in the context of the particular institution, is improving, or leading to the improvement of, the management of quality and/or academic standards, and learning and teaching. Outcomes... papers are intended to provide readers with pointers to where features of good practice relating to particular topics can be located in the published audit reports. Each Outcomes... paper therefore identifies the features of good practice in individual reports associated with the particular topic and their location in the Main report. Although all features of good practice are listed, in the interests of brevity not all are discussed in this paper. In the initial listing in paragraph 4, the first reference is to the numbered or bulleted lists of features of good practice at the end of each institutional audit report, the second to the relevant paragraphs in Section 2 of the Main report. Throughout the body of this paper, references to features of good practice in the institutional audit reports give the institution's name and the paragraph number from Section 2 of the Main report.

It should be emphasised that the features of good practice mentioned in this paper should be considered in their proper institutional context, and that each is perhaps best viewed as a stimulus to reflection and further development rather than as a model for emulation. A note on the topics identified for the first series of Outcomes... papers, to be published throughout 2005, can be found at Appendix 3 (page 18).

This first series of Outcomes... papers is based on the 70 institutional audit reports published by the end of November 2004. The second series will draw on institutional audit reports published following the 2004-05 audits. It is likely that there will be some overlap in topics between the first and second series. Papers in each series are perhaps best seen as 'work in progress'. Although QAA retains copyright in the contents of the Outcomes... papers, they can be freely downloaded from QAA's website and cited, with acknowledgement.

Programme monitoring arrangements: introduction and general overview

1 This paper is based on a review of the outcomes of the first 70 institutional audits published by 5 November 2004 (see Appendix 1, page 15). A note on the methodology used to produce this and other papers in the Outcomes... series can be found in Appendix 4 (page 19).

2 The institutional audit process introduced in England in 2002-03 requires audit reports to comment on 'the effectiveness of an institution's internal quality assurance structures and mechanisms, in the light of QAA's Code of practice for the assurance of academic quality and standards in higher education...and the way in which the quality of its programmes and the standards of its awards are regularly reviewed and the resulting recommendations implemented'. The purpose is to provide 'public information on an institution's soundness as a provider of tertiary qualifications of national and international standing' [Handbook for institutional audit: England, paragraph 11]. Audit reports therefore summarise the means used by an institution for monitoring and reviewing its programmes and for reviewing, at set intervals, their fitness to enable students to progress towards the institution's awards. In addition, and drawing on inquiries through discipline audit trails, they comment on how internal monitoring (and review) work at the discipline level.

3 This paper draws on the reports of the 70 institutional audit reports published by November 2004. It focuses on programme monitoring since the periodic review of programmes is addressed in another Outcomes... paper and, taking its cue from the structure of the audit reports, it comments on the procedures followed by institutions and the use made of the advice represented by QAA's Code of practice and, particularly, Section 7: Programme approval, monitoring and review.

Features of good practice

4 Consideration of the published institutional audit reports shows the following features of good practice relating to annual monitoring:

• 'the range and depth of links with external agencies and their contribution to the professional currency of the School's educational provision' [The School of Pharmacy, University of London, paragraph 159 i and paragraph 54]
• the detailed care given to the annual monitoring of the BMus (Hons) [Royal College of Music, paragraph 183 ii, paragraph 83]
• the institution's annual monitoring procedures which make explicit use of student input [The Surrey Institute of Art & Design, University College, paragraph 147 iii, and paragraphs 29-31]
• the use of the annual monitoring process and the associated Academic Health Day for reflection on policy and practice [Newman College of Higher Education, paragraph 210 ii and paragraph 65]
• the progressive approach to annual monitoring of support areas including peer evaluation [Norwich School of Art and Design, paragraph 154 iii and paragraph 42]
• the effective and constructive evaluation by the Royal College of Nursing Institute of its work which contributes to enhancement of its provision [Royal College of Nursing Institute, paragraph 158 iii and paragraph 47]
• 'the College's self-reflective and open approach to monitoring and review processes which are proactive, forward-looking and robust' [Trinity College of Music, paragraph 153 ii and paragraph 40-45]
• the work of the Plenary Group of [Faculty Quality Assurance Team] Chairs in identifying opportunities for enhancement and matters for further consideration at taught programme level [University of Bristol, paragraph 288 i and paragraph 57]
• the detailed and analytical reports of boards of examiners, and their availability to students in following years [University of Oxford, paragraph 247 iii and 49]
• the quality, utility and accessibility of the University's Quality Assurance Handbook [University of Southampton, paragraph 189 ii and 42]
• the design and operation of the University's [Annual Academic Healthcheck] process, including the careful arrangements to ensure attention to the annual monitoring of provision with partner colleges, and the use made by the University and its partners of the Enhancement Day component of the process [University of Brighton, paragraph 255 ii and paragraph 38]
• the flexible approach to the integration of provision offered through the [Centre for Lifelong Learning] into the formal approval, monitoring and review processes [Coventry University, paragraph 273 i and paragraph 57]
• the holistic approach to reporting and planning through the [Annual Reporting and Planning Document], combining in a single process and a single document both the academic quality and standards, and human and financial resources aspects of schools' activities, thus providing the University with a valuable instrument for managing its current and future portfolio [University of Greenwich, paragraph 270 i and paragraph 54].

5 The number of features of good practice and recommendations in programme monitoring needs to be put in the wider context of the numerous reports. While these do not explicitly identify features of good practice, the reports state that programme monitoring was being effectively conducted and provided for a clear overview of quality and standards. Among the reports offering such conclusions were:

• University of Wolverhampton [paragraph 63]
• University of Salford [paragraph 31]
• Manchester Metropolitan University [paragraph 40]
• University of York [paragraph 42]
• St Martin's College, Lancaster [paragraph 50]
• University of Leeds [paragraph 40]
• University of Warwick [paragraph 52].

Themes

6 The published reports suggest the following themes in annual monitoring:

• Frameworks and guidance
  • institutional guidance
  • monitoring of postgraduate programmes
  • monitoring of support areas
  • monitoring of collaborative provision
  • monitoring in validated institutions
  • monitoring and periodic review
• The design and operation of programme monitoring
  • monitoring systems
  • module and programme monitoring
  • monitoring reports
  • statistical data
  • external input
  • student input
  • staff input
  • the monitoring sequence
  • action plans
  • dissemination of good practice
  • filtering of information feedback on reports
• Review of monitoring processes.

Frameworks and guidance

7 Precept 1 in Section 7 of the Code of practice advises institutions to 'ensure that their responsibilities for quality and standards are discharged effectively through their procedures for...the monitoring [and review] of programmes'. Precept 7 states, inter alia, that institutions should 'ensure that appropriate actions are taken to remedy any...shortcomings' identified in programme monitoring. Audit reports, which refer to the Code of practice in the context of monitoring procedures, confirm almost without exception that institutions are either informed by or intend to review their arrangements in light of the Code of practice.

Institutional guidance

8 Audit reports generally refer to the institutional guidance provided on monitoring via handbooks and templates, and list the range of matters for inclusion in the annual procedures. These typically include data on and analysis of student admissions, progression, completion and employment destinations; teaching, learning, assessment and curricular aspects; feedback from students, external examiners, external stakeholders, staff and programme committees; staff development; consideration of module reports; reflections on internal and external reviews; good practice; action plans and responses to previous action plans. One report noted the emphasis given in a checklist to the requirement for evidence from combined degree programmes.

9 In one instance, an institution was advised to provide more explicit guidance on the annual monitoring process 'in order to improve consistency and comprehensiveness and thereby to make [the process] a more effective channel for institutional oversight within the [institution's] framework for managing quality and standards'. On the other hand, another report found that the institutional quality assurance handbook provided detailed guidance on how the annual monitoring reports should be compiled and what they should contain, including responses to external reference points. The report noted the handbook's 'quality, utility and accessibility' constituted a feature of good practice [University of Southampton, paragraph 42]. The role of such guidance and its implications for institutional oversight are discussed below.

Monitoring of postgraduate programmes

10 In some audit reports, reference is made to arrangements for monitoring both taught and research postgraduate programmes. The evidence indicates that a number of institutions have annual monitoring procedures for postgraduate teaching programmes in place comparable to those for undergraduate programmes, though with adjustments to scheduling as necessary. Some reports also noted procedures for annual overview, at school and institutional level, of postgraduate research programmes and their outcomes.

11 One report welcomed an institution's stated intention to collate data for postgraduate research students to provide cohort analysis, 'since it will allow the [institution] to have oversight of postgraduate research progress and achievement across all research institutes'. One report advised an institution to extend annual monitoring to postgraduate research, to make possible 'scrutiny of the collective experience of research students', for monitoring of trends, and for identification of matters for departmental or faculty consideration.

Monitoring of support areas

12 Some reports noted the extension of annual monitoring to support areas and departments to provide, as one report put it, 'a holistic view of the student experience'. In one institution, the monitoring of support areas was informed by a template that focused on 'review and evaluation of developments, evaluation of user surveys, quality enhancement procedures and initiatives, and action planning to deliver further improvement'. The audit report identified this development as good practice 'that gave support staff a voice in the annual review process and extended the institution's overview of its activities' [Norwich School of Art and Design, 42].

Monitoring of collaborative provision

13 The evidence of audit reports is that institutions generally integrate annual monitoring of collaborative programmes with internal procedures. One report noted the enhanced institutional scrutiny given to reports on collaborative provision. Another noted the device of an 'annual quality visit' to the collaborating centre 'to review the performance of both partners and to provide an opportunity for any centre-wide issues to be discussed with staff and students of the centre and reported to the [institution]'. The effectiveness of one institution's integration of provision offered through its Centre for Lifelong Learning into annual monitoring processes, giving emphasis to the Centre's evolving role in delivering the [institution's] widening participation strategy, was identified as a feature of good practice [Coventry University, 57].

Monitoring where provision has been validated as suitable to lead to the awards of another institution

14 Generally, in such cases, the providing institution submits annual programme reports to the awarding university via internal quality committees which monitor responses to any action requirements. One report noted the awarding university's requirement for programme-level exchanges between the institutions on all monitoring reports submitted.

Monitoring and periodic review

15 Some reports alluded to the relationship between annual monitoring and periodic review and, in particular, the role of annual monitoring in providing information such as data sets for periodic review.

The design and operation of programme monitoring

Monitoring systems

16 Audit reports state that many institutional monitoring systems were found to be effective. In various instances, overall effectiveness also included particular features in need of review and amendment. Features perceived as having a key bearing on the overall usefulness of monitoring systems included the quality and quantity of the information received by the centre (normally a quality assurance committee) and whether or not the information supplied enabled the institution to achieve an effective overview of all its provision and to take any action required. Numerous reports intimated that the information sent up might be variable in content and quality through lack of specification from the centre and too much local flexibility. In these cases, reports were likely to contain associated recommendations.

17 Another feature that influenced general effectiveness in monitoring was the extent to which procedures that earlier had been conducted informally by institutions had been systematized. Examples of this are the prescription of annual monitoring schedules, or by institutional coordination of the various means used to monitor learning and teaching. A third feature influencing overall effectiveness was the degree of complexity of the monitoring system in place, and whether or not authority and responsibilities at the various levels were clear. Some reports noted that the monitoring system in use was too new, and the evidence too limited, for any evaluation of effectiveness to be made.

18 Reports which found monitoring systems effective tended to focus on the general strength of procedures and their input to the quality of the overview achieved by the institution. Thus, among a number, one audit report stated, 'The combination of upward reporting to, and auditing by, [the institutional quality committee] has resulted in a robust and effective system, and the annual monitoring process is successful in encouraging responses at appropriate levels to issues raised and in providing the [institution] with a full and accurate picture of the quality and standards of its taught provision'. Another report concluded that the annual monitoring system was 'thorough and robust and contributes effectively to the institution's overall management of quality and standards'. A further report stated that the system 'ensured a strong institutional overview of the outcomes of [annual monitoring]', with reinforcement through annual follow-up of issues and their outcomes [University of Bristol, 57].

19 The monitoring systems employed in several institutions were judged to represent good practice. One audit report identified a self-evaluative culture in which 'improvement originated from frank, regular and timely evaluation of performance, reviewed against relevant criteria', and in which 'monitoring and review activity was the key business of all committees in the quality management structure' [Royal College of Nursing Institute, 47]. Another report concluded that the annual monitoring and review process was 'generally robust and effective' and that 'it was a feature of good practice', as was its culmination in an annual 'enhancement day' at which key themes, such as student feedback and the Academic Infrastructure, were explored [University of Brighton, 38]. Similarly, another institution had established an '"academic health day" at which samples of reports from all stages of the academic audit process were considered by an extended Academic Standards Committee'. This provided an opportunity to reflect on both the outcomes and the operation of the process. The audit report noted that the initiative had the potential to 'provide for significant enhancement of the College's processes, particularly through the identification of examples of good practice and an opportunity for their dissemination' [Newman College, 65]. A further report concluded that the annual monitoring procedures of the institution concerned were robust and that faculty reports were 'evaluative, reflective and appropriately self-critical' [Surrey Institute of Art & Design, University College, 29-31]. Finally, one report identified good practice in an individual school where, in the course of two days, 'all programmes and constituent modules were reviewed using available qualitative and statistical data' in an event encompassing 'undergraduate, taught postgraduate and postgraduate research activity', and concluding with a synthesis of the school's work.

Module and programme monitoring

20 As a general rule annual monitoring of modules/units leads into that of programmes. Review of modules incorporates such elements as data on student progression and achievement, student feedback and external examiners' reports. Three audit reports noted, however, that there was no 'explicit consideration of the particular features of each programme' in a monitoring system based on units and departments, a problem exacerbated if programmes extended across departments. In another institution, annual monitoring tended to be focused on individual components relating to programme delivery rather than on individual programmes as a whole. The report noted that it was unclear how this approach 'would necessarily reveal problems encountered by particular cohorts of students on a particular programme'. In a further case, where undergraduate provision was subject only to module review, the report suggested the extension of the use of programme review to the undergraduate phase 'to generate a more holistic perspective on quality and standards'.

Monitoring reports

21 Audit reports address the various levels of review and report production in the annual monitoring sequence and the relationships between them. There is a particular focus in many audit reports on the earlier levels, that is, those of the module/unit and programme/course. A common issue, associated with a series of recommendations, is the quality of the information provided by annual monitoring reports, in particular those at programme level.

22 Many audit reports express concerns about the variability of departmental reports and specifically of their engagement in reflection and evaluation. Given the extent of variability in content and analysis, one institution was invited to 'define the acceptable level of such variability'. One report took the view that the monitoring system in place was 'monitoring adherence to process', rather than obtaining the information about the quality and standards of programmes needed by the institution. Two reports commented on the lack or inadequacy of input on units/modules to programme monitoring, and the difficulties arising from this for programme analysis. Two other reports found it a weakness in institutional monitoring systems that no report or self-evaluation was required at departmental level. In one case this was seen as a missed opportunity for dissemination of good practice; in the other of taking away the primary monitoring responsibility from the department to a 'largely external' committee. The practice of grouping large areas of subject/programme provision within a single annual monitoring report, with the risk that issues relating to discrete parts of this provision might thereby disappear, and action plans might be incomplete, was a concern of two reports. Some reports noted that despite provision of detailed guidance on the requirements for annual monitoring reports, significant variability remained in content and quality.

23 For the most part in the above cases a direct link is made with the institution's capacity to achieve an effective overview of its provision. Thus one report, finding 'considerable variability in the depth of reflection in reports', encouraged the institution concerned 'to continue to enhance annual monitoring through rigorous self-evaluation at programme level and through review at [institutional] level'. Another report noted that an institution's annual monitoring system allowed 'variation in local practice which might limit the [Senate's] capacity to assure itself of the equivalence of the quality of provision across the [University]'. Similarly, commenting on the 'variable' manner of presentation of monitoring reports, a further audit report noted that this 'might inhibit [the institution's] capacity to make comparisons across its provision'.

24 Variability was sometimes associated with what were described as 'decentralised' processes, which are run, for example, by faculties but monitored by the centre. One audit report, noting the great variety in school report templates, concluded, 'In view of the variety of methods in place for the regular monitoring of quality and standards at programme level and the informal and unregulated nature of some of those methods, the [institution] may wish to consider the benefit of addressing variations in the annual monitoring of programmes by building on existing good practice in the [institution]'. In another case, where the annual monitoring focus is at unit level, the audit report commented, 'The degree of flexibility afforded to unit organisers and schools in the selection of matters to feature in unit reviews has the potential to lead to inconsistencies and gaps in the [institution's] understanding of its teaching and learning provision, and is unlikely to meet [the institution's] requirements for consistent and timely data on its educational provision across all its schools'.

25 In a number of institutions, however, annual monitoring reports at the various levels were found to be consistently evaluative and analytical and to contribute effectively to robust annual monitoring processes [Surrey Institute of Art & Design, University College, 29-31]. In one report, particular note was made of the identification of good practice and of areas for enhancement [Coventry University, 57] and in another the 'generally proactive and forward-looking' nature of reports [Trinity College of Music, 40-45].

Statistical data

26 In 25 of the 70 published reports, the use of statistical data is mentioned as needing further development and there are several associated recommendations. Many reports note problems with data availability. One institution - in which difficulties had been experienced with the 'integrity, reliability and timeliness' of centrally provided data - was advised to 'continue to develop its management information systems to ensure that progression and completion data for all types and groups of students...[were] standardised to facilitate consistent interpretation and to inform strategic planning'. Another institution was advised to 'introduce a more systematic gathering and reporting of cohort data', with particular reference to year-by-year student progression, in order to enhance 'academic action planning arising from annual review'. Similarly, another report noted an institution's acknowledgement of the impact of a 'lack of standardised data relating to student performance and feedback' on 'the effectiveness of annual monitoring'.

27 Several reports commented on lack or inadequacy of central data provision. This leads to dependence on departmental data, making comparative analysis across subjects and programmes unreliable. Most of these reports noted that efforts were under way to improve the quality and timeliness of management information systems and, consequently, support for analysis of student progression on a unified basis. Other reports, however, commented on the limited use made of statistical data in annual monitoring. One report recommended that an institution 'make more effective and systematic use of statistical performance indicator data in annual monitoring and programme review processes', with particular reference to 'formal analysis of student progression and assessment data'. Some reports identified the variable quality and depth of analysis of student data across courses, programmes and schools. One report commented on the disjunction between the institutional focus on student retention and the variable depth of analysis of student progression.

28 There was effective overall use of statistical data, whether across an intake from entry to completion or by year-on-year comparison [Royal College of Music, 56]. One report noted how an institution's adoption of a detailed cohort analysis would assist departments, faculties and the institution in identifying student support and teaching and learning needs, as well as good practice for dissemination. The use of progression and completion statistics in annual monitoring was identified in one report as a feature of good practice [Royal College of Music, 87]. A later paper in the Outcomes... series will be devoted to institutions' use of progression and completion statistics for monitoring the quality of educational provision and the maintenance of academic standards.

External input to monitoring

29 Over and above the general role played in monitoring by external examiners' reports, some institutions draw on feedback from external agencies and stakeholders such as professional, statutory and regulatory bodies (PSRBs), and employers. In one case, professional links were cited as a feature of good practice. Others convene annual review panels independent of the school or department monitored, which may undertake formal encounters with programme leaders or programme teams or with students. One audit report noted that such a review panel was reinforced by an adviser external to the institution whose role included submission of a report.

Student input

30 Audit reports commented variously on the quality and consistency of student input to monitoring processes. Some reports noted that student feedback was focused on units, modules or departments rather than on programmes. Others noted the participation of students in monitoring processes as members of quality assurance committees from the programme or subject to the institutional level, while one recommended formal involvement of students in the production of annual monitoring reports. One institution's 'explicit use of student input', including that of research degree students, in annual programme monitoring was found to represent good practice [Surrey Institute of Art & Design, University College, 29-31].

Staff input

31 Several reports contained recommendations advising institutions of the need for greater involvement of staff in monitoring processes. One report found staff knowledge of monitoring arrangements and documents 'limited and variable', and a lack of ownership at local levels. Others noted the general non-involvement of academic staff in the production of the annual monitoring report and action plan, as well as a lack of understanding of the purpose of monitoring. Conversely, other reports commented on the high degree of staff engagement with annual monitoring and other quality assurance processes.

The monitoring sequence

32 The links between the various stages in the annual monitoring sequence involved a number of issues and recommendations. Issues included lack of consistency in reporting between programme and faculty level and the role of effectively linking the sharing of good practice and implementation of recommended action. One report observed variable departmental understanding of the purpose of annual monitoring and matters for onward reference, with tension between reflection on quality and standards in teaching and requests for enhanced resources. In another instance, conversely, 'holistic' school reports - combining quality and standards, and human and financial resources aspects of schools' activities - provided a coherent and valuable instrument for institutional management [University of Greenwich, 54]. Another report identified the lacuna that might occur in programme monitoring where, in a departmentally focused process, a programme drew on modules from more than one department. One report noted the difficulty posed for action planning when numerous programmes were grouped together in a single synoptic report.

33 A common topic was late and uneven submission of annual reports from departments and schools to central committees, with consequences for the overall monitoring schedule and follow-up actions. In one example, it appeared that faculties might submit reports up to 18 months after the events addressed. In another example the submission deadlines were being widely missed. In this latter instance the audit report suggested that the matter be attended to as a priority.

Action plans

34 A particular facet of the monitoring sequence considered in audit reports was the identification, implementation and monitoring of action points. One report noted, as a feature of good practice, faculty arrangements for reviewing departmental outcomes of issues raised in the preceding annual cycle [University of Bristol, 57]. Elsewhere, several reports noted a lack of clarity in the conduct of action planning and follow-up. Particular concerns emerged in relation to the recording of actions and responses; the timeliness of responses, at both programme and institutional levels, to the findings of annual reviews; lack of continuity between annual cycles in action planning and lack of institutional response to issues identified at programme level; and the identification of responsibilities in action planning. One report advised an institution of the need to support its 'review culture' with a more systematic and timely approach to action.

35 Many reports, nonetheless, commented on the general rigour and effectiveness of institutional processes for reporting and acting on the issues identified in programme monitoring. One report remarked on the effectiveness of both upward reporting and monitoring of departmental outcomes from action planning [University of Bristol, 57].

Dissemination of good practice

36 The same report noted also the effectiveness of the identification and dissemination of good practice within the institution's taught provision. Many audit reports confirmed the inclusion of good practice within institutional annual monitoring report templates at programme, departmental or school/faculty levels. Several reports commented on central initiatives in collating and disseminating good practice identified through annual monitoring. One report noted a particular instance of good practice in the monitoring of a distance-learning programme for which there was no established dissemination mechanism within the institution's monitoring procedures. Another report suggested that the lack of a requirement for annual departmental reports risked forfeiting opportunities for quality enhancement through dissemination of good practice. Another institution was encouraged to focus on the opportunities provided by annual monitoring to identify and disseminate local good practice.

Filtering of information

37 Several audit reports noted the discretion exercised by schools in determining the content of summary reports. Therefore, they risk not only inconsistencies in approaches to reporting across schools, but of turning reports into subjective filters rather than comprehensive reflectors of school activities. One audit report contained a recommendation that school reports should 'allow for effective institutional oversight of issues identified for action at school level'. Similarly, another audit report, noting the expectation that faculty summary reports would focus on generic issues, advised an institution to ensure that central committees received 'sufficiently comprehensive information' to allow effective institutional oversight of quality assurance and enhancement processes undertaken at departmental level. While one report commented positively on the identification of different levels of responsibility for monitoring action plans and on the filtering out of actions 'at the appropriate level' [Salford, 30], another encouraged filtering of the detail sent up to the centre and greater emphasis on action, where appropriate, at school level.

Feedback on reports

38 Several audit reports, some with linked recommendations, commented on the delay that might occur between submission of monitoring reports to central bodies and feedback to programmes and departments. In certain cases the issue was less delay in than absence of feedback. This might be associated with lack of clarity among staff about monitoring processes beyond the faculty boundary. One report noted that the institutional requirement for feedback to staff, students and external examiners, in particular on good practice and matters for enhancement, was not always being realised. In two cases, of a validated and a collegiate institution, it appeared that access to external monitoring reports was not being used as effectively as it might to draw on external good practice in the evaluation and enhancement of internal performance. One report noted how institutional feedback processes had been improved, in the light of evaluation, by adding staff resources at the centre. In another case, the previous year's examination board reports were published for students as well as staff. These provided study guidance for students, as well as an overview for the institution of the quality of taught programmes, and were found to be good practice [University of Oxford, 247].

Review of monitoring processes

39 A number of audit reports provide evidence of improvements in the structure and content of monitoring processes stimulated by internal review. For example, one report noted how a series of reviews, internal and external, had led to greater consistency in monitoring operations across the institution and an increased focus on external reference points such as subject benchmarks and reports from PSRBs. The arrangements in two other institutions for annual 'academic health' meetings convened to review the effectiveness of annual monitoring were judged to represent good practice (see above, paragraph 14).

40 Several institutions were encouraged to continue with their reviews of annual monitoring processes in order to address matters identified in audit reports. These included consideration of and response to departmental issues beyond the department; the balance of responsibility between the centre and the parts of the institution; the involvement of staff; and rigour of outcomes and fostering of enhancement. Two audit reports advised institutions of the need to review their monitoring processes, in both cases with reference to the quality of reporting and the input of students.

Conclusion

41 The evidence of the institutional audit reports published by November 2004 is that programme monitoring is, generally, working effectively. The reports also show that reference is being made to Section 7 of the Code of practice and that monitoring arrangements have either been or are to be informed by it.

42 Overall, it is apparent that programme monitoring is contributing significantly to institutions' arrangements for overseeing and maintaining academic quality and standards.

---

Appendices