Outcomes from institutional audit published papers

External examiners and their reports
Second series

QAA 05/08

Summary

Consideration of the 59 institutional audit reports published between December 2004 and August 2006 shows that, in almost all cases, institutions had policies and practices in place to specify the roles and responsibilities of external examiners and procedures relating to their appointment and briefing, and to receive and respond to their reports. The reports show that in most cases these procedures were making a significant contribution to institutions' work to safeguard the academic standards of awards. Almost all the reports show that institutions had anticipated the introduction of a revised edition of the Quality Assurance Agency for Higher Education's (QAA's) Code of practice for the assurance of academic quality and standards in higher education (Code of practice), Section 4: External examining (hereafter, Section 4) by reviewing and, in some cases, updating their procedures. In almost all cases, institutions were already providing summaries of their external examiners' reports for publication on the HERO Teaching Quality Information (TQI) website. The TQI website was superseded by the Unistats website in 2007.

The audit reports show that institutional procedures for the appointment, briefing and induction of their external examiners were generally sound. In a number of cases reports recommended that the terms of reference provided for external examiners, and which stated their authority, should be set out more clearly. This was seen as particularly important where an institution had delegated the briefing of external examiners to subject teams, departments, schools or faculties, partners or to a combination of these.

Audit reports frequently comment on the ways in which institutions made use of the reports of their external examiners, including through programme monitoring and review. There are comments in a number of reports on the use by institutions of standard forms with 'tick boxes' to ensure that external examiners provided all the required information. In some cases the reports suggest that the use of 'tick boxes', while securing that many questions were answered, were not encouraging external examiners to provide reflective and evaluative perspectives on the characteristics of students' work and the effectiveness of institutions' assessment arrangements.

Many audit reports also comment on the steps institutions had taken to assure themselves that they had received and acted on reports from their external examiners. Some reports recommended action by institutions to ensure that external examiners receive timely formal responses to comments in their reports and some indicate the need for institutions to check that action that had been called for in external examiners' reports and approved by the institution itself had been taken.

The audit reports show that most institutions were compiling summaries of their external examiners reports to aid the distribution of information internally. Some institutions had introduced effective ways of confirming the reliability of such summaries. Again, most institutions appeared to be compiling evaluative 'overviews' of comments by external examiners to help them to identify matters for cross-institutional consideration and enhancement.

Overwhelmingly, the evidence of the audit reports indicates that institutions' external examining arrangements were working satisfactorily.

Preface

An objective of institutional audit is 'to contribute, in conjunction with other mechanisms, to the promotion and enhancement of high quality in teaching and learning'. To provide institutions and other stakeholders with access to timely information on the findings of its institutional audits, QAA produces short working papers that describe features of good practice and summarise recommendations from the audit reports. Since 2005 these have been published under the generic title Outcomes from institutional audit (hereafter, Outcomes...). The first series of these papers drew on the findings of audit reports published between 2003 and November 2004. This paper is based on the findings of institutional audit reports published between December 2004 and August 2006.

A feature of good practice in institutional audit is considered to be a process, a practice, or a way of handling matters which, in the context of the particular institution, is improving, or leading to the improvement of, the management of quality and/or academic standards, and learning and teaching. Outcomes... papers are intended to provide readers with pointers to where features of good practice relating to particular topics can be located in the published audit reports. Each Outcomes... paper therefore identifies the features of good practice in individual reports associated with the particular topic and their location in the Main report. Although all features of good practice are listed, in the interests of brevity not all are discussed in this paper. In the initial listing in paragraph 7, the first reference is to the numbered or bulleted lists of features of good practice at the end of each institutional audit report, the second to the relevant paragraphs in Section 2 of the Main report.

Throughout the body of this paper, references to features of good practice in the institutional audit reports give the institution's name and the paragraph number from Section 2 of the Main report. So that readers can readily refer to the relevant audit report, the name of the institution used when identifying references is the name that appears on the relevant audit report on QAA's website. For those institutions where a change of name has subsequently taken place, this is noted in Appendix 1 (page 17), and is the correct name at the time of publication of this paper.

It should be emphasised that the features of good practice mentioned in this paper should be considered in their proper institutional context, and that each is perhaps best viewed as a stimulus to reflection and further development, rather than as a model for emulation. A note on the topics identified for this second series of Outcomes... papers can be found at Appendix 3 (page 21).

As noted above, this second series of Outcomes... papers is based on the 59 institutional audit reports published by August 2006, and the titles of papers are in most cases the same as their counterparts in the first series of Outcomes.... Like the first series of Outcomes... papers, those in the second series are perhaps best seen as 'work in progress'. Although QAA retains copyright in the contents of the Outcomes... papers, they can be freely downloaded from QAA's website and cited, with acknowledgement.

Introduction and general overview

1 This paper is based on a review of the outcomes of the 59 institutional audit reports published between December 2004 and August 2006 (see Appendix 1, page 17). A note on the methodology used to produce this and other papers in this second Outcomes... series can be found at Appendix 4 (page 23).

2 As its counterpart in the first series of Outcomes... noted 'External examiners have a key role to play in ensuring that comparable standards are maintained in the UK's system of higher education'. Higher education institutions in the United Kingdom (UK) (for the purposes of this paper in England and Northern Ireland) are autonomous in academic matters. Each institution is therefore responsible for setting and safeguarding the academic standards of the awards for which it has responsibility, referring to the advice of the Academic Infrastructure and relying on its external examiners to confirm the integrity of its actions. As the August 2004 revision to QAA's Code of practice, Section 4 observed, external examiners constitute 'an integral and essential part of institutional quality assurance'. Across higher education in the UK there is a general view that the work of the external examiner 'provides one of the principal means for maintaining nationally comparable standards within autonomous higher education institutions' [Section 4, Introduction, paragraph 12].

3 QAA's Handbook for institutional audit: England (2002) (the Handbook) stated that it was QAA's 'first expectation' that 'institution[s] [would be] making strong and scrupulous use of independent external examiners in summative assessment procedures' [Handbook, paragraphs 17 and 59]. The Handbook also stated that where an audit team found that the institution made 'less than full use of independent external examiners' reports it would only be able to 'express limited confidence' in the institution's quality and academic standards arrangements [Handbook, Annex G, 'Limited confidence']. Further guidance was given in the report template QAA provided for its institutional audit teams, where they were asked to provide a detailed commentary on:

the use made by the institution of the opportunity to work with external examiners
institutional procedures for considering external examiners' reports
how the institution had worked with Section 4 of QAA's Code of practice
for institutions working with partnership links, any differences in the arrangements for working with external examiners in such provision
whether and how institutions sought the advice of their external examiners on their assessment policies and their implementation.

4 As noted in paragraph 1 above, the reports of the institutional audits on which this paper has drawn for its information were published between December 2004 and August 2006. For audits conducted in the 2004-05 session, QAA expected audit teams to refer to the 2000 edition of Section 4 for guidance. Section 4 was revised in 2003-04 to produce the 2004 edition, which became available as a point of reference for the purposes of audit one year later, in autumn 2005. Reports from the audits conducted between 2004 and 2005 frequently referred to work being undertaken by institutions to review arrangements for external examining against the precepts of the second edition of Section 4. Although audit teams did not begin to refer to the revised edition of Section 4 for guidance until autumn 2005 it should be noted that the two editions do not differ in substance.

5 Between 2004 and 2006 higher education institutions in England and Northern Ireland were expected to be able to provide summaries of their external examiners' reports for publication on the TQI website. This requirement was first published in October 2003 [HEFCE 03/51]. Thereafter, most institutions redesigned the ways in which their external examiners reported to them to request the examiners to provide the summary statements in addition to their reports. The audit reports considered by this paper show that almost all institutions had taken the necessary steps to publish the required summaries from their external examiners' reports (see paragraph 32). The requirement to provide this 'qualitative' material as part of TQI was dropped from October 2006, following a review of the arrangement.

6 In all cases, the audit reports published between December 2004 and August 2006 were able to confirm that institutions were operating satisfactory external examining arrangements for undergraduate and taught postgraduate programmes. An analysis and reflection on the detailed information on those arrangements forms the main part of this paper.

Features of good practice

7 Consideration of the institutional audit reports shows the following features of good practice relating to external examining:

the quality, range and accessibility of the University's staff development programme, which is well utilised by staff and includes specific training for quality assurance roles and activities [...including training programmes for newly-appointed external examiners] [University of Hertfordshire, paragraph 204 iv, paragraph 87]
...the design of its arrangements for responding to external examiners' reports and communicating to them the actions it has taken [University of Huddersfield, paragraph 317 ii, paragraph 77]
the College's approach to external examiners and their reports [Henley Management College, paragraph 213 ii, paragraphs 54-69]
the composite report of issues raised by external examiners, considered by Academic Board, as an effective vehicle for developing opportunities for enhancement [Heythrop College, paragraph 200 i, paragraph 66]
the induction of external examiners and the University procedures for considering their reports [University of Bolton, paragraph 196 bullet 5, paragraphs 62-7]
the successful efforts made, through briefing and feedback sessions, to engage external examiners in the wider aspects of their role, extending beyond their formal responsibilities in connection with assessment boards [Dartington College of Arts, paragraph 176 i, paragraph 52].

8 As noted in paragraph 3 QAA required its institutional audit teams between 2002 and 2006 to pay close attention to institutions' external examining arrangements and to report whether they were sufficiently robust to support a judgement of 'broad confidence' in the institution's arrangements to safeguard the academic standards of its awards. A review of the audit reports published between November 2004 and August 2006 shows that in addition to the reports noted above more than 30 other reports contained statements which explicitly endorsed the soundness of institutions' external examining arrangements.

Themes

9 The topics for discussion in this paper have been identified from clusters among the identified features of good practice as well as the recommendations for remedial or developmental action in the institutional audit reports. Readers will note that the nature of the themes identified and the order in which they are discussed bear a close similarity to the topics covered by the precepts of Section 4. There are two reasons for this: first, from 2004 06 Section 4 provided institutions and audit teams with a shared point of reference when institutions' arrangements for appointing external examiners and handling their reports were being audited; secondly, as the Preface to this paper notes, QAA's intention in producing Outcomes... papers is that they should be useful to readers in institutions. Hence, it has seemed sensible to order the themes discussed in this paper broadly in line with the sequence of the precepts in Section 4. From a consideration of both the features of good practice and the recommendations in the audit reports the following themes have been identified for further discussion:

institutional responsibilities:
- the roles assigned by institutions to external examiners
- institutional arrangements for the nomination, appointment, briefing and induction of external examiners
- how institutions capture information from external examiners, for example by using standard forms
- how institutions make use of the information they receive via external examiners' reports, including the use of summary or overview reports
- how institutions provide their external examiners with feedback on what has been done in response to their reports
- how institutions enable external examiners to discharge their responsibilities.

10 Readers may also find it helpful to know that there were identifiable clusters of comments and recommendations in the audit reports around two aspects of how institutions worked with external examiners' reports: how they followed up recommendations made by external examiners; and how the information provided by external examiners was being used in institutional arrangements generally. To put these in context it should be emphasised that recommendations in one or more of these areas were to be found in about one in five of the published reports.

Institutional responsibilities for designing and supporting the roles they ask their external examiners to undertake

The roles assigned by institutions to external examiners

11 Audit teams and higher education institutions both use QAA's Code of practice as a point of reference when reviewing quality management arrangements, including those for external examining. In Precept 1 of the 2004 edition of Section 4 the core functions of external examiners are described as reporting on:

the academic standards of the particular award(s) concerned
the rigour and fairness of the assessment processes
the standards of student performance in the relevant programme(s)
comparability of the academic standards and student performance with those in some other higher education institutions
any good practice they have identified.

12 The explanatory material linked to Precept 2 of Section 4 suggests that institutions should consider setting out for their external examiners clear information on the various roles, powers and responsibilities assigned to them, including the extent of their authority compared with examination boards. Where an institution operates arrangements in which external examiners may have different levels and kinds of responsibility, and which differentiate, for example, between external examiners with responsibilities for subjects or modules and external examiners with responsibilities at the level of the award, Precept 2 also advises that institutions should make these differences plain to the various external examiners so that they can understand their responsibilities properly.

13 Almost all the institutional audit reports noted that institutions had defined the roles of their external examiners. Definitions could be found set down in one or more of a 'quality manual', 'examinations handbook', or in an institutional code specially written for external examiners. The clarity with which the powers of external examiners might be specified could vary from institution to institution but, again, in almost all cases institutions expected external examiners to act as guarantors of the standards of their academic awards and the levels of student performance.

14 From the audit reports published between 2004 and 2006 the following appeared to be typical aspects of the role of an external examiner:

reviewing and approving draft examination questions
moderating a sample of scripts and coursework
conducting oral examinations where necessary
attending examination boards at which the final assessment of candidates was considered
providing an annual report in a standard format to the institution on:
- the academic standards of the institution's awards in relation to their experience in other institutions
- how the attainments of students matched the expectations set out in The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ), the relevant subject benchmark statements and the institution's own programme specifications.

15 Where an institution had clearly defined the role(s) it expected its external examiners to undertake, and audit teams encountered departures from those expectations, audit reports generally outlined the institution's expectations and the nature of the deviation from those expectations. In some circumstances reports indicated that such deviations from institutional norms might have the potential to lead to unfairness in the way students in different parts of the institution might be treated. In other circumstances they might have the potential to give rise to departments or individual subject areas operating assessment arrangements different from those sanctioned by the institution in its capacity as the awarding body.

16 Thus, one audit report identified that subject groupings across the institution were following different practices for the employment of external examiners at subject level. This was contrary to the institution's expectations as set out in its quality handbook. Hence, some external examiners appeared to be acting as additional markers of student work while others, elsewhere in the institution, had been asked to moderate the marks given by staff acting as first and second markers. In yet another part of the same institution, external examiners were asked to adjudicate in cases of disagreement between internal markers. The institution was recommended to take action to clarify for its external examiners its expectations and requirements for the performance of their role and to communicate this institution-wide.

17 Another audit report found that an institution had defined the roles and responsibilities of external examiners as 'to ensure that academic standards are maintained and that individual students are treated fairly'. The institution's examination regulations provided that examination boards 'decide students' results and classifications, but where there is disagreement the view of the external examiner shall prevail although the decision is a Board decision' [emphasis added]. The report noted some instances where external examiners had used their authority to require and conduct oral examinations for individual students, and to recommend decisions on their performance to examinations boards that had been accepted. In such circumstances it was not clear whether the institution had, in practice, delegated the powers of the board to the external examiners. The report recommended that the institution should clarify 'in formal documentation' the extent and limits of external examiners' authority to moderate the marks of individual students 'to ensure fairness to all students'.

18 In another audit report, the institution's modular undergraduate degree scheme included a number of individual 'core modules', each of which could contribute to more than one programme and award. In the institution's arrangements, external examiners were 'appointed to programmes and not modules' with the result that 'one module could be covered by as many as eight separate examiners'. The report noted that the institution had made no provision for a single external examiner to take 'lead' responsibility for each core module and the audit team could identify no arrangements to enable the external examiners to come to a common view of the academic standard to be attained in the core modules. The report recommended that the institution should revise the roles and responsibilities of external examiners with regard to the establishment and verification of the standards of individual modules shared between programmes leading to different awards.

19 Under the terms of their engagements with institutions, external examiners are generally required to 'confirm that academic standards set by the [institution] and attained by students are appropriate and comparable with those in other [higher education] institutions'. The published reports offer little information on how the comparability of the academic standards set by individual institutions is to be established by individual external examiners.

The nomination and appointment of external examiners

20 Precepts 4, 5 and 6 of the 2004 edition of Section 4 cover matters to be considered by institutions when dealing with the nomination and appointment of external examiners. They include suggestions about the need for clear criteria for the nomination and appointment of external examiners (including suggestions for establishing that they have the competence and expertise to discharge their responsibilities), for nominations to be given careful consideration and, if the nomination and offer of appointment are accepted, for the respective roles and responsibilities of the appointee and the institution to be formally embodied in a contract.

21 From the institutional audit reports published between December 2004 and August 2006, most institutions appear to have operated a process whereby the initial nomination of an external examiner would be made by a head of department or a programme leader. Nominations were usually scrutinised by a faculty or institution-level quality or examinations committee which then made a recommendation to a senior committee (often the Senate or Academic Board) for its approval. In some institutions this latter stage could be treated as a formality. Again, from the audit reports it would seem that most institutions included the criteria for the appointment of external examiners in a handbook or in their regulations, and that there was usually the expectation that nominees as external examiners should have sufficient seniority to command respect. Some reports comment explicitly that institutions had adopted the practice of requiring the nominee's curriculum vitae to be submitted with their nomination.

22 In most cases, the audit reports showed that institutional arrangements for the nomination and appointment of external examiners were consistent with the advice of Section 4. In the few cases where reports offered recommendations on this area of institutional practice, one concerned the timeliness of arrangements for appointing external examiners, and another the importance of an institution arranging to appoint external examiners for its partner links from institutions across the higher education sector in the UK, rather than a small part of the sector.

23 Institutional arrangements for nominating and appointing external examiners for programmes delivered and assessed in languages other than English were considered in one report. In this instance the external examiner was expected to oversee 'the assessment process and the standards of achievement' for the programme and the institution had been unable to appoint an external examiner with both the necessary practice-based experience and competence in the language of delivery and assessment. In this case, the report noted a need for adequate translation facilities for question papers and student work to make it possible for UK-based staff and external examiners to confirm the achievements of students taught and assessed through languages other than English. The report recommended that the institution take steps to secure the standards of its award.

24 A few audit reports discussed how institutions with a strong vocational orientation were approaching the identification and nomination of external examiners based in industry, business or the professions. Two reports noted how institutions had individually made arrangements for an external examiner with identified experience of higher education to work alongside and initially support external examiners appointed for their practice-based experience.

Briefing and induction arrangements for external examiners

25 Following the nomination and appointment of external examiners, there is an expectation (expressed in Precepts 7, 8, and 9 of the 2004 edition of Section 4) that institutions will ensure that they are provided with sufficient information and support to enable them to carry out their responsibilities effectively. In a substantial number of cases institutional audit reports commented directly on the formal induction sessions institutions provided for their external examiners. Some reports noted that institutions required the attendance of newly appointed external examiners at such induction sessions.

26 Overall, the audit reports showed that institutions were providing satisfactory and timely briefing information for their external examiners. This might be in the form of one or more handbooks or other documents, or through dedicated web pages, and might be provided at institutional-level or at the level of the faculty or department, and often at all three. In many cases reports also noted that departments provided external examiners with further information on the conventions and practices they applied to particular modules or courses.

27 One audit report noted that departments were required by the institution concerned to invite newly appointed external examiners to meet staff and to understand the departmental context [University of Bolton, paragraph 62]. Another report noted as good practice the institution's provision of a training programme for newly appointed external examiners [University of Hertfordshire, paragraph 87]. In another case, however, the report commented that although briefing and induction sessions were provided for external examiners the institution was not in a position to know which of its external examiners had availed themselves of such opportunities. The report recommended the institution to consider how it might address this.

28 Several audit reports commented on the adequacy of the information institutions provided for their external examiners and several reports offered recommendations. One report found that the information an institution had provided on its assessment arrangements for its staff, students and external examiners did not make its requirements sufficiently clear with respect to its pass mark. In this case, the institution was encouraged to ensure 'the consistent use of a single, institution-wide marking scale, so as to provide clear information to students, staff and external examiners'. Another report described the information that the institution provided on assessment criteria - again for staff, students and external examiners. In several cases the report noted that 'assessment criteria [were]...of a general nature' and the report recommended that assessment criteria be mapped onto intended learning outcomes of individual modules for the benefit of 'students and internal and external assessors'. Another report noted that information had been made available for the audit that the institution later confirmed had been superseded. This included guidance for staff, students, and external examiners about the role and powers of the latter. The institution was recommended to maintain the currency and accuracy of the guidance it provided for its external examiners.

29 Several audit reports also discussed arrangements for the end of external examiners' appointments. In one case, an institution had arranged for retiring external examiners to be required to make a closing overview report, reviewing significant occurrences during their appointment and comparing matters as they had observed them on commencing work and at the end of their term of office. Several other reports described the arrangements institutions had made to identify external examiners who had not submitted reports in a timely fashion or otherwise failed to meet the undertakings they had given when taking up their appointments.

The process of external examining

30 In several cases, audit reports found that all aspects of the processes institutions had established to support the work of their external examiners were robust [for example, Henley Management College, paragraphs 54-62]. Although not formally themselves identified as good practice, the arrangements of another institution for the appointment of its external examiners, the content of their reports, and the processes it had established to make use of the reports were identified as both effective and often reflecting good practice found elsewhere in the sector.

31 Very few audit reports found deficiencies in the way institutions' handled their external examining processes that were sufficiently serious to merit recommendations whether about the overall design of the processes, or how they were being managed. One such report concerned a reminder to an institution that in the 'interests of demonstrating equity of treatment of students', it should 'introduce procedures for ensuring that external examiners [had] full access to all relevant records of the criteria and calculations used by internal markers and moderators to establish students' results'. At another institution the report recommended that it should consider how to provide its external examiners with evidence that marking and moderation had taken place for all the modules with which they were connected. In one case, however, the deficiencies identified in the institution's handling of reports from its external examiners, extending over several years, were so serious that the audit was unable to express confidence in the institution's arrangements.

How institutions capture information from external examiners

32 As noted in paragraph 5, from 2004 to 2006, the institutions that hosted a QAA institutional audit had responded to the requirements of HEFCE 03/51 and were publishing, or preparing to publish, summaries of reports by their external examiners. Consequently, the approaches taken by individual institutions, and the progress they had made towards meeting the Higher Education Funding Council for England's (HEFCE's) requirements featured prominently in the section of the audit report reserved for comment on and analysis of the 'Reliability, accuracy and completeness of published information' made available by institutions about the academic standards of their awards.

33 In the context of the above, the report template QAA provided for the use of its audit teams between 2002 and 2006 specifically asked them to check that 'published external examiner report summaries...do justice to the full reports'. Almost all institutions had taken the view that the relationship between them and their external examiners was one where both parties needed to retain the opportunity to report to the institution fully and in confidence. Hence, almost all institutions adopted some arrangement for external examiners to provide a two-part report or two separate reports to the institution: one part summarising their key findings in a limited number of areas for publication as part of TQI; the other intended for the internal purposes of the institution.

34 The information to be provided as part of TQI had been stipulated in HEFCE 03/51 and included reporting on:

the [academic] standards set for the award(s)
the standards of student performance in comparison with those achieved in programmes or subjects at other UK institutions with which the external examiner was familiar
the soundness and fairness of the processes for assessment, examination and determination of awards.

To each of the points above, external examiners were asked to give a yes/no answer and, in the event of a negative response, to indicate the respects in which matters fell short. External examiners were also usually invited to give a description or bullet point list of any particular strengths or distinctive or innovative features worthy of drawing to the attention of external audiences.

35 In addition to the summary information outlined above, many audit reports contained information on the matters typically covered in external examiners' confidential reports to institutions. In most cases, they simply confirmed that the contents of external examiners' reports were consistent with the guidance offered in Precept 11 of Section 4, namely:

'the academic standards demonstrated by the students and, where possible, their performance in relation to students on comparable courses

the strengths and weaknesses of the students as a cohort
the quality of teaching, learning and assessment methods that may be indicated by student performance
the extent to which standards are appropriate for the award or award element under consideration
the design, structure and marking of assessments
the procedures for assessments and examinations
whether external examiners have sufficient access to the material needed to make the required judgements and whether they are encouraged to request additional information
the coherence of the policies and procedures relating to external examiners and whether they match the explicit roles they are asked to perform
the extent to which the external examiner's comments in his/her previous report have been considered and appropriately acted upon.'

In one case, this already comprehensive list of topics had been further expanded by the institution, to the point where some confusion had become evident among external examiners about how to complete their reports. The audit report therefore recommended that the institution should refine its guidance in order to assist its external examiners.

36 Several audit reports commented on the standard forms, or templates, institutions had adopted to ensure that their external examiners provided clear answers to particular questions in their reports to the institution. In this context, one audit report noted that the 'introduction of a "tick box" requirement for each question on the report form [had] ensured that all external examiners [had] to provide an explicit response to each question, but the absence of any specific requirement to substantiate these judgements, either on an individual basis or as part of a more general overview, [had limited] the opportunity to engage in qualitative monitoring of this key indicator of standards in a suitably consistent manner'.

37 In a number of instances audit reports discussed the arrangements institutions had developed to enable external examiners to deal with modular and/or institution-wide examination arrangements. In such cases, students' assessments might be scrutinised at the level of the module by external examiners appointed for their expertise in the subject, and students' overall performance might be subject to confirmation by a 'chief external examiner' appointed to take a holistic view of assessment across the modular scheme and its fairness to individuals or cohorts of students. In some cases institutions might expect chief external examiners to take a coordinating role between teams of subject-based external examiners. The title 'chief external examiner' might also be applied to an individual identified to speak for a team of external examiners in exchanges with the institution.

38 The audit reports show some difference of opinion between institutions on the role of a chief external examiner in a large modular scheme and the qualification for appointment to such a position. One institution, for example, had taken the view that a chief external examiner for a modular scheme should not simultaneously hold an appointment (at that institution) at subject level, and that only chief external examiners should normally attend award-level examination boards. This institution held that what it termed the 'separation of function' between module-level and scheme-level external examiners was a 'critical aspect' of its external examining arrangements. Another institution had taken a different view and had appointed their chief external examiners from those already appointed to act at module level. In yet another institution, chief external examiners had been appointed to provide additional robustness in the scrutiny of reports by other external examiners through confirming to the institution that no critical comments had been overlooked in compiling the institution's own summaries of the reports.

39 How institutions worked with formal written reports from their external examiners is discussed below, but it should also be noted that several institutions had established additional arrangements to gather oral reports from their external examiners at the end of examination or assessment boards. In one case the institution's practice was to produce 'succinct summaries' of oral reports from its external examiners for its 'Annual Standards Review', itself identified as a feature of good practice in the institution's work with its external examiners [Dartington College of Arts, paragraphs 51-2].

How institutions had used reports from their external examiners

40 Precept 13 of Section 4 advises institutions that 'full and serious consideration should be given by the institution to comments and recommendations contained within external examiners' reports, and the outcomes of the consideration, including actions taken, should be formally recorded'.

41 The institutional audit reports published between December 2004 and August 2006 indicated that in some institutions reports by external examiners were required to be addressed to the vice-chancellor, or principal. In other institutions, external examiners' were required to address their reports direct to the registry or to another central office. Once received by institutions, external examiners' reports were generally copied and sent to staff at faculty or school level, the head of department and the person leading the relevant programme or subject. Where the list of such postholders might not include the chair of the appropriate examination board, most institutions appeared to send copies of examiners' reports to that person, too.

42 From the audit reports, it appears that institutions had generally responded to the advice that they should give 'full and serious' consideration to external examiners' reports by ensuring that reports were scrutinised for matters of concern (and to identify good practice) at more than one level, usually - and at a minimum - by an officer based in the central administration and by postholders and committees closer to the point of delivery for the relevant provision. In some institutions, detailed consideration of the reports at the centre had been delegated to a senior officer, often a pro or deputy vice-chancellor, the institution's academic registrar, or another senior postholder. More rarely, a single and more junior administrator might discharge this function on behalf of a senior figure or a senior committee.

43 The audit reports show that most institutions had arrangements in place to enable them to confirm that the reports they required from their external examiners had been received and acknowledged, that copies of the reports had been provided for those who needed to see them, and that matters raised by external examiners in their reports had been addressed. In this latter context, almost all the audit reports published between December 2004 and August 2006 noted that reports by external examiners, and responses to them, featured prominently in institutions' internal annual monitoring reports (or their equivalents). This matter is discussed further in paragraphs 25-27 of the Outcomes from institutional audit Programme monitoring arrangements Second series paper. Again, in most institutions, an ancillary purpose served by their annual monitoring arrangements was to confirm for the institution that matters raised by external examiners in their reports had been addressed.

44 While most of the reports were able to confirm that in all but exceptional cases institutions responded promptly to comments made by external examiners in their reports, not all the audit reports were able to do so. For example, one report recommended that an institution should make more clear where responsibility lay for taking action in response to external examiners' reports and for ensuring that progress on such actions was monitored and recorded at institutional level. Another report recommended that the institution should make the methods by which matters arising from external examiners' reports were recorded, considered, and subsequently addressed more precise and consistent.

45 In the case of another institution, the audit report questioned whether its most senior committee was able to monitor external examiners' reports and responses to them in the way the institution expected. The report recommended that the institution should 'pay close attention to how actions are identified and reported on in response to external examiners' reports' (see paragraph 49).

Summary or overview reports of external examiners' findings

46 The institutional audit reports published between December 2004 and August 2006 show that almost all institutions used summaries and overviews of external examiners' reports in their internal arrangements. In most cases, summaries of matters raised by external examiners in their reports were used to focus the attention of committees and postholders on key points in the reports and reduce the volume of information circulated. Often, a report would be summarised on a single sheet of paper, sometimes referred to as a cover sheet, and sometimes external examiners themselves were asked to provide such summaries.

47 In most cases, audit reports also showed that in addition to using summaries of external examiners' reports in their internal processes, institutions were also providing their senior committees and postholders with evaluative reports (often referred to as 'overviews') of clusters of matters and common themes in examiners' reports from across the institution.

48 The audit reports also show that the availability and use made by institutions of overviews of external examiners reports, their content, and their evaluative qualities often appear to have contributed important insights for audit teams into how well an institution was safeguarding the academic standards of the awards for which it was responsible, and the likelihood that it would continue to be able to do so [Henley Management College, paragraph 58; University of Bolton; paragraph 64]. In one such case, the 'composite report' that comprised the overview, and that was provided for the information of the institution's Academic Board, was seen as an effective vehicle for developing opportunities for enhancement [Heythrop College, paragraph 66].

49 Where audit reports found that institutions were not taking steps to analyse external examiners' reports for recurring themes and generic matters, recommendations were likely to follow. For example, one institution was asked to consider undertaking a systematic analysis of generic institution-wide issues arising from monitoring and review to contribute to its quality enhancement strategy. In another report where the institution was already compiling an overview report, it was recommended, nonetheless, to review the institution-level processing and analysis of external examiners' reports to enable the relevant senior institutional committee to gain a full understanding of the nature of the matters being raised.

50 As paragraph 3 noted, where an audit identified weaknesses in institutions' arrangements to handle their external examining arrangements it would not be possible for the report to express broad confidence in the institution's current or likely future capacity to safeguard the quality of its provision and the academic standards of its awards. Such a view was reached in very few cases, and in each the crux of the matter concerned the capacity of the institutions to take an informed and sound view of the comments of their respective external examiners on the academic standards of their awards and their comparability with those in like institutions.

Feedback to external examiners

51 Precept 14 of Section 4 advises that institutions should ensure that external examiners are provided with a considered response to their comments and recommendations, within a reasonable time of submitting their report, including information on any actions taken by the institution. The institutional audit reports show that most institutions acknowledge their receipt of reports from their external examiners immediately and subsequently inform them of what action has been taken in response to matters they have raised.

52 In one audit report this particular aspect of the institution's work was identified as a feature of good practice. In this case, as part of its standard arrangements for responding to the reports of its external examiners via an 'action plan', the institution had devised 'a carefully sectionalised standard form providing a template for an interim action plan, and a standard form for recording "actions taken"'. The institution required that when both had been completed their contents be endorsed by the individual external examiner [University of Huddersfield, paragraph 77].

53 In a few cases, audit reports found that institutions' acknowledgement of having received a report from an external examiner was tardy. Several reports also found cases where external examiners might receive feedback on their reports from more than one body in the institution - for example from the registry and (at a different time) from a faculty-level body. In one particular case, the report expressed concerns about the duplication of effort involved for the institution's staff and that the external examiners would not know which communication to treat as the institution's authoritative response. In another case, a similar concern was expressed where it was not clear where responsibility lay among the affiliated parts of a confederal institution for making a formal response to each external examiner.

How well institutions enable external examiners to discharge their responsibilities

54 Institutional audit reports chiefly comment on how institutions work with the reports of their external examiners and make only passing comments on how external examiners are enabled to perform their duties. As discussed in paragraph 36, several audit reports noted that the institution's standard form for capturing external examiners' comments seem not to have encouraged external examiners to offer evaluative comments on examining arrangements and students' performance. Several other audit reports noted that institutions had identified for themselves that overly terse reports from external examiners could be unhelpful and were taking steps to encourage more reflective contributions. In one case, an institution had amended its report template to encourage its external examiners to provide more detailed information. On the evidence seen by the audit team, this initiative appeared to have been successful.

55 Only a few audit reports of institutions' quality assurance arrangements identify deficiencies in the work of external examiners. That there should be no evidence from the 59 audit reports that institutions had found it necessary to terminate the appointments of external examiners for non-performance suggests that in the period covered by the institutional audits external examiners had continued to perform their duties 'in a conscientious and professional manner'.

The findings of this paper and of its counterpart in the first series of Outcomes... papers compared

56 The first series Outcomes... paper on External examiners and their reports was compiled from reports of institutional audits conducted in the 2002-03 and 2003-04 academic sessions. This paper has drawn on the reports of audits conducted in the two following sessions, 2004-05 and 2005-06 for the latter part of which the 2004 edition of Section 4 formed a key reference point. At the same time the audit reports from 2004-05 and 2005-06 were also required to comment on how institutions were approaching the requirements set out in HEFCE 03/51 that summaries of external examiners' comments be provided for TQI. The audit reports show that most institutions had responded well to these developments.

Conclusions

57 Overall, the institutional audit reports show that institutions' external examining arrangements were generally working well and that comments by external examiners in their reports were making an important contribution to the work of safeguarding academic standards and managing the quality of provision. In some cases, audit reports recommended institutions to make greater efforts to ensure that the advice they received from their external examiners was more widely used in their quality and academic standards arrangements, and that external examiners received more timely feedback on what had been done to respond to comments in their reports.

58 The audit reports show that by 2006 higher education institutions in England and Northern Ireland had made considerable efforts to respond to two external developments: the development and subsequent publication by QAA of a revised edition of its Code of practice, Section 4 and the requirement by HEFCE for institutions to provide summaries of external examiners' reports for publication on the projected TQI website (now Unistats). In almost all cases, the audit reports found that institutions had made, or were making, the appropriate adjustments to the reporting arrangements for their external examiners and that the summary reports that had been published accurately reflected the findings of the external examiners.

59 External examining remains a key feature of the UK's approach to maintaining the academic standards of higher education awards. The evidence of the audit reports is that, almost without exception, institutions were responding satisfactorily to the information provided in their reports by 'conscientious and professional' external examiners.

Appendix >>

ISBN 978 1 84482