Introduction
Background
1 Following the Government’s announcement in March 2001 that the existing method for the quality assurance of higher education (HE) in England was to be reviewed, discussions took place between the Higher Education Funding Council for England (HEFCE), the Quality Assurance Agency for Higher Education (QAA), Universities UK and the Standing Conference of Principles to agree a new quality assurance framework.
2 Part of these discussions resulted in a review of the information available about the quality and standards of teaching in higher education institutions (HEIs). A Task Group was set up, chaired by Sir Ron Cooke, Vice-Chancellor of the University of York, and a consultation document was published in November 2001. The final report of the Task Group was published in March 2002.
3 The report set out the categories of information that should be available routinely within HEIs, and those that should also be published, as part of the revised quality assurance procedures for HE in England. The final guidance on Teaching Quality Information (TQI) was published by HEFCE in March 2003 as Information on quality and standards in higher education: final guidance (HEFCE 03/51).
4 The TQI which it was decided should be published consisted of both quantitative and qualitative data. The quantitative information, comprising mainly student input and output statistics, was to be verified and published by the Higher Education Statistics Agency. The qualitative information, relating to both institutional and subject level matters, was to be published by each institution, which would have responsibility for its accuracy and completeness. Institutions were expected to publish the first set of qualitative information on the TQI website during 2004-05, prior to its official launch in September 2005.
The nature of qualitative TQI
5 HEFCE 03/51 identified the following qualitative information for publication:
- summaries of the findings of external examiners, at programme or subject level, produced annually
- a summary of the institution’s learning and teaching strategy, as it has been presented to HEFCE under the Teaching Quality Enhancement Fund (TQEF) programme
- summary statements of the results of, and the actions taken in response to, periodic programme or departmental reviews, to be undertaken at intervals of not more than six years
- summaries of the HEI’s links with relevant employers, how the institution identifies employer needs and opinions, and how those are used to develop the relevance and richness of learning programmes.
Purpose of the review
6 Owing to the timing of the introduction of TQI, the 2002-05 round of institutional audits carried out by QAA could not include a systematic analysis of institutions’ processes for the collection and publication of the qualitative information. Consequently HEFCE asked QAA to carry out, during the course of 2005-06, a one-off sector-wide review to assess the accuracy, integrity and frankness of the qualitative TQI that had been published up to then.
7 In addition to the above, the proposed review was asked to consider:
- the experience of HEIs in ensuring the accuracy, integrity and frankness of their qualitative TQI
- the processes adopted by HEIs, and their effectiveness
- the challenges faced by institutions
- the experience and processes relating to collaborative partnerships.
8 This report presents the findings of the review, involving a sample of institutions from England and Northern Ireland, undertaken during May and June 2006. The results have been grouped according to themes which emerged during the review process.
Method
Sampling
9 Institutions were recruited to the review on a voluntary basis, prompted by the publication of a circular letter by HEFCE in January 2006 inviting them to participate.
10 The evidence sources for the review took three forms. In addition to the information published on the TQI website, 47 institutions, representing a range from across the whole sector, provided QAA with the source documentation upon which their published TQI summaries had been based. In addition, 31 representatives from 21 institutions attended three round table discussions, and a further round table discussion was held with 10 QAA auditors/audit secretaries.
11 The review used three interlinked methods to examine the evidence provided by the published TQI, the institutions themselves, and from a series of round table discussions:
- a desk-based review of documentary information
- an analysis of the round table discussions
- case-studies derived from later institutional audits.
Desk-based review
12 HEIs were requested to provide copies of either their institutional learning and teaching strategy, relevant external examiners’ reports, or internal periodic reviews. The latter were requested from one subject area per institution, using the Joint Academic Coding System and chosen to ensure a broad coverage of subjects overall and across different types of institution.
|
Source document |
No. HEIs submitting |
No. of docs received |
|
Learning and teaching strategy (Institution) |
19 |
19 |
|
External examiners’ reports (Subject) |
13 |
105 |
|
Internal periodic review reports (Subject) |
15 |
17 |
|
Total |
47 |
141 |
13 These source documents were then compared, by a team of two experienced auditors, with the relevant summaries published on the TQI website and analysed for:
- accuracy – the extent to which summaries are a true reflection of the source reports
- integrity – their comprehensiveness and consistency
- frankness – their openness and transparency.
This process included giving a rating for these factors, combined on a scale of one to five. Any emerging differences in the institutions’ approaches to producing the summaries, including good practices or weaknesses, were also noted. All documents were anonymised before analysis.
Round table discussions
14 The purpose of the round table events was to gather information on institutions’ perceptions and experiences, and on the processes they had used to produce and validate their TQI summaries. Areas explored included:
- the experience of institutions in meeting the requirements expected of the qualitative aspects of TQI
- the processes put in place for ensuring the accuracy, integrity and frankness of the qualitative TQI
- the relationship between subject and institutional responsibilities.
15 Three round table events were held for representatives from institutions and one for auditors and audit secretaries (to explore their perceptions/experiences of qualitative TQI in audit).
Case studies
16 In addition to the desk-based analysis of TQI reports and source material, it was proposed that the remaining institutional audits, due to take place in autumn 2005, could act as ‘case studies’ of the use of TQI. Because of the small number of audits remaining to be undertaken by the time of the launch of TQI (September 2005), the information gleaned through this method was necessarily limited, but the outcomes have nevertheless been included in the analysis where appropriate and supplemented by information provided by the auditors and audit secretaries present at the special round table discussion.
The impact of TQI
17 Evidence from both the desk-based review and all of the round tables indicated that, in many cases, institutions had not created new processes to produce the summaries, but had adapted existing processes for the purpose. The existing procedures for writing the learning and teaching strategy, handling external examiners’ reports and producing internal periodic review reports had typically been expanded to cater for the production and approval of summaries of the TQI source reports.
18 The adaptation, for TQI purposes, of existing procedures gave rise, not surprisingly, to significant variations amongst institutions (and even within single institutions) in the ways they managed key procedural stages. The experiences relayed through the auditors’ round table discussion suggested that variations in process existed particularly in the following key areas:
- responsibility for writing or producing summaries
- responsibility for signing-off summaries
- involvement of subject areas in reviewing and authorising summaries
- the use of templates as part of internal periodic review and external examiners’ reports
- the completion of the optional commentary within (external examiner) summaries.
19 There was noticeable variation amongst institutions in relation to the ownership of TQI production and authorisation. Informal feedback from QAA’s current collaborative audits indicates some confusion and variation about the locus of responsibility for TQI where more than one institution is involved with a programme, and institutions have asked for greater clarity about arrangements and responsibilities in this area.
20 The question of where responsibility lay was echoed in relation to the summaries of the external examiners’ reports. Both the auditors’ and HEI representatives’ round tables suggested that there was ambiguity about where responsibility lay (or should lie) for writing the summaries of external examiners’ reports.
21 In some instances the external examiners themselves took direct responsibility for writing the summary for the TQI website – either as an additional annex to their original report, or as a separate report. In contrast there were also examples where the summary was written by the institution itself, based upon either one or a number of external examiners’ reports.
22 Some of the institutions who placed the responsibility for drafting the summaries with the external examiners found that it was necessary to alter the contracts used, so that production of the summary became a contractual obligation. In some cases this meant that the process and timescale of the writing of the summary was outside the institution’s direct control. This led to problems meeting the deadlines for posting the information on the TQI website, staff time being used to ‘chase up’ missing summaries; in some cases institutions were unable to secure a summary from an examiner.
23 Where more than one external examiner reported on a programme, it was often not clear where responsibility for writing the summary lay. There was variation in practice (both amongst and within institutions), with either both examiners providing a separate summary, the ‘lead’ examiner providing the summary, or the institution producing a summary compiled from all the reports.
24 The desk-based review revealed that some institutions had remodelled their internal periodic review documents and external examiner templates to reflect TQI requirements, whilst others had used their normal institutional template. Participants in the HE staff round tables also commented that their usual procedure for drafting the institution’s learning and teaching strategy had been amended to take into account the necessity of producing a summary in the future.
25 These procedural differences are of interest, but it is also important to consider how institutions verify the accuracy, integrity and frankness of the finished summaries before they publish them. Who writes, revises, and signs off summaries determines what, and in what form, information is included. As the evidence below will show, in most cases it is not so much a question of the accuracy of the summaries, since these do not show any real errors of fact, but more of their integrity and frankness – as different authors and authorities will have different concerns and interests in relation to the uses ultimately made of TQI.
26 This variability can also make it more difficult to compare summaries from different institutions - and even, sometimes, different summaries from within the same institution - which reduces the usefulness of the information for students and other users. It also makes it harder to make judgements on accuracy, integrity and, in particular, frankness, as these features are largely determined by the ways in which the information is edited and presented.
Comparing source and summary
Learning and teaching strategies
27 Overall, the accuracy, integrity and frankness of the summaries of learning and teaching strategies were considered to be very high – all strategies submitted were ranked either 4/5 or 5/5 by the team of reviewers. No serious weaknesses were found.
28 There were, however, differences in the nature of the summaries produced. Summaries varied in length from two to 40 pages – how these related to the 1,500 word limit given in the HEFCE guidance was unclear. The majority of summaries reflected the standard structure of submissions to HEFCE’s TQEF. In a minority of cases the summary had been restructured to fit the specimen pro forma. Some summaries included future actions to be undertaken in the main report, whilst others placed these in a separate annex.
29 Despite these variations, virtually all the summaries were adapted directly, succinctly and accurately from the original documents.
Internal periodic reviews
30 The desk-based analysis found that the vast majority of summaries of internal periodic reviews offered faithful reflections of the issues and conclusions of the original documents. Most summaries submitted were ranked 4/5 or 5/5 for accuracy, integrity and frankness. This finding was supported by the feedback from the auditors as discussed during their round table event.
31 There were, however, many variations in content, especially as a result of the selection of areas of the original reports for inclusion or exclusion from the summaries. The TQI summaries analysed generally included comments on innovation and good practice but were less consistent in their inclusion of criticisms or recommendations for change. A similar situation was found when reporting action that was being taken in response to an internal periodic review. This practice of selecting only those parts of source reports which reflect well on an institution, while not seriously widespread, must inevitably limit the integrity and frankness of the published summaries.
32 The most accessible and useful summaries were from institutions which had used the TQI process to provide a forthright and focused account of the quality and standards of their programmes. By openly discussing recommendations, actions to be taken and progress to date, they provided strong evidence of their enhancement activities and demonstrated their willingness to tackle weaknesses.
External examiners’ reports
33 In contrast to the generally positive findings of the desk-based review in relation to learning and teaching strategies and internal periodic reviews, the summaries of external examiners’ reports must be viewed in a more negative light. As highlighted above, these summaries were produced in a quite different context to the other summaries under analysis and this was evident when their content was compared with the source documents from which they were derived.
34 The most noticeable weakness was the absence, from many of the external examiners’ summaries, of the optional commentary to supplement the three basic yes/no questions required by the TQI template. Although this does not mean that the examiners’ report summaries were inaccurate, it made it impossible to assess their integrity or frankness (since there was, in effect, very little to assess), and severely reduced the level of useful information available to the TQI user.
35 Even when the opportunity to provide a commentary was taken, it was almost solely used to highlight points of good practice and hardly ever to report genuine areas of concern or suggestions for improvement which had appeared in the external examiners’ original report. This presumably reflects the perhaps natural reluctance of an external examiner to criticise a department publicly, but it also meant that it was hard for the reader to gain a balanced impression of the examiners’ overall view of what they were examining.
36 In contrast to this reluctance to say more than the minimum required, some HE representatives raised concerns during the round table discussions about the opportunity presented to external examiners to include their own personal views on a programme, which were not necessarily related to its quality and standards. The round table participants also reported experiences of disagreement between an external examiner and an institution or department in respect of the content of the summaries, leading to a delay in publication.
Handling complexity
37 One particular area of TQI content that is worthy of mention, though not directly related to the translation between source and summary, is the incapacity of the system to handle more complex areas of provision such as multidisciplinary, modularised programmes, and collaborative provision
38 The HEI representative round tables and the desk-based reviews highlighted the variations in institutions’ practices for authoring summaries for their collaborative programmes (see also paragraph 19). Problems were encountered when it came to disaggregating information relating to provision delivered at the awarding institution and that delivered by collaborative partners. This, of course, had an impact on the perceived integrity of the data submitted.
39 A comparable problem was identified through the desk-based review in summaries where a programme of study covered multiple disciplines and therefore involved a number of external examiners. In these cases the majority of the TQI summaries did not differentiate between individual pathways or the comments of individual examiners. As a result the opportunity was missed either to include commendations or to identify problems relating to specific components of a programme. This absence of differentiation had a consequent impact on the integrity and usability of the information presented in the TQI summaries. A similar situation was revealed in some internal periodic review summaries. When these related to composite programmes being reviewed as a single unit, it was difficult for the reader of the TQI summary to draw out the information relating to any single programme.
40 Good practice was found in one instance where a programme covered by several external examiners had a TQI summary which provided each of their individual commentaries, while protecting their anonymity.
The nature of TQI
Purpose
41 Feedback from the round table discussions indicated that overall there was a lack of clarity about the intended purpose and target audience of TQI. This has inevitably led to variations in approach, language and content, with institutions tailoring their summaries to suit the audience for which they believed TQI is intended. This had an impact on the reviewers’ ability to assess the ‘frankness’ of the TQI summaries, since without guidance on what ought to be included - or not excluded - it was difficult to judge how ‘frank’ a summary was.
42 As is widely recognised with many quality assurance processes, there is always the potential for ‘game playing’. In this case, temptations undoubtedly exist to tailor TQI specifically to show an institution or programme in a more favourable light through the selective editing of source material. This could have a significant impact on the integrity and frankness of the summaries published, perhaps through the omission of action points or recommendations within the summaries, as these might be thought likely to be perceived negatively by the general public.
Internal documents translated for external use
43 It appeared to the desk-based reviewers and to the auditors at the round table discussion that TQI summaries, whilst intended to provide useful and usable information to the public, are currently often written in a style and language which may not be helpful to the general reader.
44 This problem mainly results from the use of internal periodic reviews and external examiners’ report summaries. These documents are produced primarily for internal audiences and for internal purposes. Translating these documents into summaries for a very different and varied external audience requires a shift in style as well as a sometimes difficult judgment as to what should be included or excluded and how that information should be presented.
45 Some external examiners’ reports included in the review were clearly written specifically to meet the requirements of professional, statutory and regulatory bodies and did not lend themselves at all readily to translation into a more ‘reader-friendly’ TQI summary. This appeared to be particularly true of programmes in education.
46 The original internal nature of periodic reviews and external examiners’ reports also has a consequential affect on the comparability of the TQI summaries amongst institutions, as they reflect the differences in content and focus of the original sources.
Workload
47 Through the desk-based review it became clear that no matter how well adapted the internal processes of an institution were to the requirements of TQI, the production of summaries created an unavoidable extra layer of activity.
48 This workload varied with the approach that individual institutions took to TQI. There was evidence that some institutions favoured a minimalist approach, getting the minimum done to fulfil the basic requirements of TQI. Others tried to be as full and frank as possible in their reporting. The latter approach had significant resource implications associated with ensuring, in particular, that all external examiners’ report summaries contained the optional commentaries.
49 HE representatives at the round table discussions considered that the resource requirements required to adhere to the qualitative requirements of TQI were not proportional to any benefit gained. They reported that significant time was spent writing summaries, chasing summaries and uploading them on to the TQI website. Even where the requirements for the TQI summaries had been incorporated within existing institutional processes, they were still considered to create an additional burden not matched by any comparable increase in resource or funding.
Conclusions
50 This review was intended to assess the accuracy, integrity and frankness of the qualitative TQI and also to examine the experience of institutions in implementing the TQI requirements.
51 In the main, the review has confirmed that the summaries of the learning and teaching strategies and the internal periodic reviews demonstrated high levels of accuracy, integrity and frankness.
52 Noteworthy concerns have, however, been raised with regard to the summaries of external examiners’ reports. Although these were found to be technically ‘accurate’, the fact that the template actually required no more than three ‘yes/no’ answers and that the optional commentaries, where offered, were frequently restricted to ‘positive’ comments, severely reduced in most cases the summaries’ direct usefulness. This, it must be emphasised, is more a consequence of the very nature of TQI, than the result of any conscious attempt on the part of institutions to misrepresent the information.
53 Other findings worthy of highlight include the following:
- the lack of clarity within HEIs about the purpose and intended audience for TQI, and the associated consequences for the language and content of the summaries
- the resource burden felt by institutions due to the demands of the production of summaries
- the variation in approach by HEIs to producing the summaries and the impact this has on the comparability of the information
- problems in adapting internal documents for external purposes
- problems in capturing more complex provision and the impact this has on the usability of the summaries to the end consumer.
Top