Annex
The Agency carries out assessments of higher education provision in England and Northern Ireland on a subject by subject basis. The programme of subject review, commenced originally by the Higher Education Funding Council for England, will cover all subjects taught in higher education institutions in a cycle lasting from 1993 to 2001. After this cycle is completed, subject review is due to be replaced by a new quality assurance method covering all provision throughout the United Kingdom.
The results of these reviews, known also as Teaching Quality Assessments, are published, and each assessment is summarised by a 'graded profile', sometimes referred to as a 'TQA score'. Published TQA scores are used by some national newspapers in constructing 'league tables' of higher education institutions. Profiles are not designed for translation into league tables, so those wishing to use them in this way should be aware of their limitations.
This note is intended to assist those wishing to make use of the graded profiles, either as a guide to the quality of an individual programme, or in making comparative judgements between programmes and the institutions that provide them. The note does not deal with the results of Scottish and Welsh subject reviews, as these are expressed largely descriptively, rather than numerically.
Anyone wishing to build up a complete picture of the quality of teaching and learning in a higher education institution should be aware that TQA reports are not the only source of information. The Agency carries out audits, which result in published reports on the overall academic management of institutions, and on collaborative links with partner organisations overseas. Institutions offering programmes of initial teacher education are reported upon by OFSTED on behalf of the Teacher Training Agency. Many programmes are accredited by professional bodies, which in some cases publish reports.
What is TQA for?
Subject review is carried out for three main reasons:
- To meet a statutory requirement.
The Funding Council is obliged, by the Further and Higher Education Act 1992, to secure that provision is made for assessing the quality of education provided in institutions for whose activities they provide financial support. This enables the Funding Council to ensure that public money is not wasted on unsatisfactory provision.
- To provide public information.
Information about individual programmes is helpful to potential students, and those who advise them, when applying to enter higher education. Information about programmes is also to helpful to employers who recruit graduates, and to professional bodies who recognise some higher education qualifications that are relevant to their field of activity.
- To help institutions enhance the quality of their provision.
An independent evaluation of the strengths and weaknesses of programmes assists institutions in learning from good practice and addressing points of relative weakness.
What does TQA measure?
TQA does not make judgements against a single standard that is of universal application in each subject. The aims and objectives of programmes having the same or similar subject titles, but offered in different institutions, will vary. Programmes will reflect the particular research interests of individual institutions, and some may have more explicitly vocational aims than others.
In common with most qualifications, degrees represent a range of attainment, not a single, absolute level. Whilst a broad comparability of degree standards is maintained through the use of external examiners, within the range of achievement that may be represented by a degree there can be some variation in demand between programmes. In the future, the comparability of standards now achieved through the external examiner system will be reinforced by the publication by the Agency of subject benchmark information.
The graded profile relates to the aims and objectives set by the subject provider, and so should be read in conjunction with those aims and objectives. Scores do not tell the reader how a programme is performing in relation to an external standard, they tell the reader how well the institution is doing in terms of meeting the objectives it has set for itself. Within the legitimate range of achievement represented by a degree, it will be harder to achieve high scores in the profile against very demanding objectives than it would be against less demanding objectives.
When TQA scores are used in league tables to make comparisons between different institutions, it is important to remember that what is being measured is performance against each institution's own objectives.
Measurement is made by considering six aspects of provision. The aspects are:
- Curriculum design, content and organisation.
- Teaching, learning and assessment.
- Student progression and achievement.
- Student support and guidance.
- Learning resources.
- Quality assurance and enhancement.
A full statement of the factors that are taken into account in assessing each aspect of provision can be found in the 'Subject Review Handbook' published by the Agency. Each aspect is graded on a scale of 1 to 4. The scores relate specifically to the aims and objectives of the programmes.
Aims are usually set at a higher level of generality than objectives. For example, the aims of a suite of degree programmes in general engineering in one institution are to provide:
'A broader engineering education than is offered in single subject departments, and to provide sufficient depth of knowledge to satisfy the accreditation requirements of the professional engineering institutions.'
A more specific objective of the same group of programmes is that on graduation a student will have:
'Developed the ability to apply knowledge and understanding in the process of engineering design'.
Aims will vary with the level of the programme. For example, an MSc programme in media and communications taught in the Department of Social Psychology of a major university aims:
'To provide a high quality postgraduate education which introduces students to major social scientific approaches to media and communications;' and 'to provide a research training, recognised by the ESRC'.
By contrast the aims of a further education college offering a range of HND programmes in communication and media studies include:
'To provide the training, education and skills experience that will enable students to work effectively and cohesively as part of a technical crew'.
Within first degree programmes there can be differing objectives depending upon the intended career path of the student. For example, one four-year accredited MEng programme has as an objective:
'To provide sufficient breadth and depth of study to satisfy the requirements of the professional institutions, leading to chartered status.'
In the same department, a three-year non-accredited BSc programme has the objective:
'To provide a shorter route for students who require a sound engineering education, but do not immediately seek chartered status.'
It is against such aims and objectives that judgements are made in each aspect of provision. The following tests are applied:
- 'To what extent do the student learning experience and student achievement, within this aspect of provision, contribute to meeting the objectives set by the subject provider?
- Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?'
The scores allocated reflect the following judgements:
-
The aims and/or objectives set by the subject
provider are not met; there are major shortcomings that
must be rectified.
-
This aspect makes an acceptable contribution to the
attainment of the stated objectives, but significant
improvement could be made. The aims set by the subject
provider are broadly met.
-
This aspect makes a substantial contribution to the
attainment of the stated objectives; however, there is
scope for improvement. The aims set by the subject provider
are met.
-
This aspect makes a full contribution to the
attainment of the stated objectives. The aims set by the
subject provider are met.
Does a number tell the whole story?
Not necessarily! Scores of 4 or 1 are pretty unequivocal. Scores of 2 or 3 need to be looked at with greater care. It will be necessary to read the narrative of the report.
A score of 3 means that the aspect in question is making a substantial contribution to the aims and objectives but there is scope for improvement. A student contemplating enrolling on a course might like to check on just what it is that was identified as needing improvement.
For example, in one programme teaching, learning and assessment scored 3. Teaching quality was reported to be high and workshop sessions were used effectively to give students the opportunity to develop and apply skills. However, there were problems with assessment and particularly in relation to providing feedback to students. The assessors criticised this and reported that external examiners had also commented adversely upon it. In this case the grade 3 tells the reader that something is not as good as it could be. The narrative discloses that there was nothing wrong with the quality of teaching or the practical learning opportunities, but there was quite a serious problem with providing feedback to students from their assessments.
In another example, a mathematics department scored only 2 in the curriculum aspect. The department provided three types of programme. The first was an access programme that assessors praised. The second was service teaching provided to science and engineering departments. Again, the assessors found this to be of quite good quality. However, the curriculum for the degree programme was weak, particularly in relation to the third year where assessors felt that it did not match up to the normal expectations of the final year of an honours degree programme. A student contemplating a single honours programme in mathematics would be best advised to look elsewhere. But a student needing to acquire the mathematical skills necessary to tackle a science or engineering programme could be well served by that institution.
Each aspect, but particularly teaching, learning and assessment, is made up of a number of elements. The numerical summary is bound to reflect a balance of strengths and weaknesses. Similarly, a subject review may cover a range of programmes, some of which may be better than others. Again, the score will reflect the balance of strengths and weaknesses. A department with a weak degree programme could be saved from an unsatisfactory marking by strong HND provision, resulting in an overall grade of 2. A small, but poorly planned master's programme could pull a department down from a 4 to a 3 on the curriculum aspect, despite excellent undergraduate provision.
Any system of numerical reporting on diverse and complex provision is bound to contain an element of compromise and averaging. It is important for users of the information to read the narrative to find out just where any weakness actually lies.
Using the Graded Profile to Compare Institutions
The graded profile is not designed for the purpose of making inter-institution comparisons. Nevertheless, it is inevitable that the figures will be used in this way; by institutions proclaiming good scores as evidence of their excellence, and by journalists constructing league tables for publication.
Any league table based upon TQA scores should come with a health warning that explains that like is not being compared, strictly, with like. At best, at equivalent levels, broadly similar is being compared with broadly similar.
The validity of comparisons can be enhanced in several ways.
SCOPE OF COMPARISON
Comparisons by subject have a greater validity than comparisons between whole institutions. Universities are large and complex organisations, all will have areas of relative strength and relative weakness. Potential students will be as much interested in the subject to be studied, as in the institution as a whole. Combining subject based TQA scores across an entire institution carries with it all the risks inherent in averaging averages.
More recent information is of greater validity than older information. A subject review reports on the state on provision at the time at which the review took place. If this is several years ago, it is likely that there will have been changes. Weak provision may have been improved as a result of inadequacies having being identified and corrected. There will be some turnover of staff, those who achieved a result reported five years ago may have retired or moved on, and could have been replaced by either stronger or weaker staff. There is a strong case for using only information of comparable age and for regarding older information as being of primarily historical interest.
When TQA was first introduced outcomes were expressed as 'excellent', 'satisfactory' or 'unsatisfactory'. Institutions carried out a self-assessment to determine the category into which each part of their provision would fall. HEFCE did not visit provision that was self-assessed as 'satisfactory'. The method was changed to one of universal visiting, with all provision being assessed directly by external assessors. The graded profile was first introduced in 1995 and has remained in use since then. There is no reliable means of equating a particular aggregate TQA score with an earlier classification of 'excellent'.
SIZE OF PROVISION
There is no standard unit of assessment. Programmes may be looked at in groups of differing size, or even individually. Much will depend upon the way in which programmes are organised within an institution, and on relative student numbers. For example, an institution having five hundred modern language students could have five separate assessments in Russian, German, French, Italian and Spanish thus resulting in five TQA scores. Alternatively, similar provision in another institution might be assessed as a whole as modern languages, thus producing only one TQA score. Some league tables attribute equal weight to each TQA score, regardless of the total student numbers involved. In such a case, an institution that was weak in modern languages would benefit by having it assessed as a single unit; whilst one that was strong would benefit by having it assessed as separate units. If TQA scores within a subject area are aggregated for the purposes of comparison, it would be appropriate to weight them by student numbers.
A SINGLE NUMBER?
There is no satisfactory way of reducing the multi-faceted judgements represented by a graded profile into a single number that can then be used to construct a league table.
A score of 21 made up of three 4s and three 3s might be regarded as quite a good result. However a score of 21 made up of five 4s and a 1 is unsatisfactory, as any score of 1 results in quality not being approved. This is not altogether a hypothetical example, there is a recent case of a profile of 4, 4, 4, 4, 3, 1. Despite the overall total of 20 the provision was deemed unsatisfactory. A simple addition can conceal significant weaknesses.
Simple addition of scores across a profile assumes that equal weight should be attached to each aspect. When a graded profile is read as such there is no need for the aspects to be weighted, because each is considered individually. However, if scores are combined across the profile, issues of relative weight can arise.
Each aspect is important in its own right. However, aspects may have differing levels of importance in relation to each other, depending upon the circumstances of individual programmes and students.
If the curriculum is not designed so as to achieve the intended outcomes of a programme, or if assessment is incapable of measuring the attainment of those outcomes, then no amount of sympathetic student guidance is going to put that right. In that sense, the first two aspects of the profile are of fundamental importance to any programme.
However, for a student choosing between two well designed programmes, other aspects could be critical. A confident student with good learning skills might attach high importance to learning resources that could be used independently. A student with less well developed study skills might regard student support as being of paramount importance.
Whilst it is relatively easy to identify aspects where failings could be very damaging to a programme, the relative significance of other aspects will vary according to institutional context and individual student needs. Any aggregation of numbers across the profile, weighted or not, cannot reflect this.
A further consideration is the impact of a bad score. A 1 in the profile means that quality is not approved. Effectively, for purposes of comparison, a 1 in the profile reduces the aggregate score to zero. However, what is the adverse weight that should be attached to a score of 2? This indicates that significant improvement could be made and, due to the effect of averaging within an aspect, could indicate that some part of an aspect is actually unsatisfactory.
All of this illustrates the degradation of data that is bound to occur when a profile representing a complex series of judgements is reduced to a single number. Newspaper league tables are a fact of life, but those reading them should be aware of the over-simplification that results from converting profiles to a singe number.
Using Graded Profiles to develop an Institutional Quality Enhancement Strategy
Subject review results for an institution across a range of subjects form a matrix. Attention should be given not only to the profiles for each subject, but also to the vertical columns which show how well the institution is delivering within each aspect across its range of provision. This information is particularly useful when looking at performance in those aspects that depend to a large extent upon institution wide services, that is the second half of the profile.
It is reasonable for a good institution to aspire to a high proportion of 4s in all columns. Every institution will have its strengths and weaknesses but, overall, strengths should predominate.
For example, if across more than twenty graded profiles, an institution achieved grades of 4 in the quality aspect in only five profiles, whilst scoring only 2 in four profiles and in one, 1, this would suggest a fairly significant weakness in institutional quality assurance. The same institution might have slightly disappointing scores in the curriculum aspect (ten 4s but three 2s) and in teaching, learning and assessment (only eight 4s and one 2). Nevertheless resources (sixteen 4s and no 2s) and student support (fifteen 4s and no 2s) might be good, with students progressing well (fifteen 4s and no 2s). This could paint a picture of an institution that is well resourced, and which is able to attract good students, perhaps due to a well-established reputation. Nevertheless, taken together these scores could suggest a potential problem, with relatively poor ratings in overall quality systems working through to some apparent under-achievement in curriculum design and teaching quality.
Similar patterns may be seen in other institutions. One having twelve graded profiles over a three year period gained only five 4s in each of the curriculum and teaching/learning/assessment aspects. The quality column had only four 4s in it but two 2s.
Even the best institutions can use the data from the graded profiles to identify areas for improvement. One institution, with eight graded profiles over the last three years, has dropped only eleven points from the maximum total available. However, six of these are in quality management and enhancement, suggesting that this is an area where some university wide attention might be needed. Similarly, another institution with nine graded profiles in the last three years dropped only fifteen points from the maximum available. Six of these were lost in the teaching, learning and assessment aspect. A reading of the reports shows that there were no problems with teaching and learning, but it is assessment that has some scope for improvement. Again, a university wide focus on this could pay dividends.
Conclusion
The graded profiles are a rich mine of information. They have most to yield when they are used in the manner for which they were originally intended, looking separately at each aspect of provision to identify strengths and weaknesses. They can be a particularly powerful aid to enhancement of quality when used to identify areas of improvement that are common to a number of subjects and which could be addressed by an institution wide enhancement strategy. They enable students and employers to identify those institutions that have a consistently good record in delivering programmes that meet their intended outcomes.
The profiles were not designed for use in the construction of league tables comparing whole institutions, and those using them for that purpose would be well advised to make clear the limitations and simplifications inherent in using the data in that way.
Top