Enhancement-led institutional review 2003-04

Independent Observer report

1 Terms of reference

The letter of appointment from the Quality Assurance Agency for Higher Education (QAA) of 23 July 2003 required me as Independent Observer ‘to undertake a project to provide an independent monitoring of the first year of implementation of the process of enhancement-led institutional review’.

2 The ELIR process

The ELIR process is fully explained in the Handbook enhancement-led institutional review (the Handbook), and it is only necessary for the purposes of this report to draw attention to a few of its salient features.

ELIR should properly be seen, not as a departure from the methods of quality assurance in operation in previous years, but as an evolutionary development which builds on the successes of the past. Teaching Quality Assessment, subject reviews, institutional audits of the management of quality and continuation audits have all played their part in successfully embedding quality criteria and objectives in the fabric of Scottish higher education. The focus now is changing to concentrate more specifically on the enhancement of teaching and learning, but not at the expense of abandoning what has already been achieved. The ELIR process includes the requirement that institutions carry out their own internal review at the subject level, but also requires institutions to have in place robust systems to ensure security of standards, and the quality enhancement of teaching provision and the student learning experience.

These requirements lie at the heart of the whole process, and inform all the stages of the review, from the Reflective Analysis to the final review report.

3 Criteria and scope

The criteria to be employed in carrying out the monitoring process were not specified in detail, though the Handbook (paras 83, 85) refers to monitoring as being ‘primarily (although not exclusively) concerned with the operational effectiveness of ELIR’. The intention is, therefore, to evaluate each part of the process by asking the question whether and to what extent that part achieved its operational objectives, and in what respects and to what extent it might be improved. It may be inferred from the fact that the report is to be submitted soon after the end of the 2003-04 cycle, and from the fact that a longitudinal study of the process has also been commissioned by the Scottish Higher Education Funding Council (SHEFC) to cover the period 2003-07, that the important question of the impact of the process on quality enhancement in the higher education sector falls outwith the scope of this project. The primary objective of this monitoring process is to ensure that the method of review has been carried out in accordance with the Handbook and associated protocols in the first cycle of institutional reviews.

It is apparent, however, that there are certain important and innovatory aspects of the ELIR process, notably the emphasis on enhancement and the role of student reviewers, which call for some special analysis and comment. Separate sections of this report will be assigned to these aspects.

4 Methodology

The methodology employed in this study was fully set out in the letter of appointment. The Independent Observer was required to familiarise himself with the ELIR process, its philosophy, purposes, and relationships with other elements of the new enhancement-led approach to quality in Scottish higher education. In addition to scrutinising the relevant documentation, he was to sample each aspect of the process of the 2003-04 reviews, attend one of the formal training sessions, and solicit feedback from the various parties involved, both in the review teams and in the institutions being reviewed.

To comply with these instructions I have:

read the appropriate documentation
attended the first training session in November, 2003
accompanied the review teams for the whole of each of the Part I visits to each of the four institutions being reviewed
accompanied the review teams on the final day of each of the Part II visits.

In terms of feedback, I have:

arranged for a formal discussion with each of the teams (except the Royal Scottish Academy of Music and Drama team) at the end of each Part II visit
sought written feedback from the members of all the review teams, and from the trainees
arranged special visits to two of the four institutions after their receipt of the initial key themes letter
sought written feedback from the remaining two institutions
attended the final debriefing and development meeting of the trained ELIR reviewers and secretaries (including members of the 2003-04 teams) organised by QAA in July 2004.

5 Selection of reviewers and review secretaries

Paragraphs 72-75 of the Handbook set out the criteria to be adopted in the selection of reviewers and review secretaries. All UK higher education institutions (HEIs) were invited to nominate individuals for appointment as reviewers and review secretaries. Existing QAA auditors were eligible for appointment, and were invited to apply via a fast track process. In the case of student reviewers, they are recruited annually and remain eligible to serve as reviewers for as long as they meet the selection criteria, of which one of the most important is that it must be not more than three years since they had experience of study in a Scottish HEI.

It is the view of the Independent Observer that the criteria for selection set out in the Handbook have been duly met. Care was also taken to have a mix of reviewers from north and south of the border, apart from the students, who were all from Scottish HEIs. The majority of staff in the review teams in fact had experience as QAA auditors, and were in any case required to have had extensive experience of managing quality procedures in their own institutions.

The Independent Observer attended the first of the training sessions arranged by QAA. Clearly, before that could take place, decisions had already been taken as to which individuals should be invited to become reviewers. It appeared that most reviewers who attended either of the November training sessions, with the exception of the student reviewers, were already very experienced in the practice of review.

A proportion of the staff reviewers came from institutions outwith Scotland, so that inevitably their experience was derived from the older type of quality assurance processes. In practice, since ELIR is a new process in Scotland, this could also be said to apply to the review staff from Scottish HEIs, though as time goes on, and more staff in Scottish HEIs become familiar with the system, a disparity of experience may well develop. Several of the reviewers had themselves been closely involved in the development of the ELIR methodology.

By agreement between QAA, Universities Scotland and SHEFC, reviewers were included from England, Wales and Northern Ireland. This brought with it several advantages, including widening the potential pool of recruits, bringing a different perspective to the process and giving it a UK wide dimension. All applications were carefully monitored in the light of gender, ethnicity and disability, so the fact that, for example, women reviewers were in a considerable minority, was simply a reflection of the nominations received from institutions.

In the case of the student reviewers, in accordance with the Handbook, all were recruited from Scottish HEIs. In terms of personal qualifications, most were undergraduate students, with a small number of postgraduates, typically well advanced towards their final examinations or thesis submissions. Several had held senior positions in their respective student associations. None had ever acted as reviewers before – prior to ELIR there had of course been no opportunity for them to do so. The student reviewers were provided with an additional day’s training at the beginning of the training session. Through a combination of circumstances, only one student attended the first training session, and the balance attended the second.

6 Role of review secretaries

It was observed that there was some debate about the precise nature of the role of the review secretaries vis-à-vis the rest of the team. One secretary commented that she did not feel that she was regarded as a full member of the team, in that she was not expected to take part in the team discussions. Neither this view nor this practice were universal, however. The role of review secretaries in quickly producing clear and accurate notes of discussions was invaluable, but most reviewers would agree that the role of the review secretary went much further than mere note taking. At the feedback meeting on 27 July there was a strong feeling among review secretaries that their role required further clarification, both in relation to that of the Assistant Director, and in relation to the rest of the team.

7 ELIR training sessions

The training sessions took the form, over a period of two days (three for student reviewers), mostly of a series of set lectures delivered by QAA staff and a number of small group practical exercises and case studies. An invited speaker gave a presentation on student participation in quality scotland (sparqs). Specially prepared documentation was provided, including a mock ELIR report for the University of Montrose.

Reviewers were invited to comment to the Independent Observer on how effective they had found the training sessions, and in particular whether they felt at the end of the session that they were confident of their ability to carry out the duties of a reviewer.

Most of the feedback received was very positive, and generally only minor suggestions for improvement were put forward. One reviewer spoke for many in describing the sessions as ‘very useful’ and QAA staff as ‘well prepared’. Another spoke of the ELIR training as ‘a positive experience. It was clear that considerable effort had gone into preparing for the event, and QAA personnel who delivered the training did so in their usual thoroughly professional manner’. Another commented on the friendly atmosphere which prevailed throughout – partly due perhaps to the small number of participants who got to know each other quickly, but also to the tone set by the trainers, both authoritative and witty. Another participant found the training session ‘enjoyable, interesting and effective. Structure and content of the sessions were very good, including blend of participation and participative workshops, opportunities for discussion and questions.’

Indeed there was a general preference for the small group sessions rather than the set piece lectures – much of the material in the lectures, it was felt, could be found by reading the documentation, while the exercises gave a taste of what the review experience would be really like. Most of the suggestions for modification of the training programme emphasised the need for as much hands-on training as possible, possibly including role-playing sessions. Specific requests included more training on chairmanship for inexperienced students, more practice in framing questions and in making judgements. One reviewer suggested the mounting of a full-scale mock review.

There was praise for the quality of the materials, which it was suggested should provide a valuable source of reference not just for the purposes of training but also for the actual reviews themselves. The University of Montrose report drew some differing reactions, some finding it an excellent guide to the kind of final report the review teams would be required to produce and of which more use might have been made in the training sessions, while others felt it smacked rather too much of the old audit style of document. At the feedback session in July, one reviewer questioned whether a sufficiently clear distinction had been drawn in training between institutional review and subject review and this is a theme we will return to later in this report. A request was also made for a full length mock version of the Reflective Analysis.

The sessions, commented another review team member, were well organised and well structured, covering all the elements of the ELIR process. As a review secretary, this individual found that the training helped in particular to clarify her role and also the distinction between the Part I and Part II visits. Other respondents confirmed that by the time they had finished the training they did indeed feel equipped to act as a reviewer.

This last point may require some qualification or amplification, since it must be remembered that, with the exception of the student reviewers, several of the participants in the training sessions already had had some experience of institutional review, both in Scotland and elsewhere. More than one reviewer questioned whether they would have felt quite so confident at the end of the training if they had not come with that experience behind them. No doubt the trainers had devised the training in the knowledge that their audience would be largely familiar with the quality assurance aspects of review and could therefore concentrate on the more distinctive and innovatory aspects. But the question puts even more emphasis on the point that where inexperienced reviewers are concerned, the maximum exposure they can receive to the actuality of the review process, the more likely they are to feel confident in their roles.

It was a feature of the November training sessions that they included staff reviewers from both sides of the border. One Scottish reviewer reported being much struck by comments from a number of participants with substantial experience of institutional audit in the English system: 'The fact that they really felt the difference in the ELIR method was significant. Also that they were positive and saw it as a major improvement'. One other respondent was not so impressed. He claimed not to be 'convinced that that the non-Scottish reviewers were fully aware of the Scottish system, and as 50 per cent of reviewers may be English it is very important that they do not slip into audit mode'. He understood, however, that this issue had been addressed in the second November session.

The student reviewers, of course, could not be expected to have had significant experience of either system of institutional review, and although they were usefully provided with an additional day’s training before the staff reviewers arrived, it is not altogether clear how effective that extra day was in practice. One of them later commented that the students were probably more familiar with quality enhancement than with quality assurance, and that perhaps more coverage of the assurance part of the process should be provided for them. He personally had had to go through a very steep learning curve to cope with the demands of the review process on the ground.

The issue of the role of the student reviewers will be discussed in more detail later in this report, but it is worth considering at this point how the extra training day might be used to better effect. Probably more time should be given to hands on case study work, but it may be that no amount of training can fully substitute for the practical experience enjoyed by the staff participants. At the July feedback session, student reviewers who had been members of review teams concluded that they learned most by actually doing the job. QAA officers have indicated that the first day of training of student reviewers was revised in the June training session.

At the end of the training sessions, participants were asked by the training staff to complete evaluation questionnaires. One such participant has specifically asked for feedback on those returns, which was one of the objects of arranging the meeting of all trained reviewers at the end of July.

8 The Reflective Analysis

It was very clear both from the explanation in the Handbook (paras 40-42), and from personal observation that the Reflective Analysis document lies at the heart of the ELIR process. On the one hand it provides the institution with complete freedom to cover the full range of its assurance and enhancement activities, while on the other it serves as the base line for the review teams in preparing for the institutional visits. (One institution commented, perhaps to their surprise, that they had found the exercise of producing the Reflective Analysis beneficial in its own right, as encouraging the institution to think hard about their own processes and activities.)

The practice by which individual reviewers could post their comments on the Reflective Analysis on the ARCS website, thus both saving time at the Part I visit, and allowing themselves and the Assistant Directors to reflect on these comments in advance of the actual meeting, was commended by reviewers as an effective way for them to exchange information quickly and so become involved as a team at an early stage. The system could be made more efficient, it was suggested, if team members were alerted by email when new material was posted on ARCS (It was noted, however, that access to ARCS was difficult for anyone not equipped with broadband). It was interesting as an observer to see how far the comments were consistent across a number of reviewers, and also how far it was possible to read between the lines of the Reflective Analysis to get a feel for the effectiveness of an institution’s own quality procedures.

As guidance for institutions the Handbook contains three paragraphs of advice, plus a suggested outline of the Reflective Analysis document. Institutions seemed generally content with this level of guidance, though most would have liked to have had more time to prepare it. One reviewer commented that in his case he would have liked more information to be included about the institutional structure – the structure of academic schools and the chain of managerial command. He suggested that QAA guidance on the Reflective Analysis in future might stress the need for a full picture of the institutional administrative and managerial systems.

The common pattern was for institutions to set up an ad hoc ELIR Planning Committee, on which students would be represented, most usually by senior officers from their students’ association. Since the ELIR process is about how institutions manage quality in the context of the student learning experience, in preparing the Reflective Analysis they are encouraged to be as candid as possible about any problems they may have encountered, as well as about their successes and achievements. One institution made a virtue of declaring that it had decided to adopt a ‘warts and all’ approach to the production of the Reflective Analysis, and this kind of approach is to be strongly commended as the only way in which real progress towards enhancement can be made. Most, but not all, institutions seem to have understood this and acted accordingly.

One institution called for earlier feedback on the Reflective Analysis to enable it to prepare fully for the visits. Officers are satisfied that in this case and all others, feedback was produced accordingly to the stated timetable.

9 Institutional visits

Part I

It was explained in an earlier section of this report that the principal criterion to be applied to each component part of the ELIR process was whether or not it could be said to have achieved its broad objectives. The primary objective of the Part I visits is to enable the review team to identify the broad themes which will underpin the agenda and meetings in the Part II visit, and to pinpoint the documentation which will be needed to support them. In that sense all the Part I visits achieved their objective, but equally there are other related issues worthy of comment.

Broadly speaking, the format of each of the Part I visits was similar. On the first half day, a series of presentations took place which had been organised, and were delivered and chaired by, the host institution. How the institution conceived these presentations varied, with some providing a wide ranging overview of the institution’s mission and functions, while others focussed more specifically on assurance and enhancement issues. The latter was preferred by the review teams who felt that in some cases too much time had been spent on setting out a ‘shop window’, rather than addressing the issues of enhancement of student learning. More guidance, it was suggested, should be provided for institutions about the presentations, which should do no more than fill in any gaps in the Reflective Analysis. It was not necessary for every institution to fill the whole three hour period. Each institution had also provided a comprehensive collection of relevant documentation to support the points made in the Reflective Analysis.

For some reviewers, though not for all, the sheer quantity of the documentation proved a real problem, even bearing in mind the need to be selective and to concentrate in the first instance on areas relating to the part of the final report assigned to the particular reviewer. For these people, not enough time was built in to the programme for reading the documents. One suggestion received was for documentation to be provided in advance of the Part I visit, either on a CD or a secure website. If additional documentation had been identified for the Part II visit, then the team should arrange their timetable of the Part II programme to allow for document reading.

Under the guidance of the Assistant Director, all groups divided responsibility for each section of the final report among themselves, and ensured that adequate coverage of the sections was achieved in the course of the meetings with institutional representatives. It was recommended by several team members as an aid to efficiency that at meetings with institutional representatives the number of the latter should not exceed eight. In all cases, the review team felt confident at the end of the Part I visit that they had identified the key themes to underpin the meetings and document selection in Part II.

From the point of view of the institutions themselves, most were happy with the way the Part I visits went, and felt that they had been given an adequate opportunity to put their case across. Two of the four, however, felt that the Part I visit would have worked better if there had been some kind of pre-meeting to set the scene for all those involved. This, it was argued, would provide an opportunity for informal discussion of the way the team was going to approach the various Part I meetings, and for discussion of the institution’s presentation session. The importance, in the light of a review process focused on enhancement, of establishing a dialogue between review team and institution was emphasised by representatives of both. Two institutions would have liked to have included a tour of their physical premises in the Part I programme.

It is for officers to consider whether the programme could be stretched to find room for these additional items. It seems unlikely that the time spent on an institutional tour could throw much light on the ability of the institution to manage the quality of the student learning experience, though selected case studies might say something about practical outcomes. That is why, presumably, encouragement is given to include such case studies within the Reflective Analysis. But on the general issue of the desirability of initiating a dialogue between the team and the institutional representatives, the suggestion of a brief and relatively informal pre-meeting has much to recommend it.

A good deal of the responsibility for establishing the right atmosphere during the institutional visits obviously lies with the review team itself, and it is gratifying to note that teams were regularly commended on the way they conducted the business, in an efficient, timely and courteous fashion.

Part II visits

The Part II visits were preceded by the preparation and delivery to the institutions of the theme letters. There was a comment from more than one institution that the theme letter was too general to allow them to plan their representation in the Part II properly, and they would like to have more specific guidance in advance of the topics to be discussed in the Part II. From the point of view of the teams, an issue in dealing with those themes was whether to organise the Part II meetings by theme, or by category of staff and students. On the whole the former was preferred, leaving the choice of the appropriate representatives to the institution itself.

The Part II visit was very intensive. Long hours were worked, with little time for relaxation, though the dinners in the evenings provided a useful opportunity for more informal discussions of the issues, as well as for team building. On reflection, several teams questioned whether they should have had so many meetings. On one occasion pressure of time led to the decision to split a team into two sub-groups, and at the July session this practice was generally agreed to be undesirable.

The dedication and professionalism of the reviewers was commented on by more than one senior manager from the institutions, and there was a general perception that the dialogue begun in Part I became fully operational in Part II. In general, the Part II visits clearly achieved their declared objectives.

10 Student reviewers

The inclusion of student reviewers represents a new and very welcome development of the quality enhancement process. Under the terms of paragraph 69 of the Handbook, 'The student reviewers will have particular, although not exclusive, responsibility for pursuing lines of enquiry relating to the student experience and the effectiveness of student involvement in the institutional processes for the strategic management of quality enhancement'.

The inclusion of student reviewers sends a very clear signal to everyone with an interest in the improvement of the student learning experience that the views of students are to be taken seriously by QAA and institutions alike. But the move must be more than symbolic. The section of the Handbook quoted above was taken by all to mean that the student reviewer in each of the teams was not there just to deal with student-related issues, still less as some kind of token, but was to be regarded as a full and equal member of the team. It is fair to say that this was how they were regarded by the other members of the team, though it was recognised that they might need some extra support and guidance if they were to be able to play a full part in the proceedings. In all cases they were given every opportunity to take part and ask questions about any of the subjects under discussion.

At the same time, however, it was clear, both from personal observation and from the comments from some institutions, that the actual participation by student reviewers was very variable. While some appeared to be fully in command of the material and the issues, and made a significant contribution to the proceedings, others were perceived to be much less effective.

If this was so, then it cannot simply be explained by lack of hard work, ability, or dedication by the individual students. Almost by definition they found themselves at a disadvantage in relation to other more experienced members of the team, and the deficit in experience was not wholly made up by the extra day’s training at the beginning of the process. Nor was the undoubted advantage they had of being more directly in touch with contemporary student views and priorities always a sufficient counterweight. One institutional commentator spoke of the impossibility of squaring this particular circle. This seems unnecessarily pessimistic. It would be preferable to talk about closing the gap. It is suggested that there are at least three ways in which this might be done.

The first of these is in the area of selection. In the selection of student reviewers QAA rightly chose those with considerable experience of being a student in their own institutions. This seems very sensible. While there might be an attraction in selecting students at an earlier stage in their academic career, the lack of experience of such students would greatly hamper their ability to take on the reviewer role. But as a selection criterion, other things being equal, one might single out the possession of a considerable degree of self confidence, and the ability quickly to get on terms of equality with the other reviewers. To make sure that this quality is possessed by appointees, where practically possible, they should be called for interview before final decisions are made.

The second area for consideration is in training. QAA is already aware of this of course, and as noted above, provided an extra day’s training for the student reviewers at the beginning of the training session. Either this was not long enough, or it somehow failed to supply just what the students really needed. As noted earlier, one of the student reviewers, himself among the most experienced in the field of quality management, suggested that current or recent students knew more about enhancement than they did about assurance. Another student reviewer asked for more training in how to go about the preparatory work prior to the Part I visit, and how to deal with the documents. It would be worthwhile, now that the first year’s cycle is complete, for the student reviewers to be asked specifically to consider what improvements they would like to see in the training schedule.

The third area is that of experience. How can the gap in experience between student reviewers and the rest be filled? Of course this is not really an issue only for student reviewers. QAA has already on occasion appointed staff reviewers with no previous experience of review and audit, and will have to again in the future. There might be more than one answer to that question. Again it may be that the training sessions should be oriented more to practical hands-on exercises and case studies than they are at present. Where possible student reviewers might be employed on more than one review. Whether they can or not, consideration should be given to enabling them to sit in as observers on actual reviews, before expecting them to take part in a fully fledged review of their own.

11 Assurance vs enhancement?

To head this section in this way is not intended to be deliberately provocative, but rather to draw attention to a key issue which lies at the heart of the ELIR process. It is an article of faith that the ELIR process does not represent an abandonment of quality assurance, but rather an evolution from the consideration of the assurance of quality to focusing on an institution’s ability to manage the enhancement of quality. ELIR is not an audit process on the old model, but it does require institutions to carry out their own programme of internal subject reviews in accordance with SHEFC published guidelines, repeated in Annex 5 of the Handbook. But as the Handbook makes abundantly clear, ‘national and institutional activities in relation to managing quality and standards have been changing to focus more explicitly on the enhancement of the learning experience of students.’ This leads to the all important final bullet of paragraph 6 of the Handbook which describes the essential nature of the institutional review process.

'This is an enhancement-led peer review process which, while providing information on the security of the institution’s management of quality and standards, is focused on the institution’s strategic management of quality enhancement.'

There is a certain amount of evidence that some of those involved in the institutions in this first round of ELIR visits were unsure as to where the emphasis lay or should lie as between assurance and enhancement. Reference was made in one institutional response to 'the tension which inevitably arises from the dual nature of ELIR – on the one hand it has its focus on enhancement, but this is coupled with its underlying judgemental aspect regarding quality and standards'. Another institutional response commented on the rather aloof stance of the review teams, at least in the early stages of the Part I visit, which was associated in the minds of some staff with an old-style audit philosophy. Within the review teams themselves there was a clear understanding that, while the focus may now be on enhancement, that did not remove their responsibility to satisfy themselves of the institution’s ability to assure quality and standards. Nevertheless the evidence suggests that there is still some work to be done to make sure that the institutions are equally clear on this issue, and that any misperceptions are corrected.

12 Final reports

The preparation of the reports and their despatch to the institutions proceeded according to the planned timetable. At the time of submission of this report, the draft final reports had been sent out to three of the four institutions under review, and only general responses had been received. In terms of ease of preparation, one reviewer noted that there was nowhere in the report template for commenting on the effectiveness of the leadership and management of the institution.

13 Administration and time management

It is very satisfying to be able to report that the administrative arrangements relating to all aspects of the ELIR process were highly commended by all those involved – institutions as well as review teams. The staff of QAA and the staff of the participating institutions deserve great praise for their efforts in this regard. The work of the Assistant Directors was spoken of very highly by many reviewers, and appreciation was expressed for the care with which the composition of the review teams was decided upon, to include reviewers who from their own background could be expected to understand the culture of the institution under their review. Similarly, the review teams managed their part of the business with great efficiency with all sessions running to schedule. The review secretaries were also commended by all concerned for their professionalism.

Some comments were received about time pressures within the system as a whole. Institutions themselves felt under considerable pressure to produce the Reflective Analysis within the time frame allowed, and others would have liked a longer period between the Part I and Part II visits (au contraire, one reviewer asked for a shorter period, to keep up the momentum). There appeared to be no slack in the timetable of any of the visits.

14 Conclusions and recommendations

To a very high degree, the various activities which make up the ELIR process achieved their operational objectives. The trainees were broadly satisfied that their training had prepared them well for the task; the Part I visits led to the desired result of identifying the agendas and documentation for the Part II; the Part II visits ended with the preparation of the draft letters to go to the institutions; and the whole process was completed with the preparation of the final reports.

That said, there are clearly some issues which would repay further consideration. It is therefore recommended that the QAA consider the following issues.

Are we quite sure that the Scottish HEIs really understand fully the philosophy and practical implications of the enhancement-led approach to institutional review? Is further guidance on this issue required in either the training programme or information to institutions? (Section 11)

Are there ways in which the initial guidance to institutions can be improved, clarified, amplified? (Sections 8, 9)

Are there ways in which the operational efficiency of the process can be improved? (Section 9)

Is there really a problem with the supply, selection and digestion of documentary evidence, and if so, how might it be addressed? Could documents be supplied in advance of the Part I visit? (Section 9)

Is QAA satisfied that the timing of each part of the process is right, and is enough time given for both the teams and the institutions to prepare? (Section 13)

Are there ways in which the selection, training and provision of relevant experience of the student reviewers can be improved? (Section 10)

Can further clarification be provided about the respective roles of the review secretary and Assistant Director, and their inter relationship? (Section 6)

David Swinfen
29 July 2004