Institutional audit - student observer pilot
This report provides an analysis of the feedback provided by student observers on their experience and perceptions of observing an institutional audit. The feedback was provided through evaluation forms and a focus group.
The report also contains feedback derived from discussions with auditors who included student observers in their teams in spring 2008.
The Quality Assurance for Higher Education (QAA) is grateful to those individuals who gave up their time to participate in institutional audit as a student observer, and provide a full evaluation of their experience. In addition, QAA is grateful to those audit teams and institutions that accommodated a student observer.
Introduction
During the spring 2008 institutional audit visits, QAA undertook a pilot project to include student observers as members of six audit teams. The purpose of the project was to gather feedback from the observers and auditors involved on the feasibility of including students as full members of audit teams. The feedback will inform any decision by QAA to include students as full members of audit teams, as part of its approach to student engagement.
Observer members of audit teams were selected from a variety of backgrounds, and included former students, current student representatives and one participant who had previously undertaken the role of a student auditor through the QAA Scottish Enhancement-led institutional review (ELIR) process.
During the audit, student observers were asked to complete an in-depth evaluation of their experience, reflecting on the various aspects of the audit process in which they were involved. This evaluation included both the briefing visit and the audit visit phases of the process. The form was supplemented by a focus group with student observers, providing an opportunity to further explore their experiences of review and their views on the potential contribution of students as members of audit and review teams. Feedback was obtained from members of the six audit teams in question through a facilitated discussion group at the Summer Solstice June 2008 - a meeting open to all QAA auditors.
The outcomes of the evaluation will be used to inform QAA’s student engagement strategy, specifically in relation to student membership on audit teams.
Key findings
In the main, the student observers were very positive about their experience with audit teams. All agreed that including students as members of audit teams would be a beneficial move, providing an enhanced focus on student issues within the context of the audit. Student observers reported that they felt comfortable and confident with the process, and they felt they would be able to participate as a full member of the audit team. However, they recognised that student auditors should not be marginalised into focusing solely on student ‘issues’, but rather should add a student perspective to the entire process.
It was reported that many of the issues relating to student members, in respect of training and support, were not dissimilar to those relating to new auditors. Many of the comments included in the feedback should be understood with that in mind and should, therefore, not be seen as a constraint to the introduction of student members of audit teams.
Auditors also commented favourably on the experience of having student observers on the team during the audit process, and on the potential opportunities afforded by including students as full members of audit teams. However, although there was a great deal of support expressed for greater inclusion of students in the audit team, auditors were concerned about the impact on the nature of peer review as currently perceived and were keen that the integrity of the audit process should not be comprised.
All but one of the student observers were confident about the contribution that they felt they could have made to the audit if they had been a full member of the team. The respondent who was least confident associated this with a perceived lack of experience and knowledge of institutional procedures and quality assurance. As a result, they felt that they were not sufficiently experienced or confident enough to contribute sufficiently to the team at all times during the audit. However, the other student observers accepted that their contribution to the audit and the team was not the same as other auditors’ contributions. Nevertheless, they recognised the valuable role they could play in respect of providing a student focus to the audit. On this basis, it was felt that the role and remit of a student auditor would need to be
clearly defined.
The student observers expressed a great deal of confidence that sufficient numbers of students with appropriate knowledge and experience would be willing to participate. However, they felt that QAA would need to be explicit about the time commitment involved.
Recruitment and selection
Recruitment and training were two other areas identified by student observers as crucial to the successful inclusion of student members in audit teams. The observers felt the role and remit of student members should be set out explicitly and clearly at the outset. This would give a clear indication of what would be involved in participating, particularly in relation to the time commitment involved. The type of student who would fulfil the role of a student auditor was also mentioned in audit team feedback. It was stated that there would need to be a careful and detailed specification of the role of the student auditor, and an associated clear specification of requirements in terms of skills and experience.
The student observers felt that insufficient information was provided before the review in terms of the demand on their time. It was suggested that such expectations should be made clear before reviews take place. Assuming that this information is explicitly stated at the outset, observers felt a student auditor should be able to manage the demands of reading the documentation, undertaking the audit and contributing to the report. Although it was acknowledged by the student observers that there may be issues for final year students in relation to assignments and examination periods, they felt that in the main this should not present a significant challenge to including a student auditor on the audit team. It was suggested that a sabbatical officer may be better placed to assume the role of a student auditor. It was also suggested that a recent graduate could undertake the role of a student auditor.
Student observers indicated that a number of ‘types’ of student would be suitable as members, including current and recently graduated (within three years) students and sabbatical officers, similar to those recruited for membership of ELIR teams in Scotland.
As per current recruitment criteria for auditors and reviewers, it was considered appropriate that student members should have recent involvement in institutional decision making. For example, experience as school or faculty representatives, sabbatical officers, members of periodic review teams or other similar roles would display knowledge and understanding of the higher education (HE) sector’s approach to managing quality and standards, along with an understanding of analytical and critical approaches.
Issues to consider:
- QAA should recruit current students, sabbatical officers and recent graduates to serve as student members of audit teams
- the criteria for recruiting student members should include:
experience of involvement in institutional decision making (for example as a faculty or senate representative or sabbatical officer)
evidence of a broad knowledge of the HE sector
evidence that they have an analytical and critical approach - the expectations and typical time commitment for auditors and reviewers should be set out clearly from the outset.
Training and support
All student observers felt that training would be crucial to the involvement and success of the student auditor and should include training on:
- HE and QAA processes
- Academic Infrastructure
- the QAA handbook
- note taking
- questioning and etiquette in meetings
- report writing
- acronyms.
The majority considered that some form of support arrangement should be available for the student auditor. There were mixed views as to whether this should be a formal or informal relationship, and on who should provide such support. During the student focus group there appeared to be a preference for the QAA Assistant Director to provide support and guidance (if required) during the early stages of the audit, in order to ease the student auditor in to the process. It was noted, however, that support required by a student member was likely to be similar to the needs of a new auditor, and could be met through informal working arrangements and relationships within the team.
Comprehensive training was considered crucial to clarifying and managing the expectations of a student auditor and providing them with the tools required to undertake the role. The student observers acknowledged that they had not participated in full QAA training and therefore had not experienced the extensive and rigorous training that is usually provided for new auditors.
Issues to consider:
- if students are to become full members of audit teams they should participate in full QAA training for auditors, including any necessary ‘top up’ training or briefing
- training should give students a clear understanding of what to expect and how to engage with senior staff in higher education institutions.
The role of students as audit team members
All student observers reported that they felt very comfortable working and socialising alongside the audit team, and were well supported by the QAA Assistant Director. However, some clarification was requested regarding expenses and payment
for meals.
All observers agreed that a student auditor would have a positive impact on the audit in respect of meetings with students. They felt that as a student auditor they could make a valuable and focused contribution in this area. However, it was agreed that the student auditor should not necessarily have responsibility for focusing on students, but should actively contribute to all areas of the audit. The auditors also concurred with this, indicating that if student auditors were to be included on the audit team, they would need to be a full and equal team member, participating fully in the entire process.
Issues to consider:
- student members should be full and equal members of audit teams and take on the same responsibilities as other members.
Documentation
Student observers reported differences of experience with respect to reading and understanding the documentation provided in support of the audit, but a common criticism centred on the use of acronyms. Observers felt that there were often too many acronyms in documentation, with little explanation of what they stood for. It was suggested that a detailed glossary should be made available, explaining all the acronyms used.
All the student observers agreed that they had not experienced any significant problems in understanding and critically appraising the documentation, and did not foresee any problems for a student auditor in doing so.
Issues to consider:
- QAA should consider developing a general guide to terms that are regularly used in the higher education sector, but which may be unfamiliar to students.
Judgements and reporting
All but one of the student observers felt that student auditors should have one section of the report to write. It was felt that this would have two primary benefits. Firstly, this would give them a specific area to focus on through the audit and, secondly, that it would aid their integration with the rest of the team. The remaining observer felt that a student auditor should not have one area on which to focus, but should adopt the role of a student advocate with involvement across the whole audit process and the ability to influence various aspects of the audit in relation to the student experience.
Referring specifically to the audit report, the auditors’ feedback highlighted a number of challenges associated with undertaking this activity. They stressed that care would need to be taken over the allocation of writing responsibilities, with perhaps a need for some form of shadowing arrangement.
Issues to consider:
- as in Scotland, QAA should consider shadowing arrangements for students, so they can call on support from other auditors when writing sections of
the report.
Derfel Owen
QAA Development Officer (students and enhancement)
Jean McLaren
QAA Development Officer (Information Unit)
Mandy Nelson
QAA Head of the Information Unit
July 2008
Top