Consultation on changes to the Handbook for institutional audit: England and Northern Ireland 2006

Student membership of institutional audit teams

PDF version (227 KB)

Background to the proposals

1    In September 2007, QAA wrote to HEFCE, HEFCW, DELNI, UUK, GuildHE, HEW, HEA, NUS, NUS Wales and the National Postgraduate Committee informing them that it would be undertaking a feasibility plan for the inclusion of student members in institutional audit teams, with a view to implementing any necessary changes during the 2008-09 academic year.

2    The QAA Board of Directors had given this matter careful thought, concluding that there were three key reasons why it was the right time to introduce this change to practice:

• QAA Scotland had, for a number of years, included students as full members of Enhancement Led Institutional Review (ELIR) teams. The external evaluation of the ELIR process indicated that student reviewers had played an important and highly successful part as a defining feature of the Scottish quality assurance model, and were fully recognised and accepted as a valuable element in the process.

• As a member of ENQA (the European Association for Quality Assurance in Higher Education), QAA is expected to act in accordance with the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) set out in 2005. The ESG clearly state that the participation of students in quality assurance activities and external assessment is an expectation and this is becoming standard practice amongst ENQA’s members.

• As the higher education landscape has evolved over recent years, so too has interest in and focus on the experience of students and on their participation in the organisation of their learning, within both their institutions and the sector as a whole. It is a natural progression of such developments that students will wish to become more deeply involved in quality assurance and more directly involved in the audit and review processes.

3    In the light of this analysis, the opinions of the funding councils and sectoral bodies were sought on:

• the likely impact of student membership on the structure and function of audit teams
• the potential role of student auditors in the audit process
• the optimal profile of students to be recruited
• the proposed timescale for implementation.

4    The responses to the 2007 consultation indicated that there was considerable support for the introduction of student members of audit teams, but that further consideration of the operational detail, especially the timetable for introduction of student members, was required. Several questions were raised about the impact of such a change on the integrity of the current audit method, its introduction during an audit cycle and the operational and logistical requirements for involving students as auditors.

5     Following this consultation, QAA agreed to have further discussions with its subscribers to explain the rationale for these proposals and to explore further some of the concerns raised in that earlier round of consultation. It was also agreed to conduct an evidence-gathering pilot exercise by including student observers on selected institutional audits in spring 2008.

6     By May 2008 the student observers had attended audits and had provided full and detailed reports to QAA about their experience. They provided information about their perception of the process, about how a student might contribute as a member of the team and identified additional training and support requirements. This feedback was analysed and in July 2008 QAA organised a conference to discuss student membership of institutional audit teams in England and Northern Ireland. The conference provided an opportunity for delegates to hear about the experience of student members of review teams in Scotland, an HEI’s experience of working with students as members of internal review teams and to hear directly from the student observers themselves about their experience. All HEIs in England and Northern Ireland were invited to send representatives, and were encouraged to include student representatives. Conference materials and reports can be seen at www.qaa.ac.uk/events/smart08

7     Overall there was strong support for the proposals, but there nevertheless remained a request for further clarity in respect of the logistics and specific detail for student membership of audit teams¹. In view of the need to consult further on this operational detail, QAA’s Board has now agreed to revise its original timetable for introduction of student members.

The proposals

8   QAA is now proposing that student members of audit teams should be introduced from the academic year 2009-10. The need for an extensive programme of recruitment and training means that the first audits which will include student auditors will be those audits taking place in the spring term 2010. Some 48 institutions will be taking part in audits from spring term 2010 to the end of the cycle. Some institutions have been concerned that the introduction of student members in audit teams in the middle of an audit cycle will create a disparity in treatment of institutions. In the light of this, QAA has agreed with Universities UK and GuildHE to consult on the possibility of institutions, for the remainder of this cycle, being able to opt out of having a student member on its audit team. Were this to be the case QAA might need to decide whether to recruit an additional academic auditor to the team, or accept that different institutions would be visited by differently-sized audit teams for the remainder of the cycle [1].

9    The student auditor will be additional to the existing number of auditors, so that the audit team would normally include five auditors (including the student member) and the audit secretary [2]. It is envisaged that the role of the student member will be the same as that of other auditors, that is, the student auditor will take part in all aspects of the audit explorations and contribute to all of its findings, and will take part in the writing of the report [3].

10   Those eligible for appointment as student auditors will be students with current or recent (within two years²) experience within a higher education institution in England or Northern Ireland amounting to a minimum of one year's full time study (or its equivalent). Student union/association/guild sabbatical officers will be eligible [4].

11   As with current auditors, student auditors will be nominated by their institution and recruited against a set of published criteria (see Appendix B). They will be required to participate in the residential training provided by QAA before they may take part in an audit. Wherever possible they will be trained alongside other new and returning members of audit teams. As well as receiving the usual audit training, student auditors will receive extra sessions dealing with the England/Northern Ireland higher education quality assurance environment and an introduction to the operation of governance in HEIs. However, we envisage that most student auditors will have some awareness of national quality assurance matters. They will be required to have had experience of engaging with senior level decision-making about quality assurance and enhancement in their own institution.

12    Some minor changes to the Handbook for institutional audit: England and Northern Ireland 2006 are required to accommodate the proposed change to the audit team. These can be seen in the draft revised handbook.

13    Some colleagues in the sector have raised concerns that the introduction of students as auditors is a significant change to the nature of institutional audit as a peer review process. QAA considers a peer to be a person of equal importance, carrying out a comparable role to others within an organisation.  Given the centrality of the student experience to the audit process, and the participation of students from an HEI in the preparation for audit, QAA sees student membership of its teams as entirely consistent with this concept of peer review.  This position is bolstered by the almost universal participation throughout the sector of students as members of HEI deliberative bodies, and increasingly as members of internal review teams.

14    QAA is currently reflecting on the method which will follow the end of the current Institutional audit method.  It is intended to include full student participation (as auditors, or whatever role the new method demands) in that method from its inception, expected in 2011-12.

Consultation questions

15    You are welcome to provide comments on any aspect of the operational detail. Answers to the following questions are specifically invited. Numbers refer to cross references in the documents.

[1]    Do you agree that from spring 2010 all audits should include a student auditor, or should an opt-out be available until the end of the audit cycle? If an opt-out option is allowed, should an extra auditor be recruited to ensure teams of equal sizes?

[2]    Do you agree that the student auditor should be an additional member of the audit team?

[3]    Do you agree with the proposed role for the student auditor?

[4]    Do you agree with the categories for eligible students? Are there others?

Appendix B

[5]    Do you agree with these selection criteria? Are there others that you would wish to suggest?

[6]    Do you consider these to be the main requirements for training of a student auditor?

---

¹ The feedback can be read in full at Appendix A.

² This is analogous to auditors remaining active until two years after retirement from their academic post

---

Appendix A

Conference to discuss student membership of institutional audit teams in England and Northern Ireland – small group sessions (18 July 2008)

Introduction

1   During July 2008, QAA hosted a conference to discuss the proposed student membership of institutional audit teams. In total, 107 delegates attended the conference. These were primarily comprised of student representatives, sabbatical officers and staff from higher education institutions, typically with an interest in student engagement.

2    The purpose of the event was to update delegates on progress regarding student involvement in quality assurance activities, and receive feedback on the proposals from the delegates. The event also sought to disseminate the findings of case study examples of student involvement, from both the English and Scottish perspectives, and to hear direct feedback from the student observers who had participated in recent institutional audits (as observers).

3    As part of the event, delegates were given the opportunity to take part in small group discussions which were intended to provide participants with the opportunity to discuss the proposals for student membership of audit teams. This report provides the key points coming out of the small groups.

4     Overall, there was support for the proposals, but never-the-less it was noted that pockets of opposition still remain within higher education institutions and that further clarity is required in respect of the logistics and specific detail for student membership of audit teams.

Findings from the small group sessions

Recruitment, selection and training

5    The selection, recruitment and training of student auditors was considered to be a crucial aspect of the process, and should not be rushed but properly planned and executed. It was noted that students selected to act as student auditors would need to have relevant experience, knowledge and skills in order to fully participate in the audit process, and that clear criteria for selection would be required.

6     Some recommended that institutions/SU/NUS should carry out the initial screening of applicants based on clear criteria to ensure consistency of approach across institutions in putting forward possible candidates. Others however, were not sure whether the selection should be based on a nomination process or through direct applications, although it was generally agreed that recruitment should initially be through the institution with QAA making the final selection. If applications were received directly from students it was proposed that the institution should be requested to provide a reference.

7     It was suggested that it might be necessary to provide an incentive to prospective student auditors in the form of ‘credit’ for undertaking an audit or a payment towards fees.

8     It was noted that the recruitment and selection of student auditors is likely to be quite expensive as each student auditor would only be able to undertake a limited number of audits. The involvement of students in the audit process should be seen as a partnership and investment between QAA and the institutions and there would need to be consistency across institutions in how they support their students in carrying out the role as a student auditor.

9    There was some discussion regarding the ‘type’ of students who would be most suitable to act as student auditors, and how to ensure the diversity of the student body is reflected in the selection and recruitment process. Some considered that a student should only audit an institution similar to their own. There were mixed views regarding the use of sabbatical officers as student auditors, with some considering they would be the most appropriate while others considered that they would not necessarily be the most appropriate as they had been elected on the basis of student popularity. It was proposed, that a sabbatical officer be trained who could be available to undertake an audit at short notice if required, to provide cover if a student auditor drops out of the team at the last moment.

10   It was considered important that the job specification be very clear regarding the time commitment that would be required of a student undertaking the role of a student auditor.

11   The training that student auditors would receive was seen as central to the success of including students as active participants of audit teams. Training would need to allow for students who come from different backgrounds and with varying experiences, which it was noted could be challenging and expensive. It was suggested by some that a two-stage training process be adopted where students receive an initial session with only student auditors and a second stage with other auditors, as it was considered important that part of the training should be conducted where the auditors and student auditors spend time together. It was noted that in Scotland the student auditors receive the same training as the other auditors but are provided with supplementary support if required.

12   It was recommended that training should cover:

• general background information on the sector
• emphasis on the role of the review of management and processes beyond the programme level
• report writing
• acronyms and terminology.

13        It was noted that where HEIs provide good training for students involved in quality assurance matters at the institution; this would provide a good basis for the QAA training. It was also suggested that the introduction of student auditors to the audit process would encourage institutions to involve students more in their internal review processes.

The role of the student auditor

14   In general there was support for student auditors to be included in audit teams, as full members of the team. It was considered that this would bring a student perspective which would strengthen the process and allow for students to be considered as peers and involved in shaping their own education. It was noted that students in many organisations are already active members of academic committees and are accepted as peers in these forums. Some however, questioned whether students could be considered as ‘peers’ and if the process would still be peer review, while others wondered if it would matter that the process was no longer strictly peer review.

15   It was also noted that students studying at an institution that was undergoing an audit felt more comfortable and reassured being questioned by a team which included a student auditor.

16   There were some reservations expressed regarding how the presence of a student on the team might skew the discussions and that they might be unable to participate in discussions relating to management of standards or contribute to discussions relating to judgements. Some also expressed concerns with respect to the writing of a section of the report by a student auditor, both in terms of the time commitment and the skills that would be required. However, it was noted that the AD would have a key role in editing the report and making any necessary amendments.

17   There was also some concern regarding the comparability of experience that would exist across a pool of student auditors as their experience would be limited by the type of HEI they had or were attending.

18   It was further proposed that student auditors’ involvement in the audit process should not be limited to just the ‘student areas’ but that they should have the opportunity to participate across all areas.

19   It was suggested by some that to encourage HEIs to accept and support the proposals of using student auditors, the benefits should be emphasised in terms of outcomes and process benefits, and through the sharing of best practice. It was also suggested that QAA undertake some research to demonstrate what the real cost/benefit from the introduction of student auditors would be for institutions.

Timing

20    With reference to when student auditors should be introduced to the audit process, mixed views were expressed regarding whether they should be introduced mid-cycle or should wait until the beginning of the next round of audits.

21    It was noted that it would be important that the introduction of student auditors to the audit process should not be rushed into, but that thorough preparation in recruitment, selection and training of students would be essential to ensure the success of the scheme and reduce the risk of it failing. It was also noted that institutions would need time to get use to the idea and time to prepare and plan for their audit if student auditors were included. It was also suggested that some acceptance testing be undertaken to help change the mindset of those institutions that might be sceptical of the scheme.

22   It was suggested by some that the inclusion of student auditors on audit teams should be piloted during the next couple of years in order that the process can be refined and allow time for institutions to become familiar with the idea, in time for the next round of audits. Others proposed that the introduction of student auditors should take place mid cycle but that institutions could be given the opportunity to ‘opt out’ if they wished. There was some concern expressed regarding the comparability of process if student auditors were introduced mid cycle for those institutions yet to be audited in this current cycle.

Practicalities of introducing student auditors to the audit process

23   One of the main areas of concern discussed in regard to the introduction of student auditors was the size of the pool of students that would be required and how many audits it would be feasible for a student auditor to carry out. If they could only undertake one or two audits, this would result in the scheme being very expensive in terms of training and recruitment costs. It was noted however, that former student auditors could act as support for new student auditors, which would help alleviate the likely high turnover of student auditors.

24   Another concern expressed through the small groups, was how representative student auditors would be of the student population as a whole, and as a consequence the concept of ‘peer review’ might be questioned. It was suggested that this could potentially lead to divisions within the pool of student auditors. It was noted that part time students might find it particularly difficult to make the time commitment that would be necessary to contribute to the process.

25    In order to become a student auditor it was suggested that students would need to have completed or be near the end of their second year and have had some involvement in quality assurance processes. However, third year students might find it difficult to commit the time required to undertake an audit, particularly around exam times. It was proposed by some that sabbatical officers would therefore be best placed to undertake the role as they would have time available to commit to the task.

26   It was considered that the composition of teams would be important to ensure the right mix of knowledge and experience. The size of the team was also discussed, and it was questioned whether the student auditor would be an additional member of the team or if they would replace one of the auditors, so keeping team size the same as it is currently, and save on costs.

Appendix B

Recruitment and training of student auditors

Introduction

1    Institutional audit is a peer review process. Student auditors are appointed by QAA from nominations made by UK higher education institutions (HEIs). The success of student involvement in the audit process relies on the recruitment and appointment of individuals who have experience of participating in substantive quality assurance activities in their own institutions. The Review of the Quality Assurance Framework published by HEFCE in July 2005 recommended that institutions should ‘be encouraged to nominate members of their staff of the highest calibre to serve as audit team members’. We expect that HEIs will wish to adhere to the spirit of this statement when nominating student auditors, so that the quality of our work will itself be assured.

2    We strongly believe that the credibility of audit depends upon the currency of the knowledge and experience of our audit teams. At the date of nomination, therefore, nominees must have current or recent (within two years) experience as a registered student within a higher education institution in England or Northern Ireland amounting to a minimum of one year's full time study (or its equivalent).

Selection criteria for all auditors

3   When selecting all potential auditors, we shall take into account the extent to which nominees are able to demonstrate:

• good oral and written communication skills.
• an ability to work with electronic and/or web-based communication systems
• an ability to work effectively as part of a team
• an ability to adhere to agreed protocols, procedures and deadlines
• an enquiring and sceptical disposition.

Additional selection criteria for student members of auditor teams

4   When selecting students as potential auditors, we shall take into account the extent to which nominees are able to demonstrate:

• current or recent (within two years) experience as a student at a higher education institution in a UK HEI amounting to a minimum of one year's full time study (or its equivalent)
• experience participating, as a representative of students’ interests, in the management and monitoring of academic standards and the quality of educational provision at institutional and/or faculty or school level in UK higher education
• general awareness of the diversity of the higher education sector, beyond their own institution and awareness of the arrangements for quality assurance and enhancement in the sector [5].

5    It should be noted that extensive and detailed training is provided to build student nominees’ experience and knowledge. In the first instance, QAA is primarily looking for applicants who have the ability to build on experience they already have (which may have been gained from a variety of sources) and to assimilate new information that will be provided during induction and training.

6    In making our selection from those nominated, we shall try to ensure that each cohort of auditors reflects sectoral discipline, geographical, gender and ethnic balances. This can only be done effectively if all HEIs nominate suitable candidates. In constructing audit teams we take account of the need to have a sufficient spread of knowledge and experience for an informed view to be taken of primary evidence relating to the academic standards of awards and the quality of educational provision and its enhancement.

Training

7   All auditors will undertake training, provided by QAA, in the process of audit. The training of newly appointed auditors will last three days. The purpose of the training is to ensure that all auditors:

• understand the nature of the England/Northern Ireland higher education quality assurance environment and are aware of the variety of quality assurance mechanisms in HEIs
• understand the aims and objectives of the audit processes
• understand the procedures involved
• are able to build upon their knowledge and experience of quality assurance methods
• understand their own roles and tasks, the importance of team coherence, QAA’s expectations of them, and the rules of conduct governing the process; and
• have an opportunity to explore the techniques of data assimilation and analysis, the development of programmes for visits, the construction and testing of hypotheses, the forming of judgements and statements of confidence, and the preparation of reports [6].

Description of services to be provided by institutional auditors

8   It is expected that, where possible, institutional auditors will participate, if requested, in three audits during a two year period.  However, we realise that pressures on students' time may mean that most student auditors will be unable to meet this expectation, and this will be taken into account in QAA’s recruitment schedule.

9    The length of an audit will vary according to an agreement between QAA and the institution being audited. All audits will include:

• preparatory work
• briefing visit to the institution
• work between visits
• audit visit to the institution
• drafting and revision of sections of the report.

10   Time required for the reading of documentation and travelling is included in the overall fee for the audit. Advance documentation will be received some 10 weeks before the audit visit. Auditors are required to submit their text for the draft report within two weeks of the end of the audit visit.

11   All communications connected with an audit, written or oral, must be treated as confidential to the team and QAA. Written communications relating to the business of an audit must be made through the Academic Review Communication System (ARCS), which is a secure section of QAA’s website.

12  Auditors are required to adhere to the procedures described in the Handbook for institutional audit England and Northern Ireland 2006, and to follow any additional guidance and/or instructions given by the Assistant Director (AD) coordinating the audit.

Preliminary work (after receipt of the institutional briefing paper)

13   An auditor will be expected to analyse the briefing paper and any other documentation provided, to gain a provisional understanding of the institution and to identify possible themes and audit trails for exploration at the briefing meeting, with a view to continuing these themes into the audit visit, or to closing them in the light of discussion at the briefing meeting.

Briefing visit to the institution

An auditor will be expected to:

• work with the audit team and the AD in private meetings to agree the main themes for testing during the visit, and to select and study evidence to support the testing
• lead and participate in discussions with staff and students of the institution on themes as agreed by the audit team as a whole, with due attention to the protocols of discussion as a peer group
• form preliminary views on the effectiveness of the institution's approach to the management of quality and standards, based on discussions and study of evidence
• propose topics and themes for investigation during the audit visit, and agree with the audit team and the AD the programme to be negotiated with the institution
• propose any further evidence required for testing during the audit
• keep sufficient personal notes to supplement the audit secretary's record of the meetings, for the purpose of undertaking the agreed writing tasks with proper attention to accuracy and reliability of evidence.

Work between visits

An auditor will be expected to:

• draft inter-visit notes on topics agreed with the audit team and its AD as a starting point for the audit visit
• exchange views with other audit team members and the AD on the inter-visit notes
• read additional documentary evidence provided by the institution in preparation for the audit visit
• assist in the identification of audit trails.

Audit visit to the institution

An auditor will be expected to:

• work with the audit team in private meetings to draw conclusions from the themes being explored during the audit visit, using oral and documentary evidence to underpin those conclusions
• lead and participate in discussions with staff and students of the institution on themes, as agreed by the audit team as a whole, with due attention to the protocols of discussion as a peer group
• through audit trails, contribute to the analysis of the effectiveness of institutional processes for assuring the quality of educational provision and the security of academic standards
• make judgements on the effectiveness of the institution's management of quality and standards, from the oral and documentary evidence gathered during the audit
• comment on institutional-level implementation of the Code of practice for the assurance of academic quality and management in higher education and on any difficulties/divergences in that implementation
• comment on institutional-level implementation of The framework for higher education qualifications in England, Wales and Northern Ireland, subject benchmark statements and programme specifications
• comment on the quality of information provided by the institutions - its accuracy and completeness
• comment on the institution’s arrangements for maintaining appropriate academic standards and quality of the provision of postgraduate research programmes
• keep sufficient and appropriate personal notes for the purpose of undertaking the agreed writing tasks with proper attention to accuracy and reliability of evidence.

After the audit visit

An auditor will be expected to:

• produce draft sections of the audit report, as allocated at the briefing visit, to agreed deadlines and to an agreed electronic format, working to the standard report template, and underpinning all conclusions and judgements with reference throughout to oral and documentary evidence
• comment on the consolidated draft report, responding to any requests from the AD for additional text or reference to evidence, to agreed deadlines
• comment on the institution's responses to the draft report, responding to any requests from the AD for amended text, a review of judgements or confirmation of evidence as necessary, to agreed deadlines.