The evaluation of the IQER pilot reviews

• A Report to HEFCE and the QAA on an Evaluation of the IQER Pilot Programme

Introduction

During 2006-07, QAA carried out 15 pilot reviews using a new method designed for higher education in further education colleges (colleges), Integrated quality and enhancement reviews (IQER). The aims of this pilot project were to assess the effectiveness of the methodology and to evaluate the benefits to institutions, staff and students. A detailed evaluation of the pilot programme was carried out in order to inform any changes or developments to the proposed method of review prior to the introduction of the full programme of IQER in spring 2008.

This report seeks to provide an overview of the key findings from both QAA’s own and the external evaluation of IQER. The findings of both evaluation activities have been used to inform revision to the method, which has subsequently been approved by the Higher Education Funding Council for England (HEFCE) Board for implementation in spring 2008.

In order to ensure maximum benefit from the evaluation of IQER, HEFCE and QAA jointly commissioned an independent organisation to carry out a detailed review of the method and provide a report with recommendations for its further development and implementation. Alongside the independent evaluation of IQER, QAA undertook its own evaluation, in accordance with its normal evaluation procedures. QAA’s evaluation focused upon the operation of the reviews, and the perceived strengths and areas for development of IQER. In addition, the evaluations examined the perceived benefits for staff, students and institutions arising from the IQER pilot. All stakeholder groups were involved in the external independent evaluation and that of the QAA.  These included the colleges, review teams, students, awarding bodies (higher education institutions and Edexcel) and QAA staff.

In order to select an independent organisation to evaluate IQER, HEFCE and QAA invited organisations or individuals to submit a tender. The evaluation was intended to focus upon the IQER process (the operation of the method) and the impacts (costs and benefits) perceived by the participant stakeholder groups.  It was anticipated that the evaluation would be accompanied by a series of recommendations for the improvement and further development of IQER. HEFCE and QAA awarded the contract to the Higher Education Consultancy Group. The objectives of the external evaluation are set out in Annex 1 and a copy of the Higher Education Consultancy Group’s full evaluation report can be found at Annex 2.

Overall, the QAA’s and the Higher Education Consultancy Group’s evaluation of IQER showed that IQER was well received by colleges, awarding bodies and students.

How the evaluation was undertaken

In accordance with normal QAA practices, following the completion of each Developmental engagement and Summative review, an evaluation questionnaire was disseminated to each of the stakeholder groups. The overall response rate to the evaluation questionnaire was 79 per cent, with 172 completed questionnaires received by QAA. The evaluation questionnaire was supplemented by a focus group event, which sought to further explore stakeholders’ perceptions and experiences of IQER.

The external evaluation also included all stakeholder groups. At the outset of the project, the Higher Education Consultancy Group held initial meetings with Edexcel and QAA. The following activities were undertaken as part of the evaluation:

• an email survey of all QAA IQER coordinators
• visits to five colleges, chosen to reflect different kinds of provision and institutional size, plus discussions with a main university partner of each college, undertaken either as a part of the visit or by a separate telephone conversation
• telephone discussions with a nominee of each of the other pilot colleges
• a survey of other university partners
• an open invitation to all IQER reviewers to make comments
• discussions with Universities UK, Guild HE and the Association of Colleges as well as with other relevant stakeholder bodies.

The combined research methodologies adopted for the evaluation of the IQER pilots were complementary and proved to be robust and thorough.

What did the evaluation find?

Overall, IQER was well received by all stakeholder groups. Benefits were identified for colleges, awarding bodies and for the student learning experience. There was a general consensus that IQER was particularly valuable in respect of the development and enhancement of colleges’ capacity to manage their responsibilities for the provision of higher education.

The evaluation confirmed that IQER was assisting colleges in identifying their strengths and limitations in the management of their quality assurance processes and the standards of awards. Although the Developmental engagement report was confidential to the college and the awarding bodies, the evaluation confirmed that the reports provided useful, independent and reliable information for stakeholders.

The evaluation confirmed that IQER would contribute to the development and enhancement of the learning experience of higher education students in colleges.         In addition, where standards were already high, IQER provided an external confirmation of good practice and aided in the maintenance of the colleges’ capacity for the management of quality and standards. While, in the main, awarding bodies commented favourably about the contribution of IQER to the development and enhancement of quality and standards of provision, some were less supportive of this assertion.

Benefits to the colleges

The evaluations identified a number of benefits to the college. The primary benefits identified showed that IQER:

• provided opportunities for critical self-reflection and for staff development
• raised the profile of higher education within the college
• acted as a catalyst for change, improvement and enhancement
• generated greater awareness of the Academic Infrastructure
• provided a mechanism for the identification and confirmation of areas of good practice, strengths and areas for development.

Benefits to awarding bodies

Awarding bodies and other stakeholder groups identified a series of benefits to awarding bodies. These mainly focused upon the role of IQER in providing an external and independent mechanism for checking the policies and procedures in place within colleges for the management of quality and standards. Additionally, benefits identified also related to the opportunities created through joint working to contribute to the continuous development and maintenance of effective relationships between the college and their awarding body(ies).

The evaluation concluded that awarding bodies benefited though IQER’s:

• contribution to the assurance of college management of programmes
• provision of a check that policies, procedures and processes are in place, and are operating effectively
• provision of an external and independent monitoring and evaluation process
• enrichment of the relationship between awarding bodies and colleges.

Benefits to students

Students and other stakeholder groups identified a series of benefits for students and the student learning experience. It was acknowledged throughout, however, that there were inherent difficulties associated with obtaining student contribution into the review process. Those students who took part in the review through either writing the student written submission or participating in meetings with the IQER team, all agreed that there were evident benefits for the student learning experience, and that other students should be encouraged to participate accordingly.

Students considered that the primary benefit of IQER was the opportunity to provide feedback to the review team, and to highlight both perceived good practice in their programmes and causes for concern. Students acknowledged that while IQER might not directly benefit them, it would inevitably benefit future students through the development and enhancement or maintenance of higher education in their colleges.

The primary benefits of IQER for students included:

• an opportunity to provide feedback on their education and highlight good practice or concerns
• an external and independent monitoring and confirmation of the quality of their learning experience
• acting as a catalyst for change, informing improvements in higher education provision for current and future students.

Challenges of IQER

Through the evaluation of IQER, a number of challenges were identified. These have been addressed in the revised method for implementation during spring 2007 and are reflected in The handbook for Integrated Quality and Enhancement Review, to be published in January 2008. The revised handbook has been agreed by the HEFCE Board, and the full programme of IQER will be implemented from spring 2008.

Challenges identified through the evaluation included the need to:

• clarify the aims and objectives of IQER within the handbook
• ensure that there is a clear and shared understanding of the role of all partners within IQER
• secure student contribution to the process through the most appropriate means
• minimise the potential burden and resource implications placed upon colleges, awarding bodies and students through the IQER process
• clarify the relationship between IQER and collaborative provision audit in higher education institutions.

The external evaluation of IQER also noted that QAA needs to be mindful of policy developments in relation to the provision of higher education in colleges, to include potential growth or the increasing diversity in the provision of higher education.

Edexcel noted their lack of partnership agreements with colleges, and that time involvement was a major concern. As a result, QAA have made appropriate revisions to the handbook to clarify the involvement of awarding bodies.

QAA is grateful to review teams, awarding bodies, college staff and students who participated in the IQER pilots, and provided constructive feedback on their perceptions and experiences through the combined evaluation activities.

Annex 1: The objectives and core questions of the external evaluation of IQER pilot

• to seek perceptions and views from relevant stakeholder groups (to include the college, awarding bodies and students), based upon their experience of IQER
• to identify the impacts of the IQER process, and to assess the extent to which it is cost-effective (to include a cost-benefit analysis). Impacts should include both benefits and costs to stakeholders (not just financial)
• to examine the extent to which IQER satisfies its aims and objectives, as set out in the handbook
• to identify good practice in the IQER process and those areas where there is scope for improvement or development, and make recommendations accordingly
• that the evaluation of the IQER pilots should not be burdensome upon the stakeholder groups and should be complementary to QAA’s own evaluation process.

The core questions for the external evaluation were:

1          Does IQER fulfil the overall aims as stated in the handbook?

2          Does the IQER method fulfil the objectives identified in the handbook?

3          Is the method cost-effective?
           
4          Does IQER enable HEFCE to meet its statutory obligations?
           
5          Are there tangible and lasting benefits for institutions, staff, students and awarding bodies?

• Are there any aspects of the IQER process that could be improved or developed, to increase the economy and efficiency of the process and minimise the burden upon institutions?

Annex 2: HECG report

A Report to HEFCE and the QAA on an Evaluation of the IQER Pilot Programme

• 1 Executive Summary
• 2 Introduction and Terms of Reference
• 3 The IQER Pilots: an Overview
• 4 IQER: The Perspective of the Colleges
• 5 IQER: The Perspective of the Awarding Bodies
• 6 IQER: The Perspective of the Reviewers
• 7 IQER and Collaborative Provision Audit
• 8 Costs and Benefits
• 9 Good Practice
• 10 The Implications of the Pilots for Full Implementation in 2008
• 11 Summary of Recommendations
• Appendix A: Institutions Taking Part In The IQER Pilot

1          EXECUTIVE SUMMARY

1.1        This report evaluates the pilot implementation by QAA of the Integrated Quality and Enhancement Review (IQER) in 15 further education colleges (FECs) in 2006-07.  IQER is a new approach to assuring the quality of higher education (HE) provision in further education (FE), and replaces the previous system of academic review.

1.2        The invitation to tender identified five core questions for the evaluation, and the    conclusions for each are summarised in turn:

            i) Does IQER Fulfil the Overall Aims and Objectives as Stated in the Handbook?

1.3        The handbook produced by QAA to guide the pilot implementation did not define the aims of IQER as such, but rather stated five tasks that the process was designed to fulfil.  We have subsequently confirmed with QAA that these statements have the formal status of aims.  We comment on each in turn:

•       To assist colleges in identifying and confirming their strengths and limitations in HE quality assurance management and self-evaluation processes:  The available data suggest that this has generally been achieved in the IQER pilot, with all colleges engaging purposefully with the process and reporting considerable value in both the developmental engagement and summative review.
•       To provide independent and reliable information about HE provided in colleges to present and prospective students and to other interested stakeholders:  This is largely achieved through the publication of the final summative report, and associated confirmation that reliance can be placed upon self-published information.  In all 14 of 15 completed reviews (one has been deferred with the agreement of HEFCE because of exceptional circumstances) a favourable summative report has been produced, although the very compressed timetable for the pilots has meant that they have not yet been published by QAA.
•       To contribute to colleges' increased and enhanced capacity to manage their responsibilities for HE, as set out in partnership agreements: There is a range of views within colleges on the extent to which IQER has met this purpose.  Overall their view is positive, with some colleges particularly enthusiastic about the enhancement in the management of HE that has been achieved.  In a small number of cases colleges (whilst generally supportive of  IQER) have found that it has largely confirmed their previous understanding of quality assurance but has not substantially increased their management capacity or changed their practice.  From an awarding body perspective the benefits of this purpose of IQER are contested.  For Edexcel and the majority of the participating HEIs for which data are available, IQER is perceived to have been beneficial, to have generally strengthened FEC-HEI relationships, and to have enhanced college management.  For a minority of pilot HEIs (mainly those with explicit franchise arrangements which provide little local autonomy for colleges) the attempt to enhance college capacity to manage HE (as currently designed within IQER) is held to be an inappropriate process which conflicts with their responsibilities as awarding bodies for all aspects of provision.  This is a sensitive issue and is considered in detail in the main text.
•       To identify features of good practice:  Summative reports do this and provide information which other colleges and stakeholders may find useful.  The extent to which such information is contained in pilot reports cannot be assessed until they are made public.  However, the notion of 'good' practice has a number of contextual factors associated with it, and thus identification and dissemination need treating with care (see Section 9). 
•       To deliver constructive reports to colleges, their awarding bodies and HEFCE:  As the summative reports are not yet public it is to early comment on this, but it is the task of the QAA to ensure an appropriate standard of publication and there is no reason to suppose that they will not do so.  However, a small number of review coordinators have raised some operational concerns about aspects of the writing and production of summative review reports which the QAA need to consider.

            (ii)  Is the Method Cost Effective?

1.4        In Section 8 this report estimates the costs involved for institutions in conducting the pilots.   Based upon data provided by a sample of the pilot institutions, estimates of the cost range for the involvement of the FECs varies (at 2006-07 prices) between approximately £57,868 and £107,500, although these figures should be treated as indicative.  Non-staff costs are very low.  If these estimates were to be applied to all 15 FECs taking part in the pilot, the total range involved would have been in the region of £868,000 to £1,612,000.  Not surprisingly, a heavier proportionate workload (and therefore proportionate cost) falls on colleges with small HE student numbers.

1.5        For HEIs the time involved - and therefore the costs - are much lower and within a cost range £897 to £7,128 per institution, with an average of £3,854.  Again non-staff costs are minimal - largely travelling.  Overall, the costs falling upon HEIs are small and if the same cost structure were extended to all 27 HEIs providing awards to the FECs in the pilots then total cost of participation would have been in the region of  £104,000.  For HEIs the major concern about costs is not associated with a single review of provision in one FE partner, but rather the overall cost of participation in reviews with multiple partners from 2008 onwards. 

1.6        Assessing future costs for the full roll out of IQER from 2008 is highly speculative.  Assuming 50 college IQER reviews a year, and that HEIs are engaged to the same extent as in the pilots and that the same ratio (27:15) of HEIs to FECs exists as in the pilots (and both assumptions are unlikely to be the case), then a very approximate estimate for the annual costs of HEI participation in IQER reviews would be in the region of £345,000 pa for 2008 or £1,725,000 over the full five years (at 2006-07 prices).

1.7        For FE the assessment of future costs is even more speculative, but based on the range identified in paragraph 1.4, then the annual costs (for 50 FECs) from 2008 might be in the range of £2,893,000 to £5,375,000.  Over a five-year period this would equate to a range of £14,467,000 to £26,875,000.  Such speculative costs take no account of what might be expected to be substantial institutional effort in addressing the outcomes of the agreed action plan.

1.8        It is not for this evaluation to make a judgement on whether the projected sector-wide costs of IQER in either FE or HE are acceptable, although we do observe that all colleges and most (not all) HEIs found their participation in IQER to provide an acceptable cost-benefit outcome at an institutional level.  Given that the expansion of HE in FE cannot be undertaken without due attention being given to quality assurance, then (particularly for HE) the additional costs involved might simply be regarded as the consequence of engaging in cross-sector partnerships.  However, there is a risk from a FE perspective that small pockets of provision may be adversely affected by the costs associated with compliance with the IQER methodology.

            (iii)  Does IQER Enable HEFCE to Meet its Statutory Obligations?

1.9        In summary, these are to ensure that the English HE that it funds meets the expected standards and that students receive an appropriate learning experience.  It discharges this responsibility by contracting with QAA to conduct reviews of the quality and standards of HE provision  in FECs and to publish reports on the outcomes.  There is no reason to suppose that the change in  methodology from the previous academic review to IQER in any way provides a risk to HEFCE in relation to its statutory obligations.  Indeed, the reverse is probably true, in that the increased focus in IQER on cross-institutional processes is generally held by all FECs involved to strengthen quality management and to give HE a greater priority in colleges.

            (iv)  Are There Tangible and Lasting Benefits for Institutions, Staff, Students and Awarding Bodies?

1.10      As already noted, almost all participating FECs think that the IQER process has produced benefits, in some cases very significantly so.  As a result, on balance, FECs have generally concluded that the costs of their involvement (mainly staff time) provided value for money and were not an excessive burden.   In most colleges it is reported that relevant staff have been heavily involved in the IQER process, and have welcomed the resulting prioritisation for HE.  The position concerning students is more mixed, in that although obtaining student engagement was a central theme of IQER, with a small number of exceptions most colleges have found this difficult.  As a result, there is a general conclusion from most data sources that the nature of student engagement needs to be recast into something more suitable for the realities of HE study in FE.

1.11      As noted above, the benefits to awarding bodies are contested, with a minority of HEIs finding that as currently designed IQER conflicts with their responsibilities as awarding bodies.  In Section 10 we conclude that this issue could be addressed by clarifying the overall  purposes of IQER and the associated three core questions (see Section 2) to place greater emphasis on capacity development in colleges rather than quality management in the narrower context of partnership agreements.

1.12      At the end of a short and rushed pilot there is - by definition - no way of knowing how lasting these benefits will be.  To the extent that in many pilot colleges IQER has contributed to the creation of a basis for the expansion of HE then the benefits are likely to be long-term and lasting.   However, a number of colleges thought that there would be less advantage on future occasions in repeating the developmental engagement methodology once good practice became embedded.  HEIs were generally unable to assess the lasting benefits until more immediate short-term gains had been achieved.

            (v) Are There Any Aspects of IQER That Could Be Improved?

1.13      Notwithstanding the benefits noted above, the full implementation of IQER could be enhanced through a number of changes in both the methodology and the operation of the process.  These are set out in the report and a number of recommendations made (Section 11), which include:

• A clarification of the aims and scope of IQER to address the concerns of some (not all) HEIs.  This would also help to eliminate any confusion with collaborative provision audit.
• Reducing the variability of the IQER process that occurred in pilot colleges through better guidance, preparation, training and briefing, and coordination with review coordinators.  Although acceptable (and perhaps inevitable) as part of a pilot, full implementation requires greater consistency to ensure robustness and the confidence of stakeholders.
• Ensuring that the QAA and the IQER Advisory Group give proactive consideration to a number of key issues central to the full implementation of IQER in January 2008.  These include - but are not limited to - the capacity of all those involved to meet the workload demands placed on them.  The position is especially acute for Edexcel and a few HEIs with a very extensive range of partnership agreements.

2          INTRODUCTION AND TERMS OF REFERENCE  

2.1        IQER is a new approach to ensuring the quality of HE provision in FE, and was piloted by the QAA in 15 FECs in 2006-07.  It replaces the previous methodology based on academic review, and seeks to take an institution-wide perspective on provision, which starts with a formal planning meeting, continues with a developmental engagement (with a focus on enhancement), and concludes with a subsequent summative review of how colleges manage their responsibilities for HE provision.  This report does not set out the full methodology of IQER, which can be found in existing QAA publications.¹

2.2        IQER applies to all HEFCE-funded HE in FECs and its purposes are set out in Section 1.   It is structured around three core questions against which specific lines of enquiry are formulated, assessment being a common line of enquiry in the pilots.  Each of the core questions focuses on the contractual obligations of colleges as follows:

• "Does the college discharge its responsibilities effectively, as set out in its partnership agreement(s), for the management and delivery of the standards of the award(s) it offers on behalf of its awarding body(ies)?
• Does the college discharge its responsibilities effectively, as set out in its partnership agreements, for the management and assurance of the quality of the learning opportunities to enable students to achieve the intended learning outcomes?
• Can reliance be placed on the accuracy and completeness of the information that the college is responsible for publishing about itself and the programmes it delivers?"

            Later in this report we comment on issues associated with these three core questions.        

2.3        The QAA commissioned an independent evaluation of the IQER pilot from the Higher Education Consultancy Group (HECG), with the work being undertaken in the summer of 2007.  The consultants involved were Allan Schofield (Head of the HECG) and Professor George Gordon (an HECG Associate).  In summary, the terms of reference for the evaluation were to review:

• Whether IQER satisfied its aims and objectives in the pilot.
• The impacts (positive and negative) of IQER on all stakeholders.
• Stakeholder experiences and perceptions of IQER.
• Emerging good practice in the operation of IQER.
• Emerging benefits of IQER for stakeholders.

2.4        In parallel, the QAA undertook a separate review of the detailed operation of the IQER pilots, which they preferred not to integrate with the external evaluation.  Accordingly this study concentrates on the policy and associated issues concerning IQER, and leaves operational details to the separate QAA enquiry.

2.5        The time available for the evaluation was small, and a straightforward methodology for the evaluation was agreed with the QAA and the IQER Advisory Group consisting of:

• Initial meetings with QAA and Edexcel.
• Undertaking an email survey of all QAA IQER review coordinators (response rate 10/11).
• Visits to five FECs (chosen to reflect different kinds of provision and institutional size), plus discussions with a main HEI partner of each FEC undertaken either as a part of the visit or by a separate telephone conversation.
• Telephone discussions with a nominee of each of the other pilot FECs.
• Undertaking a survey of other HEI partners (overall response rate was 15 of the 27 HEIs involved in IQER).
• An open invitation to all IQER reviewers to make comments (two received).
• Discussions with Universities UUK and the Association of Colleges as other relevant stakeholder bodies.

2.6        It is important to set out briefly some of the context in which IQER is being introduced.  This includes:

• A previous conclusion by QAA² that academic reviews in 2003-05 (using the previous methodology) produced evidence of "much good practice, and significant development of the provision of HE by FECs", that "colleges generally have well-established policies and procedures to maintain and enhance standards and quality", but also that "colleges have a major challenge to raise the profile and understanding of HE internally".
• The continuing need for robust quality processes which command public and stakeholder confidence, not least in the light of some of the concerns of HEFCE (in its recent consultation on HE in FE) that some HE in FE has been "opportunistic" with associated concerns about the consistency of the quality of the student learning experience.
• The diversity of the FE sector, and the wide range of relationships with HEIs, adds complexity to quality assurance and sets challenges to HE-FE partnerships.  Relevant factors here include: the size and nature of HE in FE provision (ranging from the large to very small); the variations in forms of HE-FE partnerships (eg single partnerships, multiple partnerships for both FECs and HEIs, and partnerships within the context of lifelong learning networks); the different requirements of HEIs and Edexcel as awarding bodies; and the differences in operation between validated and franchised provision.
• Government policies on the future possibility of FECs awarding their own foundation degrees, thus providing a potential stimulus for colleges taking on future responsibility for quality arrangements.
• The size of the FE sector which provides a substantial challenge for the full implementation of IQER from 2008 onwards and which is summarised in Section 10.

2.7        Finally in this section, a brief note on how this report is structured.  The three main data sources for the evaluation (FECs, the awarding bodies, and reviewers and review coordinators) all provide insights into the pilot and useful data triangulation.  Their perspectives are inevitably different, and no attempt has been made to integrate them artificially.  Therefore, in the following three sections, data from these sources is presented separately and then integrated in sections which review the costs and benefits of IQER, and key issues for the QAA to consider in relation to full IQER implementation.  This approach leads to a small amount of duplication of content, but this is probably inevitable given the nature of this evaluation.

3          THE IQER PILOTS: AN OVERVIEW

3.1        In this short section we summarise the main issues arising from the pilots, but - as subsequently noted - there are numerous specific matters which require consideration by the QAA and the IQER Advisory Group.

3.2        The main findings are that:

• 14 of the 15 pilots have been completed by the agreed dates (one being deferred with the agreement of HEFCE due to exceptional circumstances) - a not inconsiderable achievement bearing in mind a very concentrated (and - see below - much criticised) timescale. 
• In general, the strong conclusion from most of the data available to us is that the IQER process has been both valuable for and well received by the FECs taking part.  However, some with confidence in existing quality arrangements report that the benefits have been marginal although real.  The focus on cross-institutional management processes rather than narrower academic review has also been useful.  On balance, almost all participating FECs have concluded that the costs of their involvement (mainly staff time) provided value for money (see Section 8).
• The views of participating HEIs on the value of IQER are mixed.  Some have found IQER useful, think IQER has been valuable for their FE partners, and report that their relationship with colleges has strengthened as a result.  Others - particularly those with explicit franchise³ agreements - see IQER as duplicating their role as the awarding body with complete responsibility for ensuring provision.  Some HEIs are particularly exercised by the extent to which an unfavourable judgement on one aspect of provision in a college with multiple HE partners might be applied to all partners.  The issues here are complex and are discussed in Section 5.
• Despite the stated purposes of IQER and the three core IQER questions, there remains some confusion about the exact aims of IQER, and whether the three core questions most accurately capture the intentions of the QAA.  Section 10 concludes that some change of emphasis in the wording of the core questions would aid clarification.
• Both FECs and review coordinators have identified a significant number of variations or ambiguities in the way the IQER process was conducted, for example over the role of HEIs; the amount and nature of supporting documentation provided by FECs; a lack of clarity in some aspects of the documentation provided by QAA to support the IQER pilot; some inconsistency in the advice provided to FECs and HEIs by both review coordinators and some QAA staff;  and the short timescale, which amongst other things is reported to have meant that there was little time for action to be taken between the developmental engagement and the summative review.  Some of these are more operational issues (and therefore more in the scope of the QAA's own internal review).  
• The self-evaluation which preceded both the developmental engagement and summative review has been difficult for some FECs, with a lack of robustness reported in some cases. 
• Student engagement in the IQER process has been mixed and often weak.  Several FECs have observed that the nature of student involvement as suggested by QAA is unrealistic in the FE context.
• There was a general feeling amongst many participants that the time involved in IQER was not always well used, and that the process could be streamlined.  In some reviews duplication of discussion was reported between the developmental engagement and summative review meetings.
• The role of the institutional nominee (IN) was generally successfully undertaken, but tensions were evident in some FECs, and many colleges felt that the QAA needed to take more account of the real role conflict likely to exist in many colleges.
• The full implementation of IQER presents numerous challenges particularly in the scaling up for both QAA and awarding-body processes (see Section 10).

3.3        The data for these conclusions is now presented in the next three sections.

4          IQER: THE PERSPECTIVE OF THE COLLEGES

4.1        All the colleges reviewed generally welcomed the IQER process and found it helpful to some degree.  However, its value was reported to vary, with a few FECs finding the benefits modest in relation to the time involved, whereas others were much more enthusiastic - for example, one noted that it had enabled the college to do "in a few months what would otherwise have taken five years to achieve".  Particular benefits were reported to be:

• A greater focus on the institutional aspects of HE in FE provision.
• Increasing the awareness of higher education in colleges and greater priority given to it.
• Gaining greater awareness of the nature of quality assurance in higher education and the approach of the QAA.
• The encouragement gained by receiving confidence judgements as part of the IQER process.

            There was some evidence to suggest that those FECs with robust institution-wide quality assurance processes already in place have found the benefits more marginal, although real.

4.2        Overall, the developmental engagement was welcomed, as were the 'collegial' assumptions underpinning it.  The summative review process was generally well understood.  However, the fact that no review judgements were made other than those providing confidence may have meant that the methodology and evidence base was not fully tested in reaching a no confidence decision.

4.3        The benefits obtained by FECs also reflect the institutional motivation for participating in the pilots.  In general, these were reported to be a combination of the following:

• To gain benefits by being an 'early adopter' of the IQER methodology.
• To underpin a strong institutional commitment to HE, which may in some cases lead to an application for foundation degree awarding powers.
• To use the developmental engagement aspect of IQER as a conscious approach to organisational planning in expanding HE.
• In a small number of cases, to redress a previous limited confidence judgement made as part of academic review.

            In a sense, therefore, it is not surprising that participating FECs found IQER helpful, as given these motives it would have been a surprise had they not.  We consider later, in Section 10, the extent to which such motivations are likely to be typical of the whole sector when IQER is implemented fully.

4.4        Colleges report undertaking the IQER process in different ways in terms of the extent of staff consultation and participation, partly reflecting the internal organisation of HE work and whether dedicated staff were used.  In general, a senior member of college staff was responsible internally for managing the IQER process, and this was undertaken in different ways reflecting the organisational culture of colleges.  The nature of FE staff engagement with IQER varied.  In a few colleges, it was reported that IQER appeared to place greater emphasis on consultation than was normal within FE ("consultation overload" was a phrase used by one senior manager), whereas in other cases a substantial number of internal staff were encouraged to attend the initial planning meeting or were in other ways involved in the developmental engagement.  In several colleges, teamwork was felt to play an important part in undertaking IQER.   Some colleges observed that greater clarity from QAA would have been helpful in relation to which staff should be involved, and the position of library, resource centre and IT staff was generally felt to be unclear. 

4.5        FECs also reported some drawbacks to the IQER pilots, and these were of two kinds: those concerning the overall methodology, and those concerning the conduct of the pilots.  Whilst some of the latter issues are operational (and therefore within the scope of the internal QAA review) they are briefly raised below, as they may have considerable implications for full implementation, and may also affect the confidence with which the various stakeholder groups approach IQER.

4.6        Issues for some FECs concerning the IQER methodology included:

• The language and assumptions behind IQER are very much those of HE as the methodology borrows from other QAA processes.  In one sense this is very much welcome for those FECs wishing to increase their HE work; on the other hand it presented some colleges with a challenge in fully understanding what was required.  For example, the self-evaluation was difficult for some FECs, with a lack of robustness reported in some cases.  Such concerns highlight the need for appropriate briefing which (as noted below) was criticised by some - not all - colleges for being overly long and complex.
• The role of HEIs was generally perceived to be unclear, a perception shared by most HEIs themselves - see Section 5.
• There was some concern about whether the three core questions at the heart of the review process accurately represent the purposes of IQER - a view which we share and discuss in Section 10.
• Some FECs noted the difficulties of capturing the complexity of diverse and large-scale provision within a relatively short period of time (for example, substantially shorter than institutional audit in HE).  This problem was generally not faced in small colleges, but was significant in some colleges where a range of programmes on different locations involving multiple HE partners occurred.  This led to two colleges with substantial  HE provision to conclude that despite its advantages IQER was not as robust at a subject level as the previous academic review methodology.
• The concept of the academic infrastructure (AI) was reported to be not well known, although more staff were said to have become at least partly familiar with it during the IQER process.  Some colleges observed that it felt like a rather artificial construct, and that the requirement for FECs to carefully consider the purposes and intentions of the AI was - in practice - unrealistic, and that it that was likely only to be of interest to HE managers and quality assurance staff within colleges, particularly those with a small volume of provision.
• The reason for selecting lines of enquiry in relation to the developmental engagement was generally accepted, and the reasons for the QAA emphasising assessment were generally understood.  However, for some FECs, the specification of the assessment line of enquiry detracted from the potentially open-ended nature of the critical reflection, whereas for others it was simply accepted as part of the framework in which they were operating.  Further clarity about the rationale for choosing future lines of enquiry might be useful.
• In many FECs - although not all - student engagement in IQER was not very successful and few written statements were produced.  Although some innovation was demonstrated in obtaining student input (see Section 9), in general, many colleges were critical of the way that student involvement  was conceptualised and felt that IQER guidance reflected an unrealistic HE view of student engagement.  Particular issues raised included: the part-time and largely instrumental nature of FE students meant that there was little time or motivation for student participation, with the exception of small colleges or single programmes; getting robust and reliable student feedback across a number of physical locations and/or programmes was difficult; and greater clarity was needed about the extent to which existing student feedback data on programmes could be used as a proxy for additional IQER-oriented student information.  So far as the future of student engagement in IQER is concerned, then we believe there may be lessons to be learned from the ‘sparqs’ initiative in Scotland⁴ .
• Significant issues existed in relation to the role of the 'institutional nominee' (IN) - the individual whose role is both to support the development engagement and the implementation of the associated action plan, and also to be a full team member.  Several colleges noted that a major potential role conflict existed in relation to the IN, and that its conception seemed to rely on a collegial way of working that was more at home in HE than FE.  In the pilot, such role conflict was not reported to any significant extent , but there was a strong feeling that it might be in future, in colleges wishing to exercise strong managerial influence of the IQER process.  In such circumstances, several INs noted that their primary loyalty would be to their college (and often to colleagues in the management team) and not the QAA review team.   The potential role conflict is particularly acute if the IN also holds a post such as director of HE.  Accordingly we recommend that the QAA consider the role of the institutional nominee and provide guidance on how potential role conflict might be avoided.
• Several FECs commented that whilst they generally welcomed the IQER methodology, the reasons for the absence of any direct observation of teaching (as occurs in the OfSTED approach) was not clear to them, and seemed an omission.  The rationale for not observing teaching might, therefore, need to be made clearer, and where colleges wish the possibility of using relevant OfSTED data explored.  However, there are clearly some fundamental methodological issues here which need careful consideration.
• Finally, in a couple of cases, issues concerning the methodology were raised in relation to how any significant expansion in future HE provision within individual colleges was to be considered within IQER.  In this situation a confidence judgement might be made in relation to existing small scale institutional HE provision, but the question was raised: to what extent should planned future provision be taken account of, particularly when rapid growth was forecast as part of a college's strategic plan?  We recommend that QAA provide further guidance on how rapid planned growth in future HE in FE provision within individual colleges is to be taken account of within IQER reviews.

4.7        In addition to the above, numerous issues were raised by FECs concerning the operation of the IQER pilots.  In summary these included:

•       The timing was felt to be inappropriate, unrealistic and - to use the words of one FEC - "unfair".  In particular the short period of time (a few weeks) between the development engagement and the summative review meant that there was no time to test out the real linkage between the two processes when IQER goes 'live'.
•       Although most colleges were complimentary about the role played by their QAA review coordinator, there were also some concerns about perceived variations in the conduct of the IQER process from that stated in the briefing documents.  This included issues such as: an apparent lack of awareness of FE and Edexcel of some reviewers and review coordinators; the use and availability of support materials (for example over what support materials to provide to review teams); a perceived lack of preparation in some cases by reviewers (several colleges felt that documents they provided were never read by reviewers); a lack of clarity about the role of partner HEIs; the involvement of employers in the review meeting; and so on. Such variations could be explained because of the pilot nature of the activity, but emphasise the need for greater coordination and briefing by QAA.
•       At an operational level, colleges tackled IQER in different ways, some reportedly more effectively than others, and there was a general feeling that greater guidance could be provided by QAA in the preparatory meeting for the review, rather than just at the initial cycle planning meeting.  To take just two examples: first, colleges varied in the numbers of staff attending the summative review meeting, and in some cases large numbers attending were reported to have led to slightly confused discussions; second, the colleges involved HEIs in different ways and in a small number of cases did not do so in the belief that the QAA or review coordinator would make contact.  These and similar operational issues reflect the need for sound administration, and for the college to be able to demonstrate its competence and reliability in relation to IQER arrangements. 
•       Colleges expressed different views about the training and briefing provided.  For approximately one-third, both were appreciated, but another third felt that training FE staff alongside reviewers meant that some of the content was not relevant and therefore the process was more time-consuming than necessary.  Criticisms of the briefing and guidance materials by just under half of FECs included that they were both too long and also lacking information in some key areas, observing that shorter and 'punchier' documents would have been appreciated.  The QAA have reviewed separately the effectiveness of their training.
•       Whilst the summative review was accepted as a core aspect of IQER, there were concerns from colleges about the delay of a week before the review team came together to agree its judgement.   The QAA reason for this (the need for a period of reflection) was generally not accepted, and in some cases was contrasted unfavourably with OfSTED processes where judgements are provided at the end of visits (and daily feedback is given at the end of each day).   In several FECs, the compressed timescale meant that they wanted to move quickly to take any action recommended by reviewers but were unable to do so immediately after the visit, thus risking a loss of momentum. The position of review coordinators on this issue is considered below.
•       Colleges differed about the extent to which the IQER process could be reduced to take less time, but there is no obvious consensus.  For example, a few questioned the need for a lengthy initial planning meeting, although others felt this valuable. 

4.8        Whilst welcoming IQER, colleges noted that they continued to be subject to other forms of review most notably by OfSTED.  This raised several issues for them: logistical arrangements when the timing of OfSTED inspections and IQER reviews clashed (one pilot was delayed and rushed for this reason); the need for clarity about data-sharing between different approaches to reviews (for example, could data collected for OfSTED reviews be used for IQER and vice-versa); and - more generally - the desirability of trying to align more closely quality reviews in the sector.  We are aware that QAA and sector bodies are involved in discussions over this, and the experience of the IQER pilot supports their efforts.

4.9        We asked all FECs taking part in the pilot to calculate the total amount of staff time involved in undertaking IQER, although only half were able to do so.  The results are set out in Section 8 on the costs of the IQER pilot.

5          IQER: THE PERSPECTIVE OF THE AWARDING BODIES

5.1        Two kinds of awarding bodies were involved in the IQER pilot; 27 HEIs (validating and awarding foundation degrees and degrees), and Edexcel (approving and awarding directly HND/Cs).  We sought evidence from the awarding bodies in two ways: from HEIs we conducted an interview (generally by telephone but occasionally in person) with a representative of one of the awarding bodies for each of the five FECs visited, and also sought written submissions from the rest (overall we received input from 15 of the 27 institutions); and for Edexcel we conducted an individual interview with its HE Manager.

Higher Education Institutions

5.2        So far as the HEIs were concerned reactions to IQER varied, with a range of advantages and drawbacks being perceived.  In the original consultation phase of IQER, a number of universities had raised concerns about the methodology, and the extent to which it sought to deal with matters perceived to be entirely the responsibility of the HEI as the awarding body, coupled with associated concerns about possible duplication of the separate collaborative provision audit.  We have therefore considered these issues closely and comment below.  However, with some exceptions, we judge that most of the HEIs involved, and who provided information to us, are now more relaxed about IQER, although some reservations about specific issues remain.

5.3        In considering the opinions of HEIs it is important to recognise a number of factors that influence their response to IQER:

• HEIs vary in the nature and structure of their FE partnerships.  Some are very prescriptive in their franchise agreements, leaving little discretion to FECs, believing that this is the best way to ensure comparability with on-campus provision.  Conversely, others encourage greater autonomy in FECs, and believe that the student experience is primarily the responsibility of the college.  In addition, some HEIs manage FE partnerships through a specialist central office whereas others take a more devolved approach.  Pilot colleges have their own views about the different approaches adopted by HEIs; some welcome the clear structure provided in explicit franchise agreements, whilst others welcome the more flexible approach of some universities and criticise others for being over-controlling.
• Some HEI - FEC partnerships are set within lifelong learning networks or other local multi-institutional partnerships where quality assurance arrangements may apply across numerous institutions.  Such HEIs are therefore concerned about the consistency of IQER reviews, and the need to ensure that a review team in one FEC does not criticise a process found acceptable by another team elsewhere.
• The financial and associated operational arrangements between HEIs and FECs vary.  This means that both managing and reviewing cross-college provision can be complex, for example the access of college students to the library and learning resources of the HEI may vary between partnership arrangements.  
• In some multi-partner arrangements HEIs may well be in competition with each other over future validation arrangements, thus leading to potential tensions when taking part collaboratively in IQER.
• For some HEIs there may also be competitive sensitivities in relation to FE partners possibly using IQER as a stepping stone to achieving foundation degree awarding status.

5.4        Overall, HEIs varied in their views on the benefits of IQER and their satisfaction with the process.  On the one hand, one HEI noted that it was "very satisfied....it appears to us that the process lived up to its billing as developmental, principally for the FEC, but was also helpful to both the FEC and HEI in strengthening further the partnership."  Similarly another HEI noted that "overall the IQER process seemed to work reasonably effectively and helped to inform the university's own QA processes."  On the other hand, an HEI noted that "it is difficult to see any direct benefit from the process [IQER]" and another observed that "it [IQER] seems out of proportion to risk....Further, as we have extensive and satisfactory procedures for quality assurance, annual monitoring and operational management of collaborative partnerships we do not require additional external verification about the health of our collaborative provision."   Beneath these different conclusions lie a number of important issues addressed below.

5.5        The QAA guidelines for the pilot IQER deliberately did not set out to specify the role of HEIs in the process, at least partly to encourage flexibility in operation.  Although the reasons for this were generally understood, most HEIs and FECs felt that clearer guidelines would have been useful, and would have reduced some of the ambiguity that developed.  In two cases, HEIs felt that the absence of guidelines meant their involvement might have been ignored by the college concerned, and therefore took direct steps to become involved.   It is also reported that review coordinators took different views about the involvement of HEIs in the process, leading to different institutional experiences, and also frustration in some cases about the nature of their involvement and the effective use of time.  We recommend that for full implementation from January 2008, the role of HEIs be specified more clearly in guidance documents.

5.6        Part of the reason for not specifying the role of HEIs was that the QAA could not be certain how universities would respond, and how much engagement they would wish in the process.  In the event, levels of engagement were high, in some cases surprising the QAA.  The details and costs of such participation are set out in Section 8.  From the perspective of the HEIs their motivation for being actively involved in IQER was a combination of the following reasons:

• Close existing partnerships with FECs meant that they wanted to provide support during the IQER process.
• As awarding bodies, the need to preserve institutional interests in relation to IQER processes, documentation, and judgements.
• The potential reputational risk of a no confidence judgement meant that some HEIs perceived that they had no real option other than to be closely involved.
• The general perception of the need to have continued engagement with a new QAA process.

          Such factors are likely to continue with the full implementation of IQER, but need to be considered in the light of the costs of future engagement - see Section 8.

5.7        So far as the benefits of IQER are concerned, most - although not all - HEIs noted: a strengthening in the institutional relationship with FE partners (although not necessarily at subject level); the growing maturity of QA within FECs that IQER provided; and gained satisfaction from a confidence judgement in relation to  provision.  The absence of any no confidence judgements meant, of course, that most HEIs did not feel particularly challenged by the outcomes of IQER, and saw the benefits as mainly falling on FECs. 

5.8        However, several criticisms were evident - some largely operational but others more fundamental to the IQER methodology.  Issues in this latter category included:

•       Five HEIs taking part in the pilot expressed specific reservations about aspects of the purposes of IQER, and the extent to which they perceived the process to be at odds with their responsibilities as awarding bodies.  In some cases this led to a suggestion of potential duplication with CPA (see Section 7), a criticism strengthened in some reviews by review coordinators seeking to discuss the QA arrangements in place between an FEC and HE partners.   One responding HEI felt particularly strongly about this and noted that "there is a contradiction between CPA and IQER in that CPA makes it quite clear that the awarding institution is responsible for the standards of its awards and the quality of the student learning experience, wherever taught, and that the CPA is about the HEI not the partner.... IQER shifts the focus to the college and asks how they use the academic infrastructure and how they manage QA, which is only appropriate to those collaborative relationships which permit such delegated authority.  Some partners give much more autonomy to their partners.  Neither model is right or wrong, but IQER doesn't recognise the difference and (in our pilot certainly) focused only on one model."  A number of questions arise from this issue, including the extent to which it is appropriate for summative review reports and associated action plans only to make recommendations to the FEC involved.
•       Several HEIs also raised concerns about the nature of review reporting, particularly in cases where critical observations were being made.  There are two options concerning the summative review report (which is a public document): either no HEIs will be named in relation to specific provision and only comments made related to the FEC, or HEIs will be named when comments are made about their provision.  In the former case, a number of HEIs are concerned about the reputational risks to them of being associated with generalised negative comments that may not apply to their own provision, and on which they have no opportunity to comment.  In the latter case, some HEIs perceive that IQER gets near to duplicating CPA and is being more intrusive than originally intended in relation to specific institutional awards.  This is a difficult and sensitive issue, and there needs to be clarity about to whom the final judgement in IQER is aimed and who has to take responsibility for action.  Accordingly we recommend that more detailed  guidance is produced by QAA on the naming of HEIs in IQER reports.    Our comments in Section 10 may help to address this.
•       Third, a small number of HEIs were critical of the fact that there was no opportunity for them to comment on self-evaluation documentation and other material produced by colleges, and have suggested that to have done so might have led both to a better and more informed document and also have mitigated any institutional risk.  This view is particularly held by those HEIs who operate highly structured franchises, and who as a matter of principle believe that they should see all key documents about their awards before being sent to QAA.  The position is well described in the following written observation: "IQER gave ownership of the decision making regarding the sharing of information with HEIs to FECs.  Hence the awarding HEI did not have control over material that had the potential to affect corporate reputation.  This is not acceptable and should be remedied in the confirmed method for IQER."  This is a sensitive issue and we recommend that the QAA provide more detailed guidance on the circumstances in which colleges should ensure sight of documents by awarding bodies before being sent to QAA or other relevant bodies.
•       Although the issue reported in (c) is most acute in FECs with multiple HE partners, a version of the same problem also exists in FECs with a single partner.  Here a practical question is which institution in practice might lead IQER and undertake resulting action, particularly if the HEI in question operates a heavily structured franchise.  In such circumstances there is another danger that IQER might duplicate aspects of CPA.
•       There is some concern within HEIs taking part in lifelong learning networks (LLNs) that the implications of IQER have not been fully worked through.  These concerns are over two main issues; first, the relationship between the awarding body(ies) and the LLN potentially adds an additional dimension to IQER, which may complicate aspects of review.  Second, when IQER is fully implemented there is a danger of inconsistent judgements being made in different college reviews within the same LLN, causing potential damage to reputational risk to all partners.  This would be most likely to occur when a review team in one FEC was critical about a common LLN quality assurance practice that was not commented upon (or perhaps even praised) in a review of a different college.  To avoid such difficulties, we recommend that the QAA review the guidance in place to ensure consistency in the application of IQER to LLNs.
•       Such concerns also relate to the question of which institution should take the necessary steps to implement the action plans resulting from the developmental engagement and summative review.  Although the assumption of QAA is that this will usually be the college, in partnerships with a heavily structured franchise the HEI might conclude that action is largely for it to specify, depending on whether it is part of a standard institution franchise agreement or part of the agreed range of services involved in the financial agreement.  In this context, several HEIs indicated concerns that review reports might comment adversely on practice in individual FECs that was generally accepted as appropriate in other franchise arrangements.
•       More operational concerns of some HEIs included:
  • A lack of clarity in some FECs about how HEIs should be informed about IQER, including basic administrative issues concerning dates of meetings and so on.
  • A lack of consistency between review coordinators in how HEIs were dealt with (for example, in some summative review meetings they were invited to attend throughout, whereas in others they were only invited to attend for some of the time).
  • A perception by some HEIs that discussion at summative review meetings sometimes concentrated too much on the QA procedures of the awarding body rather than matters explicitly concerning the FEC.

5.9        All HEIs involved in the pilots with multiple FE partnerships indicated some concern about the time and costs involved in participating in the full future implementation of IQER.  This issue is addressed in more detail in Section 8.

Edexcel

5.10      Edexcel approves and awards HNC/D programmes undertaken in FECs and directly funded by HEFCE.  The position concerning Edexcel is slightly different from that of HEIs as awarding bodies, in that responsibility for meeting Edexcel QA requirements  resides with colleges, and is monitored through the operation of the Edexcel external examiner system.  Of the FECs in the pilot, 13 offer Edexcel awards, and the Edexcel HE Manager was involved in some way in all these reviews.  He therefore has been in a unique position to observe the pilots across institutions, and this section largely reflects the main issues as he perceives them.

5.11      On balance, the view of Edexcel is that the IQER pilots have been successful, and that most colleges have welcomed the developmental nature of the methodology.  Significant benefits are perceived in demonstrating a stronger institutional focus for HE; increasing the maturity of colleges in dealing with HE; and strengthening collaboration both between Edexcel and colleges and also - by observation - between most HEIs and colleges.

5.12      The support provided to FECs by Edexcel has been regarded as important in ensuring that colleges could meet the timetable for the pilot, particularly as some were initially slow to become fully engaged.  It is expected that continued support will be required for colleges as IQER is fully implemented from 2008; indeed a small number of self-styled  'consultants' are now offering services to colleges to help them plan IQER reviews.  Within Edexcel the provision of such future support is a major issue, and resources are not currently available for the level of personal support provided for pilot colleges.

5.13      Specific comments on the main issues arising across the pilot colleges included:

•       There was significant variability between review teams, and although Edexcel generally perceived review coordinators to be sound there was more variability in approach than was perhaps originally intended by QAA. 
•       By observation, in a small number of colleges there were initial attempts to treat IQER in the same way as an OfSTED visit, with perhaps an attempt by senior managers to present a 'united front'.  This generally diminished as confidence grew in the process, but it is to be expected that it will be a common occurrence when IQER is mainstreamed.
•       Edexcel reinforces the views of some FECs presented above, that the position of institutional nominees lends itself to significant role conflict, and where a no confidence judgement is a potential outcome their position may be extremely difficult.
•       In general, effective student engagement has been difficult to achieve and needs to be reconsidered for 2008 onwards.
•       Colleges and reviewers appear to have had some difficulty in understanding fully all the requirements of QAA, and in some cases inconsistent advice has been reported.   Accordingly, there is a need for better and clearer guidance to colleges to support full implementation from 2008, since the variations evident in the conduct of the pilots would generally not be acceptable in the final methodology.  It is also important, that such guidance be produced in a form suitable for the FE sector, rather than necessarily replicating the approach and documents used by QAA for HE.

6          IQER: THE PERSPECTIVE OF THE REVIEWERS

6.1        This section is based primarily upon the results of an email survey to all review coordinators of whom all but one replied (10/11).  An open invitation was also made to reviewers to provide information and two responses were received.  Numerous issues were raised in this data, to some extent reflecting the different experiences of those involved at reviews in different colleges.  However some of the issues most frequently emerging were:

•       The general benefit to colleges from the IQER process, for the reasons highlighted above.
•       Different levels of preparation and states of readiness of colleges for IQER.  Some review coordinators who were involved in more than one review were able to contrast college approaches, and indicated that significant variations existed in how FECs approached the exercise.  For example over the cycle planning meetings which were conducted with significant differences in style and approach; the production of the self-evaluation which was felt to be weak in some cases; the organisation of the summative review meetings; and significant variations in the quality of administration including the preparation of paperwork and the scheduling of meetings.  Much of this was felt to be due to the inexperience of some colleges in managing processes of this kind, and several respondents emphasised the importance of enhanced support by QAA in future college preparation.
•       Generally satisfactory levels of engagement by academic staff in the IQER process (although sometimes it was poorly coordinated in meetings), but less so by students (see above).   Review coordinators generally reported finding student submissions valuable where they were received, but - slightly worryingly - did not highlight issues concerning the representativeness of students’ views, which in some cases were raised with us by colleges.
•       For  some review teams, issues were reported concerning the interaction between the developmental engagement and the summative review, which reduced effectiveness.  This took a number of forms: different membership, which meant that issues dealt with as part of the developmental engagement resurfaced in some cases in the summative review; in some cases awkwardness in developmental engagement  teams when the self-evaluation was not strong (which might have been corrected by having more external reviewers); the challenge facing an individual external reviewer on a developmental engagement  team when faced with a strong college presence;  a lack of clarity in QAA guidance; and issues of chairing.  This issue may be in part resolved when a longer gap occurs between the two steps.
•       The challenges and potential role conflicts facing institutional nominees were noted by several respondents (see above).
• Several respondents queried the need for a separate judgements meeting, and suggested the timetable should be formulated to allow it to be conducted immediately at the end of the summative review.  The additional cost of the separate meeting was contrasted by some reviewers with what was seen to be a "parsimonious" approach of QAA in not paying reviewers to meet the night before a visit for planning purposes, a practice which was held to be valuable in other forms of QAA review.  We have been informed by QAA that under the Working Time Directive, QAA staff are required to have an 11-hour break within a 24 hour period.  QAA’s policy is for staff and reviewers to work within the Directive and that the advice that teams should not meet the night before the first formal meeting with the college was a logical solution to the problem of not extending the working day beyond what is legal for QAA staff.
•       Several respondents queried elements of summative review reporting, including: the word limits on core questions 1 and 2 making a sensible description of findings difficult; the overlap between core questions 1 and 2; aspects of the wording of core questions 1 and 2 particularly in relation to the role of partnership agreements; the need for reports to be written as plainly as possible and to avoid what one respondent described as "QAA speak" - particularly important in an FE environment unfamiliar with QAA; and a lack of clarity on the role of QAA assistant directors in helping to revise drafts.  In all these areas QAA could provide greater guidance.
•       Several review coordinators queried the decision of QAA to reduce the amount of time involved on IQER for colleges with fewer than 100 FTEs.  Although the reasons behind this were understood, in practice, coordinators observed that the time required to review small provision was often no less than for large institutions, and in some cases more time might be required because of the possible lack of a substantial staffing structure.  Other time pressures were faced in some institutions with large provision, and when full implementation occurs, review coordinators in such circumstances may need to be very proactive in determining well in advance how they want to structure the review, in order to ensure the best use of time and documentation.
•       More generally, some respondents also expressed concerns about the clarity of information on IQER provided by QAA, and associated briefing and training issues, for example, one observed "of all the QAA training I have had this felt like the least clear", and another noted inconsistency of advice and a lack of clarity in communication by QAA on some issues.  QAA acknowledge some difficulties in this area, caused by staffing problems.

6.2        Overall the data from review coordinators and reviewers suggests that IQER was successful in meeting its objective to assist colleges in identifying and confirming the strengths and limitations in QA management and self-evaluation, but the substantial variation in issues arising in different colleges suggests that greater consistency and stronger administration is required in a number of areas before full implementation in 2008.

7          IQER AND COLLABORATIVE PROVISION AUDIT

7.1        In the consultation for IQER, a number of HEIs commented that they perceived a potential duplication between IQER and CPA or (where only a small amount of collaborative provision existed) institutional audit.   Because of the implications for the reduction of the 'accountability burden' we have reviewed this issue in detail, and sought the views of all respondents from whom we collected data.

7.2        Overall, concerns about duplication with CPA only rest with some HEIs, and both QAA and IQER review coordinators do not see any real overlap, given that IQER focuses on college quality assurance arrangements.  The following samples of conflicting quotations from written comments received from HEIs illustrate their concerns over CPA:

• "There was no duplication in the pilot, as this particular partner was not specifically involved in our CPA.  However, if they had been, then the potential for duplication would be great.....In addition there is considerable scope for duplication of effort in the provision of documentary evidence... It is difficult to see how this duplication can be avoided if both processes are to remain and retain their current emphases."
• "There was no obvious duplication... however it does seem likely that the IQER process will be very valuable as a source of evidence for future institutional audits."
• "This was not so [in the pilot], however there would be overlap and duplication had the same partner been chosen [for CPA]."
• "There is no guarantee that the messages from IQER and from CPA will be consonant.  It is even conceivable that those messages may conflict.... If the two processes were linked in some way, and if the timings were related, then better value might be obtained from the combination."
• "Across the sector, IQER may provide an additional mechanism for quality assurance of collaborative provision.  However, were that the rationale for IQER, the process would not be proportionate, as it on the whole duplicates existing processes and could be seen as the sort of 'gold plating' counselled against by HERRG. For IQER to be a proportionate use of resources, other gains need to be recognised.  These lie in the enhancement of a further education college's capacity to take a strategic approach to collaboration provision and improve its own HE capacity."
• "There was duplication with CPA in that the University's CPA report already gave full information on the nature of the collaborative relationship with partners and IQER did not add anything to this."

7.3        Such differences of view are significant, not least because entirely different views about possible duplication were held by two HEIs who both have partnerships with the same FEC, and who both attended the same IQER meetings.

7.4        In large part we think that such differences of view are caused by what we see as some confusion in the aims of IQER coupled with some potential confusion in the wording of core questions 1 and 2.  This is explained in Section 10, but, in summary, greater clarity would be helpful, in particular about whether IQER is focusing on internal college-wide quality assurance particularly in enhancing the student learning experience, or (in the wording of core questions) for ensuring that colleges discharge their key quality assurance responsibilities effectively as set out in partnership agreements.  If the former, then no significant duplication with CPA need exist (providing the use of existing documentation for both purposes is made clear), but if the second is intended (with no change of wording) then this is an invitation for potential confusion between IQER and CPA and it will need very explicit guidance and instruction of review coordinators to avoid it.

8          COSTS AND BENEFITS

8.1        We have reported the perceived benefits of IQER in Sections 4, 5 and 6 above.  The terms of reference for the study require an assessment of the costs of IQER, and a judgement of the consequent cost-benefit return.  In order to provide the QAA with data comparable to that from previous studies, we have used a method of calculation broadly consistent to that adopted by JM Consulting in their review of the costs of external QA conducted for HEFCE in 2005.⁵   That review concluded that the main costs of QA were the direct costs (almost exclusively staff time) falling upon institutions, plus associated overheads.  They also noted non-financial costs (disruption, opportunity costs, etc) but concluded that these were difficult to measure.

8.2        Time did not permit the use of anything approaching the TRAC methodology, and we simply asked all FECs and HEIs taking part in the pilot to calculate the total amount of staff time involved in undertaking IQER (using a standard template), paying particular attention to those we visited.  Overall, only half were able to provide estimates, and only one was able to provide anything approaching a detailed calculation (reported below).   However, estimated data was obtained from four of the five FECs visited and associated partner HEIs, and is presented below.  Three colleges not visited returned low workloads with what we regard as suspect data.  All the data which follows is, therefore, approximate, and needs to be treated as purely indicative.

8.3        Colleges were asked to identify total staff time in hours for each of the main elements of IQER.  The estimated total hours for four of the five FECs visited were approximately:

College A	2,036 hours
College B	2,000 hours (estimate)
College C	1,277 hours
College D	1,096 hours

            As is evident, there was considerable variation in the staff time presented, and this appears to be due to: the range and volume of provision (the broader the range the more staff were potentially involved); the approach to data collection by the college and the extent of consultation; the numbers of staff involved in the various stages of the IQER process; the number of meetings of internal working groups; the number of developmental engagements (some FECs had more than one); and the management style of senior college staff and INs. 

8.4        Not surprisingly, a heavier proportionate workload (and therefore proportionate cost) appears to fall on colleges with small HE student numbers, as can be seen in the following ratio with estimated staff time on IQER (in hours):  College A = 29 staff IQER hours per HE student (2,036 hours @ 71 students) ; College B = 1.7 (2,000 hours @ 1,176 students); College C = 7.5 (1,277 hours @ 169 students); and College D = 1.2 (1,096 @ 936 students). 

8.5        It could be concluded from this data that because at least one college (with large student numbers) has managed to undertake IQER with a small comparative staff  workload, then this is a measure of efficiency which other FECs should strive to emulate in the full roll-out of IQER.   However, we would be wary of such a generalisation; indeed some of the benefits claimed by colleges (and assumed in the IQER methodology) concern the developmental advantages of greater engagement of  staff teaching higher education, and it is difficult to see how this could be achieved without substantial involvement in the IQER process.   In this sense, efficiency measures of introducing IQER may be very different from effectiveness measures.

8.6        Due to these variations it is inappropriate to establish a median, but the range covers 1,096 to 2,036 hours, and includes both a small FEC with limited provision and a large college with a diverse HE portfolio.  By way of illustration, the breakdown of time for College C (with limited provision) which was able to provide most detailed time estimates was:           

Cycle planning meeting	14 staff @ 3 hrs pp	42
DE planning meeting	17 staff @ 3 hrs pp	51
DE meeting I	2 INs @ 20 hrs pp	40
DE meeting II	25 staff @ average 3 hrs pp	75
SR planning meeting	14 staff @ 3 hrs pp	42
SR meeting I	1 @ 20 hrs	20
SR meeting II	25 staff @ average 3 hrs pp	75
Total IN time excluding above		100
Data preparation	16 staff @ average of 30 hrs	480
Working group meetings	18 staff @ 3 hrs @ six meetings	324
QAA briefing and training	2 staff @ 14 hrs	28
Total hours		1277

8.7        In the absence of standardised or TRAC costing data for FE, to calculate costs we have used the same approach as in the JM Consulting study, which used a standard cost for all staff involved in QA of £77,500 (at 2004-05 pay levels) consisting of £52,500 direct staff costs including 'on-costs' (superannuation and national insurance) plus £25,000 for indirect and estates costs of the institution (based on the median of a sample of 12 HEIs).  Allowing an average of 5% for pay inflation, for 2006-07 the annual cost would be £81,375.   There is no standard FE contract so we have assumed a notional yearly contract of 220 (1,540 hours).  Accordingly, the average cost per hour of staff time is therefore £52.8 (ie £81,375 divided by 1,540 hours) or £50.3 (if costed for comparative purposes with the JM Consulting study at 2004-05 pay levels). 

8.8        From these figures it can be seen that the notional cost range for the involvement of the four FECs varies (at 2006-07 prices) between £57,868 (1096 hours) and £107,500 (2036 hours).  Non-staff costs are very low, generally document production, some travel and some catering at meetings.   Of course, the actual costs in colleges may vary significantly from these estimates, depending upon local contracts and the calculation of overhead costs, and would generally be lower.  If these estimates were to be applied to all 15 FECs taking part in the pilot, the total range involved would be in the region of £868,020 to £1,612,500.

8.9        Using the ratio outlined in paragraph 8.4 of staff hours involved in IQER to HE student numbers, a notional - and very speculative - IQER cost per student can be estimated.  For the four colleges concerned this is:  College A = £1,531 per HE student (29 staff hours per student x average cost per hour of £52.8; College B = £90 (1.7 staff hours x £52.8); College C = £396 (7.5 staff hours x £52.8); and College D = £63 (1.2 staff hours x £52.8).  Even allowing for the very speculative nature of the data, it can be seen that the costs of IQER are substantial in colleges with small HE numbers.  Nonetheless, although the costs are significant all the FECs involved said that the cost benefit was positive in the light of the benefits reported in Section 4.

8.10      For HEIs the time involved - and therefore the costs - are much lower.  Data were received for 10 HEIs, with the estimates of staff time involved being as follows (in hours): 134, 125, 105, 104, 76, 70, 40, 28, 28, 17.  The variation is accounted for by the extent of participation by HEIs in the IQER process, the depth of existing partnerships, the breadth of provision involved, and the extent to which it was possible for some HEIs to separate the costs of IQER participation from general support provided to a college.  Using the same notional year contract of 220 days (or 1,540 hours) this produces a cost range at 2006-07 prices between £897 (17 hours) and £7,128 (134 hours), with an average of £3,854 (73 hours).  At 2004-05 prices, the figures would be £855 and £6,740.  Non-staff costs are very small - largely travelling.  Overall, the costs falling upon HEIs are small and for the ten institutions returning data were in the region of a total of £38,500 for 2006-07.  If the same cost structure were extended to all 27 HEIs providing awards to the FECs in the pilots then total cost of participation would have been very approximately £104,000.

8.11      For HEIs the major concern about costs is not associated with a single review of provision in one FE partner, but rather the overall cost of participation in reviews in multiple partners from 2008 onwards.  An HEI with a large number of FE partners might find itself involved in (say) five IQER reviews a year - perhaps more if part of a lifelong learning network, and depending upon local circumstances this might require a staffing commitment of up to a third of a member of staff a year.  Bearing in mind that this post needs to be at a senior level to provide the necessary support to colleges, the opportunity costs might be significant whilst the direct costs remain relatively modest.   As noted above, a similar issue concerns Edexcel, but more acutely because of its widespread involvement in the sector. 

8.12      An issue raised with us by some respondents was the extent to which there had been evidence of 'gold plating' in the conduct of IQER reviews, ie excessive staffing resources being consumed because of the perceived need to ensure a satisfactory outcome.  Our conclusion is that no robust evidence of such a practice existed, indeed the timescale for the pilots tended to mitigate against such an activity.  Moreover, because of the importance of IQER for both colleges and partner HEIs, we would have some concerns about the concept of 'gold plating' in these circumstances and would regard it as almost natural organisational behaviour to maximise resources in this area to try and ensure a favourable outcome.

8.13      Attempting to calculate possible future costs for HEIs as part of the full implementation of IQER is highly speculative, but QAA have identified approximately 268 FECs providing some form of HE awarded by HEIs.  Allowing for the 15 pilot colleges this means that up to 253 reviews may take place in the five years starting January 2008, or approximately 50 a year.  Assuming that HEIs are engaged to the same extent as in the pilots and that the same ratio (27:15) of HEIs to FECs exists as in the pilots (and both assumptions are unlikely to be the case), then a very approximate estimate for the annual costs of HEI participation in IQER reviews would be in the region of £345,000 pa from 2008 or £1,725,000 over the full five years (at 2006-07 prices).

8.14      For FE, the assessment of future costs is even more speculative, but based on the range identified in paragraph 8.8 of £57,868 to £107,500 per college, then the annual costs from 2008 might be in the range of £2,893,400 (£57,868 x 50 FECs) to £5,375,000.  Over a five-year period this would equate to a range of £14,467,000 to £26,875,000 for sector-wide implementation.  Such speculative costs take no account of what might be expected to be substantial institutional effort in addressing the outcomes of the agreed action plan.  This is a significant sum and we recommend that the implications of expenditure on this scale for HE in FE be considered by sector bodies.

8.15      The implications for future workload rather than costs coloured the view of HEIs in the pilots about the cost benefit of IQER for them.  As noted above, the majority of HEIs were generally satisfied with the IQER process, and their relatively modest involvement of time meant a favourable cost-benefit judgement.  For those that were more critical of IQER objectives and methodology, there is a small negative cost-benefit involved in the pilots which will become more substantial if they continue to engage with IQER in the ways displayed in the pilots.  However, it could also be argued that the relatively modest costs per HEI of IQER involvement should just be regarded as part of the costs of ensuring effective partnership arrangements, and therefore built into existing financial models.

8.16      The 2005 JM Consulting study concluded that with changes to the methodology of external academic quality assurance, annual costs per HEI might fall from 2005 levels of approximately £147,000 to an average of approximately £77,000pa⁶.  Assuming this has been achieved (there are no data), the projected annual IQER costs of £3,854 would increase this around £80,850 for those HEIs involved or an increase of 5%.  However, there may be some savings as data collected for IQER might be used in either institutional audit or CPA.

8.17      It is not for us to make a judgement on whether the projected sector-wide costs of IQER in either FE or HE are acceptable, although we do repeat our observation that all colleges and most HEIs found IQER to provide an acceptable cost-benefit outcome at an institutional level.  Given that the expansion of HE in FE cannot be undertaken without due attention being given to quality assurance, then (particularly for HE), the additional costs involved might simply be regarded as the consequence of engaging in cross-sector partnerships.  However, there is a risk from a FE perspective that small pockets of provision may be adversely affected by the costs associated with compliance with the IQER methodology.

9          GOOD PRACTICE

9.1        Although the terms of reference for this evaluation required good practice to be identified where it emerged, in our proposal we pointed out that we expected much of  what might be deemed good practice to concern how operational matters were dealt with, and therefore to fall within the scope of the internal QAA review rather than this evaluation.  In the event our views were confirmed, and relatively little good practice was identifiable that went beyond what might be described as generally sound management of quality processes.  Similarly good practice in quality assurance arrangements will generally be commented upon in the summative review report and is not considered here.

9.2        In addition, five caveats about what might constitute 'good' practice in IQER are  required:

• First, the information is presented second-hand - that is, we are reliant on what we have been told by colleges or reviewers. 
• Second, the concept of good practice in an emerging area of activity is elusive in that what might be regarded as 'good' in the slightly artificial context of a rather rushed pilot may not necessarily be so considered when IQER is implemented over a longer time span.  
• Third, much of what might be considered 'good' is contextual and depends on factors such as the management culture of the college or the relationship with partner HEIs.  It follows that what is 'good' and may be useful to disseminate internally in one FEC, may not always be suitable for dissemination in others.
• Fourth, there is the need to distinguish between 'good' practice in the sense that  widespread adoption is an eventual aim, and 'innovative' or 'interesting' practice that demonstrates activity that deserves to be widely known  but may not necessarily be suitable for straightforward adoption elsewhere.
• Finally, the starting point for any consideration of good practice must rest with a clarification by QAA of whether its conception of IQER is 'value-neutral' in the sense of applying to all colleges irrespective of the form of collaborative agreement, or implies a particular approach in encouraging college responsibility for QA issues.   If the latter view is taken, then the very notion of 'good' practice would be contested by those HEIs who feel that such an aim conflicts with their own franchise arrangements.

9.3        Notwithstanding these caveats, what might generally be regarded as good practice included:

• The seriousness of purpose with which IQER was conducted in many colleges, and the willingness to engage with the development process.
• A strong commitment to staff engagement and teamwork in some FECs in participating in IQER.
• The establishment of working groups which meet regularly in some FECs throughout the IQER period; for example, in one college such a group met weekly for six months.
• Sound management arrangements of the IQER process in some - not all - colleges which built on existing QA arrangements.
• In many colleges, the existence of sound partnership working with awarding bodies which supported IQER (for example, in one college it was reported that a partner HEI had seconded a member of staff half time for six months to formalise assessment procedures and programme delivery, although this was not specifically for IQER).

Some innovation to address difficulties in getting student opinion for IQER.  For example, in one college a group of students (unfortunately not representative of the college as a whole) were provided with free lunchtime pizza as an encouragement for them to attend a meeting, and in another an independent external consultant was used to gather student opinion systematically in focus groups as a way of overcoming practical obstacles to obtaining it in other ways.

10        THE IMPLICATIONS OF THE PILOTS FOR FULL IMPLEMENTATION IN 2008          

10.1      In this section we bring together the discussion above, and consider the major implications for full implementation in 2008.  The large number of issues raised in the pilot and listed above which need to be dealt with before full implementation are not repeated here, rather several major issues which are crucial to resolve are briefly set out below.

Clarifying the Aims of IQER

10.2      Although QAA has identified the main purposes of IQER and stipulated the three core questions to inform them, the pilots suggest that there is still some lack of clarity about exactly what IQER is trying to achieve, and we recommend that QAA and the IQER Advisory Group reconsider the detailed wording of both the stated purposes of IQER and the three core questions to ensure that they are appropriate to support full implementation.  Until such clarity is achieved then we think it is likely that some confusion about key processes will remain, with associated variability in operation.

10.3      In particular, we think that there needs to be greater clarity about whether IQER is focusing on internal college-wide quality assurance to support capacity building, or  (in the wording of core questions) is based on ensuring that colleges discharge their key QA responsibilities effectively as set out in partnership agreements.  At the heart of this distinction lies the concern of some of the HEIs that assessing the effective implementation of partnership agreements is a matter for them, whilst generally accepting the potential value of IQER as an approach to overall institutional capacity building and quality enhancement in colleges.

10.4      A related issue here is the role of IQER in helping to review the cross-college student learning experience.  This is not formally stated as a purpose of IQER, but a number of respondents (including some in QAA itself) felt that this was a valuable potential purpose that in most (but not all) cases would not directly conflict with specific partnership agreements.  If this were to become a stated purpose of IQER (and this would support the capacity building approach) then the methodology would need to be slightly refined and specific guidance given, with more attention being paid to the quality of student learning and less to the college-awarding body interface.   This also raises the question of the adequacy of the existing methodology for obtaining student feedback, and whether (in the specific context of FE), classroom observation has any role to play in quality assurance.  If the answer to this last question is that it does not, then it would be helpful for QAA to provide guidance on why not, as the question is likely to be asked with some regularity during full implementation.

10.5      Clarifying the issue of the main focus of IQER would also help to address the key concerns of some (not all) of the pilot HEIs, involving who recommendations in the summative report are aimed at, the terms under which HEIs might be named in reports, and how a judgement of no confidence in one aspect of college provision might relate to the involvement of multiple partners.   We note that it has been suggested by some HEIs that they should not be named at all in summative review documents, but we think that this is both undesirable and impossible as HE-FE partnerships are publicly funded and subject to freedom of information requirements.

10.6      Even if the current purposes are reconfirmed after discussion within QAA, changes might be considered to the three core questions.  Some review coordinators have found it difficult to distinguish between questions 1 and 2 in writing summative reports, and the explicit specification of the words "as set out in its partnership agreements" has led to confusion, including some review teams to focus too much on the agreements and associated HEI QA processes when this was not the intention.   It is appreciated that the original wording was designed precisely to indicate sensitivity to this issue, but the very fact of its identification may have highlighted the issue.  In any case, any review of quality assurance must by definition take account of the contractual, legal, and funding context.

10.7      As part of a review of the core questions, attention might be given to the requirements of question 3, which asks about the reliance that can be placed on published information.  Whilst understanding the reasons for the question, a number of reviewers and colleges found it out of place - both in terms of content and tone - when set against questions 1 and 2, which require a qualititative narrative account.

10.8      We think that clarification of these issues, with associated rewording of the purposes and core questions, would also help to eliminate the potential confusions that exist concerning the possible duplication of IQER and CPA (see Section 7).

The Robustness of IQER

10.9      As noted above, although all pilot FECs have received confidence decisions in their summative reviews, there have been a significant number of areas where substantial variations occurred in review processes in colleges.  These include: variation in quality and robustness of the self-evaluation; a general failure of student engagement to meet initial QAA expectations; variations in the roles of the awarding bodies; a concern by some reviews coordinators over the writing of the summative review reports; and differences in the extent to which the IQER process provided outcomes to colleges that they were not already aware of; etc.  When taken together such variations are, perhaps, inevitable outcomes of a pilot working to a tight timetable.  However, when taken overall they suggest a need to ensure that IQER is not seen as largely a confirmatory process, but one with a suitable consistency and robustness when it starts to be implemented in full in 2008.

10.10    To achieve this will require a number of measures: enhanced briefing and guidance by QAA; greater consistency of application by review coordinators and reviewers - including the need for them to understand that IQER is a different process from those in which they have been previously involved; greater clarity in the role of awarding bodies; and more consistent advice on process issues by QAA.  Overall, this points to the need - at least in 2008 - for high levels of support for colleges by QAA and the awarding bodies.  Without such measures, and the associated consistency of approach, we wonder whether, in practice, a review team might easily be able to sustain a no confidence judgement were one to be required.

10.11    Greater robustness may also be required in full implementation in that some FECs may not approach IQER with the same positive motivation generally evident amongst colleges involved in the pilot.  To this extent there is some danger - although unquantifiable - in concluding that because a small pilot was a relative success then this will necessarily be replicated across the sector.

10.12    Ensuring robustness also requires QAA and review teams to be clear about how they will deal with possible attempts to 'influence' IQER in an unreasonable way by colleges.  No significant evidence of this was noted in the pilots, and we do not suggest that it is likely to be widespread.  However, in a large and diverse sector we would be surprised if some cases did not occur during full implementation, and several respondents pointed out the dangers to us.   This might occur in a number of ways within the current methodology: pressure being placed upon INs to put primacy on their college responsibilities to ensure as far as possible a favourable IQER outcome; subtle pressure being placed upon review teams by college management, for example by information overload within a tight timetable; by the use of external IQER consultants to prepare colleges for review and perhaps to try and disguise weaknesses; and so on.  We recommend that the QAA be proactive in developing guidance on how to deal with colleges attempting to influence IQER, including briefing (from the outset) review coordinators effectively.

Other Aspects of Implementation

10.13    From 2008 IQER reviews in up to 268 FECs in England will take place over a five-year period, and QAA may also be asked to extend the methodology to some other parts of the UK.  We note above a number of key issues concerning implementation (for example, more effective briefing and guidance), and do not repeat them here.  Rather we highlight a small number of critical issues for the QAA and IQER Advisory Group to consider.

10.14    First, there will be a heavy burden falling upon various organisations to deliver this volume of reviews.  Most directly this falls upon QAA, who are aware of it and report being ready for it.  We have also noted above the burden on the awarding bodies, and the extent to which they are prepared for it.  HEIs are likely to respond to this challenge in different ways: some are likely to reduce their engagement with IQER and will respond only when invited by partner FECs; others will judge the risk involved in any particular partnership and only become involved when a significant risk is felt to exist; and others will continue to bear the additional workload on the basis that supporting effective FE partnerships is core to mission.  Whilst a more selective approach by HEIs would be understandable, the pilot suggests that it would also be unfortunate in that one of the positive outcomes for many reviews was the enhancement of relationships between HEIs and FECs.  The support provided by some HEIs to colleges has proven valuable, and it would be unfortunate if this were reduced as part of full implementation.  The same issue affects Edexcel even more acutely.

10.15    Second, assuming the QAA take steps to deal with both the policy and operational issues raised above, ensuring the optimal conduct of reviews in a very diverse sector will continue to be a challenge.  One element of dealing with this is clearly to identify in the guidance documentation the amount of flexibility permitted within the IQER methodology.  On the one hand we were repeatedly told by college staff that (unlike some of their HE counterparts) they expected to comply with standardised management procedures, and welcomed brevity and clarity in being told what was expected of them.  In the large-scale implementation of IQER that is planned, there will be a need for such an approach.  However, the diversity of the sector and of HE provision and partnerships is such that flexibility will also need to be exercised by review teams in dealing with institutionally specific issues.  The difficulty for QAA and the IQER Advisory Group will be to determine where the boundary lies between consistency and flexibility, and clearly informing review coordinators and colleges.

10.16    Third, a logistical challenge may face QAA in determining the order in which colleges are reviewed.  As originally conceived, those FECs perceived to be most at risk (based on previous academic review judgements) were planned to be reviewed first.  Whilst understandable, this presents the challenge that some of the potentially most difficult reviews might occur near the start of the process whilst new review coordinators and reviewers were still settling in.  Nonetheless, it is quite clear from the pilot that a number of FECs may want to undertake an IQER review as early as possible in order subsequently to seek foundation degree awarding powers.  Since these are likely - although not necessarily - to be some of the stronger colleges, this rationale runs directly counter to that originally articulated.   This issue could become significant, and lead to scheduling issues for QAA.  

11        SUMMARY OF RECOMMENDATIONS

11.1      The following recommendations made in the text and repeated here do not relate to operational or procedural issues, but rather to policy-related matters concerning the full implementation of IQER:

• That the QAA consider the role of the institutional nominee and provide guidance on how potential role conflict might be avoided. (para 4.6 (h) page 11)
• That QAA provide further guidance on how rapid planned growth in future HE in FE provision within individual colleges is to be taken account of within IQER reviews. (para 4.6 (j) page 11)
• That for full implementation from January 2008, the role of HEIs is specified more clearly in guidance documents. (para 5.5 page 14)
• That more detailed guidance is produced by QAA on the naming of HEIs in IQER reports.  (para 5.8 (b) page 16)
• That the QAA provide more detailed guidance on the circumstances in which colleges should ensure sight of documents by awarding bodies before being sent to QAA or other relevant bodies. (para 5.8 (c)  page 16)
• That the QAA review the guidance in place to ensure consistency in the application of IQER to LLNs. (para 5.8 (e) page 16)
• That the implications of expenditure on this scale for HE in FEbe considered by sector bodies (para 8.14 page 25)
• That QAA and the IQER Advisory Group reconsider the detailed wording of both the stated purposes of IQER and the three core questions to ensure that they are appropriate to support full implementation.  (para 10.2 page 27)
• That the QAA be proactive in developing guidance on how to deal with colleges attempting to influence IQER, including briefing (from the outset) review coordinators effectively. (para 10.12 page 29)
  

APPENDIX A:  INSTITUTIONS TAKING PART IN THE IQER PILOT

FE Colleges Involved in the IQER Pilot:

Blackpool and The Fylde College

Burton College

City and Islington College

City College Manchester

Colchester Institute

Doncaster College

Filton College

Gloucestershire College of Arts and Technology

Isle of Wight College

Kingston Maurward College

Mid-Kent College of Higher and Further Education

New College Durham

Shrewsbury College of Arts and Technology

Stephenson College

Stroud College of Further Education

HEIs Acting as Awarding Bodies for FE Colleges Involved in the IQER Pilot:

<

Bournemouth University

Canterbury Christ Church University

City University

Lancaster University

Liverpool John Moores University

London Metropolitan University