Skip navigation

Learning from AVA review 1999-2004

QAA 047 06/2004

Sharing good practice header graphic

Learning about AVA review

The review process

137 At the point of the transfer of responsibilities for Access regulation from HEQC to QAA in 1997, it was decided that the initial review cycle would be six years, a period determined, in part, by a plan for mid-term 'minor' reviews, a plan which was later overtaken by the introduction of a process of annual reporting by AVAs instead. The review process itself was initially developed by the QAA Assistant Director with responsibility for Access, with reference to other QAA review methods, and discussed and agreed in early meetings of the Access Recognition Advisory Committee and the Access Recognition and Licensing Committee (ARLC).

138 The review process typically extends over nine months from the point where an AVA is informed of its review dates to the publication of the final report. In broad terms, the process follows the model which is used by QAA for other of its review and audit activities:

  • the organisation under review produces a document analysing and evaluating its activities (the Analytical Account);
  • the document is considered by a team of peer reviewers supported by a QAA Assistant Director and a visit schedule is planned;
  • a visit takes place during which the review team gathers evidence to test the organisation's description of itself in the self-assessment document; the evidence sources comprise relevant papers and minutes, and discussions with key staff and others connected with the organisation;
  • a report is written, and a recommendation is made, on the basis of the evidence gathered.

Prior to the review visit

139 The review process begins with an initial information meeting between the AVA and the QAA Assistant Director, during which the scope of the review is described and any queries on the part of the AVA are clarified. The dates for the review are also provisionally set. The AVA then submits the evaluation of its activities, the Analytical Account, together with other supporting material which describes its quality assurance procedures and standard operational processes. The Analytical Account is prepared with reference to guidelines issued by QAA.

140 Some six weeks prior to the review, the selected review team attend a planning meeting, the principal purpose of which is to enable the team to decide on its main lines of enquiry and to establish a schedule and draft programme for the review visit. The team may also request further documentation from the AVA if it believes that this will assist in its full understanding of the AVA's structures, processes and procedures. The review team also meets on the day immediately preceding the visit to discuss any additional information which has been received.

During the visit

141 The review visit normally extends over two days. The common pattern is for the first day to be spent in discussion with groups involved in, and affected by, the AVA's activities. For each meeting, the reviewers work to a previously agreed agenda. The QAA Assistant Director makes notes of the discussion at all meetings. The review team will also allocate time to read and consider other written sources of evidence during the visit: in most cases, these comprise AVA records related to a selected number of individual Access programmes. On the afternoon of the second day, the team considers its key findings in relation to the licensing criteria and the areas it wishes to highlight in its report.

142 The recommendation to be made to the ARLC is also decided before the end of the visit. The outcomes following a review are:

  • unconditional confirmation of renewal of licence for a specified period;
  • conditional confirmation of licence with conditions to be met by a specified date or dates;
  • provisional confirmation of licence with conditions to be met and a further review visit by a specified date;
  • withdrawal of licence for operation as an AVA;
  • temporary renewal of licence with request for further information by specified date (decision suspended).

After the visit

143 According to a division of responsibilities agreed at the planning meeting, the team compiles its report in the two weeks following the review visit. Since the formal approval of the principles and accompanying criteria in 2001, reports have been written to a template which reflects those principles. The QAA Assistant Director is responsible for the compilation and editing of the final report using the contributions from the two reviewers. The confirmed report is presented to the next scheduled meeting of the ARLC by a member of the review team who answers any queries which the ARLC may have about the report or the team's recommendation. On the basis of the draft report and its own discussions, the ARLC reaches a decision on the renewal of the AVA's licence.

144 At this point, the agreed report and the decision of the ARLC is forwarded to the AVA. The AVA is asked to consider the report and to highlight any perceived factual inaccuracies or judgements which it believes are based on a misunderstanding of the facts. The response of the AVA is circulated to the review team who may then agree amendments to the text of the report. In cases where the AVA's response indicates that the security of the final judgement may be in doubt, the report and response is referred back for reconsideration by the ARLC.

145 In cases where conditional or provisional judgements are reached, the AVA supplies the requested additional material by a specified date or dates. The reviewers are asked to consider this material and indicate whether they believe that the conditions have been appropriately met. All reviewers' responses are considered by the the ARLC prior to informing the AVA of the committee's decision about the renewal of the AVA's licence.

Evaluating the review process

146 Following a licence review, QAA invites each AVA to submit an evaluation of the review process. Guidelines are offered which suggest structuring the evaluation under the three headings of: the preparation for the review, the review visit, and the review report and follow-up. By no means all AVAs took advantage of this invitation during the review cycle but where evaluations were received, they were considered by ARLC. Members of review teams are also asked to complete a Feedback Report following each review which included a self-assessment section to enable them to reflect on their own performance during the review. Evidence from AVA evaluations and reviewer feedback has been used in this section, although in neither case has it been ascribed.

The planning stages

147 It is clear that both AVAs and reviewers appreciated the advice and assistance of the QAA Assistant Director who managed the complete cycle of reviews from 1999 to 2004. AVA evaluations commented in particular on the usefulness of the initial planning meeting and the opportunity to understand fully the scope and logistics of the review process. These comments were supported by others which noted the readiness with which advice and support was offered during the whole of the review process. An appreciation of the work of the Assistant Director was also a common theme in reviewers' feedback particularly the support given to new reviewers and the general expertise provided in Access matters. There seems little doubt that this cycle of reviews benefited significantly from the consistent presence of an expert officer: it helped to ensure the overall stability of the review process and the reliability of judgements delivered by review teams.

148 In the early part of the review cycle, AVA evaluations commented on the brevity of the period between the planning meeting and the review visit. AVAs reported difficulties in preparing for the visit with what they considered to be the late receipt of a detailed programme. As the review cycle progressed, a longer period of time between the planning meeting and the review visit was built into the process. In addition, as the process developed and programmes began to share common characteristics, it became easier to provide guidance to AVAs about the likely pattern of, and participants in, the review programme. Although AVAs would wish for as long a period as possible in which to prepare, it is likely that this period has now been extended as far as is possible without creating such a large gap between the two events that the initial preparatory work of the review team is invalidated.

Review teams

149 The basic principles in the selection of review teams are that they are knowledgeable about the Recognition Scheme and Access in general, and are able to examine an AVA independently and impartially. Many reviewers commented on the importance of achieving a balance in review teams reflecting complementary interests and expertise. They had also found the training offered by QAA to be appropriate in most respects - particularly the inclusion of observation at a 'live' review - although some felt that further emphasis on report writing would have been welcome.

150 Building on one aspect of practice inherited from the first HEQC round of reviews, virtually all review teams contained at least one member of the ARLC. The functional reason for this was to facilitate the presentation of reports to the committee. However, it also reflected a belief that, in order to undertake its responsibilities properly, the ARLC needed not only to be an expert committee but also to be in constant touch with the major process which it employed for its licensing activities. In a very practical sense, the ability of the committee to undertake the licensing of AVAs, and the expert nature of its discussions, are aided considerably when all members have direct experience of the review process. A balance needed to be struck, however, between such direct experience and the consistency it adds to the process, and the possible limiting of the cohort of reviewers (in all 25 individuals were involved in AVA review of whom 12 undertook more than one review). With a relatively small number of reviews being undertaken in any one year, there were limited opportunities for drawing in new reviewers, developing the experience of others and ensuring that all review teams included at least one experienced member. That there were few negative comments made by AVAs on the composition of review teams tends to suggest that the personnel, mix and balance was generally appropriate.

Analytical Accounts

151 As might be expected across a large number of AVAs, the nature of Analytical Accounts varied greatly. The feedback from review teams frequently contained references to Accounts which were particularly helpful or unhelpful. In the former category were those Accounts which set out the major activities of the AVA in a clear and comprehensible manner; included useful guides to major changes since the previous review visit; provided evaluations of the AVA's work in key areas often indicating those where some improvements were needed or were underway; and included well-organised and informative appendices with appropriate cross-references to them in the text. Less helpful Accounts shared few or none of these qualities.

152 A good Analytical Account is valuable for a number of different reasons including its primary purpose of allowing the AVA to reflect upon its own activities as a preface to its licence review. Whatever the future development of AVA review, it is difficult to imagine that self-assessment of some kind will not continue to form an integral part of the process. One message from this cycle of AVA review is that further advice and guidance from QAA on the compilation of Accounts and further sharing of practice between AVAs would be worthwhile exercises.

The review process and review visits

153 The essence of the review process is the statement and evaluation by the AVA of how it considers it is meeting the requirements of the Recognition Scheme, and the testing of this by the review team. A further essential feature of the process is that review teams cannot cover every aspect of the operation of an AVA during a licence review and therefore must employ sampling techniques to decide on the effectiveness of the AVA's work, its quality assurance arrangements, and its overall operation of the licence. The review process seeks to ensure the robustness of its findings through the proper construction of sampling, the careful consideration of the evidence provided, consensus between team members, and the advice of the Assistant Director who ensures a level of consistency of practice between reviews.

154 One of the key sampling techniques, the range of meetings conducted during a review, appeared satisfactory to both AVAs and review teams themselves. Although there was some variability at the beginning of the cycle, on the evidence of the review reports, the nature and number of meetings settled into a reasonably predictable pattern. People invariably seen included AVA officers and staff, members of key committees including those specifically for quality and Access matters, moderators, Access programme coordinators, HE admissions tutors, and students. The only issue raised by review teams was that, not infrequently, personnel who had been invited to attend a meeting as part of the agreed schedule were then not able to be present at the review itself. The absence of key personnel, or the general overall reduction in the size of groups, may in some cases have had an effect on the conduct of the review, for example, where a review team had asked to meet eight or 10 Access programme leaders but where only two or three were eventually able to attend. In such cases, the evidence gathered from the meeting needs to be treated with particular care since it is much less likely to constitute a reasonable sample from which to draw conclusions. There is every sympathy for AVAs in this context: in all reviews, the majority of people seen were not employed by AVAs who therefore had little absolute control over their commitments and eventual presence. However, if such meetings are to continue to be used in the process as an important source of evidence, then some resolution to this issue needs to be found.

155 Other than in this area, there was much praise from review teams on the good administration of review visits by AVAs. Additional evidence for sampling, such as the papers related to a range of individual Access programmes validated by the AVA, were generally made very accessible, were well laid out and referenced, and made this part of the review team's work significantly easier.

156 There was little in the feedback from AVAs to suggest that the review visit had not been conducted satisfactorily in the majority of cases. There were some occasional comments that teams may have been overly adversarial in their tone and approach, but in many more cases teams were noted for the professional and thorough approach which they took.

Decision-making process

157 On the part of AVAs, the most common criticism of the whole review process was the lack of any immediate feedback at the conclusion of the review visit itself and the provision of some sense of closure to an exercise which, at that point, would already have extended over several months. It has been the position of QAA throughout that it is the ARLC and not the review team which has the authority to make decisions on the renewal of a licence. For this reason, QAA has considered that the review team should not share their recommendations with the AVA at the conclusion of the review visit, since those recommendations may be subject to amendment by the committee.

158 This would not be a sustainable position if the ARLC's approval of a review recommendation was a cursory action. However, as suggested in paragraph 143 above, the consideration by the ARLC of the report and recommendation of a review team is not perfunctory and may involve a good deal of detailed discussion. These discussions can lead to changes to review reports and their conditions. Part of the rationale for this method of operation is to ensure that decisions on licence renewal are as equitable and consistent as possible. Although some feedback to AVAs at the end of a review visit might be perceived as desirable, it would remain an inappropriate exercise for as long as the ARLC undertakes its responsibilities for licence renewal by the current method.

Future options

159 The purpose of this concluding section is to reflect on the nature and conduct of AVA review and to discuss the possible options and developments for the next cycle. An initial question, of course, is whether there should be any changes at all. With some necessary amendments, can the basic review process used between 1999 and 2004 be employed again? The issue is perhaps not the durability of the review process, but the nature of what is being reviewed. There have been substantial developments and significantly new challenges for AVAs in the period since 1999. Some of these, for example, the growth of internal verification methods, the introduction of internal audit, the changing requirements of regulatory and funding bodies, have been touched upon in this report. There would seem little doubt that the process of licence review must also change in order to ensure that it is appropriate for the next stage in the development of AVAs and Access provision.

160 It might be argued that there are three factors which should influence changes to the nature of AVA review:

  • the development of the relationship between periodic licence review and the annual reviews which QAA requires from each AVA;
  • the report of the Access to Higher Education Development Project;
  • the impact of the 1999-2004 cycle of reviews.

161 The annual review process applied to AVAs has been progressively developed by the ARLC since its piloting in 1999. ARLC devotes a substantial amount of time to the consideration of these reviews and, increasingly, uses them as a significant indicator of the continuing health of an AVA. The logic of this development is that the use of annual reviews must be allowed to influence the conduct and application of periodic review. In a context where annual review is well established, and where the ARLC is increasingly confident of the evidence which it provides, should an AVA which consistently demonstrates its effectiveness and suitability to hold and operate its licence, be subjected to a different form of periodic review? If the two processes of annual and periodic review are to be seen as coherent and mutually supportive, then this option would appear to be an appropriate one to explore in more detail. In so doing, the nature and content of the annual review submitted by an AVA would need to be the subject of further consideration to ensure that the report is capable of supplying the necessary and consistent level of assurance for the ARLC.

162 The Access to Higher Education Development Project has already been mentioned several times in this report. If the recommendations of the Project report are accepted and implemented, they will constitute a substantial development of Access and will perhaps lead to a different concept of the AVA. Some new or expanded responsibilities would be assigned to AVAs leading to changes to the licensing criteria and, consequently, changes to the nature of AVA review.

163 The impact of the 1999-2004 cycle of reviews should not be ignored, particularly where good and consistent practice has been identified, or where relatively straightforward operational features have been checked. Inevitably, part of the cycle of reviews has been devoted to straightforward monitoring against the criteria (a process which has been made significantly easier for review teams since the formal articulation of the principles and licensing criteria). If the 1999-2004 cycle of AVA reviews has been effective, then the next cycle should not need to monitor in detail, for example, the process of recognition (a clear point of consistent and good practice) but might wish to focus on how AVAs assist in targeting the development of new programmes; equally, there should no longer be the need simply to check the existence of an equal opportunities policy in an AVA, but more emphasis on how such a policy, and others like it, is implemented across the activities of the AVA.

164 Although the nature of what is reviewed may be changing, and given that any process can always be enhanced and improved, there is little from AVA review to suggest that the fundamentals of the current method need to be radically amended. A peer process, the catalyst for which is self-assessment, involving a visit and a variety of sampling techniques, might still provide the basic platform for review. Considering the three factors above, however, the themes which might dominate the next cycle and the philosophy of the review method might change. It is likely, for example, that such themes might include:

  • self-assessment and a greater use of, and focus on, an AVA's internal quality audit or assessment practices as an indicator of its maturity as an organisation;
  • the academic standards of Access programmes and the ways in which AVAs use the internal moderation practices of their providers and their own external moderation in order to ensure that the Access award is confirmed as a national qualification indicating clearly a readiness for higher education;
  • the regional role of an AVA, its involvement in broader policies and strategies for adult learning, the purposeful targeting of disadvantaged groups, and the generation and analysis of data in order to achieve this;
  • the conception of the AVA as a 'training' body as well as a validating body, being equipped and willing to provide a broad programme of initial and updating opportunities for the development of all those involved in Access provision;
  • if the recommendations of the Project are implemented, the ways in which AVAs adopt and implement the qualification and subject benchmarks that are envisaged.

165 As for the 'philosophy' of an amended review process, QAA has within its own operational experience what might appear to be a helpful model. The first complete round of academic audits of HEIs from 1991 to 1997 was characterised in part by its relatively detailed scrutiny of quality assurance processes and their practical operation. QAA's reflection on the nature of the second round of audits led it to develop and then implement the process called 'continuation audit'. In essence, the difference between the two processes was one of focus; where first-round audit teams checked for themselves whether quality assurance processes were operating effectively, continuation audit teams were more interested in scrutinising an HEI's own methods for providing such assurance. Key questions, therefore, were how does an institution know that it is discharging effectively its corporate responsibility for the standard of each award granted in its name? And how does it assure the sufficiency, validity and reliability of the evidence that it is relying on for this purpose?

166 AVA review between 1999 and 2004 had already adopted some of the key characteristics of continuation audit, particularly the reliance on the AVA's own account of its activities as a basis for the conduct of the review. It is an option for the future for AVA review also to adopt that approach of continuation audit which scrutinises not the detail of processes, but the ways in which an AVA itself undertakes the assurance of the quality of those processes.

 

Appendix A

Authorised Validating Agencies

 

Authorised Validating Agency

Acronym (as used in this report) Further information Notes
Anglia South Open College Network ASOCN www.asocn.co.uk  
Assessment and Qualifications Alliance AQA www.aqa.org.uk  
Cambridge Access Validating Agency CAVA www.cava.ac.uk  
Chiltern Region Open College Network and Access Consortium CROCNAC www.crocn.org.uk  
Greater Manchester Open College Network GMOCN www.gmocn.ac.uk  
Hampshire Authorised Validating Agency HAVA    
London Open College Network LOCN www.locn.org.uk  
Merseyside Open College Network MOCN www.mocn.co.uk  
North East Midlands Open College Network NEMOCN www.nemocn.org.uk  
North Wales Open College Network NWOCN www.nwocn.org.uk Known as North Wales Access and Credit Consortium (NWACC) a tthe time of its review in April 2000.
Open College Network (North and East London and Hertfordshire) OCN (NELH) www.ocnetwork.co.uk  
Open College Network (SouthYorkshire and Humber Region) OCNSYH www.shu.ac.uk/ocn  
Open College Network (TROCN) TROCN www.trocn.co.uk  
Open College Network for Central England OCNCE www.ocnce.warwick.ac.uk  
Open College Network Kent and Medway OCNKM www.ocnkm.ac.uk  
Open College Network North West Midlands OCNNWM ocnnwm@staffs.ac.uk  
Open College Network of the South West OCNSW www.ocnsw.org.uk  
Open College Network of the West Midlands OCNWM ocnwm.wlv.ac.uk  
Open College Network South East Midlands OCNSEM ocnsem@ocnsem.com  
Open College Network West and North Yorkshire OCNWNY www.wnyocn.org.uk Known as West and NorthYorkshire Open College Network at the time of its review in November 2001.
Open College of the North West OCNW www.ocnw.com  
Oxfordshire Open College Network OxOCN ocn@oxocn.org.uk  
South East Wales Open College Network SEWOCN croeso@sewocn.co.uk  
South of England Open College Network SEOCN www.brighton.ac.uk/seocn/  
Thames Region Accrediting Consortium Open College Network TRACOCN www.tracocn.co.uk  
Western Region Open College Network WROCN info@wrocn.co.uk Known as Open College Network West at the time of its review in March 2002

Appendix B

Details of AVA reviews and licence applications, 1999-2004

The titles of AVAs are those in use at the time of the review or licence application

AVA Review date Review team Outcome
1998-99 (academic year)      
Open College of the North West 26-27 April Professor Beverly Sand (University of Derby) Dr Clive Linnett (Bradford & IlkleyCommunity College) Provisional Renewal
Open College Network of the South West 20-21 May Dr Peter Easy (Cheltenham & Gloucester College of HE) Ms Miriam Griffiths (Open University in Wales) Conditional Renewal
Open College Network for Central England 16-17 June Professor Steve Bristow (Shrewsbury College of Arts& Technology) Ms Margaret Davidson (Charles Keene College) Unconditional Renewal
Bedfordshire Access Consortium* 24-25 June Dr Pete Johnston (University of Essex) Ms Carole Stott (LOCN) Provisional Renewal
1999-2000      
North East Midlands Open College Network 9-10 November Dr Peter Easy (Cheltenham & Gloucester College of HE) Ms Sue Georgious (OCNC) Conditional Renewal
Open College Network NorthWest Midlands 17-18 November Dr Pete Johnston (University of Essex) Mr Alan Smith (NWACC) Conditional Renewal
University of Ulster Authorised Validating Agency* 8-9 December Professor Beverly Sand (University of Derby) Ms Miriam Griffiths (Open University in Wales) Conditional Renewal
Open College Network (North and East London and Hertfordshire) 10 February Dr Pete Johnston (University of Essex) Professor Steve Bristow (Shrewsbury College of Arts & Technology) Licence Awarded
South East Wales Open College Network 1-2 March Dr Peter Easy (Cheltenham & Gloucester College of HE) Mr Anthony McClaran (UCAS) Provisional Renewal
South West Wales Open College Network* 15-16 March Dr Geoffrey Copland (University of Westminster) Mr Steve Babbidge (SEOCN) Provisional Renewal
North Wales Access and Credit Consortium 6-7 April Professor Steve Bristow (Shrewsbury College of Arts & Technology) Ms Christine Davies (Nottingham Trent University) Conditional Renewal
London Open College Network 1-2 June Professor Beverly Sand (University of Derby) Mr Mike Farmer (Cheltenham & Gloucester College of HE) Conditional Renewal
Tyne and Border Counties Access Partnership* 14-15 June Professor Janet Finch (University ofKeele) Mr Malcolm Barry (Goldsmiths College, University of London) Licence Withdrawn
2000-01      
Open College Network Kentand Medway 8-9 November Dr Pete Johnston (University of Essex) Mr Malcolm Barry (Goldsmiths College, University of London) Provisional Renewal
Oxfordshire Open College Network 22-23 November Mr Steve Babbidge (SEOCN) Ms Priscilla McGuire (OCNWNY) Provisional Renewal
South of England Open College Network 22-23 March Mr Anthony McClaran (UCAS) Ms Sue Georgious (OCNCE) Conditional Renewal
Assessment and Qualifications Alliance 27-28 February Dr Peter Easy (Cheltenham & Gloucester College of HE) Mr Alan Smith (NWOCN) Provisional Renewal
Thames Region Accrediting Consortium Open College Network 7-8 June Professor Beverly Sand (University of Derby) Ms Gill Evans (OCNNWM) Conditional Renewal
Chiltern Region Open College Network and Access Consortium 13 June Dr Pete Johnston (Independent) Professor Tony Chapman (University of Wales Institute Cardiff) Licence Awarded
2001-02      
West and North Yorkshire Open College Network 20-21 November Dr Pete Johnston (Independent) Mr Anthony McClaran (UCAS) Conditional Renewal
Open College Network West 5-6 March Mr Steve Babbidge (SEOCN) Mr Anthony McClaran (UCAS) Provisional Renewal
Hampshire Authorised Validating Agency 15-16 May Dr Pete Johnston (Independent) Dr Philip Bentley (Shrewsbury College of Arts & Technology) Conditional Renewal
Cambridge Access Validating Agency 28-29 May Dr Peter Easy (University of Gloucestershire) Dr Rob Allen (University of Greenwich) Provisional Renewal
2002-03      
South West Wales Open College Network* 5 November Dr Pete Johnston (Independent) Dr Andy Thompson (NESCOT) Application Referred
Open College Network South East Midlands 19-20 November Mr Steve Babbidge (SEOCN) Professor Colin Raban (Edge Hill College of HE) Conditional Renewal
Greater Manchester Open College Network 11-12 February Ms Sue Georgious (OCNCE) Mr David Burtenshaw (HAVA) Conditional Renewal
Open College Network (TROCN) 18-19 February Dr Pete Johnston (Independent) Ms Jill Ward (University of Derby) Conditional Renewal
Open College Network of the West Midlands 13-14 May Dr Peter Easy (University of Gloucestershire) Mr Alan Smith (NWOCN) Conditional Renewal
Merseyside Open College Network 7-8 May Mr Mike Farmer (University of Gloucestershire) Dr Philip Bentley (Shrewsbury College of Arts & Technology) Provisional Renewal
2003-04      
Open College Network (South Yorkshire and Humber Region) 12-13 November Mr Steve Babbidge (SEOCN) Ms Jill Ward (Wedgwood Memorial College) Provisional Renewal
Anglia South Open College Network 19-20 January Dr Peter Easy (University of Gloucestershire) Mr Keith Fletcher (OCNSW) Conditional Renewal

Normally, a review team included one member of the ARLC in order to involve the committee directly in the process of licence review (and to facilitate the presentation of the draft report to the committee). ARLC members are identified by the use of italics. In cases where no ARLC representative was present, the team included at least one member with previous experience of AVA review, who was invited to report on the review outcomes to the ARLC. In the case of applications for new licences, both team members are drawn from the ARLC.

* = no longer operating as an AVA.

 

ISBN 1 84482 083 1

TopTop