Foreword
1 The Quality Assurance Agency for Higher Education (QAA) is responsible to the Department for Education and Skills (now the Department for Innovation, Universities and Skills) for the recognition of Access to Higher Education courses. QAA exercises this responsibility through a national network of authorised validating agencies (AVAs), which are licensed by QAA to recognise individual Access to Higher Education (HE) courses, and to award Access to HE qualifications. The AVAs are responsible for implementing quality assurance arrangements in relation to the quality of Access to HE provision and the standards of student achievement. QAA has developed a scheme for the licensing and review of AVAs, the principles and processes of which are described in the QAA Recognition Scheme for Access to Higher Education in England, Wales and Northern Ireland. The Recognition Scheme is regulated and administered by the Access Recognition and Licensing Committee (ARLC), a committee of the QAA Board of Directors.
2 The ARLC is responsible for overseeing the processes of AVA licensing and periodic review and relicensing. The criteria applied by the ARLC and by review teams operating on the Committee's behalf, in reaching judgements about whether and under what terms an AVA licence should be confirmed or renewed, are provided within the Recognition Scheme documentation. These criteria are grouped under the four principles that provide the main section headings of this report.
3 Following an AVA review, a member of the team presents the team's report to the ARLC. The Committee then makes one of four decisions:
i unconditional confirmation of renewal of licence for a specified period
ii conditional confirmation of licence with conditions to be met by a specified date
iii provisional confirmation of licence with conditions to be met and a further review visit by a specified date
iv withdrawal of licence for operation as an AVA.
4 This is a report on the review for the Assessment and Qualifications Alliance (AQA). QAA is grateful to AQA and to those who participated in the review for the willing cooperation provided to the team.
The review process
5 The review was conducted in accordance with the process detailed in the QAA Recognition Scheme for Access to Higher Education in England, Wales and Northern Ireland. The preparation for the event included an initial meeting between a QAA officer and AQA representatives to discuss the requirements for the Overview Document (the Overview) and the review process; the preparation and submission by AQA of its Overview, together with a selection of supporting documentation; a meeting of the review team to discuss the Overview and supporting documentation and to establish the main themes and confirm the programme for the review; and negotiations between QAA and AQA to finalise other arrangements for the review.
6 The review visit took place on 12 to 14 March 2007. The visit consisted principally of meetings with representatives of AQA, including AVA officers; members of the AQA Council, the Quality Assurance Committee and Access Management Group; moderators for Access to HE programmes; Access to HE coordinators; and representatives from HE.
7 The review team consisted of Professor Brian Anderton, Associate Dean (Academic Planning and Development), University of Central England in Birmingham; Mike Farmer, education consultant, Farmer Associates, member of the ARLC and lead reviewer; Professor James Hughes, formerly Professor of Industrial Relations, University of Kent. The review was coordinated for QAA by Kath Dentith, Head of Access.
The AVA context
Background and developments since the last review
8 AQA is a company limited by guarantee and a registered charity. The company in its present form was created in April 2000, following a merger between the Associated Examining Board and the Northern Examinations and Assessment Board (NEAB). AQA is accredited by the Qualifications and Curriculum Authority to award a broad range of qualifications, including GCSE and GCE A-levels. The original AVA licence was granted to the Joint Matriculation Board (one of four constituent bodies which were amalgamated to form NEAB) in 1991, transferred to the NEAB when it was established in 1992 and to AQA in 2000.
9 The organisation is the largest of the three English examination boards. Its annual turnover is in excess of £130 million and it employs around 900 full-time staff and a substantial number of part-time staff. Income directly attributable to Access to HE is less than 0.1 per cent of the overall total.
10 AQA has offices in Guildford, Manchester and Harrogate. The AVA functions are coordinated and managed from the Harrogate office, having been relocated from the Manchester office since the previous review in 2001.
11 The last review resulted in the provisional renewal of the AVA licence, subject to nine conditions. The conditions were approved by QAA as having been met in four stages over the period July 2002 to July 2004. The review team saw evidence to indicate that action to meet these conditions had led to a number of improvements. In particular, positive developments have taken place with regard to clarification and strengthening of governance structures for the AVA; the development of regular strategic planning processes relating to Access to HE; the collection and analysis of statistical information from providers; and clarification of moderators' roles.
12 The AVA has provided regular annual reports on its operations to QAA. One was not approved because of the amount of material that was missing or not presented on the required format; one was approved after further information had been submitted; and three reports were approved by QAA with no further action required.
AVA statistics 2005-061
Providers and programmes
Total number of providers 11
Total number of Access to HE programmes available 30
Total number of Access to HE programmes running 30
Total Access to HE learner registrations 1,985
Total Access to HE certifications 880
13 Over the period since the last review, until 2005-06, the number of providers remained fairly static at ten or 11, while the number of programmes rose from 16 to 30 and the number of registered learners rose significantly from 590 to 1,985. At the time of the review, three new providers had joined the AVA, bringing the total to 14 and the number of programmes available to 37.
Principle 1
The organisation has governance structures which enable it to meet its legal and public obligations, to render it appropriately accountable, and to allow it to discharge its AVA responsibilities securely
Legal and constitutional status
14 AQA is a company limited by guarantee and has charitable status. The governing body is the Council, consisting of 29 trustees, including representatives of education, local government and business. The Articles of Association and terms of reference of its committees identify the constitutional arrangements and governance structures, the appointment of Council members, and the appointment of the Director General and Deputy Director General.
Council
15 Of the 29 Council members, eight are drawn from schools and colleges, and seven from higher education institutions (HEIs). The Director General, his Deputy and other members of the Executive Board (see paragraph 29, below) also attend Council meetings, thereby ensuring that, whatever subject is under discussion, there will always be someone with direct line management responsibility present to answer any queries that might arise during the Council's deliberations.
16 The Council is the locus of all decision-making in AQA, including all matters relating to Access to HE. Thus it has ultimate authority for the AVA licence. The composition of its membership and the relationship between the Council and the committee and management structures which are responsible for Access to HE matters are, therefore, as noted in the 2001 report, of some importance. Of the eight Council places reserved for schools and colleges, none is currently filled by a representative of a further education (FE) member of the AVA 'partnership' (see paragraph 24, below), while only one of the university places is filled by a representative of one of the AVA's 'partner' institutions. AVA partnership representation on Council is therefore limited. Clearly, from an AVA point of view, having a Council member drawn from one of its FE partners with direct and current understanding of Access to HE, would be seen as a positive development.
17 The Articles of Association distinguish between governance committees, which take decisions and determine policy because Council has delegated specific powers to them, and advisory committees, which have no decision-making powers. Council has recently reviewed the committee structure and terms of reference of all committees and, as a consequence, both types of committee are now chaired by a trustee and sometimes include additional trustee members. The main governance committees which have a bearing on the work of the AVA are the Council Access Review Group (CARG) and the Access Management Group (AMG). Although the AVA does not have any advisory committees, there is a Quality Assurance Committee (QAC), which is a subcommittee of AMG, and its Chair, who is elected by the membership, becomes an ex officio member of the AMG.
Council Access Review Group
18 CARG is not listed as part of the formal committee structure but is a small subgroup of Council, consisting of five trustees, including the Chair of Council. It meets only once a year and its sole function is to consider the AVA's annual report to QAA on behalf of Council, and advise the Chair of Council on its approval. The minutes of CARG confirm that it discharges its scrutiny and amending role very thoroughly before approving the draft report on behalf of Council. While the report does not, therefore, go to a full Council meeting for approval, the review team considered that this development was a positive one in ensuring that at least five members of Council, including its Chair, gave the report detailed scrutiny and thereby received and considered detailed information about the AVA's activities.
Access Management Group
19 The AMG is responsible for managing all AVA activities and advising Council on the strategic direction for AQA's Access to HE work. In addition to the trustee Chair appointed by Council, there are ten members, divided equally between representatives of HE and FE. Nine of these members are currently drawn from, and nominated by, partnership institutions and this ensures that the partners have a prominent role in both the management of Access to HE and the development of strategy within the AVA. The authority of AMG is reinforced by the fact that it is chaired by a trustee and reports directly to Council.
20 In its management role, AMG is involved with all aspects of Access to HE, including approving the validation of programmes; reviewing programme reports from college coordinators; approving the draft AVA annual report; approving and monitoring the action plan; and managing external relations with QAA and members of the partnership. It is assisted in its work by the QAC, which reports to it.
Quality Assurance Committee
21 The QAC has eight members, all of whom are appointed by AMG from nominations submitted by the partners. As noted above, these members elect a Chair from among themselves. Its other seven members are appointed from nominations submitted by the partners. As QAC is formally neither a governance nor an advisory committee of AQA (see paragraph 17, above), its terms of reference are not included in AQA's main document, which details the terms of reference for those committees in a consistent format with common categories of information. Rather, the QAC's terms of reference are provided in a separate document with a different format. QAC's terms of reference specify that 'Members will be appointed for a period of three years'. However, unlike committees for which terms of reference appear in the main document, no limit to the number of times a member might be reappointed is specified. Presumably this omission is an oversight, and the team recommends that this be addressed when the committee's terms of reference are next reviewed. In making appointments to QAC, AMG ensures there is 'an appropriate spread of expertise and experience', not only between HE and FE, but also between those with expertise in curriculum matters, quality issues and experience of running Access to HE courses. That the majority of its members are from partnership institutions and collectively have a wide spread of expertise ensures that QAC is appropriately constituted to keep AMG informed about developments affecting Access to HE within the partnership.
22 The Committee, which meets three times a year, is an important advisory committee to AMG. Its key functions are to monitor the operation of all Access to HE courses within the AVA, to oversee quality assurance processes and, where necessary, make recommendations to AMG. QAC considers in detail all moderator reports, validation and revalidation reports, as well as any other issues affecting the quality of courses. As well as providing oversight of the quality of the AVA's provision in these ways, through the work of the Committee, individual QAC members are directly involved - and are therefore a key resource - in the validation process itself, since all validation panels include two members of QAC (see paragraph 57, below).
The AVA partnership
23 Membership of the AVA partnership consists of all those providing institutions that deliver Access to HE courses validated by the AVA and some of the HE institutions (HEIs) that admit students from these courses. Currently there are 14 partners in each group, although it is part of the AVA's strategy to increase both (see paragraph 36 and 40, below).
24 The partnership acts in a purely advisory capacity to AMG, although its members play a central role in electing AMG as well as providing the majority of members for both AMG and QAC. The AVA partnership was originally established as a mechanism to identify the AVA's stakeholders and allow them to be involved in the strategy and development of the AVA. The partnership is now defined by a Memorandum of Association which provides categories of membership, rights, and duties and obligations of members. The introduction of the Memorandum clarifies a number of matters about the AVA's expectations of those it works with. However, the constitutional significance of the Memorandum is unclear. Indeed, with the strengthening of the governance structures and procedures through which AQA meets its formal responsibilities as an AVA, the purpose of the partnership itself - which stands outside those structures - has become less certain.
25 The annual partnership meeting provides a useful forum for discussion of both local and national developments in Access to HE, and staff believe that communication and feedback from the meeting inform the day-to-day thinking of AVA officers and influence longer-term strategy. Attendance at these meetings has been low in recent years, particularly among HE representatives. An explanation for this poor attendance might be that offered by the AVA itself in its self-evaluation document, namely that the 'precise remit (of the partnership) still needs further clarification'. A further contributory reason for poor attendance by HE representatives is that the AVA may not be targeting those in HE with most direct interest in the development of Access to HE.
26 Members of AMG expressed the view that more could be done to make the partnership meetings more relevant and proposed that they should be used as a forum for consultation on strategic issues. While, in the past, the AVA has not consciously used partnership meetings to either consult on or promote its strategic objectives, the minutes of the March 2007 meeting suggest that, in future, strategic issues might figure more prominently as agenda items.
27 The partnership meeting has the potential to provide an important forum for facilitating the flow of information and the exchange of ideas, but it is not currently fulfilling that potential. The review team considers that, in order to do so, there would need to be a change of focus, with agendas reflecting strategic issues, and an increase in the participation of the HE members.
28 Although there are still matters relating to partnership and the development of strategy to be resolved (see also paragraphs 40 to 46, below), the review team considers that there have been several positive developments in governance since the last review. In particular, the appointment of a trustee to chair AMG and the clarification of the Group's role in the governance structure; the establishment of CARG; and the increased representation of partnership institutions on QAC and AMG are all positive developments which lead the team to believe that the AQA Council is a responsible location for the AVA licence. The team also noted the statement in the 2005-06 annual report, that the AVA 'remains an institutional priority for AQA'.
Principle 2
The organisation is able to manage its AVA responsibilities effectively, and to maintain an appropriate structure to support them
Management
29 The Director General and Deputy Director General of AQA, together with five divisional directors, constitute the Executive Board. The Executive Board is responsible for the management of the organisation and the implementation of the Council's strategy. Each director has line management responsibility for three or four assistant directors. Together, the members of the Executive Board and the assistant directors form the Senior Management Team (SMT). Within the SMT, the Assistant Director, Business Management (Harrogate), has some responsibility for Access to HE including line management of the Access to HE officer. The Assistant Director reports to the Director of Finance, although the review team heard that Access to HE matters are also reported to the Director of the Qualifications Development and Support Division, when appropriate. The job description states that the Assistant Director has 'overall responsibility for all aspects of the work of the Harrogate office and, in particular, for the overall management of staff and the development of new business opportunities in the Harrogate office', as well as a wide range of 'specific accountabilities', including 'encouraging best practice and most effective use of resources in the overall management' of a number of specified areas, including Access to HE. The post holder is also required to 'provide continuing leadership' for the teams which manage the various qualifications administered from the Harrogate office. Inevitably, the time which the Assistant Director is able to dedicate to managing and providing leadership for Access to HE activities will be limited.
Access to HE section
30 The Access to HE section consists of four individuals, three of whom are on fractional, fixed-term contracts. There are two section heads and a clerical assistant, all working half-time (0.5) and, at a more senior grade, an Access to HE Officer on a 0.6 contract. Thus, ignoring grade differences, the Access to HE team is equivalent to 2.1 full-time staff. This amounts to almost a doubling of dedicated staff on the previous year although, at that earlier point, administrative support was being provided to Access to HE by staff outside the section. According to the job descriptions for these posts, the team is based in the Division of Qualifications Development and Support, and the Access to HE Officer reports to a Principal Subject Manager. In discussion with the Access to HE Officer, the review team heard that although the Access to HE Officer once reported to a Principal Subject Manager, she now reports directly to the Assistant Director, Business Management. The job descriptions for staff in the Access to HE section would appear to be in need of updating. The review team recommends that job descriptions are reviewed to ensure that they reflect reality and, in particular, that reporting and line management responsibilities are clear.
31 The two half-time section heads, who are the first points of contact for partners and are responsible for the day-to-day running of the Access to HE section, report to the Access to HE Officer whose overarching responsibility is to 'administer and develop' Access to HE within the AVA. This responsibility embraces a range of duties, including servicing committees and validation panels; coordinating moderation arrangements; collecting and analysing colleges' data; and preparing the annual report to QAA.
32 The review team noted that the AVA had suffered from a high turnover of Access to HE Officers during the last two years and, for substantial periods of time, this post has remained unfilled. The post remained unfilled for several months in late 2005 and the failure to fill the post quickly prompted the Chair of AMG to express his concern to Council about the level of Access to HE staffing and the need for Council's support. While the post was then filled, it again fell vacant again in August 2006.
33 A lack of dedicated Access to HE staff and, in particular, the availability of appropriate expertise for the future, was identified as an issue in the 2001 review. Now, although the number of staff with dedicated Access to HE posts has increased, the team remains small. When members of Council met the review team they expressed some 'concern' about whether, with current staffing levels, the AVA would have the capacity to take on additional provision, as planned, as well as taking forward the work to implement the Access to HE Diploma. The review team was informed that the staffing situation would be reviewed later in the year and, if the expansion in AVA activity that is planned was realised, then a case for additional staff would be made. The team considered this a positive move.
34 The review team concurred with the Council representatives' view that, given the demands associated with the AVA's responsibilities, including demands relating to the introduction of Access to HE Diploma, there must be some concern about whether the current level of staffing could provide both the management and support necessary to ensure that the AVA is able to discharge its responsibilities fully and effectively. The team's concerns were reinforced by recent instances of lapses in quality assurance procedures (as described later in this report), which suggested that the Access to HE team was working under significant pressure. Given these circumstances, the team concluded that the Access to HE staff would be unlikely to have any spare capacity for engaging in additional development activity.
35 The AVA is therefore required, as a condition of licence, to carry out a review and analysis of its staffing arrangements for Access to HE, and report on the outcomes of the review to demonstrate how it will ensure that its responsibilities as an AVA licence holder will be met, including responsibilities for quality assurance and development, actions identified in its action plan (see paragraph 36, below), actions relating to the implementation of the Access to HE Diploma, and actions relating to implementing the conditions of this report.
Strategic planning
36 AQA operates a regular strategic planning process for its Access to HE activity, and an annual action plan is considered and approved by the AMG in November each year. The 2006-07 plan sets out specific actions and targets under five broad headings: strategic development; quality assurance, enhancement and development; financial; statistical data; and preparation for QAA review. The specific targets include actions relating to developing the necessary structures for, and implementation of, the Access to HE Diploma, as well as attracting new business during the period of introduction of the Diploma and establishing links with more HEIs. The plan also includes reducing expenditure and reviewing charges. Within the plan, responsibilities for undertaking actions and achieving targets are assigned and review dates for each are set. The plan does not identify resources for meeting the targets either in terms of finance or in staff time, and the relationship of this plan to budget setting within the organisation was not explored by the review team, although the Access to HE section has a separate budget, for which the Access to HE officer has monitoring responsibility. However, discussion of the plan at the AMG meeting which approved it did not appear to have addressed the question of resources. It was consequently not clear to the review team whether the 2006-07 targets were realistic and achievable, given the current staffing base and the plan to reduce expenditure.
37 Achievement of the plan is monitored by AMG and an assessment of the achievement of targets is included in the AVA's annual report. The plan is therefore considered by the members of AQA's Council through the CARG at its annual meeting to approve the annual report, and in this way Council is involved in the planning process. At the most recent meeting of CARG, it expressed concern about the level of resources, particularly staffing, necessary to meet the AVA's plans, although it did not amend the plan itself.
38 In parallel with this regular planning process, AQA Council has been conducting a review over the last three years of a number of its awards which were not central to its main business, including Access to HE, in particular to ensure that these areas did not make a loss. In December 2006, Council considered a report which assessed the financial position of Access to HE, the potential for growth and the anticipated financial and staffing implications, and the resource implications for the development of the Access to HE Diploma. The review team sought information on the outcome of this review. Members of Council accepted that Access to HE is a very small part of AQA's business, but confirmed their continued commitment to Access to HE.
39 The review team concluded that the AVA's strategic planning processes were generally appropriate for their purposes, involving AQA's Council and therefore the potential to ensure that resources can be matched to the planned actions. An example of this is the decision to invest in data resources to support the development of the Access to HE Diploma. However, there is a requirement for the AVA to generate sufficient income not only to pay for its use of AQA's general infrastructure, but also to make a financial surplus. In practice, therefore, there is not a sufficiently clear articulation between the AVA's plans and the resources to enable these plans to be implemented. The review team recommends that the AVA reviews its approach to strategic planning and makes explicit the articulation between plans and resources to support these plans.
Regional strategy
40 The 14 providers at the time of the review (see paragraph 13, above) were located in five English regions, with the largest single group in Yorkshire and the Humber. The 14 HEI members were located in five regions, in four of which there were also providers of Access to HE programmes validated by AQA. The AVA is seeking additional HEI members and two new members have recently joined.
41 Council members highlighted the differentiation that they were trying to create from other players in Access to HE and the importance of the AQA brand as an alliance between schools, FE and HE. As an AVA operating in a national context, AQA is open to any provider, irrespective of the region in which it is located. The AVA stresses its national scope and the view was expressed by officers and Council members that, since AQA is a national brand, its mission is to facilitate Access to HE wherever there is a demand. They stated that they had decided not to prioritise any particular regions, and that the AVA would respond to approaches from any provider in any region.
42 At the same time, the AVA accepts the need to have a regional focus and the plan for 2006-07 (see paragraph 36, above) includes the development of regional links, and commits the AVA to identifying and publicising relevant regional priorities and agendas to Access to HE providers. The AVA states that the 'regional skills agenda should be at the centre of strategic development', and it has committed itself to developing regional subgroups of the AVA, using a bulletin board to aid communication, to working with Aimhigher partnerships and Lifelong Learning Networks, and to exploring ways of using AQA's network of regional forums and regional officers to respond to this agenda. Plans in these areas are still at an early stage, however.
43 While AQA has stated its intention to develop regional networks in these ways, the AVA acknowledged in its Overview that, as a national organisation, it has problems in promoting a regional agenda and stated that although regional skills agendas are key elements in policy and planning, in practice, the labour market and progression to HE is more localised. Its main approach in responding to regional agendas is, therefore, through its approach to course development. This approach focuses on giving providers flexibility to develop Access to HE provision tailored to meet local priorities and local HE progression opportunities in the travel-to-work area.
44 The review team noted that the AVA was considering expansion to include a number of particular providers in a region in which the AVA was not currently active. The team therefore sought to clarify the AVA's position, including how it proposed to undertake the regional agenda it had set itself, which region or regions it proposed to operate, how it would collaborate with the regional organisations in these regions and, in particular, how in response to the licensing criteria it would analyse the needs of each region in which it proposed to operate, identify those groups in these regions which have most need of opportunities to progress to HE, and how it would communicate the results of this analysis to its providers to assist them in their planning, and what resources it planned to devote to these developments. AMG members indicated that strategies in each region would be developed from the bottom up, and would need to reflect the interests of providers in each region. They reported that a number of ideas had been discussed about how to develop the regional dimension, including greater use of the annual partnership meeting.
45 The review team came to the view that the AVA licensing criteria which relate to regional planning pose a particular challenge for AQA as a national organisation. Moreover, the AVA's approach to regional strategy, in particular its decision not to prioritise any particular region or regions, and to accept applications for membership from any provider without any regionally-based criteria, add to this challenge. Its plans for expansion are to attract providers, principally by relying on its reputation for flexibility and responsiveness, rather than to provide strategic leadership and direction in the planning, development and promotion of Access to HE at a regional level. It was not clear to the team that the AVA had committed sufficient resources to sustain this competitive approach, or to the necessary joint work with such agencies as Aimhigher and Lifelong Learning Networks, or to the work required to analyse the needs for Access to HE development in all the five regions in which it currently operates, and support the development of new provision. Indeed, should its plans for expansion be successful, the particular difficulties in engaging with, and responding to, regional strategies, and providing robust quality assurance of provision over a wider geographical area, are likely to be exacerbated if the new providers attracted are drawn from a still wider range of regions.
46 It is therefore a condition of licence that the AVA should review its approach to developing a regional strategy to ensure that it can meet its licence obligations in those regions in which it operates, and that it should report, in particular, on:
- the regions in which it intends to operate, including any phasing of expansion
- the means whereby it will engage and work collaboratively with other organisations and institutions working to promote widening participation in each region
- its approach to the analysis of regional priorities and needs for Access to HE development
- its approach to the promotion of Access to HE in each region
- the role of its regional officers, regional forums and bulletin boards in this strategy
- the resources necessary to support the implementation of this strategy.
Principle 3
The organisation is able to assure the quality and fitness for purpose of Access to HE programmes at the point at which they are granted formal approval, and to have effective means to develop, evaluate and review the Access to HE provision for which it has responsibility
Access to HE Handbook
47 The AVA publishes an Access to HE Handbook, which is updated on an annual basis and constitutes the main means through which it advises its members about its processes for assuring quality and standards. The latest version, at the time of the review, was dated September 2007. During the course of the review, the review team was also provided with a copy of the AVA's Annual Office Procedures which includes some of the procedural aspects of validation and approval of courses, but which had not, at that time, been published.
Fees and charges for validation
48 The Handbook describes the AVA's charging policy, stating that there are separate fees for initial validation and for subsequent amendments to validated courses. The review team was provided with a more detailed charging schedule. This showed that the validation fee comprises two elements - a basic validation fee and variable fees according to the number of units to be validated. It also showed a standard basic fee for validation at the provider's own premises and for an 'office-based' validation at AQA's offices if the validation used already validated units, in which case the fee would be reduced by just over 70 per cent.
Course development
49 The Handbook provides detailed guidance for providers on course development. This sets out the AVA's requirements and provides comprehensive guidance on the development work necessary prior to submission for validation, including preliminary research and identification of target groups, course structure and content, organisation and management, assessment, and consultation with representatives of HE. Access to HE coordinators whom the review team met expressed satisfaction with the level of support that they had received from AVA officers in the developmental phase of the courses for which they were responsible. The team noted that the advice on course structure had been updated to address the requirements for the Access to HE Diploma, including detailed guidance on rules of combination.
Validation and revalidation procedures
50 The Handbook also provides guidance on validation and revalidation procedures, and for course approval by the AVA. The Handbook provides a standard timetable starting with a visit by an AVA officer in October and culminating with a validation panel meeting in May/June. The procedures for revalidation are the same as for validation.
51 Before a provider can proceed with a submission, the centre first needs to be approved as an AQA Examination Centre, using the standard procedures that AQA uses for all its centres, and linked to AQA's General Regulations. The review team considered that the requirements for centre approval, whilst a valuable initial check on a centre's standing, are mainly concerned with matters relating to examination centres for AQA's main qualifications, rather than requirements for offering Access to HE courses, and that this also applied to a range of other standard publications. The general requirements for Access to HE providers are covered in the Memorandum of Association for the AVA partnership (see paragraph 24, above), which places certain obligations on providers in terms of resources, student support, reporting to the AVA, and payment of annual subscription fees. AVA officers, in responding to questions from the team about procedures for approval and withdrawal of approval for Access to HE courses, referred to the terms of the Memorandum of Association.
52 The review team noted that neither the requirement to be approved as an AQA Examination Centre, nor the requirement to adhere to the terms of the Memorandum of Association, are specifically referred to in the Handbook as necessary prerequisites for validation or for continuing approval as a centre. In view of the importance of some of the expectations articulated in these documents for providing the context for securing quality and standards, the team considered that the AVA should clarify these requirements (in the Handbook, or other formal document), and that they should be verified and confirmed as part of the validation process.
53 The AVA's procedures for validation provide for scrutiny of units by subject specialists and feedback to the centre, before a validation meeting. Members of the QAC stated that by undertaking this prevalidation scrutiny of units, it was unusual for detailed issues on the units to be raised during the validation event itself. However, the review team noted that there is no procedure for new or revised units to be formally approved prior to the meeting of the validation panel, that there appeared to be some overlap between the issues on which subject specialists were asked to comment and those that the validation panels were asked to address, and it saw at least one example of a validation report where detailed issues relating to the content of the units were addressed by the validation panel.
Membership of validation panels
54 The AVA's procedures provide for a validation panel to be appointed by QAC for each Access to HE course for which a provider is seeking validation, comprising three members: two members of the QAC and one from an HEI. Although the AVA's own requirements are for a validation panel for each course, the review team noted examples of validation events where the same panel had considered a number of courses and, in the most recent case, the same panel had considered seven courses at three different colleges. As noted in paragraph 58, the AVA was unable to provide written reports of these three validation events. Membership of these panels is also supposed to include a representative from an HEI outside the partnership, although finding such volunteers has recently proved difficult and members of QAC have suggested that this requirement might be abandoned in future.
55 The Handbook states that, in addition to the normal complement of panel members, one or more external subject specialists may also be co-opted. The panel concluded that the precise relationship between the role of subject specialists and that of the validation panel in relation to the subject content and approval of units was not clear.
56 Given that, under the new Access to HE Diploma, each programme might include a number of different pathways leading to different named awards, the review team sought clarification about whether the AVA would regard each pathway as a separate course, for which a separate validation panel would be appointed, or whether it would add additional members to a validation panel to take account of the range of pathways and awards under consideration. QAC members stated that the number of pathways under consideration would not necessarily have any bearing on the size of the validation panel. Since a range of subjects could be covered by different pathways at an individual provider, and the AVA's own procedures allow for additional subject specialists to be added to validation panels, the team considered that the AVA should review this practice, particularly if units continue to be formally validated by validation panels.
Validation meetings
57 The role of the validation panel is to make a recommendation via the QAC to the AMG about the suitability of the course as preparation for entry to HE. The Handbook provides guidance on the issues which validation panels should address in order to make that judgement. However, the review team noted that although there is guidance to providers on course development in relation to the requirements of the Access to HE Diploma, the guidance to validation panels does not appear to have been updated to ensure that the validation process confirms that a course meets the standard requirements for the Diploma, including the specifications for the common credit framework, the rules of combination, or the methods of recording individual student achievement. The team concluded that guidance to validation panels needs to be revised to ensure, at the point of validation, that all courses will meet the requirements of the Diploma.
58 The Handbook, in describing the validation procedures, states that a validation panel will visit the provider, or that there will be an 'office-based panel'. However, although the Handbook provides an outline agenda for a validation panel meeting on the provider's premises, none is provided for an office-based meeting, nor does the Handbook specify under what circumstances there would be an office-based panel. QAC members explained that this procedure would be adopted with an existing provider in circumstances where most of the units had already been formally approved by the AVA, and had been developed partly for the Access to HE Diploma, although not all revalidations would necessarily be office-based. The review team noted, however, that office-based validation had been used in summer 2006 for three centres which were new to the AVA. Members of the QAC described this procedure as 'experimental' and explained that normal procedures had not been followed in that instance. AMG members referred to them as 'atypical'. The AVA provided a paper setting out the rationale for these validations which referred to an intention to 'test run and streamline procedures' and to address 'excessive' 'costs to AQA and institutions'. Unfortunately, there is no formal written record of these three validation events, so the team was unable to confirm that the AVA's procedures had been followed fully. AVA officers stated that the decision on which type of validation would take place would be taken by an AVA officer, although it would probably also go to the Chair of QAC. The AVA's Overview referred to plans to move towards a risk-based approach to validation and revalidation and AMG members confirmed that this was planned, although no details were provided to the team. The team formed the view that, in the absence of clear published criteria for determining when a validation could be office-based, or what that procedure should involve, and a lack of involvement by the appropriate committee in decisions about what kind of validation procedure should be followed, there was a real possibility that an office-based validation which included insufficiently robust quality assurance procedures might be adopted.
59 After the validation, the AVA officer attending the event writes a report. The Handbook describes the possible recommendations of a validation panel, including approval with or without conditions or recommendations. An officer writes to the provider informing them of the outcome and of the deadlines which the panel has set by which conditions are required to be met. The Handbook does not describe this process but the Office Procedures document states that conditions have to be met by 20 July for courses starting in September. AVA officers confirmed that the decision on whether conditions are met is taken by the AVA officer. Once officers are satisfied that conditions have been met, the AVA officer writes to the provider to confirm this and takes the report through the AVA's committee structures for approval.
Course approval
60 The report of the validation meeting, including the panel's recommendations, is presented to the QAC and then, through receipt of the QAC's minutes, the AMG grants formal approval. The review team noted that the meetings of AMG to approve courses had routinely taken place around November each year, and that courses therefore had not been approved until after they had started. AMG members acknowledged that this was late. The Office Procedures document provides a revised timetable, with AMG meeting in June/July to consider validations. The Chair of AMG stated that, in order to ensure that all courses were approved prior to their commencement, Chair's action might need to be considered. However, this process would depend on a formal written record of the validation event being available.
61 The audit trails also raised concerns about the process for withdrawal of course approval (see paragraph 88, below). The AVA will wish to consider how the particular events came about, and the process by which the different decisions taken were communicated to the provider, and address any weaknesses in procedures to ensure that such events cannot recur.
62 The AVA is required, as a condition of licence, to reconsider its procedures for validation and revalidation and revise the AVA's formal documentation to clarify:
- the role of standard AQA centre approval and of the AVA's Memorandum of Association, and requirements relating to these
- the distinction between approval of a centre and the validation and approval of particular courses and pathways offered by a centre
- composition and formal approval of the validation panel membership, and the process for ensuring that the panel membership is appropriate for the range of courses and/or pathways which it is to consider
- the point at which units are formally approved including the distinctive roles of subject specialists prior to the validation event and the validation panel
- matters to be addressed by a validation panel, including requirements relating to the Access to HE Diploma specification
- expectations for the content and standard format of the panel report, and responsibility for its production
- process for the confirmation of course and pathway validation
- process and responsibility for monitoring conditions and recording that conditions have been met and formally approved
- processes and timing for formal approval, and for the formal withdrawal of approval, including notification to the provider
- the rationale for, and distinction between, different styles of validation event and the criteria and process used to decide on the style for a particular event
- how its proposed 'risk-based' approach to validation will be implemented.
Progress with plan for implementing the Access to HE Diploma
63 The AVA produced a plan for introducing the Access to HE Diploma and provided an update in its annual report for 2005-06. This included a schedule which provided for the revalidation of courses at nine providers during 2006-07; five between November 2006 and February 2007, and four between April and June 2007, with a further five during 2007-08. The review team sought clarification on progress with the plan. It was reported that none of the scheduled revalidations had taken place by the time of the review and the team was provided with a revised schedule, with five revalidation events to take place during February to June 2007, and nine during 2007-08. However, during the review, some members of QAC suggested that only two of the five validations scheduled for 2006-07 might take place before June 2007. The team concluded that there had been significant slippage from the original schedule, and there were indications of potential further slippage from the revised schedule. Some members of AMG considered that these revalidations would not be substantial events, as they would only involve a few units, but some providers indicated that the amount of work could be substantial. No analysis of the volume of work envisaged for each event was provided by the AVA. The possibility of new pathways had not been included in the schedule since it only listed providers. The team also noted that one of the AVA's strategic objectives in its 2006-07 Action Plan was to attract new business during the period of the introduction of the Diploma.
64 The review team noted that the AVA's standard method of planning validations and revalidations, and recording them in AVA committee minutes, in correspondence with providers, and on validation schedules, was to refer to a particular provider rather than the titles of the individual courses or pathways, although in the most recent AMG minutes full course titles had been recorded. The team noted that this was also true for the revalidation schedule for the introduction of the Access to HE Diploma (see paragraph 56, above). The team came to the view that the practice of recording only the name of the provider makes it difficult to track decisions about individual pathways, and to judge the appropriateness of membership of individual validation panels, and concluded that the AVA should review this practice, with particular reference to the requirements of the Access to HE Diploma.
65 The review team concluded that, in addition to the pressures it will place on AVA staff, the revised revalidation schedule will pose a significant logistical challenge to QAC, given that current validation procedures require two members of each validation panel to be members of QAC, from a total membership of nine at the time of the review. The team came to the view that the delays in revalidations, and the consequent 'bunching' of validation events, together with the possibility of new or substantially-revised pathways to be validated, and new providers, would add to this challenge, given the AVA's current practice of considering each provider separately.
66 The AVA is therefore required, as a condition of licence, to review and develop its plan for the introduction of the Access to HE Diploma, to ensure that all its providers are able to offer the Diploma from 2008 onwards, and to provide a revised schedule with confirmed dates for all stages of validation, with particular reference to:
- the revalidation schedule for existing providers and for pathways
- the incorporation of new providers into the schedule
- the style of validation (provider or office-based) in each case
- AVA officer support for providers
- scrutiny of new and revised units
- membership of, and officer support for, validation panels
- the AVA's database of units
- the training of moderators and subject specialists.
Annual review
67 Providers are required to submit an annual review report which is considered by AMG. Immediately before the review, AMG had reviewed the content of these reports and had agreed that more detailed guidance to providers should be developed, a view with which the review team concurred. The review team also noted that the process for the consideration of moderators' reports undertaken by QAC, was considerably more detailed than the process for the consideration of providers' annual reports undertaken by AMG. While the revised guidance to providers is an improvement, the team recommends that the AVA should review its processes to ensure that that it is explicit about how matters of concern are addressed, how data on Access to HE courses and students are formally considered, the level of scrutiny by AMG, and how the AVA disseminates good practice identified though annual review. It further recommends that these procedures are published to providers in the Handbook or in some other appropriate way.
Access to HE promotion and development
68 The AVA provides opportunities to share good practice through its moderator and coordinator training events, development days, and its annual conference. These cover a range of matters and the most recent have addressed, in particular, matters relating to the introduction of the Access to HE Diploma. The events appear to be generally well-attended, and Access to HE coordinators reported that they welcomed the support provided by the AVA and its responsiveness to queries.
69 The review team identified a tension between the AVA's desire to be flexible and responsive to the needs of its providers on the one hand (see paragraph 43, above), and the need to provide leadership and direction in the promotion of Access to HE at regional level on the other. This tension is heightened by the fact that the AVA operates in a number of government regions, but has not developed distinctive promotion and development policies for each (see also paragraphs 44 and 45, above).
Principle 4
The organisation is able to secure the standards of achievement of students awarded the Access to HE qualification
70 At the time of the last review in 2001, the AVA had a system of both moderators and external examiners which led to some uncertainty about roles. Subsequent to the 2001 review, the AVA modified its approach to external programme monitoring and assessment by phasing out the role of external examiner and by redefining the role of moderator. Under these new arrangements, moderators fulfil the roles which were previously undertaken by external examiners and, where several moderators are appointed to one provider, they operate as a team coordinating their visits to the provider and their reporting to the AVA. A group of moderators whom the review team met indicated that they found these new team-based arrangements a positive development. A report presented to the AVA with the collected views of two or three moderators tended to carry more weight, while the team-based approach was also helpful in mentoring new moderators. The team also met with a group of coordinators who were similarly positive about the changes in moderation. They considered them a significant improvement on the previous system, being rigorous yet supportive and generating constructively critical comments on the provision.
71 The role and function of moderators are detailed in the AVA's Handbook. These include ensuring conformity of the course as delivered, with the course as approved by the AVA; oversight of the conduct of assessment including approval of assessment materials, moderation of assessment outcomes and attendance at examination boards; and reporting twice yearly to the AVA through QAC. The review team was of the view that the AVA's Handbook provided moderators with a clear specification of their duties in relation to the Access to HE provider. In particular, there is an explicit requirement that moderators must visit colleges for three full days each year and that they must attend all examination boards. The Handbook provides moderators with a detailed checklist of activities which must be undertaken on each of their three visits, geared to the annual academic cycle, and there are templates for reporting to the AVA twice yearly. There is also clear guidance on the moderation of assessment outcomes, including sampling of student work. Although they operate within a team, there is no hierarchy of moderators, and each moderator takes responsibility for particular subject areas, with all the moderators contributing to the report to the AVA.
Moderator appointment and training
72 Moderators are appointed by the AVA and are independent of the provider institution. The review team examined the profile of current moderators, and it was able to confirm this independence. However, the team did note an instance of a 'back-to-back' moderation arrangement, whereby the moderator of a discipline area in College A came from College B, and the moderator in the same discipline area in College B came from College A. Asked whether there was an explicit policy relating to such arrangements, the AVA confirmed there was not. The team was of the view that the AVA should avoid situations of 'back-to-back' moderation arising, and recommends that it should develop an explicit policy statement to this effect.
73 While there is a general statement of the qualities which the AVA seeks when appointing moderators, there is no explicit statement of the criteria against which the suitability of a potential moderator might be evaluated. QAC has identified the need for transparent criteria for the appointment of moderators, and has indicated it would undertake a review and approve new criteria and procedures. However, the review team saw no evidence that this work had so far been undertaken. The QAC is charged with reviewing the role and practice of moderators but it is not part of its terms of reference to make appointments of moderators. The team was told that appointment of moderators is the responsibility of the Access to HE Officer. While there was no suggestion that current practices were leading to the appointment of inappropriate moderators, the team was of the view that this process placed too great a reliance on a single officer.
74 The review team was told that, through information provided on application and through taking up references, the AVA could consider a combination of factors including academic qualifications and experience, and to seek to match appointments to the courses and subject areas in the provider for which a new moderator was required. The range of subject areas covered by individual moderators is, in some cases, quite wide ranging. The team asked the group of moderators with which it met for their views on the breadth of subject coverage within their areas of responsibility. Generally, the moderators felt comfortable that they had the necessary expertise to cover the range of subjects which had been allocated to them. However, one moderator indicated that he felt one subject for which he had responsibility was outside his subject competence, and another moderator indicated she had refused a request from the AVA to cover a subject area for the same reason. To ensure that provision is moderated by appropriately experienced moderators, the AVA is required, as a condition of licence, to develop clear criteria, including curriculum expertise, and process for the selection of moderators. including recommendation to the QAC, or other appropriate committee for appointment.
75 Moderators are appointed initially for one year. Their appointments are, thereafter, renewed annually until they have completed a maximum period of four years. It is possible for the AVA not to renew a contract if it has concerns about the performance of a moderator. Moderators receive a letter of appointment and schedule of required duties, but there is not a formal contract document. The team recommends that this letter of appointment should incorporate a prohibition on subcontracting of duties by the moderator.
76 The AVA is able to make judgements about the performance of its moderators through the reports which they make during the year to QAC. In the past, QAC has expressed concerns about variability in the quality of moderator reports, but introduction of standard reporting templates was said to have improved the quality and consistency of reports. It was also possible for college coordinators to raise concerns about moderation in their annual reports to the AMG.
77 The AVA provides an annual Moderator Training Programme and recent events have covered updating, good practice and procedural issues. Moderators with whom the team met confirmed they found the annual training events helpful and that they provided a sufficient level of support. While the level of attendance by moderators was reasonable, the AVA has recognised that it is an area of weakness that attendance is not compulsory. The team was told that the AVA intends to introduce compulsory attendance at the annual training programme for all moderators from 2007-08 and, recognising that unforeseen circumstances or late appointment may prevent some moderators from attending, it also intends to introduce online training materials. The AVA is required, as a condition of licence, to implement its plans to make moderator training mandatory.
Standardisation
78 The review team was of the view that, through the move to working on a team-based approach, moderators within one college would be able to compare standards and judgements in the different subject areas which made up the Access to HE provision in the college. The team members sought to understand how moderators would be able to make comparisons of standards and judgements between different colleges. They were advised that moderator training had included standardisation exercises using actual student assessments, and this would be carried over into the mandatory training programme scheduled to commence in 2007-08. However, the AVA confirmed that there was no individual or body within the AVA explicitly charged with oversight of comparability of standards between different college providers, and the team recommends that the AVA reviews the terms of reference of the QAC (or other appropriate body), in order to ensure that responsibility for comparability of standards between providers is explicitly located within the AVA.
Moderator reports
79 Moderators are required to complete two reports each year. The first focuses on the college's response to the previous year's moderator reports, while the main, summer report, made after the examination board has taken place, comments on assessment performance during the year, and makes recommendations to the college for future action. The reports are received by the QAC for consideration. The AMG is made aware of the outcomes of moderation through receiving the minutes of QAC which reports to it.
80 The summer moderator reports are analysed by members of the QAC working in pairs in order to produce summaries of the key points. A draft copy of the report is sent to the college coordinator for checking for factual accuracy before the reports and summaries of key points are presented to the September QAC meeting for consideration. The summaries and comments of QAC are then sent to the principal of the relevant college, and they form the basis for the first moderation visit of the following academic year. The review team heard from moderators that this system of reporting had proved very effective, and they were able to cite examples of instances to demonstrate that the quality of provision had been enhanced as a result. Responses to recommendations for action are reflected in the coordinator's annual report. The moderator also monitors and reports back on the response of the college to the recommendations made in the previous moderator report. Clearly, it could take up to another year before it became clear that a college was not responding appropriately. In this case, the Access to HE Officer would raise the issues directly with the college and, should a college not be receptive, the AVA has the ultimate sanction of withdrawing approval, although the team was told this had not been necessary to date.
81 The AVA's Handbook provides clear guidance on the membership of college examination boards. There is also guidance on the agenda for the examination board. The AVA requires colleges to have their own internal verification procedures to ensure internal moderation and standardisation of assessments prior to their external moderation. Access to HE awards are only confirmed after the exam board has been concluded. The chair of the exam board and the moderator are required to sign the schedule of results to signify agreement to the awards, and this schedule is sent to the AVA in order to trigger the issue of certificates.
Certification
82 The review team was able to inspect copies of the certification issued by the AVA to students and to confirm that these were in the standard format and used the wording specified by QAA for Access to HE certificates. The team asked officers about the office procedures for the issue of certificates to students. Certification is prepared against the schedule of results sent by the colleges, after the results have been checked for accuracy. It was clear that, while the approach used to manage the issuance of certificates was satisfactory, there were no written procedures to which reference might be made, and the procedure relied on the tacit knowledge of the member of staff responsible for this activity. The team recommends that the process be codified in the form of a set of written procedures.
83 The team was told that blank certificates were issued to the responsible officer in batches from AQA's secure central store, for preparation in the Access to HE section and onward transmission to the colleges who issued them to the individual students. The certificates carry a unique serial number, and the numbers of each batch are recorded. The team asked whether there was a system whereby details were recorded of to which student each certificate was issued. The answer was that there was no such system, and the AVA had no record of to whom each certificate was issued. The team considered that this was a serious omission. There was no system whereby the issue of blank certificates from the central store and the issue of completed certificates to students was audited or reconciled, and the AVA had no records of the serial numbers of the certificates which it issued to individual students. The team was of the view that this weakness in the AVA's administrative systems was sufficiently serious as to require the AVA to introduce a robust and effective system to manage and record the issue of its Access to HE certificates, as a condition for renewal of the licence.
Audit trails
84 In the course of the visit, the review team conducted audit trails on eight of the AVA's provider colleges. The purpose of these trails was to enable the team to consider the consistency and effectiveness of the AVA's processes at centre level.
85 The providers selected were seven public sector colleges, and one private sector college. Three of the public sector colleges had an extensive portfolio of Access to HE programmes, covering a diverse range of discipline areas. The private sector provider had one Access to HE course in business. Geographically, while four of the providers including the private sector college were located in Yorkshire, the other four providers were in the Midlands, the South and the South-West of England.
86 The review team was presented with audit trail files which included, where available, moderation, monitoring and evaluation documentation for the years 2004-05 and 2005-06. The files included validation and revalidation reports as appropriate, moderators' and coordinators' reports, letters to college principals regarding the outcomes of moderation, and course handbooks.
87 In the examples of the validation and revalidation processes presented, the review team saw evidence of variability in practice. There were examples of rigorous and effective processes, with scrutiny of modules by subject specialists, and course approval by panels which included external academic expertise. However, the team also saw examples of validations where the AVA did not follow its own procedures, for example, in relation to panel membership and the production of written reports. It was also the case that, from the communications the team saw in some of the audit trail files, there was an absence of standardised letters which would provide clarity of communication, for example, formal letters of confirmation of approval following a course validation. It was not always possible for the team to confirm, from the documentation presented, that validation conditions had been followed through. Also, there was minimal communication from the partner colleges presented in the audit trail documentation, suggesting an incompleteness of record-keeping.
88 The review team saw one example within the audit trails which gave rise to significant concerns about the robustness of the AVA's procedures and the consistency with which it followed these. The validation panel had not been constituted in accordance with the AVA's own procedures as laid down in its Access to HE Handbook with, in particular, no external representation on the panel. The team saw documentation confirming approval of the provision in the AMG's minutes but, following the examination board for the first student cohort in September 2005, the moderator made recommendations that amendments should be made to the programme. The moderator's report was sent to the principal of the college with a covering letter in November 2005. The team noted that, while the moderator report pro forma referred to 'future action required by the college', the covering letter stated only that QAC 'offered the following recommendations' while hoping that student numbers would be sufficient to run the course again in the new academic year. Subsequent discussion in the AVA's QAC evidenced concern that the college had recruited a further cohort of students in January 2006 'without the course being approved', approval being related not to the initial validation but rather the requirements conveyed to the college in the moderator's report. The team saw several subsequent communications sent to the college seeking clarification of whether it would be submitting revised documentation to meet the AVA's requirements, although no further cohort of students had, in fact, been recruited. As late as June 2006, the minutes of the AMG stated that 'students had been enrolled on the un-validated (sic) course. The college had been written to more than once but no reply had been received'. The team subsequently sought clarification from the AVA as to the current status of the provision. The team was told that the college was no longer a part of the AVA's approved provision, and that this had come about because the college had not undertaken the modifications to the programme that had been required following the moderator's visit, but also because the college had been unable to recruit a further cohort of students. It was confirmed that the AVA did not have a formal procedure, as such, for the withdrawal of approval. The team concluded that the AVA had not made a distinction which was sufficiently clear to all parties, either in these particular circumstances or in its own formal procedures, between: an approved course with conditions to be met; an approved course, but not running; and a course where approval had been formally withdrawn.
89 The AVA requires college providers to present an annual report, and this is the responsibility of the college coordinator. The AVA does not currently specify a format for the production of coordinator reports and, while they generally covered key issues such as action plans responding to moderators' reports, self-assessment of the provision and student feedback, there was some unevenness in the quality of reports presented in the audit trails. The AVA has recognised this as a weakness, and is proposing to introduce a new template for coordinator reports (see paragraph 67).
90 Through the audit trails, it was possible to track the transition from the original system of external examiners and moderators to the current system of team-based moderation. The audit trails provided confirmation that moderators fulfil their obligations, in some cases visiting on more than the required number of times. The AVA has recently introduced new templates to support the twice-yearly reports which moderators are required to write. From the examples the review team saw in the audit trails, this system appeared to be working well, and teams of moderators working in a specific college are taking the opportunity to produce joint reports on the college provision.
91 Overall, from the information provided by the AVA in the audit trails, while the review team saw examples of good practice in course validation and revalidation and in moderation, it was not able to form a firm judgement about the effectiveness of the quality assurance framework and its application.
Conclusions
92 AQA's position as a large national awarding body provides it with obvious potential advantages for its role as an AVA. Not least, it has very substantial experience in managing the award of qualifications, and monitoring and maintaining the academic standards of awards. It also has the structures and resources typical of such an organisation, with extensive, specialist support services in a range of areas. Activity relating to Access to HE benefits from some of these services, for example, marketing and website development, financial management, the development of information management systems to support the implementation of the Access to HE Diploma, and the expertise available within the organisation to provide sound analysis of the AVA's data in its annual report to QAA.
93 In some respects, an AVA may also be disadvantaged by a situation in which Access to HE is a very small part of the organisation's total endeavour, and where the associated activity depends on a very small number of individuals, especially if that activity is outside the main stream of the organisation's other activities. Clearly, there are dedicated, committed and able staff working in AQA's Access to HE section and in its partner colleges. The committees and moderators overseeing the activity are also evidently committed to providing a service for the benefit of Access to HE students. But AQA faces challenges, both structural and procedural, in ensuring that mechanisms are in place to monitor and develop Access to HE activity and ensure that equally rigorous procedures are followed to equally high standards for this, as for any other, part of its activity. It is a matter of some surprise, and of particular concern, that the benefits of AQA's awarding body experience have not always been evident in the quality assurance of Access to HE, particularly in relation to the key processes of course validation and certification.
94 The last review report expressed concerns about staffing levels and, in particular, about leadership and management for Access to HE. While the overall staffing complement has increased slightly, there have been recurrent changes in staffing and management for Access to HE in the period since the last review, with the inevitable disruption and discontinuity to operations that follow such change within a small team. These changes have occurred at a time when major changes to the Access to HE qualification itself have occurred, as well as changes in the AVA licensing requirements. In these circumstances, it is not surprising that there has been some slippage in the AVA's timetable for the implementation of the Access to HE Diploma. It is essential that this timetable is now reviewed and all necessary action taken to ensure that all courses are consistent with the specifications of the Access to HE Diploma by 2008-09, if AQA is to be able to maintain its service to providers and students as a licensed AVA.
95 Although engagement in development activity is now a regulatory requirement, it is not foregrounded in AQA's work and little notable progress has been made since the revised licensing criteria were introduced. There are particular difficulties for AQA as it attempts to find an appropriate way of meeting those AVA licensing criteria which require it to be responsive to the development of regional strategies for widening participation in HE. However, while AQA describes itself as adopting a 'flexible' approach, to provide for different providers' needs and demands, and respond to different opportunities at the local level, there is little evidence to date that this has contributed to substantial or innovative development, regional or otherwise.
96 The last review report expressed concerns about the AVA's governance arrangements. These have been addressed through the creation of the Council Access Review Group and a more direct and active link between the Council and the Access Management Group. These structures have clarified and confirmed the Council as the appropriate locus of responsibility for the AVA licence, and improved the capacity of the Council to monitor Access to HE activity. This has enabled Council members to raise concerns, if they feel the need to do so, and also to ensure that the Council is more fully informed about the AVA's work and its responsibilities.
97 The continuing financial viability of Access to HE as an area of work for AQA has been a natural and proper concern of the AQA Council. In its deliberations about the AVA's future, AQA will, no doubt, consider the effectiveness of measures taken to date, including any impact on the security of quality and standards, as well as on the financial return from Access to HE. In assessing any further potential for growth, the AVA may recognise three possibilities: attracting pre-existing provision from other AVAs; generating new provision in the areas in which it is already active; or supporting development of provision outside its current area of activity. All three options are likely to require some investment of resources, for development and validation, if AQA is to be able to maintain the quality of provision and standards of the awards it makes as an AVA.
The AVA licence
Review outcome
98 The review team recommends that AQA be granted a provisional renewal of its AVA licence, with conditions to be met by the dates specified below and revisit in spring 2008.
99 The AVA's licence is restricted in relation to the award of the Access to HE qualification, until Conditions iii and iv have been met to QAA's satisfaction. While the AVA may continue to award Access to HE certificates, it may not award the Access to HE Diploma or indicate on its publicity that Access to HE Diplomas will be awarded until these conditions have been met.
The award of the Access to HE Diploma
Conditions
100 The AQA AVA licence is renewed on condition that it:
i carries out a review and analysis of its staffing arrangements for Access to HE, and reports on the outcomes of the review to demonstrate how it will ensure that its responsibilities as an AVA licence holder will be met, including responsibilities for quality assurance and development, actions identified in its action plan, actions relating to the implementation of the Access to HE Diploma, and actions relating to implementing the conditions of this report (paragraph 35)
ii reviews its approach to developing a regional strategy to ensure that it can meet its licence obligations in those regions in which it operates, and reports, in particular, on:
- the regions in which it intends to operate, including any phasing of expansion
- the means whereby it will engage and work collaboratively with other organisations and institutions working to promote widening participation in each region
- its approach to the analysis of regional priorities and needs for Access to HE development
- its approach to the promotion of Access to HE in each region
- the role of its regional officers, regional forums and bulletin boards in this strategy
- the resources necessary to support the implementation of this strategy (paragraph 46)
iii reconsiders and revises its procedures for validation and revalidation and revise the AVA's formal documentation to clarify:
- the role of standard AQA centre approval and of the AVA's Memorandum of Association, and requirements relating to these
- the distinction between approval of a centre and the validation and approval of particular courses and pathways offered by a centre
- composition and formal approval of the validation panel membership, and the process for ensuring that the panel membership is appropriate for the range of courses and/or pathways which it is to consider
- the point at which units are formally approved including the distinctive roles of subject specialists prior to the validation event and the validation panel
- matters to be addressed by a validation panel, including requirements relating to the Access to HE Diploma specification
- expectations for the content and standard format of the panel report, and responsibility for its production
- process for the confirmation of course and pathway validation
- process and responsibility for monitoring conditions and recording that conditions have been met and formally approved
- processes for the formal withdrawal of approval, including notification to the provider
- the rationale for, and distinction between, different styles of validation event and the criteria and process used to decide on the style for a particular event
- how its proposed 'risk-based' approach to validation will be implemented (paragraph 62)
iv reviews and develops its plan for the introduction of the Access to HE Diploma, to ensure that all its providers are able to offer the Diploma from 2008 onwards, and to provide a revised schedule with confirmed dates for all stages of validation, with particular reference to:
- the revalidation schedule for existing providers and for pathways
- the incorporation of new providers into the schedule
- the style of validation (provider or office-based) in each case
- the resources necessary to implement this plan, in particular
- AVA officer support for providers
- scrutiny of new and revised units
- membership of, and officer support for, validation panels
- the AVA's database of units
- the training of moderators and subject specialists (paragraph 66)
v develops clear criteria, including curriculum expertise, and process for the selection of moderators, including recommendation to the QAC, or other appropriate committee for appointment (paragraph 74)
vi implements the intention to make moderator training mandatory (paragraph 77)
vii introduces a robust and effective system to manage and record the issue of its Access to HE certificates (paragraph 83).
Conditions iii and iv to be met by 1 October 2007.
Conditions i, ii and v-vii to be met by 1 December 2007.
Recommendations to the AVA
101 The review team recommends that the AVA:
i specifies rules for the reappointment of members to the QAC when the committee's terms of reference are next reviewed (paragraph 21)
ii reviews and updates job descriptions for staff in the Access to HE section to ensure that they reflect reality and, in particular, that reporting and line management responsibilities are clear (paragraph 30)
iii reviews its approach to strategic planning and makes explicit the articulation between plans and resources to support these plans (paragraph 39)
iv reviews its procedures for annual review reports from providers to ensure that that it makes explicit how matters of concern are addressed, how data on Access to HE courses and students are formally considered, the level of scrutiny by AMG, and how the AVA disseminates good practice identified as through annual review, and to ensure that these procedures are published to providers in the Handbook or in some other appropriate way (paragraph 67)
v develops an explicit policy statement to avoid situations of 'back-to-back' moderation arising (paragraph 72)
vi keeps under careful review the span of subject areas it expects moderators to cover, in order to ensure moderation of student achievement is sufficiently rigorous and fair (paragraph 74)
vii incorporates a prohibition on subcontracting of duties in the letter of appointment for moderators (paragraph 75)
viii reviews the terms of reference of the QAC (or other appropriate body), in order to ensure that responsibility for comparability of standards between providers is explicitly located within the AVA (paragraph 78)
ix codifies the process for the issue of certificates to students in the form of a set of written office procedures (paragraph 82).
Appendix
Aims and objectives of AVA review
The aims of the system of AVA review are:
i to provide the basis for an informed judgement by the ARLC about the fitness of the AVA to continue as a licensed agency
ii to promote public confidence in Access to HE as a properly regulated and respected route into HE by assuring
- the quality and adequacy of AVAs' systems and procedures
- the quality, comparability and range of AVAs' operations
- the adequacy and comparability of AVAs' standards for approval, moderation and monitoring of programmes
- consistency across AVAs in the operation of criteria for the granting of the Access to HE award
iii to stimulate reflective and self-critical perspectives within AVAs as an instrument to promote quality enhancement
iv to provide an opportunity to identify and disseminate good practice of AVA operations
v to provide a mechanism for ensuring necessary, and encouraging desirable, improvements and developments in AVAs.
The objectives of each AVA review are:
i to examine, assess and report on:
- the development of, and changes in, the AVA since its last review or initial licence, and its plans and targets for the future
- the organisation's continuing viability and robustness and the ways in which the AVA demonstrates sound governance
- the efficiency and effectiveness of the AVA's operational and quality assurance systems
- the range and scope of the AVA's activities, and the appropriateness and value of these activities
- the ways in which the AVA approves and monitors programmes and the ways in which these processes take account of the need for consistency and comparability
- the ways in which the AVA satisfies itself of the adequacy and comparability of standards achieved by students gaining the Access to HE certificate
- the evidence available to indicate the AVA's success in achieving its aims and targets
ii to identify and report on:
- strengths and good practice in procedures and operations
- areas which would benefit from further development
- areas requiring attention.
1 As provided by the AVA in its annual report to QAA in December 2006
ISBN 978 1 84482 733 6
Top