Preface
Quality assurance of overseas collaborative provision
The Quality Assurance Agency for Higher Education (QAA) was established in 1997 to contribute to the maintenance and improvement of the quality and standards of all UK higher educational provision, wherever and however this is offered to students. To this end, QAA undertakes regular academic quality audits of individual institutions to review the operation and effectiveness of arrangements for assuring quality and standards.
Quality audits also cover the arrangements which institutions use to assure the quality and standards of their awards and programmes offered in collaboration with other partners, both within and outside the UK. As part of this process, QAA has extended its audit procedures enabling audit teams to visit overseas partners of UK institutions so that the same enquiries can be made of arrangements for safeguarding UK awards and programmes offered to students outside the UK as are made of UK-based provision. This initiative has been designed to help provide enhanced confidence in the work of UK universities and colleges operating overseas.
QAA's enquiries have been assisted by the publication in December 1996 of the former Higher Education Quality Council's revised Code of Practice for Overseas Collaborative Provision in Higher Education. This offers guidance on good practice and a framework within which institutions can review and consider their current and future activities. The Code of Practice has been widely welcomed and has been used as a common point of reference for the programme of overseas visits. Shortly after this overseas audit visit, QAA published a Code of practice for the assurance of academic quality and standards in higher education: Collaborative provision. This document will form the basis of overseas auditing activities undertaken with effect from Summer 2000.
The UK universities and colleges, with the agreement of their overseas partners, were voluntary participants in the programme of overseas visits. Their collaborative links cover between them a range of programmes and subjects, levels of award and different forms of institutional partnership.
This report is one of a number of reports published from the summer 1999 overseas audit programme to Ireland.
Foreword
1 This is a report of an academic audit carried out by the Quality Assurance
Agency for Higher Education (QAA) of the quality assurance arrangements
for a collaborative partnership between Staffordshire University (the
University) and Dublin Institute of Technology (the Institute) to deliver
two Specialist Routes of a taught postgraduate programme of studies, through
a franchise arrangement, leading to an MSc in Computing Science award
of the University.
2 The University was asked to provide a Commentary on its arrangements which would describe the collaborative link, making reference to the extent to which it was considered to be representative of its procedures and practice in all its overseas collaborative activity, or was specific to its collaborative activities in Ireland. The University was also asked to provide a view of the effectiveness of the means by which it assured itself of the quality of the learning opportunities and student support offered through this overseas link and to offer its view of the effectiveness of the means by which it assures itself of the standards of awards gained through this overseas link.
3 At the end of its Commentary, the University was asked to list the sources and nature of the evidence on which it relied to assure itself that the quality of the collaborative educational provision leading to its awards (or awards for which it had responsibilities) was meeting its requirements and expectations, and that the standard of the relevant awards were being safeguarded. The University was asked to make this evidence available to the QAA audit team on the occasion of its visit.
4 The full membership of the Ireland audit team was: Dr F R Burnet, Professor R A Pearce, Mr J C P Raban and Dr D Timms, auditors; and Mr G Clark, audit secretary. Additional audit secretary support was provided by Ms T Hazard, QAA. The audit was co-ordinated for QAA by Dr D W Cairns. The members of the team who visited the University on 22-23 April 1999 were Professor R A Pearce and Dr D Timms, auditors; and Mr G Clark, audit secretary. Dr D W Cairns accompanied the team for QAA. In the course of the visit members of the team held discussions with more than 30 members of the University, some on more than one occasion; they included: the Deputy Vice-Chancellor with responsibility for academic quality and international strategy; senior members of staff with University-wide responsibilities for assuring the quality of the University's collaborative provision in the UK and elsewhere; and members of the teaching and support staff of the School of Computing. The University also arranged a discussion between members of the team and the external examiners for the programmes leading to the MSc Computing Science (Specialist Routes), including the franchised provision, via a video conference.
5 Members of the audit team visited the University's partner, Dublin Institute of Technology, on 18 May. On this occasion, the members of the audit team were Professor R A Pearce and Mr J C P Raban, auditors; and Mr G Clark, audit secretary; Dr D W Cairns accompanied the team for QAA. In the course of its visit the team met: the President of the Institute; the Dean of the Faculty of Science; the Head of the School of Mathematics, Statistics and Computer Science; members of the Institute's teaching and support staff associated with the franchised MSc Computing Science Routes; and a group of students. In all, the team met some 30 members of the Institute, some on more than one occasion.
6 QAA is grateful to the staff of the University and to staff and students at the Institute for their assistance and co-operation with the audit team.
The University's Commentary
7 The University's Commentary was produced in two sections. The
Chair of the Academic Collaboration Committee (ACC), with assistance from
the Quality Support Unit, produced a section dealing with the University's
general arrangements for the assurance of quality and standards in relation
to overseas links, while the section dealing with the University's partnership
with the Institute was produced by the School of Computing. The version
of the Commentary sent to QAA was produced following discussions
within the School of Computing and a meeting between the Quality Support
Unit and others involved in its production. Its contents had also been
seen and agreed by the Deputy Vice-Chancellor as Chair of the Quality
Development Committee.
8 The Commentary described the development and operation of the University's link with the Institute and served as a useful starting point for the audit. The Commentary did not, however, provide a critical evaluation of the means by which the University assured the quality of the provision and the academic standards of its awards in relation to the overseas link, nor did it indicate the degree to which the arrangements to safeguard academic standards and the quality of the students' learning opportunities were typical of those operated in respect of other University franchise arrangements, in all respects other than the appointment of a Monitor (see below, paragraph 32). In the course of the visit, senior members of the University were invited by members of the audit team to assess the degree of confidence they would place in the University's quality assurance arrangements for its collaborative provision generally, and in this particular link. The team was initially told that the University had full confidence in all aspects of its quality assurance arrangements, but it was later told that the University had identified a number of matters to which it was giving further attention (see below, paragraphs 13 and 43).
The University context for collaborative provision
9 The University's academic quality assurance arrangements were the subject
of an HEQC academic audit report in 1995. Its quality assurance arrangements
for its collaborative provision were the subject of a further HEQC report,
published in 1997. This later report noted, within the context of the
University's development of its collaborative provision, that its 'overseas
partnerships were a more recent development for the University than UK
regional ones and that they had grown out of academic links between individual
staff and groups of staff with colleagues in potential partner institutions'.
The report also noted that while the link between the University's regional
and UK partnerships and its Strategic Plan was clearly demonstrable,
'a University strategy for overseas provision in general' was difficult
to perceive. From the instances of overseas collaborative provision that
had been scrutinised, however, the 1997 report was able to conclude of
the University's overseas collaborative provision that, with 'few exceptions
[its] informal and formal procedures for quality assurance were well structured,
carefully implemented and well understood by the University staff involved'.
10 Since 1996-97, the University has made a number of changes to its academic management, among which has been the assumption of responsibility for managing the quality assurance of the University's extensive portfolio of international collaborative arrangements by its recently-appointed Deputy Vice-Chancellor. The University established an International Strategy Task Group in 1998 to review all aspects of its overseas recruitment and collaborative activities. At the time of the present audit the University was completing its new Strategic Plan and international strategy. Among the changes since the 1997 report, the Commentary noted the renaming of the Accreditation Sub-Committee of the University's Quality Development Committee (QDC) as the ACC, and the revision of its terms of reference to include overseeing annual monitoring. This change was stated to have been made in response to recommendations contained in the 1997 report.
11 The University's Quality Assurance Handbook for Award Bearing Awards (the Handbook) was first issued in September 1994. Subsequent changes to procedures and policies have been communicated within the University by means of guidance notes. The Handbook was familiar to academic managers at the University and the Institute but did not appear to be used by members of the teaching staff at the latter. In February 1998, a summary of the main changes made to the Handbook since 1994 had been produced, although not all members of staff at the University and at the Institute appeared to have received this document. The University was reviewing the Handbook and a new edition was due to be published in Autumn 1999.
The University's approach to collaborative provision
12 The University recognises four general categories of collaborative provision: franchises or licences; dual awards; joint awards; and external validation. They are more fully described in the Handbook and in paragraph 12 of the 1997 report. The arrangement with the Institute is currently considered by the University to be a franchise: an arrangement whereby an award or part of an award of the University is offered by a partner institution on the latter's campus. The Handbook makes it clear that the curriculum of a programme or courses to be franchised is not required to be identical to that offered within the University.
13 The University's newly-formulated strategy for overseas collaboration, including collaborative provision, was about to be promulgated at the time of the audit and the University provided the audit team with a draft copy of its most recent revision. This stated that the University's strategy was guided by a number of principles, including a determination to 'ensure that the quality of provision for international students, wherever located and however provided, will be measured against the same standards as all the University's provision'. The draft strategy sets out the University's intention to 'maintain [its] position as a major provider of quality UK higher education overseas'; and 'to enhance the University's reputation through the quality of its international activities, provision and partnerships'. One of the University's objectives for its next planning period (2000-2004) includes the development of 'a method for review and evaluation of overseas provision and links'.
The partnership between the University
and Dublin Institute of Technology
Dublin Institute of Technology
14 The Institute has been offering vocationally-relevant awards since the 1950s and has been awarding degrees under authority from the University of Dublin (Trinity College) since 1975. The Institute was designated as a third level institution in 1992 and, in 1997, it was granted degree awarding powers. In 1998-99, approximately 22,000 students were registered for study with the Institute, of whom some 10,000 were registered for full-time higher education programmes and courses.
15 According to the Commentary, formal links between the University and the Institute have been 'based around the external validation of postgraduate awards in computing science' a view confirmed by members of the University who stated that there were no other formal links with the Institute. In 1992, however, the University and the Institute had concluded a Memorandum of Co-operation (Memorandum) under which students who had completed a higher diploma at the Institute could be admitted directly to the final year of degree studies at the University. Members of the Institute confirmed that this articulation arrangement remained in force, even though no students had transferred to the University in the three years immediately preceding the audit. The present audit focused on two Routes to a specialist MSc in Computing Science, which had been franchised by the University to the Institute: the MSc Computing Science (IT for Strategic Management) franchised in 1995, and the MSc Computing Science (Distributed Systems) franchised in 1997. These two Routes, together with a further six Routes, available only at the University's Stafford campus, are generically described by the University as 'the MSc (Specialist Routes)'.
16 Informal relations between the two institutions date from 1986, when links had been established by the University (Staffordshire Polytechnic, as it then was) to support courses in the Institute leading to examinations of the British Computer Society. Throughout their collaboration, links between the University and the Institute have been centred on the University's School of Computing and the Department of Mathematics, Statistics and Computer Science at the Institute. Formal collaboration between the University and the Institute dates from 1991, with a curriculum development and a joint research project later extended to include staff and student exchange arrangements under the EU Erasmus and Tempus schemes.
17 At the time of the present audit, the Institute had recently been granted powers to offer taught postgraduate awards. The University and the Institute intend to continue their collaboration, however, and were planning to develop the present franchise as a joint award at the end of the 1998-99 session. Staff at the Institute believed that because of the high level of autonomy which they enjoyed under current arrangements, the significance of a change to a joint award would be more symbolic than real.
18 Both institutions continue to see clear academic benefits in their partnership and merit in continuing with the association through a joint award. In the University's view, its franchise with the Institute, like its other franchises, has been 'forged in the context of a partnership recognising the needs of students and the strengths of both partners'. In this instance, the two postgraduate Routes to the MSc in Computing Science which the University offers in association with the Institute are flagship programmes for the School of Computing, and provide a basis for enhancing the research profile of each institution. Additionally, there have been occasions when one of the franchised Routes has recruited students when the home-based programme has not, and the Dublin-based route has, therefore, sustained the viability of the Route overall. Members of staff and students at the Institute saw its partnership with the University as providing access to a wider range of expertise than could be available in a single institution and in adding the broader perspectives that were possible through the international dimension.
Initial approval of the franchise
19 The Commentary stated that the initial approval of the collaborative arrangement followed the standard pattern for the franchise of University programmes. In the case of the first of the two MSc Routes franchised to the Institute, a panel was convened under the authority of the School of Computing to give initial consideration to the franchise proposal. Following the successful outcome of this School-level scrutiny, a second panel was convened under the auspices of the Accreditation Sub-Committee (ASC) (now the ACC) to visit the Institute in order to extend consideration of the franchise proposal, and to advise the University's QDC on its approval.
20 At the time of the audit visit, the University did not require that the suitability of a potential partner institution should be formally established in advance of any process of approving a franchise or other form of collaboration, a position it has subsequently revised. The University does expect, however, that members of its staff will report on the learning environment available in the prospective partner to the panel charged with testing the franchise proposal. A formal application in support of approval was provided for the ASC panel which scrutinised the first franchise proposal, the IT for Strategic Management (ITSM) Route. This 'Application for Approval of a Franchise Arrangement' provided a comprehensive summary of information on the proposed partner, the learning environment for the proposed franchise and the staff support available.
21 The University uses the term 'validation' both when referring to its processes for approving the curriculum of a programme of study or an individual syllabus, and sometimes, (as in the Commentary) when referring to the process through which a proposal for a franchise is tested prior to approval. When the Commentary states, therefore, that: 'Validation panels always include external subject specialists from other UK universities and, on occasion, industrial advisors', it is not possible to be sure whether such a requirement applies to panels convened to scrutinise a franchise proposal.
22 The scrutiny of the first franchise to the Institute, in 1995, was conducted by an ASC panel which included an external member. In the case of the second franchise proposal, for the Distributed Systems Route, the only record provided for the audit team took the form of a report of an 'Enhanced School of Computing Validation', at which no member external to the University was present. The outcome of this event in 1997 is stated in terms of a recommendation to the successor of the ASC, the ACC, that it should approve the proposed franchise. It was not clear to the team whether this School-level, 'enhanced validation', had been followed by - or had stood in place of - the visit of an ACC panel and, if the latter, how the University might have determined and recorded that this was an appropriate arrangement.
23 The event at which it was decided to recommend approval of the ITSM Route franchise to the University's QDC (on behalf of its Academic Board) took place in Dublin, in November 1995. Although the relevant Route Leader's report to the School of Computing for the 1995-96 session stated subsequently that the programme had been in operation in Dublin since September 1995, members of the Institute and the University told the audit team that the first cohort of students was admitted to the programme only after approval of the franchise, and that although this was just two days after the recommendation to approve the franchise, students who had previously been interviewed for admission had been made offers 'subject to validation'.
24 In addition to addressing matters relating to administration, communication and staffing, the ASC panel which tested the 1995 franchise proposal questioned the adequacy of the descriptions of the content of some of the modules in the documentation provided for them in the form of a programme handbook. This did not prescribe the topics to be covered in individual modules, which were to be (as they are currently) negotiated annually between staff at the Institute and their University counterparts, a practice which mirrors that for the University-based Routes. Members of the ASC panel viewed this practice with some reservations, but the minuted response from the 'presenting team' provides no indication as to whether the ASC panel accepted the arguments advanced. In practice, however, the negotiation of the content of modules from year to year (and the use of local exemplar case studies) appears not to have caused any serious difficulty in the context of this particular collaboration. In circumstances where the University might not be able fully to rely upon the professional judgement of the staff of a partner institution, however, operating under a similar arrangement might provide the University with insufficient means to ensure that its expectations would be met by such a partner.
25 In the 'Enhanced School of Computing Validation' in 1997, the panel reviewed the accommodation and resources available to the staff and students at the Institute, and the need for a full-time administrator to support the programme, and made improvements to these areas conditions of its recommendation of approval to ACC. The need for these improvements appears to have arisen from the success of the existing ITSM Route in attracting more students than had been initially anticipated (see below, paragraph 51). The Institute appeared to the audit team to have met the 1997 panel's conditions in full. It also appeared, however, that the University did not have any formal means to ensure its partner's compliance with the conditions or recommendations specified in the panel's report, should that be necessary, other than through reporting by exception in the annual monitoring process and through observations by staff during visits to the Institute. If the University's arrangements with the Institute are typical of those for other partnerships, particularly overseas partnerships, it is not clear how the University would be able to satisfy itself that conditions attached to a franchise approval and/or a validation had been met in the absence of frequent visits to its partners by University staff.
Memorandum of Co-operation and Franchise Implementation Document
26 The formal documentation for the link between the University and the Institute in relation to the MSc Computing Science (Specialist Routes) comprises a Memorandum of Co-operation relating to the institutional link, with schedules relating to each award or award Route. The University considers that the documentation follows the standard format set by the University for collaborations of this nature. The formal documentation supporting the partnership was signed some 10 months after the admission of the first students to the programme. It was not clear whether such a delay is typical in the University's arrangements.
27 The Memorandum makes it clear that formal responsibility for the awards offered in collaboration with the Institute rests with the Academic Board of the University and that 'the award will be subject to the same rigorous annual monitoring, evaluation and periodic review as are all University awards'. Under the terms of the Memorandum, the Institute undertakes to provide all such reports 'as are laid down in the University's quality assurance guidelines'.
28 The arrangements established by the University in respect of the assurance of the quality of the franchised provision and the academic standards of its awards appear to be in accordance with the provisions of the Memorandum, although there are some striking instances where the University has not operated the link in accordance with those provisions (see below, paragraph 70). The audit team was not convinced that the formal agreement between the two institutions was capable of preserving for the University the capacity to intervene, where necessary, to ensure the provision of satisfactory learning opportunities for students and the academic standard of the awards for which they are working. References to the duration of the franchise (see below, paragraph 29), the powers of the University to influence staffing for the franchised provision (see below, paragraph 44), or to seek information on, or to control student admissions (see below, paragraph 50), student tracking (see below, paragraph 55), and publicity (see below, paragraphs 70 and 72) appeared to be either missing or unclear in the Memorandum and the associated schedules.
29 Each of the three signed and dated documents contained within the formal documentation (the Memorandum and the two associated schedules) stated that they were 'signed for a period of three years'. It was not made clear in the University's documents whether these periods referred to three successive intakes of students, or whether they ran from the date of signing, or from the effective date of commencement of the link. There was, in addition, a potential technical problem in that the Memorandum was expressed to be made for a single term of three years and contained no stated provision for extension or renewal. This would therefore terminate by passage of time before the end of the agreed period for the operation of the second schedule which was signed some 15 months after the Memorandum, in this case for a renewable term of three years. The schedules were not consistent with the Memorandum in that they did not specify the number of intakes approved for the franchised award. It was not apparent to the audit team where responsibility lay for ensuring that a Memorandum of Co-operation and its schedules were complete and internally consistent.
30 The Memorandum provides explicitly for the termination of the agreement and provides also that 'acceptance of the termination will be subject to appropriate arrangements being in place for continuing students to complete the programme(s)'. The University considered that in the unlikely event of termination, suitable arrangements for continuing students could readily be made, either by transfer to the equivalent programme at the University, or by the latter arranging for its own staff to travel to Dublin to deliver the taught elements of the programme and for project supervision. The audit team concurred with this view.
31 One noteworthy and commendable aspect of the University's documentation
is that each of the schedules relating to the MSc (Specialist Routes) incorporates
a Franchise Implementation Document. This contains information on
a number of matters pertinent to the operation of the link, including a
summary of the responsibilities of key staff, a schedule of key dates and
visits, a description of the arrangements for assessment, including the
criteria for assessment, and copies of the forms to be used for reporting
assessment decisions.
32 The introductory section of the Commentary supplied by the University stated that once a franchise arrangement has been approved 'it is customary for the University to appoint...a University Monitor'. The Commentary ascribed a number of roles to the Monitor, including assisting the staff delivering the franchise in the partner institution in the monitoring of the award, attending boards of studies and boards of examiners, and advising the staff in the University receiving reports from the team in the partner. The Monitor is expected to report 'annually to the University on the operation and administration of the...[franchise]'; to advise staff in the partner institution on the University's teaching and learning strategies, assessment strategies, and UK subject development; and to assist 'University admission tutors in considering applications from students graduating from franchise/validated awards'. It appeared to the audit team that the reliable fulfilment of these responsibilities might make a valuable contribution to assuring the quality of the franchised provision and safeguarding the standard of the University's awards.
33 The second section of the Commentary addressed the detail of the partnership between the University and the Institute. It stated that while the 'normal practice of the University is to have a monitor specifically identified with each course...[in]...view of the specialist nature of these particular courses, this monitoring role is undertaken by the leader for the corresponding course operating at Staffordshire'. There is no reference to a Monitor in the Memorandum and no reference to this matter in the full record of the meeting of the validation panel; nor is there scope in the Memorandum for the subsequent appointment of a Monitor, with the result that there was no provision for a member of the University to attend boards of study other than as the nominee for the University's Route Leader.
34 In view of the significance of the role of the Monitor in the University's quality assurance arrangements for its collaborative provision, the audit team sought additional information which might explain the omission of such an appointment for the franchised Routes, but received conflicting accounts of the attention given to the matter during the initial franchise approval process, which it could not reconcile. The team was therefore not able to satisfy itself that the matter had been given full and explicit consideration then, or that the University's QDC had been notified that the partnership was operating without the benefit of a Monitor and, therefore, outwith its usual quality assurance arrangements for collaborative provision. The team concluded that the University's systems for initial validation might not have been sufficiently robust on this occasion to have ensured full and adequate consideration of all matters relevant to the successful operation of the collaborative arrangement. The team has no way of knowing, however, whether this was an isolated lapse. In addressing this issue, the University might wish to consider whether clarification is required as to the level at which approval must be given for any final decision to depart from the usual arrangements, such as the appointment of a Monitor.
Quality assurance of academic provision
35 The University's chief means of satisfying itself that the quality
of its educational provision meets its expectations consist, according
to the Commentary, of the processes of annual review (the University's
more usual term for which is 'annual monitoring'), to which all modules
and programmes of study are subject, and periodic review. The University
does not stipulate that collaborative programmes are to be subject to
periodic review and a periodic review has not yet taken place covering
the franchised Routes. In addition there may be some scope for monitoring
the wellbeing of collaborative provision through the annual review of
external examiners' reports for collaborative programmes conducted by
the ACC (see below, paragraph 64). The Commentary states that the
University's schools are required to review on an annual basis each collaborative
arrangement for which they are responsible, and to produce an annual monitoring
report (AMR), although the Handbook contains no special rules or
procedures relating to the annual monitoring of collaborative provision.
36 The Handbook requires schools conducting the annual monitoring of a programme, or programmes, leading to a named award, or awards, to take account of matters including student satisfaction data, admission, progression and performance data, and the reports of external examiners. The Handbook does not specify who is to be responsible for the compilation of the AMR, although the AMRs seen by the audit team had been produced on a standard form, and each included a report completed by the member of staff responsible for the academic management of the provision being monitored. In the case of the Routes franchised to the Institute, although some members of the University were initially unsure how the AMRs had been produced, it appeared that they had been completed by the University's Route Leader after informal discussions with the Institute.
37 From its scrutiny of the University's papers, and its discussions with staff in Stafford and Dublin, it appeared to the audit team that there had been no opportunity for the Institute to make a formal contribution to the report, or one unmediated by a member of the University. Members of the Institute told the team that they produced their own annual report on the programme for the Institute's internal annual monitoring purposes. Under present arrangements there appeared to be no way in which the Institute's internal quality assurance processes and those of the University could mesh.
38 The Handbook recognises the right of schools to conduct annual monitoring in the light of their own preferences. In the case of the School of Computing, AMRs are considered by the Monitoring Sub-Committee of the School's QDC, the membership of which includes a rapporteur appointed by the University Graduate School and a rapporteur appointed by the University's central QDC. The minutes of the sub-committee, and the rapporteur's reports, are considered by the Quality Development Committee of the School of Computing.
39 The introductory section of the Commentary stated that in the course of the annual monitoring process, the quality committee of the relevant school would consider the report of the Monitor, together with the reports of internal and external examiners. In the case of the specialist Routes franchised to the Institute, the absence of a Monitor caused the Monitoring Sub-Committee to place greater reliance on the reports of the University's Route Leader and denied it an important, and quasi-independent, source of information on the operation of the franchises.
40 Within the University there are two lines of reporting from the School Quality Development Committee: to the ACC, which in turn reports to the University QDC, and to the Postgraduate Education Committee (PEC) of the University's Graduate School (established in 1997-98) which takes responsibility for monitoring all postgraduate awards annually. It appeared that the respective roles of the two committees had originally been unclear but that this had now been resolved: ACC is now primarily responsible for oversight of the monitoring processes and responsible for ensuring that action is taken on matters arising from the annual review process, while PEC is concerned with more general matters relating to postgraduate awards.
41 The strong links between staff at all levels in the two institutions, and the professionalism of staff at the Institute, have made it unlikely that problems would not be identified and addressed at an early stage; nonetheless the audit team sought to establish the effectiveness of the University's annual monitoring arrangements for the franchised Routes. The University's formal quality assurance arrangements had been unable to bring to its attention matters such as those referred to in paragraph 37 above and paragraphs 43, 48 and 57, below, but this single instance of the University's collaborative provision did not provide the team with sufficient evidence to judge what value the University's annual monitoring process was adding to its other arrangements for assuring the quality of its collaborative and the academic standards of its awards.
42 The University does not solicit the opinions of students at the Institute following the franchised Routes through questionnaires or focus groups. Feedback from students on the franchised Routes is secured in part through the attendance of University staff (in this case, the Route Leader) at boards of study held at the Institute, on which students are represented. In addition, students from the Institute visit the University at Stafford, annually, as part of their programme of studies (see below, paragraph 68). Students following the franchised Routes who met the audit team in Dublin recalled that their views had been canvassed informally on their first visit to the University. They were not, however, aware of any other mechanism for making their views known except through their representatives on the boards of studies. The team was told by some members of the University that minutes of boards of studies meetings were attached to the AMR and were seen by the School Quality Committee, but members of the School of Computing Quality Committee advised the team that they did not see such minutes. There was no provision for comment on the outcomes of student feedback or consultation exercises in the AMRs seen by the team.
43 In the course of its meeting with students following the franchised Routes at the Institute it became apparent to the audit team that the students had constructive comments to make about the operation of the partnership which they had not fully had the opportunity to express through the University's consultation processes. Senior members of the University recognised that more could - and should - be done to enhance formal access by the University to feedback from students engaged in collaborative provision.
The learning environment for the franchised
Routes
Academic staff appointment
44 As part of its initial process to approve the franchise of the ITSM Route, the University considered the suitability of the Institute's staff for delivering a postgraduate programme of study with a significant research component. The Commentary stated that the University is regularly consulted about staffing issues and claimed that the provisions of the Memorandum confer on the University the power to 'ensure that any change in the teaching staff is approved by the relevant University School': this is incorrect. The Memorandum, does, however, reserve for the University the right to specify the minimum qualifications and experience required of staff teaching on the programme (although it has never sought to do so). Members of the School of Computing were under the erroneous impression that they had, if necessary, a power of veto over the selection of staff involved in the delivery of the award (see above, paragraph 28). The University might wish to review the terms of its Memoranda of Co-operation to make clear its responsibilities in respect of the appointment of staff to teach on its franchised programmes.
45 The Commentary stated that in addition to reviewing the CVs of staff provided at each validation event, senior members of the University have been included as external members of interview panels when new academic staff have been appointed at the Institute. In its discussions with senior members of the Institute, the audit team learnt that the involvement of the University in the selection of new staff by the Institute had been relatively limited. In most cases, any consideration of new appointments by members of the University had been made after the fact of appointment, and by informal means. In practice, therefore, the University appears to have relied upon the professional judgement and quality procedures of the Institute in selecting appropriate staff. In this instance the University's confidence in its partner's judgement has been well-founded.
46 The Commentary states that 'the qualifications of staff involved in the delivery of the courses is an area of natural concern, particularly as the courses are at postgraduate level. This is an area which has been explored at each validation, and staff with appropriate postgraduate and industrial experience have been identified as the main tutors and supervisors for the course'. The record of the 1995 franchise approval process conducted by the ACC panel does not refer to staff qualifications or staff development. The record of the discussions of the 1997 'Enhanced School of Computing' panel notes only that the panel was told that 'DIT staff have £3-4,000 each as their financial support and this enabled attendance at a variety of seminars and conferences to update their knowledge and understanding'. The report of neither event made it a condition of approval that relevant staff development should be undertaken, or included a recommendation to that effect, which might have been expected had the qualifications of staff indeed been a matter of concern. The audit team was told that the University would not participate in, or initiate any kind of review of staff development needs at the Institute unless a matter had been explicitly raised in the AMR; nor had it established any milestones or benchmarks against which the progress of the Institute in developing its staff could be measured.
47 Members of the Institute told the audit team that because its teaching staff had little previous experience of postgraduate teaching and supervision when the Routes had originally been franchised, staff development had been accorded a high priority. The team was told that the Computing Mathematics and Statistics Department at the Institute spent around IR£10,000 annually on staff development for its 18 teaching and research staff and that, by the end of the current year, all such staff were expected to hold postgraduate qualifications some, interestingly, as the result of themselves enrolling as students on the specialist Routes. Comments from external examiners pointed to evidence of the work that had been done to develop the subject expertise and skills of staff at the Institute, with support from the School of Computing at the University.
Academic and personal tutorial support
48 The Commentary makes no reference to arrangements for the academic and pastoral support of students following the franchised Routes. Students following the Routes in Dublin told the audit team that their greatest need for academic support arose at the project and dissertation stage and that in addition to the support then provided by staff at the Institute, students were frequently put in touch with an appropriate member of the University staff. Students following the franchised Routes could not recall any means by which the University monitored or enquired about learner support and pastoral arrangements. The Institute has its own arrangements for student support and pastoral counselling. Students seen by the team could not recall being asked by the University for their views on the adequacy or appropriateness of these arrangements. Similarly, members of staff who met the team at the Institute could not recall any means by which the University monitored or enquired about learner support and pastoral arrangements.
The academic standards of credits and awards
49 The University considers that its external examiners are its main means
of safeguarding academic standards and the external examiners who discussed
the matter over the video link with members of the audit team were unaware
of any other means by which the University safeguarded the academic standards
of its awards. Members of the Institute identified a number of other factors
in safeguarding the academic standards of the franchised Routes, including
the fact that all written work was double marked within the Institute
and seen by University staff, who in most cases also double marked it
against agreed common criteria. In addition, members of the Institute
cited the fact that University staff were present at the majority of presentations,
and that they and the external examiner attended every final viva voce examination,
as contributing to the security of the academic standards of the University's
award. The audit team concluded that members of the Institute had developed
a sound understanding of the mechanisms needed to safeguard academic standards
and that in view of the flexibility possible within the franchised Routes,
this was important in safeguarding the academic standards of the resulting
awards.
Student admission to the franchised Routes
50 The Memorandum states that all students are required to complete a standard University registration form, to be submitted to the University, which includes a statement of admission qualifications. The Commentary states that it 'is therefore possible for the University to monitor at first hand the level of entry to the courses'. The Memorandum states that admissions correspondence and interviews are to be conducted at the Institute in accordance with the definitive documentation for the franchised Routes and the accompanying schedules state that this concerns the admissions process. There is no explicit reference in the Memorandum, or the accompanying documents, to admissions decisions being taken by the Institute, and it is unclear from the language used whether the intention was that the Institute should have the authority to make decisions on admissions to the programme without reference to the University. The student entry requirements are stated in the Commentary, but the audit team was unable to find any description of these requirements in the other documentation supplied to it. The team was told by staff at the Institute that the entry requirements applied for the franchised Routes were those it applied generally for admission to its own postgraduate programmes of study. In practice, decisions on entry to the franchised Routes have been made at the Institute.
51 The University's papers did not indicate that it placed any limit on the number of students to be enrolled on the programme by the Institute at the commencement of the franchise, nor that there were any means by which such a limit could be imposed if needed. The audit team found this surprising, since the establishment of a learning environment appropriate to the nature of the franchised Route(s) had been discussed by the ASC panel which had recommended approval of the 1995 franchise with a recommended maximum cohort of 15 students. In practice, about 15 students appear to have been recruited into each cohort and the University appears to have relied upon the professional judgement of staff at the Institute to ensure that no more students were admitted to the programme than could properly be supported by the physical and staffing resources available.
52 Members of the University told the audit team that in order to avoid the delays attendant on posting registration forms to Stafford from Dublin, students provisionally admitted to the franchised Routes were asked to complete these only during their first visit to the University, which takes place some three weeks after the commencement of the programme. At this time students would have already made a substantial commitment of money and time to the programme, and it appeared to the team that deferring registration until the visit to Stafford would make it difficult for the University to refuse registration. In this particular instance it may be that the University's long association with the Institute provides it with grounds for confidence that decisions taken by colleagues in Dublin would be soundly based. It would be fortunate should that be the case, since the University cannot have any such confidence in the ability of its own systems to identify that admissions decisions have been properly made in time to rectify any errors. Nor could the University be confident from the language used in the Memorandum that it had the power to intervene other than by terminating the collaboration should the confidence which it reposes in its partner prove to be misplaced.
53 Students following the franchised Routes at the Institute have invariably been admitted to the MSc programme on a full-time basis and have been required to follow the taught components of the programme according to a predetermined timescale set out in the Franchise Implementation Document (see above, paragraph 31). On completion of this stage of the programme (including the presentation of a project proposal), students are eligible for the award of a Postgraduate Diploma, whether or not they progress to the MSc award. The Postgraduate Diploma is not subsumed in the later award of the degree of MSc, so that students are entitled to receive and use both the Postgraduate Diploma and MSc awards. The audit team considered this to be an unusual arrangement and therefore confirmed that this was the University's normal practice with members of the University and the Institute.
54 Once students have completed the Postgraduate Diploma stage of the franchised Routes, they continue their studies through independent supervised research. Members of the University indicated that students could complete this final project and dissertation stage either 'on-campus' or 'off-campus' with no clear distinction between the two modes of study. The University subsequently informed the audit team that the 'off-campus' mode of study was approved when the MSc Routes were introduced in 1994. However, there was no mention of study by this 'off-campus' mode in any of the documents relating to the formal approval of the link with the Institute.
55 The Memorandum requires all students to be registered with the University after having completed University registration forms and states that student records will be held centrally by Staffordshire University. Members of the University supporting the franchised Routes told the audit team that there was a well-established procedure for changes of student status, involving the completion of a standard form to be returned to the University for authorisation by the Route Leader in the School of Computing. No-one among the University's academic staff, at any level, who met the team was aware of this procedure, however, or the requirement that changes in student status should be initiated and authorised by the University's Route Leader. The team was satisfied that when students in Dublin did switch their mode of attendance, the fact was notified by staff at the Institute to the University through informal rather than formal channels and that, in consequence, some changes in student status may well be decided outside the formal arrangements that govern the franchises.
56 It appeared from the documentation provided by the University that the number of students based on its own campus failing to complete the final project and dissertation stage of the MSc was a matter of concern. The problem had been debated a number of times at School level with a view to finding a solution, and it had been agreed that there should be regular reviews of individual student progress. The pattern of student completion within the Routes franchised to the Institute was not dissimilar in pattern or cause, in both cases being traceable to high market demand for people with computing skills.
57 The Institute has addressed the matter of completion by requiring that the progress of each student shall be reviewed at each examination board, at which the reasons for any deferral of assessment are considered, and that all students receive an annual progress interview. The Institute has also required, via the board of examiners for the franchised Routes, that there should be a two-year time limit for the completion of the project and dissertation stage. The audit team was told that as this deadline approached, students received reminders of an increasing degree of urgency. It seemed to the team that these measures to underpin satisfactory student progression were at least as robust as those adopted by the University itself. The team noted, however, that the board of examiners for the franchised Routes did not appear to have the authority to impose a time limit for the completion of the project and dissertation stage, which fell short of the general University requirement that study for a degree had to be completed within a maximum period of seven years from first registration. No limit similar to that introduced in Dublin had been imposed for students based at the University's own campus.
58 The University considers that its external examiners play a critical part in the maintenance of the academic standards of its awards and in providing feedback to the University in the event of any problem arising in the operation of the collaboration. Reports of external examiners were one of the few sources of data considered in an unmediated form by members of the University who were not immediately involved in the operation of the link. The University's procedures for the appointment of its external examiners were scrutinised in earlier HEQC audits; they were not re-examined in the context of the present audit.
59 In addition to opportunities to scrutinise written work produced by
students following the franchised Routes, external examiners pay regular
visits to the Institute and, through their meetings with staff and students
in Dublin, are in a position to provide the University with useful additional
information. The University can be confident in the ability of external
examiners to provide feedback on the operation of the collaboration.
60 In view of the research orientation of the MSc Computing Science Routes, assessment of students is by oral presentations and written papers. The topics for these papers are negotiated by individual students with staff of the Institute in consultation with their colleagues at the University. The Franchise Implementation Document identifies assessment sheets in a standard format for the marking of all assessments undertaken by students. All student papers, proposals and dissertations are assessed by academic staff from both institutions, and a complete portfolio of all the work is sent to the external examiners, together with the completed assessment sheets. The assessments for oral presentations are also included, along with any audio-visual materials used by the student. University staff are present at most student presentations, some of which are scheduled to coincide with the students' visit to the University.
61 One of the functions of a University Monitor appointed to a collaborative programme is to act as an additional and authoritative internal examiner. Members of the University frankly admitted that in view of the Route Leader's existing role as an internal examiner, he was disqualified from assuming this additional internal examiner role and of fulfilling this part of the Monitor's tasks. No other arrangements had been made by the School or the University to substitute for this function.
62 The external examiner conducts viva voce examinations on submission of the research proposal by the student and on completion of the MSc dissertation. The same external examiners are used for students at the University and at the Institute providing, as the Commentary notes, means for 'ensuring consistency of standards between the two institutions and in relation to comparable awards elsewhere in the UK'. The external examiners confirmed that they were present at all viva voce examinations and that, although on some occasions it was their practice only to view a sample of work, all the students' work was made available to them. The University can be generally confident in its measures for assessing the work of students following the franchised Routes.
63 In the course of the annual monitoring process, Schools are required to respond to the Quality Support Unit and the Deputy Vice-Chancellor on matters arising from external examiner reports, a process described in the 1995 HEQC report (see above, paragraph 9). For collaborative provision, reports from external examiners which refer to collaborative provision are copied to the Administrative Officers responsible for supporting ACC and its predecessor ASC, and to the Chair of that Committee. The School is expected to respond to the Chair of ACC on action taken.
64 The Commentary stated that 'from time to time, the Academic Collaboration Committee will also audit all external examiner reports for collaborative partnerships (eg December 1998)'. Having sought and consulted the report of this exercise by the University, the QAA audit team discussed the frequency of such 'audit' activities with senior members of the University. The 'audit' conducted by the University in December 1998 appeared to have been linked to the University's commitment, dating back to 1995, to initiate a process of internal audit, and to have been not specifically related to collaborative provision. Members of the University when invited to discuss this exercise used the term 'audit', confusingly, to refer instead to what was later described by the University as 'a systematic annual review of external examiners' reports' conducted by the ACC, independent evidence of which the team was unable to discover. The team was unable to judge the effectiveness of either of the activities referred to by members of the University as 'audits'.
65 External examiners' reports are sent to the Institute shortly after receipt by the University. The reports are then circulated within the team delivering the franchised Routes and meetings are arranged to discuss any necessary action. These responses and actions take place in addition to the steps which the Institute is formally required to take under the University's quality assurance procedures. Members of the University told the audit team that the annual monitoring process, and in particular AMRs, were the means by which the University could satisfy itself that matters raised by external examiners were being addressed. The external examiners indicated that they received informal and ad hoc feedback on the outcomes of recommendations they have made, but made no reference to seeing AMRs. They confirmed from their personal knowledge that their recommendations were usually acted on by the University. Since the feedback on responses to their reports received by the external examiners appeared to be chiefly provided through informal communications, the team was unable to establish the facts of the matter.
66 Under the formal agreements for the franchised Routes, decisions relating to awards for students following them at the Institute were to be made at examination boards held in Dublin, attended by University staff and external examiners. According to the Franchise Implementation Document, the Dublin-based board is authorised to make recommendations on progression, Diploma and MSc awards, and in the latter case, on whether the MSc should be awarded with distinction. Recommendations of the Dublin examination board do not appear to have been considered at the main MSc Computing Science examination board at the University, however, and the audit team was unable to identify any other body within the University which considered the recommendations. Members of staff at the Institute considered the two examination boards to be of co-equal status.
67 It was not clear to the audit team from the University's documentation what measures there were, beyond the presence of University staff and external examiners at the Dublin boards, to ensure that decisions made relating to students at the Institute were aligned with those relating to students following the specialist Routes at the University's own campus. Although both external examiners and members of the University have attended most examination boards for the franchised Routes held at the Institute, the team was told that members of the Institute had never attended an examination board at the University. Neither the University nor external examiner members of the Dublin examination board had any special position or authority in the matter of decisions of the Dublin examination board.
Internal and external communications
68 There is frequent contact between members of the University and the
Institute at all levels, and academic staff from each institution have
acted as external examiners at the other. Opportunities are also provided
for the staff supporting the programmes at the University and the Institute
to accompany the academic staff on visits to the other institution. University
teaching staff visit the Institute to participate in the assessment of
the student presentations and attend examination boards and boards of
study and, twice each year, students following the franchised Routes visit
the University, accompanied by staff from the Institute for a week-long
study period. Members of the Institute told the audit team that there
had been discussions between the University and the Institute at a very
senior level to establish a joint research fund and that these discussions
had reached an advanced stage, although no formal agreement had yet been
signed. The team was also told that the reciprocal visits that had taken
place between the Vice-Chancellor of the University and the President
of the Institute, and between other senior staff, had helped to develop
a climate of support for further collaboration between the institutions
and high-level interest in their existing partnership.
69 The Commentary stated that 'advertising of the awards in both prospectuses and newspaper advertisements is integrated with [that for] other postgraduate courses delivered by [the Institute]. The advertisements have been seen by the University and are considered not to misrepresent the University or the awards on offer'. The Commentary did not make it clear whether the advertisements had been seen by the University before or after their publication, and members of the University offered no view about their confidence in the effectiveness of the current arrangements.
70 The audit team learnt from members of the University that advertisements for the franchised Routes had not been seen prior to publication and had not, therefore, been reviewed by the University, as required under the terms of the Memorandum. The view of some members of the University's staff was that the frequency of their visits to the Institute enabled effective informal monitoring of promotional material. The University's present arrangements for the review of promotional material operate retrospectively. Senior members of the University were themselves of the view that it could have no confidence in its current arrangements to ensure that materials issued in its name describing collaborative provision met its requirements. In the light of the substantial level of collaborative work overseas in which the University is involved, it will wish to reconsider as a matter of urgency its arrangements for monitoring the accuracy of promotional materials for such provision.
71 The Institute issues its own Quality Assurance Handbook which the Commentary considers is 'complementary to Staffordshire University regulations, and explicitly does not supersede them'. Members of the Institute were clear that in the case of a conflict between the Institute's Handbook and that of the University, the latter would prevail.
72 The Institute also issues students following the two franchised Routes with a Student Handbook, which contains no reference to Staffordshire University, and which sets out the composition of examination boards for the MSc awards in a way which is inconsistent both with the Memorandum of Co-operation between the two institutions, and with the practice that has been followed by them. The University appeared to operate no procedure to approve the contents of the Institute's Student Handbook prior to publication; indeed, Route Leaders at the University were unaware of its contents. Notwithstanding the absence of any reference to Staffordshire University in the Institute's Student Handbook, students who discussed arrangements for appeals and complaints were aware that although any requests for special consideration or appeals would initially be routed through staff at the Institute, they would ultimately fall to be determined by or on behalf of the University.
Conclusions
73 The University considers that an international outlook and international
activity are essential elements of present-day University life for students
and staff, and that participation in international activity presents the
University with an opportunity to contribute to the economic and cultural
well-being of the West Midlands Region and the immediate locality of Staffordshire.
Much of the University's existing overseas collaborative provision has
hitherto developed from school-level initiatives, founded on individual
relationships forged between members of staff in the University and colleagues
in overseas institutions. The University's partnership with Dublin Institute
of Technology has followed such a pattern, developing from staff meetings
dating back to 1986 to discuss interests in teaching and research, to
an articulation arrangement in 1992, allowing successful diploma students
from the Institute to enter undergraduate degree programmes in computing
at the University with advanced standing.
74 In 1995, when the University agreed to franchise the first of two of its taught postgraduate Routes to the Institute, the development of the franchise was able to draw on almost nine years of contacts between teaching staff in the two institutions. Since 1995 the mutual knowledge of the staff teams in Dublin and Stafford, and the professionalism of each has underpinned the successful and continuing development of the franchises, the quality assurance of which largely depends on the shared determination of the two staff teams to safeguard and enhance the quality of the educational provision and the standard of the awards with which they have been entrusted. In this they have been ably and enthusiastically supported by the external examiners appointed to the programmes, whose ability to monitor student performance across all the Routes of the MSc, including those franchised to the Institute, provides a sound basis for accepting that the academic standards of the University's awards have been safeguarded. It is regrettable that the University's formal quality assurance arrangements appear to be not as well designed as they might be for the purpose of assuring the quality of the educational provision which comprises the franchised Routes and the academic standards of the awards to which they lead.
75 The Commentary which the University provided to support the audit had been produced in two sections: the first, provided from the centre of the University, taking the form of a statement of its general approach to assuring the quality of its collaborative provision; the second, information from the University's School of Computing concerning the detailed development and operation of the two franchised Routes. In some particulars, the contents of the two sections of the Commentary do not fit easily together. The Commentary did not offer a view of the strengths or limitations of the University's academic quality assurance arrangements, or the degree to which its partnership with the Institute was typical of other overseas franchise arrangements.
76 The University needs to recognise that at present the academic well-being of the franchised Routes depends upon the professionalism of the staff directly involved in their delivery, and their conduct of assessments and awards, rather than its general academic quality assurance arrangements for its franchises. The University's arrangements for monitoring the admission of students, controlling the development of the curriculum, approving new staff to support the franchised provision, and monitoring the content of student handbooks and promotional materials for students studying through collaborative provision require attention. The information provided by the University indicates that it can have only limited confidence in its capacity to know what is being done in its name by its schools in the matter of collaborative provision and, therefore, in its capacity to intervene effectively when this is necessary, to ensure that its expectations are met.
77 The need for the University to review its quality assurance arrangements for its overseas collaborative provision had not formally been identified through its own quality assurance arrangements or reported in the University's Commentary; nonetheless, in the course of meetings at the University it was apparent that members of its senior staff had independently begun to identify some of the weaknesses in its quality assurance arrangements outlined in this report. There were also indications that the same senior members of staff were determined to remedy the weaknesses outlined in the previous paragraphs.
Annex
Note of changes implemented since the visit or about to be implemented
1 The University is committed to the production of an updated Quality Assurance Handbook by the New Year. It is also committed to the production of a new Quality Assurance Handbook for Collaborative Provision which will take account of the recently published QAA Code of practice for the assurance of academic quality and standards in higher education. Section ii: Collaborative provision (July 1999) and the draft Handbook on Collaborative Provision for Practitioners produced by the Council of Validating Universities. The process of consultation involved in the production of each of these documents, in which the University has participated, has created a high level of debate and awareness about the issues involved in collaborative audit across the sector. This, taken with the University's own experience of operating its quality assurance procedures through the Quality Development Committee and its sub-committees and its first audit report on collaborative provision, makes the next few months the ideal time for a review of procedures and the production of a new Handbook.
2 During the 1998-99 session the Quality Development Committee established a working group to review the Annual Monitoring Report form. A new electronic version of the Report form has been agreed by Quality Development Committee. Key changes include the inclusion of a specific section on formal Student Feedback and an emphasis on Action Points and Planning. The new format will be implemented from September 1999 for all provision, including collaborative.
3 The University established a new institutional procedure for assessing the suitability of a potential new partner institution during 1998-99 and has implemented this procedure on two occasions to date. The formal decision about proceeding to a Memorandum of Co-operation is taken by Academic Board.
4 The University will implement new arrangements for monitoring the accuracy of promotional materials for its collaborative work overseas during the 1999-2000 session. This arrangement will require the University to be active in ensuring that it scrutinises promotional material in advance of its publication and that it does so systematically across all its collaborative provision.
5 The School of Computing has appointed a Monitor to the Dublin programmes.
6 The procedure for changes to student status will be tightened up [paragraph 53].
7 The University created the new post of Director of Quality Improvement during 1999 with the appointee taking up post in September 1999. The post was created to provide the University with increased leadership capacity in quality enhancement and management matters.
Top