1 The Quality Assurance Agency for Higher Education (QAA) is a UK organisation
which aims to promote public confidence that the quality of provision and
standards of awards in higher education are being safeguarded and enhanced.
It provides public information about quality and standards in higher education
to meet the needs of students, employers and the funders of higher education.
One of QAA's activities is to carry out quality audits of collaborative
links between UK higher education institutions and some of their partner
organisations in other countries. In the spring of 2001, QAA audited selected
partnership links between UK higher education institutions and institutions
in Hong Kong. The purpose of the audits was to provide information on the
way in which the UK institutions are maintaining academic standards and
quality of education in their partnerships with institutions in Hong Kong.
The process of audit of overseas partnership links
2 In planning these audits of overseas collaborative provision, QAA invited all UK institutions to provide a list of their collaborative links with partners in Hong Kong. On the basis of the information provided on the range and scale of the links, computing and information technology was selected as the subject focus for the audit. Each of the UK institutions whose collaborative link had been selected for the audit provided for QAA a Commentary describing the way the partnership operated, and commenting on the effectiveness of the means by which the UK institution assured quality and standards in the link. In addition, each institution was asked, as part of its Commentary, to make reference to the extent to which the link was representative of its procedures and practice in all its overseas collaborative activity or specific to this link, subject or country. QAA identified four UK institutions which had established arrangements where most or all of the educational provision was delivered through a partner in Hong Kong, and these institutions were visited by small teams of auditors to discuss the arrangements with appropriate staff and to look at relevant documentation.
3 Audit teams visited the partner in Hong Kong to gain further insight into the experience of students and staff, and to supplement the view formed by the team from the institution's Commentary and from the UK visit. During each of the visits in Hong Kong, further documentation about the link was made available to the team, and discussions were conducted with key members of staff, lecturers and students. In addition, members of the team, who included subject specialists in computing and information technology, were able to see facilities and resources available to the students. Examples of students' work were seen by members of the team, including the subject specialists either in Hong Kong or in the UK. QAA is grateful to the UK institutions and their partners in Hong Kong for the willing cooperation provided to the team.
4 Institutions were invited, in their Commentaries, to make reference
to the way in which their arrangements met the precepts of QAA's Code
of practice on the assurance of academic quality and standards in higher
education, Section 2: Collaborative provision (1999) (QAA's Code).
QAA's Code contains precepts and guidance about the assurance of
quality and standards in all collaborative provision, and it is expected
that institutions would be adhering to the precepts of the Code.
In the context of these audits, the Code was used as a reference
point by the audit team, and its contents are reflected in the observations
in this report.
The context of collaborative provision with partners in Hong Kong
5 Under Hong Kong legislation introduced in 1997, all courses conducted
in Hong Kong leading to the award of non-local higher academic or professional
qualifications have to be registered or exempted. This means that all programmes
offered by UK institutions in partnership with institutions in Hong Kong
have to be registered or exempted. The criteria for the registration of
a course include the requirement that there are effective measures in place
to maintain the standard of the course at a comparable level with that
of a similar course conducted in the home country. Those courses that are
conducted in collaboration with a local institution of higher education
may be exempted from registration. Registration is approved by the Registrar
of non-local higher and professional education courses, and the decision
is based on advice from the Hong Kong Council for Academic Accreditation
(HKCAA). Registration or exemption does not mean that the course is also
recognised by the HKCAA or employers in Hong Kong. Courses which are considered
to be 'purely distance learning' are excluded from the registration requirement.
The collaborative link between London Guildhall University, NCC Education
Ltd and the Hong Kong Management Association
6 The collaborative provision considered in this report relates to the programme of part-time study delivered through the Hong Kong Management Association (HKMA) leading to the award of BSc (Honours) in Computing and Information Systems of London Guildhall University (the University). The University's Commentary on the collaborative link described the programme as a joint programme with NCC Education Ltd (NCC) leading to an external degree of the University. HKMA is NCC's accredited delivery centre in Hong Kong. HKMA is a non-profitmaking organisation primarily engaged in the provision of management training. It offers a series of higher education programmes in association with universities in the United Kingdom, Australia and China. The programme is registered with HKCAA. The language of instruction and assessment is English.
7 An initial visit to London Guildhall University took place on 25-26 January 2001. The audit team comprised Dr D J Buckingham, Dr D H Furneaux, and Dr S Jackson. The team held two formal meetings during the visit. The first was with University staff centrally responsible for the link and representatives from NCC Education Ltd, and the second with staff responsible for the programme aspect of the link. The University had suggested to QAA that the team should visit NCC in Manchester, on the basis that its direct link was with NCC rather than with the centre in Hong Kong. QAA decided that it should not take up this suggestion on the basis that its responsibility was to consider the way in which the University, as the awarding body, exercised its stewardship of quality and standards in the collaborative link.
8 A visit to HKMA took place on 21 March 2001. The audit team comprised
Dr D J Buckingham, Professor R S Burgess, Mrs N J Channon, Dr S Jackson
and Professor B S Lee. The team met with senior staff of HKMA, staff with
responsibility for administration and management of the programme, the
NCC Regional Moderator in Hong Kong, academic staff and a representative
group of students.
Background
9 The University established in February 1996 a relationship with NCC Education Ltd (then known as the National Computing Centre) whereby students could study for an external degree of the University at NCC-accredited centres worldwide. The partnership with NCC was established prior to the formal arrangements described in the University's Quality Assurance Handbook. In its Commentary, the University stated that, as part of its initial approval and validation process, it satisfied itself of the good standing of partner institutions, and that in assessing the proposed partnership with NCC 'legal advice was not sought, but the Head of Department elicited opinions from a number of sources which confirmed the high standing of NCC'.
10 The validation of the external degree in Computing and Information Systems took place in 1996. The report recommended that the NCC International Diploma in Computer Studies (IDCS) should be credit rated at 120 specific credit points at level 1, that the NCC International Higher Diploma in Computer Studies (IHDCS) should be credit rated at 120 specific credit points at level 2, and that the IDCS and IHDCS taken together should be considered as the entry qualification for the level 3 course Computing and Information Systems. The validation report recommended that the University should approve a one-year level 3 course, jointly developed by the University and NCC, designed to run at centres specially accredited by NCC to deliver at this level. A contractual Agreement was signed after the initial validation. The level 3 course was first delivered in January 1998 at three NCC-accredited centres to a total of approximately 40 students. NCC's accredited centre in Hong Kong is operated by HKMA, and delivery of the level 3 course at HKMA began in July 1999. At the time of this audit there were 13 NCC-accredited centres, and the June 2000 examinations had been taken by over 600 students. The audit team was informed that the number of students was projected to grow to around 1,500 over the next two years.
11 The planning and establishment of the University's collaborative arrangements have, since 1998, been guided by its International Strategy, and its Strategy, Principles and Framework for Collaborative Course Provision. Detailed procedures for the consideration and approval of collaborative arrangements are contained in the University's Quality Assurance Handbook. The audit team considered these procedures to be detailed and comprehensive, and generally to meet the expectations of QAA's Code of practice on collaborative provision.
12 The University refers to the programme as a 'joint' programme because
the management is shared with NCC, although the ownership of the programme
remains primarily with the University. The programme also has aspects of
external degree provision in that the University does not have responsibility
for delivery of the teaching (see below, paragraph 19). In addition the
programme has many aspects of a franchise arrangement in that the basic
curriculum content is similar to an equivalent programme offered at the
University, although with some significant differences (see below, paragraph
29). In its Commentary, the University included an analysis of how
representative the association with NCC is of its normal collaborative
arrangements. It stated that, in many ways, arrangements for this link
are fully representative in that: the relationship was specified in a formal
contractual agreement; the programme was required to follow the normal
procedures for periodic review and annual course monitoring; and the University
determined the wording on the award certificates. However, it explained
that the link is not representative in that: there was no direct link between
the University and the delivery centre (see below, paragraph 21); there
was a reliance upon the partnership with NCC for assurance of the quality
of learning opportunities and student support (see below, paragraph 19);
the assessment was set by the University, marked by local staff and moderated
by University staff and the external examiner (see below, paragraph 47);
and the University had no direct involvement in the delivery of the programme.
The University subsequently commented that its statement in its Commentary that
'there is not a direct link between the University and a delivery centre...was
in no way meant to signal a lack of acknowledgement by the University of
its responsibilities with regard to standards, or a devolution of any of
those responsibilities to NCC, or that communication does not take place
directly between the University and HKMA'. The audit team recognised that
the University had direct communication with HKMA in addition to its communications
via NCC (see below, paragraph 28).
Formal arrangements
13 The University's link with NCC is the primary association in the collaborative arrangement which is the subject of this report. The Agreement to 'offer a joint programme leading to...an external degree of BSc Computing and Information Systems' is between the University and NCC Education Ltd. The University relies on NCC's procedures for the accreditation of centres such as HKMA for the delivery of the joint programme, and for the monitoring of local activity. NCC's procedures for the accreditation of centres were initially designed for the delivery of its IDCS and IHDCS programmes (the IHDCS has since been renamed the International Advanced Diploma in Computer Studies - the IADCS). Further guidance has been produced by NCC for the accreditation of its centres to deliver programmes to degree level. These procedures cover: the academic and professional standards of the programme; qualifications and experience of staff; standard and extent of teaching and computing facilities; lecturer and student materials; adherence to the NCC Code of practice; and effectiveness of management and administration. They are also applied in annual renewal of accreditation. The University stated in its Commentary that it had approved these procedures.
14 The programme is managed by a Joint Steering Group, which comprises three University staff and three NCC staff who are involved with the operation of the programme together with the University's Institutional Liaison Officer. The Group is chaired by the NCC representative of NCC's Academic Steering Group, and meets at least once every examination cycle.
15 A Joint Examination Board is responsible for the assessment of students and the conferment of awards. Its membership includes several members of the Joint Steering Group together with the University's course unit examiners and the external examiner(s). The Board is chaired by the University's Head of Computing, Information Systems and Mathematics and meets at the end of each examination cycle.
16 The University controls the issue of award certificates and transcripts. The audit team was shown examples, which were in accord with the recommendations of QAA's Code of practice on collaborative provision. The Agreement states that the degree certificate shall indicate that the award is an external degree of the University and name the accredited centre where the programme was delivered.
17 The Agreement stipulates that all publicity information should be forwarded to the University's Academic Registrar and to NCC's Programme Manager for approval. Publicity material was discussed during the audit team's visits to the University and HKMA, and the control of paper-based material appeared to the team to be secure. The control of web-based material appeared to be less well-addressed. For example, the team noted that the NCC web site described the final year of study as 'the final year of NCC Education's honours degree programme'. The University may wish to discuss web site publicity with its partner, and ensure it has the opportunity to approve all such publicity material before it is published, as required by the Agreement.
18 During a periodic course review in July 2000 (see below, paragraph 27)
the opportunity was taken to update the Agreement, and the current
version covering the period August 2000 to August 2005 was made available
to the audit team. The team considered this version to adhere to the relevant
precepts of QAA's Code of practice on collaborative provision. In
the context of the award being an external degree of the University, the
team concluded that the arrangements for this collaborative provision generally
adhered to the precepts of the Code.
Responsibility for quality and standards
19 The Commentary and additional documents provided to the audit
team described a clear division of responsibility between the University
and NCC. The Commentary stated that, in broad terms 'the University
is responsible for the standard of the award and NCC is responsible for
the quality of learning opportunities and student support'. This division
of responsibility, to NCC for quality of provision and to the University
for the standards of awards, is made clear in the Agreement.
Quality of learning opportunities and student support
20 NCC's role as specified in the Agreement is that 'all matters
relating to students taught at the accredited centres shall be the responsibility
of the accredited centres', and that 'the NCC shall be responsible for
the accreditation of all training centres'. A member of the University's
staff from the initial validation panel observed one of NCC's initial centre
accreditation events. Thereafter, the University has reviewed the accreditation
procedures rather than having a University representative at accreditation
events, although in the formal agreement it reserves the right to take
part in the process. NCC conducts its accreditation and annual re-accreditation
against guidance set out in its Procedures Manual, a copy of which
was available to the audit team.
Liaison and administration
21 The University's Quality Assurance Handbook states that each
link should have an Institutional Liaison Officer for ensuring the implementation
of the contract at institutional level, and a Course Liaison Officer responsible
for providing appropriate subject specific and pedagogic advice to staff
in the partner institution. In the collaborative link with NCC and HKMA,
the role of the Course Liaison Officer is non-standard, as the University's
link is with NCC and not directly with the centre delivering the programme.
The link between the University and NCC's delivery centres is indirect,
but there are several examples of direct communication between the University
and HKMA, particularly in matters of assessment (see below, paragraphs
33 and 47 et seq). The audit team noted that the sections on collaborative
provision in the Handbook were generally detailed and comprehensive.
These sections did not, however, identify overseas collaborative provision
separately, and the additional challenges overseas collaboration might
present. The University drew the team's attention to the coverage in its Handbook of
the importance of considering issues of compatibility of systems and ethos
when selecting a collaborative partner, and of political, cultural and
legal issues that may arise from overseas collaborations. Nevertheless,
it seemed to the team that there was still an assumption that, in the operation
of a link, the University's processes within the UK could be applied to
the overseas situation without modification. The team considered that this
assumption might be unsafe, especially when some of the University's links
with the overseas delivery centre are direct, but others are through its
collaborative partner.
Monitoring of provision
22 A formal visit to accredited centres is normally made each year by a small team of University staff (at least two), in order, according to the Commentary, 'to have first hand experience of the course and the QA procedures in operation in the overseas centres'. The University visited a number of the NCC-accredited centres in May and June 2000, not including HKMA. The reports of these visits usefully identified a number of outcomes to be addressed by the University and its partners, including a clarification of the responsibilities of the University, NCC and NCC regional moderators, and the consideration of a closer direct relationship between the University and the accredited centres.
23 The first cohort of students on the level 3 course at HKMA completed in June 2000, and the University conducted its first monitoring visit in the following November. The visiting panel consisted of three members of the University staff and two members of NCC who joined the team in the capacity of observers. The panel met senior staff at HKMA, members of the course team and a group of students. It also reviewed the resources and teaching facilities, although teaching observation was not possible during the visit as it happened to coincide with a half-term break. The report of the monitoring visit showed that the panel had given careful attention to the quality of provision available to students at HKMA, and made recommendations for enhancement. The report identified positive features, including the good academic performance of the first graduating cohort, and the students' positive view of the level 3 course. Matters identified for further consideration included the production of staff and student handbooks, enhancements to software and improvements in the timing of reporting results to students. At the time of its visit, in March 2001, the audit team was able to confirm that many of these matters had since been acted on by HKMA.
24 A feature of NCC's monitoring arrangements is the use of regional moderators appointed by NCC to cover a number of centres in a designated geographical region. The Commentary explained that a regional moderator was responsible for providing 'an effective communication link between NCC, the accredited centres, students, the local computing community, and local government'. Their duties include involvement in centre accreditation, ensuring standards are monitored through classroom observation, considering staff changes, providing monitoring reports on the centres, and providing guidance and a point of contact for students. Regional moderators report directly to NCC through the mechanism of an annual report. According to NCC's Procedures Manual, regional moderators are expected to visit each centre twice a year, at which time they chair staff-student liaison committees, and gather student feedback from the anonymous questionnaires that students are asked to complete at the end of each taught unit. The report of the monitoring visit to HKMA in November 2000 noted that 'formal staff-student committees, chaired by the NCC Regional Moderator, had not taken place', and recommended that regular meetings of the staff-student committee should be started immediately. The Regional Moderator for Hong Kong subsequently met a group of 26 students from the second cohort and discussed a variety of issues relating to the programme including delays with approval for project topics, concerns about a lack of practical exercises in the programme and requests for booklists for each course. A member of the teaching staff was present at the meeting, which the team considered to be a departure from good practice in seeking student feedback. The matters raised through this student feedback were forwarded to both the University and NCC for consideration. In the case of the programme at HKMA the Regional Moderator had also participated in the project presentations by students. The audit team considered that, when a regular schedule of meetings with students has been established, the Regional Moderator in Hong Kong would play a useful role in monitoring the quality of learning opportunities and in the provision of student support.
25 It is a University requirement that all the University's provision is subject to annual course monitoring. The annual monitoring report, covering all delivery centres, for the joint programme leading to the University's external BSc in Computing and Information Systems is produced by the Joint Steering Group in accordance with the guidance given in the University's Quality Assurance Handbook. The report brings together information from four main sources: the Joint Examination Board; individual centre annual course monitoring reports; the regional moderators' reports; and reports from University monitoring visits. In the case of the monitoring report for 1999-2000, the report of the periodic review of the level 3 course held in July 2000 (see below) was an additional information source. The contribution by teaching staff to the annual monitoring report is indirect. In accordance with the agreed line of communication between the University, NCC and the delivery centres, teaching staff are encouraged to report any concerns to NCC rather than directly to the University. The University's Commentary was realistic in its evaluation of the outcomes of the monitoring process, and identified several matters requiring attention, mainly associated with the variable quality of the individual centre reports. It stated that further attention would be given to staff development in the centres. The audit team noted that further guidance on annual monitoring procedures was provided during the University's monitoring visit to HKMA in November 2000.
26 The report of the Joint Steering Group is submitted to NCC's Academic Steering Group and to the University's Working Group to Coordinate Ongoing Institutional Links (WGCOIL), which is a sub-group of the University's Quality Management Committee (QMC). WGCOIL produced a report in March 2000, with subsequent updates in August 2000 and January 2001, on its annual monitoring of all the University's collaborative provision during 1998-99. In respect of the joint programme with NCC, the documents available to the Group for the production of that report included NCC's overview report, annual monitoring reports from all delivery centres and minutes of the meeting of the Joint Steering Group of November 1999 at which the centres' monitoring reports were considered. The audit team considered that WGCOIL had made a careful analysis of the available documents. In respect of the joint programme overall, WGCOIL's report noted the good academic standards of student achievement. It considered, however, that the monitoring reports were not adequate, and 'insufficient to judge the quality and health of the course'. The Group's collective report on the University's collaborative provision for 1998-99 included a number of substantive developmental points and concerns about procedural matters in relation to the joint programme with NCC, on the basis of which it noted that 'the Group had some concerns as to the ability of NCC to communicate the University's quality assurance requirements to the accredited centres'. The minutes of the QMC meeting of March 2000 noted that it had received WGCOIL's report, but did not refer to any discussion of matters raised in the report, despite the report's observation, in relation to the joint programme with NCC, that the monitoring reports 'raised several issues of serious concern regarding the link itself'. From the minutes of the meeting of QMC, the team was unable to form a view of the level of engagement of the University's senior quality management committee with the matters raised by its sub-group's comprehensive monitoring report for 1998-99 in relation to the joint programme with NCC.
27 Under the University's standard procedures for periodic review, the joint programme was reviewed in July 2000. The review was based upon a study of the documentation and student work from all centres delivering the level 3 course, and was held at the University. The audit team considered that the review report gave a useful analysis of this collaborative provision. The report noted the variable quality of annual monitoring reports from centres, and discussed with the course team the measures that were being taken to address this problem. The panel agreed 'that these measures were required'. The review panel noted the comments in WGCOIL's report on the monitoring of collaborative provision for 1998-99 in respect of centres' annual monitoring reports, and 'was convinced that ... appropriate action [had been taken] to resolve the problems raised by the monitoring reports'. The review panel 'was impressed by the new procedural framework that had been established for the management and quality assurance of the accredited centres', but advised that 'special attention should be paid to monitoring mechanisms culminating in the annual monitoring report to ensure that the new procedures were successful'. The review report recommended that the course should continue with indefinite approval, and that the next review of the course should be in session 2005-06.
28 Although the Commentary was clear about the division of responsibilities
and the linear line of communication between the University, NCC and NCC's
accredited delivery centres, it appeared to the audit team that there were
several areas where, as the relationship has developed, it has not been
easy to make this division as discrete in practice as it is described in
the Commentary and in the Agreement. The outcomes of the
University's monitoring procedures tend to confirm that view. The WGCOIL
monitoring report on collaborative provision in 1998-99 noted, in respect
of the joint programme with NCC, that 'many centres were unsure of communication
lines between themselves, NCC and the University'. It also noted that 'centre
reports suggested that some centres were keen to communicate directly with
the University' but that 'the arrangement specifically stated that there
would be no direct communication between the University and the centres'.
The report of the monitoring visit of November 2000 to HKMA concluded that
the 'visit revealed a lack of clarity about the roles and responsibilities
of certain people within the link model', but the visiting team 'was able
to provide the Centre with a clear list of responsibilities'. In the annual
course monitoring report of the Joint Steering Group for 1999-2000, the
Group reported that it had received an adequate monitoring report from
HKMA, but commented that 'the overview showed an incomplete understanding
of the responsibilities within the partnership'. The report of the July
2000 periodic review of the joint programme observed, in respect of the
programme's accredited centres generally, that the University 'has recognised
that it is necessary to make an annual visit to all centres, as the recent
visits had emphasised the great benefit for centre development of having
contact with the University'. The team's discussions at HKMA supported
these findings of the University's monitoring and review of the link. The
team noted that University staff were taking action to improve the situation
through a more direct engagement with HKMA in matters relating to student
learning and support. In the light of its monitoring and review of the
link between the University, NCC and the NCC-accredited HKMA delivery centre,
the University will wish to satisfy itself that the statement of the roles
of itself and NCC (see above, paragraphs 12 and 19) in relation to HKMA
continues to provide appropriate guidance, reflecting operational practice,
on the respective responsibilities and lines of communication between the
three parties engaged in the link.
Curriculum
29 The University stated in its Commentary its view that 'the quality of educational provision is...equivalent to that offered within the University', and that the 'LGU/NCC level 3 course is equivalent to the University-based level 3 course'. There are a number of differences between the level 3 course delivered at the University and that delivered at HKMA. The University programme has six taught units and a double project unit. Students must follow the core elements but have a choice from a number of option units. The level 3 course at HKMA has been repackaged to include the core material and material selected from the option units which was thought 'most appropriate to NCC IDCS and IHDCS'.
30 Assessment of the achievement of HKMA students in the taught units is
based solely on formal examinations, and coursework assignments do not
contribute to summative assessment. In the University's own version of
the level 3 course, assessment of coursework assignments contributes at
least 40 per cent of the summative assessment of each taught unit. In respect
of opportunities for formative assessment through assigned coursework,
the HKMA web site noted that 'although coursework is not directly assessed
it will be set, marked and returned to students in order to provide formative
feedback'. The panel that conducted the course review of July 2000 explored
with the course team the lack of a coursework element in the joint level
3 course, and accepted that the 'principal barrier to including assessed
coursework was cost'. The draft student induction presentation (see below,
paragraph 39) emphasised the formative value of coursework, and explained
that the coursework was not counted as part of the summative assessment
because the University could not satisfy itself of the standards of assessed
coursework. The audit team appreciated the University's arguments for deciding
not to include coursework as part of the assessment of the joint level
3 course, but noted that the decision had put more emphasis on the individual
project (see below, paragraph 33) as the vehicle for giving students experience
of assessed investigative independent work at honours level.
Learning resources
31 Production of the teaching material for the programme is the responsibility of NCC. The University informed the audit team that NCC had offered commissions to staff of the University to write the material for the taught units of the joint programme, and with the exception of one unit this offer had been accepted. In the view of the team the material is of good quality, and provides a helpful resource for the teaching staff at HKMA. The teaching staff are free to decide how best to use the allocated class-time, generally for lectures and discussion of the resource materials, question and answer sessions, small group exercises and occasional demonstrations of software. However, in discussion with the teaching staff it was confirmed that students are generally reluctant to read beyond the core materials because of the pressure of their full-time jobs; they tend to focus on the requirements of the formal examinations and adopt learning strategies accordingly. The University might wish to reinforce its guidance on wider reading around the subject in order to satisfy itself that students act upon this guidance and engage in study beyond the core teaching material in a way that would be appropriate to study at honours level.
32 The report of the monitoring visit of November 2000 to HKMA had noted
that IT resources and classroom facilities were good, but made some recommendations
for the purchase of CASE tools to 'significantly enhance two of the taught
units', and the provision of Oracle to 'improve the course', although this
was seen as less significant. The audit team supported these observations,
and considered that the provision of advanced software such as Oracle or
Rational Rose was needed to support the course. The report of the November
2000 monitoring visit also noted that library resources at HKMA were limited.
The team agreed that the library was limited in the provision of books
and journals that would support reading beyond the core material, although
several students who met the team explained that they had made use of sites
on the internet for up-to-date information to support project work. By
acting upon the recommendations of the report of the November 2000 monitoring
visit in respect of access to library resources the University will encourage
students at HKMA to support their studies with wider reading appropriate
to the final year of an honours degree programme.
Project
33 The arrangements for the management and assessment of the project are contained in the University's Project Handbook. At HKMA, students complete the individual project in the third semester of the programme, alongside two other taught units (see below, paragraph 46). When asked about the use of object-based methods within their projects, students indicated that they believed that they had sufficient knowledge from the IADCS to tackle their projects. Topics are chosen following discussion with staff at HKMA and are forwarded to the University for approval and comment. The University gave this as an example of direct communication between the University and a delivery centre without routing via NCC, although the draft induction presentation (see below, paragraph 39) indicated that approved proposals were sent to the University via NCC. Students frequently, and reasonably, choose topics which are relevant to their employment. At the time of the audit visit, all projects were supervised by one member of staff at HKMA, with assistance from other staff where appropriate.
34 The report of the July 2000 review of the joint programme overall noted that 'a number of the projects provided in the base room did not include sufficient testing and evaluation'. The course team had informed the review panel that 'further reference to this aspect had now been included in the project handbook'. Three projects from the first cohort of 12 graduating HKMA students were seen by the audit team. This sample represented the top, middle and bottom of the mark range; the top and bottom marks had been moderated slightly downwards. The team considered that the projects in this sample contained less critical evaluation or evidence of research than would be expected at honours level, or demonstrated having achieved the aim expressed in the Project Handbook of making 'a critical appraisal of the work done and reached sound conclusions'. The team took the opportunity during its visit to HKMA to study a further sample of five projects. It considered that these were, on the whole, well-executed practical projects, but they showed little evidence of research and critical evaluation. Projects were described to the team by HKMA teaching staff as practical in their approach, with students preferring implementation tasks. In discussion with the students, it emerged that they expected the assessment of the project to reward a professional approach to developing a product rather than research-based analysis or critical evaluation, regarding the project as similar in its requirements to practical projects completed for the diplomas. They did not express awareness of the change in level or demand that was represented by the progression from diploma level studies to the final year of the honours degree programme. The July 2000 review panel reported a need for centres to 'be made more aware of the difference between the level 2 NCC IADCS and the final year of the degree'. The review report noted that the course team planned to address this problem with an induction programme 'particularly emphasising the demarcation between this and the previous year's study' (see below, paragraph 39).
35 The audit team was confident that the practical and problem-solving
nature of the projects would be in keeping with the British Computer Society's
(BCS) requirements for exemption and accreditation. It was less confident
that the sample of project reports that it had studied demonstrated sufficient
critical evaluation of project outcomes or process, or evidence of research
into the problem domain, to meet in full the BCS expectation that the project
reports would include 'an in-depth investigation of the context/literature/other
similar products', and 'a critical appraisal of the project'. Given that,
in the absence of assessed coursework (see above, paragraph 30), the project
unit of the joint programme is the principal opportunity for students to
demonstrate their achievement of the investigative and evaluative skills
expected at honours-level, the University may wish to review its guidance
to NCC about its expectations for achievement in the project unit in order
to satisfy itself that the unit offers students learning opportunities
and challenges appropriate to a level 3 course.
Staffing and staff development
36 Staff are recruited and appointed in Hong Kong by HKMA against criteria set by NCC. The University has no direct involvement in appointment but has agreed criteria with NCC with regard to staff quality. These criteria put emphasis on the appointment of competent and up-to-date practitioners. The staff who met the audit team in Hong Kong had experience of working in computer-related occupations, and worked for HKMA on a part-time basis. They confirmed that they had been interviewed by HKMA and observed in the classroom, and their performance was monitored by student feedback questionnaires.
37 The University's Commentary identified the appointment of staff to the teaching team as a responsibility of the NCC-accredited delivery centre. Individual centres are responsible for sending the CVs of appointed staff to the University and to NCC by way of advising of staff changes. The report of the November 2000 monitoring visit to HKMA noted that staff stability was good, and that new staff were interviewed and their teaching observed, although their later performance was monitored only by student feedback questionnaires. The audit team learnt from its discussions at the University that there had been occasions when there had been considerable delay in the passing-on of information about staff changes at NCC-accredited delivery centres. The University subsequently reported that 'lecturing staff, and therefore new staff, are identified six-monthly on the marks spreadsheets, and also on the annual monitoring visit'. The team considered this to be a positive development, although it did not specifically address the matter of informing the University of staff changes at the time of appointment.
38 NCC has responsibility for staff development at its accredited centres,
although the University and the centres themselves also have a role in
staff development. The report of the visit to HKMA by University staff
in November 2000 indicated that staff development was 'seen as a joint
responsibility of HKMA and individual staff', including the identification
of courses to meet development needs. It appeared to the audit team that,
in terms of staff development directly related to the programme, the major
effort had focused on achieving a good understanding of the assessment
processes and marking criteria. This aspect of staff development has been
delivered by University staff during their visits, another example of direct
contact between the University and HKMA in matters of quality in the quality
of provision. Following the November 2000 monitoring visit, a handbook
has been produced by HKMA to provide staff with guidance on teaching methods
and assessment.
Student information and support
39 Students are supplied with the Student Handbook which was produced by HKMA following the University's monitoring visit in November 2000. The Handbook includes details of the University, NCC and HKMA, the programme structure and calendar and assessment arrangements. The Handbook also includes an outline bibliography for each of the taught units. The audit team acknowledged the usefulness of the Handbook, but considered that many of the references provided for students were not as recent as might be expected for a subject which is experiencing rapid development. The team saw a draft version of an induction presentation that would be made to students entering the joint programme. The draft presentation emphasises that students should not approach the University with queries and problems about the programme, which should be channelled through NCC, via the delivery centre or regional moderator. The team considered that the presentation would be helpful in informing students about the programme and its expectations, and about the relative roles of the University, NCC, delivery centres and regional moderators.
40 The HKMA Course Handbook contains information on appeals procedures; the University has not yet had an appeal from an HKMA student. The Commentary stated that students had to appeal to NCC initially. If that is not satisfactorily resolved they can appeal to the University under the University's regulations, but 'normally at the expense of the student'. The matter of expense was described by the University as the cost for the student to travel to the UK. In Hong Kong, the audit team learnt that the appeal fee of £80.00 per unit would be charged by HKMA. The team was told by a member of NCC staff that initial appeals would be to HKMA, and that NCC would filter out appeals to see if they needed to proceed to the University, but the University subsequently explained that a student can take a case directly to the University if not satisfied with the action taken by the partner. As the University is responsible for the standards of awards, the team considered it was inappropriate for students to have to appeal via NCC on academic issues. It might be more logical if appeals were routed through the Course Liaison Officer initially, and for complaints about provision, rather than academic appeals, to be dealt with by HKMA and NCC. The University might also wish to review the policy of allowing its collaborative partners to charge fees for appeals.
Assurance of the standards of awards
41 In its Commentary, the University made it clear that 'it is responsible for the standard of the award'. Equivalence of standards was discussed in the Commentary, which stated that the collaborative programme was operated in accordance with the requirements of the University's Quality Assurance Handbook to 'ensure that such programmes are of an academic standard which is equivalent to comparable qualifications within the University'.
42 While it is clear from the Agreement that the responsibility for standards remains with the University as the awarding body, in the operation of the partnership it appeared to the audit team that there were several areas where, as the relationship has developed, it has not been easy to make the division of responsibility between the University and NCC as clear as this in practice. There are several examples of NCC taking a role in the oversight of academic standards. For example, the Commentary explained that NCC's procedures for the accreditation of its centres included 'academic and professional standards', and that the University had approved these procedures. The University subsequently explained that the accreditation procedures were jointly formulated, and that the University receives the report and 'makes the final decision on accreditation'. Another example relates to the regional moderator's involvement, stated in the Commentary, of 'ensuring standards are maintained, by classroom observation'. The team questioned whether it was appropriate to charge regional moderators with this responsibility, but the University later explained that classroom observation by regional moderators supplemented that carried out by University staff during monitoring visits. The University, bearing in mind its position as the degree awarding body, may wish to review the statements and views on the respective responsibilities of itself and NCC to ensure they still provide clear manageable guidelines, realistic in practice, for safeguarding the academic standards of the programme.
43 The Joint Examination Board considers the outcomes of the moderated assessment, and is responsible for ensuring that 'assessment is appropriate to the aims and objectives of the course' and for the classification and approval of awards. The Joint Examination Board includes NCC representatives. In view of the University's awarding body responsibility, the audit team did not understand the role of NCC staff as members of the Board, but it was explained to the team that the NCC representatives were there as observers. Nevertheless, the current membership of the Board might suggest that NCC staff are entitled to have an input to decisions about the grading of students and the conferment of awards. The University might wish to clarify the membership of the Board to ensure that the status of all participants is clearly stated.
44 The decisions of the Joint Examination Board are recorded by the University.
The audit team saw minutes of the Joint Board held on 24 October 2000,
whose purpose was to review students' performance at the end of academic
year 1999-2000. The minutes were accompanied by an assessment spreadsheet
for students at all delivery centres, and the degree classifications awarded.
The minutes recorded that 'examiners considered the results of students
at all the centres and made end of year decisions on performances'. The
minutes, which did not record the external examiner among those attending
the meeting (see above, paragraph 15), contained little detail of the matters
considered by the Board. The University will wish to ensure that the minutes
of the Joint Examination Board are kept in sufficient detail to provide
a full record of the Board's deliberations and decisions.
Entry requirements
45 Admission to the programme can only be achieved via the NCC IADCS (formerly the IHDCS). Management of the admissions process is the responsibility of NCC. HKMA has identified other potential students, with qualifications similar to the IADCS, who would be attracted to the final year of the Computing and Information Systems degree programme. The audit team was told that NCC is not, at present, considering opening-up the programme to students who have not followed the previous stages of the programme. It was explained to the team that the package is seen as a complete programme with continuity of learning between the different levels. Students are not registered with the University until they enrol for level 3.
46 Two cohorts of students are recruited each year, with starting dates
in January and July. All students are part-time. Most attempt to complete
the programme in one year - although they are permitted to take up to three
years. Some students expressed to the audit team that they felt 'under
pressure', particularly with regard to work on the individual project being
carried out alongside taught units in Decision Support Systems and Object
Oriented Analysis and Design.
Assessment of students
47 The assessment of students remains the responsibility of the University. Draft examination papers are produced by University staff and passed to the external examiner for comment. Detailed marking schemes are also produced by the University to assist the process of assessment in local centres. In its discussions at the University, the audit team was told that the University relied on NCC to manage the assessment process, and that this followed a well-laid out set of procedures. Examination papers are sent to the regional moderator who takes them to the examinations, and examination invigilators are hired and supervised by NCC. Adherence to this process was confirmed by the team during its visit to HKMA.
48 Examination scripts are marked by staff at the centre. Internal checking of assessed work for completeness and accuracy of assessment spreadsheets is undertaken at HKMA before marked scripts and their spreadsheets are sent directly, not via NCC, to the University for moderation. In the view of the audit team, the examination papers set by the University were generally appropriate to honours level and, in some cases, expected knowledge of relevant wider reading beyond the core teaching materials, although there was little specific requirement for comparison and evaluation.
49 The sample size for moderation normally varies from 35 per cent to 5 per cent according to the centre's past performance, although 100 per cent sampling was used for the first two examination cycles. The external examiner also contributes to the moderation process by sampling the moderated papers. The Commentary evaluated the University's experience of moderation, noting that up to 40 per cent of local marking could require blanket adjustment of marks, normally downwards, and that 10 per cent might have to be returned for re-marking. The external examiner's report of October 2000 for the outcomes of the joint programme across all centres commended the moderation process, and commented that University staff 'have clearly ensured that appropriate quality control is in place, although this is very time consuming'. The University recognises that it has a significant on-going task in the moderation of assessed work, particularly as the throughput is planned to increase in the future. The University indicated, in its discussions with the audit team, that the intensity of sampling could be reduced for selected centres where the experience of moderation was good. In view of the emphasis placed by the University on the careful moderation of assessed work as its principal control on academic standards, the team considered that the viability of any reduction in intensity of sampling would need to be assessed very carefully.
50 Assessment of projects is by two members of staff from HKMA. Assessment should follow an agreed marking scheme, but the report of the November 2000 monitoring visit noted that HKMA staff had not been using project assessment proformas. The University subsequently reported that the procedures were now working correctly. Project assessment sheets are sent by HKMA direct to the University, not via NCC, and a sample of projects, selected by the University on the basis of those assessment sheets, is forwarded by HKMA directly to the University for moderation. The report of the July 2000 periodic review of the joint programme noted that 60 per cent of the mark awarded is based on the final report and 40 per cent is awarded for the interim presentation and development work. The University is able to moderate the marking of the report, but is not able to moderate the remaining 40 per cent of marks awarded for this double unit. The course team agreed to consider the review panel's suggestions for viewing the interim presentation or workbooks in order to gain some oversight of this aspect of project assessment. The NCC Course Handbook that was seen by the audit team at HKMA informed students that the expected length of a project report was 5,000 to 8,000 words, although the Project Handbook for the joint programme suggested a report 8,000 to 10,000 words in length. The University will wish to ensure that information given to students about its expectations for the project report is consistent across all delivery centres, and in line with its expectations for projects undertaken by its own students.
51 The audit team noted variations in syllabus descriptions between the version that it saw of the NCC Course Handbook and that for the University-based level 3 course. It noted that in the syllabus statements for Software Engineering and for Advanced Database Management Systems, reference in the UK handbook to critical evaluation and to current research was lacking in the HKMA version. While the expectation of achievement that was made clear to University-based students was in keeping with the Subject Benchmark Statement and with the honours level qualification descriptor in QAA's Framework for higher education qualifications, it was less clear that the expectation for HKMA-based students fully matched those reference points.
52 The University claims that the comparability of academic standards in
this collaboration is safeguarded by the NCC level 3 course being a repackaged
version of one delivered in UK, with the assessment for the award being
under the control of the University. The audit team formed the view that
the adaptation of the University's level 3 units to produce the joint level
3 course with NCC represented more than a reorganisation of course content;
it reflected a change in the nature of the programme and a shift in emphasis
towards a more practically focused set of outcomes with less emphasis on
critical evaluation than would be normally expected at honours level. The
University may wish to reflect on whether the changes introduced in the
creation of the joint programme have influenced the learning opportunities
available to students who are striving to achieve the academic standards
of an honours award, and whether it can be fully confident about its statement
that the programme is equivalent to the programme delivered at the University.
External examiners
53 The University's normal procedure for allocating external examiners to collaborative provision is to draw on the external examiners for equivalent programmes in the UK. This procedure was followed initially for the joint programme in Computing and Information Systems with NCC. The University's Quality Assurance Handbook specifies that nomination of replacements is normally by the Head of Department in consultation with the Course Leader. The University's Commentary stated that the external examiners are nominated by the Degree Steering Group and appointed by the University. The audit team could not appreciate the reasons for the Degree Steering Committee's involvement in the appointment process since the nominations for the post(s) of external examiner were made by the Head of Department, and the University was required to ratify the appointment. The involvement of the Degree Steering Committee places NCC in the loop for the selection of external examiners. The University might wish to consider if this is an appropriate activity for the Degree Steering Group, given the University's sole responsibility for the security of academic standards.
54 A single external examiner was used in the early stages of the joint programme, but the University has decided that two external examiners should now be appointed because of the growth in student numbers. The external examiner has not been involved in a visit to HKMA, and staff at the centre have had no direct contact with this aspect of the assessment process. The students who met the audit team at HKMA were unaware of the role played by the external examiner in the moderation of grades and awards, and in the maintenance of academic standards, but the new induction presentation (see above, paragraph 39) should improve future students' awareness of these mechanisms for the control of standards.
55 External examiners are members of the Joint Examination Board, and are involved in the setting and moderation of examination papers and in the recommendations for the conferment of awards. They submit annual reports which are sent to and read by the University's Deputy Provost, the Head of Computing, Information Systems and Mathematics, and are referred to the Degree Steering Group for consideration and inclusion in the annual review process. The audit team formed the view that the external examining process satisfied the precepts of QAA's Code of practice, and that the University had an effective process for identifying points for action. The University might consider whether at least one visit to a delivery centre might not enable the external examiners better to appreciate cases for mitigation that might be brought before them. The opportunity for the teaching staff to engage with the external examiner could also present a staff development opportunity.
56 The BSc (Honours) in Computing and Information Systems is described by London Guildhall University as a joint programme operated by the University and NCC Education Ltd (NCC) leading to an external degree of the University. It is a variant of the level 3 course of the University's Computing and Information Systems programme forming the final year of part-time study of an honours degree programme, for which the entry requirement is the NCC International Advanced Diploma in Computer Studies. The University's collaborative partner in this arrangement is NCC, and the joint programme is delivered by NCC-accredited centres, one of which is operated by the Hong Kong Management Association (HKMA). The collaborative link between the University, NCC and HKMA was the subject of this audit.
57 The University prepared for this audit a Commentary on the collaborative link which provided a helpful starting point for the audit. The University's procedures for the consideration and approval of collaborative arrangements, which are set out in its Quality Assurance Handbook, meet the expectations of QAA's Code of practice on collaborative provision. The Commentary explained that, in many ways, arrangements for this link were fully representative of the way in which the University managed its collaborative provision, but that in other ways they were not representative in that the University's partnership was with NCC, not with the delivery centre. Responsibilities for different aspects of the link have been shared between the University and NCC.
58 In its Commentary, the University stated that its procedures for quality assurance ensured that its collaborative programmes 'are of an academic standard which is equivalent to comparable qualifications within the University'. The University's joint programme with NCC has a sufficiently different structure from the University's own level 3 course to indicate that the difference does not represent a simple re-organisation of the programme content. The findings of this audit suggest that the joint programme meets the needs of local students at HKMA, offering them good learning opportunities and support. The findings of the audit also suggest that aspects of the summative assessment of taught units, of expectations for student achievement in projects and project reports, and of student attainment in respect of the skills of analysis and critical evaluation that would be expected at level 3 combine to raise questions about whether the learning experience, and challenge, offered to students at HKMA fully satisfy the expectations that the University would have for students on its own programme leading to an honours award in the subject.
59 The University sets examination papers for the taught units of the joint programme, provides marking schemes against which staff of the delivery centre mark students' scripts, and moderates a sample of scripts. The sample size for moderation of scripts from HKMA was initially 100 per cent, and the University indicated that it would not reduce the intensity of sampling until its experience of moderation of scripts gave it confidence that such reduction would not put the reliability of assessment at risk. Projects are also marked locally before a sample is moderated by the University, although the interim local mark, which contributes to the summative assessment, cannot be moderated. The University puts considerable effort into moderation as a principal means of satisfying itself of the security of academic standards. There can be broad confidence that the assessment procedures are securely managed by the University in its position of the awarding body in this collaborative arrangement.
60 The University shares responsibility with NCC for the management of the quality and standards of the joint programme. In its Commentary on the partnership link the University explained that 'in broad terms the University is responsible for standards of the award and NCC is responsible for the quality of learning opportunities and student support'. In practice, this simple division of responsibility is not easy to maintain. The University undertakes direct annual monitoring through visits to delivery centres and periodic review of the joint programme as a whole. The findings of this audit suggest that the monitoring and review activities conducted by the University have been conducted with care, and have generated perceptive reports with specific recommendations. The process of monitoring and review, and identification of action upon certain recommendations, involves the University in a more direct engagement with the education provided by HKMA and other NCC-accredited delivery centres than would be envisaged by its stated responsibility for the standards of the award of its external degree. Similarly, NCC's stated responsibility for the quality of learning opportunities and student support offered by its accredited delivery centres cannot be isolated from the standards of awards since the educational process is a major factor in determining the standard of the experience achieved by students. There would seem to be potential for confusion about the division of responsibility between the University and NCC, and in the lines of communication between HKMA, NCC and the University. The findings of the audit suggest there would be merit in the University reflecting upon the terms of its collaborative agreement with NCC to ensure that the stated division of responsibility properly reflects actual operation and practice.
Comments*
Comments on the audit report as supplied by London Guildhall University
The University welcomes the opportunity to comment on the QAA's audit of the one-year level 3 study, offered in partnership with NCC Education Ltd and delivered by Hong Kong Management Association, which leads to the external London Guildhall University award of BSc (Honours) in Computing and Information Systems.
It should be noted the version of the course subject to QAA audit is in the process of being phased out and replaced with a version that incorporates modifications already made to the 'London version' of the course. These changes are in accordance with University plans that were already in agreement at the time of the review by the QAA in spring 2001.
The University is pleased that the report acknowledges that the issues identified by the auditors had already been identified by the University through its own quality assurance processes, and that appropriate actions had commenced. The University has full confidence in the robust mechanisms that it has in place to safeguard academic standards and provide high quality learning for students. The University welcomes the auditors' additional recommendations for consideration in the further enhancement and development of, what the University considers, a very successful partnership that meets the needs of students.
Top