Executive summary
The purpose of this report is to provide an overview of the findings arising from the review of research degree programmes (RDPs) in 2005-06. The findings provide a picture of overall confidence in the management of RDP programmes by higher education institutions (HEIs) in England and Northern Ireland. The findings also show examples of good practice and identify areas for improvement in the institutional management of the quality and standards of RDPs.
In all cases the review teams found that, overall, the procedures in place to secure and enhance the quality and standards of RDPs were appropriate and satisfactory. This is not to say that the review teams found every aspect to be appropriate and satisfactory, but that, on balance, any concerns identified were not sufficient to threaten the overall management of the quality and standards of RDPs.
There was widespread evidence that HEIs had approached the Code of practice for the assurance of academic quality and standards in higher education (Code of practice), Section 1: Postgraduate research programmes, in a positive and constructive manner. Feedback from institutions suggested that the review gave institutions the opportunity to reflect upon their policies and procedures and where necessary revise them to meet the expectation of the Code of practice.
The overall conclusions can be found in paragraphs 115 to124.
The purpose of the review of research degree programmes
1 The main purpose of the review was to ensure that all HEIs receiving funding for RDPs, from the Higher Education Funding Council for England (HEFCE) and the Department for Employment and Learning Northern Ireland (DEL), have policies and procedures in place that are robust and effective in securing and enhancing the quality and standards of RDPs. The review was announced in the HEFCE publication Postgraduate research degree programmes: minimum standards and funding (HEFCE 18/2004) as part of a broader strategy for securing the quality of RDPs.
2 In September 2004, QAA published a revised Code of practice, Section 1: Postgraduate research programmes (the precepts of the Code of practice are included as Appendix C). The review of RDPs was intended as a means to gauge the extent to which the policies and procedures of HEIs are in alignment with this section of the Code.
3 The review applied to all HEIs in England and Northern Ireland in receipt of funding for RDPs from either HEFCE or DEL (for a list of the 114 participating institutions see Appendix D). Within institutions the review included all RDP students registered for a PhD (including the New Route PhD and PhDs awarded on the basis of published work), and all forms of taught or professional doctorate and research master's degrees where the research component (including a requirement to produce original work), is larger than the taught component when measured by student effort.
4 Those HEIs which are not in receipt of funding for RDPs were not obliged to participate in the review, but had the option to do so.
5 The review of RDPs was a one-off exercise. All further consideration of the quality of RDPs will be part of the revised institutional audit process which started in 2006-07. The reports produced as a result of the review will form part of the evidence base for institutional audit.
6 A summary of the review methodology is in Appendix A. An evaluation of the review method was undertaken by QAA and the findings have been summarised in Appendix B.
7 An identical review process was conducted in Wales. For further details see the QAA website (www.qaa.ac.uk/reviews/postgraduate).
Learning from the review of research degree programmes
Institutional arrangements
8 This section considers the arrangements that institutions have in place to safeguard the academic standards of their RDPs and enable these to be delivered successfully according to national and, where relevant, international expectations. The review teams found that almost all institutions had appropriate and satisfactory arrangements in place to align with precepts 1 to 4 of the Code of practice.
9 In the Code of practice, Precept 1 states that institutions are expected to put in place effective arrangements to maintain appropriate academic standards and enhance the quality of RDPs. In their response to the questionnaire, institutions described their deliberative and administrative structures with respect to research programmes, explaining how these provided institutional oversight of such programmes. A number of institutions register research students in one or more graduate schools, some of which have dedicated facilities to encourage interaction among researchers. There was an expected and, in general, acceptable degree of variation in the extent to which research degree oversight and administration is centralised or devolved to schools or faculties, with more devolution usual in institutions with large numbers of research students. However, in a small number of cases review teams indicated that institutions should consider means of ensuring clarity and consistency of approach across schools and faculties.
10 In the Code of practice, Precept 2 states that institutional regulations for postgraduate research students will be clear and readily available to students and staff and, where appropriate, supplemented by similarly accessible guidance at faculty, school or department level. Review teams found this to be the case in almost all institutions with many indicating that their regulations had been reviewed recently to ensure that they were in line with The framework for higher education qualifications in Engalnd, Wales and Northern Ireland (FHEQ) and with the revised Code of practice. Exceptions tended to occur as a result of lack of clarity with respect to specific issues: for example, the appointment of examiners and time guidelines for complaints and appeals. These are dealt with in more detail elsewhere in this report.
11 A significant number of institutions make provision for students to study for research awards at sites other than their own campus. For example, part-time students may be based in a professional or industrial post with much of their research forming a part of their normal working role. In general, such students are subject to the same regulations as those who are full-time and campus based with, in some cases, an additional requirement that a minimum amount of time be spent on the university campus. In other cases, students pursue their research at a partner institution: an HEI or a research institute that does not have its own degree awarding powers. Again review teams noted that common regulations were generally in place with provision in some cases for minor variation to meet local circumstances.
12 Key sections of regulations frequently form a part of an institutional code of practice as required by Precept 3 of the Code of practice, published by QAA. Institutions have addressed this precept in a variety of ways: mapping their regulations against the Code; providing separate staff and student handbooks. Almost all indicated that they had reviewed practice to ensure alignment with the revised section of the Code.
13 Precept 4 is concerned with institutional level oversight of RDPs in terms of monitoring and benchmarking success against appropriate indicators and targets. Most institutions receive information about progression via the annual monitoring process, including the analysis of external examiners' reports, but few described the use of targets or benchmarks. Where these were referred to, the most common choice was research council guidelines or national averages for completion rates.
14 With respect to institutional arrangements, review teams identified a number of points of good practice in the production, updating and use of a code of practice for RDP students. In some institutions the review teams noted the clarity and comprehensive nature of their codes, including the availability of different versions for students and members of the supervisory team and in others, the mechanisms in place to review internal codes and processes against the revised Code of practice.
15 Further good practice was noted in the arrangements for validating RDPs and the relationships between the awarding body and validated institution and in the formation of groups to support administrative staff concerned with RDPs and encouraging the dissemination of good practice among such staff.
16 Review teams identified two areas to which a number of institutions needed to give further consideration. The most common of these was the need to ensure that regulations were explicit, clearly stated and applied consistently across the institution. The second was the need to record and monitor success rates for RDPs and, in particular, to benchmark these against internal and external targets.
The research environment
17 This section considers the ways in which institutions ensure that RDP students are provided with an appropriate environment for their research. The review covered a range of institutions, from those which are clearly research intensive and research led to those where research plays a relatively small part in overall activity. However, review teams found almost all institutions had appropriate and satisfactory arrangements in place with respect to Precept 5.
18 Almost half of institutions used their performance in the 2001 Research Assessment Exercise to demonstrate that some or all of their research activity was of a high standard. In some cases RDPs were only approved in areas of activity rated at, or above, a specific level in the exercise; in others broader measures of research excellence, including knowledge transfer activity, were used. External approval of the suitability of the research environment in a wider sense was offered by some institutions, generally in the form of recognition for research student funding by one or more of the research councils.
19 In their response to the questionnaire a number of institutions indicated that they restrict research student admission to a limited number of areas of activity as indicated in the preceding paragraphs and/or include a check on resource and supervisor availability as part of the admissions process. Many institutions also described information technology (IT) facilities and study/office space dedicated to research students. In some cases less research intensive institutions described mechanisms for using electronic access or sharing resources with others to ensure research students had access to adequate book and journal stock. Many specialist institutions such as art and music colleges also made reference to the availability of specialised collections to their research students. In general, a range of mechanisms for bringing research students together with their peers was described. Graduate schools offering social space such as common rooms often play a part in this, as do research seminars and internal research conferences aimed specifically at RDP students.
20 With respect to Precept 5 of the Code of practice, review teams identified a number of points of good practice. The review teams noted several institutions were sharing resource with respect to postgraduate skills training and other facilities. One has written requirements for the equivalence of the research environment of students not based on campus and monitors adherence to these requirements on an annual basis. Another institution was introducing a well considered set of requirements to ensure the presence of a critical mass of students and a suitable research environment despite having a small RDP provision.
21 Review teams identified areas for further consideration in some cases. These included: the need to ensure research students were more fully integrated with the overall research environment of the institution; the adequacy of the research environment for international distance learning students and the appointment of local supervisors for such students; and the need to make available a base for research students from which to work.
Selection, admission and induction
22 Precepts 6 to 10 are concerned with ensuring that institutions have clear admissions and induction procedures and requirements for their RDPs, and that their policies in this area are applied consistently and fairly. While review teams found that most institutions had appropriate and satisfactory arrangements in place with respect to these precepts and many demonstrated examples of good practice, more than 20 per cent were asked to give further consideration to certain aspects of their selection, admissions and induction processes.
23 Precept 6 states that 'admissions requirements will be clear, consistently applied and will demonstrate equality of opportunity'. Review teams noted that most institutions had clearly documented regulations and guidelines about admission to their RDPs and that this information was readily available to potential students. In many cases, the material was available electronically as well as in hard copy. Some institutional questionnaire responses indicated that special arrangements are made to ensure information is available to potential students with disabilities. These included contact with the institution's disability support officers and the provision of admissions material in the form of media appropriate to candidates with disabilities.
24 Entry qualifications for RDP students are covered by Precept 7. Almost all HEIs require research students to offer at least an Upper Second class honours degree in an appropriate subject and, in some cases, the requirement is set at master's level for direct entry to a PhD. In some cases, institutions indicated that, often in respect of their widening participation policies, they had arrangements in place to admit students who did not meet the normal criteria, but who offered significant research or professional experience. Many institutions also indicated that they have particular English language requirements for international students.
25 Precept 8 expects admissions decisions to involve at least two members of staff who have received instruction, advice and guidance in respect of the institution's selection and admissions procedures. Consideration should be given to the place of interviews in the process including arrangements for those based overseas or working at a distance. Institutions described a variety of practice in their response to the questionnaire. Many interviewed all potential RDP students who were based in the UK; in a few cases, overseas students were also interviewed. Telephone and video link interviews, or email dialogue, were cited as being used in some cases where a visit was not practicable, for example where the potential student was overseas.
26 Some institutions make use of an admissions panel including at least one senior member of staff such as a head of department, research degrees coordinator or graduate school director to promote consistency while others use potential supervisors. In some cases specific training, including briefing on issues of equality and diversity, is offered, either as part of more generic supervisor induction and training or as a separate activity. A number of institutions indicated that, although training for RDP admissions was not mandatory at present it was planned or under consideration.
27 Precept 9 requires that the entitlements and responsibilities of a research student undertaking a postgraduate research programme be defined and communicated clearly. Most institutions indicated that successful applicants received a formal letter of acceptance, accompanied by other material such as research student handbooks, and that further information was provided at induction. There is a variety of practice as to the source of acceptance letters: in some cases these are handled centrally; in others at school or faculty level. Several institutions have a policy of offering feedback to unsuccessful applicants.
28 Precept 10 states that once a student has been selected for an RDP the institution should provide them with sufficient information to enable them to begin their studies with an understanding of the academic and social environment in which they will be working. Review teams found a variety of practice in such induction procedures depending on the nature of the institution and the number of RDP students. Institutions with large numbers of students starting their research programmes in any one year were most likely to offer a mix of university-wide and faculty or school-specific events, sometimes running programmes more than once a year. Where new student numbers were small, induction was more often informal in nature.
29 Graduate schools often play a role in bringing new students together to provide information on regulations, student entitlements and student responsibilities in addition to social events which offer opportunities for students to meet each other. In other instances, induction forms part of the research student training programme. Some institutions indicated that they made specific efforts to ensure induction material was available to part-time and/or distance learning students by issuing it in hard copy or electronic form and in a few cases enrolment was limited to one or two intakes per year to ensure that appropriate induction could be provided to new research students as a cohort when they first joined the institution.
30 Review teams identified a number of features of good practice with respect to admissions, selection and induction. At three institutions it was noted that particular care was taken with respect to ensuring equal opportunities and ensuring merit, capability and diversity in the selection process. At two institutions the review teams noted good practice with respect to procedures to advise students who could not be offered a place on an RDP. Further examples were: the timeliness and transparency of the selection process at one institution; the guidance given to candidates on the formulation of the research proposal as part of the application form; the making of a clear statement to applicants that admission would, in part, depend on fitting with the institution's expertise and the requirement that the candidate outline how they would expect to benefit from the institution's resources; and the use of an interview grid to ensure consistency between applicants and induction arrangements.
31 A number of areas were identified for further consideration. The two most common were a lack of clarity about the level of entry qualification set by the institution for admission onto RDPs and the need to ensure that at least two staff who had received appropriate instruction, advice and guidance on the institution's policies and procedures, were involved in admissions decisions about RDP candidates. Other points for consideration included the need to ensure greater clarity and consistency for admissions, the place of interviews in the admissions process, more formalised induction and better monitoring of school level induction processes.
Supervision
32 This section deals with the policies and procedures that institutions have in place for the supervision of research students. The review teams found that in most institutions arrangements for supervision are appropriate and satisfactory, and aligned with precepts 11 to14 of the Code of practice.
33 In the Code of practice, Precept 11 states that institutions are expected to appoint supervisors who have the appropriate skills and subject knowledge to support, encourage and monitor research students effectively. Most institutions made explicit (for example, in their Code of Practice for Research Students) their expectations and requirements, particularly for principal supervisors. This typically includes being research active, holding a doctorate (although a number of institutions have special arrangements for supervisors who do not), and experience of having supervised at least two research students to successful completion. Most institutions expect their supervisors to be full-time permanent members of academic staff, and a few explicitly state this as a requirement.
34 The review teams found some variability between institutions in the training and development of supervisors. Mostly, new supervisors are required or expected (as a condition of probation) to undertake a formal programme of induction and training. Some institutions also provide mentors for new supervisors, although very few mentioned how long this was for (typically during probation). In a handful of institutions new staff are explicitly expected to continue to take opportunities to update their skills and knowledge of supervision.
35 Variability of expectations and requirements was found to be greatest in relation to the training and development of established supervisors and was one of the most frequently mentioned themes in the reports. In many institutions there is still work to be done engaging many established supervisors in supervision development programmes. In most institutions, established supervisors are encouraged or expected, but rarely required to engage in personal development activities in this area. In a few institutions, they are required to take an appropriate training course at least every two years as a condition of being allowed to continue to serve as a supervisor.
36 One institution offers a dedicated Postgraduate Certificate (PgCert) in Research Degree Supervision, and a number of others include supervision within broader PgCerts in Teaching and Learning or, in one case, in a Professional Doctorate in Academic Practice. New and established supervisors in many institutions also have access to training offered by staff development units. Review teams noted two interesting ways of incentivising supervisor development. One institution runs an annual competition for 'best' student/supervisor relationship, based on nominations, and one has introduced a Vice-Chancellor's Award for Excellence in Doctoral Supervision. The review teams considered both these initiatives to be features of good practice.
37 Precept 12 states institutions are expected to provide each research student with at least one main supervisor, and one clearly identified point of contact. Most institutions now have supervisory teams, generally comprising two or three people, including a designated principal supervisor (who is sometimes referred to as the Director of Studies). Some institutions explicitly allow one external supervisor who can bring specialised knowledge and expertise. Many have formalised arrangements for when supervisors are away on leave, absent for prolonged periods, or leave the institution.
38 Review teams found fairly wide variability in the guidance given to supervisors for their role. Providing academic support and guidance and formally reviewing student progress are usually implicit roles, and responsibility for keeping a record of supervisory sessions was mentioned in a small number of cases. Few institutions specify a minimum frequency with which students and supervisors should meet; at least every four to six weeks is typical of the few that do so. Similarly, few mention what, if any, special arrangements are in place for supervising part-time or distant research students.
39 Precept 13 states institutions are expected to ensure that the responsibilities of all research student supervisors are clearly communicated to supervisors and students through written guidance. Review teams reported that most institutions make this information available in a variety of ways, including through institutional codes of practice, student handbooks and guides for research degree supervision.
40 Precept 14 expects institutions to ensure that the quality of supervision is not put at risk as a result of an excessive volume and range of responsibilities assigned to individual supervisors. Here again review teams found variability between institutions, particularly in terms of workload allocation models. Institutions often take supervisory loads into account, either formally or informally, but there is no generally agreed load for a typical supervisor. Some institutions allow supervisors 40 hours a year for each full-time student and 20 hours a year for each part-time student, but several note that this is difficult to apply in practice. Review teams also found variations in institutional maxima on the number of research students a principal supervisor is allowed to supervise. Many institutions state that a supervisor is not allowed to supervise more than a stated 'normal maximum' without specific approval or permission. A typical 'normal maximum' is six for principal supervisor and up to 12 as a member of a supervisory team, but it can often be higher for a supervisor with little other 'teaching' or on a research contract.
41 The review teams found examples of good practice in supervision the appointment of supervisors. In some institutions they maintain a Register of Approved Supervisors. In many institutions documentation is provided setting out the supervisors roles and responsibilities in relation to supervision practice. A small number of institutions have a formal learning agreement between student and supervisors that clearly spells out the expectations and responsibilities of each.
42 In supervisor training and development, review teams found numerous examples of good practice. Many institutions now have formal arrangements for the training of new and established supervisors, a small handful require all supervisors to be trained, some have accredited training programmes for new supervisors, and a small number have regular events (including update seminars, master classes, and annual Supervisors' Conferences or away-days) for established supervisors. One institution even includes good supervisory practice as a key criterion for promotion to Readership and Professor.
43 Good practice was also found in the monitoring and review of supervision. Examples include the annual progress review being conducted by two independent assessors excluding the supervisor, and termly formal reporting on supervision to the institution. A particularly interesting example of good practice, found in two institutions, is the formal review of supervisory practice and the supervisory team when a thesis is failed or requires major revisions.
44 Review teams identified two main areas for further consideration, which were widely distributed between institutions. Most commonly mentioned was the need to enhance the monitoring and management of supervisor workloads, in terms of the number of postgraduate research students that they are allowed to supervise (as primary supervisor or member of supervisory team), and/or defining a minimum number of supervisory meetings or contact hours per term. The second most common theme was the need to ensure that appropriate formalised training and development is not only available to, but taken advantage of, by all supervisors, new and experienced. Within this, two issues were mentioned most frequently: the need to find ways of engaging all experienced supervisors in appropriate personal development; and the need for greater consistency of provision and expectations across all schools and faculties.
45 A third area for further consideration, more varied in expression and mentioned by review teams in fewer institutions, is the formalisation of supervisory team arrangements and feedback mechanisms on supervisor performance. Comments ranged from the need to introduce supervisory teams, the need for more systematic supervision arrangements and clearer definition of roles and responsibilities, through practical matters such as keeping records of outcomes of supervisory meetings, to the need for more effective ways of sharing good practice among supervisors.
Progress and review arrangements
46 This section deals with the policies and procedures that institutions have in place for monitoring and supporting student progress. The review teams found that most institutional arrangements for monitoring and supporting the progress of research students are appropriate and satisfactory and aligned with precepts 15 to 17 of the Code of practice.
47 Precept 15 states that institutions should put in place and bring to the attention of students and relevant staff clearly defined mechanisms for monitoring and supporting student progress. The Code of practice noted the main purpose of the monitoring process is to provide overall support for the student to complete the research programme within an appropriate timescale. The purpose and frequency of monitoring arrangements need to be made clear from the outset. Should a student's progress be unsatisfactory, the monitoring process should include ensuring that support is available for the student to make improvements. In respect of the process of institutional review of monitoring arrangements, good practice was identified in three institutions, noting especially the integration of the review of RDP provision into the general annual and periodic review processes.
48 Precept 16 states that institutions are expected to put in place and bring to the attention of students and relevant staff clearly defined mechanisms for formal review of student progress including explicit review stages. The Code of practice notes the desirability of establishing procedures for reviewing student progress that involve individuals independent of the supervisor and student, suggesting the possibility of an annual review by a panel at which the student would be present. The review teams noted that annual reviews of some description were now virtually universal in the institutions participating in the review. The main feature requiring attention is the lack of consistency between departments or faculties within a single institution in the conduct of annual reviews.
49 Mostly, institutions require students to be initially registered for the award of MPhil or other research master's qualification prior to upgrading to PhD. The upgrading process, which may take place at the end of the first (normally) or second years, is highly formalised, requiring the provision of substantial progress reports by student and supervisor(s), and the involvement of a panel empowered to make recommendations to an institutional or faculty research committee.
50 Many institutions also require a formal report to be submitted by the student near the mid-point of the first year for the department to assess his/her progress. Others differentiate between 'enrolment' and 'registration', with transfer being assessed after a period of six to nine months.
51 The arrangements for monitoring of student progress and the manner of annual reporting of RDPs was identified as a feature of good practice in a small number of institutions. The feature which struck the review teams most forcefully was those cases where monitoring reports included sections reporting on the integration of generic skills and academic progress. The use of a panel for annual review that excludes the supervisor was also considered good practice by the review teams. Conversely, some institutions are asked to consider introducing a larger element of independence into their procedures for annual monitoring. A related item of good practice was the use of a third party monitoring system which enables students to discuss their research degree studies with a third party who is neither their supervisor nor their head of discipline. The integration of skills development with academic progress in annual review was also considered as good practice by the review teams.
52 Precept 17 states that institutions are expected to provide guidance to students, supervisors and others involved in progress monitoring and review about the importance of keeping appropriate records of the outcomes of meetings and related activities. The Code of practice notes that the kind of record that might be kept of regular, informal meetings between student and supervisor might be different from and less detailed than that of a formal meeting such as annual review.
53 The majority of institutions require or encourage students and supervisors to keep records of their meetings. Some institutions have moved, or are moving, to electronic recording of meetings in logbook form, while others use a standard pro forma approach. The keeping of a logbook of meetings in some form by the student is becoming well established. At others, the recording of meetings is seen by review teams as needing to be more systematic or to be enhanced.
Development of research and other skills
54 This section deals with the way in which institutions ensure that appropriate development opportunities are available to their research students. While the review teams found quite wide variations in practice between institutions, they found that in most cases the arrangements for the development of research and other skills are appropriate and satisfactory, and aligned with precepts 18 to 20 of the Code of practice.
55 Precept 18 states that institutions are expected to provide research students with appropriate opportunities for personal and professional development. Most institutions now have formal research training programmes that are informed by the Research Councils UK (RCUK) Joint Skills Statement, financed at least in part by the RCUK Career Development and Transferable Skills Training Payments (sometimes referred to as Roberts Money), and combine both institutional and faculty/school-based provision. The review teams found that, in a handful of institutions, skills training was still being planned but not fully in place at the time of the RDP review. They also found that in most institutions, provision for the development of research skills is better developed than that for generic skills, and uptake by students follows the same trend.
56 The structure, content and processes of research training were found to vary greatly from one institution to another. In a relatively small number of institutions, students are required to take a prescribed number, or credit weight, of courses and activities, although it is not always clear whether attendance, participation or completion are required or expected. More commonly, institutions make available a wide variety of training opportunities (formal and informal, assessed and non-assessed) from which their students are expected to select an appropriate set, guided by their supervisors. Most institutions make explicit their expectations about research training in their postgraduate regulations, code of practice and in student handbooks.
57 Practices vary in some important areas. For example, not all institutions have formal procedures for monitoring student uptake of, and participation in, research training. Many institutions do not explicitly define the amount of training they expect their students to engage in; some expect (and fewer require) the 10 days training per year that the research councils regard as appropriate, and some institutions expect this only of their Research Council funded students, whereas others expect it of all full-time research students. A relatively small number of institutions require the student to pass all or a prescribed amount (usually defined by weight of credit) of the research training element before they are allowed to progress, submit or receive their degree. Only a handful of institutions explicitly state that they grant accreditation of prior learning of skills development for mature students. A small number of institutions award a PgCert in Research Training (or similar) to students who have successfully completed a defined training programme.
58 Most institutions publish handbooks of training courses that are available to their students, and this information is often made available on intranets and virtual learning environments (VLEs). A fairly large number of institutions are developing materials for delivery by e-learning and through VLEs. Some institutions make special arrangements for the delivery of training opportunities to distant (off-campus) and part-time research students for example, by running workshops at weekends and in the evenings, and via residential summer schools or training weeks.
59 Review teams also found diversity of practice relating to the training of research students who teach. In some institutions such training is required before the student is allowed to teach, whereas in others the training is encouraged but not required.
60 Most institutions have appropriate training and skills development opportunities available on-site, many also encourage their students to take advantage of opportunities outside or, in the case of associated colleges, at the parent institution, and some have established collaborative arrangements for the sharing of courses and resources. Many institutions also make good use of the UK GRAD national and regional graduate schools.
61 Many institutions have in recent years reviewed their research and skills training, against the revised Code of practice and the RCUK Joint Skills Statement. Institutions are spread out along a spectrum in terms of fully meeting the external expectations; some have much work still in progress, while most now have well-developed skills development and research training programmes, which operate at both institutional and faculty levels. Many institutions have appointed, or plan to appoint, a Research Training Programme Manager or Coordinator (usually funded by Roberts Money) to deliver appropriate training centrally and to coordinate training in faculties and schools.
62 Precept 19 says institutions are expected to have effective procedures for the initial identification and regular review of students' development needs, agreed jointly by the student and appropriate academic staff. The review teams found that many institutions now have formal training needs analysis (TNA) processes, and most of the rest have informal processes. Some of the latter are piloting more formalised processes.
63 While all institutions have procedures for the identification and review of training and development needs, and most claimed that the procedures are informed by the RCUK Joint Skills Statement, the review teams found considerable variations in practices and procedures. Some TNA systems are web based and others are paper based. Some are compulsory while others are recommended or encouraged. Systems in most institutions are based on initial TNA soon after registration, with annual reviews subsequently; a few institutions expect reviews every six months. Many institutions expect TNA evidence to be included in the annual progress review process.
64 Precept 20 states that institutions are also expected to provide opportunities for students to record personal progress and to reflect on the potential application of the skills they have acquired. All institutions provide such opportunities, but here again review teams found wide variability in practices and procedures and there is no consensus about the most appropriate or effective approach to adopt. A number of institutions had recently introduced or were running pilot projects on new procedures at the time of the RDP review.
65 Many institutions have introduced personal development planning (PDP) tools (sometimes called training logs, diaries or portfolios, or skills records) over the last year or so; some are online and others paper-based. Typically, the student is expected (in some institutions required) to use these tools to record and reflect on their personal progress and skills development, on a confidential basis, and to discuss these with their supervisors. Most institutions expect students to evidence this record and reflection in their annual monitoring and progress assessment processes; one or two require a statement of training undertaken to be included with the submitted thesis. A few institutions have plans to integrate records of skills training into the registry information system, to facilitate the creation of transcripts of skills training as records of achievement.
66 Review teams found that most institutions had only recently enhanced their programmes of research training and skills development, and had introduced PDP and TNA systems, in response to the revised Code of practice. As a result, it had often not been possible for an institution to review the effectiveness of its new arrangements as part of the RDP review, and review teams sometimes commented on the need for an institution to ensure it put appropriate review mechanisms in place.
67 Good practice in many institutions reflected positive engagement with the skills agenda and was evidenced in the provision of cohesive, high quality, flexible and accessible programmes of research training, which often blended central and faculty or school-based courses and other activities, and recognised both formal and informal activities. Fairly widespread good practice was also identified in the ways in which institutions made special arrangements to enable part-time, distant and overseas students to access appropriate training in research and other skills for example, by delivery at weekends and residential weeks. Formal accreditation of training programmes; for example, in the form of a PgCert in Research Degree Supervision, or a Key Skills Award for Researchers, was a feature of good practice in a handful of institutions.
68 A range of good practice was identified in the delivery of skills training, such as the development of particular materials; for example, training videos and web-based support materials, evaluation of existing provision, and the appointment of specialist training staff. The review teams found many examples of good practice in the provision of information to students on what training opportunities were available, within and beyond the institution, and on what is expected or required of students, both at induction and subsequently, in both printed and online formats, for example, through induction materials, Research Student Handbooks, and the institutional code of practice.
69 Encouragement to take advantage of training opportunities beyond the institution, including collaborative schemes, often with financial support to do so, was noted as good practice in a number of institutions. Good practice was also highlighted in the provision of multiple sources of advice and guidance (including personal tutors, other members of the supervisory team, research student networks and graduate schools).
70 Other examples of good practice include: opportunities in some institutions for students to make presentations about their work and to receive feedback on it, for example, through an annual research student conference or poster session; and the Researchers in Residence scheme coordinated by one institution, which arranges short-term placement programmes for researchers in secondary schools to help them develop a range of transferable skills.
71 The second main area of good practice in the development of research and other skills relates to institutional PDP systems to support and record personal development. Review teams found many examples of good practice as institutions develop new processes and procedures. Approaches to personal development planning, procedures (including the introduction of learning contracts, and formal PDP systems and research student logbooks, in both electronic and paper formats) to assist that, the embedding of PDP in regulatory frameworks, use of PDP evidence in progression reviews, and regular monitoring to ensure that processes were operating effectively, were highlighted as good practice in different institutions.
72 TNA is the third main area of good practice in the development of research and other skills, although here review teams found fewer examples, mainly because most institutions are still developing their procedures and practices in this area. The formal assessment of research student training needs, the accreditation of prior learning, and the ways in which TNA is used to monitor the acquisition of skills, were regarded as good practice in a few institutions.
73 The review teams identified a large number and wide range of areas of further consideration in respect of the development of research and other skills, relating particularly to the development and delivery of training programmes, and personal development planning. While most institutions have now formalised research training programmes, the review teams concluded that many of them could and should be further enhanced to make them more fit for purpose. This applies to the general approach to the development, delivery and monitoring of skills development and provision of training opportunities to research students, in a number of large and small institutions. Specific areas for improvement, scattered between many institutions, include better embedding of skills development within RDPs, developing more formal and/or compulsory programmes of skills development, better alignment of programmes with the requirements of the RCUK Joint Skills Statement, and making sure that programmes are appropriate for and accessible to particular groups of students, such as part-time and international distance learning research students.
74 Many institutions were encouraged to give further consideration to the introduction or further development of suitable systems of PDP for research students, particularly in terms of monitoring and review of procedures, integrating evidence from PDPs into the annual progress review process, and ensuring that similar systems are available to all students.
75 Other areas for further consideration, mentioned much less frequently, include the use of induction sessions to raise awareness of institutional requirements or expectations relating to research training, the provision of appropriate information about research training to students, and the provision of appropriate training for research students involved in teaching.
Feedback mechanisms
76 This section deals with the policies and procedures that institutions have in place to secure formal feedback from research students and is covered by Precept 21. The review teams found that while all institutions had established some mechanisms for securing formal feedback in addition to feedback received informally, there was much variation for receiving feedback between institutions, and in the formality and efficacy of these mechanisms. Recommendations by the review teams aimed at considering their improvement were directed at some 25 per cent of the HEIs involved in the review, a higher fraction than occurred in any other area of investigation.
77 The Code of practice, Precept 21, sets out the expectations on institutions to put in place mechanisms to collect, review and, where appropriate, respond to feedback from all concerned with postgraduate research programmes. They will make arrangements for feedback to be considered openly and constructively and for the results to be communicated appropriately.
78 The review teams found evidence of utilisation of a wide variety of means of obtaining student feedback. Most institutions now have a section on the annual student progress form for comment by the student on the supervisory arrangements and infrastructural support (particularly library resources, IT, skills training). Annual or exit questionnaires are used widely, sometimes of a general character but often directed towards some particular element of the provision. Student opinion is routinely sought at periodic review or in departmental annual review. Student feedback at departmental level is often gathered through postgraduate staff-student liaison committees (which may also include taught postgraduate students) and at faculty/institutional level by student membership of committees responsible for postgraduate affairs and/or research matters generally. Some institutions employ student forums, postgraduate societies, or research student committees as opportunities to gain student feedback. Mostly, institutions have some process for collating and analysing reports from external examiners, and some proformas seek an opinion on the institutional processes as well as the candidate's performance.
79 Points of good practice on the eliciting of research student feedback were identified in respect of: the use of on-line processes for discussion fora and student feedback; an annual anonymous student satisfaction survey; the existence of a Faculty or institutional Research Students' Committee focusing on student issues; the opportunity for research students to provide feedback directly to the Board of Graduate Studies as part of the annual review process; the inclusion of formal reports from the student representatives on departmental research committees; the detailed nature of local feedback provision; the communication of student evaluation and university response via the regular newsletters; the mechanism to obtain feedback from recent RDP graduates; and the use of school employers' fora as a vehicle for providing feedback on RDPs.
80 Areas for further consideration included asking a large number of institutions to consider their arrangements for securing feedback from stakeholders other than current research students, such as employers, sponsors, supervisors and recent graduates. There were also a large number of instances which drew suggestions for enhancing the mechanisms for obtaining student feedback. These fell into two main groups: the absence at a number of institutions of an appropriate level of student representation on senior institutional committees dealing with research degree matters (research committees, research degree committees, and graduate schools); and the lack of staff-student liaison committees (SSLCs) dedicated to research students, or even postgraduates as a whole, at departmental level.
81 While a number of institutions had a research student representative on a school or departmental SSLC, most students in such joint committees were undergraduates and the meeting therefore dominated by undergraduate concerns. This point has been recognised by one institution and is being acted upon. The review teams accept that in institutions with relatively small numbers of research students, the scope or necessity of multiple means of student representation may be limited, but noted that some institutions with large numbers of RDP students had no departmental SSLCs dedicated to research students.
82 Some institutions were asked to consider reviewing the effectiveness of research student representation in more general terms, others were asked to consider strengthening their mechanisms and others to widening the range of mechanisms. Some institutions have found that the return rate of student questionnaires is disappointingly low and are exploring the use of focus groups as one means of obtaining better feedback; others have found attendance by students' representatives can be sporadic.
83 Other areas for consideration included the need to obtain data from institutional questionnaires that relate more specifically to research students; elsewhere there has been a recognition by the institution that the library needs of RDP students were being inadequate resourced. These examples, with the representational issues mentioned above, indicate a certain tendency for research students to be overlooked in certain regards in some institutions with relatively small numbers of research students but very large undergraduate populations.
Assessment
84 This section deals with policies and procedures that institutions have in place for the assessment and examination of research students. The review teams found that, in most institutions, the arrangements for assessment and examination are appropriate and satisfactory and aligned with precepts 22 to 24 of the Code of practice.
85 Precept 22 states that institutions are expected to use criteria for assessing research degrees that enable them to define the academic standards of different degree programmes and the achievements of their graduates. While most research postgraduate students are registered for PhD, often after an initial period of registration for MPhil or other research master's qualifications, there are an increasing number of other research awards such as professional doctorates, MPhil, MRes and Doctorates through Published Work, which have different criteria and which are structured differently. The criteria used by institutions to assess these various research degrees must be clear and readily available to students, staff and external examiners.
86 The review teams found that in setting criteria for different types of research programmes, institutions usually referred to the FHEQ as their starting point for developing policy. Institutional criteria are published in institutional codes of practice, research student handbooks, research handbooks and similar publications as well as being available on the local website. Typical of the criteria applied for the award of PhD are statements such as 'the candidate must show convincing evidence of his/her capacity to pursue research and scholarship and make an original contribution and substantial addition to knowledge'.
87 The topic of criteria, rather than that of examination procedure, drew limited comment, either positive or negative, from review teams, signifying an overall satisfaction. In one case, the review team noted the clarity with which levels of attainment are set out for students, while another operated an initial Period of Study followed by an assessment of the candidate's suitability to proceed. On the other hand, one institution was asked to consider including more specific assessment criteria in its Regulations for individual degrees, while in another the institution was encouraged to implement the recommendations of its Research Board that proposed introducing arrangements for the assessment of students more closely aligned with the Code of practice, published by QAA.
88 Precept 23 indicates that institutions' research degree procedures must be clear; operated rigorously, fairly and consistently; include input from an external examiner; and be carried out on a reasonable timescale. One area of good practice highlighted in the Code of practice is the point that 'the institution will want to consider carefully including the appointment of an independent non-examining chair to help ensure consistency between different vivas and in providing an additional viewpoint if the conduct of the viva should become the subject of a student appeal'.
89 The review teams found that there is much commonality in the UK system for the assessment of research degrees. However, practice in the appointment of an independent chair varies considerably. Although the practice is firmly entrenched and operated across the faculties in a number of institutions, in others the practice has either been adopted only in parts of the institution, or it is not mandatory. Some institutions have considered adopting independent chairs, but have rejected the practice on the grounds that with very large graduate schools the resulting additional workload would be unacceptable. Some institutions were asked to consider how best to assure themselves that the viva voce process is fair and transparent in the absence of an independent chair.
90 The make-up of the group carrying out the viva attracted comment beyond the appointment of an independent chair. A significant number of institutions were asked to review their practice of allowing a supervisor, member of a supervisory team, or external collaborator in the research, to act as internal examiner, a practice that would not appear to meet the spirit of the Code of practice. Others were asked to review their arrangements for the oral examination of members of staff. One institution was asked to review its requirement for the supervisor to be present (not as an examiner) at the oral, with or without the student's agreement. Evidently practice in formulating the membership of such examining bodies is variable across the UK, with some elements of practice seeming to fall outside the guidance of the Code.
91 Other points which emerged from the review teams' analysis which required consideration by institutions were: the appropriateness of the level of information being sent to external examiners; the need to state explicitly in the institution's guidelines that the examination process is completed within a reasonable timescale.
92 Points of good practice, other than the use of an independent non-examining chair referred to above, were: a requirement for new internal examiners to attend the Internal Examiners' Briefing session; the use of an Enquiry Panel to review students failing to achieve the target award to see if any lessons could be learnt; the use of independent moderators to overcome problems that might arise from a relatively small research base; the clarity of the guidelines for the conduct of research degree examinations; the requirement to review supervisory practice when a thesis requires major revision prior to resubmission; the arrangements for note-taking in an oral examination.
93 While few institutions require a mock viva to be held, this is a growing practice; some institutions provide training sessions or workshops to assist students to prepare for their viva. The review teams regard these developments as valuable in supporting students to cope with what many see as a daunting experience. We note the making available to the student of the preliminary reports of the examiners by one institution, and the introduction of the Freedom of Information Act has prompted a more universal disclosure of the final (joint) report to both student and supervisor(s).
Student representations, complaints and appeals
94 This section deals with the arrangements that institutions have in place for research student representations, complaints and appeals, and is covered by precepts 25 to 27 in the Code of practice.
95 Institutions are expected, according to Precept 25, to have in place and publicise procedures for dealing with student representations that are fair, clear to all concerned, robust and applied consistently. Such procedures will allow all students access to relevant information and an opportunity to present their case. Review teams found that arrangements for student representations in most of the institutions which described them were indeed fair, clear, robust and applied consistently.
96 Most reports described institutional procedures, usually briefly. Typical procedures involve a ladder of stages, starting with informal representations within the department (usually to a personal tutor, supervisor or departmental director of studies or research tutor), and rising to more formal representations at faculty and institutional levels. At least one institution advises its students that they can also can make representations through the monitoring and review processes. A small number of institutions advise students of their right to take cases to the Office of the Independent Adjudicator for Higher Education where internal procedures have not resolved the issue. One institution was able to evaluate the effectiveness of its system for representations through the periodic quality review process.
97 A number of reports also outline how information about how to make representations is made available to students, for example through the student charter, in the code of practice, in research student handbooks and departmental handbooks, and on graduate school websites. Some institutions also brief students on this at induction. The review team in one institution found that it was not clear whether students were aware of the stages and processes for raising issues.
98 The review teams identified one example of good practice in respect of student representations: an institutional Research Students' Committee which focuses on student issues. Only one institution was encouraged to review the effectiveness of student representation and to ensure that all students are aware of the mechanisms available, in light of Precept 25 of the Code of practice.
Complaints
99 This section deals with the arrangements that institutions have in place for dealing with student complaints. Review teams found that in most institutions arrangements for dealing with complaints are appropriate and satisfactory, and aligned with Precept 26 of the Code of practice.
100 Precept 26 expects institutions to have in place independent and formal procedures to resolve effectively complaints from research students about the quality of the institution's learning and support provision. Most institutions, in their institutional responses to the review questionnaire, provided very brief descriptions of their complaints procedures. Most have also recently reviewed their complaints procedures in light of the revised Code of practice.
101 Complaints procedures vary slightly between institutions, but all HEIs have formal procedures that are codified in institutional regulations. Most institutions use their standard undergraduate complaints procedures for research students, which involve an informal first stage (typically by discussing the matter with staff in the department) and more formal subsequent stages, in which complaints must be put in writing and are usually dealt with at faculty or institutional level by a senior academic or administrator, who either acts in person or (more typically) refers the matter on to a standing group or committee. In many institutions students have access to 'independent' advice in student advice centres or Students' Unions.
102 Many institutions did not define an indicative time scale for research student complaints, although review teams found that some are reviewing their position on that. Some institutions allow an appeal against the outcome of the formal complaints procedure, and some advise students of their right to send a complaint to the Office of the Independent Adjudicator for Higher Education where internal procedures have not resolved the issue.
103 In almost all institutions the review teams found that regulations are clear about complaint procedures, what is expected from each party in the process, and the steps through which any process must proceed. Information about complaints procedures is often given to students on registration, and is usually published in research student handbooks, in institutional codes of practice and on institutional websites.
104 Few institutions routinely monitor and report the number and outcomes of complaints by research students, although some are putting systems in place to do this. In a few institutions, summaries of complaints are presented annually to named institutional-level committee (such as the Graduate School Committee) and to senior academic and administrative staff. A number of institutions reported that they received few if any complaints by research students; the maximum reported was four complaints over the last five years.
105 Review teams found a small number of examples of good practice in respect of complaints in a few institutions. These relate to clarity of procedures, clarity of information about procedures, the training and briefing of staff who deal with complaints, the existence of separate complaints regulations for applicants, and the development of improved policy as a result of use of the complaints procedure.
106 Review teams found a number of areas for further consideration. The most common of these (mentioned in a handful of institutions) was the need to define indicative timescales for the complaints process, to ensure that complaints are dealt with in a timely manner. Individual institutions were encouraged to more clearly define the formal complaints process for students, make the process more readily available to students, make more explicit what constitutes the basis for an appeal, or consolidate institutional procedures into a single complaints procedure. Among associated institutions, whose degrees are validated by a 'parent' institution, one was encouraged to review the status of students in their pre-registration period (when they have no access to complaints and appeals procedures), and one was encouraged to review the alignment of its complaints procedures and those of its 'parent'.
Appeals
107 This section deals with the ways in which institutions deal with appeals by research students against specific academic outcomes or decisions. Review teams found that in most institutions arrangements for dealing with appeals are appropriate and satisfactory, and aligned with Precept 27 of the Code of practice.
108 Institutions are expected, Precept 27, to have in place formal procedures to deal with any appeals made by research students, and to clearly define the acceptable grounds for appeals. As with complaints, most institutions provided very brief descriptions of their appeals procedures, and most have recently reviewed their appeals procedures in light of the revised Code of practice.
109 Appeals procedures vary slightly between institutions, but all have formal procedures that are fully or largely aligned with Precept 27 of the Code of practice, and are codified in institutional regulations. Typically, appeals are heard by a formal panel or committee that is usually chaired by a senior member of staff, either academic, or less frequently, administrative. Very few reports include information on panel membership. In most institutions, appeals must be made in writing, and some have separate processes for appeals against examination results. It is not uncommon for an institution to use its standard undergraduate appeals processes for research students, but some have specific procedures for this group. Few reports mention time limits for appeals; in those that do, the time limit is usually within a month, and occasionally as short as two weeks. Provision of training and briefing to staff who deal with appeals was mentioned in only one report.
110 Acceptable grounds for appeal are clearly stated in most institutions. Most explicitly state that appeals against academic judgements are not allowed. Many institutions specify what can be appealed against, and typically this covers examination decisions, and decisions about exclusion decisions, progress review and upgrade or transfer. Most make explicit reference to the fact that appeals can only be made on procedural grounds or where there are extenuating or mitigating circumstances that were not known at the time of the evaluation or examination. A few also allow appeals if there is specific evidence of prejudice or bias of examiners.
111 Information about grounds, stages and possible outcomes of appeals is usually made available to students on registration or at induction. It is typically published in postgraduate regulations and handbooks, institutional codes of practice and on institutional websites. In some institutions the review teams found that students are made aware of their right to take appeals to the Office of the Independent Adjudicator for Higher Education where internal procedures have not resolved the issue.
112 Few institutions routinely monitor and report the number and outcomes of appeals, although some are putting systems in place to do this. In those that do, reports normally go to appropriate university committees (such as Graduate School Committees) and senior academic and administrative staff. One institution plans to review appeals cases to inform enhancement of its current procedures. A handful of institutions mentioned that they receive an average of up to two or three appeal cases each year.
113 Review teams found examples of good practice in relation to appeals in three institutions, all of them reflecting the clarity of information provided to research students on appeals procedures (particularly information on the right to appeal and the outcome of appeals).
114 Review teams suggested to 12 institutions that they should review their arrangements for appeals in light of Precept 27 of the Code of practice. For most cases this was to clearly define timescales for appeals procedures, and ensure that appeals are heard in a timely manner. Some institutions were advised to review their arrangements for appeals against progression decisions. Individual institutions were encouraged to define more clearly the appeal process, make more explicit the grounds that may form the basis of an appeal, adapt the standard institutional appeal procedure specifically for use by research students, consider whether having a member of the supervisory team as a member of the review panel is compatible with the need for impartiality, and review the status of students in their pre-registration period (as with complaints).
Conclusions
Themes emerging from the review
115 The overall picture emerging from the outcomes of this review is highly positive. Institutions show good levels of engagement with the Code of practice in a thoughtful manner, with appropriate reference to the institutional environment and student diversity. There is a clear feeling of evolution of effective practice, rather than revolution.
116 As with any effective self-evaluation or self-assessment, it appears that the process of preparing for the review was developmental for institutions and helped them to know more about, and perhaps improve, their understanding of strengths and weaknesses.
117 In only one or two cases, institutional practice is still directly contrary to the principles set out in the Code of practice. For example, some institutions require or allow supervisors to be present at the oral examination. In all cases, the review team were reassured that institutions were taking steps to address divergent practice and it is encouraging that institutional alignment with the Code generally is so positive.
118 There is evidence of good practice relating to all precepts of the Code of practice, which is particularly notable in respect of two areas: the clear and comprehensive information provided for research students, either in institutional codes or other media; and arrangements for monitoring and reviewing research degrees.
119 Under some headings, for example, Supervision and Assessment, there appear to be significant similarities in practice, with few differences between types of institution. However, in other areas such as workload allocation for supervisors, there are wide variations in practice between institutions
120 The broad areas for further consideration identified by the review teams included: the need for consistency and fairness in applying policies and procedures; the importance of making available clear information which is accessible to all audiences; the provision of effective support for staff in fulfilling their roles and responsibilities; and the need for regular review of practice to assure continuing effectiveness in research education.
121 Some of the areas where institutions are finding it challenging to respond to the principles in the Code of practice could have been expected. For example, providing development opportunities for established supervisors; finding ways of assuring fairness and consistency in the oral examination; implementing more detailed assessment criteria. In these (and other) areas, it is appropriate for institutions to take time to assure themselves that they are taking the right steps for the institution and its students and staff, and to learn about practice elsewhere before finalising policy. This approach may also be likely to encourage ownership of new developments.
122 The review outcomes show that it is possible for different types of institutions to use the Code of practice to support their management of RDPs and to assure quality of provision. The Code seems to be applicable to students in different disciplines and from different backgrounds. This is encouraging as it was designed to accommodate diversity and enable flexibility of approaches.
123 The review process also seems to have encouraged some institutions to engage more fully with their research students and to put in place mechanisms to solicit and act upon student views.
124 The wide consultation that has taken place within institutions in preparing for the review, including making available draft submissions to a wide range of staff and students, has helped to raise awareness and spread good practice internally.
Integration of review of research degree programmes in the external quality framework
125 The review of RDPs, commissioned by the funding bodies of England, Wales and Northern Ireland, made specific reference to the precepts of the Code of practice, Section 1: Postgraduate research programmes and it was understood that the review would be followed up by further audits.
126 The revised QAA institutional audit method therefore requires audit teams explicitly to assess and report upon the extent to which institutional arrangements for securing the academic standards of awards and the quality of provision in postgraduate research degree programmes are in alignment with guidance given in the Code of practice, Section 1: Postgraduate research programmes. The introduction to this explains that the section 'is written in a firmer style than some other sections, especially the precepts, to give institutions clear guidance on the funding councils', research councils' and the QAA's expectations in respect of the management, quality and academic standards of research programmes'.
127 Audit teams will have access to the report on the outcomes of each institution's review, and the institution may wish to make reference to its report, updating the team on developments since the review took place. In coming to a conclusion about institutional arrangements for postgraduate research students, audit teams will focus specifically on the following areas: the use made of external examiners; internal and external review of research provision; research students as partners in quality management; the Academic Infrastructure and other reference points; management information including feedback and other relevant topics. In the final report, audit teams will comment specifically on the institution's arrangements for maintaining appropriate academic standards and quality of provision of postgraduate research programmes.
Appendix A: Review methodology
1 The RDP review method was primarily a desk-based exercise. It involved an evaluation by review teams of an institution's response to a questionnaire about the extent to which the institution had aligned its policies and procedures with the revised Code of practice, Section 1: Postgraduate research programmes. When completing the questionnaire institutions were asked to provide supporting evidence to substantiate the claims made in their response to the questionnaire. On the basis of the evidence submitted by the institution the review teams were asked to form a judgement about the extent to which the institutions' approach to the management of its RDPs was appropriate and satisfactory.
2 The reviewers worked in teams, with the work of each team coordinated by a review coordinator. The review coordinators helped ensure consistency both within and across the review teams. A QAA Assistant Director assumed overall responsibility for the management of the review, providing guidance and support as and when required.
3 The review teams were selected by QAA from nominations made by awarding HEIs. Some existing institutional auditors for England and Northern Ireland and review teams in Wales and Scotland, who met the criteria, were also invited to participate in the review. For a list of reviewers please see Appendix E.
4 The outcome of each institutional submission was a report to the institution and HEFCE or DEL. The individual reports identified areas for further consideration and features of good practice.
Appendix B: Evaluation of the review of research degree programmes methodology
1 Following the review of postgraduate research degree provision, all participating institutions were asked to complete an evaluation questionnaire. In total, 86 out of 114 institutions responded to the evaluation questionnaire.
2 The questionnaire provided the opportunity for institutions to comment upon:
- the information available about the review process
- the operation of the review activity, to include information requirements
- communication
- judgements and findings.
3 Institutions were also asked to identify the strengths and weaknesses of the review process, and to identify ways in which the review findings would be of specific benefit to the institution in respect of postgraduate research degree provision.
4 Almost all respondents (95 per cent) agreed that the review had achieved its aim, and 86 per cent agreed that the outcomes of the review will assist the institution with development and enhancement of quality and standards of postgraduate research programmes.
5 The lowest level of satisfaction with the operation of the review process was expressed in respect of the communication between QAA and the institution throughout the review. A quarter of respondents disagreed that the communication had been effective. Similarly, a quarter of respondents considered that prior to the review, there was not a clear and shared understanding about the criteria the team would use for undertaking evaluation and making judgements.
6 Almost all (98 per cent) agreed that prior to the review, there was a clear and shared understanding about the research programmes to be included in the review.
The appropriateness of a desk-based approach to the review of postgraduate research degree provision
7 In the main, the desk-based approach to the review was considered to be appropriate. There was a general consensus that the desk-based approach, utilising existing documentation and a questionnaire completed by the institution was fit for purpose and significantly less burdensome than other 'visit based' forms of review.
Preparation for the review
8 Respondents reported that the process of having to complete a questionnaire was of immense value, providing an opportunity both for general reflection and for consideration of programmes against the Code of practice. Institutions reported that the requirement to complete a self-evaluation questionnaire acted as the impetus for them to reflect upon existing processes, policies and systems.
9 However, some institutions reported that the preparation for the review had generated significant burdens upon a small number of individuals. The preparation of the questionnaire and the gathering of appropriate supporting documentation and evidence had generated a significant work load for staff. Additionally, concern was expressed regarding the need to consult with a wide body of individuals from across the institution in preparation for the review. This added to the burden and was further compounded by the relatively short timescales allowed for such preparatory activities.
10 There were a number of institutions which had recently experienced an institutional audit and this group considered the RDP review as an additional and unnecessary burden.
11 It was noted by respondents from institutions that greater clarity regarding the requirements for the submission of information and evidence would have aided in the preparation for the review. Similarly, institutions queried the requirement for progression and completion information, which they considered to be largely inappropriate and out of date, and the use subsequently made by the review team of this information as part of the review.
12 The ability to provide information and documentation to QAA electronically was welcomed by institutions.
Identification and dissemination of good practice
13 A positive feature of the review related to the opportunities provided for the identification of good practice. Institutions highlighted that the reviews identified both good and innovative practice in respect of postgraduate research degrees. It was reported that the review not only provided a benchmark through the identification of relative baselines, but also there was the opportunity for the identification of good and innovative practice across the sector. Institutions considered that the outcomes of the reviews provided further opportunities for the dissemination of good practice, which could be used by institutions for benchmarking purposes and informing enhancement agendas.
Judgements
14 Some respondents from institutions suggested that a simple 'pass'/'fail' judgement category adopted for the review was too benign. It was suggested that greater use could have been made of a graded judgement category, so enabling greater differentiation between review outcomes.
The report
15 It was reported that timescales were very tight for the review. Some institutions, considered that the reports were too descriptive and as such, the 'added value' provided by the reports was deemed to be limited. While it was welcomed that the reports teased out good and innovative practice and areas for enhancement and further development, there was some suggestion that greater emphasis could have been placed upon these within the body of the report.
16 In their response to the draft report, many institutions highlighted steps that they propose to take to address any recommendations to emerge for the report, as part of ongoing enhancement activities.
17 Institutions agreed that there was significant value in having good practice confirmed, and from receiving guidance on areas where there was scope for development and enhancement of policies, processes and systems.
Summary
18 There was a general consensus that the review had been of benefit to institutions providing an impetus for the ongoing review of regulations, policies and procedures. In addition, the highlighting of good practice and those aspects of provision that could be further enhanced was also considered beneficial. The recommendations arising from the reports were of significant value to institutions and would be incorporated within future development plans aimed at enhancing and improving postgraduate research degree provision.
19 It was felt that preparation for the review had assisted in identifying the strengths and weaknesses of individual institutions. Additionally, it was perceived that the involvement of a range of staff in the preparation of the institutions' submissions further helped to raise awareness and increase ownership of policies and procedures.
'In general we found the exercise helpful to us in ensuring that we have developed our arrangements in an appropriate manner and that our future plans are in accord with QAA best practices'.
'We have found the exercise to be useful in focussing our attention on the research degrees area of our activity. This activity along with your helpful comments on areas where we can improve our practice, policy or procedures will be incorporated in to our development plans'.
'I would like to take this opportunity to thank the reviewers for their time and efforts in reviewing the submission from the University. The whole exercise has been very useful and enlightening'.
Appendix C: The precepts of the Code of practice
Institutional arrangements
1 Institutions will put in place effective arrangements to maintain appropriate academic standards and enhance the quality of postgraduate research programmes.
2 Institutional regulations for postgraduate research degree programmes will be clear and readily available to students and staff. Where appropriate, regulations will be supplemented by similarly accessible, subject-specific guidance at the level of the faculty, school or department.
3 Institutions will develop, implement and keep under review a code or codes of practice applicable across the institution, which include(s) the areas covered by this document. The code(s) should be readily available to all students and staff involved in postgraduate research programmes.
4 Institutions will monitor the success of their postgraduate research programmes against appropriate internal and/or external indicators and targets.
The research environment
5 Institutions will only accept research students into an environment that provides support for doing and learning about research and where high-quality research is occurring.
Selection, admission and induction of students
6 Admissions procedures will be clear, consistently applied and will demonstrate equality of opportunity.
7 Only appropriately qualified and prepared students will be admitted to research programmes.
8 Admissions decisions will involve at least two members of the institution's staff who will have received instruction, advice and guidance in respect of selection and admissions procedures. The decision-making process will enable the institution to assure itself that balanced and independent admissions decisions have been made, that support its admissions policy.
9 The entitlements and responsibilities of a research student undertaking a postgraduate research programme will be defined and communicated clearly.
10 Institutions will provide research students with sufficient information to enable them to begin their studies with an understanding of the academic and social environment in which they will be working.
Supervision
11 Institutions will appoint supervisors who have the appropriate skills and subject knowledge to support, encourage and monitor research students effectively.
12 Each research student will have a minimum of one main supervisor. He or she will normally be part of a supervisory team. There must always be one clearly identified point of contact for the student.
13 Institutions will ensure that the responsibilities of all research student supervisors are clearly communicated to supervisors and students through written guidance.
14 Institutions will ensure that the quality of supervision is not put at risk as a result of an excessive volume and range of responsibilities assigned to individual supervisors.
Progress and review arrangements
15 Institutions will put in place and bring to the attention of students and relevant staff clearly defined mechanisms for monitoring and supporting student progress.
16 Institutions will put in place and bring to the attention of students and relevant staff clearly defined mechanisms for formal reviews of student progress, including explicit review stages.
17 Institutions will provide guidance to students, supervisors and others involved in progress monitoring and review processes about the importance of keeping appropriate records of the outcomes of meetings and related activities.
Development of research and other skills
18 Institutions will provide research students with appropriate opportunities for personal and professional development.
19 Each student's development needs will be identified and agreed jointly by the student and appropriate academic staff, initially during the student's induction period; they will be regularly reviewed during the research programme and amended as appropriate.
20 Institutions will provide opportunities for research students to maintain a record of personal progress, which includes reference to the development of research and other skills.
Feedback mechanisms
21 Institutions will put in place mechanisms to collect, review and, where appropriate, respond to feedback from all concerned with postgraduate research programmes. They will make arrangements for feedback to be considered openly and constructively and for the results to be communicated appropriately.
Assessment
22 Institutions will use criteria for assessing research degrees that enable them to define the academic standards of different research programmes and the achievements of their graduates. The criteria used to assess research degrees must be clear and readily available to students, staff and external examiners.
23 Research degree assessment procedures must be clear; they must be operated rigorously, fairly, and consistently; include input from an external examiner; and carried out to a reasonable timescale.
24 Institutions will communicate their assessment procedures clearly to all the parties involved, ie, the students, the supervisor(s) and the examiners.
Student representations
25 Institutions will put in place and publicise procedures for dealing with student representations that are fair, clear to all concerned, robust and applied consistently. Such procedures will allow all students access to relevant information and an opportunity to present their case.
Complaints
26 Independent and formal procedures will exist to resolve effectively complaints from research students about the quality of the institution's learning and support provision.
Appeals
27 Institutions will put in place formal procedures to deal with any appeals made by research students. The acceptable grounds for appeals will be clearly defined.
Appendix D: A list of institutions that participated in the review of research degree programmes
Anglia Ruskin University
Arts, London, University of the
Aston University
Bath, University of
Bath Spa University
Birkbeck College
Birmingham, University of
Bolton, The University of
Bournemouth University
Bradford, University of
Brighton, University of
Bristol, University of
Brunel University
Buckinghamshire Chilterns University College
Cambridge, University of
Canterbury Christ Church University
Central England in Birmingham, University of
Central Lancashire, University of
Chester, University of
Chichester, University of
City University
Courtauld Institute of Art
Coventry University
Cranfield University
Creative Arts at Canterbury, Epsom, Farnham, Maidstone and Rochester, University
College for the
Dartington College of Arts
De Montfort University
Derby, University of
Durham, Univerity of
East Anglia, University of
East London, University of
Edge Hill University
Essex, University of
Exeter, University of
Gloucestershire, University of
Goldsmiths College
Greenwich, University of
Hertfordshire, University of
Huddersfield, University of
Hull, University of
Imperial College, London
Institute of Cancer Research
Institute of Education
Keele University
Kent, University of
King's College London
Kingston University
Lancaster, University of
Leeds Metropolitan University
Leeds, University of
Leicester, University of
Lincoln, University of
Liverpool Hope University
Liverpool John Moores University
Liverpool, University of
London Business School
London Metropolitan University
London School of Economics and Political Science
London School of Hygiene and Tropical Medicine
London South Bank University
London, University College
London, University of
Loughborough University
Manchester Metropolitan University, The
Manchester, University of
Middlesex University
Newcastle, University of
Northampton, The University of
Northumbria at Newcastle, The University of
Norwich School of Art and Design
Nottingham Trent University
Nottingham, University of
Open University
Oxford Brookes University
Oxford, University of
Plymouth, University of
Portsmouth, University of
Queen Mary, University of London
Queen's University of Belfast, The
Reading, University of
Roehampton University
Royal Academy of Music
Royal College of Art
Royal College of Music
Royal College of Nursing Institute
Royal Holloway, University of London
Royal Northern College of Music, The
Royal Veterinary College, The
Salford, University of
School of Oriental and African Studies, University of London
School of Pharmacy, The
Sheffield Hallam University
Sheffield, University of
Southampton, University of
St George's Hospital Medical School
St Martin's College
St Mary's College
Staffordshire University
Sunderland, University of
Surrey, University of
Sussex, University of
Teesside, University of
Thames Valley University
Trinity and All Saints College
Ulster, University of
Warwick, University of
West of England, Bristol, University of the
Westminster, University of
Wimbledon School of Art
Winchester, University of
Wolverhampton, University of
Worcester, University of
York, University of
Appendix E: Review of research degree programmes: a list of review team members
Professor Pamela Abbott
Professor John Beeby
Professor Nick Brewin
Professor Anne Curry
Professor Tony Davies
Dr Phil Garnsworthy
Professor Jean Gilkison
Professor Sally Glen
Professor Ian Haines
Dr Sue Hallam
Professor Sandra Kemp
Professor Terry Kemp
Ms Ann Kettle
Dr Martin Le Voi
Professor Roger Linford
Professor Paul Luker
Professor Graham Moon
Professor Chris Park
Professor David Phoenix
Professor Hefin Rowlands
Professor Gaynor Taylor
Professor Christopher Thomas
Professor Bob Usherwood
ISBN 978 1 84482 643 8
Top