Warwick Certificate in Diabetes Care: An evaluation

It is recognised that continuing professional development, including education in diabetes care, for healthcare teams is essential for maintaining quality of care provision (Standing Committee on Postgraduate Medical and Dental Education, 1994). The National Service Framework (NSF) for diabetes (Department of Health [DoH], 2003) indicates that care can be improved and education is a necessary tool to facilitate this.

The educated healthcare professional can help the person with diabetes to effectively self-manage his or her condition. Self-management skills can lead to improved blood glucose control and hence a delay in the onset or a reduction of complications of diabetes (DoH, 2003). These include damage to the vascular system and to major organs such as the eyes, kidneys or heart. Any reduction in complications may reap huge financial benefits for health services as well as having the potential to improve the quality of life for many people living with diabetes. This has been recognised by healthcare professionals themselves (Agarwal et al, 2002).

It is known from the literature that education can be effective in changing clinical and organisational behaviour (Freudenstein and Howe, 1999). Professional development in the healthcare system benefits from an organisational climate in which development is part of normal practice (Gibson, 1998), yet there are restricting factors including the lack of time, resources, support and recognition of the need for continuing professional development (Brown et al, 2002).

The aim of continuing professional development for healthcare professionals in all disciplines is to provide a planned and ongoing process which improves patient care. This improved care is achieved by maintaining knowledge and skills, improving existing competencies and learning new ones (Guly, 2000). Good outcomes can be obtained from interprofessional learning which is well planned, has clearly stated objectives, and is relevant to and recognises each participant’s experiences. It also needs to be led with enthusiasm (Parsell et al, 1998). Interprofessional learning for healthcare professionals has been shown to improve understanding of the roles of others and increase the effectiveness of collaborative working in shared tasks (Pearson, 1999; Headrick et al, 1998).

Benefits have been identified in providing education in diabetes care for practice nurses as part of a practice team (Hearnshaw et al, 2001). Nurses identified the benefits of learning alongside GPs and other professionals, such as dietitians and podiatrists, in order to develop an integrated service for patients. Lack of protected time and high workload are major reasons why diabetes education had not been undertaken, especially in practices with a small number of practice nurses (Gorrod, 1996).

Certificate in Diabetes Care 
The Certificate in Diabetes Care (CIDC) at the University of Warwick aims to provide healthcare professionals of all disciplines with the practical knowledge and skills necessary to provide an effective and efficient service for people with diabetes. It supports participants in extending their knowledge, increasing their skills and developing their confidence in delivering high quality diabetes care to enable effective self-management. The CIDC comprises five taught days of supported study sessions and self-directed learning, spread over 9 months (visit http://www2.warwick.ac.uk/fac/med/healthcom/diabetes/ for more information [accessed 08.12.2005]).

Assessment comprises completion of a case study, an audit, a project and a written examination. Achieving the CIDC requires 80% attendance and satisfactory completion of all coursework. The course has been running since 1996 and over 5000 health professionals have achieved the Certificate (Hearnshaw et al, 2004). Although obviously popular, it is also appropriate to evaluate its effectiveness in order to be sure that the education is worth investing in. The evaluation is reported here.

Evaluation design
Evaluation can be done in many ways, but it is important to use a method which is theory-based and informs people of what data are necessary for a valid evaluation. Measures for evaluating a programme should include (from Peyrot, 1996):

• programme exposure
• content delivered
• quality of delivery
• participant involvement
• participant views.

All the above should preferably be measured at the start and end of the programme. The principles of the systematic method to evaluate patient education programmes for people living with diabetes are also applicable in evaluating healthcare professional education. There is a distinction between the evaluation of an education programme and assessment of an individual’s educational needs.

Thus, an appropriate way to evaluate the effectiveness of the CIDC was to gather and assess the knowledge levels of participants before and after the course, and measure completion rates. Also, as intention to change and having made changes are both strong indicators of likelihood of changing (Peyrot and Rubin, 1995), measuring both actual changes made in diabetes care and intended changes is valuable. This paper reports such evidence on all these aspects in order to evaluate the effectiveness of the CIDC, and hence its suitability to support the implementation of high quality diabetes care, in particular meeting the requirements of the NSF.

Methods
Baseline measures
All 2111 participants attending CIDC courses from September 2000 to August 2003 were asked to complete a questionnaire on the first day of attendance (questionnaire 1). The questionnaire covered self-assessed knowledge of 68 diabetes-related topics, rated on a 4-point scale:

• 0 = no knowledge
• 1 = limited knowledge
• 2 = good knowledge
• 3 = extensive knowledge.

The scores on this scale were converted to a score out of 100 for ease of comprehension. The questionnaire asked for demographic details and characteristics of the responder’s practices including: list size; whether or not the practice is a training or teaching one; whether the practice has dedicated diabetes clinics; and whether services for foot care, eye care, dietary advice or blood pressure care were provided. The questionnaire was developed from existing tools (MacKinnon, 2000).

Outcome measures
Nine months later, when handing in the final completed piece of coursework, all participants were asked to complete a second questionnaire (questionnaire 2). This had all the same questions as questionnaire 1 but with additional questions on changes in practice made since the course, changes planned, and any identified obstacles to making the changes. Comments on the course were also invited. The participants were asked whether they would recommend this course to others.

Analysis
Changes reported by responders and comments on the course were in open text format. These were categorised from the responses by three researchers independently for the first 40 questionnaires. A list of category codes were agreed and a further 20 questionnaires coded by two researchers independently. Coding was compared and no discrepancies were found. The codes were then used by one of the researchers to code all subsequent responses. When a new code was deemed necessary the researchers discussed and agreed its introduction and any required amendments to previous codes.

Administrative staff recorded data on attendance and coursework completion. Data entry and cleaning ensured that outliers, missing data and illegible or ambiguous responses were handled systematically and consistently. To monitor the maintenance of quality of data input, members of the research team carried out checks by double entry of data samples. SPSS software (SPSS Inc., Chicago, USA) was used for all data management.

Results
Levels of participation
During the data collection period, 99 CIDC courses were completed and 2111 healthcare professionals from across the UK commenced the course (56% of whom were nurses, 36% doctors, 2% podiatrists, % dietitians, 1% pharmacists and 4% other). Of these, 1965 (93%) completed all course work satisfactorily, passed the written examination and so achieved the CIDC (5% did not complete the course and 2% failed the final examination).

Responses to the questionnaires
A total of 1398 (71%) of those gaining the CIDC responded to both evaluation questionnaires. In the following analyses mean values have been calculated for these participants.

Changes in knowledge over the duration of the course
The differences in scores between CIDC questionnaires 1 and 2 showed overall changes in self-reported knowledge on 68 diabetes-related items. The mean scores, over 1319 participants, for all items, increased from 40% (range 5–93%) to 66% (range 32–99% P≤0.01).

Data were collected and analysed for the first 684 responders; we investigated whether these increases in self-reported knowledge were the same for responders from practices of different sizes and with different practice organisation for diabetes care. For these analyses, scores of knowledge were first converted to five score bands: 0–40%, poor knowledge; 41–50%, satisfactory; 51–60%, adequate; 61–70%, good; 71–100&%, excellent.

Practice size
Responders were grouped by quartiles of their list size: up to 3900 patients, small; 3900–7300, medium-small; 7300–10 200, medium-large; 10 200 or above, large. A comparison of knowledge scores with list size showed that before the course, the level of knowledge was related to the list size (Chi squared=25.33, 9 degrees of freedom [df], P<0.005). Responders from larger practices showed higher scores. However, after the course these differences disappeared (Chi squared=8.43, 9 df, P>0.5).

Practice diabetes care organisation
Several aspects of care, such as whether the practice was a teaching practice, whether it had dedicated diabetes clinics and whether it provided eye care, were reported at the start of the course. Comparisons were made between mean knowledge scores of responders who did or did not offer these aspects (Table 1). In general, the results show that the level of knowledge increased for all sub-groups, to reach a common level at the end, even though the starting levels were different.

Changes in organisation of diabetes care over the duration of the course
The mean number of changes in diabetes care actually made was 2.6, with a mean of 1.4 changes planned, as found from questionnaire 2 (n=1398; total frequency of organisation and clinical changes implemented = 3635; total frequency of organisation and clinical changes planned = 1957). Thus there was a mean of 4.0 service development changes in diabetes care, per respondent, either made or planned.

For the first 684 responders, the changes identified were categorised by the researchers. Thirty-nine categories of change were identified. Frequency of responses on changes made ranged from one to 415. Frequency of changes planned ranged from one to 111. The 12 most frequently reported items are listed in Table 2.

Obstacles to implementing planned changes were identified by 473 respondents. There were 19 categories of obstacles and the frequencies ranged from one to 279. The 12 most frequently reported categories to implementing changes are listed in Table 3.

Participants’ views
Of 1379 responders 90% said ‘yes’ to recommending the course to others, 5% ‘maybe’ and 5% ‘no’.

For the first 684 responders, comments made were categorised by the researchers. Thirty-nine categories were identified. Frequency of comments in categories ranged from one to 236. The 15 most frequently reported comments are listed in Table 4.

Discussion
Evidence of positive effect
The evidence presented here is from a systematic evaluation of an educational programme in health care. The evidence demonstrates that the CIDC increases healthcare professionals’ self-perceived knowledge of diabetes care. It raised the participants’ knowledge towards a common level, independent of their starting level.

The authors found levels of initial knowledge in teaching practices, practices describing themselves as having a special interest in diabetes, those running dedicated diabetes clinics and those having registers which are used for call and recall of patients to attend higher than in practices where these aspects were not present. Nevertheless, by the end of the course all groups showed similar levels of knowledge. It is clear that with the final levels of knowledge being around 66%, participants still recognised scope for increasing knowledge levels even after completing the CIDC.

There was evidence that the course had stimulated numerous improvements in organisation and clinical practice for diabetes care. The improvements in process of care should, in turn, produce improvements in patient outcome (Clinical Governance Research and Development Unit, 2002). Furthermore, direct investigation into the effects of the course on patient outcome is gathered by course participants through clinical audit of their own patient groups.

Critique of the evaluation
The finding that 95% of course participants would, or may, recommend the course to others is both pleasing and challenging for the course presenters. The courses leading to the CIDC undergo continuous quality improvement in line with normal university teaching quality assessments. All the data presented here were also used in that process. Changes have been made in light of the results from these analyses. For example, marking has been more carefully structured, training in marking has been increased and the assessment processes have been redesigned.

It could be argued that, since the measures used in this evaluation are self-assessment of knowledge and self-report of changes made, these values may be higher than an independent assessor would rate them. These views, though, are validated by the assessed coursework that participants produced (a case study, an audit, a written exam and a project) for the CIDC, the quality of which has generally been very high. Nevertheless, even if the levels were lower, the reported changes in levels are both real and important.

Further evaluations should attempt to measure the impact on practice, knowledge decay, the development of continued learning strategies, and the relationship between the learning outcomes and assessment.

Relevance to primary care
Primary care organisations in the UK that are aiming to deliver the service required by the NSF for diabetes (DoH, 2003) will value the evidence presented here on the effectiveness of this model to meet their needs in workforce development. The course is suitable for meeting the demands of the NSF for diabetes in improving care for people living with diabetes. The courses can meet those needs in many parts of the UK (Hearnshaw et al, 2004).

Acknowledgments
The authors would like to thank all the responders, advanced leaders, and past and present teaching and administrative staff at Warwick Diabetes Care. They would also like to thank Jan Hopkins for setting up the processes, Celia Creagh for all the data handling, Susan Gatward for contribution to data interpretation and Ursula Edgington for invaluable secretarial support. Gareth Jones conducted data analysis as a University of Warwick Undergraduate Research Studentship scholar.

Conflict of interest
Warwick Diabetes Care is partly funded by fees of students attending the courses leading to the Certificate in Diabetes Care, at Warwick Medical School, University of Warwick.