- Open Access
A mixed methods study to evaluate the feasibility of using the Adolescent Diabetes Needs Assessment Tool App in paediatric diabetes care in preparation for a longitudinal cohort study
Pilot and Feasibility Studies volume 4, Article number: 13 (2018)
An evaluation study was carried out to determine the feasibility of integrating the Adolescent Diabetes Needs Assessment Tool (ADNAT) App into UK paediatric diabetes care, to ascertain best practice standards and to determine methodological recommendations for a future cohort study.
A non-randomised, cohort, mixed methods study design was used to ensure equality of access to ADNAT for all participants at three sites in the North West of England. Following UK Medical Research Council guidance, the RE-AIM (reach, effectiveness (potential and perceived), adoption, implementation, maintenance) framework was used to guide study objectives and feasibility outcomes. Patients who completed ADNAT (completers) were compared with those who failed to complete (non-completers). Patients’ glycaemic control (HbA1c) was accessed from their clinical data at baseline and at 6 months, alongside their ADNAT scores which were correlated with changes in HbA1c levels. The diabetes teams (respondents) completed a web-based survey and attended focus group interviews.
Eighty-nine patients were recruited. Withdrawal rates were low at 4.5% (n = 4). Forty-four patients (49.4%) completed ADNAT, leaving 45 (50.6%) non-completers. There were large baseline differences in HbA1c and variable rates of change at 6 months. After adjusting for baseline HbA1C and site in an analysis of covariance, completers had a lower post-ADNAT mean HbA1C level than non-completers at 6 months (-5.42 mmol/mol, 95% CI −11.48, 0.64). Patients’ glycaemic control (HbA1c) at 6 months correlated reasonably well with their ADNAT scores (Spearman’s rho = 0.46). Survey and focus group data showed that ADNAT was judged to be an effective clinical tool by the diabetes teams. Value to patients was perceived by the teams to be linked to parental support, age and previous diabetes education. The combined data triangulated. It served to capture different dimensions which were used to define changes to achieve practice standards and methodological recommendations.
The combined data showed that ADNAT has the potential to be a clinically viable tool. It has demonstrated the need for a randomised design that is tailored for a ‘hard to reach’ adolescent population. A cluster randomised controlled trial that involves sequential but random rollout of ADNAT over multiple time periods may be the most appropriate and is currently being considered for the larger study.
NIHR Children’s Clinical Research Network, UKCRN ID 6633
This paper reports on a study which evaluated the feasibility of using the Adolescent Diabetes Needs Assessment Tool (ADNAT) App in three Paediatric Diabetes Units (PDUs) in the North West of England. The study used a realist evaluation approach  to address two issues: firstly, whether the ADNAT App could be integrated into paediatric diabetes care and, secondly, to determine best practice standards and methodological recommendations for a future cohort study. A core assumption of a realist perspective is that phenomena such as ADNAT are complex interventions introduced into constantly changing systems, which has particular relevance to paediatric diabetes care in the UK. Comparisons within and between clinical sites were therefore required to determine what did and did not work and why, in order to establish local modifications necessary to ensure effectiveness in practice. This relationship is expressed in Medical Research Council (MRC) guidance  for process evaluation of complex interventions which defines evaluation of context, implementation and mechanisms of implementation as the primary aims of such studies.
Type 1 diabetes (T1D) is one of the most common endocrine and metabolic conditions in childhood within Europe. The UK, which has 27,600 children and young people living with the condition, alongside the Russian Federation and Germany, makes the largest contribution to the overall numbers in T1D in young people with an incidence rate of 25.9 per 100,000 of the general population aged 0–15 years and a prevalence rate of 195.4 per 100,000 [3, 4]. Alongside this, young people in the UK also have one of the worst rates of glycaemic control in Europe, which is associated with later micro- and macrovascular risk . This has been demonstrated in successive National Paediatric Diabetes Audits (NPDA) for England and Wales for those aged 0–25 years, with the latest for 2015–2016  reporting improving but still disturbing figures alongside the need to reduce variability in outcomes:
Only 26.6% achieved recommended glycaemic targets of less than 58 mmol/mol, with 17.9% having levels above 80 putting them at high risk of complications.
Glycaemic variability is due to service related factors, including standards and delivery of diabetes self-care education which showed wide Paediatric Diabetes Unit (PDU) level variability with 30% receiving no structured education.
For those aged 12 years and over: 26.3% had high blood pressure, 19.7% had high total cholesterol, 9.7% had albuminuria (sign of kidney disease), over 13.8% had early signs of eye disease, and 18.1% were overweight and 20.8% were obese.
Overall, just 35.5% of children and young people received all seven of the recommended key care processes including blood glucose (HbA1c), body mass index, blood pressure, urinary albumin, cholesterol, eye screening and foot examination.
In general, those in the most deprived areas, were of non-white ethnicity, were adolescent, or female had poorer outcomes.
These findings have been supported by the UK’s National Peer Review Programme  which highlighted inequity of service provision to young people. They also fit with the UK’s Kennedy Review  which described teenagers as a ‘forgotten group’, reporting that their health care needs are given low priority by commissioners, policymakers and clinicians alike, and recommended investment in youth friendly services. This is particularly important since many adolescents lack the skills and strategies to avoid behaviours that carry health risks which has specific relevance to those with diabetes given their added risk of future debilitating complications.
In 2012, the UK government responded to the problem by introducing a paediatric diabetes ‘Best Practice Tariff’ (BPT) which defines 13 mandatory care standards including tailored education . For adults, over a 10-year period, such education has been shown to save the National Health Service (NHS) £2200 per patient, breaking even at 4 years [9, 10]. No such data are available for young people, although Swift  reported that education for young people has greater effects than for adults with small to medium effects on glycaemic control and larger effects on psychosocial outcomes.
Given these changes, instigated by the BPT, alongside the annual National Paediatric Diabetes Audit , and a Peer Review Quality Assurance Programme , paediatric diabetes clinical practice was undergoing extensive changes. These emerging changes meant that ‘routine care’ was neither standardised nor constant at the time of the study so that the outcomes of using ADNAT depended upon clinical context and health professionals’ responses to its implementation, making the setting a mediator of outcomes.
In the UK, six paediatric educational trial interventions have been completed [12,13,14,15,16]. They all followed traditional didactic face-to-face approaches, reported considerable variations in outcomes, and no significant long-term changes in glycaemic control. Recommendations included the need to review research methodology and to modernise paediatric care through the use of technology enhanced learning (TEL) to support long-term patient training. This latter recommendation is supported by a review of technology enabled approaches to diabetes management which endorsed self-assessment tools and tailored education based on patients’ unique histories and their immediate needs . In support of this, a meta-analysis of 46 studies found that a blend of TEL and face-to-face instruction had stronger learning outcomes than did face-to-face instruction alone in primary/secondary/tertiary education . However, there are few validated diabetes websites for young people, the majority being directed toward adults; there is wide variation in the quality of evidence provided, and they offer didactic information at high reading levels with little problem-solving assistance [19, 20]. Social networking, as a patient-led tool, is growing in popularity and starting to be used by patients and practitioners but research in children with T1D in all these areas is lacking, both in terms of quantity and quality, reflecting the complex issues of using social media as a clinical tool . Systematic reviews [22, 23], including our own , have consistently highlighted an absence of rigorous UK-based research, minimal use of theory and no reporting of process, health inequalities, dose response and cost-effectiveness data. In addition, findings highlighted the need to personalise learning in alignment with developmental stages, i.e. age-related reasoning and cognitive abilities, making regular needs assessment a core requirement. No instrument to assess such needs was located in the UK. We therefore developed, validated and psychometrically tested the ADNAT App. The App provides secure username and password protected access to ADNAT through mobile devices, e.g. smart phones and tablets, delivers immediate feedback to users and emails confidential patient data to practitioners.
The ADNAT App
Development, validation and psychometric testing of ADNAT have been reported elsewhere [25,26,27]. The research programme, followed Medical Research Council (MRC) guidance for complex interventions . It has included studies of adolescent diabetes self-care [29,30,31] and technological methods of learning  and theory . ADNAT combines reflective questioning with needs assessment to raise self-awareness to support adolescent decision-making in relation to diabetes self-care. It consists of 117 questions divided between six domains including all about me, physical activity, eating, monitoring blood glucose, medication taking and living with diabetes. The number of questions answered by users is filtered according to, for example, insulin regimen and lifestyle factors. Thirty-six of the questions, hidden amongst the total, provide self-care and psychosocial health scores Needs Assessment Ratings (NARs) (the self-care questions with examples of some of the shorter range of responses are shown later in Table 6). Classical test theory and item response analysis validated the use of simple additive scores which we translated into traffic light responses. Scoring for each item was 0 for green, 1 for amber and 2 for red responses, so that high scores indicated high need, moderate scores intermediate need and low scores low need.
From our previous research which qualitatively analysed the perceptions of the clinical usefulness of ADNAT with young people, parents and health professionals , ADNAT was theoretically determined to have the following mechanisms of action:
Mediator for facilitating tailored education and support by raising patients’ self-awareness about their diabetes self-care and coping mechanisms, identifying patient-led foci for conversation in the clinical consultation and providing practitioners and patients with data to guide individual health care planning.
Augmenting resource efficiency through flexibility of access for patients and practitioners using mobile phones and tablets, auto-saving function for ease of use by patients, large data storage capacity and provision of ‘connected information’ for all members of the multidisciplinary team including patients.
Strengthening professional accountability through standardisation of needs assessment, promotion of team working and provision of educational audit data.
Based on these premises, ADNAT was included in the UK National Paediatric Diabetes Improvement Plan for 2013–2018 . This inclusion stipulated the need to evaluate ADNAT’s use in clinical practice prior to long-term implementation for which we proposed to follow MRC’s guidance for process evaluation of complex interventions  to support its on-going development. This guidance identifies three areas for evaluation which are informed by the causal assumptions of the intervention and interpretation of context, implementation and mechanisms of implementation. This process evaluation model and the theories underpinning the intervention (experiential learning theories and the transtheoretical change cycle ) guided the aims of the evaluation study.
Aims and objectives
The aim of this study is to evaluate the feasibility of integrating ADNAT into paediatric diabetes care with respect to:
Resources and processes that influence the clinical implementation of ADNAT
Methodological issues in preparation for a large scale study
The objectives of this study are to evaluate:
ADNAT’s clinical utility in relation to delivery of paediatric diabetes care
How paediatric diabetes health care staff (nurses, doctors, psychologists and dieticians) perceive use of ADNAT within the context of their clinical experiences
Key methodological issues that influence sustainability and best practice
The objectives were defined more specifically using the RE-AIM framework , as recommended in MRC guidance. RE-AIM stands for reach, effectiveness, adoption, implementation and maintenance and included the following:
Reach: we assessed the number of participants recruited and retained and response rates to ADNAT, i.e. number completed divided by total number of recruits. Data were obtained from research nurses’ and patients’ ADNAT (monthly) data returns.
Effectiveness (potential and perceived): we used NPDA data to assess (pre-study) the functional status of each site, a survey to measure site and practitioners’ views on ADNAT’s effectiveness and collected pre/post-glycated haemoglobin (HbA1c) data, taken from patients’ notes by the research nurses, to determine any potential changes in patients’ glycaemic control.
Adoption: we carried out a survey to assess system and information quality, accessibility, social norms, data protection and intention of PDUs to use ADNAT in the future. Focus group interviews explored resources needed to set up and sustain use of ADNAT, staff perceptions of factors affecting adoption and their training needs.
Implementation/maintenance: the survey and focus groups also explored staff responses to working with ADNAT including perceived value and health improvement outcomes, and the focus group interviews looked at facilitators and barriers to use.
The study was conducted between January 2013 and February 2015. Setup and delivery was supported by the Cheshire and Merseyside Children’s Clinical Research Network (CRN). This support included access to the NIHR CRN-funded research nurses in post at the three NHS Trust sites which were selected based on geography and positive responses to invitation letters. The UK National Research Ethics Service (NRES) defined the study as a service evaluation  (see Additional file 1 for the copy of the letter). Approvals from the R&D departments at the three sites were received. The research team had no access to identifiable information for any patient consenting/assenting to participate.
The participants were young people with type 1 diabetes aged 12–18 years.
Respondents, sites and support
Respondents were health professional members of the diabetes teams including paediatric diabetes specialist: nurses, doctors, dieticians and psychologists at three paediatric diabetes centres in the North West of England. These three sites allowed the study to capture diversity of feedback data and ensured adequate representation based on information provided in the 2013–2014 NPDA data (see Table 3). The approach to the implementation of ADNAT was tailored according to team dynamics but each site had a named research nurse for the study, and all members of the team were trained informally by HC to use ADNAT. On-going support was provided by the on-site research nurses and by the ADNAT technologist via email. Each site commenced at a different time point with site 1 starting in March 2013, site 2 in June 2013 and site 3 in February 2014.
We planned to recruit a minimum of 80 patients attending clinic appointments over an 18-month period. Recruitment was undertaken by the research nurses working in liaison with the diabetes teams to identify young people who met the inclusion/exclusion criteria, as shown in Table 1. A letter of explanation and the study information sheet were posted to eligible young people who were later targeted at their clinic appointments. If they agreed to participate, those under 16 gave signed assent, whilst those over 16 years provided informed consent. Proxy consent of parents/guardians of young people under 16 was also obtained. All information sheets and consent forms were produced in age-and-stage-of-development appropriate formats and were checked before use by an audit team at one of the participating sites. Copies of the signed assent/consent documentation were given to the young people and, where appropriate, their parent/guardian for their records; the original copy was filed in participants’ medical notes, and copies were kept in the site study files held by the research nurses.
Delivery of ADNAT
Participants were provided by the research nurses with username and password access to the ADNAT App, alongside their usual standard care based on the BPT  criteria (3-monthly follow-up including HbA1c, and tailored self-care education; annual review of body mass index, blood pressure check and screening for eye and kidney problems from age 12, plus psychological assessment). The ADNAT App was accessed through the Internet using a PC, laptop or mobile technological devices including participants’ mobile phones or tablets. They could choose where to complete it: at home and/or in clinic on their own smart phones or using iPads loaned to them by the research nurses. All participants were followed up at their diabetes outpatient clinics or at a home visit. In both cases, the ADNAT outcome data was used to guide their health care plans. Those who successfully completed and submitted their ADNAT questionnaires were called the ‘completers’, whilst those who chose not to submit were called the ‘non-completers’ and were used as the comparative group.
Quantitative feasibility outcome data
A range of feasibility outcomes were measured including:
ADNAT data to measure reach in terms of response rates across the PDUs. All data, which was automatically downloaded onto a secure central database, were encrypted to ensure anonymity. Data included number of participants recruited and retained and response rates to ADNAT and ADNAT NARs for self-care and psycho-social health.
National Paediatric Diabetes Audit (NPDA) data  to assess the functional status (effectiveness) of each site.
Glycaemic control to compare potential effectiveness pre/post-intervention levels of glycaemic control using baseline HbA1c levels (means/standard deviations over previous 12 months), and 6-month post-ADNAT levels obtained from patients’ clinical notes by the research nurses.
A 67-item survey to collect information on perceived effectiveness, adoption, implementation and maintenance. Adapted from a validated survey developed by Okazaki et al , it has seven domains including system and information quality, accessibility, perceived value, data protection, health improvement, subjective norms and intention to use in the future. We also included an open-ended question at the end asking if there was anything they would change. The survey was facilitated by the Audit Department at one of the participating sites using SNAP software (http://www.snapsurveys.com/) and was pilot tested by two researchers. It was sent out by the Audit Department via an email link to all respondents. Responses were returned directly to the Audit Department where analysis of the data was completed using SNAP software.
Qualitative process evaluation data
Three focus groups were run at the end of the study period, one at each of the three sites. All respondents were invited to participate, and they were sent information sheets and an interview schedule prior to the meetings. The schedule was informed by the RE-AIM domains. It aimed to evaluate training needs, staff perceptions of facilitators and barriers to using ADNAT, and staff responses to working with ADNAT including perceived value and perceived health improvement outcomes. Consent forms, prior to participation, were signed for tape-recording the interviews.
We specified a priori progression criteria that should be met in order for continuation to the main study. These included the following: that recruitment to the study should be ≥30% of the PDUs’ eligible 12–18 years populations attending clinic, that there should be no deterioration in mean HbA1c levels at 6 months in participants and that diabetes teams should report positive feedback on ADNAT’s perceived effectiveness including system and information quality and accessibility and be tailoring ADNAT to meet their site needs.
The encrypted quantitative ADNAT data were collated, coded and analysed using R or SPSS. All quantitative data taken from the ADNAT questionnaires were checked for missing or unusual values and for internal consistency of the scoring items. Participant glycaemic control (HbA1C) was monitored pre and post use of ADNAT at each site and across all sites using summary statistics for completers and non-completers. Analysis of covariance was performed with post-HbA1C as the dependent variable and baseline HbA1C, completer status and site as the independent variables to assess whether any preliminary change in HbA1C levels was apparent. Correlations between high/moderate self-care needs (based on the self-care NAR) and poor/moderate levels of (pre-baseline and at 6 months) HbA1c were analysed using Spearman’s rho statistic .
Qualitative data were analysed using an inductive thematic content analysis, assisted by QSR NVivo software. First, an evolving set of themes was created and linked to respondents’ ‘quotes’. These themes were then categorised within the RE-AIM domains. To assure trustworthiness of the analysis, respondent validation was used by cross-checking findings with respondents and triangulating it with the quantitative outcomes.
Data from recruitment of patients, the NPDA, the survey and focus groups comprise the results of the study. The RE-AIM domains are used to present both the quantitative and the qualitative data.
Comparison of ADNAT completers versus non-completers
Table 2 shows recruitment rates and participant characteristic data. We planned to recruit a minimum of 80 patients and we recruited 89 in total, with an uptake of 65–70% of those who were screened as eligible to participate. The graph in Additional file 2 shows that actual recruitment rates were above our monthly anticipated target rates. Of those recruited, there were twice as many females to males and the withdrawal rate was low at 4.5% (n = 4). Reasons for withdrawal included patient transfer to other areas (n = 1) and not wanting to continue with the study (n = 3). Forty-four young people (49.4%) submitted their completed ADNAT questionnaires. There were more female than male completers and non-completers (ratio ~1 male to 2 females), and their average age was 14.3 years compared to 14.5 years for the non-completers. Forty-eight of the young people were aged 11–14 years (25 completers, 23 non-completers), and 41 were aged between 15 and 18 years (19 completers, 22 non-completers).
Effectiveness (potential): glycaemic control
Data in Table 3 are taken from the 2013/14 NPDA . It shows disparity between the sites for the percentage of young people who have a mean HbA1c of the recommended level of less than 58 mmol/mol (range 8.1–26.5%), with the mean (range 65.5–78.7 mmol/mol) and median values (range 64–74 mmol/mol) for all sites above the recommended level. The NPDA ascribes such variability (despite statistical adjustments for known confounding influences, such as ethnicity, social deprivation, gender, age and diabetes duration) to differences in service provision and delivery which has particular relevance for this study. In relation to this, care process records, which are used to monitor diabetes management and detect long-term complications at the earliest treatable stage, were also significantly different in terms of incomplete records (range 25.4–69.1%), again highlighting disparity between the three sites. Of note is the fact that the two with poorer HbA1c audit results (sites 1 and 3) had interruptions in team functioning during the study period due to staff changes and/or long-term staff absences owing to sickness.
Table 4 shows the glycaemic control data pre- and 6 months post-ADNAT for the completers versus the non-completers. For both groups, subject-specific profile plots (not shown) and the range of pre- and post-HbA1C levels indicated that the young people had very different pre-glucose levels and variable rates of change leading to their post-HbA1C levels. Overall, summarising across all three sites, there was a non-significant reduction in the post-ADNAT mean and median HbA1c levels for the completers versus a non-significant increase in the mean and median levels for the non-completers. The mean HbA1C levels are illustrated in Fig. 1 and suggest a potential decreasing trend in HbA1C for ADNAT completers. This trend is encouraging given that our progression criteria defined ‘no deterioration in HbA1c’ as the outcome at 6 months.
The results of the analysis of covariance (ANCOVA), presented in Table 5, show how post-ADNAT mean HbA1C levels changed after adjustment for pre-HbA1C, completer status and site. In general, the model explained a reasonable amount of the overall variability in post-HbA1C levels with a R 2 value of 0.52. Only pre-HbA1C mean level was a strong predictor of post-HbA1C mean level (p < 0.001), which is to be expected since the two measures are correlated. On average, post-HbA1C levels increased by 0.71 mmol/mol for each unit increase in baseline HbA1C. The completer status variable reached borderline significance (-5.42, 95% confidence interval −11.48, 0.64), indicating that on average, completers had a post-ADNAT mean HbA1C level of 5.42 mmol/mol lower than non-completers. Mean differences between site 2 and site 3 compared to site 1 indicated a lower average post-HbA1C mean difference by 1.75 mmol/mol at site 2, and a higher average mean difference by 1.50 mmol/mol at site 3, compared to site 1, after adjusting for baseline HbA1C and completer status, but these differences were non-significant. Please note that the results from the above model should be interpreted with caution due to the small numbers at each site.
Figure 2 shows a scatter plot of the correlation between HbA1C levels and self-care scores. The Spearman rho coefficient is 0.46 suggesting a good moderate correlation with higher (worse) self-care scores indicating higher HbA1C levels overall. Only at one site was very little correlation observed due to several outlying young people with high HbA1C levels but generally lower self-care scores.
All questions comprising the self-care score are listed in Table 6 together with the number of children and percentage scoring green, amber and red at each site using the ADNAT scoring algorithm. Examples of the response categories, comprising green, amber and red coding, are also given for a selection of the items. The table indicates that a large proportion of young people were scoring green for each item, indicating reasonable management, but box plots by item and site (not shown) suggested that there was variability across items and sites and that it was not necessarily the same group of young people scoring green across all items.
Eleven people (two males, nine females) out of 14 completed and submitted the survey. They included seven nurses, one doctor, two dieticians and one research nurse. When asked about years worked in paediatrics, two had worked for five or less years, two for 6–10 years, five for 11–20 years and two for 21–25 years. Two people had a recognised adult teaching certificate (English National Board 998 Adult Teaching and Assessing course), but no one had a paediatric teaching qualification.
Table 7 shows responses to statements relating to adoption of ADNAT. Given the ad hoc nature of the scales used, single item responses are reported (as opposed to scale-scores) which cover the following areas:
Information and system quality: the majority of respondents (n = 7–9) gave a positive response to the five statements relating to information quality, suggesting confidence in ADNAT. However, there was uncertainty for four participants in relation to ADNAT being sufficiently complete to meet their patients’ needs. Overall, the majority of participants strongly agreed or agreed with the 13 statements relating to system quality, but some ambivalence was expressed in relation to ADNAT launching and running quickly and not crashing.
Intention to use ADNAT in the future, accessibility and capability with technology: ten respondents said they intended to use ADNAT when it is available at their work place. However, the majority (n = 9) indicated that accessing Wi-Fi is a problem in their work place, and four people felt that their workplace is not good in the way it uses technology. Nine reported that patients had completed ADNAT at home and five in clinic. All respondents felt that ADNAT was secure with regard to data protection. When asked about using technology in clinical practice, nine respondents reported capability with computers, tablets and mobile devices; the remaining two were ambivalent. Nine people regarded technology to be an important element of their patients’ education.
Social norms (not shown in Table 7): when asked to give an opinion on whether their National Children and Young Peoples’ Diabetes Network, their managers, their colleagues, and their patients and families would think they should be using ADNAT, all participants (n = 11) responded positively to the Network and patients and families options, and seven positively responded to the managers and colleagues options.
Table 8 summarises survey responses to questions relating to implementation and maintenance of ADNAT which included the following:
Perceived value: overall, respondents judged ADNAT to be effective, practical, useful and efficient, with nobody judging it to be poor. In relation to factors that influenced its value to patients, respondents were unsure about gender, insulin regimen and hospital admissions, but more confident with regard to parental support, age and previous diabetes education.
Health improvement outcomes: statements here were based on the transtheoretical change cycle. The majority of participants indicated that ADNAT had an effect at each of the eight different stages of the cycle with the majority indicating that effects happened sometimes, often or regularly. The ‘not applicable’ responses came from the research nurse who was not involved at the clinical level and were therefore not included in the calculations.
When asked to add any comments about what they would change about ADNAT (not included in Table 8), five responses were received including the need for iPads and improved Wi-Fi access in clinics, access to on-line reports and inclusion of a section for patients to ask for immediate feedback/help from the diabetes team.
Each site and participant was coded (based on roles and numbers) for reference purposes as follows: Paediatric Diabetes Specialist Nurses (PDSN 1–6), Doctors (Dr 1,2), Researcher (R 1,2), Psychologist (P) and Dietician (Di). Twelve people in total attended the three groups (site 1: n = 5, site 2: n = 3, site 3: n = 4) providing a total of 160 min of recorded conversation. Analysis of the focus group data produced seven sub-themes which were aligned to the RE-AIM framework (themes). Findings are presented using anonymised quotes to capture the essence of the phenomena and are summarised at the end in Tables 9 and 10 which provide a combined summary of the quantitative and qualitative data and a synopsis of practice recommendations.
Tension for change
Respondents found ADNAT to be a viable option for clinical practice and wanted to change the way they engaged with their patients by using technology, recognising that web-based applications play a crucial role in adolescent life. They perceived technology as a way of overcoming communication barriers, as this nurse commented,
I remember being quite impressed at what they were saying they didn’t know. So it seems to get past that barrier when it is face to face. They are more likely to be honest even though they know we’re going to see it (PDSN4).
The three sites all felt that ADNAT fit within their teams’ values and goals, with participants suggesting that it could standardise educational assessment allowing for comparisons between PDUs. As a policy driver, the BPT enhanced motivation to use ADNAT, linking it to the education criteria. Other respondents agreed with this thinking suggesting that it also met with the peer review process but questioned its practical potential as an audit tool given that it assesses those aged 12–18 years only making it ‘difficult to draw any conclusions’ (Dr1).
Operationally, decision-making regarding how to implement ADNAT was devolved to the teams to see what emerged. Two different methods were used: individual nurses reviewing their own patient returns versus using a generic email for all returned questionnaires which was reviewed by one PDSN only. For the former, choice of treatment was determined by the individual nurses but they also discussed their approaches within their teams. For the latter, the PDSN identified urgent cases, i.e. red traffic light returns for discussion at team meetings. Both approaches therefore embraced working as a team, as the following quote highlights,
We did bring the red ones to the team meeting and there were actions..generated from it, and we did implement those actions. I think the ones that came through green reinforced what we felt but it was good to get the teenagers’ perspectives married up with ours (PDSN2).
Team capacity varied owing to sickness absence and/or new staff starting. Site 2 was not affected by these problems and had an established team. It was notable that this site had the best 2013/2014 audit returns in relation to glycaemic control (as highlighted in Table 3) and also the best ADNAT return rates (69.2%), compared with the other two sites (46.4 and 37.1%). With regard to team working, participants commented on how ADNAT encouraged a standardised approach which supported consistency in the messages given to patients. They argued that integrating ADNAT into their team work would normalise its use, although lack of time, given the current politically driven changes, had impacted upon feelings of being in control. Respondents talked about ‘time constraints’, ‘feeling too busy’ and ‘a continual focus on problems’. Receptivity for change therefore varied across the sites although respondents felt that ADNAT had the potential to drive change.
The time taken for patients to complete ADNAT was discussed given the large number of questions to be answered, but two respondents had asked patients for their perspectives and both reported positive responses,
I asked quite a few of them, was it a waste of 30 minutes of their life and they all said ‘no’ they felt it was useful that they’d done it, and many of them said it made them think… (Dr2)
..a lot of them came back and said it was a good use of their time and gave them that refreshment of the advice that we gave them previously (PDSN6).
For the practitioners, the time taken to review patients’ outcome results was helped by ADNAT’s scoring and traffic light feedback systems, alongside the drop-down menus and navigation commands to allow selection of scoring questions only and/or questions relating to the different domains. However, when asked whether they felt the traffic light system was good for the children, there was a mixed response. It was viewed as both a facilitator and a barrier, with the barrier relating to its potential to raise young peoples’ anxiety.
Embedding ADNAT into practice
There were mixed responses with regard to where ADNAT should be completed. Location was seen as important affecting both uptake and practitioner feedback to patients. Home completions brought problems in relation to patients being willing to complete it once they left clinic, and time between completion and feedback was deemed important, as the following quote highlights,
Because they did it at home and sometimes then a week after their previous clinic appointment, then it would be reviewing it again much later. …and actually they couldn’t remember the results (PDSN4)
Theoretically, completions in clinic prior to their consultation were thought beneficial, but practically, this was not an option given time limitations. Home visits were appropriate for two of the sites, but at the third site, home visits were being discouraged by management. These comments highlighted a barrier to embedding ADNAT into practice. When questioned about how this could be overcome, integrating ADNAT into patients’ health care plans was seen as a viable option, with patients completing ADNAT prior to their next clinic appointment at home. Suggestions included gaining consents in clinic and sending instructions on how to complete ADNAT with their clinic appointment letters, followed by text reminders. The role of the lead clinician was seen as crucial for embedding ADNAT into routine practice, alongside mandating its use through, for example, including it in the BPT criteria. Tailoring ADNAT to fit each site was seen as important, paying attention to the whole team being involved. To meet this goal, training (up to a maximum of 4 h) was considered essential. Web-based instruction was not popular given the need for self-motivation and personal time, but face-to-face training was deemed superior in that it would,
..help to promote the team aspect of it because discussion could be had about how to make it cohesive as a team (PDSN4).
Another suggestion was to include previous users of ADNAT, i.e. expert patients in the training programme.
Respondents felt that ADNAT mapped on to what they are aiming to achieve in clinic including getting patients to “create (their own) agendas and identify things” (P). Other respondents felt that it provided the link between all the different components of diabetes self-management commenting that ADNAT got patients “thinking about aspects of the condition which they might normally not really think about” (PDSN 3). There were comments that in clinic, the focus tends to be on blood glucose and insulin doses whilst ADNAT promoted reflection on all aspects of their diabetes, including their feelings. One person summarised ADNAT’s perceived value in the following way,
the opportunity for self-evaluation of learning, reflection, and for young people to actually get feedback on what they know, and also for the teams to have feedback on what they know as young people (Di).
This process of self-evaluation was a strong theme throughout the focus groups with one nurse commenting that ADNAT, “…reminded them (patients) about the right ways to manage their diabetes” (PDSN6). There was an agreement that ADNAT promotes behaviour change, and in terms of why it is effective, one person summarised her opinion by saying that,
It gives them (patients) a chance to identify. They’re doing the identifying, possibly prioritising things for themselves …and if it has come from them, then they are much more likely to engage in conversations about what could be done differently…. (P)
Respondents questioned the traditional (misplaced) focus on glycaemic control with one person stating that education is more about quality of life at this age and being able to ‘… get a balance between their diabetes and being a teenager…’ (PDSN4). This point was agreed by others who felt that a single educational intervention is not going to impact upon glycaemic control because there are ‘an awful lot of things that affect someone’s HbA1c’ (Dr.2). Education was seen as beneficial in other ways including improving quality of life and self-care processes, and the example of carrying glucose to treat hypoglycaemia was used to highlight this point.
Having open-ended text responses at the end of each question was seen as important, because it allowed patients to express their feelings of knowing more and being in control. This concept of ‘control’ was an important theme, with ADNAT being viewed as a way of accessing patients’ needs without removing their sense of control, as the following quote highlights,
..it might be a question that they (patients) might not have thought about, but felt a bit embarrassed, or thought well, I shouldn’t think like that, or maybe other people don’t feel or think like that, I should know that. (PDSN3)
Accessing patients’ needs meant that the teams could tailor conversations with their patients, focusing on their raised self-awareness with regard to what they did and did not know, providing a base on which to progress joint health care planning.
Combining the data
We used the RE-AIM framework to combine the data from recruitment of patients, the NPDA, the survey and the focus groups. This summary of the quantitative results and the qualitative findings is provided in Table 9. We also identified the main points raised for how best to integrate ADNAT into clinical practice (summarised in Table 10).
This study aimed to evaluate the feasibility of integrating ADNAT into paediatric diabetes care focusing primarily upon resources and processes that influenced its implementation taking into account context and perceived impact. We met each of our progression criteria, recruiting over 65–70% of those screened as eligible to take part; there was no deterioration in mean HbA1c levels at 6 months; and diabetes teams reported positive feedback on ADNAT’s perceived effectiveness and tailored the use of ADNAT within their clinics to meet their site needs.
The study also aimed to determine methodological recommendations for a future large-scale study. Its strength lies in its mixed methods design and the fact that there was overlap between the different data sets which supports the findings and helps to explain the feasibility outcomes. The summary of the quantitative results and the qualitative findings in Table 9 shows that from a quantitative (reach and potential and perceived effectiveness) outcome perspective, ADNAT met the proposed mechanisms of action and progression criteria from a staff perspective. The survey and qualitative findings indicate that ADNAT was acceptable to the diabetes teams. However, these results need to be interpreted with caution due to the study design and the associated confounding factors.
Methodological implications: strengths and limitations
A limitation of this study is the small number of sites, participants and respondents involved, alongside the non-randomisation to treatment. Both of these limitations raise statistical issues concerning the accuracy of the outcome data. Differences in characteristics between those patients who chose to complete ADNAT compared to the non-completers are unknown, and it can be argued that the former group may be more compliant generally with regard to their diabetes self-care when compared to the latter group. However, this potential difference was accounted for by controlling for HbA1c levels at baseline. Findings therefore provided an insight into ADNAT’s potential in relation to glycaemic control (HbA1c), particularly for those poorly controlled. However, it can also be argued that those young people who completed ADNAT may have wanted a different kind of management for their diabetes, one that fits more effectively with a digital culture and their learning styles. This suggests that completers may have been ready to make changes compared to the non-completers. Reasons why young people choose to engage or not in using ADNAT therefore need to be researched to improve future response rates.
Questions regarding the number of young people who completed ADNAT also need to be addressed. Whilst the percentage of completers (49.4%) is good in terms of figures quoted for the response rate of the general population to web-based surveys (24.8%) , it remains questionable as to how typical the completers were relative to the non-completers and to those who declined to participate. It highlights the need to identify ways of engaging more effectively with what is typically a ‘hard to reach’ population. Systematic reviews [40, 41] suggest extended timeframes, recruitment techniques suited to young people and the need to work in close co-operation with the community.
The study identified the research setting as a mediator of outcomes making treatment heterogeneity a confounding factor. Whilst ADNAT is a standardised intervention, the responses of the teams to the outcomes of using ADNAT are heterogeneous. This demonstrated a need to include non-participant clinic observations in a future study to evaluate what is actually happening in clinical practice when ADNAT is used. It also validated the need to research comparative effectiveness of the different research sites by using the historical NPDA data to provide evidence on the value of different response options to ADNAT.
Previous research has highlighted the need for adult diabetes education to have broad patient-based outcomes and not to be expected to have lasting benefits on glycaemic control unless it is repeated. The focus group data reinforced these points highlighting the need to include other outcomes such as quality of life and patients’ greater self-involvement in their care. Findings from previous trials of paediatric diabetes education have also affirmed this point . Measures of effectiveness therefore need to include but not be limited to glycaemic control.
This evaluation study has provided evidence on the feasibility of using ADNAT in clinical practice and has demonstrated a number of limitations which have provided practice and methodological guidance. It has shown that a randomised design that fits the needs of a ‘hard to reach’ adolescent population is necessary. A cluster randomised controlled trial that involves sequential but random rollout of ADNAT over multiple time periods may the most appropriate and is currently being considered for the larger study.
Adolescent Diabetes Needs Assessment Tool
Analysis of covariance
Best Practice Tariff
Clinical Research Network
Medical Research Council
Needs Assessment Rating
National Health Service
National Institute for Health Research
National Paediatric Diabetes Audit
National Research Ethics Service
Paediatric Diabetes Unit
Research and Development
Reach, effectiveness, adoption, implementation, maintenance
Type 1 diabetes
Technology enhanced learning
Pawson R. The science of evaluation: a realist manifesto. Los Angeles, London, New Delhi, Singapore: Sage; 2013.
Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D, Baird J. Process evaluation of complex interventions: Medical Research Council guidance. London: MRC Population Health Science Research Network; 2014.
International Diabetes Federation. IDF diabetes. 7th ed. Brussels, Belgium: International Diabetes Federation; 2015. http://www.diabetesatlas.org. Accessed 7 Feb 2016.
NHS Information Centre for Health and Social Care. National Diabetes Paediatric Audit Reports. London: Department of Health. Available at: http://www.rcpch.ac.uk/improving-child-health/quality-improvement-and-clinical-audit/national-paediatric-diabetes-audit-n-0. Accessed 15 May 2017.
Diabetes Control and Complications Trial Research Group (1994) Effect of intensive diabetes treatment on the development and progression of long term complications in adolescents with insulin-dependent diabetes mellitus. Journal of Pediatr. 1994;125:177–188.
National Peer Review Programme. National peer review report: paediatric diabetes services 2013/2014. London: National Peer Review Programme; 2015.
Kennedy I. Getting it right for children and young people. Overcoming cultural barriers in the NHS so as to meet their needs. London: Department of Health; 2010.
Department of Health. Payment by Results team. Payment by results guidance for 2011-12. London: Department of Health; 2011.
Diabetes UK (2014). The cost of diabetes report. In: http://www.diabetes.org.uk/Documents/Diabetes%20UK%20Cost%20of%20Diabetes%20Report.pdf. Accessed 7 Feb. 2016.
Li et al. Cost-effectiveness of interventions to prevent and control diabetes mellitus: a systematic review. (2010) Diabetes Care. In: http://stacks.cdc.gov/view/cdc/7107/. Accessed 7 Feb 2016.
Swift P. Diabetes education in children and adolescents. Pediatr Diabetes. 2009;10 Suppl 12:51–7.
Waldron S. Education and other interventions for diabetes management in children and young people: detailed overview of randomised controlled trials. Diab Care Child Young People. 2012;1(2):73–4.
Price KJ, Wales J, Eiser C, Knowles J, Heller S, Freeman J, Brennan A, McPherson A, Wellington J. Does an intensive self-management structured education course improve outcomes for children and young people with type 1 diabetes? The Kids In Control OF Food (KICk-OFF) cluster randomised controlled trial protocol. BMJ Open. 2013;3:e002429.
Murphy HR, Wadham C, Rayman G, Skinner TC. Approaches to integrating paediatric diabetes care and structured education. Diab Med. 2007;24:1261–8.
Chaney D, Coates VE, Shevlin M. Running a complex intervention—lessons learned. J Diab Nurs. 2010;14:370–9.
Hawthorne K, Bennert K, Lowes L, Channon S, Robling M, Gregory JW, on behalf of the DEPICTED Study team. The experiences of children and their parents in paediatric diabetes services should inform the development of communication skills for healthcare staff (the DEPICTED study). Diab Med. 2011;28:1103–8.
Kaufman N. Internet and information technology use in treatment of diabetes. Int J Clin Pract. 2010;64 Suppl 166:41–6.
Means B., Toyama Y., Murphy R., Bakia M., Jones K. (2010) Evaluation of evidence-based practices in online learning. A meta-analysis and review of online learning studies. In: http://repository.alt.ac.uk/629/1/US_DepEdu_Final_report_2009.pdf. Accessed 8 Feb 2016.
Seidman JJ, Steinwachs D, Rubin H. Design and testing of a tool for evaluating the quality of diabetes consumer-information web sites. J Med Internet Res. 2003;5:e30.
Bull SS, Gaglio B, McKay HG, et al. Harnessing the potential of the Internet to promote chronic illness self-management diabetes as an example of how well we are doing. Chronic Illn. 2005;1:143–55.
Pal BR. Social media for diabetes health education—inclusive or exclusive? Curr Diabetes Rev. 2014;10(5):284–90.
Boren SA, Gunlock TL, Peeples MM, Krishna S. Computerized learning technologies for diabetes: a systematic review. J Diabetes Sci Technol. 2008;2(1):139–46.
Bass AM, Farhangian ME, Feldman SR. Internet-based adherence interventions for treatment of chronic disorders in adolescents. Adolesc Health Med Ther. 2015;6:91–9.
Cooper H, Cooper J, Milton B. Technology-based approaches to patient education for young people living with diabetes: a systematic literature review. Ped Diab. 2009;10:474–83.
Cooper H, Spencer J, Lancaster J, Titman A, Johnson M, Lwin R, Wheeler S. Development and psychometric testing of the online Adolescent Diabetes Needs Assessment Tool (ADNAT). J Adv Nurs. 2014;70(2):454–68.
Cooper H, Spencer J, Lancaster J, Johnson M, Lwin R. Perceptions of the clinical usefulness of the Adolescent Diabetes Needs Assessment Tool (ADNAT). Diab Care Children Young P26eople. 2013;1(2):55–61.
Cooper H, Spencer J, Lancaster J, Titman A, Johnson M, Lwin R, Wheeler S. (2015) Development and psychometric testing of the online Adolescent Diabetes Needs Assessment Tool (ADNAT). World Biomed frontiers Obesity and Diabetes. In: http://biomedfrontiers.org/diabetes-obesity-2015-1-21/. Accessed 1/3/2016.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:979–83.
Spencer J, Cooper H, Milton B. A qualitative phenomenological study to explore the lived experiences of young people (13-16 years) with type 1 diabetes and their parents. Diabetic Med. 2013;30:e17–24.
Spencer J, Cooper H. A multidisciplinary paediatric diabetes health care team: perspectives on adolescent care. Pract Diabetes Int. 2011;28(5):210–4.
Spencer J, Cooper H, Milton B. Qualitative studies of type 1 diabetes in adolescence: a systematic literature review. Ped Diab. 2010;11(5):364–75.
Cooper H, Geyer R. Using ‘complexity’ for improving educational research in health care. Soc Sci Med. 2008;67:177–82.
Diabetes NHS. National Paediatric Diabetes Service Improvement Plan 2013-2018. London: Department of Health; 2013.
Ruggiero L, Prochaska JO. Readiness for change: application of the transtheoretical model to diabetes. Diabetes Spectrum. 1993;6:21–60.
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7 (www.RE-AIM.org).
NRES: Determine whether your study is research. In: http://www.hra.nhs.uk/research-community/before-you-apply/determine-whether-your-study-is-research/. Accessed 7 Feb 2016.
Okazaki S, Castañeda JA, Sanz S, Henseler J. Factors affecting mobile diabetes monitoring adoption among physicians: questionnaire study and path model. J Med Int Res. 2012;14(6):e183.
Altman DG. Practical statistics for medical research, London: Chapman and Hall/CRC; 1991.
Penwarden R. (2014) Response rates for online surveys—what numbers should we be aiming for? In: http://fluidsurveys.com/university/response-rate-statistics-online-surveys-aiming/. Accessed 7 Feb 2016.
Billie B, Adeleine Randell M, Paul C, Chapman K, Twyman L, Bryant J, Brozek I, Hughes C. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Med Res Methodol. 2014;14:42.
Shaghaghi A, Raj S, Bhopal RS, Aziz SA. Approaches to recruiting ‘hard-to-reach’ populations into research: a review of the literature. Health Promot Perspect. 2011;1(2):86–94.
We thank HealthImo for their work on developing the website and the App under the direction of Eric Link. We thank all the patients who completed ADNAT and the health professionals and research nurses who contributed at the three study sites and also the National Institute for Health Research (NIHR) Alder Hey Clinical Research Facility which supported the study under the direction of Professor Matthew Peak and Chris Walsh, portfolio manager at the NIHR Clinical Research Network (CRN) for Children. We also thank the Audit Department at one of the sites which supervised the writing, distribution and analysis of the survey under the direction of Steven Riley.
The study was supported by Alder Hey Children’s NHS Foundation Trust which funded the purchase of 10 iPads and the employment of three NIHR Children’s Research Nurses.
Availability of data and materials
Please contact the author for data requests.
HC wrote the protocol, carried out the training, analysis of the qualitative data and interpretation of the survey data and drafted the initial manuscript. GL and PG were responsible for the analysis and interpretation of the statistical data and revisions of the questionnaire. GL was also involved in the design of the study and reviewed and revised the manuscript with HC. MP was involved in the design of the study, approved funding for the study and reviewed the final manuscript. All authors read and approved the final manuscript.
Helen Cooper was seconded to the NIHR Alder Hey Clinical Research Facility, Alder Hey Children’s NHS Foundation Trust, whilst this study was conducted.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
To meet the requirements of the Research Governance Framework, approval for the study was sought from the UK National Research Ethics Service (NRES). The protocol was submitted to the North West – Cheshire NRES Committee where it was agreed that the study was not considered to be research according to their guidance  and was defined as a service evaluation not requiring ethical review (see Additional file 1 for the copy of the letter). Their decision was based on the fact that the study involved an intervention only with no allocation or randomisation procedures, that the choice of treatment (education/support) was determined by the clinician(s) and patients, that it measured current service without reference to a standard and that data collection involved analysis of existing data together with administration of interviews and questionnaires. Following this, the proposal was submitted to the NHS Research and Development Departments at each of the three sites for their approvals and agreement that the study was a service evaluation rather than a research study. Approvals were received from all three sites based on the requirement that appropriate ethical standards were followed. We therefore used site files, information sheets and consent/assent forms. Confidentiality and anonymity were maintained by using a coding system for all participants (allocated by the Research Nurses), and data were stored in a locked cabinet. The research team had no access to identifiable information for any patient taking part in the study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
A correction to this article is available online at https://doi.org/10.1186/s40814-017-0183-2.
Additional file 1:
Ethics. (PDF 169 kb)
Additional file 2:
Recruitment. (DOCX 14 kb)
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Cooper, H., Lancaster, G.A., Gichuru, P. et al. A mixed methods study to evaluate the feasibility of using the Adolescent Diabetes Needs Assessment Tool App in paediatric diabetes care in preparation for a longitudinal cohort study. Pilot Feasibility Stud 4, 13 (2018). https://doi.org/10.1186/s40814-017-0164-5
- Type 1 diabetes
- Needs assessment
- Patient education
- Glycaemic control