Skip to main content

Fidelity evaluation of the compared procedures for conducting the PVS-PREDIAPS implementation strategy to optimize diabetes prevention in primary care

Abstract

Background

Assessing the fidelity of an implementation strategy is important to understand why and how the strategy influences the uptake of evidence-based interventions. The present study aims to assess the fidelity of the two procedures for engaging primary care (PC) professionals and for the deployment of an implementation strategy for optimizing type 2 diabetes prevention in routine PC within the PREDIAPS randomized cluster implementation trial.

Method

We conducted a mixed-method fidelity evaluation study of the PVS-PREDIAPS implementation strategy. Nine PC centers from the Basque Health Service (Osakidetza) were allocated to two different procedures to engage physicians and nurses and deploy a implementation strategy to model and adapt the clinical intervention and its implementation to their specific contexts: a Global procedure, promoting the cooperation of all healthcare professionals from the beginning; or a Sequential procedure, centered first on nurses who then pursued the pragmatic cooperation of physicians. Process indicators of the delivery and receipt of implementation strategy actions, documented modifications to the planned implementation strategy, and a structured group interview with centers’ leaders were all used to assess the following components of fidelity: adherence, dose, quality of delivery, professionals’ responsiveness and program differentiation.

Results

Generally, the procedures compared for professionals engagement and deployment of the implementation strategy were carried out with the planned differentiation. Nonetheless, some unexpected between-group differences were observed, the initial rate of collaboration of nurses being higher in the Sequential (93%) than in the Global (67%) groups. Exposure rate to the programed implementation actions (% of hours received out of those delivered) were similar in both groups by professional category, with nurses (86%) having a higher rate of exposure than physicians (75%). Professionals identified half of the planned discrete strategies and their rating of strategies’ perceived usefulness was overwhelmingly positive, with few differences between Sequential and Global centers.

Conclusions

The PVS-PREDIAPS implementation strategy has been implemented with high fidelity and minor unplanned reactive modifications. Professionals’ exposure to the implementation strategy was high in both groups. The centers’ organizational context (i.e., work overload) led to small mismatches between groups in participation and exposure of professionals to implementation actions.

Trial registration

Clinicaltrials.gov identifier: NCT03254979. Registered 16 August 2017.

Peer Review reports

Background

Implementation fidelity has, until recently, been defined as the degree to which a given program is implemented as it was originally planned and it has been mostly focused in measuring the deployment of the evidence-based intervention under study [1, 2]. Yet, in complex interventions, the program includes the evidence-based intervention and the supporting implementation strategies aimed at facilitating the adoption of this intervention by those responsible for delivering it [3, 4]. Consequently, factors related to adherence to the planned implementation strategy, dose received, i.e., the extent to which the recipient was exposed to the implementation strategy, participant responsiveness and actual involvement, as well as modifications made and the role of context, become central issues for understanding the impact of implementation initiatives to improve real-world clinical practice [3, 5]. As the field of implementation expands and the use of hybrid trial designs grows [6], the distinction between intervention-level fidelity and implementation strategy-level fidelity is becoming increasingly important.

Evaluating the degree to which implementation strategies are operated as designed within implementation trials are key in order to determine the internal and external validity of implementation studies [1, 3,4,5]. They are necessary to investigate both the receipt and the scope of an implementation strategy to improve the adoption of evidence-based practice in routine settings [4, 7]. Additionally, they help in the interpretation of outcome results of interventions translated to real practice and inform the optimization of both the clinical intervention and/or implementation strategy to favor adoption of the intervention and implementation and future scale-up in other contexts and settings [8,9,10,11]. Fidelity evaluation is especially necessary in multisite trials, where the “same” implementation strategy may be enacted and received in different ways [12, 13].

However, despite the importance of implementation fidelity evaluation, first, there is currently no framework explicitly establishing either a set of procedures or specific requirements to guide the evaluation of the fidelity of an implementation strategy [4, 14]. And second, likely linked to this first issue, under-reporting of fidelity of implementation strategies is the rule rather than the exception in the implementation research literature [4]. Among general existing frameworks to guide fidelity evaluations [2, 3, 5, 7, 8], the framework stated by Dane et al. [8] and its adaptation by Dusenbury et al. [5], has been successfully used by others for rating the fidelity of implementation strategies [4]. This framework points to the following five main fidelity dimensions: a) “adherence”, which strategies were actually used during implementation versus which were planned; b) “dose”, the quantity of the implementation strategy delivered and extent to which the recipients actually received it; c) “quality of program delivery”, how well the components of the implementation strategy have been delivered and if modifications occurred; d) “participant responsiveness”, how the delivery process of an implementation strategy is received by its recipients; and e) “program differentiation”, defined as the degree of enactment of differentiated procedures or strategies in one condition from that in the other condition should be also considered as an important element of implementation fidelity evaluation [4, 5, 8].

The aim of the present study is to evaluate the fidelity of two procedures being compared for deploying the PVS-PREDIAPS implementation strategy to optimize type-2 diabetes prevention in primary care (PC) [15]. Briefly, the PVS-PREDIAPS implementation strategy consist in conducting externally facilitated collaborative modeling process through a set of planned implementation actions with PHC professionals in order to adapt and integrate an evidence-based healthy lifestyle promotion intervention to prevent type-2 diabetes within the routine primary care services. This fidelity evaluation will help to gain understanding about the quality of the implementation of the PVS-PREDIAPS strategy and specifically regarding the two procedures compared for engaging professionals and deploying the implementation actions. This in turn will help to explain and interpret future results of the PREDIAPS trial (ie., to reject a possible type III implementation failure error), will identify what has been changed from the original implementation plan and how changes may impact outcomes, and will inform how future dissemination and scale-up can be improved. Specifically, the objectives of the present study were to:

  • To describe the process indicators of the PREDIAPS trial and deployment procedures of the PVS-PREDIAPS implementation strategy [15], including a clear description of the context in which it has been implemented

  • To assess the fidelity of the delivery of the PREDIAPS implementation strategy among the groups compared, in terms of the dose of both the strategy delivered and that actually received by professionals and centers involved, the planned and unplanned modifications that took place, i.e., adherence, and the degree to which elements can reliably differentiate one type of procedure for engagement and deployment of the strategy from another, i.e., program differentiation.

  • To assess the perceived usefulness of the implementation strategy by professionals involved (responsiveness)”

Methods

Design

This is a mixed-method fidelity evaluation study of the 2.5-year enactment of an implementation strategy to improve the adoption of an evidence-based healthy lifestyles intervention for the prevention of type-2 diabetes in primary care - the PREDIAPS Trial. Briefly, the PREDIAPS is a randomized cluster implementation trial conducted in nine Basque Health Service (Osakidetza) PC centers that aims to assess the effectiveness and feasibility of different engagement procedures to perform a facilitated interprofessional collaborative process - the PVS-PREDIAPS implementation strategy- to optimize type-2 diabetes prevention in routine PC. The protocol has been published elsewhere [15].

Headed by a local leader and an external facilitator, centers were expected to perform a collaborative structured process to adapt an evidence-based intervention to promote healthy lifestyles and its implementation to their specific context. Centers were randomly allocated to one of two arms. One arm was to apply this strategy to engage staff globally, promoting the cooperation of all healthcare professionals from the beginning, while the other arm applied the strategy sequentially, centering first on nurses, who then sought the pragmatic cooperation of physicians. The research protocol was approved by the Basque Country Clinical Research Ethics Committee (Ref. no.: 08/2015) and the protocol was registered in Clinicaltrials.gov (identifier: NCT03254979, registered 16 August 2017).

Participants

PC centres: Osakidetza PC centers identified by managers as having adequate organizational readiness for change seeking to optimize the primary prevention of T2D were considered eligible. Centers were finally included if, after a session introducing the intervention, they succeeded in gaining written consent and agreement to participate from the majority (> 51%) of the nursing staff of the center and a substantial proportion of the physicians whose patients would be involved through the nurses.

PC Users: Patients aged 30 years or more who sought medical attention at least once through the participating centers between 2 March 2017 and 2 March 2018, and had been classified as at high risk of developing T2D and/or prediabetes (abnormal fasting glucose level or glucose intolerance plus an additional known cardiovascular risk factor) but did not have a documented diagnosis of T2D in their health record were eligible to participate in the T2D prevention program at the centers. The final definition of the target population was the result of actions implicit in the implementation strategy.

Recommended evidence- and clinical practice guideline-based clinical intervention

A program was recommended based on scientific evidence and the clinical practice guidelines available, reviewed by our research group in 2016 [16]. This program involved first screening for T2D risk. Then, patients identified as at high risk should be invited, and if they agree, participate in an intensive structured intervention program focused on the prescribing of personalized plans for lifestyle change (low-energy low-fat diet or Mediterranean-type healthy diet; and at least 150 min of moderate physical activity a week). Lastly, patients should be followed up, initially with frequent contact and then annual check-ups. The 5 A’s (Ask, Advise, Agree, Assist, and Arrange follow-up) intervention framework was used to standardize the provision of the evidence-based behavior modification techniques used to promote changes in physical activity (150 min of moderate physical activity a week) and diet (Mediterranean-type healthy diet) to prevent type-2 diabetes in high-risk patients (see Appendix Table 4).

Implementation strategy for facilitating the adoption of the recommended intervention in routine clinical practice

The PVS-PREDIAPS strategy is based on the creation of an inter-professional community of practice that undertakes a process of modelling, adapting and integrating the recommended clinical intervention into the local context, led by the clinicians themselves, a local leader and an external facilitator [15]. This translates into the following actions to be carried out over the course of an implementation phase lasting 1 year (including holiday periods):

  • Selection by consensus in the center and subsequent training of the local leader (3 sessions, 15 h in total). The goal is for the leader to acquire knowledge and skills concerning the primary prevention of T2D and planning for the implementation of primary prevention in practice, as well as communication and leadership skill and techniques. Further, seeking to offer ongoing support and facilitation to the local leaders, 4-h Ongoing leader support meetings are to be held monthly, for preparing the actions to be taken in each of the sessions held in the centers and reviewing the progress made.

  • Training for the clinical intervention (90-min theoretical and practical training session) and the software support tool (6-h practical training session on the PVS-OSABIDE_AP tool)

  • Collaborative modeling of the local T2D prevention program: creation of an inter-professional community of practice lead by the local leader and external facilitator, completion of a needs assessment and process to identify areas for improvement, and mapping of actions, parties responsible, flows and procedures for the intervention program (three 90-min study, discussion and consensus sessions)

  • Plan-Do-Study-Act (PDSA) piloting cycles: process involving short pilot studies of specific actions seeking to assess their efficiency and, thereby, optimize the set of actions in the program (each cycle involving a 90-min session for planning and/or evaluation of the cycle)

  • Standardization and integration of the program: final specification of the objectives, intervention target population and screening or identification strategy, key components of the intervention and follow-up: detailed mapping of actions and processes; staff involved, material and organizational components, as well as components related to the context of the center, etc. (one 90-min study, discussion and consensus session)

The full set of actions of the implementation strategy take a total of 20 h. Once the program has been deployed in the centers and seeking to assess progress and provide ongoing support to encourage intervention programs being sustainably integrated into each center’s portfolio of services, a set of Regular audits and ongoing facilitation sessions are to be held over the second year post-implementation (six 90-min sessions, a session every 2 months). The specific discrete implementation strategies used in the PREDIAPS trial, as cataloged by the Expert Recommendations for Implementing Change (ERIC) taxonomy [17], are described elsewhere [15].

Random allocation to the procedures compared

As described elsewhere [15], with the aim of isolating the effect of two different procedures to engage professionals and perform the PVS-PREDIAPS facilitated interprofessional collaborative implementation strategy to optimize T2D prevention in routine PC, centers were randomly assigned to: a Global strategy, seeking involvement and cooperation between physicians and nurses from the outset; or a Sequential strategy, first led by nurses, and then seeking the pragmatic involvement and cooperation of physicians later in the process. Specifically, physicians within the Sequential group were involved from the second PDSA cycle, and consequently, they only participated in 4.5 h of the core implementation strategy actions estimated to take 20 h in total. The allocation process was conducted using a random number sequence generated by computer prior to the start of the trial by an external researcher from the Primary Care Research Unit of Bizkaia.

Measures

In order to evaluate the fidelity of the two procedures to perform the PVS-PREDIAPS collaborative modeling implementation strategy to optimize T2D prevention in PC [15], the following factors were measured in accordance to the fidelity evaluation framework stated by Dane [8] and Dusenbury [5].

Adherence

The degree of adherence to the planned execution of the implementation strategy for each procedure for conducting the PVS-PREDIAPS implementation strategy and its active ingredients are assessed by comparing three sources of information: i) the protocol for the implementation strategy [15]; ii) actual process indicators; and, iii) the reports, material and products resulting from the sessions. The FRAME framework [18] for reporting modifications (planned or reactive) to evidence-based interventions is used in order to evaluate and describe modifications made to the planned implementation strategy. Specifically, for each of the modifications, the following were specified: the reason, by whom they was requested or encouraged, when and how they took place, and the degree to which they affect fidelity depending on whether they were (or were not) planned in advance. In addition, a strategy mapping for complex interventions [19] has been used to specify the strategies deployed and the timing in which the strategies have been applied.

Dose

The following indicators related to the process of conducting the PREDIAPS trial and the actions embedded to deploy the implementation strategy are specified for reporting on the dose:

  1. a.

    Percentage of centers included out of all those approached

  2. b.

    Percentage of healthcare professionals who initially collaborated out of the total number of professionals at each center

  3. c.

    Actions carried out over time (training, work sessions, etc.).

  4. d.

    Participation of collaborating healthcare professionals in each action and actual exposure to the implementation strategy actions compared to that originally planned (% of hours/action received out of the total number of hours that would be implied by participation in all the actions of the strategy, i.e., 20 h or 4.5 h).

Quality of program delivery/participant responsiveness

A structured group interview was carried out with the local leaders from the centers involved (n = 9) in order to assess the perceived usefulness of the implementation strategy among healthcare professionals. Six open-ended questions focused on the implementation strategies perceived to be part of the PREDIAPS trial, the perceived value of these strategies, and recommendations for their optimization. This interview was video-recorded with prior consent from the participants. Specifically, the following questions were used:

a) personal rating of the implementation process and associated strategies (through questions such as, “Specifically, which aspects of the process for optimizing practice do you consider the most important or useful for doing your job?”)

b) recommendations for optimizing the PVS-PREDIAPS implementation strategies for achieving the following among the staff: (i) build personal competence, (ii) engage staff and (iii) enhance interprofessional cooperation (through questions such as, “In your opinion, which aspects should be strengthened to build competence among staff for putting the innovation into practice in a sustainable way in your center.

Program differentiation

Lastly, program differentiation is determined by comparing the degree to which indicators of adherence, dose and modifications can be reliably differentiated between one type of procedure for engagement and deployment of the strategy and another [5].

Other measured variables

The following variables were used in order to describe collaborating centers’ characteristics: (1) center size, measured in terms of the number of users assigned to receive care at each center (the catchment population) and the number or practitioner lists; (2) center location (Suburban, City center or Town-rural); and (3) the aggregated socioeconomic status of people assigned to each center as measured by the Deprivation Index [20]. This index is an ordinal variable, categorized into five levels (deprivation quintiles; 1 representing high and 5 low socio-economic status), providing a relative measure of the socioeconomic characteristics of the population of census tracts. Its design allows socioeconomic and environmental inequalities between residents to be estimated by census tract in Spain. The calculation takes into account the percentages of residents in a tract who, according to the most recent data available (2016 census), are manual workers, unemployed, or on temporary contracts, or overall or specifically among young people, have a low level of educational attainment.

Analysis

Frequencies and proportions were used to describe characteristics and process indicators related to professional participation and exposure rates for each collaborating center and for the Global and Sequential groups. The mean deprivation quintile for all patients under the care of each of the collaborating centers in 2016 was used to estimate the Deprivation Index at center level.. Regarding the qualitative study based on a structured group interview, responses to the questions were extracted from both the paper surveys and the audio of the discussion that emerged. For the analysis and interpretation of this information, the ERIC strategies [17] referred to in the healthcare professionals’ responses were identified and coded positively or negatively. These strategies were then classified by center and question grouping.

Results

Of the 12 PC centers put forward by the management of the 4 participating Osakidetza integrated healthcare organizations, 9 were recruited to the project. Of these nine centers, six were classified as being located in an “Urban city” area and three were allocated to each of the comparison groups, while the only center classified as “Residential”, which had the lowest mean Deprivation Index, was allocated to the Global strategy group, and the only two “Urban-rural” centers were allocated to the Sequential strategy group. Centers described as “large” based on the catchment population and number of practitioner lists were more represented in the Global than in the Sequential group, which contained the two smallest centers.

Across the 9 centers, 137 physicians and nurses originally gave written consent and agreed to collaborate (70% of all the physicians and 82% of all the nurses assigned to these centers) (Table 1). The initial collaboration rate among nurses was higher in the Sequential (94%) than in the Global (69%) group. The overall exposure rate to the programed implementation strategy actions (% of hours received out of those delivered; maximum = 20 h) was slightly higher in the Sequential than the Global group for both categories, with nurses (89.4% vs 85%) having a higher rate of exposure than physicians (82.6% vs 75%). Assessing the exposure of collaborating professionals in each category to the implementation actions, the percentage of nurses exposed to at least 80% of the actions was again a higher in the Sequential group (70% vs 60% in the Global group).

Table 1 Collaborating primary care centers’ characteristics and professionals participation and exposure rates

Tables 2 and 3 describes the actual execution of the actions of the implementation strategy and its components, including modifications (planned or reactive), over the course of the 2.5 years during which the deployment of the strategy and setting up of the intervention program were due to take place (March 2017–May 2019). All nine centers held the three leader training sessions and the 10 sessions led by the local leader supported by the external facilitator, which comprise the core strategies and actions within the implementation phase. The rates of participation in each of the sessions of the strategy were over 50% of the staff in each category in almost all cases and confirm differing patterns of execution of the actions by centers and by group (Global vs Sequential) (see Appendix Fig. 1). In the post-implementation phase, five sessions were held to support and monitor the setting up of the intervention program. As a component of the strategy, befovre each of the sessions held in their center,11 coordination and preparation sessions were run for the leaders, led by the facilitator, of which 5 were for following-up the setting up of the program and monitoring progress.

Table 2 PVS-PREDIAPS implementation strategy and action timing mapping
Table 3 Reporting modifications of the PVS-PREDIAPS implementation strategy

In the process of executing the strategy, various unplanned and reactive modifications were made (see Table 3). Specifically, three actions were carried out in addition to those planned: two were related to training, one of these being a session for one center and the other for new nursing staff from several centers; and one was a repeat session for encouraging participation of medical staff in one of the Sequential strategy centers, on this occasion involving managerial staff of the integrated healthcare organization, seeking to boost the involvement of physicians. All of these modifications were requested by the centers themselves via the local leader. On the other hand, three of the planned actions were not carried out as intended. Specifically, of the planned six ongoing monitoring and facilitation sessions, a maximum of five were held, with fewer in some centers, mainly due to scheduling constraints and heavy workloads. Additionally, not all centers managed to organize and run one of the planned Ongoing supportive training sessions.

Table 2 displays the original implementation plan involving 11 ERIC discrete implementation strategies and the intervention mapping, which consisted of a combination of 3 ERIC discrete strategies, yielding a total of 14 discrete strategies. In the group surveys/interviews regarding the perceived usefulness of the implementation strategies, the healthcare professionals recognized half of these 14 planned strategies. Appendix Table 5 lists the 14 planned strategies, as well as 3 additional ERIC implementation strategies that were perceived by the healthcare professionals to be part of the implementation process. In general, the strategies that were identified were rated positively, with 10 ERIC strategies described by the healthcare professionals as useful for their own professional development, valuable for the optimization process, and/or specifically helpful to build competence, engage professionals, and/or enhance inter-professional collaboration. Room for improvement was noted for only two ERIC strategies delivered - audit and feedback (ERIC 5) and conducting educational meetings (ERIC 15). While centers assigned to both groups identified the need for increased frequency, length, or quality of educational sessions, only one center (assigned to the Global arm) indicated a need for more audit and feedback sessions. Few differences were observed in the perception of strategies received between centers assigned to each arm. Healthcare providers in the centers assigned to the Global strategy identified the importance of conducting cyclical small tests of change (ERIC 14) and revising professional roles (ERIC 59), while none in the Sequential group identified these strategies. In contrast, one Sequential group center noted the value of promoting adaptability (ERIC Strategy 51), while no Global group centers did. Facilitation (ERIC 33) and need to mandate change (ERIC 44) were identified as important by more Global than Sequential arm centers, while providing audit and feedback (ERIC 5) and conducting local needs assessment (ERIC 18) were identified by more Sequential than Global arm centers. See Appendix Table 5.

Discussion

Fidelity evaluation is necessary to advance knowledge and understanding about the effectiveness of implementation strategies designed to facilitate adoption of evidence-based interventions and practices [2, 4]. The PREDIAPS project seeks to generate scientific evidence concerning the optimization of healthcare practice in primary prevention of T2D in Osakidetza PC centers through the application of implementation science as a way to achieve feasible, sustainable and effective translation of the recommended evidence-based clinical intervention to clinical practice [15]. A first step in this process is to ascertain whether the implementation strategy and the procedures for putting it into practice in the groups and centers compared has been executed as planned or there have been modification or variations that could have an impact on the outcomes of interest and in the future reproducibility of the study [4, 5, 8]. Considering that, we lack a specific framework to guide implementation strategy-level fidelity evaluations, we have adopted the framework for fidelity evaluations stated by Dane and Dusensbury with the purpose of evaluating and reporting on the fidelity of PVS-PREDIAPS implementation strategy and the two procedures for its deployment [15]. This framework considers the following elements in fidelity evaluations: adherence, dose, quality of delivery, participant responsiveness and program differentiation. Results of the present study seem to indicate that the PVS-PREDIAPS implementation strategy to improve T2D primary prevention has been carried out with high degree of fidelity. Despite some differential exposure to overall strategy within the nursing staff of compared groups, professionals involved have been notably exposed to the implementation strategy and the planned program differentiation related to engagement of professionals and deployment of the implementation strategy has been attained.

Part of the present evaluation of PVS-PREDIAPS implementation strategy’s fidelity involves examining the implementation strategy dosage, that is, the degree of passive exposure to the planned implementation strategies and actions. In general, we can state that the professionals involved in both comparison groups had a notably high degree of exposure to the implementation strategy and that, as planned, the procedures for involving the professional groups and delivering the PVS-PREDIAPS strategy actions were executed differently in the two arms. Additionally, the exposure indicators suggest that professionals assigned to execute the strategy through sequential engagement of colleagues (starting with nurses who later sought the engagement of the physicians) had an unexpectedly higher degree of exposure to the implementation strategy actions, particularly in the case of nurses, the staff responsible for providing the active element of the clinical intervention. As described in the literature, commonly reported obstacles faced by physicians to fully engaging in implementation actions included heavy workload, staff turnover, difficulties in investing time and effort improvement initiatives beyond providing care, and existing practice priorities [21,22,23].

Dosage can also be evaluated subjectively, in terms of perceived receipt of the implementation strategies by the healthcare professionals. Although all healthcare professionals participating in the PREDIAPS trial were exposed to the 14 ERIC discrete implementation strategies with minor modifications, our qualitative evaluation of their experience indicates that they only recognized having received half of them. Many of the strategies that did not emerge from the surveys and group discussion related to “more structural” implementation actions like developing a formal implementation blueprint, changing record systems, or developing and organizing quality monitoring systems. This may be due to the fact that healthcare professionals are not implementation specialists and did not pay attention to or notice these changes. They also failed to identify several “ongoing” implementation activities, such as ongoing training and ongoing support, and local discussion and consensus sessions (collaborative modeling sessions). It is possible that these activities were seen to be typical, or standard, implementation tools that were too obvious to mention in the evaluation session. It is also possible that these strategies were not strong enough to be detectable. In any case, differential participation or exposure to the strategy could compromise the future implementation of the clinical intervention that we are seeking to promote [24]. Therefore, future analysis and interpretation of the main outcomes of interest should be adjusted for the degree of participation of professionals in general and/or exposure to the actions of the implementation strategy as a function of study arm and professional group [13, 23].

A second factor of interest in the evaluation of the fidelity of the execution of an implementation strategy is the extent of changes and adjustments that may have been made in the process of putting it into practice. Nevertheless, an adequate fit between “fidelity” of the strategy and the necessary “adaptability” to the local context of centers remains a great challenge in implementation trials [25]. Few modifications were made to the PVS-PREDIAPS implementation strategy. With respect to the planned implementation strategy [15], it proved unfeasible to carry out some activities, for example, the total number of planned monitoring and ongoing facilitation sessions. As commented previously, difficulty in scheduling meetings due to work overload or other competing priorities are common barriers faced by facilitators in improvement initiatives [23, 26]. There was demand among professionals for additional actions related to specific core strategies within the overall implementation strategy like training actions regarding the clinical intervention. Contextual factors, for example, site characteristics, needs and priorities are considered to be among the main drivers for tailoring implementation strategies [27], and one of the approaches used is to permit flexibility in order to enhance alignment and involvement while offering support and guidance towards change [26, 27].

Regarding participant responsiveness, all the ERIC discrete implementation strategies identified by the healthcare professionals in the evaluation session were described as beneficial. Specifically, 10 ERIC discrete strategies were perceived to be useful by at least one center. Such a positive evaluation is indicative of the high quality of the delivery of all strategies identified. Notably, however, two of these 10 strategies (conducting educational meetings and audit and feedback) were also mentioned as needing improvement by at least one individual in five of the nine centers. Nonetheless, the same two strategies were the most mentioned overall, and therefore also the most positively evaluated. The high frequency of mention within the evaluation session likely correlates with high exposure to these strategies, and this may imply more time for critical thinking about these specific strategies that were an important part of the implementation plan.

Interestingly, healthcare professionals also perceived receipt of ERIC strategies that were not necessarily specified a priori in the implementation plan or incorporated as part of a planned modification. These previously unidentified strategies included creating a learning collaborative, facilitation, and the mandating of change. Given the emphasis of the implementation strategy on facilitation to create a learning collaborative to develop and adapt the implementation strategy to an individual center’s context, it is not surprising that these two strategies were identified by at least half of the centers. The positive evaluation of these strategies seems to indicate the perceived value of external support and building teamwork in the successful implementation of complex interventions in the healthcare setting [26, 28]. Moreover, the participants felt that mandating change (socially and/or organizationally) fostered engagement in half the centers in the Global arm and built competence in one center in the Sequential arm. In the PREDIAPS trial, the perception of these additional discrete implementation strategies provides further evidence of an appropriate dose having been received and suggests further ecological validity of the overall implementation plan. Lastly, the lack of large qualitative differences in perceived receipt of implementation strategies observed between centers assigned to the Global and Sequential collaborative processes provides subjective evidence that exposure to planned implementation strategies seems to have been fairly similar, regardless of the deployment procedure.

The present study has some important limitations. The first, and possibly most important, is the small number of participating centers, which limits the potential generalizability of the findings to other PC centers in the Basque Country or other health systems. Second, the emphasis on the beneficial aspects of the implementation strategies identified in the qualitative inquiry may have biased our evaluation of participant responsiveness. The qualitative evaluation was carried out exclusively with the local leaders, who, though best positioned to provide related information, might have a different perspective to that of their colleagues. Furthermore, most of the open-ended questions posed in the evaluation session tended to be phrased positively in terms of usefulness. Future studies of participant responsiveness should consider researcher/interviewer bias in the design and realization of the evaluation. In interpreting our results, potential social desirability bias in participant responses should also not be ruled out.

Despite these limitations, a major strength of this study is the nature of the results obtained regarding fidelity, as they demonstrate that professionals involved were capable of identifying and rating the implementation actions conducted. Moreover, the data presented in this manuscript provides a practical example from the PREDIAPS trial that brings to life the core components of fidelity assembled from various existing frameworks. These components include adherence to the planned implementation strategies, dose/exposure to the strategies, quality of delivery, participant responsiveness to the strategies received, modifications made and program differentiation. Given the lack of operational definitions and existing frameworks to evaluate the fidelity of implementation strategies, this paper helps advance scientific research on fidelity.

The present fidelity evaluation has fulfilled some of its most important goals. In this sense, it seems to confirm the high quality of the implementation of the PVS-PREDIAPS strategy and of the two procedures for its deployment. Further, it will help to explain and interpret future results of the PREDIAPS trial, by rejecting the possibility of an implementation failure and by informing about potential confounding factors due to differences observed between comparison groups, these being potentially associated with both exposure and results. Lastly, it points out to some core elements of the implementation strategy that should be improved for future dissemination and scale-up, as for example the training component. In this sense, there is missed opportunity regarding the assessment of the training component of the strategy (eg., overall quality, satisfaction, etc.) that could lead to an improvement to ensure that professionals of any background and skills to be appropriately trained to deliver the intervention in a standardized way. Nevertheless, the majority of the demanded modifications were related to additional actions related to training in the clinical intervention.

Conclusions

The PVS-PREDIAPS implementation strategy to improve T2D primary prevention of the collaborating PC centers has been carried out with high degree of fidelity in each of the main measured dimensions. Despite some differential exposure to overall strategy in comparison groups, mainly in the nursing staff, professionals involved in both comparison groups have been notably exposed to the implementation strategy and the planned program differentiation related to engagement of professionals and deployment of the implementation strategy has been attained. Some minor unplanned reactive modifications have been required within the strategy responding to contextual circumstances related to centers’ work overload. Future analysis and interpretation of results pertaining to the main study will need to consider the mentioned differences in actual degree of exposure to implementation strategy’s actions within group and professional levels. The framework for fidelity evaluations stated by Dane [8] and adapted by Dusenbury [5], is a valuable framework that can be used to evaluate the implementation strategy-level fidelity. The FRAME framework [18] may be a useful complement in order to identify and report modifications made within the planned implementation strategy.

Availability of data and materials

Since data supporting the present study will mostly concern to process data and reports from specific professionals of the Basque Health Service-Osakidetza, it will be only shared upon justified request to the study guarantors.

Abbreviations

T2D:

Type-2 diabetes

PC:

Primary Care

MRC:

Medical Research Council

FRAME:

Framework for Reporting Adaptations and Modifications-Enhanced

PDSA:

Plan-Do-Study-Act

ERIC:

Expert Recommendations for Implementing Change

References

  1. 1.

    Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2:40.

    Article  Google Scholar 

  2. 2.

    Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38:65–76.

    Article  Google Scholar 

  3. 3.

    Cross WF, West JC. Examining implementer fidelity: conceptualizing and measuring adherence and competence. J Child Serv. 2011;6:18–33.

    Article  Google Scholar 

  4. 4.

    Slaughter SE, Hill JN, Snelgrove-Clarke E. What is the extent and quality of documentation and reporting of fidelity to implementation strategies: a scoping review. Implement Sci. 2015;10:129.

    Article  Google Scholar 

  5. 5.

    Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18:237–256. 2.

  6. 6.

    Kemp CG, Wagenaar BH, Haroz EE. Expanding hybrid studies for implementation research: intervention, implementation strategy, and context. Front Public Health. 2019;7:325.

    Article  Google Scholar 

  7. 7.

    Gearing RE, El-Bassel N, Ghesquiere A, Baldwin S, Gillies J, Ngeow E. Major ingredients of fidelity: a review and scientific guide to improving quality of intervention research implementation. Clin Psychol Rev. 2011;31:79–88.

    Article  Google Scholar 

  8. 8.

    Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are implementation effects out of control? Clin Psychol Rev. 1998;18:23–45.

    CAS  Article  Google Scholar 

  9. 9.

    Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14:15.

    Article  Google Scholar 

  10. 10.

    Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6:134–47.

    Article  Google Scholar 

  11. 11.

    Dobson KS, Singer AR. Definitional and practical issues in the assessment of treatment fidelity. Clin Psychol Rev. 2005;12:384–7.

    Google Scholar 

  12. 12.

    Glasgow RE, Lichenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–7.

    Article  Google Scholar 

  13. 13.

    Sánchez A, Silvestre C, Campo N, Grandes G. PreDE research group. Type-2 diabetes primary prevention program implemented in routine primary care: a process evaluation study. Trials. 2016;17(1):254.

    Article  Google Scholar 

  14. 14.

    Butel J, Braun KL, Novotny R, Acosta M, Castro R, Fleming T, Powers J, Nigg CR. Assessing intervention fidelity in a multi-level, multi-component, multi-site program: the Children’s healthy living (CHL) program. Transl Behav Med. 2015;5(4):460–9.

    Article  Google Scholar 

  15. 15.

    Sanchez A, Grandes G, Pablo S, Espinosa M, Torres A, García-Alvarez A. PREDIAPS Group. Engaging primary care professionals in collaborative processes for 28rioritiza type 2 diabetes prevention practice: the PREDIAPS cluster 28rioritiza type II hybrid implementation trial. Implement Sci. 2018;13(1):94.

    Article  Google Scholar 

  16. 16.

    Sanchez A, Silvestre C, Cortazar A, Bellido V, Ezkurra P, Rotaeche Del Campo R, et al. Prevención de la diabetes mellitus tipo 2 en Atención Primaria de Salud mediante estilos de vida saludables: cuidados e intervención clínica recomendada. Vitoria-Gasteiz: Department of Health, Government of the Basque Country; 2016.

    Google Scholar 

  17. 17.

    Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. 2015;10:21.

    Article  Google Scholar 

  18. 18.

    Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58.

    Article  Google Scholar 

  19. 19.

    Huynh AK, Hamilton AB, Farmer MM, Bean-Mayberry B, Stirman SW, Moin T, Finley EP. A pragmatic approach to guide implementation evaluation research: strategy mapping for complex interventions. Front Public Health. 2018;6:134.

    Article  Google Scholar 

  20. 20.

    Domínguez-Berjón MF, Borrell C, Cano-Serral G, Esnaola S, Nolasco A, et al. Constructing a deprivation index based on census data in large Spanish cities (the MEDEA project). Gac Sanit. 2008;22:179–87.

    Article  Google Scholar 

  21. 21.

    Parchman ML, Anderson ML, Dorr DA, Fagnan LJ, O’Meara ES, Tuzzio L, Penfold RB, Cook AJ, Hummel J, Conway C, Cholan R, Baldwin LM. A randomized trial of external practice support to improve cardiovascular risk factors in primary care. Ann Fam Med. 2019;17(Suppl 1):S40–9.

    Article  Google Scholar 

  22. 22.

    Taylor EF, Peikes D, Genevro J, Meyers D. Creating Capacity for Improvement in Primary Care. The Case for Developing a Quality Improvement Infrastructure. Rockville: Agency for Healthcare Research and Quality; 2013. http://www.ahrq.gov/professionals/prevention-chronic-care/improve/capacity-building/pcmhqi1.html. Published Apr 2013. Accessed May 9, 2020.

  23. 23.

    Mold JW, Walsh M, Chou AF, Homco JB. The alarming rate of major disruptive events in primary care practices in Oklahoma. Ann Fam Med. 2018;16(Suppl 1):S52–7.

    Article  Google Scholar 

  24. 24.

    Egeland KM, Ruud T, Ogden T, Färdig R, Lindstrøm JC, Heiervang KS. How to implement illness management and recovery (IMR) in mental health service settings: evaluation of the implementation strategy. Int J Ment Health Syst. 2017;11:13.

    Article  Google Scholar 

  25. 25.

    Cohen DJ, Crabtree BF, Etz RS, Balasubramanian BA, Donahue KE, Leviton LC, et al. Fidelity versus flexibility: translating evidence-based research into practice. Am J Prev Med. 2008;35:S381–9.

    Article  Google Scholar 

  26. 26.

    Liddy C, Singh J, Guo M, Hogg W. Physician perspectives on a tailored multifaceted primary care practice facilitation intervention for improvement of cardiovascular care. Fam Pract. 2016;33(1):89–94.

    Article  Google Scholar 

  27. 27.

    Nguyen AM, Cuthel A, Padgett DK, Niles P, Rogers E, Pham-Singer H, Ferran D, Kaplan SA, Berry C, Shelley D. How Practice Facilitation Strategies Differ by Practice Context. J Gen Intern Med. 2019 Oct 21. Doi: https://0-doi-org.brum.beds.ac.uk/10.1007/s11606-019-05350-7. [Epub ahead of print].

  28. 28.

    Rogers ES, Cuthel AM, Berry CA, Kaplan SA, Shelley DR. Clinician perspectives on the benefits of practice facilitation for small primary care practices. Ann Fam Med. 2019;17(Suppl 1):S17–23.

    Article  Google Scholar 

Download references

Acknowledgements

The Prediaps Group:

ALANGO PC center (OSI Uribe): Nurses: Amaia Bengoetxea, Olga Galarza, Elsa Martínez, Itziar Zalduegi, Dorothea Chausson, Agurtzane Gorroño, Alicia Pollán, Marisol Bernabéu, MªYolanda Calvo, Ander Artiagoitia, Nerea Zaramillo, Lidia Gonzalez, and Asier Aurrekoetexea; Family Physicians: Jon Azkarate, María Muñoa, Mar Bilbao, Vicki Camineiro, Gonzalo Gómez de Iturriaga, Fernando Gago, Iciar Ochoa de Retana, Ana Zorrilla, MªLuisa Gutiérrez, Jone Capetillo Serra, and Mª Nieves Lopez.

ERANDIO PC center (OSI Uribe): Nurses: Nekane Iguerregui, Dolores López, Maite Gastañaga, Antonia Flores, Marcos Pereda, Amaya González, Ana Castresana, Laura Saiz, Nerea Regulez, and Estibaliz Peciña; Family Physicians: Jasone De la Plaza, Lucía Irastoza, Jose Contreras, Idoia Etxebarria, Begoña Oleaga, Cristina Herrero, Nora Cabezón, Fátima Calvo, J Manuel Llamazares, Mª Ángeles Gutierrez, and Monica Prieto.

ZUAZO PC center (OSI Barakaldo-Sestao): Nurses: Concepción Estébanez, María José Cordovilla, Alicia Domínguez, Isabel Lázaro, Elena Resines, Yolanda Villalba, and Begoña Ayerdi; Family Physicians: Florencia Martín, Magdalena Presmanes, Floreal Crespo, Araceli Benito, Mª Belén Molina, Mª Mar García, Mª Gracia Díaz, Mª Luisa Rodriguez Ortiz de Zarate, Rebeca San Cristóbal, and M Zugazaga Prieto.

SAN VICENTE PC center (OSI Barakaldo-Sestao): Nurses: Pedro Martínez, Mercedes Crespo, Estíbaliz Albitre, Adelina García-Roldán, Amaia García, Maite Castro, Iñaki Gorospe, Amelia V. Hernández, Maite López, and Mirian Sainz; Family Physicians: Irene Marín, María Jesús Aragón, Leire Ulayar, Encarnación Santamaría, Carmen Sánchez, Javier Bayo, Begoña Urkullu, Ana Inés Pereda, and Mercedes Garcia.

PORTUGALETE PC center (OSI Ezkerraldea-Ekarterri-Cruces): Nurses: Pilar Blanco, Silvia Soler Valverde, Jose I. Atela, Hiart Trespalacios, Anabel Llarena, and Verónica Ruiz; Family Physicians: Begoña Cabieces, Concepción Ugarte, Guadalupe Icaza, Edurne Zubeldia, Idoia González, and Ángeles Gayo.

ZALLA PC center (OSI Ezkerraldea-Ekarterri-Cruces): Nurses: Itxaso Arévalo, Gloria Intxausti, Esther García, Teresa Sánchez, Igone Lobato, Noelia Fuente, Naiara Ortolachipi, and Edelweiss Sánchez; Family Physicians: Victoria Cosgaya, Ángeles Gayo, Arantza Azazeta, Patricia Zaballa, Ana Isabel Ramila, and Teresa Rodeño.

SODUPE PC center (OSI Ezkerraldea-Ekarterri-Cruces): Nurses: Inmaculada Rodríguez, Teresa Vázquez, Raquel Ruíz, Rosa Herrero, María Valvanera, and Saioa Setién; Family Physicians: Begoña Ruíz, Juan José Casas, Joana Clemente, Javier Amiama, and Javier Angulo.

IZTIETA PC center (OSI Donostialdea): Nurses: Belén Aramendia, Soledad Asenjo, Mª Sonia Mayoral, Remedios Oyarzun, Bergoi Calvar, Belinda Zulueta, Edurne Elola, Josu Egaña, Gemma Díaz, Mª Ángeles Sola, Laura Balague, Mª Ángeles Ganzarain, Arantxa Aramburu, Ana Mª Guinea, Edurne Lizarazu, Inma Valverde, Nekane Arenas, and Susana Alonso; Family Physicians: Rosa Salaberria, Javier Merino, Mercedes Álvarez, Ester Lázaro, Juncal Izcara, Leire González, Ainhoa García Leunda, Idoia Sánchez, Esther Usandiaga, and Eluska Yetano.

EGIA PC center (OSI Donostialdea): Nurses: Jaione Larrea, Inés Mendinueta, Asún Uria, Ana Belén Gaztañaga, Eva Mayo, Onintza Aranzadi, Eulalia Medina, Rosa González, Ione Gutiérrez, Arantxa Perez, and Mª Ángeles Izquierdo; Family Physicians: Alejandro García, Ainhoa Ugarte, Mª Teresa Zubeldia, Bingen Uriondo, Mª Carmen Aranegui, Arantxa Mendiguren, Yolanda Fernández, Maite Zapirain, Mª Jose Garín, Aitziber Ayerbe, and Jon Urkia.

COORDINATING RESEARCH TEAM: Director Team: Alvaro Sánchez, Josep Cortada (Deusto PC center), Esther Gorostiza (Matiena PC center), Susana Pablo, Heather Lynn Rogers, Arturo García-Alvarez, Gonzalo Grandes; Clinical Commitee: Alicia Cortazar (Cruces Hospital), Virginia Bellido (Cruces Hospital), Patxi Ezkurra (Zumaia PC center), and Rafa Rotaetxe (Alza PC center).

Consent to publication

Not applicable.

Funding

This study has been funded by Instituto de Salud Carlos III through several project grants: “PI15/00350; PI18/00952; RD12/0005/0010 and RD16/0007/0002” (Co-funded by the European Regional Development Fund “A way to make Europe “); and by the Health Department of the Basque Government (EXP: 201711086). The funding body had no role in the design of the study, collection, analysis, or interpretation of data or writing of the manuscript.

Author information

Affiliations

Authors

Consortia

Contributions

GG and AS conceived the idea and are the study guarantors. They were primarily responsible for the study design and planning, obtained funding, and are responsible for project coordination and supervision, analysis and interpretation of results and manuscript preparation. HLR and SP collaborated in the study design, obtained funding, and contributed to study coordination, interpretation of results and manuscript preparation. AS, SP and HLR were responsible for the analysis of results and critically reviewed the manuscript. They led the implementation process and participated in fieldwork at each of the collaborating centers. They also collaborated in obtaining data and in the analysis and interpretation of study results. The PREDIAPS group contributed to the fieldwork for this study in relation to patient recruitment, intervention deployment and data collection. This study is an action-research project and all collaborators are members of the research team. All contributors approved this version submitted for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Alvaro Sánchez.

Ethics declarations

Ethics approval and consent to participate

The research protocol has been approved by the Basque Country Clinical Research Ethics Committee (Ref. no.: 08/2015). The Primary Care Research Unit of Bizkaia is explicitly authorized by the Healthcare Management of the Basque Health System to extract and use data from the electronic health records for research purposes. Written informed consent to participate in the study was obtained from all the participating Physicians and Nurses.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Fig. 1.
figure1

Participation of PC professionals in the PREDIAPS strategy in the planned implementation sessions (n = 10)

Table 4 The PVS-PREDIAPS clinical intervention reported according to the Template for Intervention Description and Replication (TIDieR) checklist
Table 5 Evaluation of PVS-PREDIAPS discrete implementation strategies as cataloged by the Expert Recommendations for Implementing Change (ERIC) taxonomy

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sánchez, A., Rogers, H.L., Pablo, S. et al. Fidelity evaluation of the compared procedures for conducting the PVS-PREDIAPS implementation strategy to optimize diabetes prevention in primary care. BMC Fam Pract 22, 34 (2021). https://0-doi-org.brum.beds.ac.uk/10.1186/s12875-021-01378-z

Download citation

Keywords

  • Fidelity
  • Implementation strategy
  • Diabetes prevention
  • Primary healthcare