Skip to main content

Formative evaluation and adaptation of pre-and early implementation of diabetes shared medical appointments to maximize sustainability and adoption

Abstract

Background

Understanding the many factors that influence implementation of new programs, in addition to their success or failure, is extraordinarily complex. This qualitative study examines the implementation and adaptation process of two linked clinical programs within Primary Care, diabetes shared medical appointments (SMAs) and a reciprocal Peer-to-Peer (P2P) support program for patients with poorly controlled diabetes, through the lens of the Consolidated Framework for Implementation Research (CFIR). We illustrate the role and importance of pre-implementation interviews for guiding ongoing adaptations to improve implementation of a clinical program, achieve optimal change, and avoid type III errors.

Methods

We conducted 28 semi-structured phone interviews between September of 2013 and May of 2016, four to seven interviewees at each site. The interviewees were physician champions, chiefs of primary care, pharmacists, dieticians, nurses, health psychologists, peer facilitators, and research coordinators. Modifiable barriers and facilitators to implementation were identified and adaptations documented. Data analysis started with immersion in the data to obtain a sense of the whole and then by cataloging principal themes per CFIR constructs. An iterative consensus-building process was used to code. CFIR constructs were then ranked and compared by the researchers.

Results

We identified a subset of CFIR constructs that are most likely to play a role in the effectiveness of the diabetes SMAs and P2P program based on our work with the participating sites to date. Through the identification of barriers and facilitators, a subset of CFIR constructs arose, including evidence strength and quality, relative advantage, adaptability, complexity, patient needs and resources, compatibility, leadership engagement, available resources, knowledge and beliefs, and champions.

Conclusions

We described our method for identification of contextual factors that influenced implementation of complex diabetes clinical programs - SMAs and P2P. The qualitative phone interviews aided implementation through the identification of modifiable barriers or conversely, actionable findings. Implementation projects, and certainly clinical programs, do not have unlimited resources and these interviews allowed us to determine which facets to target and act on for each site. As the study progresses, these findings will be compared and correlated to outcome measures. This comprehensive adaptation data collection will also facilitate and enhance understanding of the future success or lack of success of implementation and inform potential for translation and public health impact. The approach of using the CFIR to guide us to actionable findings and help us better understand barriers and facilitators has broad applicability and can be used by other projects to guide, adapt, and improve implementation of research into practice.

Trial registration

ClinicalTrials.gov ID: NCT02132676.

Peer Review reports

Background

Understanding the many factors that influence implementation of new programs, in addition to their success or failure, is extraordinarily complex. Formative evaluation, defined as a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts, is an essential means to systematically approach this complexity [1].

Many implementation studies rely exclusively on summative data-- or data outputs, products, and outcomes—to determine program success or failure. While summative data is useful, it is not adequate to understand critically important implementation processes [1, 2]. When used in isolation, summative data often leads to the term “implementation black box” [3]: there is no way to understand the specific reasons the intervention succeeded or failed, how it was actually implemented, and how local contextual factors affected implementation. This is where formative evaluation comes in, to fill in these gaps and to systematically examine key features of the local implementation setting, detect and monitor unanticipated events and adjust if necessary in real-time, optimize implementation to improve potential for success, and avoid type III errors—the failure to detect differences between the original intervention plan and the ultimate manner of implementation that lead to failure to achieve outcomes [1, 4]. This understanding is essential for efforts to sustain, scale up, and disseminate any new program–otherwise there is potential for failure to account for specific contextual issues in program implementation.

Adaptations have been found to be necessary for sustainable implementation [5] and are considered part of the traditional translation pipeline (adaptations as required) [6]. Progress has been made in advancing the science of implementation, but too often the complexity of translating research into practice is overlooked or follows an overly simplistic model [7]. In the words of implementation experts Dr. Chambers and Dr. Norton, “Rather than assuming that adaptation of a manualized intervention is at odds with good implementation, the field can systematically collect information on the impact of adaptation to individuals, organizations, and communities and use this information to extend the knowledge base of implementation of evidence-based practices as well as ongoing improvement of the evidence-based practices themselves” [8]. Indeed other efforts in implementation have been described where adaptations are catalogued on an ongoing basis. [9] While the static view of adaptation has been that it is bad or to be avoided and/or eliminated, the dynamic sustainability model believes adaptation to be “inevitable and encouraged.” [7] Likewise, simplified intervention implementation overlooks the complexities of translating research to practice and relies on a set of assumptions that limits enhancement of fit between evidence-based interventions and delivery setting. [10]

Additionally, while the evidence-to-practice gap for interventions is receiving attention, it tends to be understudied in primary care. [11] A recent study highlighted the importance of paying attention to context, which was noted as frequently failing to be acknowledged, described, or taken into account during implementation in primary care. [11]

Accordingly, to fill this gap in the literature, we sought to illustrate the role and importance of pre-implementation (early) interviews for guiding ongoing adaptations to improve implementation of a clinical program, achieve optimal change, and avoid type III errors. We gathered detailed pre-implementation data across five health system sites, within primary care settings, that had each committed to institute Shared Medical Appointments (SMAs) and in some SMA cohorts an additional offered mutual peer support program (P2P) for adult patients with poorly controlled diabetes. In particular, we examined modifiable barriers and facilitators to implementation. We then documented the adaptations that were made in real-time to attempt to improve implementation. This comprehensive data will also facilitate and enhance understanding of the future success or lack of success of implementation of new innovative clinical programs such as SMAs and inform potential for translation and public health impact of these [2].

Methods

This qualitative study is part of a larger implementation study, “The Shared Health Appointments and Reciprocal Enhanced Support (SHARES) study” [12]. The SHARES study is a multi-site cluster randomized trial of five geographically diverse Veterans Affairs (VA) health systems evaluating the effectiveness and implementation of diabetes Shared Medical Appointments (SMAs) with and without an additional reciprocal Peer-to-Peer (P2P) support program, when compared to usual care.

SMAs bring patients with the same chronic condition together with an interdisciplinary team of providers to provide shared education and support. The diabetes SMAs consist of a series of 1–2 h sessions of 8–10 patients led by a team of health professionals. At each site, participants have a total of approximately 8 h of sessions, and the sessions are intended to be interactive and focus on key diabetes self-management topics (e.g., diet, medications, physical activity, self-monitoring). The P2P program comprises periodic peer support, [13, 14] group sessions, and telephone contact between SMA participant pairs to promote more effective diabetes self-management and sustain gains achieved through the SMAs after completion of these sessions. Outcomes will be examined across three different treatment groups: (1) SMAs; (2) SMAs plus P2P; and 3) usual care.

We undertook a type of formative evaluation (FE), labeled implementation-focused evaluation, of the SMAs and of P2P [1]. This type of FE occurs throughout implementation of the project plan: before, during, and after implementation. This manuscript focuses on the real-time FE that took place during the early stages of implementation and pre-implementation.

The overarching framework for the SHARES qualitative discovery-- including the interviews, formative evaluation, implementation, and analysis-- was the Consolidated Framework for Implementation Research (CFIR) [15]. The CFIR provides a framework of 39 constructs from across published implementation frameworks that describe the organizational and contextual setting and are believed to influence implementation. We are tracking the CFIR constructs across the sites throughout the project and will ascertain how they influence implementation success during future program evaluation.

Institutional Review Board (IRB) approval was obtained from the Central IRB.

Data collection and analysis

A brief phone survey was conducted with key informants from each site to identify potential key CFIR constructs. We then conducted 28 semi-structured phone interviews with participants at five VA health systems, four to seven interviewees at each site (see Table 1). Interviews were conducted between September of 2013 and May of 2016 and lasted between 22 and 56 min (mean of 34 min). Two interviewers participated in all interviews; detailed notes were taken by both, while most of the questions were asked by the first interviewer. The interviewees were physician champions, chiefs of primary care, pharmacists, dieticians, nurses, health psychologists, P2P facilitators, and research coordinators from the local sites. After each initial interview, the interviewee was then asked to recommend other informants. This type of recruitment, called snowball sampling, means that rather than determining individuals to interview ahead of time, we asked everyone that we interviewed to recommend other potential participants, including key clinical opinion leaders. Interviewees from each site were continually recruited until the research team felt that qualitative data discovery had reached saturation—the point at which new data only confirmed the themes and conclusions already achieved [16].

Table 1 Qualitative interviewee titles

Phone interviews were qualitative and, unlike a survey, all questions were open-ended; interviewees were encouraged to share their experiences in detail to enable a thorough understanding of the implementation experience from their varying perspectives. Conducting multiple interviews at each site enabled us to understand how perspectives compared from staff in different positions.

Our interview guide included questions about the interviewee’s role in the diabetes SMAs and P2P groups, aspects of the program they would like to change, barriers and facilitators with regards to implementing/expanding the diabetes SMAs and P2P programs, what kinds of resources or tools they would need for implementation, need for and awareness of evidence for the P2P program in addition to SMAs, existence of a clinical champion or local opinion leader, and types of feedback they would like to receive as the study progressed. Although CFIR-relevant questions were asked, there were opportunities to explore non-CFIR issues in that the interviewees could discuss anything related to their implementation experience, which was described by the participants in detail. After each interview, we developed a list of barriers, facilitators, and tasks that we needed to accomplish or follow up on with key staff at each of the local sites as part of the implementation process.

The initial lists of barriers and facilitators were developed from the detailed interview notes with a quick turnaround time to implement an immediate feedback loop to the sites because the primary consideration was not collecting data for “research,” but rather to implement adaptations to overcome barriers and improve the implementation process. After these barriers were reviewed, the implementation team worked with the local staff from each site to make adaptations as necessary. Of note, some of the adaptations were driven by local clinical staff (see Table 2 for these details). We shared feedback with each local site continually while the phone interviews progressed so that suggestions could be incorporated in a timely manner. Each of these items was addressed to the extent that it could be prior to a mid-implementation site visit. For any barriers that we had not been able to address before each site visit, a detailed summary packet was written by the project qualitative analyst (CPK) and distributed to the implementation team members attending the site visit, with background information and a list of questions and tasks that still needed to be addressed. In addition, conference calls with individual sites occurred throughout this study period and detailed notes about important issues were documented and included in analysis.

Table 2 Detailed adaptations made in real-time by site and CFIR construct

Subsequent data analysis started with immersion in the data to obtain a sense of the whole and then by cataloging principal themes that emerged according to the CFIR framework constructs [17, 18]. This is a type of qualitative directed-content analysis [19] (directed initially by the CFIR constructs). Given the CFIR has 39 broad constructs related to implementation, although we analyzed, we did not discover any strong themes outside of the organizational parameters outlined in the framework.

Two authors (CK and MV) used an iterative consensus-building process to code; first each team member independently coded transcripts, and then met as a group to discuss and reconcile codes, identify emergent themes, and resolve discrepancies through consensus. Each of the CFIR constructs was ranked on a scale of − 2 to + 2 (Table 3) independently by CK and MV for each site and then were discussed until consensus ranking was reached, taking all of the qualitative data into account. The valence of each construct reflects the impact on implementation (negative or positive); the numbers provide a reference for the impact on implementation as weak or strong, with 2 being the strongest. [20]

Table 3 CFIR construct ranking after pre-implementation phone interviews. Rating − 2 to + 2

Results

Key constructs or areas of focus varied across sites. Table 2 is organized by site and CFIR construct and within that row, highlights how the findings (barrier column) informed our actions to improve or facilitate implementation (facilitator column) and what actions were taken (adaptations).

Based on our work with the participating sites to date, we identified a subset of CFIR constructs that are most likely to play a role in the implementation and possibly effectiveness of the diabetes SMAs and the P2P program, including evidence strength and quality, relative advantage, adaptability, complexity, patient needs and resources, compatibility, leadership engagement, available resources, knowledge and beliefs, and champions (Table 4). Although data from any organizational aspect mentioned by interviewees, and all CFIR constructs, were coded (see Table 2), these specific constructs formed the basis of primary qualitative analyses due to the depth and frequency of the construct throughout the interviews and qualitative analysis. A definition of each CFIR construct is included in Table 2.

Table 4 CFIR construct importance as ranked by local sites

Evidence strength and quality

Diabetes SMA

The evidence strength and quality construct constitutes stakeholders’ perceptions of the quality and validity of evidence supporting the belief that the innovation will have desired outcomes. This construct was rated positively for all sites except 1002 (see Table 2), where every interviewee mentioned that the diabetes SMAs were seen as extra work without added value or belief that the innovation will have the desired outcomes. 1002 nurse summarized these thoughts: “I’ve got to tell you, it’s a hard sell with physicians. Even now, I don’t have a champion for the diabetes SMA. They see it as extra work. They don’t see the added value. It troubles me a lot that it’s so hard to get the docs involved.” After this barrier was discovered, our facilitation team made a visit to present evidence and met with the local primary care team to help educate and influence the physicians.

Diabetes SMA and P2P

The remaining four sites’ interviews demonstrated local staff belief in the positive evidence quality for both the diabetes SMAs and P2P. A leader from site 1004 stated, “I think [SMA and P2P] is another means for providing guidance and motivation for patients with diabetes struggling with their glucose to meet goals. I think we do a lot of telling patients what to do, and I really think there’s a benefit to hearing from ‘equals’ rather than somebody else—someone lateral as opposed to top down with the guiding and coaching.” A Primary Care Physician (PCP) from site 1005 explained, “No matter how much I say, ‘Yeah, I know diabetes does this, diabetes does that,’ I don’t have to deal with it and there’s one guy saying, ‘Yeah, my blood sugar’s like this, I made this little change and it dropped like, you know, dropped 20 or 30 points,’ and they believe them more.” Sites 1001 and 1003 also mentioned that local evidence had shown that having a peer or buddy for support, as is the case in the diabetes SMAs and the P2P pairs and group sessions, helps patients with diabetes.

Relative advantage

Diabetes SMA

Perceptions of the relative advantage of diabetes SMA programs were mixed across sites. Those sites that had an overall positive ranking for stakeholder’s perception of the advantage of these programs were 1003 and 1004. As an example, a 1003 health psychologist stated, “We do have also a diabetes education class… There’s a great bit of information given. There is only a very small percentage of diabetes patients that have not taken that class because it is essentially mandated. But, we’re finding that for some reason this doesn’t translate into action.” The need for and potential advantages of the peer support program in this population was mentioned at site 1004: “…especially in Veteran population because they’re deployed in a unit and they come back in a unit. So the peer support would be even more effective in theory than in the general population…”.

There were also some references from clinicians about advantages to leading group visits instead of one-on-one patient appointments and that format provided a means to hear more detailed information about patient behaviors, “I love talking to patients but I get tired of half-hour slots, so anything that kind of breaks up my clinic, it’s a slightly different format…and it’s a chance to kind of listen to the more of the social stories and kind of what’s going in [food] why do they eat at Coney Island every day and their diet hasn’t changed, or why they’re not going to change that.”

Sites 1001 and 1002 had negative ratings; staff did not see an advantage compared to usual care or likewise because they saw the SMAs as added worked, without any added value. For those sites with a negative ranking, we asked local staff to present information about the value and advantage to their local PCPs. During our site visits, we also presented evidence to the primary care staff from the literature, engaged them, and answered any questions they had.

Diabetes SMA and P2P

Another noted advantage (see Cost in Table 2) was the belief that the SMAs and P2P program will improve efficiency of care and, therefore, increase patient access and decrease cost. A physician leader from site 1004 said, “I also think that it’s a good way to cut costs from healthcare because if a peer mentor can help the patient, he can remind him to follow his appointments, take his medications, exercise; it’s much simpler than a health professional trying to do the same thing while juggling other things. I think it helps from the clinical aspect, as well as having a financial benefit…” A site 1005 clinical leader said, “Let that dietician go over information with 8-12 people at one time, instead of one at a time. Those kinds of efficiencies are really great and I think it will help my dieticians, my clinical pharmacists, and my psychologist a lot, and I think it will help the physicians to manage their population of patients.”

Adaptabiliy

Diabetes SMA

Sites 1001 and 1002 had concerns initially that the SMA program was not adaptable. They thought that the overseeing site would be dictating the content and manner that each of the SMA sessions would be run. We worked with both sites continually through phone calls and virtual meetings to explain that the local team had flexibility and control over the SMA sessions and that our fidelity assessment would help to account for any differences across sites.

Site 1004 had a positive rating for adaptability of SMAs because they realized the importance of flexibility and had tailored their SMA sessions to be adaptable so Veterans could get what “they want and need out of each session.” Staff at this site also had a good understanding that we wanted the local clinical programs to be adaptable to fit local context.

P2P

Views on the potential adaptability of the peer support component of the program were mixed. Site 1001 had concerns that the program could not be adapted for patients who did not mesh well with their assigned peer. We worked with that site to come up with an adapted plan whereby a patient could be easily re-paired, even with a patient outside their cohort if need be, or assigned to a new group of 3 patients. We also implemented a way of working with the local nurses and PCPs to get their recommendations on patient pairs that would work well together. Site 1001 also had some issues surrounding their locally developed recruitment plan that the clinical and research teams thought was not feasible. During the site visit, we discussed this with the local team and they adapted their recruitment plan to be more realistic.

Complexity

Diabetes SMA and P2P

Interestingly, site 1001 staff had no concerns, even when prompted, about staff being so busy that it would be difficult to do anything additional or complex. However, that was the only site with a positive ranking. The other four sites expressed concerns that if the programs were complex, this would make the programs much more difficult to implement. An Associate Chief of Staff at 1005 said, “I am very supportive of this, but it can’t add work to my people. It will have to be very efficient.” One specific complexity concern was in terms of the clinical notes and documentation required for each SMA visit and that the notes would be cumbersome for the clinical pharmacist, who was the documenter at one site. “I just want the pharmacist who will be involved to write very short, patient specific changes. That is going to be VERY important to me. If it is burdensome to them, they are not going to be able to continue.” The facilitation team worked with the site through team meetings and in-person discussions with the clinical pharmacists to streamline the documentation and share diabetes SMA clinical note templates across the sites.

Sites 1002, 1003, 1004, and 1005 all expressed concern with staff being busy and that it would be difficult to add additional programs and find staff time to help facilitate the SMA and/or P2P group sessions. To impact the complexity barriers, the facilitation team worked with the local sites through team meetings and phone calls to streamline documentation and shared diabetes SMA clinical note templates across the sites. Additional efforts were made to integrate the process within existing workflow--for example, allowing the clinician who documented the diabetes SMA notes to be adaptable: in some sites a clinical pharmacist, others a nurse practitioner, or primary care physician. Likewise, the role of the P2P facilitator varied.

Patient needs and resources

Diabetes SMA and P2P

All sites except 1005 perceived the new programs as facing multiple barriers within the CFIR construct of addressing patient needs and resources. This was the most negatively ranked construct overall. There were a plethora of barriers across the sites, including staff concerns about patients being guarded and not wanting to pair up or exchange numbers, lack of motivation in patient population, low patient attendance especially without financial compensation for the P2P or SMA visits, difficulties with patient recruitment, long distance drives for patients to the hospital, patients’ lack of financial resources, patient resistance to change, and lack of patient follow-through when asked to bring in or complete materials or goals. A clinical pharmacist from site 1003 expressed it this way, “There are many obstacles to bringing the patients in… [to SMAs or P2P group sessions] Our population tends to be older and on the financial scale of things, having more difficulty. Those factors set into place some natural barriers to being compliant with appointments.”

A primary care physician at site 1001 stated, “Definitely the top barrier will be convincing the patients to show up. We invite an average of 10 people and we usually have between 4 and 7 who come and continue to show up. I think patient buy-in is definitely a barrier.” Site 1001 health psychologist, “I can say generally we have a hard time getting patients to come to groups here. We’re trying to hold them first thing in the morning so that parking will be easier, but parking is a huge barrier. Patients don’t want to come to anything that they perceive as extra a lot of the time, because they find it so challenging to actually physically get here, get parked and get to their appointment.”

Site 1004 research coordinator, “People with diabetes don’t feel well and getting them involved in something—it is difficult. And some of our patients are still working, so if we have classes during the day, that’s an obstacle. And transportation. We’ve had that experience in the past where they’d like to join but they can’t get here.

To overcome these barriers the facilitation team worked to make sure that patients would be eligible for travel pay for attendance to the SMA clinical appointments and helped with attendance by making additional reminder calls and sending reminder letters. We worked with all sites to ensure that, when possible, appointments were scheduled at a convenient time for patients (also considering which time of day each facility has the most parking availability). We consulted with staff from the sites to consider their perspective on matching peers together and who would work best together and again, allowing for adaptability and patient re-pairing. We also worked with sites to ensure whenever possible that these facilitators would be sustainable across time when the research team would no longer be involved (transference of some of these tasks in time to local clinical and administrative staff).

Compatibility

Diabetes SMA and P2P

Perceptions of the compatibility of the SMA and P2P programs were either mixed, neutral or positive for all sites. Site 1001 and 1003 staff were confident that the implementation process would be smooth because both programs were designed in a way that they believed was compatible and fit within their existing workflow and programs. At site 1004, we heard from multiple staff that it was very important for us to make sure the programs were integrated into their suite of already existing programs or there would be push-back from staff. To do so, we spoke with several front-line staff about how to best integrate the programs with their existing work. The Chief of Primary Care at this site also worked with us to present information at their monthly staff meetings.

Other compatibility adaptations meant that we were very flexible about the type of clinicians who could facilitate the SMAs—the main facilitators varied between clinical pharmacists, nurses, dieticians, and primary care providers; as long as multiple types of clinicians were involved, the lead was adaptable to best suit their local needs and staffing considerations. Additionally, we were flexible about the role of the P2P facilitators; Veterans with and without diabetes, research associates, and volunteers.

Leadership engagement

Diabetes SMA and P2P

Leadership engagement was ranked positively at all sites except one. Leaders were considered engaged when multiple interviewees expressed that a leader did things such as: help with convincing providers to enroll patients, block out time for clinicians to facilitate the SMAs and staff to facilitate the P2P groups, guarantee space for the SMA and P2P programs to be held, garner general support from staff, help with results/outcome feedback to staff, lead from a high level, and express to staff why they feel these programs are important. At site 1002 where leadership was lacking, a named leader for the project was mentioned by local staff as not being influential. Additionally, there were some issues with general lack of high leadership support at this site (beyond the scope of this project).

Available resources

Diabetes SMA and P2P

The availability of resources was ranked the same negative value across all sites (− 1). Issues of concern included space constraints for SMA and P2P group visits, lack of patient parking, SMA and P2P facilitator staff leaving after trained, staff generally busy and stretched thin, competing initiatives, overall information overload, and being short-staffed, “We are short-staffed right now and we are unable to hire people.” Space was so limited at site 1002 that SMA sessions had to be scheduled based on room availability, rather than staff availability.

The facilitation team helped to find guaranteed rooms, scheduled SMAs and P2P sessions early so rooms could be booked well in advance, wrote scripts for staff, and had monthly training for the P2P facilitators to ease their workload and make it easier to understand their role.

Knowledge and beliefs

Diabetes SMA and P2P

There were concerns from several sites stemming from beliefs that the patients in the SMAs or as part of their pairing would share incorrect clinical information. Site 1002 health psychologist explained: “And one other barrier potentially could be misinformation, people talk and sometimes myths get out there and misconceptions and what they hear by word of mouth which is not very accurate information so…the group sessions, we don’t know what’s going to be said between the Veterans and I think it’s important to make sure proper education being shared among them but I mean that’s going to be hard to control, so that’s another downfall.”

Site 1005 had the most concerns about the patients being paired and chatting in the SMAs, largely based on their prior experience: “There’s been two people that used to call each other and kind of hold each other accountable, however, they were both very non-compliant and didn’t give off the best information, so we were kind of like, ‘Eh, that didn’t work out very well.’” This site also had a concern about peer interaction during the SMAs, as a dietician talked about a negative belief about the interaction of peers and how one could be over-bearing and change the tone. “Then there’s this other guy that was coming to diabetes SMA and he made so many good changes, [But] he kind of came off really hard to others. He was losing weight, he was improving his A1C and we first told him, ‘Can you share your story, can you try to motivate these people?’ and it became really aggressive…someone would say, ‘No, I haven’t started exercising,’ and he’s like, ‘Why not?? Why can’t you do that?’ and it was really offending patients. We actually had to talk to him after…he was giving advice that more a provider should’ve given and it was not supportive and we had several patients call and complain about it.”

At site 1001, 1002, 1005, we held separate conference calls, where we gave a detailed outline of the P2P program, we explained that patients would receive orientation materials and instruction and be advised not to exchange any clinical advice, and allowed staff to discuss any of their concerns with facilitators.

“In the group [SMA] we do try to set goals each time. We’ll go over the goals that they made last month…goal-setting holds them accountable, we say, ‘Mr. so and so, did you get on the treadmill like you said you were going to last month?’ and if he hasn’t, it’s kind of like, ‘Oh, I let my group down.’ It might motivate them to try again this month and them kind of working off of each other and holding each other accountable, which is nice.”

P2P

Site 1001 had what we termed a “wait and see” approach to observe if the peer program had any benefit. When prompting interviewees on their beliefs about the peer program, we were able to clear up some misperceptions and confusion about how the patients would be communicating.

Site 1003 was the most confident in the benefits of the peer program—they stated that they saw an absolute need for the peer program based on other local work with patient groups that were fruitful. They believed having a peer would help with attendance and motivate patients to attend. Furthering their confidence in the peer program, a previous local veteran-paired smoking group had been successful.

There was pushback at sites 1001, 1002, and 1005 stemming from beliefs that matching peers would be difficult for multiple reasons. Site 1001 nurse, “Matching the patients up will probably be the hard part, I mean if one person doesn’t like their partner, I could see them wanting to stop.”

Champions

Champions were ranked positively at all but one site (1002). Interestingly, the champions’ organizational role varied across sites: Director of Primary Care (3), Clinical Health Psychologist (1), and Nurse Practitioner (1).

Positive champions were defined as those working to push through any barriers and positively advocating for the clinical programs to gain staff buy-in. The Chief of Primary Care for 1004 was deemed a good champion by staff at his site. He also said, “I see my role as making sure that primary care, as a service, sees this [SMA and P2P] as a benefit. Leading it from a high level rather and being able to tell others why we’re doing this. I can say, ‘this is important work.’” Descriptions of what the champions did included: help with presentations, help convince providers to enroll patients, block out time for providers to facilitate/attend the groups, secure space, and oversee local management of the project.

Many interviewees at site 1002 saw their site champion as non-influential. Facilitators worked with the local site to try to gain physician buy-in for the SMAs and P2P groups. In addition, we tried to pull in and speak with a physician who was named as being influential by interviewees and staff we met on site visits.

Discussion

In this article, we described our method for identification of contextual factors that influenced implementation of complex diabetes clinical programs -SMAs and P2P. The qualitative phone interviews aided implementation through the identification of modifiable barriers or conversely, actionable findings. Implementation projects, and certainly clinical programs, do not have unlimited resources and these interviews allowed us to determine which facets to target and act on for each site.

Likewise, our facilitation team used formative evaluation to understand the context and organizational issues at each of these VA health systems. Our approach used the CFIR to guide us to actionable findings and help us better understand barriers and facilitators, and variations of those, across constructs and sites. Using the CFIR to do this allowed us to improve the generalizability and efficiency of our findings by highlighting factors that prior research have identified as influencing implementation (each of the published constructs). Additionally, our project team and the local sites benefited from the use of formative evaluation throughout the early implementation process; we identified, in an ongoing manner, problems that we had not anticipated but that needed to be addressed to optimize implementation.

The implementation of a new clinical program is very complex and the field has recognized the need to utilize theoretical bases of implementation to facilitate implementation itself and there have been more calls for researchers to utilize existing frameworks to gain insights into the mechanisms by which implementation is more likely to succeed and to achieve common terminology. [21] We believe our approach of using the CFIR to accomplish those goals has broad applicability and can be used by other projects to guide, adapt, and improve implementation of research into practice.

We have illustrated how pre- and early-implementation FE is critically important in preparing for and gaining early understanding of key factors that influence implementation processes, and future success or failure. This early formative research shaped our implementation to minimize type III failures. Our rich examples highlight areas that were challenging as well as those that facilitated implementation of both shared medical appointments, and peer-to-peer programming. It is important to note that there was no site that was universally positive or negative across constructs, as often is assumed of “laggard” or “early adopting” sites. Evidence strength and quality was a negative issue at only one site (1002), but it was very impactful there (see Table 3, − 2 ranking) and important for the facilitators to be aware of and to work to overcome. As a result, our team presented evidence during the site visit and during a local primary care team meeting to help educate and influence the physicians. In contrast patient needs and resources was a negatively rated construct at 4 of the 5 sites, but at the 5th (1005) was a positively ranked construct—illustrating that implementation scientists need to be very cautious of labeling any construct as universally problematic.

Because of our intentional broad range of interviewees, CFIR constructs could be mixed within a site. When this was the case, the findings were discussed, weighed and used to come up with one overall score as per CFIR guidelines. The process is similar to consensus-based coding; “Analysts apply a summary rating, taking all the individual ratings and supporting qualitative summary and rationale into consideration, and then discuss ratings to achieve consensus” [22].

There are several limitations to this study. Constructs were assigned only one rating per site using weighted data from all respondents. Additionally, it is challenging for implementation researchers to identify when modifications create an additional intervention; however, in this case we classified these as local adaptations because the core underlying conceptual nature of the interventions was maintained in each case. Program drift is sometimes thought of as resulting in lower intervention success due to lack of fidelity [7]. However, experts in the field recognize that view is overly simplistic and encourages an unnecessarily rigid view of fidelity; “this designation decreased opportunities to learn from evidence-based intervention adaptations that result in improvements beyond what is expected” [10].

Conclusions

As the SHARES study progresses, these findings will be compared and correlated to outcome measures. This comprehensive adaptation data collection will also facilitate and enhance understanding of the future success or lack of success of implementation and inform potential for translation and public health impact. Crossing the bridge from research to practice in primary care and family practice settings is crucially important because in many ways we are not reaping the full public health benefits of our investment in research. [23] While the evidence-to-practice gap for interventions in primary care is receiving attention, it tends to be understudied. [11] We believe our approach of using the CFIR to guide us to actionable findings and help us better understand barriers and facilitators, has broad applicability and can be used by other projects to guide, adapt, and improve implementation of research into practice in primary care and other clinical settings.

Abbreviations

CFIR:

Consolidated Framework for Implementation Research

FE:

Formative Evaluation

IRB:

Institutional Review Board

P2P:

Peer-to-Peer

PCP:

Primary Care Physician

SHARES:

Shared Health Appointments and Reciprocal Enhanced Support

SMA:

Shared Medical Appointments

VA:

Veterans Affairs

References

  1. Stetler CB, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21(Suppl 2):S1–8.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93(8):1261–7.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci. 2010;5:67.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Hulscher ME, Laurant MG, Grol RP. Process evaluation on quality improvement interventions. Qual Saf Health Care. 2003;12(1):40–6.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  5. Mitchell SE, et al. Implementation and adaptation of the re-engineered discharge (RED) in five California hospitals: a qualitative research study. BMC Health Serv Res. 2017;17(1):291.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  6. Brown CH, et al. An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38:1–22.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Chambers, D.A., R.E. Glasgow, and K.C. Stange, The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci, 2013. 8: p. 117.

  8. Chambers DA, Norton WE. The Adaptome: advancing the science of intervention adaptation. Am J Prev Med. 2016;

  9. Aarons GA, et al. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Chambers DA, Norton WE. The Adaptome: advancing the science of intervention adaptation. Am J Prev Med. 2016;51(4 Suppl 2):S124–31.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Lau R, et al. Achieving change in primary care--causes of the evidence to practice gap: systematic reviews of reviews. Implement Sci. 2016;11:40.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Heisler M, et al. The shared health appointments and reciprocal enhanced support (SHARES) study: study protocol for a randomized trial. Trials. 2017;18(1):239.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Heisler M, et al. Diabetes control with reciprocal peer support versus nurse care management: a randomized trial. Ann Intern Med. 2010;153(8):507–15.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Heisler M, et al. Randomized controlled effectiveness trial of reciprocal peer support in heart failure. Circ Heart Fail. 2013;6(2):246–53.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Damschroder LJ, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Maxwell, J.A., Qualitative Research Design: An Interactive Approach. 3 ed. applied social research methods. 2013: Sage Publications. 232.

  17. Mason J. Qualitative researching. Thousand Oaks: Sage Publications; 2002.

    Google Scholar 

  18. Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res. 1999;34(5 Pt 2):1189–208.

    PubMed  PubMed Central  CAS  Google Scholar 

  19. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

    Article  PubMed  Google Scholar 

  20. Damschroder LJ, et al. Implementation evaluation of the telephone lifestyle coaching (TLC) program: organizational factors associated with successful implementation. Transl Behav Med. 2017;7(2):233–41.

    Article  PubMed  Google Scholar 

  21. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.

    Article  PubMed  PubMed Central  Google Scholar 

  22. CFIRguide. Consolidated Framework for Implementation Research - Qualitative Data Analysis. [cited 2017 April 18, 2017]; available from: http://cfirguide.org/qual.html.

  23. Lenfant C, lecture S. Clinical research to clinical practice--lost in translation? N Engl J Med. 2003;349(9):868–74.

    Article  PubMed  Google Scholar 

Download references

Funding

The SHARES study is funded by the VA Health Services Research & Development (HSR&D) IIR 15–321. The study funder did not design the study or data collection, and did not participate in the analysis or interpretation of data.

Availability of data and materials

The dataset generated and/or analyzed during the current study are not publicly available due to the confidential nature of qualitative data, but are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

Authors

Contributions

CPK lead all the interviews, lead the coding and analysis of the qualitative data, and wrote this manuscript. MV helped in the interviewing process, participated fully in the qualitative coding and analysis of data, and editing this manuscript. MH conceived and designed the overall SHARES study and critically revised this manuscript. All authors read and approved the final manuscript. CPK, MV, and MK, all agree to be accountable for all aspects of the work and in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to Christine P. Kowalski.

Ethics declarations

Author’s information

CPK was trained in epidemiology at the University of Michigan and works for the Center for Clinical Management Research (CCMR) at the Ann Arbor Veterans Affairs Healthcare System. She leads a national Implementation Research Group (IRG) of 250 members for the Center for Evaluation and Implementation Resources (CEIR) that provides continuning education, training, and sharing of best practices in implementation science. Her expertise over the last 15 years includes formative evaluation, qualitative interviewing and analysis, adaptations and fidelity, and implementation science.

MV obtained her Bachelor of Science from the University of Michigan and currently works as the research associate on the SHARES study.

MK is a physician Research Scientist at the Ann Arbor Center for Clinical Management Research (CCMR). She is also Director of the Community Engagement and Outreach Core of the Michigan Center for Diabetes Translational Research (MCDTR), one of five DTRs funded by the National Institutes of Health to provide assistance to researchers conducting novel interventions to improve diabetes care. MK has expertise in the development and evaluation of health system and behavioral interventions to improve between-clinic visit chronic disease self-management and outcomes. She has served as PI on multiple multi-site effectiveness and implementation studies evaluating different peer support models and health team outreach programs to improve glycemic, blood pressure, and other risk factor control in diabetes. She is also PI on an AHRQ grant that developed a diabetes web-based decision aid that peer mentors and other outreach workers can use with patients to improve diabetes treatment decision-making.

Ethics approval and consent to participate

The SHARES study was approved by the Veterans Affairs Central Institutional Review Board (C-IRB) (reference number 13–21). All participants consented to participate and were informed that their participation was completely voluntary. The consent process was executed as governed by the Central IRB. Staff members were provided with a Study Information Sheet at the time of recruitment and verbal consent was obtained prior to the start of the interview. Verbal consent for staff interviews was approved by the CIRB, as the study is considered minimal risk and obtaining written consent would place an additional burden on participants.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kowalski, C.P., Veeser, M. & Heisler, M. Formative evaluation and adaptation of pre-and early implementation of diabetes shared medical appointments to maximize sustainability and adoption. BMC Fam Pract 19, 109 (2018). https://0-doi-org.brum.beds.ac.uk/10.1186/s12875-018-0797-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12875-018-0797-3

Keywords