Skip to main content

Using a modified nominal group technique to develop general practice

Abstract

Background

There are few areas of health care where sufficient research-based evidence exists and primary health care is no exception. In the absence of such evidence, the development of assisted support must be based on the opinions and experience of professionals with knowledge of the relevant field. The purpose of this research project is to explore how the nominal group technique can be used to establish consensus by analysing how it supported the development of structured, knowledge-based, electronic health records for preventive child health examinations in Danish general practice.

Methods

We convened an expert panel of five general practitioners with a special interest in the preventive child health examinations. We introduced the panel to the nominal group technique, a well-established, structured, multistep, facilitated, group meeting technique used to generate consensus. The panel used the technique to agree on the key clinical and socioeconomic themes to include in new electronic records for the seven preventive child health examinations in Denmark. The panel met three times over a four-month period between 2013 and 2014 and their meetings lasted between two-and-a-half and five hours.

Results

1) The structured and stepwise process of the nominal group technique supported our expert panel’s focus as well as their equal opportunities to speak. 2) The method’s flexibility enabled participants to work as a group and in pairs to discuss and refine thematic classifications. 3) Serial meetings supported continual evaluation, critical reflection, and knowledge searches, enabling our panel to produce a template that could be adapted for all seven preventive child health examinations.

Conclusion

The nominal group technique proved to be a useful method for reaching consensus by identifying key quality markers for use in daily clinical practice. Our study focused on the development of content and a layout for systematic, knowledge-based, electronic health records. We recommend the method as a suitable working tool for dealing with complex questions in general practice or similar settings, and we present and discuss modifications to the original model.

Peer Review reports

Background

General practitioners (GPs) regularly make difficult choices about treatment options. Guidelines are one way of assisting GPs in decision-making and, in an ideal world, guidelines would be based on evidence derived from rigorously conducted empirical studies. In practice, there are few areas of health care where sufficient research-based evidence exists or can even be produced [1], and this is especially so within primary healthcare [2]. In such situations, the development of assisted support will inevitably be based, largely or in part, on the opinions and experience of clinicians and others with knowledge of the relevant field [3].

Since 2013, a group of Danish GPs has worked on producing systematic and knowledge-based electronic health records for the seven preventive child health examinations (PCHEs) held in general practice.

The Danish National Board of Health provides guidelines for PCHEs, but the recommendations are extensive and cover all aspects of a child’s health [4]. There is no structured and systematic process in place to determine which of the comprehensive recommendations is the most important to focus on during the limited time available to carry out the PCHEs, nor is there a nationally aligned process for keeping journal notes. The vision is to make new electronic journal records available to all Danish GPs and the idea is that GPs’ use of an electronic health record, with its potential for decision support and easy access to previous findings, will support their work and make it easier to keep an overview of the patient’s case history.Footnote 1 The development of electronic records therefore holds the potential for a quality development in child healthcare in Denmark.

Given the likely diversity of opinion that any group of people may display when considering a topic, formalised methods, such as consensus techniques, are essential for organising subjective judgments in group work. Consensus techniques have been successfully used by several research groups in their work to develop quality markers in complex clinical areas, such as angina [5], emergency care [6], cancer [7], and also within the field of child healthcare [8,9,10].

The three most common consensus methods used for medical and health services research are the Delphi method, the consensus development conference, and the nominal group technique (NGT) [11]. The Delphi method is a forecasting method based on several rounds of questionnaires sent to a panel of experts. The anonymous, written responses are aggregated and shared with the group after each round [12]. The consensus development conference brings together practitioners, researchers, and consumers over a period of several days to seek general agreement, or consensus, on the efficacy, safety, and appropriate conditions for the use of various medical and surgical procedures, drugs, and devices [13]. The third method, and the one we selected in the present study, is the NGT.

The NGT is a structured, well-established, multistep, facilitated, group meeting technique used to generate and prioritise responses to a specific question by a group of people who have expert insight into a particular area of interest [2, 11, 14, 15]. It is an organised process that gives participants an equal opportunity to contribute their personal views before inviting them to build on the reflections of others to develop their own thoughts, and finally to reach consensus about the issues raised in the original question [8]. The NGT has been applied on several occasions for projects in general practice [16,17,18,19]. It has an advantage in that its format resembles the way Danish GPs are accustomed to collaborating in network groups, where experiences and challenges from everyday working life in practice are shared and discussed [20].

In this study the NGT was used with a twofold purpose. First, it was a way to systematise and develop the content of PCHEs; and second, it was a method to develop the format of electronic health records to be used in PCHEs. These two parallel purposes were strongly interlinked.

To bridge the gap between research and practice, evidence as well as its applicability should be considered when formulating recommendations. It is important that recommendations are compatible with existing norms and values and it is therefore essential that practitioners, in this case the future users of electronic health records, participate in the development of practice [21].

In this paper we explore how the NGT can be used to establish consensus in a complex clinical field by analysing how it supported the development of structured electronic health records for PCHEs in Danish general practice.

Methods

This study applied group discussions based on the NGT method in an adapted serial meeting design. The adapted design complies with the checklist created by Humphrey-Murto et al. to ensure methodological rigour when using consensus group methods, with only one deviation concerning anonymous re-ranking of feedback [22]. In-depth descriptions of the original steps in the NGT method have been reported extensively elsewhere [8, 23].

The project was conceived and designed by KL and RE, who are both experienced GPs specialising in research on children’s health. During the meetings, KL was in the facilitator’s role and RE participated as a member of the NGT panel. RE participated on the same terms as the other panel members, meaning that e.g. she waited for her turn to speak in the rounds, and her opinion carried no more weight than any of the other participants. More importantly, RE was aware of her double role in the project and its potential downfalls. This demanded a continuous reflection on her position, which we shall return to in the discussion section.

As well as RE, we purposively identified four GPs known for their broad knowledge and expertise in general practice and their specific interest in the PCHEs. We invited them to constitute the expert panel. Verbal informed consent was given before the four recruited GPs freely and informed chose to participate in the project. All five participating GPs worked either in Region Zealand or in the capital area of Copenhagen in Denmark. Prior to the first meeting the GPs were asked to read the report: Evaluation of the Preventive Child Consultations in General Practice [4] and before each meeting they were also asked to read the chapters in the Danish National Board of Health’s guidelines on PCHEs [24] relevant for that particular meeting’s focus. In this way we pursued a systematic method combining evidence and expert opinion.

In addition to background reading, we asked the participating GPs to be extra observant when carrying out PCHEs in the period leading up to the first meeting. We encouraged them to ask parents about their needs and expectations during the PCHEs. Throughout the working period of four months, the expert panel met three times. The meetings lasted five hours, five hours, and two-and-a-half hours respectively, and the four invited participants were offered compensation for their time. All meetings took place at facilities convenient to the practice of two of the participants.

The main aim of NGT is to generate themes and issues, which are discussed and ranked by the group. At the first meeting, KL described the NGT as a method to the panel members who had the opportunity to ask questions. This introduction was a factual description of the method’s different steps and did not have any content or comment that would influence participants and the task in hand. After the introduction, KL asked the panel the nominal question: What do you consider important to prioritise in the preventive child health examination at five weeks, and what do parents think is important, according to your experience and knowledge? The question was developed based on KL’s extensive work with the PCHEs [25,26,27] and RE’s previous experience with developing electronic health records for antenatal care visits in general practice in Denmark. At this stage, the panel was given no guidance on how broad or narrow their focus should be.

The original plan was to work on the first three PCHEs, which take place when a baby is five weeks, five months, and twelve months old; one PCHE per scheduled meeting. However, while working through the steps of the NGT during the first meeting, the group found it necessary to make adjustments and deviations to the original model, outlined by Gallagher et al. [23].

The five hours allocated to develop a health record for the first PCHE were not enough to meet the project’s combined objective: exploratory research involving a qualitative understanding of the priorities; and the development of a concrete product in the form of a systematic electronic health record. As a result, it was agreed in plenum that KL and RE should work with the draft produced by the panel between the first and the second meeting. This work solely concerned linguistic and structural aspects and a conscious effort was made to keep the content unchanged. The re-edited draft was then presented to participants at the second meeting, where it was critically evaluated, adjusted, and approved in plenum. In this way, a mutual understanding was secured in a forum in which the participants were both informants and collaborators. This pattern was repeated between the second and third meetings and became the model for working with drafts of the succeeding electronic health records (Fig. 1). Consensus was defined as having been achieved when there were no further comments or suggestions for corrections from any of the participants. Achieved consensus determined the process. The continual and circular re-evaluation of the drafts enhanced the process and secured communicative validity [28].

Fig. 1
figure 1

Three NGT meetings were planned from the beginning; one for each of the first three preventive child health examinations in general practice. Between meetings RE and KL continually worked with the document from the previous meeting, which was then discussed, adjusted and approved at the following meeting

The group’s experience from working with the content and structure of the first electronic health record was used strategically at the next meeting when the focus shifted to the succeeding PCHE. During the meetings, the atmosphere was jovial and enthusiastic. The participation of KL and RE as the project’s initiators did not seem to affect or influence the four invited GPs. All participants took part in the discussions equally and appeared confident in their roles as well as eager to contribute with their individual perspectives on the work.

In addition to working papers from the meeting rounds and the different draft versions of the electronic health records for the first three PCHEs, material for the present article also consisted of field notes produced during one of the meetings by ES, who participated as an observer. Having an observer in the research project enhanced opportunities for noticing aspects of interpersonal communication and group dynamic that are taken for granted or missed by participants due to their immediate obviousness [29]. The observations and field notes permitted an extra level of abstraction in the discussion of the group’s use of the NGT, particularly with regard to the steps of the model where the group deviated from the original structure.

Based on the output of the meetings, including descriptive field notes, we conducted a thematic text analysis to identify important areas of new knowledge and to better understand what the modified version of the NGT meant for the validity of the method.

Results

The use of the NGT made it possible to combine idea generation and problem solving as two complementary parts of the same process. This makes the method well suited for development work in general practice with its complex characteristics and demands for applicability. Three main categories of experience were identified and these are described and appraised below. For clarity, the third category is divided into four sub-categories.

Keeping focus and supporting equal opportunities to speak

The structured and stepwise process of the NGT ensured that the energetic expert panel kept focus on the defined purpose, while the repeated table rounds supported opportunities for participants to be equally heard. The method’s face-to-face approach integrated non-verbal communication, such as laughter; while the structured design minimised potential power structures that can appear when participants already know each other, or when one of the panelists is also the initiator of the project, which was our experience on this project.

Generating new perspectives on clinical practice

During steps 4 and 5 (Table 1) the participants became aware of the potential to re-use knowledge previously obtained about the patient (Table 2, column 3). Prior to the seven PCHEs, the Danish preventive healthcare programme has three antenatal care visits and, in principle, all ten visits are conducted by the same GP. Data from antenatal care visits are recorded in the mother’s journal, which is not automatically consulted in the PCHEs that follow. The process of the group discussions generated an awareness of the prospective re-use of knowledge gathered during the antenatal care visits, such as the pregnancy’s development or the family’s socioeconomic situation.

Table 1 An outline of the steps in the original NGT model and the deviations and attributions made to the structure during the working process with the development of an electronic health record for the examination of babies at 5 weeks
Table 2 In the first column, all the participants’ ideas are shown in the order they appeared during the table rounds and in the short formulations the proposers found adequate

Flexibility of the NGT model

The NGT proved to be a highly flexible model well suited to the complex research question we asked, and conducive to detailed discussion and elucidation of themes and issues.

Discussions and thematic classification in pairs

The panel recognised early in the working process that it would not be favourable to strictly follow the model’s original outline. For example, the discussions and clarifications carried out in step 4 (Table 1) revealed that it did not make sense to produce a prioritised list, as the model prescribes. Since all the suggested themes were important, the expert panel found it more relevant to organise them into broader thematic categories and line them up in that way. The work with these categories was carried out first as a group and then the panel divided into pairs to further discuss the categories. This resulted in an outline of the first draft of the electronic health record (Table 2, column 2).

Serial meetings

Serial meetings provided time for continual evaluation and the search for more information.

The original time allocated to work with electronic health records turned out to be too short to produce adequate content and a format for each record. As a consequence, drafts were linguistically and structurally reorganised by KL and RE between meetings (Fig. 1). At the same time, participants had opportunities to test in their practices aspects that had been discussed during the meeting, and to return to the next session with new experience-based knowledge. Participants became aware that some instructions in the guidelines from the National Board of Health were not fully up-to-date; the recommendations for congenital cataract, for example. The serial nature of the meetings meant that such questions of doubt could be checked in the interim and discussed at the next meeting.

Reflections and ethical considerations

The serial character of the meetings and the continual re-evaluation of the drafts (Fig. 1) made room for participants to further reflect between meetings on the topics discussed. During the first meeting, the idea that information collected at the mother’s antenatal care visits could automatically be transferred to the child’s health record was presented and calmly received in the group. However, at the third meeting an intense discussion arose concerning ethical issues raised by the idea of transferring certain kinds of information to the child’s record, e.g. alcohol abuse in the family. The serial application of the method provided time for important critical reflection on themes and, in this case, the ethical challenges around a potential transfer of data.

Adjustable template

Although the group did not succeed at the first meeting in producing a final model for the PCHE at five weeks, the NGT secured the production of a fruitful draft which was applied as a model for the first and all succeeding PCHEs (See Table 2). The flexibility of the NGT therefore led to the production of a template that could be used and tailored during the development of systematic electronic health records for all seven PCHEs. The template was based on the experience of frontline professionals, the guidelines from the National Board of Health, and best evidence. The findings were practice-near, experience-based, and therefore directly applicable to PCHE work in general practice.

Discussion

Our main findings from working with the NGT relate to its flexibility and modifiability. The flexibility of the method confirmed its suitability for complex research questions, such as ours; while the production of an adjustable template with consensus results made the meetings’ outcomes both manageable and tangible. The functionality of the modified serial meeting design provided fruitful time for continued reflection on the results and previous discussions, as well as providing opportunities for relevant checks between meetings where a lack of knowledge or doubts had become apparent. The latter provided openings for systematic development of knowledge.

Some of our results concur with findings from previous projects working with the NGT and the model’s flexibility has been recognised by other studies that also successfully modified the original NGT design and experienced an improvement [5,6,7,8,9,10]. One study had difficulty with the ranking in step 5 (Table 1) [8], which we also report in our findings. They ended up voting when consensus could not be reached through ranking, while the present study chose to divide the panel into pairs to work with the themes, before returning once more to discussion in plenum. In line with our findings, other studies have also found that the original NGT structure, with one meeting allocated to reach final recommendations, was not sufficient for an in-depth elaboration of themes [8, 30]. The serial character of the meetings in this study is comparable to the Delphi method [12, 15] where consensus is obtained through evaluation of written documents that are sent back and forth among participants a number of times until consensus is reached. Therefore, in the present study we incorporated strength from the Delphi method into the modified version of the NGT.

Experiences emerged during the working processes that, to our knowledge, have not previously been reported by other studies. A central feature of using the NGT is that a question is investigated extensively from a broad spectrum of viewpoints and thereby creates awareness of overlaps, knowledge-sharing, gaps in knowledge, or unproductive working patterns. In this project, our participants became aware of the possibility of using existing knowledge about their patients, but also about the potential ethical downside of such a practice. By documenting all proposals and ideas, the NGT model ensures that no insights are lost through the potential uncertainty of some participants, while at the same time tangible products in the form of written documents are produced.

Implication of findings for future research

Related to the specific research project

It is anticipated that the possibility of using electronic health records as a support when carrying out PCHEs in the future will systematise and develop both the structure and the content of the PCHEs in general practice. However, experience and intuition are fundamental and effective elements of everyday working life in general practice, not least when it comes to diagnosing children [27, 31, 32]. It is therefore crucial that the electronic health records do not compromise this practice, which is why the use of the records will be a supportive option and not a mandatory practice. Lippert et al. have pointed to a need for further discussion about the relationship between situatedness and standardisation in primary care and for further empirical investigations of the possible consequences of standardisation processes [33]. DanChild, of which the present study is a part, has a combined vision to investigate GPs’ responses to electronic health records as well as to develop child health through cohort research. While the NGT as a method encourages consensus and practice-near solutions, it is important to emphasise that the success of the electronic health records is dependent on a continued and reciprocal collaboration with general practice [34, 35].

Related to the applicability of the modified method

The modifications we made to the NGT were feasible and did not lose the method’s advantageous structure. We believe this was because the participants and the facilitator shared a common professional background as GPs, limiting the perspectives to one professional grouping. Participants had been asked to read relevant chapters in the Danish National Board of Health’s guidelines on PCHEs as well as a thematically relevant report, further enhancing a mutual starting point. However, we did not check whether or not they had read the documents. We felt that this would unnecessarily highlight the fact that one of the participants, RE, was also one of the project’s initiators. Therefore we cannot guarantee that all participants had a common starting point for discussion. Finally, the project had a well-defined goal, namely the production of content and a format for electronic health records supporting the PCHEs, and that concrete purpose enabled a softening of the original NGT model’s mechanical steps, without the group losing its focus. Based on our experience with the modified NGT all three aspects are crucial for future researchers implementing similar changes to the original NGT model.

Strengths and limitations of the study

By asking highly professionally engaged GPs with a specific interest in the PCHEs to reach consensus and suggest a way forward for all GPs to follow, our results may be ambiguous for the average practitioner. The project group is aware of this risk and will incorporate it into their continued work with electronic health records by pilot testing the product and by giving individual practitioners flexibility to use the record in their own way.

In this project two of the article’s authors participated either as facilitator (KL) or as a member of the panel (RE), and they worked with the drafts between the meetings. Double roles like these are not uncommon in similar projects [8], but still worth critical reflection and consideration. Any data collection, analysis, and conclusion are inextricably entwined with the researcher’s presuppositions as well as the positions adopted while collecting the data [36, 37]. According to Skjervheim, researchers can only gain access to social phenomena of interest by recognising themselves as a contributing participant [38]. In this case, RE is an experienced GP with a known research interest in child health, and took part in the panel as an equal to the other participants. RE was, however, aware of her double role in the process and continuously reflected on the effect it might have on the way her suggestions and comments were received by the group, and, ultimately, on how it might have affected consensus. One could discuss if KL’s and RE’s work between the meetings minimised the democratic process, by giving them more influence than the rest of the group. This risk was reduced as much as possible, by the process of repeatedly evaluating their work in plenum at subsequent meetings. During these evaluations the other participants actively suggested critical corrections and this leads us to believe that the final product meets a satisfactory degree of representativeness. Furthermore, we attempted to minimise bias by enrolling co-authors who were not actively implicated in the data producing process. One of them even participated as an observer at one of the meetings. Finally, the authors recognise that objective knowledge in the form of true consensus is a naïve understanding of reality. Following Haraway, it might be more fruitful to think of knowledge as situated within a context [39]. While the point of view within a context has a more limited range than disembodied objectivity, situated points of view are richer in content as they take into account the numerous bits of information constituting the context and the environment of that point of view. For the present project, it means acknowledging the influence KL and RE had on consensus, while at the same time recognising this as a given condition that simultaneously supported the success of the group’s progressive work.

Conclusions

To our knowledge, this study is the first to report on Danish GPs using the NGT to identify key areas of focus and to structure quality marker development in general practice. The structured interactive process used in this study supported equal opportunities for experienced professionals to significantly contribute to the development of electronic health records to support PCHEs in Danish general practice. By using the modified NGT, participating GPs actively expressed their views through structured discussions as a group, through working in pairs, and through the process of reaching final consensus. In accordance with previous studies [3] we therefore argue that the original NGT model developed in the late 1960s [5] can be modified advantageously and used to explore developmental work and changes in general practice. Due to the integration of experienced professionals from the very beginning of the process the results are practice-based and applicable. We are confident that the NGT model can be useful for capturing group perspectives in complex working areas such as general practice, and we recommend the NGT as a working tool in general practice development in the future.

Notes

  1. Creating electronic health records is part of DanChild, a research and quality development project, with the goal to develop a new national birth cohort with potential to nest randomised trials in general practice. The cohort will support the development of knowledge about child health which is highly relevant to the work in general practice.

Abbreviations

ES:

Elisabeth Søndergaard

Fig.:

Figure

GPs:

General practitioners

KL:

Kirsten Lykke

NGT:

The nominal group technique

PCHEs:

Preventive child health examinations

RE:

Ruth K. Ertmann

References

  1. Chassin M. How do we decide whether an investigation or procedure is appropriate? In: Hopkins A, editor. Appropriate investigation and treatment in clinical practice. London: Royal College of Physicians; 1989. p. 21–9.

    Google Scholar 

  2. Campbell SM, Braspenning J, Hutchinson A, Marshall MN. Research methods used in developing and applying quality indicators in primary care. BMJ. 2003;326(7393):816–9.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  3. Mann T. Clinical guidelines : using clinical guidelines to improve patient care within the NHS. London: Department of Health; 1996.

    Google Scholar 

  4. Michelsen S, Kastanje M, Flachs EM, Søndergaard G, Biering-Sørensen S, Madsen M, AMN A. Evaluation of the preventive child consultations in general practice. Copenhagen: Danish National Board of health, Statens Institut for Folkesundhed, Syddansk Universitet; 2011.

    Google Scholar 

  5. Campbell SM, Ludt S, Van Lieshout J, Boffin N, Wensing M, Petek D, et al. Quality indicators for the prevention and management of cardiovascular disease in primary care in nine European countries. Eur J cardiovasc Prev Rehabil. 2008;15(5):509–15.

    Article  PubMed  Google Scholar 

  6. Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, et al. Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators. CJEM. 2011;13(5):300–9. E28–43

    Article  PubMed  Google Scholar 

  7. Malin JL, Asch SM, Kerr EA, McGlynn EA. Evaluating the quality of cancer care: development of cancer quality indicators for a global quality assessment tool. Cancer. 2000;88(3):701–7.

    Article  PubMed  CAS  Google Scholar 

  8. Gill PJ, Hewitson P, Peile E, Harnden A. Prioritizing areas for quality marker development in children in UK general practice: extending the use of the nominal group technique. Fam Pract. 2012;29(5):567–75.

    Article  PubMed  Google Scholar 

  9. Giannini EH, Ruperto N, Ravelli A, Lovell DJ, Felson DT, Martini A. Preliminary definition of improvement in juvenile arthritis. Arthritis Rheum. 1997;40(7):1202–9.

    Article  PubMed  CAS  Google Scholar 

  10. Wang CJ, McGlynn EA, Brook RH, Leonard CH, Piecuch RE, Hsueh SI, et al. Quality-of-care indicators for the neurodevelopmental follow-up of very low birth weight children: results of an expert panel process. Pediatrics. 2006;117(6):2080–92.

    Article  PubMed  Google Scholar 

  11. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  12. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

    PubMed  CAS  Google Scholar 

  13. Wortman PM, Vinokur A, Sechrest L. Do consensus conferences work? A process evaluation of the NIH consensus development program. J Health Polit Policy Law. 1988;13(3):469–98.

    Article  PubMed  CAS  Google Scholar 

  14. Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62(3):337–42.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  15. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74(9):979–83.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  16. Stolper E, van Leeuwen Y, van Royen P, van de Wiel M, van Bokhoven M, Houben P, et al. Establishing a European research agenda on 'gut feelings' in general practice. A qualitative study using the nominal group technique. Eur J Gen Pract. 2010;16(2):75–9.

    Article  PubMed  Google Scholar 

  17. Shipman C, Gysels M, White P, Worth A, Murray SA, Barclay S, et al. Improving generalist end of life care: national consultation with practitioners, commissioners, academics, and service user groups. BMJ. 2008;337:a1720.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Zhao Y, Chen R, Wang B, Wu T, Huang Y, Guo A. General practice on-the-job training in Chinese urban community: a qualitative study on needs and challenges. PLoS One. 2014;9(4):e94301.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  19. Laughlin T, Wetmore S, Allen T, Brailovsky C, Crichton T, Bethune C, et al. Defining competency-based evaluation objectives in family medicine: communication skills. Can Fam Physician. 2012;58(4):217–24.

    Google Scholar 

  20. Gannik D. Kvalitet i almen praksis: Praktiserende lægers syn på fagets essens, vilkår og udvikling i Viborg Amt. Forskningsenheden for Almen Praksis i København; 2004.

  21. Burgers JS, Grol RP, Zaat JO, Spies TH, van der Bij AK, Mokkink HG. Characteristics of effective clinical guidelines for general practice. Br J Gen Pract. 2003;53(486):15–9.

    PubMed  PubMed Central  Google Scholar 

  22. Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and nominal group in medical education research. Med Teach. 2017;39(1):14–9.

    Article  PubMed  Google Scholar 

  23. Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: a research tool for general practice? Fam Pract. 1993;10(1):76–81.

    Article  PubMed  CAS  Google Scholar 

  24. Vejledning om forebyggende sundhedsydelser til børn og unge. National Board of health: Copenhagen, Denmark; 2011.

  25. Lykke K, Christensen P, Reventlow S. The consultation as an interpretive dialogue about the child's health needs. Fam Pract. 2011;28(4):430–6.

    Article  PubMed  CAS  Google Scholar 

  26. Lykke K, Christensen P, Reventlow S. GPs’ strategies in exploring the preschool child’s wellbeing in the paediatric consultation. BMC Fam Pract. 2013;14:177.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Lykke K, Christensen P, Reventlow S. "this is not normal ... "--signs that make the GP question the child's well-being. Fam Pract. 2008;25(3):146–53.

    Article  PubMed  Google Scholar 

  28. Steinar K. The social construction of validity. Qual Inq. 1995;1(1):19–40.

    Article  Google Scholar 

  29. Corbin Dwyer S. The space between: on being an insider-outsider in qualitative research Int J Qual Methods. 2009;8(1):54–63.

  30. He G, Huang WY, Wong SH. Understanding neighborhood environment related to Hong Kong Children’s physical activity: a qualitative study using nominal group technique. PLoS One. 2014;9(9):e106578.

  31. Van den Bruel A, Aertgeerts B, Bruyninckx R, Aerts M, Buntinx F. Signs and symptoms for diagnosis of serious infections in children: a prospective study in primary care. Br J Gen Pract. 2007;57(540):538–46.

    PubMed  PubMed Central  Google Scholar 

  32. Van den Bruel A, Haj-Hassan T, Thompson M, Buntinx F, Mant D. Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet. 2010;375(9717):834–45.

    Article  PubMed  Google Scholar 

  33. Lippert ML, Reventlow S, Kousgaard MB. The uses and implications of standards in general practice consultations. Health. 2015;21(1):3–20.

    Article  Google Scholar 

  34. Dahler-Larsen P. The Evaluation Society Standford: Standford University; 2012.

  35. Munro E. The impact of audit on social work practice. Br J Soc Work. 2004;34(8):1075–95.

    Article  Google Scholar 

  36. Reventlow S, Tulinius C. The doctor as focus group moderator - shifting roles and negotiating positions in health research. Fam Pract. 2005;22(3):335–40.

    Article  PubMed  CAS  Google Scholar 

  37. Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet. 2001;358(9280):483–8.

    Article  PubMed  CAS  Google Scholar 

  38. Skjervheim H. Objectivism and the study of man. Oslo: Universitetsforlaget. 1959.

  39. Haraway D. Situated knowledges: the science question in feminism and the privilege of partial perspective. Fem Stud. 1988;14(3):575–99.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank all the other members of the expert panel; Morten Jakobsen, Johannes Larsen, Majbrit Brouer and Pia Birgitte Koefoed for their engaged participation.

Funding

The work with developing the electronic health records for the preventive child health examinations was funded by Region Zealand.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

Authors

Contributions

KL and RE conceived and designed the study and RE participated in the panel while KL moderated the meetings. ES collected and interpreted the data and drafted the article. RE, KL, and SR contributed to drafting of the article, critical revisions to the article, and approved the final draft.

Corresponding author

Correspondence to Elisabeth Søndergaard.

Ethics declarations

Ethics approval and consent to participate

Danish legislation has waived the need for formal ethics approval for studies that do not include biomedical tissue or sensitive data (Komitéloven § 2 nr. 1). Furthermore, projects that solely involve non-sensitive data about participants do not have to be notified to the Danish Data Protection Agency. Verbal informed consent was given before the four recruited GPs freely and informed chose to participate in the project.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Søndergaard, E., Ertmann, R.K., Reventlow, S. et al. Using a modified nominal group technique to develop general practice. BMC Fam Pract 19, 117 (2018). https://0-doi-org.brum.beds.ac.uk/10.1186/s12875-018-0811-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12875-018-0811-9

Keywords