- Study protocol
- Open Access
- Open Peer Review
Identification of factors associated with diagnostic error in primary care
BMC Family Practice volume 15, Article number: 92 (2014)
Missed, delayed or incorrect diagnoses are considered to be diagnostic errors. The aim of this paper is to describe the methodology of a study to analyse cognitive aspects of the process by which primary care (PC) physicians diagnose dyspnoea. It examines the possible links between the use of heuristics, suboptimal cognitive acts and diagnostic errors, using Reason’s taxonomy of human error (slips, lapses, mistakes and violations). The influence of situational factors (professional experience, perceived overwork and fatigue) is also analysed.
Cohort study of new episodes of dyspnoea in patients receiving care from family physicians and residents at PC centres in Granada (Spain). With an initial expected diagnostic error rate of 20%, and a sampling error of 3%, 384 episodes of dyspnoea are calculated to be required. In addition to filling out the electronic medical record of the patients attended, each physician fills out 2 specially designed questionnaires about the diagnostic process performed in each case of dyspnoea. The first questionnaire includes questions on the physician’s initial diagnostic impression, the 3 most likely diagnoses (in order of likelihood), and the diagnosis reached after the initial medical history and physical examination. It also includes items on the physicians’ perceived overwork and fatigue during patient care. The second questionnaire records the confirmed diagnosis once it is reached. The complete diagnostic process is peer-reviewed to identify and classify the diagnostic errors. The possible use of heuristics of representativeness, availability, and anchoring and adjustment in each diagnostic process is also analysed. Each audit is reviewed with the physician responsible for the diagnostic process. Finally, logistic regression models are used to determine if there are differences in the diagnostic error variables based on the heuristics identified.
This work sets out a new approach to studying the diagnostic decision-making process in PC, taking advantage of new technologies which allow immediate recording of the decision-making process.
Missed, delayed or incorrect diagnoses are considered to be diagnostic errors . Although the incidence of diagnostic error is difficult to establish , it is estimated to be between 5% and 20% , depending on the medical speciality analysed. Berner and Graber  distinguish between “perceptual” specialities (radiology and anatomical pathology), where the diagnosis is made based on the perception of an image, and other clinical fields (family medicine, internal medicine, emergency department). The incidence of diagnostic error in the former is between 2% and 5%, while the incidence in the latter can be up to 15% . In the United States, it is estimated that more than 150,000 patients per year may undergo misdiagnosis-related harm, and 50,000 missed diagnostic opportunities occur each year in primary care (PC) alone . Diagnostic errors lead to more deaths than any other medical error type , and this can have major legal repercussions: diagnostic errors are the leading cause of malpractice claims in the United States .
However, few studies on the diagnostic process have been published, probably because it is still considered to be more of an individual art than a science. The complexity of the diagnostic process and the lack of established methods for its analysis also play a part. There are even fewer studies on the diagnostic process in the PC setting .
In order to provide the right clinical care, a correct diagnosis is essential, although not sufficient on its own. To reach the right diagnosis, physicians must take a complete medical history and perform the right physical examinations and further tests. Not ordering the correct tests is potentially just as dangerous as performing unnecessary tests, which can lead to iatrogenic effects and overdiagnosis .
Studies on decision-making under conditions of uncertainty in the fields of psychology and behavioural economics have very slowly started to spread into the medical field. Stanovich and West  coined the terms “system 1” and “system 2” to refer to the 2 systems of cognitive function. System 1 operates automatically and quickly, with little or no effort and no sense of voluntary control. System 2 allocates attention to the effortful mental activities that demand it, and its operations are often associated with the subjective experience of agency, choice and concentration. Diagnostic error is typically viewed as a cognitive failing , often caused by biases linked to system 1 (non-analytical, intuitive thinking). Some authors  believe that these cognitive biases can be a warning sign for possible diagnostic errors.
Heuristics are “the strategies that people use deliberately in order to simplify judgemental tasks that would otherwise be too difficult for the typical human mind to solve” . Kahneman, Slovic and Tversky  identified different types of heuristics involved in the decision-making process. However, not all researchers believe that the use of heuristics (and by extension, system 1) in clinical reasoning necessarily leads to a higher risk of error .
This study uses only the 3 general heuristics described initially by Kahneman and Tversky, the validity of which has been confirmed by many empirical findings over the last few decades : 1) Availability, i.e. the diagnostic alternatives available during the reflective process at a particular point in medical care; 2) Representativeness, i.e. the case’s similarity to a particular known diagnostic category; and 3) Anchoring and adjustment, i.e. how well the information generated during the diagnostic process fits with the initial diagnostic hypothesis.
“Suboptimal cognitive acts” (SCAs) are cognitive errors that deviate from the optimal cognitive process, and which could lead to an adverse event . Based on Reason’s model of unsafe acts , cognitive errors can occur as a result of 2 types of action: unintentional and intentional.
1) Unintentional actions: these occur when routine activities are inadequately performed because of failings in attention (slips) or memory (lapses). In these cases, the error is caused by a failure in the storage and/or execution stage in the action sequence, regardless of whether the action is appropriate or not. It is therefore largely influenced by working conditions, especially overwork and fatigue.
2) Intentional actions: deliberate actions that may lead to an error because the person lacks the knowledge required to tackle the situation (mistakes), or because the person disregards the appropriate procedure (violations). In these cases, the error is due to a lack of knowledge (in the first case) or compliance (in the second). They are therefore influenced by experience and attitude.
Factors associated with the diagnostic process in PC
PC physicians work under vastly different conditions from their colleagues in emergency or hospital departments. In PC settings, patients usually present with poorly defined, often psychological or social symptoms. Some patients attend in the initial stages of the disease, while others have confirmed diagnoses. PC physicians see patients whose problems have not yet been reduced to a specific category, so there is a very broad range of possible diagnoses . In order to tackle this diverse range of problems appropriately, PC must fulfil 4 essential criteria : accessibility, comprehensiveness of services, coordination of care within the healthcare system, and longitudinality (the capacity to provide continuous (ongoing) care throughout a patient’s lifetime). In addition to these specific aspects, the diagnostic process in PC is determined by factors that influence any care context: professional experience, overwork, fatigue and stress [18, 22]. The increase in PC activities often leads to overwork, and this can have a direct influence on the diagnostic process.
Studying the diagnostic process in PC requires an analysis of the complete process a physician follows when faced with symptoms, not diseases, from the moment care starts to the moment a diagnosis is reached. Dyspnoea was selected for this study because it is a frequent PC problem and provides an opportunity for a differential diagnostic procedure.
The aims of this study are to:
Identify and describe the SCAs that occur during care of patients with dyspnoea attending PC centres.
Analyse cognitive aspects of the process by which PC physicians diagnose dyspnoea using heuristics.
Analyse the relationship between situational factors (perceived overwork, fatigue, professional experience) and SCAs during the diagnostic process in patients with dyspnoea in PC settings.
Analyse the relationship between those situational factors and the use of heuristics during the diagnostic process in patients with dyspnoea in PC settings.
We designed a prospective cohort study of new episodes of dyspnoea. The cohort monitoring period begins when the patient first attends because of a new episode and ends when a concrete diagnosis is reached. This period is estimated to be between 2 days and 8 months . The study procedure is shown in Figure 1.
Selection of centres
We selected 4 PC centres in the province of Granada (Spain). The family physicians working at the centres have different levels of experience, the centres have different patient numbers, and the patients come from different socioeconomic backgrounds (lower, middle-lower, middle-upper and upper classes). 2 distinct physician types were selected: residents and experienced family physicians.
The study analyses diagnostic processes in patients registered at the practices of the participating family physicians, attending because of a new episode of dyspnoea. A consecutive sampling method was used, recruiting diagnostic procedures from the accessible population of patients attending because of a new episode of dyspnoea.
We calculated the number of cases required based on the method set out by Zwaan et al. . In that study, in order to obtain a substantial number of SCAs in each of the 5 categories (slips, lapses, mistakes, violations, patient record problems), a confidence interval was calculated assuming an equal distribution of suboptimal cognitive acts in each category (20% of SCAs in each category). An average of 1.5 SCAs was expected per patient record, so 250 records of dyspnoea (375/1.5) were required. Based on this, the number of cases of dyspnoea expected in our study is 384.
As independent variables, we are going to explore factors related with the patient, the setting and the professional. The dependent variables are those related with the errors. These are described in Table 1.
The study period deemed necessary to recruit the number of patients required and carry out the diagnostic process is 1 year. The audit is carried out at the end of each diagnostic process.
The data are collected in the following phases and through the following sources:
PHASE 1. Record of the diagnostic process (provided by the physician)
Data from the physician
Once they have agreed to take part in the project, the PC physicians fill out records detailing variables about themselves (listed in the first column of Table 1). They also fill out the patients’ electronic clinical records in the software application used by all healthcare professionals working for the Andalusian Health Service, the region’s public health service.
Data from the diagnostic process
For each patient attending with a new episode of dyspnoea, the physicians fill out 2 questionnaires via a tablet computer with an internet connection. The questionnaire is available as a wufoo application (http://www.wufoo.com) saved on the tablet’s desktop. Tables 2 and 3 describes the questionnaires used.
PHASE 2. Review of the diagnostic process (provided by the evaluators)
Identification of the optimal diagnostic process
The methodology used by Zwaan  has been used as a reference for the audit and adapted to the PC setting.
The clinical practice guidelines set out by the Spanish Society of Family and Community Medicine (semFYC) were reviewed. As no clinical guidelines on the management of dyspnoea in PC were found, an exhaustive review of the literature on dyspnoea care in PC was carried out. The dyspnoea care procedure in PC was revised based on the articles found.
These reviewed guidelines were sent to 4 family physicians with more than 20 years of clinical experience, who made all the necessary corrections to create a consensus reference document.
Creation of the audit questionnaire
Based on the consensus document created, a diagnostic process audit questionnaire  analysing the following 7 phases of the process was created: 1) medical history; 2) physical examination; 3) laboratory test results; 4) scans; 5) diagnosis; 6) treatment; 7) follow-up.
The questionnaire was revised and piloted by the participants in the audit process, and a second questionnaire was created based on the suggestions made after the pilot.
The final audit questionnaire will be filled out via wufoo when the audit is carried out.
Each of the cases will be assessed by 2 physicians (not including the physician who performs the diagnosis). If the evaluations of the 2 physicians are different, the case will be assessed by a third evaluator. The physician-evaluators have accredited clinical care experience and have been trained in how to use the audit guide. The stages of data collection will be analysed based on two criteria: 1) whether the correct information was collected; and 2) whether the information was interpreted correctly.
Variables associated with the possible use of heuristics, the performance of suboptimal cognitive acts and the possible existence of diagnostic error will be collected.
Review of the diagnostic process with the physician in charge of the patient’s care
The diagnostic process audit will be reviewed by the evaluators and the physicians who performed the diagnostic process through an interview in order to analyse any cognitive acts that took place, as well as any possible diagnostic errors.
PHASE 3. Review of centre reports, to obtain information about centre characteristics
A descriptive statistical analysis of diagnostic errors, suboptimal cognitive acts and use of representativeness, availability and anchoring heuristics will be carried out.
A bivariate analysis will be performed to establish if there are any differences in the diagnostic error variables based on the heuristics identified and the independent variables. To do this, a Chi-square or Fisher’s exact test will be used for samples with fewer than 5 individuals.
Multivariate models will be used to determine if there are any differences in the diagnostic error variables based on the heuristics identified, adjusting for the other independent variables, using logistic regression models.
Confidentiality and ethical approval
This project has been approved by the Regional Research Ethics Committee of the province of Granada (Spain).
All participating patients agreed to take part in the research project through the corresponding informed consent process.
All participating physicians also signed consent forms in order to take part, expressly agreeing to keep all the information collected confidential. None of the data collected could be used to identify patients. ID numbers only were used.
The database created, using information exclusively from the questionnaires available through wufoo, was not linked in any way to the electronic medical records of the patients studied.
The empirical evidence gathered over the last few decades about cognitive biases and the use of heuristics in decision-making under conditions of uncertainty  has not yet been satisfactorily transferred to the medical field. In fact, there are major differences of opinion about the role of heuristics in clinical decision-making and the effectiveness and risks of their use [24, 25].
Furthermore, some recent reviews [26, 27] warn of the lack of studies analysing the relationship between the reasoning process and diagnostic errors, despite the relevance of the link. This lack of empirical research is particularly pronounced in PC, even though it is the setting where most clinical encounters take place and where the clinical decision-making process is performed under the greatest conditions of uncertainty. Some authors  believe that the next step in diagnostic error research would be to better understand how hypotheses developed under experimental conditions translate into real-world situations. Despite its methodological limitations, our study could lead to interesting conclusions to better understand the diagnostic process in real-world PC settings, especially with regard to its determining factors, both situational (overwork, fatigue, stress) or experience-related (training, years of experience).
It can be difficult to collect information about the clinical reasoning process at the time it takes place. However, this can be overcome through the use of recently developed and widely used information technologies in the form of hardware (tablets) and software (apps with links to websites that collect data).
The methodology proposed allows the diagnostic process as a whole to be assessed, using both prospective and retrospective sources . One major strength of the method is the involvement of the participating physicians themselves, particularly the physician being assessed in the interview analysing his/her cognitive processes. This approach makes it possible to check, adjust and increase the information available to the evaluators, turning the audit into a process of ongoing improvement of the physician’s practice.
The study is affected by volunteer selection bias, the Hawthorne effect, and the selection criteria used (only cases described as dyspnoea by the physician at the time of consultation are included, excluding false negatives). There is also a differential classification bias, in the sense that it is known that diagnostic error will be underestimated for several reasons: the physician already knows which symptom is being studied, is aware that he/she is going to be observed, and over time there may be a learning effect in the care of patients with this symptom.
There are no objective criteria defining the optimal diagnostic process in patients with dyspnoea attended by PC physicians, so there is no gold standard with which the processes performed can be compared. However, in order to have a reference with which to compare the diagnostic processes performed, we followed the procedure used in other similar studies : review of the best scientific evidence available, review and consensus by expert professionals, and design, pilot and application of a questionnaire for the audit.
Given the study period used in the analysis, it is possible that in some cases the diagnostic process may be deemed finished when it is not, and this means that certain errors may not be detected. There are 3 main causes of losses during follow-up: a) leaving the study; b) death by another cause; and c) “administrative losses” caused by early termination of the study due to reasons other than those initially predicted. To reduce this bias, all of the information available about the lost cases will be collected in order to have as much data as possible about the losses, to quantify their causes and thus evaluate the validity of the study.
Widely available, easy-to-use, uncomplicated electronic devices have been used to facilitate recording of the cognitive and diagnostic processes. Each physician uses a tablet with a 3G connection to record the information in the health centres and in patients’ homes. By storing the data on a website as soon as it is recorded, it is immediately available for analysis. However, the possibility of under-recording and delays in the recording process cannot be ruled out, and these could limit the validity of the results obtained.
The medical history review process may also be subject to hindsight bias, i.e. analysis of the diagnostic processes may be affected by prior knowledge of the final result.
The medical history audit is currently being carried out.
This paper sets out a methodology for studying the diagnostic process in PC adapted from previous works by Zwaan et al. [18, 23]. It also takes advantage of the possibilities that new technologies offer in terms of immediate recording of the clinical decision-making process, in order to assess the possible use of heuristics (representativeness, availability and anchoring) during the diagnostic process. Reason’s taxonomy, which categorises actions into unintentional (slips and lapses) and intentional ones (mistakes and infractions), is used as a reference for identifying possible suboptimal cognitive acts that physicians may have performed during the dyspnoea diagnostic process. Discussing the results obtained in the audit with each physician will facilitate learning about the diagnostic process, increasing knowledge of the key factors determining clinical reasoning.
The methodology also makes it possible to analyse the influence of clinical experience and situational factors (overwork, fatigue, stress) on the diagnostic process.
Suboptimal cognitive acts
Spanish society of family and community medicine.
McDonald KM, Matesic B, Contopuolos-Ionnidis DG, Lonhart J, Schmidt E, Pineda N, Ionnidis JP: Patient safety strategies targeted at diagnostic errors: a systematic review. Ann Intern Med. 2013, 158: 381-389. 10.7326/0003-4819-158-5-201303051-00004.
Graber ML: The incidence of diagnostic error in medicine. BMJ Qual Saf. 2013, 22 (Suppl 2): doi:10.1136/bmjqs-2012-001615
Graber ML, Wachter RM, Cassel CK: Bringing diagnosis into the quality and safety equations. JAMA. 2012, 308: 1211-1212. 10.1001/2012.jama.11913.
Berner ES, Graber ML: Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008, 121: S2-S23.
Elstein AS: Heuristics and biases: selected errors in clinical reasoning. Acad Med. 1999, 74: 791-794. 10.1097/00001888-199907000-00012.
Newman-Toker DE, Makary MA: Measuring diagnostic errors in primary care: the first step on a path forward. Comment on “Types and origins of diagnostic errors in primary care settings”. JAMA Intern Med. 2013, 173: 425-426. 10.1001/jamainternmed.2013.225.
Leape LL, Brennan TA, Laird N, Lawthers AG, Localio AR, Barnes BA, Hebert L, Newhouse JP, Weiler PC, Hiatt H: The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991, 324: 377-384. 10.1056/NEJM199102073240605.
Newman-Toker DE, Pronovost PJ: Diagnostic errors–the next frontier for patient safety. JAMA. 2009, 301: 1060-1062. 10.1001/jama.2009.249.
Singh H, Giardina TD, Meyer AN, Forjuoh SN, Reis MD, Thomas EJ: Types and origins of diagnostic errors in primary care settings. JAMA Intern Med. 2013, 173 (6): 418-425. 10.1001/jamainternmed.2013.2777.
Welch G, Schwartz L, Woloshin S: Overdiagnosed: Making People Sick in the Pursuit of Health. 2011, Boston: Beacon Press
Stanovich KE, West RF: Individual differences in reasoning: implications for the rationality debate. Behav Brain Sci. 2000, 23: 645-665. 10.1017/S0140525X00003435.
Graber ML, Franklin N, Gordon R: Diagnostic error in internal medicine. Arch Intern Med. 2005, 165: 1493-1499. 10.1001/archinte.165.13.1493.
Balla J, Heneghan C, Goyder C, Thompson M: Identifying early warning signs for diagnostic errors in primary care: a qualitative study. BMJ Open. 2012, 2: e001539-
Gilovich T, Griffin D, Kahneman D: Heuristics and Biases. The Psychology of Intuitive Judgment. 2002, Cambridge: Cambridge University Press
Kahneman D, Slovic P, Tversky A: Judgement Under Uncertainty: Heuristics and Biases. 1982, Cambridge: Cambridge University Press
Norman GR, Eva KW: Diagnostic error and clinical reasoning. Med Educ. 2010, 44: 94-100. 10.1111/j.1365-2923.2009.03507.x.
Kahneman D: Thinking, Fast and Slow. 2011, New York: Farrar, Straus and Giroux
Zwaan L, Thijs A, Wagner C, van der Wal G, Timmermans DR: Design of a study on suboptimal cognitive acts in the diagnostic process, the effect on patient outcomes and the influence of workload, fatigue and experience of physician. BMC Health Serv Res. 2009, 9: 65-10.1186/1472-6963-9-65.
Reason J: Human Error. 1990, New York: Cambridge University Press
Knottnerus JA: Medical decision making by general practitioners and specialists. Fam Pract. 1991, 8: 305-307. 10.1093/fampra/8.4.305.
Starfield B: Primary Care: Concept, Evaluation, and Policy. 1992, New York: Oxford University Press
Wachter R: Patient Safety At Ten: Unmistakable Progress, Troubling Gaps. Health Affairs. 2010, 29: 165-173. 10.1377/hlthaff.2009.0785.
Zwaan L, Thijs A, Wagner C, van der Wal G, Timmermans DR: Relating faults in diagnostic reasoning with diagnostic errors and patient harm. Acad Med. 2012, 87: 149-156. 10.1097/ACM.0b013e31823f71e6.
Croskerry P, Singhal G, Mamede S: Cognitive debiasing 1: origins of bias and theory of debiasing. BMJ Qual Saf. 2013, 22 (Suppl 2): 1-7. doi:10.1136/bmjqs-2012-001712
Eva KW, Norman GR: Heuristics and biases–a biased perspective on clinical reasoning. Med Educ. 2005, 39: 870-872. 10.1111/j.1365-2929.2005.02258.x.
Graber ML, Kissam S, Payne VL, Meyer AN, Sorensen A, Lenfestey N, Tant E, Henriksen K, Labresh K, Singh H: Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf. 2012, 21 (7): 535-557. 10.1136/bmjqs-2011-000149.
Singh H, Graber ML, Kissam SM, Sorensen AV, Lenfestey NF, Tant EM, Henriksen K, LaBresh KA: System-related interventions to reduce diagnostic errors: a narrative review. BMJ Qual Saf. 2012, 21 (2): 160-170. 10.1136/bmjqs-2011-000150.
Zwaan L, Schiff GD, Singh H: Advancing the research agenda for diagnostic error reduction. BMJ Qual Saf. 2013, 22 (Suppl 2): 1-6. doi:10.1136/bmjqs-2012-001624
The pre-publication history for this paper can be accessed here:http://0-www.biomedcentral.com.brum.beds.ac.uk/1471-2296/15/92/prepub
The authors gratefully acknowledge funding of this research from the Spanish Research Agency. Ministry of Health (Fondo de Investigaciones Sanitarias) FIS PI10/01468 and the European Regional Development Fund (ERDF). The funder of this study had no role in study design, data analysis, data collection, data interpretation, or writing of the report. The views expressed are those of the authors and not necessarily of the funding bodies. We are also grateful to the physicians who have taken part in the study and Professor Laura Zwaan for her willingness to discuss some aspects of the project.
The approach used in this project was presented as a poster presentation at the Diagnostic Error in Medicine 6th International Conference held in Chicago (2013).
The authors declare that they have no competing interests.
SM directed the study design process, planned the review and adaptation, drafted the manuscript and wrote the final version. CB participated in the study conception and design, revised the draft manuscript and critically revised the manuscript. AF participated in the study conception and design, revised the draft manuscript and critically revised the manuscript. JJM supervised the study protocol and its design, and critically revised the final manuscript. VB performed the bibliographical review and participated in the design of the audit. MM participated in the study conception, reviewed the design and implementation of the audit, and participated in the design of the dyspnoea protocol and the evaluation questionnaire. AC participated in the study design. MJO reviewed the design and implementation of the audit, and participated in the design of the dyspnoea protocol and the evaluation questionnaire. MAP reviewed the design and implementation of the audit, and participated in the design of the dyspnoea protocol and the evaluation questionnaire. JED reviewed the design and implementation of the audit, and participated in the design of the dyspnoea protocol and the evaluation questionnaire. RM reviewed the design of the initial project. All the authors revised the final version of the manuscript. All authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Minué, S., Bermúdez-Tamayo, C., Fernández, A. et al. Identification of factors associated with diagnostic error in primary care. BMC Fam Pract 15, 92 (2014) doi:10.1186/1471-2296-15-92
- Primary care
- Diagnostic errors