This article has Open Peer Review reports available.
Readiness of the Belgian network of sentinel general practitioners to deliver electronic health record data for surveillance purposes: results of survey study
© Boffin et al; licensee BioMed Central Ltd. 2010
Received: 26 January 2010
Accepted: 25 June 2010
Published: 25 June 2010
In order to proceed from a paper based registration to a surveillance system that is based on extraction of electronic health records (EHR), knowledge is needed on the number and representativeness of sentinel GPs using a government-certified EHR system and the quality of EHR data for research, expressed in the compliance rate with three criteria: recording of home visits, use of prescription module and diagnostic subject headings.
Data were collected by annual postal surveys between 2005 and 2009 among all sentinel GPs. We tested relations between four key GP characteristics (age, gender, language community, practice organisation) and use of a certified EHR system by multivariable logistic regression. The relation between EHR software package, GP characteristics and compliance with three quality criteria was equally measured by multivariable logistic regression.
A response rate of 99% was obtained. Of 221 sentinel GPs, 55% participated in the surveillance without interruption from 2005 onwards, i.e. all five years, and 78% were participants in 2009. Sixteen certified EHR systems were used among 91% of the Dutch and 63% of the French speaking sentinel GPs. The EHR software package was strongly related to the community and only one EHR system was used by a comparable number of sentinel GPs in both communities. Overall, the prescription module was always used and home visits were usually recorded. Uniform subject headings were only sometimes used and the compliance with this quality criterion was almost exclusively related to the EHR software package in use.
The challenge is to progress towards a sentinel network of GPs delivering care-based data that are (partly) extracted from well performing EHR systems and still representative for Belgian general practice.
The Belgian network of sentinel general practitioners (GPs) was developed in 1979, drawing on experiences of the Weekly Returns Service of the Royal College of General Practitioners and the Dutch Sentinel Stations in the Netherlands . In the first European study among similar networks, this type of sentinel surveillance was described as "a system that keeps a watchful eye on a sample of the population by supplying regular and standardized reports on the incidence and main epidemiological characteristics of specific diseases and procedures in primary health care" .
Nowadays, the importance of epidemiological research on health problems in primary care is well accepted and the characteristics of primary care based research networks have been described more than once [3–6]. In 2002-2003, site visits to seven "well-established surveillance networks" in primary care showed that four of them, among which the Belgian network, were still using paper forms . Automatic data collection obtained from electronic medical records (EHR) was one of the recommendations of the inventory. Considerations and requests to shift to a paperless registration have been uttered since several years by both managers and members of the Belgian sentinel network. Proposals ranged from web-based data entry to automatic extraction of EHR data, either routinely collected or recorded purposively for the surveillance system. While EHR systems are primarily designed for clinical information management they might also be used for research. Opportunities and challenges of EHR data for research purposes have already been described [7–9].
In order to develop a surveillance system that is (partly) based on extraction of EHR, a substantial number of sentinel GPs should use a limited, manageable number of the 20 government-certified EHR systems on the Belgian market. Also, the well-guarded representativeness of the network should be preserved. Our annual surveys showed that age and sex distributions in the network of sentinel GP were highly similar to those of all Belgian GPs and those living in the Northern and Southern region .
At present there is no report providing an overview of the distribution and features of EHR systems in Belgian general practice and the functions used by GPs. The rate of use of certified EHR systems in the Belgian sentinel GPs is known by our annual surveys. But so far we did not analyze the pooled data set of all sentinel GP's over the past years. Neither did we know to what extent the EHR data of the sentinel GPs are fit for research purposes. This exploitability should be evaluated by data quality criteria and in reference to features of Belgian general practice and the (health) policy system.
Belgium is known for its high rate of home visits and out-of-hours care in general practice . The policy system has evolved towards a federal model composed of communities with specific cultural identities and different languages. The Dutch speaking community (northern part of Belgium) and the French speaking community (southern part of Belgium) have jurisdiction over matters that are linked to people rather than territory, such as health and social support insofar as they are not part of the social security system. The sentinel network is supported by the two concerned public health Ministries, having set their own health priorities based on the health needs of their communities. This explains the importance of a random sample of sentinel GPs in both communities.
In order to deliver data fit for research, the sentinel GPs should comply with some quality criteria on the management of patient information. For instance, free text notes might be vital in daily patient care, for research purposes they are rather useless. Easy retrievability of drug treatment codes and diagnostic codes were expressed in rate of use of the prescription module and rate of use of the system-provided (uniform) subject headings for symptoms and diagnoses. A government certification requirement for EHR systems is that subject headings should be exchangeable to codes of the International Classification of Primary Care (ICPC). A third criterion measures the rate by which data of patients seen in home visits are recorded in the EHR system. Omission of cases from home visits could bias study results as they often concern aged people. Due non-existing valid measures of the fit of EHR data for research, we copied these criteria from a survey of a random sample of Belgian GPs by the Belgian ResoPrim project (2003-2008) involving co-author VVC [12, 13].
Our research questions were: How many sentinel GPs were using a certified EHR system the last year they participated and what specific software package? To what extent is this use associated with the key characteristics of the sentinel GPs? What is the level of compliance with three data quality criteria? To what extent is this compliance associated with the key characteristics of the sentinel GPs and the EHR software package?
This study was part of a larger research project, i.e. the postal survey held yearly among all participating sentinel GPs. The data of this study were collected by five annual surveys between 2005 and 2009. All data concern the last year of participation, apart from two variables concerning 2009. For the variables practice organisation (solo/group), use of a (certified) EHR system (yes/no), EHR software package and frequency of recording of home visits in EHR, data from the last year of participation were used. Data on frequency of use of prescription module and subject headings are from spring 2009. No validation of the main outcome measures, three data quality criteria, was available. Frequency of compliance to data quality criteria were measured by Likert scales [never (1), sometimes (2), usually (3), always (4)]. A shortened version of the questionnaire can be viewed in additional file 1. For each quality criterion a dummy was constructed with a value of 1 if the rate of use equalled or exceeded the median value.
Bivariate correlation between the three quality criteria compliance rates was expressed in Spearman's r. Odds ratio's were calculated to explore the relation between independent variables, i.e. four key characteristics of the sentinel GPs (age group, gender, community, practice organisation), EHR software package and compliance with data quality criteria. Four key characteristics were included in a multivariable logistic regression with the use of a certified EHR system as a dependent variable. The compliance with three quality criteria was equally measured by multivariable logistic regression models. Only the top three of EHR software packages was included in the models by constructing a categorical variable with the remaining certified EHR software packages as reference group.
Data were collected from 221 sentinel GPs. Median age was 52 years (interquartile range, 47-58), 67% of the GPs were men, 53% were living in the Dutch community and 28% were organised in a group practice. More than half (55%) of them participated in the surveillance without interruption from 2005 onwards, i.e. all five years and 78% of the GPs were participating in 2009. Three of 215 GPs having participated between 2005 and 2008 did not respond to the survey (participation rate in 2009 was still unknown in November 2009).
Gender, age, language community and practice organisation as determinants of the use of certified EHR systems: results from a multivariable logistic regression (N = 221)
Most frequently used certified EHR systems by language community of the sentinel GPs (N = 173)
French speaking sentinel GPs
Dutch speaking sentinel GPs
N = 51 (of 65)°
N = 85 (of 108)°
Compliance rates with three data quality criteria
Use of subject headings
Use of prescription module
Recording of home visits in EHR
N = 141,
median = 2
N = 141,
median = 4
N = 186,
median = 3
The highest correlation (all p < 0.001) was found between the use of uniform subject headings and prescription module (Spearman's r = 0.516) and the lowest between recording of home visits and, respectively, prescription module (Spearman's r = 0.283) and uniform subject headings (Spearman's r = 0.346).
EHR software package as determinant of compliance with three data quality criteria: results from multivariable logistic regression analyses
EHR software package
Use of subject headings (N = 134)*
Use of prescription module (N = 134)*
Recording home visits (N = 173)*
This study shows that a vast majority of the sentinel GPs was using a certified EHR system in the last year of participation between 2005 and 2009. This is largely due to the high use in the Dutch speaking community compared to the French. Of all 16 certified EHR systems in use, only one (Health One©) ranked in the top three in both communities. Overall, the prescription module was always used and home visits were usually recorded. Uniform subject headings were only sometimes used and the compliance with this quality criterion was exclusively related to the EHR software package in use.
Our study throws a new light on the quality of EHR data for research purposes by showing the relation with the EHR system in use. A previous study in Belgian general practice also found that data quality indicators, i.e. the completeness of drug treatment data, were quite different according to the (anonymous) EHR system . As far as we know, the relation between quality of EHR data and software package had only once been examined . This study compared the medication registration after an upgrade of the EHR software package and found an improved completeness.
Our main outcome measures were not validated in previous studies. So far, characteristics of EHR data affecting their usefulness for research were described only in a tentative way but our criteria are very similar .
Our assessment of compliance with data quality criteria was based on self reports and possibly overestimated by memory effects and social desirability. We did not observe actual practice patterns of GPs nor did we study the content of EHR, i.e. the correctness and completeness of EHR morbidity data. Yet, the quality of extracted EHR data in Belgian general practice was assessed in the ResoPrim project (2003-2005). Its preliminary conclusion was that it is still a long way from using routinely collected EHR data for epidemiological studies, mainly due their incompleteness and less to their incorrectness .
Our findings are comparable with results of three Belgian studies. The proportion of sentinel GPs using a certified EHR system (78%) is high compared to the proportion of certified and full time working Belgian GPs that were entitled in 2005 to receive a lump sum for using a certified EHR system (66%) . A survey in a random sample of 448 Belgian GPs in 2006 showed that 70% was using an EHR system, 76% of the Dutch speaking GPs and 62% of the French speaking GPs . This survey showed identical EHR systems in the top three among Dutch and the top five among French speaking GPs with equally only Health One© in common. Finally, the survey from which we copied the three quality criteria showed similar results on the frequency of use of uniform subject headings and the prescription module among 1028 Belgian GPs . Only the proportion of GPs recording home visits usually or always in their certified EHR system was higher among sentinel GPs compared to the random sample (51% versus 41%, p < 0.05).
As expected, no significant relation was found between EHR system software and the level of recording of home visits, a practice presumed to be completely GP related. This criterion has a weak correlation with the two other criteria on diagnosis and therapy, key components of patient data. The high use of the prescription module is not surprising: a systematic review of studies on EHR data quality in primary care confirmed that prescription data have the highest rate of recording . The attribution of uniform subject headings to symptoms and diagnoses is only fully achieved by a minority of sentinel GPs. Two EHR systems appear to improve the use of the provided uniform subject headings. It seems self-evident that the human-computer interfaces of Medidoc© and Health One© are worth being studied and, possibly copied by other EHR software package vendors. Also, Health One© seems the first choice EHR software package in the transition process to (semi-) automatic data extraction as it is used both by French and Dutch speaking sentinel GPs. Yet we can not exclude that EHR systems have their own user profiles, independent of the system or not. As a matter of fact, Medidoc© is used by Intego, an EHR-based registration network of Dutch speaking GPs, but only four sentinel GPs appeared to be also trained participants in Intego .
From lessons learnt about the use of EHR for research (in networks), we retain several, multifaceted approaches to increase the quality of data . These include starting with relative simple research questions, use of pop-ups to force correct data entry, motivation of network members and support from peers and academics, financial incentives and so on. A literature review on methods of data quality improvement in general practice concluded that empirical knowledge is still missing . Although there seems to be ample space for improvement, criteria and acceptable levels of data quality are lacking. Quality assurance interventions were not described in detail but ranged from individual feedback with or without a comparison with group performance, self-auditing, group meetings to training sessions.
This study confirms that the network of sentinel GPs is highly representative of Belgian general practice, including the use of a high variety of certified EHR systems. This representativeness has a high price since developing and implementing data extracting modules for all certified EHR systems in use seems unfeasible due high costs. The challenge lies in the reconciliation between a representative network, covering both communities and high EHR data quality.
This survey shows that current use of certified EHR systems among Dutch speaking sentinel GPs was much higher compared to French speaking GPs. The level of compliance with three data quality criteria was mixed and partly related to the EHR software package. The challenge is to progress towards a network of sentinel GPs delivering care-based data that are (partly) extracted from well performing EHR systems and still representative for Belgian general practice.
The Belgian Network of Sentinel GPs is supported by the Flemish and Walloon Ministry of Welfare, Public Health and Family. We wish to thank all sentinel GPs for providing the study data.
- Van Casteren V: Thirty years Registration Network of Sentinel General Practitioners. Arch Public Health.Google Scholar
- Van Casteren V: Inventory of Sentinel Health Information Systems with GPs in the European Community. Situation up to March 1990. 1991, Brussels: Institute of Public HealthGoogle Scholar
- Deckers JG, Paget WJ, Schellevis FG, Fleming DM: European primary care surveillance networks: their structure and operation. Fam Pract. 2006, 23: 151-158. 10.1093/fampra/cmi118.View ArticlePubMedGoogle Scholar
- Fleming DM, Schellevis FG, Paget WJ: Health monitoring in sentinel practice networks: the contribution of primary care. Eur J Public Health. 2003, 13: 80-84. 10.1093/eurpub/13.suppl_1.80.View ArticlePubMedGoogle Scholar
- van Weel C, de Grauw W: Family practices registration networks contributed to primary care research. J Clin Epidemiol. 2006, 59: 779-783. 10.1016/j.jclinepi.2005.08.022.View ArticlePubMedGoogle Scholar
- Thomas P, Griffiths F, Kai J, O'Dwyer A: Networks for research in primary health care. BMJ. 2001, 322: 588-590. 10.1136/bmj.322.7286.588.View ArticlePubMedPubMed CentralGoogle Scholar
- de Lusignan S, van Weel C: The use of routinely collected computer data for research in primary care: opportunities and challenges. Fam Pract. 2006, 23: 253-263. 10.1093/fampra/cmi106.View ArticlePubMedGoogle Scholar
- Vandenberghe HE, Van Casteren V, Jonckheer P, Bastiaens H, Van der Heyden J, Lafontaine MF, De Clercq E: Collecting information on the quality of prescribing in primary care using semi-automatic data extraction from GPs' electronic medical records. Int J Med Inform. 2005, 74: 367-376. 10.1016/j.ijmedinf.2005.02.004.View ArticlePubMedGoogle Scholar
- De Clercq E, Van Casteren V, Jonckheer P, Burggraeve P, Lafontaine MF, Vandenberghe H, Lorant V, Artoisenet C, Degroote K: Research networks: can we use data from GPs' electronic health records?. Stud Health Technol Inform. 2006, 124: 181-186.PubMedGoogle Scholar
- Boffin N, Bossuyt N, Van Casteren V: Current characteristics and evolution of the Sentinel General Practitioners: data gathered in 2007. [Huidige kenmerken en evolutie van de peilartsen en hun praktijk. Gegevens verzameld in 2007]. 2009, Brussels: Institute of Public HealthGoogle Scholar
- Corens D: Health system review: Belgium. Health Systems in Transition. 2007, 9: 1-172.Google Scholar
- De Clercq E, Van Casteren V, Jonckheer P, Burggraeve P: Primary healthcare research network: the Belgian ResoPrim recommendations. Stud Health Technol Inform. 2009, 150: 38-42.PubMedGoogle Scholar
- Van Casteren V, Declercq E, Jonckheer P, Burggraeve P, Lafontaine MF: ResoPrim: sampling tool. 2009, Brussels: Institute of Public HealthGoogle Scholar
- Hiddema-van de WA, Smith RJ, van der Werf GT, Meyboom-de JB: Towards improvement of the accuracy and completeness of medication registration with the use of an electronic medical record (EMR). Fam Pract. 2001, 18: 288-291. 10.1093/fampra/18.3.288.View ArticleGoogle Scholar
- Terry AL, Chevendra V, Thind A, Stewart M, Marshall JN, Cejic S: Using your electronic medical record for research: a primer for avoiding pitfalls. Fam Pract. 2010, 27: 121-126. 10.1093/fampra/cmp068.View ArticlePubMedGoogle Scholar
- Anonymous: Registry of general practitioners: number and profile in 2005. [Register van de huisartsen: aantal en profiel van de huisartsen in 2005]. Thematic discussions. [Thematische uiteenzettingen]. 2008, Brussels: RIZIV/INAMI, 67-73.Google Scholar
- Boffin N, Cornelis E, Hubens V, Laperche J, Spinnewijn B, Vaes B, et al: Rapid Assessment: Cardiovascular Primary Prevention in Belgian Family Practice. [Cardiovasculaire Primaire Preventie in de Belgische huisartspraktijk] Bijlage voor hoofdstuk 5. Good Clinical Practice (GCP). 2007, Brussels: KCEGoogle Scholar
- Thiru K, Hassey A, Sullivan F: Systematic review of scope and quality of electronic patient record data in primary care. BMJ. 2003, 326: 1070-10.1136/bmj.326.7398.1070.View ArticlePubMedPubMed CentralGoogle Scholar
- Bartholomeeusen S, Kim CY, Mertens R, Faes C, Buntinx F: The denominator in general practice, a new approach from the Intego database. Fam Pract. 2005, 22: 442-447. 10.1093/fampra/cmi054.View ArticlePubMedGoogle Scholar
- Brouwer HJ, Bindels PJ, Weert HC: Data quality improvement in general practice. Fam Pract. 2006, 23: 529-536. 10.1093/fampra/cml040.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://0-www.biomedcentral.com.brum.beds.ac.uk/1471-2296/11/50/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.