Abstract
Introduction
Electronic health (eHealth) applications have become a very large repository of health information which informs critical decisions relating to the diagnosis, treatment and prognosis of patients. Poor information quality (IQ) within eHealth may compromise patient safety. Evaluation of IQ in eHealth is therefore necessary to promote patient safety. An IQ framework specifies what aspects of information to assess and how to conduct the assessment. This systematic review aims to identify dimensions within existing IQ frameworks in eHealth and develop a new IQ framework for the assessment of eHealth.
Method and analysis
We will search Embase, Medline, PubMed, Cumulative Index to Nursing and Allied Health Literature, Maternity and Infant Care, PsycINFO (American Psychological Association), Global Health, Scopus, ProQuest Dissertations and Theses Global, Health Management Information Consortium and reference lists of relevant publications for articles published in English until November 2018. Studies will be selected by two independent reviewers based on prespecified eligibility criteria. Two reviewers will independently extract data in each eligible study using a prepiloted Microsoft Excel data extraction form. Thematic synthesis will be employed to define IQ dimensions and develop a new IQ framework for eHealth.
Ethics and dissemination
Ethical approval is not required for this systematic review as primary data will not be collected. The result of the review will be disseminated through publication in an academic journal and scientific conferences.
PROSPERO registration number
CRD42018097142.
Keywords: health informatics, telemedicine, quality in health care, systematic review, information quality
Strengths and limitations of this study.
This study will contribute an evidence-based framework for assessing information quality (IQ) in electronic health.
The protocol is based on Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols guidelines.
We used a theoretical framework to develop the search strategy.
The review will not provide specific information on the level of relevance of each IQ dimension included in the new IQ framework.
Introduction
Electronic health (eHealth), defined as the use of information and communication technology (ICT) in healthcare, is regarded as a modern driver of universal health coverage and quality healthcare delivery.1 A range of eHealth applications including telemedicine, electronic health records (EHRs), clinical decision support systems (CDSS), mobile health (mHealth) applications, computerised physician order entry (CPOE), electronic prescribing systems (EPS) and web-based health services (WHS), have all recorded varying levels of success in promoting access to quality health services.2 3 Over time, eHealth applications have become a very large repository of health information which informs critical decisions relating to the diagnosis, treatment and prognosis of patients.1 4 Against the backdrop of its increasing adoption, there are concerns that poor information quality (IQ) in eHealth may compromise patient safety.5 6 A number of patient safety problems associated with eHealth have been reported in the UK and the USA.7 8 These problems are classified as human factors, which are predominantly data entry errors; and technical factors, which are majorly IQ issues such as incorrect, partial and/or delayed information output.7 8 For example, incomplete information by CPOE led to medication overdose and subsequent acute renal failure in a patient.7 Evaluation of IQ in eHealth is therefore necessary to promote patient safety. Although human errors contribute to patient safety problems associated with eHealth, these factors could be addressed through clinical governance and other interventions which are beyond the scope of this review.
IQ describes the extent to which information is fit for purpose.9 Each dimension of IQ describes an aspect of information.10 11 For example, completeness is the extent to which data are sufficient for the task at hand, and timeliness is the extent to which up-to-date data are available when needed.12 An IQ framework is a systematic integration of IQ dimensions for the purpose of evaluating a specific information system.9 11 An IQ framework traditionally specifies what aspects of information to assess and how to conduct the assessment.10 11 An IQ framework also depicts the relationship existing among IQ dimensions by categorising them.12 However, some frameworks only conceptualise IQ without providing guidance on its assessment.9 11 For example, one IQ framework for EHR conceptualises IQ using 11 dimensions.12 The IQ framework depicts the relationship among ‘privacy’, ‘confidentiality’ and ‘secure access’ dimensions by grouping them in ‘security’ category.12 The dimensions and categories in the IQ framework are presented on table 1.
Table 1.
Dimensions and categories of an existing information quality (IQ) framework for electronic health record
| IQ category | IQ dimensions |
| Information | Accuracy |
| Completeness | |
| Consistency | |
| Relevance | |
| Timeliness | |
| Usability | |
| Communication | Provenance |
| Interpretability | |
| Security | Privacy |
| Confidentiality | |
| Secure access |
A number of IQ frameworks have been developed to evaluate different types of health information systems.13 However, IQ frameworks for newer types of eHealth, such as the mHealth apps, are virtually non-existent14 15 and there is no generic IQ framework for eHealth which is applicable across different eHealth applications. Also, there is no consensus on IQ dimensions that are relevant to eHealth and their definition. It is therefore necessary to synthesise the definition of the IQ dimensions within existing frameworks. Identification and definition of IQ dimensions are the first critical steps towards developing an IQ framework.16 Thus, this systematic review aims to identify and define dimensions within existing IQ frameworks in eHealth. In addition, the review will develop a new IQ framework for eHealth using the dimensions synthesised from the existing IQ frameworks for eHealth applications.
Methods and analysis
The protocol is based on the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) Protocols checklist17 presented as online supplementary file 1. The review team are healthcare and ICT professionals with research, teaching and clinical experience in eHealth. AM, JTA, KPF and JG are medical practitioners with hands-on experience in the use of eHealth applications in clinical practice. SO has a multidisciplinary background in nursing and information system. PAW is a nutritional and chronic disease epidemiologist with expertise in digital health technologies. JO has an expertise in ICT and IQ. CC is an expert in health technology assessment, systematic review and evidence synthesis. AM, JG, JO, PAW and SO also have a vast research and teaching experience in eHealth.
bmjopen-2018-024722supp001.pdf (172.4KB, pdf)
Review questions
What IQ frameworks currently exist for evaluating eHealth applications?
How are dimensions within these existing IQ frameworks defined by the authors?
Which IQ dimensions indicate how well information in eHealth is fit for diagnostic, therapeutic or prognostic purposes?
How are these IQ dimensions in eHealth related to one another?
Eligibility criteria
The traditional systematic review approach based on Population Intervention Comparator and Outcome (PICO) is not fully applicable to this study as we aim to synthesise frameworks rather than interventions. The eligibility criteria are therefore based on the Behaviour/phenomenon of interest, Health context and Model/ Theory (BeHEMoTh) procedure, which is an approach recommended specifically for identifying frameworks, theories and models in systematic review.18 19 The inclusion and exclusion criteria are presented in table 2.
Table 2.
Inclusion and exclusion criteria
| Concept | Inclusion | Exclusion |
| Phenomenon of interest | Information or data quality | Information or data quality of administrative and non-clinical data |
| Health context | Use of eHealth for clinical purposes (ie, diagnostic, therapeutic or prognostic). | Online search for health-related information, e-learning, eHealth applications for self-management. |
| Model/theory | Multidimensional framework | Individual dimension assessment |
| Language | English | Non-English |
| Publication status | Published and grey literature | None |
| Date of publication | Any | None |
| Type of study | Any | None |
eHealth, electronic health.
We will only include IQ frameworks for assessing eHealth applications used for clinical purposes. We will exclude IQ framework of eHealth applications that manage only non-clinical or administrative data, which are less likely to directly affect patient safety. We will also exclude IQ frameworks that assess online health-related information and e-learning because they are not directly used in clinical management of the patient at the point of care. We will exclude self-management applications, used by patients for health education and disease tracking purposes, as their IQ requirements are probably different compared with the applications used by healthcare professionals for clinical purposes. In addition, we will include multidimensional frameworks, but not individual dimension assessment. IQ is a multidimensional concept and individual dimension assessment cannot provide information about existing relationship between IQ dimensions. Both published and grey literatures will be included. There will be no restriction based on the date of publication. Thus, all relevant studies until November 2018 will be included. There will be no restriction based on study type as there is no evidence that one study type is superior to another when developing a framework. In addition, restriction based on study type may lead to exclusion of potentially relevant IQ frameworks.
Information sources
We will search Embase, Medline, PubMed, Cumulative Index to Nursing and Allied Health Literature, Maternity and Infant Care, PsycINFO and Global Health which are bibliographic databases for healthcare. We will search Scopus to identify eHealth publications in non-healthcare disciplines such as engineering and computer science. In addition, we will search Health Management Information Consortium and ProQuest Dissertations and Theses Global that are considered as good sources of grey literature.20 21 Finally, we will manually search the references of included studies and track their citations to identify other eligible studies using Scopus and Google Scholar.
Search strategy
The search terms will be based on three key concepts, information quality (behaviour or phenomenon of interest), eHealth (health context) and framework (models or theories). Search terms relating to each of these concepts will be developed based on the literature and thesauruses. A librarian will be consulted for inputs in the search strategy. Both Medical Subject Headings and free-text terms will be searched. Truncation and adjacency searching will be used to increase the sensitivity of the search as appropriate. The initial search strategy is available from https://www.crd.york.ac.uk/PROSPEROFILES/97142_STRATEGY_20180521.pdf
Data management
The search results will be imported into the Endnote reference management software (https://endnote.com) which will be used to delete duplicates. Duplicates not identified by the Endnotes will be manually removed. The study selection will be done with Covidence (https://www.covidence.org), a review-management software programme which is in partnership with Cochrane collaboration.
Study selection
Titles and abstracts of the studies will be screened for eligibility by two independent reviewers (KPF and JA) using the criteria outlined in table 2. Conflicts will be resolved by discussion between the two reviewers, and, if needed, by adjudication of a third independent reviewer (JO, SO or PAW). The full-text of all studies selected during screening will be reviewed independently by two reviewers (KPF and JA) with disagreement resolved as earlier described. A PRISMA flow chart will be used to show the details of the selection process.22
Data extraction
Two reviewers (KPF and SO) will independently extract data in each eligible study using a prepiloted Microsoft Excel data extraction form. Other reviewers (JO, CC, PAW, JG and AM) will review the extracted data to ensure accuracy and completeness of the data. Study details that will be extracted will include: author(s), year of publication, country, affiliation, study aim, study design and publication status. We will also extract data on the IQ framework and these will include: method of development; method of validation (if any); type of eHealth technology (eg, telemedicine, CDSS, WHS, EHR and EPS); IQ dimensions and their verbatim definition; categories of IQ dimensions (if any) and metrics of IQ dimension measurement (if any). The main data elements are further defined below:
IQ frameworks for eHealth applications: a systematic integration of IQ dimensions with the purpose of evaluating health information technologies used in the diagnosis, treatment and prognosis of patient.
IQ dimensions within the frameworks in eHealth: these are the evaluation criteria within the IQ frameworks that specify the extent to which health information technologies are fit for clinical use.
Definition of IQ dimensions in eHealth: a clear description of what aspect of information each dimension assesses.
Categories of dimensions within IQ frameworks in eHealth (if provided): IQ dimensions are often categorised to depict relationship among IQ dimensions in an IQ framework.
Metrics of measurement of IQ dimensions in eHealth (if provided): How each IQ dimension is measured, for example, questionnaire, mathematical formulae, and so on.
Quality assessment
We will assess the quality of the included studies using the appropriate Critical Appraisal Skills Programme checklist based on study design.23 Studies will not be excluded based on quality assessment outcome as this is unlikely to have any major impact on the ultimate definition of the dimensions and the construction of the IQ framework. However, the assessment is intended to provide a general idea about the quality of the existing IQ frameworks and the strength of evidence.24
Data synthesis
The IQ framework for eHealth will be developed using a thematic synthesis approach which comprises three stages.25
In the first stage, codes will be generated from the verbatim definition of IQ dimensions extracted from the existing frameworks. This will involve identification of unique concepts from each definition of IQ dimension.
In the second stage, the codes will be grouped into categories based on observed similarities and differences, and a descriptive theme will be created to capture the meaning of each category. These descriptive themes will be the IQ dimensions for the proposed framework. Each of the IQ dimensions will be defined based on the meaning of the original codes from which they were developed.
In the final stage, we will generate the analytical themes from the descriptive themes. Analytical themes are interpretation of the descriptive themes which usually go beyond the findings of the original studies. The analytical themes will be inferred from the descriptive themes (IQ dimensions) based on the interrelationship observed from the definition of the dimensions. This stage will involve organisation of the IQ dimensions (descriptive themes) into different categories conceptualised by the reviewers based on their understanding of the definition of the dimensions. Thus, the analytical themes will be the IQ categories in the new framework. All the reviewers will initially generate the analytical themes independently and then collectively as a group so as to minimise bias.25
Thus, the new IQ framework for eHealth will be derived from the thematic synthesis of the verbatim definition of IQ dimensions. The study details and other extracted framework-related information will provide an understanding of the context of the new IQ framework.
Patient and public involvement
Patients and the public will not be involved directly in the design and conduct of the review. However, the development of the review questions was informed by patient safety concerns and the experience of health professionals using eHealth applications in clinical practice.
Ethics and dissemination
Ethical approval is not required for this systematic review because primary data will not be collected. This systematic review protocol is registered in the International Prospective Register of Systematic Reviews (http://www.crd.york.ac.uk/PROSPERO).26 The result of the review will be disseminated through publication in an academic journal and scientific conferences.
Discussion
This systematic review aims to identify and define IQ dimensions as well as construct a new IQ framework for eHealth. This newly developed framework will specify aspects of eHealth information that should be assessed to determine if such information is fit for diagnostic, therapeutic or prognostic purposes.
This review is the first attempt to develop an evidence-based IQ framework using a systematic review approach, to the best of our knowledge. The use of a theoretical framework to develop the search strategy may also be considered as a strength of the review. However, the generation of analytical themes from descriptive themes in thematic synthesis has been described as controversial because it is influenced by the insight and judgement of the reviewers,25 but we believe that the multidisciplinary perspectives and vast experience of the reviewers will rather add values to data synthesis in this study. A limitation of this review is that the new IQ framework will be unable to provide specific information on the level of relevance of each IQ dimension. We are planning a subsequent international online Delphi study to address this limitation.
Finally, it is expected that the adoption of a transparent and rigorous systematic review approach methodology in this study will result in an evidence-based IQ framework for eHealth. Assessment of eHealth using the evidence-based IQ framework could identify poor IQ issues and potentially forestall associated patient safety problems.7 8
Supplementary Material
Acknowledgments
The authors thanks to Rebecca Jones, library manager and liaison librarian at the Charring Cross Library, Imperial College London, for useful advice on strategies for our literature search.
Footnotes
Contributors: KPF and JO conceived the study. KPF drafted the protocol manuscript. JTA, SO, PAW, CC, AM, JG and JO revised the manuscript for important intellectual content; and contributed to the methodology including search strategy, study selection, data extraction and data analysis. AM is the clinical lead and JO is the guarantor of the review.
Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests: None declared.
Provenance and peer review: Not commissioned; externally peer reviewed.
Patient consent for publication: Not required.
References
- 1. World Health Organization. Global diffusion of eHealth: making universal health coverage achievable. Report of the third global survey on eHealth; 156 http://www.wipo.int/amc/en/mediation/rules%0AReport of the third global survey on eHealth Global Observatory for eHealth. [Google Scholar]
- 2. Black AD, Car J, Pagliari C, et al. . The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011;8:e1000387 10.1371/journal.pmed.1000387 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Alotaibi YK, Federico F. The impact of health information technology on patient safety. Saudi Med J 2017;38:1173–80. 10.15537/smj.2017.12.20631 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. US Department of Health and human Services. Expanding the reach and impact of consumer e-health tools. 2006. http://www.health.gov/communication/ehealth/ehealthtools/default.htm.
- 5. Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med 2009;6:e1000126 10.1371/journal.pmed.1000126 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Clarke A, Adamson J, Watt I, et al. . The impact of electronic records on patient safety: a qualitative study. BMC Med Inform Decis Mak 2016;16:62 10.1186/s12911-016-0299-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Magrabi F, Ong MS, Runciman W, et al. . Patient safety problems associated with heathcare information technology: an analysis of adverse events reported to the US Food and Drug Administration. AMIA Annu Symp Proc 2011;2011:853–7. [PMC free article] [PubMed] [Google Scholar]
- 8. Magrabi F, Baker M, Sinha I, et al. . Clinical safety of England’s national programme for IT: a retrospective analysis of all reported safety events 2005 to 2011. Int J Med Inform 2015;84:198–206. 10.1016/j.ijmedinf.2014.12.003 [DOI] [PubMed] [Google Scholar]
- 9. Lee YW, Strong DM, Kahn BK, et al. . AIMQ: a methodology for information quality assessment. Inf Manage 2002;40:133–46. 10.1016/S0378-7206(02)00043-5 [DOI] [Google Scholar]
- 10. Alkhattabi M, Neagu D, Cullen A. Information quality framework for e-learning systems. Knowl Manag E-Learning An Int J 2010;2:162–70. [Google Scholar]
- 11. Eppler MJ, Wittig D. Conceptualizing information quality: a review of information quality frameworks from the last ten years. Conference on Information Quality 2000. https://www.researchgate.net/publication/220918665_Conceptualizing_Information_Quality_A_Review_of_Information_Quality_Frameworks_from_the_Last_Ten_Years. [Google Scholar]
- 12. Almutiry O, Wills G, Alwabel A. Toward a framework for data quality in cloud-based health information system. Int Conf Inf Soc 2013:153–7. [Google Scholar]
- 13. Chen H, Hailey D, Wang N, et al. . A review of data quality assessment methods for public health information systems. Int J Environ Res Public Health 2014;11:5170–207. 10.3390/ijerph110505170 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Boudreaux ED, Waring ME, Hayes RB, et al. . Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl Behav Med 2014;4:363–71. 10.1007/s13142-014-0293-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Stoyanov SR, Hides L, Kavanagh DJ, et al. . Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3:e27 10.2196/mhealth.3422 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Kerr K. The institutionalisation of data quality in the New Zealand health sector. 2006.
- 17. Shamseer L, Moher D, Clarke M, et al. . Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:g7647–25. 10.1136/bmj.g7647 [DOI] [PubMed] [Google Scholar]
- 18. Carroll C, Booth A, Leaviss J, et al. . "Best fit" framework synthesis: refining the method. BMC Med Res Methodol 2013;13:37 10.1186/1471-2288-13-37 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J 2015;32:220–35. 10.1111/hir.12108 [DOI] [PubMed] [Google Scholar]
- 20. Haig A, Dozier M. BEME Guide no 3: systematic searching for evidence in medical education–Part 1: Sources of information. Med Teach 2003;25:352–63. 10.1080/0142159031000136815 [DOI] [PubMed] [Google Scholar]
- 21. Boukacem‐Zeghmouri C, Schöpfel J. Document supply and open access: an international survey on grey literature. Interlending & Document Supply 2006;34:96–104. 10.1108/02641610610686012 [DOI] [Google Scholar]
- 22. Moher D, Liberati A, Tetzlaff J, et al. . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535 10.1136/bmj.b2535 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Critical Appraisal Skills Programme. CASP checklists. 2018. https://casp-uk.net/casp-tools-checklists/.
- 24. Popay J, Roberts H, Sowden A, et al. . Guidance on the Conduct of Narrative Synthesis in Systematic Reviews, 2006. [Google Scholar]
- 25. Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 2008;8:45 10.1186/1471-2288-8-45 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Fadahunsi K, Akinlua J, O’Connor S, et al. . Information quality frameworks in eHealth: a systematic review. 2018. https://www.crd.york.ac.uk/prospero/export_record_pdf.php. [DOI] [PMC free article] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2018-024722supp001.pdf (172.4KB, pdf)