research paper on decision support system

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 06 February 2020

An overview of clinical decision support systems: benefits, risks, and strategies for success

• Reed T. Sutton   ORCID: orcid.org/0000-0002-3009-1914 1 ,
• David Pincock 2 ,
• Daniel C. Baumgart 1 ,
• Daniel C. Sadowski 1 ,
• Richard N. Fedorak 1 &
• Karen I. Kroeker 1  

npj Digital Medicine volume  3 , Article number:  17 ( 2020 ) Cite this article

374k Accesses

1047 Citations

114 Altmetric

Metrics details

• Drug regulation
• Health services
• Medical imaging

Computerized clinical decision support systems, or CDSS, represent a paradigm shift in healthcare today. CDSS are used to augment clinicians in their complex decision-making processes. Since their first use in the 1980s, CDSS have seen a rapid evolution. They are now commonly administered through electronic medical records and other computerized clinical workflows, which has been facilitated by increasing global adoption of electronic medical records with advanced capabilities. Despite these advances, there remain unknowns regarding the effect CDSS have on the providers who use them, patient outcomes, and costs. There have been numerous published examples in the past decade(s) of CDSS success stories, but notable setbacks have also shown us that CDSS are not without risks. In this paper, we provide a state-of-the-art overview on the use of clinical decision support systems in medicine, including the different types, current use cases with proven efficacy, common pitfalls, and potential harms. We conclude with evidence-based recommendations for minimizing risk in CDSS design, implementation, evaluation, and maintenance.

Similar content being viewed by others

The potential of artificial intelligence to improve patient safety: a scoping review

Why do probabilistic clinical models fail to transport between sites

Effect of genetics clinical decision support tools on health-care providers’ decision making: a mixed-methods systematic review

Introduction: what is a clinical decision support system.

A clinical decision support system (CDSS) is intended to improve healthcare delivery by enhancing medical decisions with targeted clinical knowledge, patient information, and other health information. 1 A traditional CDSS is comprised of software designed to be a direct aid to clinical-decision making, in which the characteristics of an individual patient are matched to a computerized clinical knowledge base and patient-specific assessments or recommendations are then presented to the clinician for a decision. 2 CDSSs today are primarily used at the point-of-care, for the clinician to combine their knowledge with information or suggestions provided by the CDSS. Increasingly however, there are CDSS being developed with the capability to leverage data and observations otherwise unobtainable or uninterpretable by humans.

Computer-based CDSSs can be traced to the 1970s. At the time, they had poor system integration, were time intensive and often limited to academic pursuits. 3 , 4 There were also ethical and legal issues raised around the use of computers in medicine, physician autonomy, and who would be at fault when using the recommendation of a system with imperfect ‘explainability’. 5 Presently, CDSS often make use of web-applications or integration with electronic health records (EHR) and computerized provider order entry (CPOE) systems. They can be administered through desktop, tablet, smartphone, but also other devices such as biometric monitoring and wearable health technology. These devices may or may not produce outputs directly on the device or be linked into EHR databases. 6

CDSSs have been classified and subdivided into various categories and types, including intervention timing, and whether they have active or passive delivery. 7 , 8 CDSS are frequently classified as knowledge-based or non-knowledge based. In knowledge-based systems, rules (IF-THEN statements) are created, with the system retrieving data to evaluate the rule, and producing an action or output 7 ; Rules can be made using literature-based, practice-based, or patient-directed evidence. 2 CDSS that are non-knowledge based still require a data source, but the decision leverages artificial intelligence (AI), machine learning (ML), or statistical pattern recognition, rather than being programmed to follow expert medical knowledge. 7 Non-knowledge based CDSS, although a rapidly growing use case for AI in medicine, are rife with challenges including problems understanding the logic that AI uses to produce recommendations (black boxes), and problems with data availability. 9 They have yet to reach widespread implementation. Both types of CDSS have common components with subtle differences, illustrated in Fig. 1 .

They are composed of (1) base: the rules that are programmed into the system (knowledge-based), the algorithm used to model the decision (non-knowledge based), as well as the data available, (2) inference engine: takes the programmed or AI-determined rules, and data structures, and applies them to the patient’s clinical data to generate an output or action, which is presented to the end user (eg. physician) through the (3) communication mechanism: the website, application, or EHR frontend interface, with which the end user interacts with the system 9 .

CDSS have been endorsed by the US Government’s Health and Medicare acts, financially incentivizing CDS implementation into EHRs. 10 In 2013, an estimated 41% of U.S. hospitals with an EHR, also had a CDSS, and in 2017, 40.2% of US hospitals had advanced CDS capability (HIMSS Stage 6). 11 Elsewhere, adoption rates of EMRs have been promising, with approximately 62% of practitioners in Canada in 2013. 12 Canada has had significant endorsement from the government level, as well as Infoway, a not-for-profit corporation. 13 England has also been a world leader in healthcare IT investment, with up to 20 billion euros invested back in 2010. 13 Several countries have also managed to implement national health records, at least for patient-facing data, including Denmark, Estonia, Australia, and others. 14

The scope of functions provided by CDSS is vast, including diagnostics, alarm systems, disease management, prescription (Rx), drug control, and much more. 15 They can manifest as computerized alerts and reminders, computerized guidelines, order sets, patient data reports, documentation templates, and clinical workflow tools. 16 Each CDSS function will be discussed in detail throughout this review, with the potential and realized benefits of these functions, as well as unintended negative consequences, and strategies to avoid harm from CDSS. Methodology used to inform the review is shown in Box 1 .

Box 1. Methods and sources used for this overview

MEDLINE search 1980-January 2018. Key words: CDSS, diagnostic decision support system/DDSS, personal health record/PHR decision support, EHR decision support

Hand searches of the references of retrieved literature

University libraries searching for texts on clinical decision support systems and other keywords mentioned above

Personal and local experience working with healthcare technology and decision support systems

Functions and advantages of CDSS

Patient safety.

Strategies to reduce medication errors commonly make use of CDSS (Table 1 ). Errors involving drug-drug interactions (DDI) are cited as common and preventable, with up to 65% of inpatients being exposed to one or more potentially harmful combinations. 17 CPOE systems are now designed with drug safety software that has safeguards for dosing, duplication of therapies, and DDI checking. 18 The types of alerts generated by these systems are among the most disseminated kind of decision support. 19 However, studies have found a high level of variability between how alerts for DDIs are displayed (e.g., passive or active/disruptive), which are prioritized, 20 , 21 and in the algorithms used to identify DDIs. 18 , 22 Systems often have varying degrees of irrelevant alerts presented, and there is no standard for how best to implement which alerts to providers. The US Office of the National Coordinator for Health Information Technology has developed a list of ‘high-priority’ list of DDIs for CDS, which has reached various levels of endorsement and deployment in CDSS’ of other countries including the U.K., Belgium, and Korea. 20 , 21 , 23

Other systems targeting patient safety include electronic drug dispensing systems (EDDS), and bar-code point-of-care (BPOC) medication administration systems. 24 These are often implemented together to create a ‘closed loop’, where each step of the process (prescribing, transcribing, dispensing, administering) is computerized and occurs within a connected system. At administration, the medication is automatically identified through radio-frequency identification (RFID) or barcodes and crosschecked with patient information and prescriptions. This presents another target for CDSS and the potential benefit is the prevention of medication administration errors occurring at the ‘bedside’ (opposed to further upstream). Adoption is relatively low, partly due to high technology requirements and costs. 25 However; studies show good efficacy for these systems in reducing errors. 26 Mohoney et al. showed that many of these systems can be combined with CPOE and CDSS simultaneously, with reduced prescribing error rates for drug allergy detection, excessive dosing, and incomplete or unclear ordering. 24 As with most CDSS, errors can still be made if providers omit or deliberately work around the technology. 27

CDSS also improve patient safety through reminder systems for other medical events, and not just those that are medication related. Among numerous examples, a CDSS for blood glucose measurement i n the ICU was able to decrease the number of hypoglycemia events. 28 This CDSS automatically prompted nurses to take a glucose measurement according to a local glucose monitoring protocol, which specified how often measurements should be done according to specific patient demographics and previous glucose levels/trends. 28

Overall, CDSS targeting patient safety through CPOE and other systems have generally been successful in reducing prescribing and dosing errors, contraindications through automated warnings, drug-event monitoring and more. 29 Patient safety can be considered a secondary objective (or requirement) of almost all types of CDSS, no matter the primary purpose for their implementation.

Clinical management

Studies have shown CDSS can increase adherence to clinical guidelines. 30 This is significant because traditional clinical guidelines and care pathways have been shown to be difficult to implement in practice with low clinician adherance. 31 , 32 The assumption that practitioners will read, internalize, and implement new guidelines has not held true. 33 However, the rules implicitly encoded in guidelines can be literally encoded into CDSS. Such CDSS can take a variety of forms, from standardized order sets for a targeted case, alerts to a specific protocol for the patients it pertains to, reminders for testing, etc. Furthermore, CDSS can assist with managing patients on research/treatment protocols, 34 tracking and placing orders, follow-up for referrals, as well as ensuring preventative care. 35

CDSS can also alert clinicians to reach out to patients who have not followed management plans, or are due for follow-up, and help identify patients eligible for research based on specific criteria. 36 A CDSS designed and implemented at Cleveland Clinic provides a point-of-care alert to physicians when a patient’s record matches clinical trial criteria. 37 The alert prompts the user to complete a form which establishes eligibility and consent-to-contact, forwards the patient’s chart to the study coordinator, and prints a clinical trial patient information sheet.

Cost containment

CDSS can be cost-effective for health systems, through clinical interventions, 38 decreasing inpatient length-of-stay, CPOE-integrated systems suggesting cheaper medication alternatives, 39 or reducing test duplication. A CPOE-rule was implemented in a pediatric cardiovascular intensive care unit (ICU) that limited the scheduling of blood count, chemistry and coagulation panels to a 24-h interval. 40 This reduced laboratory resource utilization with a projected cost savings of $717,538 per year, without increasing length of stay (LOS), or mortality.

CDSS can notify the user of cheaper alternatives to drugs, or conditions that insurance companies will cover. In Germany, many inpatients are switched to drugs on hospital drug formularies. After finding that 1 in 5 substitutions were incorrect, Heidelberg hospital developed a drug-switch algorithm and integrated it into their existing CPOE system. 41 The CDSS could switch 91.6% of 202 medication consultations automatically, with no errors, increasing safety, reducing workload and reducing cost for providers.

Administrative functions

CDSS provide support for clinical and diagnostic coding, ordering of procedures and tests, and patient triage. Designed algorithms can suggest a refined list of diagnostics codes to aid physicians in selecting the most suitable one(s). A CDSS was conceived to address inaccuracy of ICD-9 emergency department(ED) admission coding (ICD is International Statistical Classification of Diseases, standardized codes used to represent diseases and diagnoses). 42 The tool used an anatomographical interface (visual, interactive representation of the human body) linked to ICD codes to help ED physicians accurately find diagnostic admission codes faster.

CDSS can directly improve quality of clinical documentation. An obstetric CDSS featured an enhanced prompting system, significantly improving documentation of indications for labor induction and estimated fetal weight, compared to control hospital. 43 Documentation accuracy is important because it can directly aid clinical protocols. For example, a CDSS was implemented to ensure patients were properly vaccinated following splenectomy, to combat the increased risk of infections (including pneumococcal, Haemophilus influenzae , meningococcal, etc.) that comes with spleen removal. However, the authors found that 71% of patients with the term ‘splenectomy’ in their EHR did not have it documented on their problem list (which was what triggers the CDSS alert). 44 A supplemental CDSS was then developed to enhance problem list documentation of splenectomy, 45 and improve the utility of the original vaccination CDSS.

Diagnostics support

CDSS for clinical diagnosis are known as diagnostic decision support systems (DDSS). These systems have traditionally provided a computerized ‘consultation’ or filtering step, whereby they might be provided data/user selections, and then output a list of possible or probable diagnoses. 46 Unfortunately, DDSS have not had as much influence as other types of CDSS (yet) for reasons including negative physician perceptions and biases, poor accuracy (often due to gaps in data availability), and poor system integration requiring manual data entry. 47 , 48 The latter is improving with better EHR-integration and standardized vocabulary like Snomed Clinical Terms.

A good example of an effective DDSS is one which was created by Kunhimangalam et al. 49 for diagnosis of peripheral neuropathy using fuzzy logic. Through 24 input fields which include symptoms and diagnostic test outputs, they achieved 93% accuracy compared to experts at identifying motor, sensory, mixed neuropathies, or normal cases. While this has great utility, especially in countries with less access to established clinical experts, there is also a desire for systems that can supplement specialist diagnostics. DXplain is an electronic reference based DDSS that provides probable diagnosis based on clinical manifestations. 50 In a randomized control trial involving 87 family medicine residents, those randomized to use the system showed significantly higher accuracy (84% vs. 74%) on a validated diagnosis test involving 30 clinical cases. 50

Given the known incidence of diagnostic errors, particularly in primary care, 51 there is a lot of hope for CDSS and IT solutions to bring improvements to diagnosis. 52 We are now seeing diagnostic systems being developed with non-knowledge-based techniques like machine learning, which may pave the way for more accurate diagnosis. The Babylon AI powered Triage and Diagnostic System in the U.K. is a good example of the potential, but also of the work that still has to be done before these systems are ready for primetime. 53 , 54

Diagnostics support: imaging

Knowledge-based imaging CDSS are typically used for image ordering, where CDSS can aid radiologists in selecting the most appropriate test to run, providing reminders of best practice guidelines, or alerting contraindications to contrast, for example. 55 An interventional CDS for image ordering at Virginia Mason Medical Center was shown to substantially decrease the utilization rate of lumbar MRI for low back pain, head MRI for headache, and sinus CT for sinusitis. 56 The CDS required a series of questions to be answered by providers prior to image ordering (POC), to verify appropriateness. Importantly, if an image was denied, an alternative was suggested by the system. Another commercialized example is RadWise®, which guides clinicians to the most relevant imaging order by analyzing patient symptoms and matching them with a large database of diagnoses, while also providing appropriate use recommendations at the point of care. 57

There is great interest in non-knowledge based CDS for enhanced imaging and precision radiology (‘radiomics’). 58 , 59 With images accounting for increasing amounts of medical data, but requiring extensive manual interpretation, providers need technologies to aid them in extracting, visualizing, and interpreting. 60 AI technologies are proving capable of providing insights into data beyond what humans can. 61 To do so, these technologies make use of advanced pixel recognition and image classification algorithms, most prominently: deep learning (DL). 62 IBM Watson Health, DeepMind, Google, and other companies are at the forefront, developing products for use in tumor detection, 63 medical imaging interpretation, 64 diabetic retinopathy diagnosis, 65 Alzheimer’s diagnosis through multimodal feature learning, 62 and countless more. IBM Watson’s ‘Eyes of Watson’, has been able to combine image recognition of a brain scan with text recognition of case descriptions to provide comprehensive decision support (or what IBM describes as a ‘cognitive assistant’). 60

Several projects have been able to demonstrate performance that is disputably ‘on par’ with human experts. 65 , 66 , 67 , 68 For example, Google’s team trained a deep convolutional neural network (CNN) to detect diabetic retinopathy (blood vessel damage in the eye) from a dataset of 130,000 retinal images with a very high sensitivity and specificity. 65 The algorithms performance was on par with US board certified ophthalmologists. Another study just recently published by the Stanford group demonstrated a CNN for detecting arrhythmias on electrocardiogram that exceeded the accuracy (F1 and sensitivity with matched specificity) of the average cardiologist on all rhythm classes. 68 With the current rate of progress, some experts controversially speculate that in 15–20 years, the majority of diagnostic imaging interpretation will be done (or at least pre-processed) by computers. 69 For the time being however, we should think of these early systems as an addition or augmentation to a clinician’s available toolset.

Diagnostics support: laboratory and pathology

Another subset of diagnostics where CDSS can be useful is laboratory testing and interpretation. Alerts and reminders for abnormal lab results are simple and ubiquitous in EHR systems. CDSS can also extend the utility of lab-based tests for the purpose of avoiding riskier or more invasive diagnostics. In Hepatitis B and C testing, liver biopsies are considered the gold standard for diagnosis, while non-invasive lab tests are not accurate enough to be accepted. However; AI models are being developed that combine multiple tests (serum markers, imaging, and gene tests) to produce much greater accuracy. 70 There is also application for CDSS as an interpretation tool where a test’s reference ranges are highly personalized, for example age, sex, or disease subtypes. 71

Pathology reports are crucial as decision points for many other medical specialties. Some CDSS can be used for automated tumor grading. This was done for urinary bladder tumor grading and estimating recurrence, with up to 93% accuracy. 72 The same has been done for brain tumor classification and grading. 73 There are many other examples including computerized ECG analysis, automated arterial blood gas interpretation, protein electrophoresis reports, and CDSS for blood cell counting. 46

Patient-facing decision support

With the advent of the ‘Personal Health Record’ (PHR), we are seeing CDS functionality integrated, similar to EHRs, with the patient as the end user or ‘manager’ of the data. This is a great step towards patient-focused care, and CDS-supported PHRs are the ideal tool to implement shared decision-making between patient and provider, specifically because CDSS can remove a ‘lack of information’ as a barrier to a patient’s participation in their own care. 74 PHRs are frequently designed as an extension of commercial EHR software, or as standalone web-based or mobile-based applications. 75 When connected to EHRs, PHRs can have a two way relationship, whereby information entered directly by the patient can be available to their providers, and also information in the EHR can be transmitted to the PHR for patients to view. 76

One of the earliest PHRs, the “Patient Gateway”, was simply a dashboard for patients to view medications and labs, and communicate with their physicians. 77 This has expanded and some systems now allow patients to modify their own record of care, effecting the EHR data as well. 78 Another example is Vanderbilt University’s MyHealthAtVanderbilt, a PHR fully integrated into the institutional EHR. In addition to disease-targeted delivery of patient educational materials, they incorporated a Flu Tool for patients with flu-like symptoms to decide the level of care they need and then help them seek treatment. 79 Symptom tracking is a useful and common feature of PHRs, but the variety of collected data is virtually limitless, from allergies to insurance coverage to prescription and medication information. 80 Furthermore, PHRs and other patient monitoring applications can be designed to collect information from health devices and other wearables, to create actionable insights for providers. An excellent example exists in diabetes care. Many systems are already in use, 81 but one in particular pioneered by the Stanford School of Medicine uses a wearable glucose monitor which transmits data to an Apple device (HealthKit). 82 Apple has made HealthKit interoperable with the Epic EHR and Epic PHR, “MyChart”. This successfully allows providers to monitor glucose trends in their patients in between visits, and contact them through MyChart for follow up or urgent recommendations. The pilot study demonstrated improved provider workflow, communication with patients, and ultimately quality of care. 82 Various other medical fields are deploying similar systems for monitoring that combines PHR/EHR, wearable technologies, and CDSS, including but not limited to heart failure (cardiology), hypertension, sleep apnea, palliative/elder care, and more.

It is worth noting that as PHRs have become more advanced with CDSS capabilities, there has also been increasing emphasis on the design of these systems to serve shared decision making between patient and provider, and to be interactive tools to make patients more knowledgeable/involved in their own care. PHRs that only serve as a repository for health information are now seen as missing the mark, particularly by patients themselves. 75

Pitfalls of CDSS

Fragmented workflows.

CDSS can disrupt clinician workflow, especially in the case of stand-alone systems. Many early CDSS were designed as systems that required the provider to document or source information outside their typical workspace. CDSS also disrupt workflow if designed without human information processing and behaviors in mind. In response, CDSS have been designed using the ‘think-aloud’ method to model practitioners’ workflow and create a system with better usability. 83

Disrupted workflow can lead to increased cognitive effort, more time required to complete tasks, and less time face-to-face with patients. Even when CDSS are well integrated within existing information systems, there can be disconnect between face-to-face interactions and interaction with a computer workstation. Studies have found that practitioners with more experiential knowledge are less likely to use, and more likely to override CDSS. 84

Alert fatigue and inappropriate alerts

Studies have found up to 95% of CDSS alerts are inconsequential, and often times physicians disagree with or distrust alerts. 85 Other times they just do not read them. If physicians are presented with excessive/unimportant alerts, they can suffer from alert fatigue. 86

Disruptive alerts should be limited to more life-threatening or consequential contraindications, such as serious allergies. However; even allergy alerts can be incorrect, and clinicians will often verify themselves, especially if the source is another site/hospital/practitioner. 85 , 87 Medication alerts can also be specialty specific, but irrelevant when taken out of context. For example, an alert against using broad-spectrum antibiotics such as vancomycin may be inappropriate in ICU. 85 An alert against duplicate medications may be inappropriate in inflammatory bowel disease clinics, where the same class of drug can be applied through different administration routes for increased effect.

Impact on user skill

Prior to CPOE and CDSS, healthcare providers, pharmacists, and nurses were relied upon exclusively to double-check orders. CDSS can create the impression that verifying the accuracy of an order is unnecessary or automatic. 85 This is an important myth to dispel.

It is also important to consider the potential long-term effect of a CDSS on users. Over time a CDSS can exert a training effect, so that the CDSS itself may no longer be required. Coined the “carry-over effect”, it is most likely with CDSS that are educational in nature. 88 Conversely, providers may develop too much reliance or trust on a CDSS for a specific task. 89 This could be compared to using a calculator for mathematical operations over a long period of time, and then having poorer mental math skills. It is potentially problematic as the user has less independence and will be less equipped for that task should they switch to an environment without the CDSS.

CDSS may be dependent on computer literacy

Lack of technological proficiency can be hindering when engaging with a CDSS. 90 , 91 This can vary by the design details of the CDSS, but some have been found to be overly complex, relying too much on user skill. 90 , 92 Systems should aim to stay as close to the core functionality of the pre-existing system as possible. Regardless, all new systems have a learning period, and so baseline evaluations of users’ technological competence may be appropriate. Further training can then be provided to facilitate full use of CDSS capabilities, 93 or more explicit guidance incorporated into the CDSS’ recommendations themselves. 94 This information could be implemented as info buttons to be non-disruptive. 95

System and content maintenance

Maintenance of CDSS is an important but often neglected part of the CDSS life-cycle. This includes technical maintenance of systems, applications and databases that power the CDSS. Another challenge is the maintenance of knowledge-base and its rules, which must keep apace with the fast-changing nature of medical practice and clinical guidelines. Even the most advanced healthcare institutions report difficulty keeping their systems up to date as knowledge inevitably changes. 85 Order sets and the algorithmic rules behind the CDSS have been identified as particularly difficult. 85

Operational impact of poor data quality and incorrect content

EHRs and CDSSs rely on data from external, dynamic systems and this can create novel deficiencies. As an example, some CDSS modules might encourage ordering even when the hospital lacks adequate supplies. In a study by Ash et al. 85 , a number of experts indicated that at their hospital, Hemoccult tests or pneumococcal vaccine inventories run out quickly, but this is not communicated to the CDSS.

Medication and problem lists can be problematic, if not updated or used appropriately. At one site, the medication list might be a list of dispensations, which means patients may or may not be taking them(and thus must still be asked in person). 85 Other medication lists are generated from CPOE orders only, thus still requiring manual confirmation that patients are taking the medication. Systems that make it easy to distinguish these are ideal. It is also a major area where PHRs could create a solution, by collecting medication adherence data directly from patients.

In poorly designed systems, users may develop workarounds that compromise data, such as entering generic or incorrect data. 85 The knowledge base of CDSS is dependent on a centralized, large clinical data repository. Quality of data can affect quality of decision support. If data collection or input into the system is unstandardized, the data is effectively corrupted. You may design a system for use at the point-of-care, but when applied to real world environments and data, will not be utilized properly. The importance of using informational standards such as ICD, SNOMED, and others, cannot be understated.

Lack of transportability and interoperability

Despite ongoing development for the better part of three decades, CDSS (and even EHRs in general) suffer from interoperability issues. Many CDSS exist as cumbersome stand-alone systems, or exist in a system that cannot communicate effectively with other systems.

What makes transportability so difficult to achieve? Beyond programming complexities that can make integration difficult, the diversity of clinical data sources is a challenge. 96 There is a reluctance or perceived risk associated with transporting sensitive patient information. Positively, interoperability standards are continuously being developed and improved, such as Health Level 7 (HL7) and Fast Healthcare Interoperability Resources (FHIR). These are already being utilized in commercial EHR vendors. 97 Several government agencies, medical organizations and informatics bodies are actively supporting and some even mandating the use of these interoperability standards in health systems. 98 , 99 , 100

The cloud also offers a potential solution to interoperability (and other EHR ailments such as data sync, software updating, etc. 101 ). Cloud EHRs have open architecture, newer standards, and more flexible connectivity to other systems. 102 It is also a common misconception that data stored on a cloud is more vulnerable. This is not necessarily true. Web-based EHRs are required to store data in high-level storage centers with advanced encryption and other safeguards. They must comply with national data security standards including the Health Insurance Portability and Accountability Act (HIPAA) in the USA, Personal Information Protection and Electronic Documents Act (PIPEDA) in Canada, or the Data Protection Directive and General Data Protection Regulation (GDPR) in Europe, to name a few. 103 They can be just as safe (or just as vulnerable) as traditional, server-based architecture. 103 In fact, there are often fewer people who have access to unencrypted data in cloud storage centers vs. server-based records. 103

Financial challenges

Up to 74% of those with a CDSS said that financial viability remains a struggle. 104 Outset costs to set up and integrate new systems can be substantial. Ongoing costs can continue to be an issue indefinitely as new staff need to be trained to use the system, and system updates are required to keep pace with current knowledge.

Results from cost analyses of CDSS implementations are mixed, controversial, and sparse. 105 , 106 , 107 , 108 Whether an intervention is cost-effective depends on a wide range of factors, including those specific to the environment, both political and technological. 105 Cost benefit assessment in itself can be limited, with challenges such as a lack of standardized metrics. 107 This is an emerging research area and much work needs to be done to advance our understanding of the financial effects of CDSS.

CDSS have been shown to augment healthcare providers in a variety of decisions and patient care tasks, and today they actively and ubiquitously support delivery of quality care. Some applications of CDSS have more evidence behind them, especially those based on CPOE. Support for CDSS continues to mount in the age of the electronic medical record, and there are still more advances to be made including interoperability, speed and ease of deployment, and affordability. At the same time, we must stay vigilant for potential downfalls of CDSS, which range from simply not working and wasting resources, to fatiguing providers and compromising quality of patient care. Extra precautions and conscientious design must be taken when building, implementing, and maintaining CDSS. A portion of these considerations were covered in this review, but further review will be required in practice, especially as CDSS continue to evolve in complexity through advances in AI, interoperability, and new sources of data.

Osheroff, J. et al. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide . (HIMSS Publishing, 2012).

Sim, I. et al. Clinical decision support systems for the practice of evidence-based medicine. J. Am. Med Inf. Assoc. Jamia. 8 , 527–534 (2001).

Article   CAS   Google Scholar  

De Dombal, F. Computers, diagnoses and patients with acute abdominal pain. Arch. Emerg. Med . 9 , 267–270 (1992).

Shortliffe, E. H. & Buchanan, B. G. A model of inexact resoning in medicine. Math. Biosci. 379 , 233–262 (1975).

Google Scholar  

Middleton, B., Sittig, D. F. & Wright, A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb. Med. Inform. 25 (S 01), S103–S116 (2016).

Article   Google Scholar  

Dias, D. Wearable health devices—vital sign monitoring, systems and technologies. https://doi.org/10.3390/s18082414 (2018).

Berner, E. S. (Ed.). Clinical Decision Support Systems (Springer, New York, NY, 2007).

Osheroff, J., Pifer, E., Teigh, J., Sittig, D. & Jenders, R. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. (HIMS, 2005).

Deo, R. C. Machine learning in medicine. Circulation 132 , 1920–1930 (2015).

Article   PubMed   PubMed Central   Google Scholar  

HITECH Act Enforcement Interim Final Rule. https://www.hhs.gov/hipaa/for-professionals/special-topics/HITECH-act-enforcement-interim-final-rule/index.html .

Electronic medical record adoption model requirements. https://www.himssanalytics.org/emram , Accessed 25 Aug 2019 (2017).

Chang, F. & Gupta, N. Progress in electronic medical record adoption in Canada Recherche Les progrès dans l’ adoption du dossier médical électronique au Canada. Canadian Family Physician. 61 , 1076–1084 (2015).

Healthcare Information and Management Systems Society (HIMSS). Electronic Health Records: A Global Perspective, 2nd edn. https://s3.amazonaws.com/rdcms-himss/files/production/public/HIMSSorg/Content/files/Globalpt1-edited%20final.pdf (2010).

Nøhr, C. et al. Nationwide citizen access to their health data: analysing and comparing experiences in Denmark, Estonia and Australia. 1–11. https://doi.org/10.1186/s12913-017-2482-y (2017).

Omididan, Z. & Hadianfar, A. The role of clinical decision support systems in healthcare (1980-2010): a systematic review study. Jentashapir Sci.-Res Q. 2 , 125–134 (2011).

Kabane, S. M. Healthcare and the Effect of Technology: Developments, Challenges and Advancements: Developments, Challenges and Advancements. Medical Information Science Reference (2010).

Vonbach, P., Dubied, A., Krähenbühl, S. & Beer, J. H. Prevalence of drug-drug interactions at hospital entry and during hospital stay of patients in internal medicine. Eur. J. Intern. Med. 19 , 413–420 (2008).

Article   PubMed   Google Scholar  

Helmons, P. J., Suijkerbuijk, B. O., Nannan Panday, P. V. & Kosterink, J. G. W. Drug-drug interaction checking assisted by clinical decision support: a return on investment analysis. J. Am. Med. Inf. Assoc. Jamia. 22 , 764–772 (2015).

Koutkias, V. & Bouaud, J. Contributions from the 2017 Literature on Clinical Decision Support. Yearb. Med. Inf. 27 , 122–128 (2018).

Phansalkar, S. et al. High-priority drug – drug interactions for use in electronic health records. J. Am. Med. Inform. Assoc . 19 , 735–743 (2012).

Cornu, P., Phansalkar, S., Seger, D. L., Cho, I. & Pontefract, S. International Journal of Medical Informatics High-priority and low-priority drug – drug interactions in di ff erent international electronic health record systems: a comparative study. Int. J. Med. Inf. 111 , 165–171 (2018).

McEvoy, D. S. et al. Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J. Am. Med. Inf. Assoc. 24 , 331–338 (2017).

Cho, I., Lee, J., Choi, J., Hwang, H. & Bates, D. W. National rules for drug-drug interactions: are they appropriate for tertiary hospitals?. J. Korean Med. Sci. 31 , 1887–1896 (2016).

Mahoney, C. D., Berard-Collins, C. M., Coleman, R., Amaral, J. F. & Cotter, C. M. Effects of an integrated clinical information system on medication safety in a multi-hospital setting. Am. J. Health Syst. Pharm. 64 , 1969–1977 (2007).

Peris-Lopez, P., Orfila, A., Mitrokotsa, A. & van der Lubbe, J. C. A. A comprehensive RFID solution to enhance inpatient medication safety. Int. J. Med. Inf. 80 , 13–24 (2011).

Levtzion-korach, O. et al. Effect of bar-code technology on the safety of medication administration. N. Engl. J. Med. 362 , 1698–1707 (2010).

van der Veen, W. et al. Association between workarounds and medication administration errors in bar-code-assisted medication administration in hospitals. J. Am. Med. Inf. Assoc. 25 , 385–392 (2018).

Eslami, S. et al. Effects of two different levels of computerized decision support on blood glucose regulation in critically ill patients. Int J. Med. Inf. 81 , 53–60 (2012).

Jia, P., Zhang, L., Chen, J., Zhao, P. & Zhang, M. The effects of clinical decision support systems on medication safety: an overview. PLoS ONE 11 , 1–17 (2016).

Kwok, R., Dinh, M., Dinh, D. & Chu, M. Improving adherence to asthma clinical guidelines and discharge documentation from emergency departments: Implementation of a dynamic and integrated electronic decision support system. Emerg. Med. Australas. 21 , 31–37 (2009).

PubMed   Google Scholar  

Davis, D. A. & Taylor-Vaisey, A. Translating guidelines into practice: a systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. Can. Med. Assoc. J. 157 , 408–416 (1997).

CAS   Google Scholar  

Michael, C., Rand, C. S., Powe, N. R., Wu, A. W. & Wilson, M. H. Why don’ t physicians follow clinical practice guidelines? a framework for improvement. Jama,. 282 , 1458–1465 (1999).

Shortliffe, T. Medical thinking: what should we do? http://www.openclinical.org/medicalThinking2006Summary2.html (2006).

Lipton, J. A. et al. Impact of an alerting clinical decision support system for glucose control on protocol compliance and glycemic control in the intensive cardiac care unit. Diabetes Technol. Ther. 13 , 343–349 (2011).

Salem, H. et al. A multicentre integration of a computer led follow up in surgical oncology is valid and safe. BJU Int . https://doi.org/10.1111/bju.14157 (2018).

Health Information Technology Foundations Module 28: Clinical Decision Support Basics. Carnegie Mellon University Open Learning Initiative. https://oli.cmu.edu/jcourse/workbook/activity/page?context=e6f7c0b180020ca600c0f4e5957d6f8c .

Embi, P. J., Jain, A., Clark, J. & Harris, C. M. Development of an electronic health record-based Clinical Trial Alert system to enhance recruitment at the point of care. AMIA Annu. Symp. Proc. 2005 , 231–235 (2005).

Calloway, S., Akilo, H. & Bierman, K. Impact of a clinical decision support system on pharmacy clinical interventions, documentation efforts, and costs. Hosp. Pharm. 48 , 744–752 (2013).

McMullin, S. T. et al. Impact of an evidence-based computerized decision support system on primary care prescription costs. Ann. Fam. Med. 2 , 494–498 (2004).

Algaze, C. A. et al. Use of a checklist and clinical decision support tool reduces laboratory use and improves cost. Pediatrics 137 , e20143019 (2016).

Pruszydlo, M. G., Walk-Fritz, S. U., Hoppe-Tichy, T., Kaltschmidt, J. & Haefeli, W. E. Development and evaluation of a computerised clinical decision support system for switching drugs at the interface between primary and tertiary care. BMC Med. Inf. Decis. Mak. 12 , 1 (2012).

Bell, C. M., Jalali, A. & Mensah, E. A decision support tool for using an ICD-10 anatomographer to address admission coding inaccuracies: a commentary. Online J. Public Health Inform. 5 , 222 (2013).

Haberman, S. et al. Effect of clinical-decision support on documentation compliance in an electronic medical record. Obstet. Gynecol. 114 , 311–317 (2009).

Turchin, A., Shubina, M. & Gandhi, T. NLP for patient safety: splenectomy and pneumovax. In Proc. AMIA 2010 Annual Symposium (2010).

McEvoy, D., Gandhi, T. K., Turchin, A. & Wright, A. Enhancing problem list documentation in electronic health records using two methods: the example of prior splenectomy. BMJ Qual Saf . https://doi.org/10.1136/bmjqs-2017-006707 (2017).

Berner E. Clinical Decision Support Systems: Theory and Practice 3rd edn. https://doi.org/10.1007/978-0-387-38319-4 (2016).

Berner, E. S. Diagnostic decision support systems: why aren’t they used more and what can we do about it? AMIA Annu. Symp. Proc . 2006 , 1167–1168 (2006).

Segal, M. M. et al. Experience with integrating diagnostic decision support software with electronic health records: benefits versus risks of information sharing. EGEMs Gener. Evid. Methods Improv. Patient Outcomes 5 , 23 (2017).

Kunhimangalam, R., Ovallath, S. & Joseph, P. K. A clinical decision support system with an integrated EMR for diagnosis of peripheral neuropathy. J. Med. Syst. 38 , 38 (2014).

Martinez-Franco, A. I. et al. Diagnostic accuracy in Family Medicine residents using a clinical decision support system (DXplain): a randomized-controlled trial. Diagn. Berl. Ger. 5 , 71–76 (2018).

Singh, H., Schiff, G. D., Graber, M. L., Onakpoya, I. & Thompson, M. J. The global burden of diagnostic errors in primary care. BMJ Qual. Saf. 26 , 484–494 (2017).

Singh, H., Meyer, A. N. D. & Thomas, E. J. The frequency of diagnostic errors in outpatient care: Estimations from three large observational studies involving US adult populations. BMJ Qual. Saf. 23 , 727–731 (2014).

Razzaki, S. et al. A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis. arXiv preprint arXiv:1806.10698 (2018).

Fraser, H., Coiera, E. & Wong, D. Safety of patient-facing digital symptom checkers. Lancet 392 , 2263–2264 (2018).

Georgiou, A., Prgomet, M., Markewycz, A., Adams, E. & Westbrook, J. I. The impact of computerized provider order entry systems on medical-imaging services: a systematic review. J. Am. Med. Inf. Assoc. 18 , 335–340 (2011).

Blackmore, C. C., Mecklenburg, R. S. & Kaplan, G. S. Effectiveness of clinical decision support in controlling inappropriate imaging. JACR 8 , 19–25 (2019).

DSS Inc. Radiology Decision Support (RadWise®). https://www.dssinc.com/products/integrated-clinical-products/radwise-radiology-decision-support/ .

Giardino, A. et al. Role of imaging in the era of precision medicine. Acad. Radiol. 24 , 639–649 (2017).

Oakden-rayner, L. et al. Precision Radiology: predicting longevity using feature engineering and deep learning methods in a radiomics framework. Sci. Rep . https://doi.org/10.1038/s41598-017-01931-w (2017).

From Invisible to Visible: IBM Demos AI to Radiologists. https://www-03.ibm.com/press/us/en/pressrelease/51146.ws . Accessed Aug 2019 (2016).

Greenspan, H., Ginneken van, B. & Summers, R. M. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35 , 1153–1159 (2016).

Suzuki, K. & Chen, Y. Artificial intelligence in decision support systems for diagnosis in medical imaging. https://doi.org/10.1007/978-3-319-68843-5 (2018).

IBM Watson Health - IBM Watson for Oncology. https://www.ibm.com/watson/health/oncology-and-genomics/oncology/ . Accessed 25 Aug 2019 (2018).

Lunit Inc. https://lunit.io/en/ . Accessed Aug 2019 (2018).

Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA - J. Am. Med. Assoc. 316 , 2402–2410 (2016).

Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542 , 115 (2017).

Article   CAS   PubMed   Google Scholar  

Rajpurkar, P. et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint ar Xiv:1711.05225 (2017).

Hannun, A. Y. et al. FOCUS | Letters FOCUS | Letters Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network letters | FOCUS letters | FOCUS. Nat. Med . https://doi.org/10.1038/s41591-018-0268-3 (2019).

Erickson, B. J. Machine Intelligence in Medical Imaging (Society for Imaging Informatics, SIIM, 2016).

Keltch, B., Lin, Y. & Bayrak, C. Comparison of AI techniques for prediction of liver fibrosis in hepatitis patients patient facing systems. J. Med. Syst . https://doi.org/10.1007/s10916-014-0060-y (2014).

Mørkrid, L. et al. Continuous age- and sex-adjusted reference intervals of urinary markers for cerebral creatine deficiency syndromes: a novel approach to the definition of reference intervals. Clin. Chem. 61 , 760–768 (2015).

Article   PubMed   CAS   Google Scholar  

Spyridonos, P., Cavouras, D., Ravazoula, P. & Nikiforidis, G. A computer-based diagnostic and prognostic system for assessing urinary bladder tumour grade and predicting cancer recurrence. Med. Inf. Internet Med. 27 , 111–122 (2002).

Tsolaki, E. et al. Fast spectroscopic multiple analysis (FASMA) for brain tumor classification: a clinical decision support system utilizing multi-parametric 3T MR data. Int. J. Comput. Assist Radio. Surg. 10 , 1149–1166 (2015).

Davis, S., Roudsari, A., Raworth, R., Courtney, K. L. & Mackay, L. Shared decision-making using personal health record technology: a scoping review at the crossroads. J. Am. Med. Inf. Assoc. 24 , 857–866 (2017).

Fuji, K. T. et al. Standalone personal health records in the United States: meeting patient desires. Health Technol. 2 , 197–205 (2012).

Tang, P. C., Ash, J. S., Bates, D. W., Overhage, J. M. & Sands, D. Z. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J. Am. Med. Inf. Assoc. 13 , 121–126 (2006).

Wald, J. S. et al. A patient-controlled journal for an electronic medical record: issues and challenges. Stud. Health Technol. Inform. 107 (Pt 2), 1166–1170 (2004).

Hanauer, D. A., Preib, R., Zheng, K. & Choi, S. W. Patient-initiated electronic health record amendment requests. J. Am. Med. Inf. Assoc. 21 , 992–1000 (2014).

Rosenbloom, S. T. et al. Triaging patients at risk of influenza using a patient portal. J. Am. Med. Inf. Assoc. 19 , 549–554 (2012).

Roehrs, A., Da Costa, C. A., Da Rosa Righi, R. & De Oliveira, K. S. F. Personal health records: A systematic literature review. J. Med. Internet Res . https://doi.org/10.2196/jmir.5876 (2017).

Benhamou, P. Y. Improving diabetes management with electronic health records and patients’ health records. Diabetes Metab. 37 (Suppl. 4), S53–S56 (2011).

Kumar, R. B., Goren, N. D., Stark, D. E., Wall, D. P. & Longhurst, C. A. Automated integration of continuous glucose monitor data in the electronic health record using consumer technology. J. Am. Med. Inf. Assoc. 23 , 532–537 (2016).

Kilsdonk, E., Peute, L. W., Riezebos, R. J., Kremer, L. C. & Jaspers, M. W. M. Uncovering healthcare practitioners’ information processing using the think-aloud method: From paper-based guideline to clinical decision support system. Int. J. Med. Inf. 86 , 10–19 (2016).

Dowding, D. et al. Nurses’ use of computerised clinical decision support systems: A case site analysis. J. Clin. Nurs. 18 , 1159–1167 (2009).

Ash, J. S., Sittig, D. F., Campbell, E. M., Guappone, K. P. & Dykstra, R. H. Some unintended consequences of clinical decision support systems. AMIA Annu Symp. Proc. AMIA Symp. AMIA Symp. 2007 , 26–30 (2007).

Khalifa, M. & Zabani, I. Improving utilization of clinical decision support systems by reducing alert fatigue: Strategies and recommendations. Stud. Health Technol. Inform. 226 , 51–54 (2016).

Van Laere, S. et al. Clinical decision support systems for drug allergy checking: systematic review. https://doi.org/10.2196/jmir.8206 (2018).

Wyatt, J. & Spiegelhalter, D. Field trials of medical decision-aids: potential problems and solutions. American Medical Informatics Association. 3–7. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2247484/ (1991).

Goddard, K., Roudsari, A. & Wyatt, J. Automation bias - A hidden issue for clinical decision support system use. Stud. Health Technol. Inform. 164 , 17–22 (2011).

Devaraj, S., Sharma, S. K., Fausto, D. J., Viernes, S. & Kharrazi, H. Barriers and facilitators to clinical decision support systems adoption: a systematic review. J. Bus Adm. Res . https://doi.org/10.5430/jbar.v3n2p36 (2014).

Leslie, S. J. et al. Clinical decision support software for management of chronic heart failure: Development and evaluation. Comput Biol. Med. 36 , 495–506 (2006).

Murray, E. et al. Why is it difficult to implement e-health initiatives? A qualitative study. Implement Sci. 6 , 6 (2011).

Lai, F., Macmillan, J., Daudelin, D. H. & Kent, D. M. The potential of training to increase acceptance and use of computerized decision support systems for medical diagnosis. Hum. Factors J. Hum. Factors Erg. Soc. 48 , 95–108 (2006).

Ojeleye, L. Ensuring effective computerised clinical decision support. Prescriber 27 , 54–56 (2016).

Cook, D. A., Teixeira, M. T., Heale, B. S. E., Cimino, J. J. & Del Fiol, G. Context-sensitive decision support (infobuttons) in electronic health records: a systematic review. J. Am. Med Inf. Assoc. 24 , 460–468 (2017).

Sujansky, W. Heterogeneous database integration in biomedicine. J. Biomed. Inform. 34 , 285–298 (2001).

Index - FHIR v3.0.1. (2018). https://www.hl7.org/fhir/index.html . Accessed July 2019.

Katehakis, D. G. Towards the Development of a National eHealth Interoperability Framework to Address Public Health Challenges in Greece. Proceedings of the First International Workshop on Semantic Web Technologies for Health Data Management, SWH@ISWC. 2164 , 1–9 (2018).

EHRIntelligence. 5 Ways States Mandate Health Information Exchange. https://ehrintelligence.com/news/5-ways-states-mandate-health-information-exchange . Accessed Aug 2019 (2015).

European Commission Report. Commission Recommendation on a European Electronic Health Record Exchange Format (C(2019)800) of 6 February 2019. https://ec.europa.eu/digital-single-market/en/news/recommendation-european-electronic-health-record-exchange-form (2019).

Bresnick, J. HealthITAnalytics. Interoperability, Low Costs Make Cloud-Based EHRs a Favorite. https://healthitanalytics.com/news/interoperability-low-costs-make-cloud-based-ehrs-a-favorite . Accessed July 2019 (2015).

Fernández-Cardeñosa, G., De La Torre-Díez, I., López-Coronado, M. & Rodrigues, J. J. P. C. Analysis of cloud-based solutions on EHRs systems in different scenarios. J. Med. Syst. 36 , 3777–3782 (2012).

Rodrigues, J. J. P. C., de la Torre, I., Fernández, G. & López-Coronado, M. Analysis of the security and privacy requirements of cloud-based electronic health records systems. J. Med. Internet Res . 15 , e186 (2013).

Kabachinski, J. A look at clinical decision support systems. Biomed. Instrum. Technol. 47 , 432–434 (2013).

O’Reilly, D., Tarride, J.-E., Goeree, R., Lokker, C. & McKibbon, K. A. The economics of health information technology in medication management: a systematic review of economic evaluations. J. Am. Med. Inf. Assoc. 19 , 423–438 (2012).

Bright, T. J. et al. Effect of clinical decision-support systems: a systematic review. Ann. Intern Med. 157 , 29–43 (2012).

Jacob, V. et al. Cost and economic benefit of clinical decision support systems (CDSS) for cardiovascular disease prevention: a community guide systematic review. J. Am. Med. Inf. Assoc. Jamia. 24 , 669–676 (2017).

Main, C. et al. Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems. Health Technol. Assess. 14 , 1–227 (2010).

CAS   PubMed   Google Scholar  

Scheepers-Hoeks, A. M., Grouls, R. J., Neef, C. & Korsten, H. H. Strategy for implementation and first results of advanced clinical decision support in hospital pharmacy practice. Stud. Health Technol. Inform . 148 , 142–148 (2009).

Edlin, R., McCabe, C., Hulme, C., Hall, P. & Wright, J. Cost effectiveness modelling for health technology assessment. https://doi.org/10.1007/978-3-319-15744-3 (2015).

Vermeulen, K. M. et al. Cost-effectiveness of an electronic medication ordering system (CPOE/CDSS) in hospitalized patients. Int J. Med, Inf. 83 , 572–580 (2014).

Okumura, L. M., Veroneze, I., Burgardt, C. I. & Fragoso, M. F. Effects of a computerized provider order entry and a clinical decision support system to improve cefazolin use in surgical prophylaxis: a cost saving analysis. Pharm. Pract. 14 , 1–7 (2016).

Osheroff, J. A. et al. A roadmap for national action on clinical decision support. J. Am. Med. Inf. Assoc. 14 , 141–145 (2007).

Greenes, R. A. Clinical Decision Support 2nd edn. The Road to Broad Adoption https://doi.org/10.1016/B978-0-12-398476-0.00035-X (2014).

Bonney, W. Impacts and risks of adopting clinical decision support systems. In: Efficient Decision Support Systems - Practice and Challenges in Biomedical Related Domain. IntechOpen https://doi.org/10.5772/711 (2011).

Bates, D. W. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J. Am. Med. Inf. Assoc. 10 , 523–530 (2003).

Kawamoto, K. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. Bmj 330 , 765–0 (2005).

Health Level Seven International - Homepage. http://www.hl7.org/ . Accessed 29 Aug 2019 (2018).

IHTSDO. History of SNOMED CT. Ihtsdo. http://www.ihtsdo.org/snomed-ct/what-is-snomed-ct/history-of-snomed-ct (2015).

Marco-Ruiz, L. & Bellika, J. G. Semantic Interoperability in Clinical Decision Support Systems: A Systematic Review. Stud. Health Technol. Inf. 216 , 958 (2015).

Angraal, S., Krumholz, H. M. & Schulz, W. L. Blockchain technology: applications in health care. Circ. Cardiovasc. Qual. Outcomes https://doi.org/10.1161/CIRCOUTCOMES.117.003800 (2017).

Ivan, D. Moving toward a blockchain-based method for the secure storage of patient records. ONC/NIST Use of Blockchain for Healthcare and Research Workshop . Gaithersburg, Maryland, United States: ONC/NIST (2016).

Eichner, J. & Das, M. Challenges and Barriers to Clinical Decision Support (CDS) Design and Implementation Experienced in the Agency for Healthcare Research and Quality CDS Demonstrations. Agency Healthc Res. Qual. Website . 29. https://healthit.ahrq.gov/sites/default/files/docs/page/CDS_challenges_and_barriers.pdf (2010).

Khalifa, M. Clinical decision support: strategies for success. Procedia Comput. Sci. 37 , 422–427 (2014).

Sittig, D. F. Electronic Health Records: Challenges in Design and Implementation . (CRC Press, 2014).

Harper, B. D. & Norman, K. L. Improving User Satisfaction: The Questionnaire for User Interaction Satisfaction Version 5.5. Proceedings of the 1st Annual Mid-Atlantic Human Factors Conference, (pp. 224–228), Virginia Beach, VA. (1993).

Lewis, J. R. The system usability scale: past, present, and future. Int J. Hum.-Comput. Interact. 34 , 577–590 (2018).

Lewis, J. R. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J. Hum.-Comput. Interact. 7 , 57–78 (1995).

Lewis, J. R. Measuring perceived usability: the CSUQ, SUS, and UMUX. Int J. Hum.-Comput Interact. 34 , 1148–1156 (2018).

Download references

Acknowledgements

The author’s (R.T.S.) work was supported by a Canada Institute of Health (CIHR) Research Graduate Scholarship (CGS-M). Thank you to Nathan Stern for discussion and initial review of the manuscript.

Author information

Authors and affiliations.

Department of Medicine, Division of Gastroenterology, University of Alberta, Edmonton, Canada

Reed T. Sutton, Daniel C. Baumgart, Daniel C. Sadowski, Richard N. Fedorak & Karen I. Kroeker

Chief Medical Information Office, Alberta Health Services, Edmonton, Canada

David Pincock

You can also search for this author in PubMed   Google Scholar

Contributions

R.T.S. reviewed the literature, drafted and revised the manuscript. D.P., D.C.S., D.C.B., K.I.K., and R.N.F. provided critical input, discussion, and revision of the manuscript. All authors approved the final submitted version of the manuscript.

Corresponding author

Correspondence to Karen I. Kroeker .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Sutton, R.T., Pincock, D., Baumgart, D.C. et al. An overview of clinical decision support systems: benefits, risks, and strategies for success. npj Digit. Med. 3 , 17 (2020). https://doi.org/10.1038/s41746-020-0221-y

Download citation

Received : 21 February 2019

Accepted : 19 December 2019

Published : 06 February 2020

DOI : https://doi.org/10.1038/s41746-020-0221-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

The association between population health management tools and clinician burnout in the united states va primary care patient-centered medical home.

• Lucinda Leung
• Susan E. Stockdale

BMC Primary Care (2024)

Health professionals’ acceptance of mobile-based clinical guideline application in a resource-limited setting: using a modified UTAUT model

• Addisalem Workie Demsash
• Mulugeta Hayelom Kalayou
• Agmasie Damtew Walle

BMC Medical Education (2024)

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

• Fabio Dennstädt
• Johannes Zink
• Nikola Cihoric

Systematic Reviews (2024)

Improve the efficiency and accuracy of ophthalmologists’ clinical decision-making based on AI technology

• Yingxuan Guo
• Changke Huang

BMC Medical Informatics and Decision Making (2024)

Exploring the role of professional identity in the implementation of clinical decision support systems—a narrative review

• Sophia Ackerhans
• Thomas Huynh
• Carsten Schultz

Implementation Science (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.10(2); 2023
• PMC10685930

Harnessing the power of clinical decision support systems: challenges and opportunities

Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China

Haili Zhang

Xingyu zong, yanping wang, associated data.

No data are available.

Clinical decision support systems (CDSSs) are increasingly integrated into healthcare settings to improve patient outcomes, reduce medical errors and enhance clinical efficiency by providing clinicians with evidence-based recommendations at the point of care. However, the adoption and optimisation of these systems remain a challenge. This review aims to provide an overview of the current state of CDSS, discussing their development, implementation, benefits, limitations and future directions. We also explore the potential for enhancing their effectiveness and provide an outlook for future developments in this field. There are several challenges in CDSS implementation, including data privacy concerns, system integration and clinician acceptance. While CDSS have demonstrated significant potential, their adoption and optimisation remain a challenge.

Introduction

Clinical decision support systems (CDSSs) have evolved significantly over the past few decades, 1 2 providing clinicians with essential tools for making informed decisions in patient care. 3 CDSSs have emerged as a promising tool for improving patient outcomes and reducing healthcare costs. These systems utilise electronic health records (EHRs), 4 medical knowledge databases and advanced algorithms (artificial intelligence (AI), machine learning (ML), etc) to assist clinicians in making more informed decisions by providing evidence-based 5 and patient-specific recommendations at the point of care. 6–8 Despite their potential benefits, there are several challenges in CDSS implementation, including data privacy concerns, system integration and clinician acceptance. 9–11 While CDSS have demonstrated significant potential, their adoption and optimisation remain a challenge.

These systems leverage AI, ML and data analytics to assist clinicians in making more informed decisions by providing evidence-based recommendations at the point of care. 12–15 While CDSS have demonstrated significant potential, their adoption and optimisation remain a challenge. This review discusses the current state of CDSS, ethical considerations and the opportunities for enhancing their effectiveness, exploring their benefits, limitations, and future prospects.

Current state of CDSS

History of cdss.

CDSSs have undergone significant development since their inception, evolving from rule-based expert systems to more advanced AI-driven tools. 16 This overview traces the history of CDSSs, highlighting key milestones and technological advancements. 17

The evolution of CDSSs has been marked by significant milestones and technological advancements, from the early rule-based expert systems to the sophisticated AI-driven tools of today. 18 As CDSSs continue to evolve, they hold tremendous potential for improving patient outcomes, 19 reducing healthcare costs and revolutionising the way healthcare providers make clinical decisions 20 ( figure 1 )

The history of CDSS. AI, artificial intelligence; CDSS, clinical decision support system.

Early beginnings (1950s–1960s)

The initial concept of CDSS emerged with the advent of electronic computers. In the late 1950s, Ledley and Lusted introduced the idea of using computers for medical decision-making in their paper ‘Reasoning Foundations of Medical Diagnosis’. This was a turning point that paved the way for future developments in the field.

Early expert systems (1970s–1980s)

The emergence of CDSSs can be traced back to the 1970s, when researchers began developing expert systems using AI techniques. Early examples of these systems include MYCIN, an antibiotic selection support system, and INTERNIST-1, which aimed to assist physicians in diagnosing complex medical cases. These systems were primarily rule-based, relying on knowledge encoded by medical experts in the form of ‘if-then’ rules.

Integration with EHRs (1990s–2000s)

As EHRs became more prevalent in the 1990s and 2000s, the integration of CDSSs with EHRs emerged as a priority. This integration enabled more seamless access to patient data, allowing CDSSs to provide context-specific recommendations based on individual patient information. Standards such as Health Level Seven and Clinical Document Architecture were developed during this period to facilitate data exchange between EHRs and CDSSs.

The rise of evidence-based medicine (late 1990s–2000s)

The late 1990s saw a growing emphasis on evidence-based medicine, which aimed to use the best available evidence to inform clinical decision-making. 21 Evidence-based medicine (EBM) is a process of systematically reviewing, appraising, and using clinical research findings to aid the delivery of optimum clinical care to patients. This shift prompted the development of CDSSs that incorporated evidence-based guidelines and clinical practice recommendations, helping clinicians to make decisions based on the latest research findings.

Advancements in AI and ML (2010s–present)

The 2010s witnessed rapid advancements in AI and ML techniques, which have significantly impacted the development of CDSSs. By leveraging large-scale data sets and advanced algorithms, these AI-driven CDSSs can provide more personalised and accurate recommendations. 22 Examples include IBM Watson Health and Google’s DeepMind, which have demonstrated the potential of AI and ML in transforming healthcare decision-making.

Mobile health and telemedicine (2010s–present)

With the widespread adoption of mobile technology and the growth of telemedicine, CDSSs have expanded beyond traditional clinical settings. 23 Mobile health (mHealth) applications and remote monitoring tools have integrated CDSSs to support patients and healthcare providers outside the clinical environment, enabling more proactive and personalised care.

Development of CDSS

The evolution of CDSS, from its inception to the modern sophisticated systems we witness today, provides a rich tapestry of progress and technological integration. Diving deeper into its development, it becomes evident that the nexus between AI, ML and data analytics plays a pivotal role in this transformation. 24 With the advent of robust ML algorithms, contemporary CDSSs have transcended these boundaries. These systems now possess the capability to not only process vast datasets 25 but also refine their recommendations continually, ensuring that they remain relevant and actionable.

As CDSS continue to evolve, research and development efforts should focus on several key areas to maximise their potential impact on healthcare. These areas include:

• Personalised medicine: CDSS can play a significant role in the growing field of personalised medicine, 26 which seeks to tailor treatments to individual patients based on their unique genetic, environmental and lifestyle factors. Integrating genomic, proteomic and other -omics data into CDSS can help clinicians identify the most effective therapies for each patient, minimising adverse effects and improving treatment outcomes.
• Predictive analytics: The incorporation of predictive analytics into CDSS can enable healthcare providers to anticipate potential complications and disease progression, facilitating early intervention and preventative care. Developing CDSS that can accurately predict outcomes based on historical patient data and other relevant factors will be critical in this regard.
• Natural language processing (NLP): As much of the clinical data stored in electronic health records is unstructured, advancements in NLP can help unlock valuable insights from these sources. By extracting and analysing relevant information from free-text clinical notes, CDSS can provide more comprehensive and accurate recommendations to clinicians.
• Real-time data integration: Integrating real-time patient data from various sources, such as wearable devices and remote monitoring systems, can enable CDSS to provide timely and actionable insights to clinicians. This data can help inform treatment decisions and enhance patient monitoring, ultimately improving patient outcomes.
• Multi-modal data analysis: The analysis of multi-modal data, including medical imaging, laboratory results and patient-reported outcomes, can provide a more holistic view of a patient’s condition. CDSS that can effectively integrate and analyse data from diverse sources will be better equipped to support clinical decision-making.
• Advancements in AI and ML: As AI and ML technologies continue to advance, 27 CDSS will likely benefit from these developments. The integration of more advanced AI and ML techniques can enable CDSS to process and analyse large volumes of data more efficiently, improve the accuracy of their recommendations and identify previously unrecognised patterns and associations. Future research should focus on developing and evaluating novel AI and ML methodologies for CDSS and exploring their potential applications in various clinical contexts.

In summary, the future of CDSS research and development should focus on addressing current limitations, expanding the use of these systems to diverse settings and adapting to emerging technologies and data sources. By fostering collaboration among stakeholders and exploring innovative solutions, CDSS can continue to evolve and play an increasingly vital role in shaping the future of healthcare delivery.

Implementation and integration

Implementing and integrating CDSS into existing healthcare systems is a complex process that requires careful planning and execution. 28–31 Here is a step-by-step guide to help you with the process:

Assess the needs and goals

Before selecting a CDSS, it is important to evaluate the specific needs and goals of your healthcare organisation. 32 Identify the areas where the CDSS can have the greatest impact and determine the desired outcomes. 33

Choose the appropriate CDSS

Evaluate various CDSS solutions available in the market based on their features, compatibility with existing systems, ease of use and scalability. Select a system that aligns with your organisation’s needs, goals and budget.

Assemble a multidisciplinary team

Form a team comprising clinical experts, IT professionals and administrative staff to oversee the implementation and integration process. This team should be responsible for developing a comprehensive plan, setting timelines, and ensuring that the project stays on track.

Develop a comprehensive plan

Create a detailed project plan, including timelines, milestones and success metrics. This plan should outline the steps needed for successful implementation and integration of the CDSS, such as data migration, system configuration, training, and pilot testing.

Data migration and integration

Migrate relevant patient data and integrate the CDSS with existing EHR systems, 34 ensuring seamless data exchange and interoperability. This step may require collaboration with CDSS vendors and EHR providers to ensure proper integration and data security. 35

System configuration and customization

Configure the CDSS to align with your organisation’s clinical workflows and preferences. Customise the system to accommodate the unique needs of your healthcare setting, such as local practice guidelines, specific diagnostic criteria, and preferred treatments.

Training and support

Provide comprehensive training to healthcare professionals who will be using the CDSS. This may include workshops, webinars and hands-on sessions. Develop a support system to address any questions or concerns that arise during the implementation process.

Pilot testing

Conduct a pilot test to evaluate the performance of the CDSS in a controlled setting. Use the feedback from the pilot test to refine the system and address any issues before full-scale implementation.

Full-Sscale implementation

Roll out the CDSS across the organisation, monitoring its performance and impact on patient care. Continuously evaluate the system’s effectiveness and make necessary adjustments to ensure that it meets the desired goals.

Continuous improvement and evaluation

Regularly assess the CDSS’s performance and gather feedback from users to identify areas for improvement. Stay up-to-date with advancements in the field and incorporate new features and updates to ensure that the system remains effective and relevant.

Beyond the aforementioned steps, integrating a CDSS requires a careful understanding of the organisational culture, including the willingness of staff to adapt to change. Recognising that each healthcare setting has its unique set of challenges, whether in terms of infrastructure, patient demographics, or prevailing practices, is pivotal. 36 37

By following these steps, healthcare organisations can successfully implement and integrate a CDSS into their practice. It is also imperative to understand that integrating CDSS does not negate the significance of human intuition and judgement. In fact, the efficacy of CDSS is maximised when human expertise synergises with technology. Regular feedback loops, wherein clinicians and healthcare professionals provide insights about the system’s functionality, can be instrumental in refining CDSS. 38 39

The future of CDSS will likely involve further advancements in AI 40 and ML. By staying attuned to these developments and continuing to address the challenges and opportunities outlined in this article, healthcare organisations can harness the full potential of CDSS to enhance patient care and optimise healthcare delivery.

Moreover, as technology continues its rapid advancement, ensuring the CDSS remains updated is paramount. This includes software updates for improved functionality, incorporating new research findings to keep the decision-making process current and integrating with newer patient care technologies.

Benefits of CDSS (PRECISE-CARING)

A CDSS is a health information technology tool that provides doctors, nurses and other healthcare professionals with clinical decision-making support in real-time. 41 CDSS can assist with diagnosis, treatment and care management by leveraging patient data, evidence-based guidelines and best practices. 42 CDSS have been shown to improve patient outcomes by streamlining clinical workflows, reducing mortality rates and facilitating evidence-based decision-making. They can also enhance clinician satisfaction by providing real-time feedback and reducing cognitive burden. 43 Moreover, there are numerous benefits of CDSS, 44 including patient-centric care, 45 46 reduced medical errors, 47 enhanced decision-making, 48–50 cost savings, 51 increased efficiency, 5 52–54 scalability, 55–57 enhanced patient safety, 58–60 compliance with guidelines and regulations, 61 62 adaptive approaches, 5 resource optimisation, 63 64 interoperability and data sharing 17 65 66 , networked collaboration, 67–69 global knowledge access and gaining foresight 70–72 (PRECISE-CARING) ( figure 2 ).

PRECISE-CARING for the benefits of CDSS. CDSS, clinical decision support system.

• Patient-centric care: CDSS facilitates the delivery of personalised care by providing tailored treatment recommendations based on each patient’s unique circumstances and medical history.
• Reduced medical errors: By providing alerts and reminders for potential medication interactions, allergies or contraindications, CDSS can help prevent medical errors and enhance patient safety.
• Enhanced decision-making: CDSS can reduce cognitive overload and human error by assisting healthcare providers in analysing complex patient data, synthesising relevant information and offering tailored treatment suggestions.
• Cost savings: CDSS can help healthcare providers reduce healthcare costs by identifying unnecessary tests, avoiding duplicate procedures and preventing complications that can result from medical errors. By supporting more accurate diagnoses and treatment plans, CDSS can reduce unnecessary tests and procedures, leading to decreased healthcare costs.
• Increased efficiency: CDSS can reduce the time spent on manual tasks, such as searching for information or calculating dosages, by providing quick access to relevant information and automating certain processes. This can save time for healthcare providers and allow them to focus on more critical aspects of patient care.
• Scalability: CDSS can be implemented across various healthcare settings, from large hospitals to small clinics, allowing healthcare providers of all sizes to benefit from improved decision-making and patient care.
• Enhanced patient safety: By reducing the risk of diagnostic errors and adverse drug events, CDSS can help minimise harm to patients and improve overall safety. 73 74
• Compliance with guidelines and regulations: By incorporating evidence-based guidelines and regulations into the system, CDSS can help healthcare providers stay compliant with industry standards and avoid penalties.
• Adaptive approaches: With CDSS, medical professionals deliver personalised care by customising treatment plans based on individual patient details and medical history.
• Resource optimisation: CDSS streamlines healthcare tasks and automates certain processes, leading to more efficient resource allocation and reduced time spent on manual tasks for healthcare professionals.
• Interoperability and data sharing: CDSS can facilitate communication between different healthcare systems, ensuring that providers have access to the most up-to-date patient information when making clinical decisions.
• Networked collaboration: CDSS fosters better communication and cooperation among healthcare teams by centralising patient data and providing a platform for sharing insights, ultimately improving patient outcomes.
• Global knowledge access and gaining foresight: CDSS serves as a valuable educational resource, connecting medical professionals to the latest research, clinical guidelines and best practices from around the world. CDSS assists in identifying patients who may be at risk for developing certain conditions, allowing for earlier interventions and potentially preventing more severe health issues in the future. 75

CDSS offer a wide range of benefits that can be captured by the acronym PRECISE-CARING. By leveraging these benefits, CDSS has the potential to revolutionise healthcare practices and significantly improve patient care quality and outcomes.

PRECISE-CARING serves as a useful reminder of how CDSS can help healthcare professionals make more informed decisions, reduce errors, streamline processes and facilitate collaboration. It also emphasises the importance of adapting to changing patient needs, optimising resource allocation and fostering a continuous learning environment to ensure the most up-to-date and evidence-based care possible.

When considering the comprehensive advantages of CDSS, it is also worth noting the empowerment of patients. As healthcare transitions towards a more patient-centric model, CDSS can significantly improve patient engagement by providing them access to easy-to-understand information, allowing them to be active participants in their care journey. 76

Furthermore, CDSS reduces variations in practice, ensuring that irrespective of the caregiver, patients receive consistent, high-quality care. By flagging potential deviations from best practice guidelines, CDSS ensures a standardised yet personalised approach to care. 77 78 In addition, as the global health community moves towards value-based care, the role of CDSS in improving healthcare quality while reducing costs becomes more pronounced. It aids in eliminating wasteful spending, optimising resource use and ensuring each patient interaction is maximally beneficial. 79 80

In summary, PRECISE-CARING highlights the key benefits and clinical significance of CDSS, underscoring the potential for these systems to revolutionise healthcare practices and enhance patient care quality and outcomes.

Evaluating the impact of CDSS

The current state of evaluating the impact of CDSSs is evolving as technology and methodologies continue to develop. 81 To determine the true value of CDSS, it is crucial to conduct rigorous evaluations that measure their impact on patient outcomes, 82 healthcare processes 9 83 and costs. These evaluations should involve the use of appropriate research designs and methodologies, such as randomised controlled trials, observational studies and cost-effectiveness analyses. The results of these evaluations can be used to inform decision-making and identify areas for improvement in CDSS design and implementation.

As CDSS adoption increases in healthcare settings, evaluating their impact becomes more crucial for ensuring positive outcomes and optimising system performance. The evaluation process is comprehensive, involving multiple factors such as clinical effectiveness, user satisfaction, cost-effectiveness and integration with existing workflows. This requires a combination of quantitative and qualitative methods to assess the CDSS’s impact accurately.

While the existing methodologies for evaluating CDSS are robust, considering the global variances in healthcare delivery is crucial. CDSS implemented in a tertiary care hospital in an urban setting might differ significantly in its impact compared with a primary care setting in a rural environment. 84 85

The evaluation of CDSS impact is an ongoing process, 86 with healthcare organisations and researchers continuously monitoring system performance, gathering user feedback and making necessary improvements to ensure the system remains effective and relevant. 87 While several studies have demonstrated the positive effects of CDSS on patient outcomes and clinical efficiency, 88 more research is needed to assess their long-term impact. 89 Future studies should examine the effects of CDSS on healthcare costs, patient satisfaction and the overall quality of care, helping to build a stronger evidence base for their implementation in practice. 90

Furthermore, the onset of global pandemics, like COVID-19, underscores the importance of agility in CDSS evaluations. Such systems should be nimble enough to incorporate new findings rapidly and ensure healthcare providers are equipped with the most recent and relevant information at all times. 91 92 Moreover, as patient care becomes increasingly digital, the role of cybersecurity in CDSS cannot be overstated. Evaluating the impact of CDSS should also encompass its resilience against cyber threats, ensuring patient data privacy and system functionality remain uncompromised. 93 94

In summary, the current state of evaluating the impact of CDSS is characterised by an increasing focus on evidence-based methodologies, data-driven analytics, data privacy, standardisation, collaboration and continuous improvement to ensure that these systems contribute to better patient care and improved healthcare outcomes. 95

Ethical considerations

The increasing use of CDSS raises a number of ethical considerations, 96 including concerns related to algorithmic bias, transparency and accountability. Future research should explore ways to address these ethical challenges, such as developing transparent and explainable algorithms, 97 incorporating diverse patient populations in the development process and establishing guidelines for the responsible use of CDSS in practice.

As CDSS become more prevalent, ethical considerations must be addressed. Issues such as patient privacy, data security and informed consent need to be carefully considered. 98–100

In conclusion, as CDSS continue to advance and evolve, addressing the challenges in data privacy, system integration and clinician acceptance will be crucial for realising their full potential in improving patient care and reducing medical errors.

CDSS have demonstrated significant potential to improve healthcare delivery, but their widespread adoption remains limited by several challenges. 101 Overcoming these obstacles will require innovative solutions and sustained commitment from healthcare providers, developers and policymakers. 102

In this review, we have highlighted the potential of CDSSs to improve patient outcomes, reduce medical errors and enhance clinical efficiency. The discussion emphasises various aspects of CDSS, including their history, development, implementation, and integration, as well as the benefits and challenges associated with their use.

While CDSS hold promise for improving patient outcomes and reducing healthcare costs, the challenges associated with their implementation cannot be ignored. 103 To overcome these challenges, a comprehensive and systematic approach is required, addressing not only technical issues but also organisational and human factors.

The potential of CDSS to transform healthcare is significant, but the challenges to their adoption and optimisation are substantial. By addressing these challenges and harnessing the opportunities outlined in this article, it is possible to create more effective CDSS that improve patient outcomes and reduce healthcare costs. Future developments in this field should focus on interoperability, transparent and explainable AI, user-centred design, continuous improvement and collaboration.

Geographical disparities in CDSS implementation and adherence

An often-underemphasised aspect of CDSS implementation is the geographical disparities that influence its adoption and effectiveness. Our affiliation and insights from various landscapes allow us to delve deeper into these nuances.

Publication bias

One of the most glaring issues is the publication bias that tends to favour high-income, English-speaking countries. The majority of CDSS literature emerges from these regions, potentially overshadowing significant findings and insights from non-English speaking countries. This bias can skew our understanding of CDSS’s universal applicability and challenges. It is vital for future research to actively seek out and incorporate studies and experiences from a wider range of geographical areas to provide a more balanced global perspective.

Cultural differences

Cultural nuances play a pivotal role in the reception and reliance on CDSS. For instance, certain cultures might lean heavily on traditional medical practices, viewing CDSS recommendations with scepticism. Conversely, in some settings, there might be an over-reliance on technology, potentially overshadowing the clinician’s expertise. Understanding these cultural subtleties is critical to customise CDSS interfaces and recommendations, ensuring better alignment with regional beliefs and practices.

Training paradigms

The varied clinician training frameworks across different geographical terrains further compound these challenges. Clinicians trained in regions where protocol adherence is paramount might find it easier to trust and follow CDSS recommendations. In contrast, those from more flexible training backgrounds might exercise more clinical judgement, potentially overlooking CDSS insights. Recognising and addressing these training paradigms can better inform CDSS design and integration strategies.

In summary, while the potential of CDSS in transforming healthcare remains undeniable, it is crucial to acknowledge and address the geographical, cultural and educational nuances that influence its global adoption. As we move forward, a more inclusive approach, taking into account these factors, will be instrumental in realising the full potential of CDSS across diverse healthcare landscapes.

Challenges and future directions

Future research should focus on addressing the current limitations of CDSS, developing new approaches for system integration and exploring novel ways to enhance clinician acceptance. 104 Additionally, more studies are needed to evaluate the long-term impact of CDSS on patient outcomes, 105 healthcare costs and clinician satisfaction. As CDSS continue to evolve, they will likely play an increasingly vital role in shaping the future of healthcare. 106

Challenges associated with CDSS implementation can be broadly categorised into technical, organisational and human factors. 28 Technical challenges include data quality and interoperability, algorithm transparency and system integration. Organisational challenges encompass resistance to change, financial constraints and regulatory issues. Human factors involve user acceptance, usability, and training.

Data privacy concerns

Despite their potential benefits, CDSS face several challenges in practice, including data privacy concerns, system integration issues and clinician acceptance. Addressing these challenges requires ongoing collaboration between stakeholders and investment in CDSS development and evaluation.

Data privacy and security are crucial concerns for healthcare providers and CDSS developers. 94 Ensuring that patient information remains confidential and secure is essential for building trust among clinicians and patients. Some potential solutions to address these concerns include adopting advanced encryption techniques, implementing strict access controls and adhering to relevant regulatory frameworks such as the Health Insurance Portability and Accountability Act (HIPAA). 107–109

Clinician acceptance

Clinician acceptance of CDSS is crucial for their successful implementation and adoption. To enhance acceptance, CDSS should be designed with a focus on usability, relevance and non-intrusiveness. Involving end-users in the development process can help to ensure that CDSS meet the needs and preferences of clinicians, ultimately promoting their acceptance and use in practice. 110

Incorporating patient preferences and values

Another important area for future CDSS research is the incorporation of patient preferences and values into the decision-making process. By integrating patient-reported outcomes and preferences, CDSS can support shared decision-making 111 and enhance patient-centred care. 112 This will require the development of new methodologies and techniques to elicit and incorporate patient preferences into CDSS algorithms effectively.

Expanding CDSS applications to underserved populations and settings

One of the future directions for CDSS research should involve expanding their use to underserved populations 113 and settings, such as rural healthcare facilities and low-resource environments. Developing CDSS solutions that are adaptable to varying resource levels and local contexts can help to address healthcare disparities and ensure that the benefits of these systems are accessible to a broader range of patients.

System integration

Integrating CDSS into existing clinical workflows and electronic health record systems 114 can be challenging, particularly in complex healthcare settings. Future research should explore innovative approaches to improve system integration, 31 such as utilising standardised data formats, adopting service-oriented architectures, and employing user-centred design principles.

The successful adoption of CDSS requires a supportive ecosystem, including strong leadership, a culture of innovation and the availability of resources for training and education. Creating this environment may involve the development of policies and incentives that encourage the use of CDSS and foster collaboration among stakeholders.

Training and education

To ensure the successful adoption of CDSS in clinical practice, clinicians must be adequately trained and educated on the use of these systems. This includes understanding the capabilities and limitations of CDSS, interpreting their recommendations and integrating these recommendations into their clinical decision-making processes. 3 Healthcare organisations should invest in training and educational resources for clinicians to promote the effective use of CDSS and ensure that their potential benefits are realised.

Tailoring CDSS to local contexts

One of the essential aspects of CDSS implementation is ensuring that the system is tailored to the specific needs and requirements of the local healthcare environment. 115 This includes adapting the system to local clinical guidelines, workflows and preferences. Customisation of CDSS can lead to higher user acceptance and better integration with existing practices.

CAUCICETCI: multifaceted strategies for CDSS advancement

The current state of research on CDSS is vibrant and multifaceted, with ongoing efforts to address various challenges and seize opportunities to enhance its effectiveness in clinical settings. 116 To enhance CDSS effectiveness, several opportunities can be explored:

• Customisability: Researchers are examining the potential for customisable CDSS to address unique clinical contexts and user preferences, ultimately enhancing CDSS adoption and utility.
• Addressing ethical and legal concerns (responsibility): As CDSS evolves, researchers are exploring ethical and legal issues, such as data privacy, patient consent, and liability, to ensure responsible use and widespread adoption.
• User-centred design: Designing CDSS that are intuitive and easy to use, with an emphasis on reducing cognitive burden and information overload for clinicians.
• Collaboration: Encouraging multi-disciplinary collaboration between healthcare professionals, software developers and data scientists to create more effective CDSS solutions.
• Integration with EHRs: Researchers are working on improving the integration of CDSS with EHRs, focusing on data exchange standards, interoperability and data security. Efforts are being made to create more efficient data flows between systems and ensure real-time access to patient information.
• Continuous updates and knowledge expansion: Research on knowledge management and maintenance for CDSS is ongoing, focusing on efficient ways to incorporate the latest medical research and best practices.
• Evaluation and feedback mechanisms: Studies are being conducted to assess the impact of CDSS on clinical outcomes, user satisfaction and cost-effectiveness, with the goal of improving CDSS design and performance.
• Transparent and explainable AI: Developing CDSS that provide not only recommendations but also explanations for their reasoning, which can help build trust and improve user acceptance.
• Continuous improvement: Incorporating feedback loops and real-time performance metrics to facilitate ongoing system refinement and adaptation to changing clinical needs.
• Interoperability and standardisation: Ensuring seamless integration with EHRs and other healthcare systems through standardised data formats and application programming interfaces.

By extracting the initials of each phrase and rearranging them, we can form the word ‘CAUCICETCI’ ( figure 3 ), which is related to our topic of improving CDSS effectiveness. In conclusion, the current state of research on CDSS is vibrant and multifaceted, with ongoing efforts to address various challenges and seize opportunities to enhance its effectiveness in clinical settings.

CAUCICETCI: multifaceted strategies for CDSS advancement. AI, artificial intelligence; CDSS, clinical decision support system; EHRs, electronic health records.

In summary, enhancing the effectiveness of CDSSs relies on addressing key opportunities, such as seamless integration with EHRs, interoperability, leveraging AI and ML, continuous knowledge updates, user-friendly interfaces, customisability, evaluation and feedback, education and training, ethical and legal considerations, and stakeholder engagement. 117 By focusing on these areas, healthcare providers can ensure that CDSS remains a valuable tool for improving clinical decision-making, optimising patient outcomes, and transforming the overall quality of care. 118

Future directions

As technology continues to evolve, the potential for CDSS to advance healthcare will grow. Future developments in AI and ML can further enhance the diagnostic and predictive capabilities of CDSS. 119 120 Additionally, expanding CDSS applications to underserved populations and settings can help address healthcare disparities and improve access to quality care. Collaborative efforts among healthcare providers, policymakers, researchers and industry partners are crucial to realise the full potential of CDSS in the years to come.

This review has provided a comprehensive overview of the current state of CDSSs, examining their development, implementation, benefits, limitations and future directions. We have discussed the challenges associated with data privacy, system integration, clinician acceptance, incorporating patient preferences, expanding CDSS applications to underserved populations and the need for training and education. Furthermore, we have explored the opportunities for enhancing CDSS effectiveness through seamless integration with EHRs, interoperability, leveraging AI and ML, continuous knowledge updates, user-friendly interfaces, customisability, and evaluation and feedback mechanisms.

In conclusion, harnessing the power of CDSS requires a multifaceted approach that addresses the barriers to implementation and optimises their effectiveness. By considering the ethical aspects, ensuring seamless integration with other healthcare IT systems, promoting clinician acceptance, focusing on continuous improvement and fostering collaboration among stakeholders, CDSS can become a powerful tool for transforming patient care and improving overall healthcare outcomes.

Acknowledgments

We would like to thank anonymous reviewers for their valuable comments and improvement suggestions that further led us to improve this paper.

ZC and NL contributed equally.

Contributors: ZC, YW and NS conceived and designed this review. ZC, HZ, HL and YY did the search. XZ and YC selected the studies for inclusion. ZC and NL drafted the manuscript. ZC, HZ and HL edited and approved the final version.

Funding: We thanked the China Academy of Chinese Medical Sciences Independent Selection Project (Z0830) and Institute of Basic Research in Clinical Medicine Independent Selection Project, China Academy of Chinese Medical Sciences Independent Selection Project (Z0830-1) for their support for this work.

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data availability statement

Ethics statements, patient consent for publication.

Not applicable.

Ethics approval

• Open access
• Published: 10 September 2024

Human factors methods in the design of digital decision support systems for population health: a scoping review

• Holland M. Vasquez 1 ,
• Emilie Pianarosa 2 ,
• Renee Sirbu 2 ,
• Lori M. Diemert 2 ,
• Heather Cunningham 3 ,
• Vinyas Harish 2 , 4 ,
• Birsen Donmez 1 &
• Laura C. Rosella 2 , 4 , 5  

BMC Public Health volume  24 , Article number:  2458 ( 2024 ) Cite this article

59 Accesses

Metrics details

While Human Factors (HF) methods have been applied to the design of decision support systems (DSS) to aid clinical decision-making, the role of HF to improve decision-support for population health outcomes is less understood. We sought to comprehensively understand how HF methods have been used in designing digital population health DSS.

Materials and methods

We searched English documents published in health sciences and engineering databases (Medline, Embase, PsychINFO, Scopus, Comendex, Inspec, IEEE Xplore) between January 1990 and September 2023 describing the development, validation or application of HF principles to decision support tools in population health.

We identified 21,581 unique records and included 153 studies for data extraction and synthesis. We included research articles that had a target end-user in population health and that used HF. HF methods were applied throughout the design lifecycle. Users were engaged early in the design lifecycle in the needs assessment and requirements gathering phase and design and prototyping phase with qualitative methods such as interviews. In later stages in the lifecycle, during user testing and evaluation, and post deployment evaluation, quantitative methods were more frequently used. However, only three studies used an experimental framework or conducted A/B testing.

Conclusions

While HF have been applied in a variety of contexts in the design of data-driven DSSs for population health, few have used Human Factors to its full potential. We offer recommendations for how HF can be leveraged throughout the design lifecycle. Most crucially, system designers should engage with users early on and throughout the design process. Our findings can support stakeholders to further empower public health systems.

Peer Review reports

Interactive decision aid systems, such as dashboards, are vital digital interfaces that support decision-makers across diverse sectors like healthcare, energy, and finance [ 1 ]. In these dynamic and often unpredictable settings, professionals must make swift and accurate decisions under pressure, where the cost of an error can be substantial. Given that human cognitive and perceptual constraints can lead to decision-making errors, these systems aim to minimize errors and enhance user decision-making.

Human Factors (HF) Engineering, along with its subdiscipline of Human Computer Interaction (HCI), represents a field poised at the intersection of human behavior and system design [ 2 ]. It is predicated on the principle of tailoring systems to match user capabilities and characteristics, thereby minimizing the mismatch between humans and the tools they use. This alignment aims to reduce cognitive and physical strain, facilitating improved performance and satisfaction. The methodologies encompass understanding user-system interactions, crafting solutions responsive to user needs, and evaluating these solutions against criteria such as decision-making accuracy, task efficiency, mental workload, and user satisfaction [ 2 ].

The emergence of HCI as a distinct field in the late 20th century represents an evolution of the HF tradition, with a specific focus on the interfaces between humans and computers [ 3 ]. While the field of HF broadly addresses the design of systems with human users, HCI hones in on the complexities of human interactions with computer systems. HCI researchers examine how individuals interact with computers, striving to make these interactions more intuitive, efficient, and pleasant. This includes studying user behavior, developing new interaction techniques, designing user interfaces, and evaluating user experiences. The relationship between HCI and HF is synergistic; while HF provides the overarching principles of user-centered design and system optimization, HCI applies these principles specifically to the design and evaluation of software systems. Throughout the manuscript we refer to HF in a broad sense, thereby encompassing HCI.

The system design process for software systems begins with a needs assessment and design requirements phase, where user, task, environment, and stakeholder analyses are conducted to define functional, non-functional, user, and regulatory requirements. This is followed by design and prototyping, involving conceptual and detailed design, as well as creating low-fidelity and high-fidelity prototypes to visualize and test concepts. Next, testing and evaluation occur through formative and summative evaluations, including usability testing and user acceptance testing to ensure the system meets requirements. Deployment involves implementation, integration, training, and launching the system. Post-deployment evaluation includes monitoring, maintenance, gathering user feedback, and implementing updates and patches based on feedback and issues, as well as planning for new releases and the system’s end-of-life [ 4 ]. Frequently, system designers employ agile methods in the software design process, which emphasizes iterative development, frequent collaboration with stakeholders, and adaptability to change throughout the project lifecycle [ 5 ].

In the context of Decision Support Systems (DSS), the contribution of HF is significant. These systems often involve complex user interfaces that must present information in a clear and actionable manner. Human-centered methodologies have advanced DSS for healthcare, aiding clinicians in making better diagnostic and therapeutic decisions and promoting patient safety [ 6 , 7 ]. Yet, challenges remain in the adoption of DSS in clinical environments due to issues rooted in usability and integration into existing workflows domains where HF provides essential insights [ 8 , 9 , 10 , 11 ]. Absent a strong emphasis on human factors principles such as user interface design and interaction paradigms, users may not adopt these systems.

The intersection of HF and HCI is particularly potent in public health, where decisions affect large populations. Public health officials undertake complex tasks that require synthesizing vast arrays of data, and here, the role of HF is to ensure that DSS are not only functionally aligned with these tasks but are also accessible and engaging for the users. As such, DSS designed for public health need to accommodate broader determinants of health, from socioeconomic factors to healthcare services. This scoping review explores the applications of human factors in the design of evidence-based DSS in population health.

Our scoping review was based on the methodological framework described by Arksey and O’Malley [ 12 ], with refinements by Levac and colleagues [ 13 ]. We also followed the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P and PRISMA-S, respectively) reporting guidelines to facilitate understanding and transparency [ 14 , 15 ]. Our detailed study protocol was published in BMJ Open in March 2022 [ 16 ]; we briefly describe these methods below.

Search strategy

Our search included peer-reviewed literature databases, manual searches, and grey literature. First, we searched 7 interdisciplinary indexed databases: Ovid MEDLINE, EMBASE, Scopus, PsycINFO, Compendex, IEEE Xplore, and Inspec. Our team included a librarian specialising in health science, and we further consulted with an engineering & computer science librarian to ensure both disciplines were captured. Details on our search strategy for each database can be found in the published protocol [ 16 ] and Supplemental Material. The MEDLINE search strategy was validated against a key set of 8 articles [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ], pre-determined by the authors and was peer reviewed using PRESS [ 25 ] by another librarian, not associated with this study to ensure accuracy and comprehensiveness. We then manually searched the reference lists of included articles and relevant reviews. The search was completed in September 2023.

Our grey literature search started with a pilot review of several public health dashboards for infectious disease surveillance, modeling and forecasting, where we identified that the information presented on these websites were insufficient for the HF aspect of this review. Thus, our grey literature search included full-text conference proceedings papers, identified through Compendex (Engineering Village), IEEE Xplore, and Inspec (Engineering Village).

Eligibility criteria

We sought to describe the HF applications to the field of population health, thus we excluded clinical applications, such as those discussing patient safety, monitoring of an individual’s health, or clinical DSS. Since HF applications in healthcare began to emerge in the 1990’s [ 26 , 27 ], our search started in 1990 to capture the potential evolution of HF applications in the public health domain. As detailed in our study protocol [ 16 ], we included studies published in English since 1990 that described the development, validation, or application guided by HF principles in the field of population health. Exclusion criteria included articles whose end-user was not public health, articles not related to HF, articles that did not describe a digital evidenced-based DSS, as well as conference abstracts, reviews (including commentaries and discussion pieces), and articles not written in English.

Screening process

The search results were integrated into Covidence [ 28 ], a systematic review management software, and duplicates were removed. Two reviewers independently screened the title and abstract of all articles according to the inclusion and exclusion criteria. Disagreements were resolved through team discussion and included a third independent reviewer as necessary. Using a similar process, selected articles then underwent full text screening by two independent reviewers, resulting in the final studies for inclusion [ 16 ].

Data abstraction and synthesis

As outlined in the published protocol [ 16 ], a data abstraction form was developed and pilot-tested by two researchers working independently of each other. The abstracted data were synthesized according to three themes: study characteristics, population health characteristics, and human factors characteristics (Table  1 ). A reviewer used the form to extract data from each article; a second reviewer verified the extraction.

We computed descriptive statistics for all extracted items, calculating the total number and percent of all studies in a particular category. We also conducted a narrative synthesis of the included studies and the application of HF in population health.

Our search yielded 21,581 unique studies, of which 153 studies met our inclusion criteria [ 19 , 21 , 22 , 23 , 24 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141 , 142 , 143 , 144 , 145 , 146 , 147 , 148 , 149 , 150 , 151 , 152 , 153 , 154 , 155 , 156 , 157 , 158 , 159 , 160 , 161 , 162 , 163 , 164 , 165 , 166 , 167 , 168 , 169 , 170 , 171 , 172 , 173 , 174 , 175 , 176 , 177 , 178 , 179 , 180 , 181 ]. Figure  1 provides a modified PRISMA flow diagram of our screening workflow. Raw data from the extraction process for the 153 included studies can be found in the Supplementary Materials.

PRIMSA flow chart for screening workflow

Study characteristics

Academic discipline of authors and year of publication.

The academic disciplines of the authors were diverse, with the majority being from Public Health (56%). Other disciplines included Multidisciplinary teams (23%), which consisted of researchers from both Public Health and Computer Science/Human-Computer Interaction/Informatics fields. Authors from solely Computer Science/Human-Computer Interaction/Human Factors or Informatics (CS/HCI/HF/Informatics) made up 20%, and those from Geographic Information Science/Geographic Science (GIS/Geographic Science) comprised 1%. The distribution of publications over the years showed that 3% were published between 2000 and 2004, 11% between 2005 and 2009, 24% between 2010 and 2014, 29% between 2015 and 2019, and 32% between 2020 and 2023 (Table  2 ).

Publication type

The types of publications varied, with peer-reviewed journal articles being the most common (76%). Conference proceedings accounted for 18% of the publications, while other types of publications made up 6% (see Table  2 ).

Publication venue type

Publications were most frequently found in Public Health (65%) venues. Additional publication venues included Informatics (13%), Computer Science/Engineering (12%), Geospatial (5%), Human Factors/Human-Computer Interaction (HF/HCI) (1%), and other disciplines (10%); see Table  2 ).

Study location

Most studies were conducted in North America (50%). Other study locations included Europe (16%), Africa (11%), Asia (10%), South America (4%), Oceania (4%), and global or multiple locations (4%; see Table  2 ).

Population health characteristics

Population health topic area.

The studies covered a range of population health topic areas. The most frequently addressed topic was infectious disease, representing 35% of the studies. Public health data and indicators were covered in 14% of studies, while maternal, newborn, and child health were the focus of 10%. Non-communicable diseases were addressed in 10% of the studies, and vaccines and drugs were the topic of 6%. Other areas included injury (4%), mental health (3%), nutrition (3%), and substance abuse (2%). Various other topics were covered in 13% of studies (Table  3 ).

Most of the studies utilized health surveillance tools, accounting for 69% of studies. Program evaluation tools and predictive modeling tools were each used in 8% of the studies. Other types of tools were employed in 14% of the studies (see Table  3 ).

Population health end-user

The end-users of the population health tools and interventions were predominantly multidisciplinary teams, representing 35% of the studies. Program planners were the end-users in 27% of studies, while public health professionals (not otherwise specified) accounted for 12%. Policy makers were the end-users in 8% of the studies, community health workers in 4%, and academia in 3%. Other end-users were identified in 12% of studies (see Table  3 ).

Population health setting

The settings refer to where the tools are intended to be used. Multiple levels of public health were the most common setting, reported in 25% of the studies. Local public health units were the intended setting in 17% of the studies, and regional public health in 16%. Public health (not otherwise specified) was the setting in 16% of the studies, while federal public health accounted for 12%. Hospitals were the intended setting in 4% of the studies, community health centres in 3%, and other settings in 8% of studies (see Table  3 ).

Human factors characteristics

Researchers primarily engaged with users during the testing and evaluation phase, followed by the post-deployment evaluation phase, the needs assessment and requirements gathering phase, and the design and prototyping phase (Table  4 ). The majority of studies ( n  = 96) involved users at only one point in the design lifecycle; 36 studies engaged users in two phases, 17 studies in three phases, and only 4 studies involved users in all four design lifecycle phases. Detailed results for how users were engaged within each phase are presented in the subsequent sections.

User needs assessment and requirements gathering

During the needs assessment and requirements gathering phase, various methods were employed to engage users and gather necessary information. Interviews were the most frequently used method, cited in 26 studies, with an average sample size of 15 participants, although 19% of these studies did not specify the sample size. Meetings, workshops, and discussions were used in 21 studies, with an average of 13 participants, but a significant number of these studies (67%) did not report sample sizes. Focus groups were conducted in 11 studies, averaging 21 participants, with 36% not specifying sample sizes. Questionnaires were used in 6 studies, with a mean sample size of 27 and all studies reporting their sample sizes. Observations and the Delphi method were each employed in 5 studies. Observations averaged at 10 participants with 20% not reporting sample sizes, while the Delphi method had a notably higher average of 84 participants, with 60% not specifying sample sizes. Less frequently used methods included usability assessments of baseline tools and task analysis (see Table  4 ).

Design and prototyping

In the design and prototyping phase, several methods were utilized to engage users and gather feedback. The most frequently used method was design-based workshops, reported in 16 studies, with an average sample size of 25 participants. Expert and stakeholder reviews were conducted in 9 studies, averaging at 3 participants. Heuristic evaluations were used in 4 studies, with an average of 4 participants. Focus groups and questionnaires were each employed in 3 studies, with focus groups averaging at 7 participants and questionnaires at 13 participants. Interviews were conducted in 2 studies with the average sample size of 18. Informal feedback was gathered in 2 studies, with an average sample size of 5 participants. The Delphi method was used in 1 study, but no information on sample size was provided. Overall, many of the qualitative methods for engaging users in the design and prototyping processes neglected to indicate their sample size (see Table  4 ).

User testing and evaluation

In the user testing and system evaluation phase, various methods and measures were employed to assess system performance and user experience. User testing, was the most frequently used method, appearing in 49 studies with an average sample size of 16 participants. Of the 49 studies that conducted user testing, 1 study [ 69 ] used an experimental framework, 11 collected quantitative data [ 21 , 47 , 48 , 65 , 69 , 117 , 132 , 138 , 146 , 162 , 177 ] including: task completion time (8 studies; [ 21 , 47 , 48 , 65 , 117 , 138 , 146 , 177 ] ), task success/accuracy (6 studies; [ 21 , 48 , 65 , 69 , 132 , 162 ]), efficiency (1 study; [ 69 ]), and the number of clicks (1 study; [ 146 ] ). Questionnaires were utilized in 43 studies, with an average sample size of 22 participants, while interviews were conducted in 21 studies, averaging at 14 participants. Informal feedback was gathered in 17 studies, with an average sample size of 8 participants, and focus groups were used in 12 studies, with an average of 13 participants. Log data was analyzed in 3 studies, and experiment [ 69 ] 1 study, with an average sample size of 33 participants. The Delphi method was used in 2 studies, with an average sample size of 15 participants. Notably, many studies using these methods neglected to specify their sample sizes, particularly for qualitative methods such as informal feedback sessions, like the qualitative methods applied in the design and prototyping phase (see Table  4 ).

Post-deployment evaluation

In the post-deployment assessment and evaluation phase, various methods were employed to gather feedback and assess system performance after it was deployed for use by end-users. Questionnaires were the most frequently used method, reported in 33 studies with an average sample size of 71 participants. Interviews were conducted in 28 studies, averaging at 44 participants, and focus groups were used in 9 studies with an average sample size of 22 participants. User testing was employed in 7 studies, with an average sample size of 11. Quantitative metrics were used in 4 of the 7 studies that conducted user testing and included task success/accuracy (3 studies; [ 23 , 149 , 152 ]), the number of clicks (1 studies; [ 146 ]), and task completion time (1 study; [ 146 ]). Additional methods included log data analysis (5 studies), informal feedback (4 studies), and observations (3 studies) with an average sample size of 15 participants. App issue reporting and experiment and A/B testing were each conducted in 2 studies, with the latter having an average sample size of 105 participants. Heuristic evaluations were used in 1 study, with an average sample size of 4 participants. Notably, again many studies, particularly those relying on qualitative methods such as informal feedback, did not specify their sample sizes (see Table  4 ).

Have HF methods been used to their full potential?

Over the past 20 years, HF methods have been increasingly applied throughout the design lifecycle of DSS for public health contexts. A variety of qualitative and quantitative methods were used, with qualitative methods used more frequently during the needs assessment and design and prototyping phases, while quantitative methods more frequently used in the two evaluation phases: user testing and evaluation and post-deployment evaluation. Indeed, qualitative methods, such as interviews and observations, provide deep, contextual insights into user needs and behaviors, ensuring a user-centered design process. They allow for flexibility and iteration, uncovering unmet needs and fostering empathy, which leads to more inclusive and effective solutions. These methods help designers create systems that truly resonate with and benefit users, which is why they are advantageous in the early phases of the design lifecycle. On the other hand, quantitative methods provide objective, measurable data that allow for statistical analysis, and benchmarking, ensuring rigorous evaluation during user testing and post-deployment phases. These methods offer precision, reproducibility, and the ability to identify trends, enabling data-driven decisions and continuous improvement of system performance.

However, our findings indicate that researchers have not been using quantitative human factors methods to their full potential in the two evaluation phases. Importantly, most user testing and evaluation approaches did not collect direct measures of performance with the system. Additionally, only 3 studies [ 69 , 121 , 127 ] employed A/B testing or experimental methods to compare new or current tools with alternatives in public health contexts. Furthermore, no study evaluated whether these DSS help public health professionals make better decisions. As such, despite following some best practices in engaging users in the system design process, there is little evidence for the efficacy of these tools in supporting users in decision making tasks. Furthermore, a large proportion of studies did not report their sample size, particularly for qualitative methods. Those that reported sample sizes for qualitative studies generally followed best practices (e.g. 6–8 participants per focus group) [ 182 ]. Most studies reported sample sizes for quantitative methods, which followed best practices using larger sample sizes than qualitative methods (e.g., 20 + per questionnaire).

Human factors vs. agile Software Development

In the field of HF, researchers have thoroughly and rigorously assessed system design in the context of safety-critical systems such as those encountered in the aviation, surface transportation, military, and nuclear domains. However, as demonstrated in this study, this approach is lacking in the design of DSS in public health. This may in part be attributed to several constraints such as time and resources. For instance, funding opportunities are more limited for public health DSS than in other domains such as military DSS. In turn, this limits the number of public health staff available to develop and systematically evaluate these systems. Against these constraints, agile approaches to development afford user engagement and feedback throughout the design lifecycle, however, they may fall short in providing robust evidence for the efficacy of DSS. Indeed, most studies identified in this review were from researchers in the public health domain. Multidisciplinary teams may open-up additional funding opportunities in addition to fostering synergy between public health domain expertise and engineering technical skills.

When should we conduct HF experiments?

In systems design, it is best practice to engage with users throughout the design lifecycle. Encouragingly, we have seen an increase in the engagement of users in the design of DSS for public health. While it may not always be feasible to conduct A/B testing or experimentation, especially under time and funding constraints, some circumstances may warrant a more thorough approach. For example, more rigorous testing may be beneficial in the context of DSS intended to support high-stakes decision-making processes. Additionally, introducing novel technologies, such as artificial intelligence (AI) and machine learning (ML), in public health necessitates thorough testing to validate their efficacy. AI and ML models can potentially enhance the speed and accuracy of epidemiological insights, enabling quicker decision-making during time-critical events like the COVID-19 pandemic [ 183 ]. However, HF challenges such as the “black box” that characterises many AI/ML tools can hinder the ability of epidemiologists to explain results and decision-makers to take confident action.

Strengths and limitations

Our scoping review has numerous strengths. Since it was designed to capture studies in both engineering and public health over the last twenty years, it has considerable breadth and comprehensiveness. Importantly, the long review period allowed us to track changes in this area over time. Our search strategy was reviewed by two librarians in both the public health sciences and engineering domains, which also improves the rigour of our search and address challenges with different nomenclature with this interdisciplinary research. We were also able to ensure that each record was reviewed by both a team member from HF engineering and one from public health, with the ability to discuss potential conflicts with a third member of the study team. This approach reduces the likelihood of false positives or negatives in terms of the studies deemed to meet inclusion criteria. Finally, our study protocol was previously peer-reviewed and published [ 16 ] and we did not deviate from our study protocol.

Our review also has some important limitations. We were only able to include studies published in English and thus we may be under-capturing studies from the Global South. Our review also does not include a full appraisal of methodological quality or risk of bias as such checklists do not exist for the study of HF in health. As DSS continue to be developed for use in clinical medicine and population health, the development of a checklist to guide rigorous Human Factors evaluations may represent a fruitful area of future work, especially by groups such as the EQUATOR network [ 184 ]. While it was difficult to summarize all potentially relevant details of our included studies due to space restrictions, we aimed to cover the most salient details for stakeholders in this space. We also present our full table of the 153 studies that met our inclusion criteria in the Supplementary Materials.

While we identified many studies that applied HF methods to design decision support tools for population health, few leveraged HF methods to their full potential. We offered several recommendations for how HF methods can be leveraged at different points within the design lifecycle. The key is to engage with users early on and throughout the design process rather than simply bringing in end-users for usability testing. In terms of testing, there is a need to consider additional metrics beyond usability and tool utility. This includes measuring task performance, mental workload, situation awareness, and, ultimately, the quality of decisions made. Furthermore, there is a greater need for more rigorous evaluations, to generate the level of evidence needed to determine if and how DSS improve public health decision-making. Overall, HF methods have great potential for enhancing the impact of dashboards and data-based decision support tools and efforts are needed to adopt best practices in design and evaluation.

Data availability

All data generated or analyzed during this study are included in this published article and its supplementary information.

Abbreviations

Artificial Intelligence

Computer Science

Decision Support Systems

Human Computer Interaction

Human Factors

Machine Learning

Bonczek RH, Holsapple CW, Whinston AB. Foundations of decision support systems. Academic; 2014.

Lee JD, Wickens CD, Liu Y, Boyle LN. Designing for people: an introduction to human factors engineering. CreateSpace; 2017.

Harrison S, Tatar D, Sengers P. The three paradigms of HCI. Alt Chi Session at the SIGCHI Conference on human factors in computing systems San Jose, California, USA. 2007. pp. 1–18.

Meister D. Human factors testing and evaluation. Elsevier; 2014.

Salah D, Paige RF, Cairns P. A systematic literature review for agile development processes and user centred design integration. Proceedings of the 18th international conference on evaluation and assessment in software engineering. 2014. pp. 1–10.

Salwei ME, Carayon P, Hoonakker PLT, Hundt AS, Wiegmann D, Pulia M, et al. Workflow integration analysis of a human factors-based clinical decision support in the emergency department. Appl Ergon. 2021;97:103498.

Article   PubMed   PubMed Central   Google Scholar  

Carayon P, Hoonakker P, Hundt AS, Salwei M, Wiegmann D, Brown RL, et al. Application of human factors to improve usability of clinical decision support for diagnostic decision-making: a scenario-based simulation study. BMJ Qual Saf. 2020;29:329–40.

Article   PubMed   Google Scholar  

Karsh B-T. Clinical practice improvement and redesign: how change in workflow can be supported by clinical decision support. Volume 200943. Rockville, MD: Agency for Healthcare Research and Quality; 2009.

Google Scholar  

Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inf Assoc. 2003;10:523–30.

Article   Google Scholar  

Sittig DF, Belmont E, Singh H. Improving the safety of health information technology requires shared responsibility: It is time we all step up. Healthcare [Internet]. 2018;6:7–12. http://www.journals.elsevier.com/healthcare-the-journal-of-delivery-science-and-innovation

Kilsdonk E, Peute LW, Jaspers MWM. Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. Int J Med Inf. 2017;98:56–64.

Article   CAS   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:1–9.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1–9.

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Syst Rev. 2021;10:1–19.

Vasquez HM, Pianarosa E, Sirbu R, Diemert LM, Cunningham HV, Donmez B et al. Human factors applications in the design of decision support systems for population health: a scoping review. BMJ Open [Internet]. 2022;12. https://bmjopen.bmj.com/content/12/4/e054330

Revere D, Dixon BE, Hills R, Williams JL, Grannis SJ. Leveraging health information exchange to improve population health reporting processes: lessons in using a collaborative-participatory design process. EGEMS (Wash DC). 2014;2:1082.

PubMed   Google Scholar  

Pike I, Smith J, Al-Hajj S, Fuselli P, Macpherson A. The Canadian atlas of child and youth injury: mobilizing injury surveillance data to launch a national knowledge translation tool. Int J Environ Res Public Health. 2017;14:982.

de Lima TFM, Lana RM, Carneiro TGS, Codeço CT, Machado GS, Ferreira LS et al. Dengueme: a tool for the modeling and simulation of dengue spatiotemporal dynamics. Int J Environ Res Public Health. 2016;13.

Ola O, Sedig K. Beyond simple charts: design of visualizations for big health data. Online J Public Health Inf. 2016;8.

Yuan M, Powell G, Lavigne M, Okhmatovskaia A, Buckeridge DL. Initial Usability Evaluation of a Knowledge-Based Population Health Information System: The Population Health Record (PopHR). AMIA Annu Symp Proc [Internet]. 2017;2017:1878–84. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85058748371&partnerID=40&md5=b859b042f33ad16f747528f271aacac0

Al-Hajj S, Pike I, Riecke BE, Fisher B. Visual analytics for public health: Supporting knowledge construction and decision-making. Proceedings of the Annual Hawaii International Conference on System Sciences. 2013. pp. 2416–23.

Scotch M, Parmanto B, Monaco V. Usability Evaluation of the Spatial OLAP Visualization and Analysis Tool (SOVAT). J Usability Stud [Internet]. 2007;2:76–95. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=pmnm2&NEWS=N&AN=26613012

Harris JK, Hinyard L, Beatty K, Hawkins JB, Brownstein JS, Nsoesie EO, et al. Evaluating the implementation of a twitter-based foodborne illness reporting tool in the city of St. Louis department of health. Int J Environ Res Public Health. 2018;15:833.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

Leape LL. Human factors meets health care: the ultimate challenge. Ergon Des. 2004;12:6–12.

Cafazzo JA, St-Cyr O. From discovery to design: the evolution of human factors in healthcare. Healthc Q. 2012;15:24–9.

Veritas Health Innovation. Covidence systematic review software. Veritas Health Innovation.

Accorsi P, Lalande N, Fabrègue M, Braud A, Poncelet P, Sallaberry A et al. HydroQual: Visual analysis of river water quality. 2014 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE; 2014. pp. 123–32.

Adini B, Verbeek L, Trapp S, Schilling S, Sasse J, Pientka K et al. Continued vigilance - development of an online evaluation tool for assessing preparedness of medical facilities for biological events. Front Public Health [Internet]. 2014;2:35. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=pmnm3&NEWS=N&AN=24783192

Al-Hajj S, Fisher B, Smith J, Pike I. Collaborative visual analytics: a health analytics approach to injury prevention. Int J Environ Res Public Health. 2017;14.

Ali H, Waruru A, Zielinski-Gutierrez E, Kim AA, Swaminathan M, De Cock KM, et al. Evaluation of an HIV-Related Mortuary Surveillance System - Nairobi, Kenya, two sites, 2015. MMWR Surveill Summ. 2018;67:1–12.

Anderson B, Coulter S, Orlowsky R, Ruzich B, Smedley R, Purvis M et al. Designing user experiences for policymakers in serious games in the domain of global food security. University of Virginia, Charlottesville, VA, United States BT – 2017 Systems and Information Engineering Design Symposium (SIEDS), 28 April 2017: IEEE; 2017. pp. 89–94.

Andersson SR, Hassanen S, Momanyi AM, Onyango DK, Lutukai MN, Chandani YK, et al. Using human-centered design to Adapt Supply chains and Digital Solutions for Community Health Volunteers in nomadic communities of Northern Kenya. Glob Health Sci Pract. 2021;9:S151–67.

Anema A, Druyts E, Hollmeyer HG, Hardiman MC, Wilson K. Descriptive review and evaluation of the functioning of the International Health regulations (IHR) annex 2. Global Health. 2012;8:1.

Azofeifa A, Yeung LF, Duke CW, Gilboa SM, Correa A. Evaluation of an active surveillance system for stillbirths in metropolitan Atlanta. J Registry Manag. 2012;39:13–36.

Bhowmick T, Robinson AC, Gruver A, MacEachren AM, Lengerich EJ. Distributed usability evaluation of the Pennsylvania Cancer Atlas. Int J Health Geogr. 2008;7:36.

Bollaerts K, De Smedt T, Donegan K, Titievsky L, Bauchau Kaatje. ORCID: http://orcid.org/0000-0001-7704-0527 VAO-B. Benefit-Risk monitoring of vaccines using an interactive dashboard: a methodological proposal from the ADVANCE Project. Drug Saf. 2018;41:775–86.

Boonchieng W, Tuanrat W, Aungwattana S, Tamdee D, Budda D. Development of a Community-based Geographic Health Information System via Mobile Phone in Saraphi District. J Computers (Taiwan). 2019;30:84–92.

Borges HL, Malucelli A, Paraiso EC, Moro CC. A physiotherapy EHR specification based on a user-centered approach in the context of public health. AMIA. Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2007;61–5.

Brownson RC, Kemner AL, Brennan LK. Applying a mixed-methods evaluation to healthy kids, Healthy communities. J Public Health Manag Pract. 2015;21:S16–26.

Butchart A, Peden M, Matzopoulos R, Phillips R, Burrows S, Bhagwandin N, et al. The South African National Non-natural Mortality Surveillance System–rationale, pilot results and evaluation. S Afr Med J. 2001;91:408–17.

CAS   PubMed   Google Scholar  

Carr ECJ, Babione JN, Marshall D. Translating research into practice through user-centered design: an application for osteoarthritis healthcare planning. Int J Med Inf. 2017;104:31–7.

Cesario M, Jervis M, Luz S, Masoodian M, Rogers B. Time-based geographical mapping of communicable diseases BT – 2012 16th International Conference on Information Visualisation, IV. 2012, July 11, 2012 - July 13, 2012. Graduate Programme on Health Promotion, University of Franca, BrazilDepartment of Computer Science, University of Waikato, New ZealandSchool of Computer Science and Statistics, Trinity College Dublin, Ireland: Institute of Electrical and Electronics Engineers Inc.; 2012. pp. 118–23.

Chirambo GB, Muula AS, Thompson M, Hardy VE, Heavin C, Connor YO et al. End-user perspectives of two mHealth decision support tools: Electronic Community Case Management in Northern Malawi. Int J Med Inf. 2021;145.

Cinnamon J, Rinner C, Cusimano MD, Marshall S, Bakele T, Hernandez T, et al. Evaluating web-based static, animated and interactive maps for injury prevention. Geospat Health. 2009;4:3–16.

Cleland B, Wallace J, Bond R, Muuraiskangas S, Pajula J, Epelde G et al. July. Usability Evaluation of a Co-created Big Data Analytics Platform for Health Policy-Making. Ulster University, School of Computing, United Kingdom BT - Human Interface and the Management of Information. HIMI 2019, held as part of the 21st HCI International Conference, HCII 2019, 26–31 2019: Springer International Publishing; 2019. pp. 194–207. https://doi.org/10.1007/978-3-030-22660-2_13

Concannon D, Herbst K, Manley E. Developing a data dashboard framework for population health surveillance: Widening access to clinical trial findings. JMIR Form Res [Internet]. 2019;3. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85096868744&doi=10.2196%2F11342&partnerID=40&md5=0596 d9738c51cdb49ebf2e5f2a2d010f.

Cox R, Sanchez J, Revie CW. Multi-criteria Decision Analysis Tools for prioritising emerging or re-emerging infectious diseases Associated with Climate Change in Canada. PLoS ONE. 2013;8:e68338.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Cretikos M, Telfer B, McAnulty J. Evaluation of the system of surveillance for enteric disease outbreaks, New South Wales, Australia, 2000 to 2005. N S W Public Health Bull. 2008;19:8–14.

Cruden G, Frerichs L, Powell BJ, Lanier P, Brown CH, Lich Gracelyn. ORCID: http://orcid.org/0000-0002-1737-5867 KHAI-O http://orcid.org/Cruden. Developing a multi-criteria decision analysis tool to support the adoption of evidence-based child maltreatment prevention programs. Aarons Baumann, Belton, Bonabeau, Buffett, Cruden, Glasgow, Marsh, Muhlbacher, Palinkas, Sheldrick, Stoltzfus, Thokala, Tversky A, editor. Prev Sci. 2020;No-Specified.

Cunningham PM, Cunningham M, van Greunen D, Veldsman A, Kanjo C, Kweyu E et al. Oct. Implications of baseline study findings from rural and deep rural clinics in Ethiopia, Kenya, Malawi and South Africa for the co-design of mHealth4Afrika. Stockholm University, Dept. of Computer and Systems Sciences, 13 Docklands Innovation Park, 128 East Wall Road, Ireland BT – 2016 IEEE Global Humanitarian Technology Conference (GHTC), 13–16 2016: IEEE; 2016. pp. 666–74.

Dalle Carbonare S, Cerra C, Bellazzi R. Development and representation of health indicators with thematic maps. Stud Health Technol Inform [Internet]. 2012;180:220–4. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emed13&NEWS=N&AN=366370560

Fico G, Hernanzez L, Cancela J, Arredondo MT, Dagliati A, Sacchi L, et al. What do healthcare professionals need to turn risk models for type 2 diabetes into usable computerized clinical decision support systems? Lessons learned from the MOSAIC project. BMC Med Inf Decis Mak. 2019;19:163.

Finch CF, Goode N, Salmon PM, Shaw Caroline F. ORCID: http://orcid.org/0000-0003-1711-1930 LAO-F. End-user experiences with two incident and injury reporting systems designed for led outdoor activities - challenges for implementation of future data systems. Inj Epidemiol. 2019;6:39.

Fisher RP, Myers BA. Free and simple GIS as appropriate for health mapping in a low resource setting: a case study in eastern Indonesia. Int J Health Geogr. 2011;10:15.

Foldy SL, Barthell E, Silva J, Biedrzycki P, Howe D, Erme M, et al. SARS Surveillance project–internet-enabled multiregion surveillance for rapidly emerging disease. MMWR Morb Mortal Wkly Rep. 2004;53:215–20.

Foldy SL, Biedrzycki PA, Baker BK, Swain GR, Howe DS, Gieryn D, et al. The public health dashboard: a surveillance model for bioterrorism preparedness. J Public Health Manag Pract. 2004;10:234–40.

Gagnon M-P, Lampron A, Buyl R, Implementation. and adoption of an electronic information system for vaccine inventory management BT – 49th Annual Hawaii International Conference on System Sciences, HICSS 2016, January 5, 2016 - January 8, 2016. Universite Laval, CanadaCHU de Quebec Research Center, CanadaVrije Universiteit, Brussel, Belgium: IEEE Computer Society; 2016. pp. 3172–8.

Gerrits RG, Klazinga NS, van den Berg MJ, Kringos Reinie G. ORCID: http://orcid.org/0000-0001-8030-2882 DSAO-G. figure interpretation Assessment Tool-Health (FIAT-health) 2.0: from a scoring instrument to a critical appraisal tool. BMC Med Res Methodol. 2019;19:160.

Gesteland PH, Livnat Y, Galli N, Samore MH, Gundlapalli AV. The EpiCanvas infectious disease weather map: an interactive visual exploration of temporal and spatial correlations. J Am Med Inform Assoc. 2012;19:954–9.

Gourevitch MN, Athens JK, Levine SE, Kleiman N, Thorpe LE. City-Level Measures of Health, Health determinants, and equity to Foster Population Health Improvement: the City Health Dashboard. Am J Public Health. 2019;109:585–92.

Grossberndt S, Bartonova A, Van Den Hazel P. Application of social media in the environment and health professional community. Environ Health. 2012;11:S16.

Guthrie JL, Marchand-Austin A, Lam K, Whelan M, Lee B, Alexander DC, et al. Technology and tuberculosis control: the OUT-TB web experience. J Am Med Inform Assoc. 2017;24:e136–42.

Hawver JE, Rocheleau B, Wyllie TT, Waller KN, Bailey R, Smith MC. Mental health resources and the criminal justice system: Assessment and plan for integration in Charlottesville, Virginia - Phase III expansion BT – 2009 IEEE Systems and Information Engineering Design Symposium, SIEDS ’09, April 24, 2009 - April 24, 2. University of Virginia, Charlottesville, VA 22904, United StatesDepartment of Systems and Information Engineering, University of Virginia, Charlottesville, VA 22904, United States: IEEE Computer Society; 2009. pp. 197–202. https://doi.org/10.1109/SIEDS.2009.5166183

Ha YP, Tesfalul MA, Littman-Quinn R, Antwi C, Green RS, Mapila TO et al. Becerra Chapman Creswell Daemen Denkinger Fox Ha Karlesky Kayiwa Khan Labrique Lewis Puryear Timimi B Vella editor 2016 Evaluation of a mobile health approach to Tuberculosis contact tracing in Botswana. J Health Commun 21 1115–21.

Heidebrecht CL, Wells GA, Tugwell PS, Engel ME. Tuberculosis surveillance in Cape Town, South Africa: an evaluation. Int J Tuberculosis Lung Disease. 2011;15:912–8.

Hundley VA, Avan BI, Ahmed H, Graham WJ, Group BKW. Clean birth kits to improve birth practices: development and testing of a country level decision support tool. BMC Pregnancy Childbirth. 2012;12:158.

Hu PJ, Zeng D, Chen H, Larson C, Chang W, Tseng C et al. System for infectious disease information sharing and analysis: design and evaluation. IEEE Trans Inf Technol Biomed [Internet]. 2007;11:483–92. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emed10&NEWS=N&AN=47306402

Hussain-Alkhateeb L, Olliaro P, Benitez D, Kroeger A, Sewe MO, Rocklov J, et al. Early warning and response system (EWARS) for dengue outbreaks: recent advancements towards widespread applications in critical settings. PLoS ONE. 2018;13:e0196811.

Ilesanmi OS, Fawole O, Nguku P, Oladimeji A, Nwenyi O. Evaluation of Ebola virus disease surveillance system in Tonkolili District, Sierra Leone. Pan Afr Med J. 2019;32:2.

Jaroensutasinee M, Jaroensutasinee K, Jinpon P. Integrated information visualization to support decision-making in order to strengthen communities: design and usability evaluation. Inf Health Soc Care. 2017;42:335–48.

Joshi A, de Araujo Novaes M, Machiavelli J, Iyengar S, Vogler R, Johnson C, et al. A human centered GeoVisualization framework to facilitate visual exploration of telehealth data: a case study. Technol Health Care. 2012;20:457–71.

Joyce K. To me it’s just another tool to help understand the evidence: public health decision-makers’ perceptions of the value of geographical information systems (GIS). Health Place. 2009;15:801–10.

Kadam R, White W, Banks N, Katz Z, Kelly-Cirino C, Dittrich S. Target product profile for a mobile app to read rapid diagnostic tests to strengthen infectious disease surveillance. PLoS ONE. 2020;15:e0228311.

Karavite DJ, Miller MW, Ramos MJ, Rettig SL, Ross RK, Xiao R, et al. User testing an information foraging Tool for Ambulatory Surgical site infection surveillance. Appl Clin Inf. 2018;9:791–802.

Kealey CM, Brunetti GM, Valaitis RK, Akhtar-Danesh N, Thomas H. A severe Acute Respiratory Syndrome extranet: supporting local communication and information dissemination. BMC Med Inf Decis Mak. 2005;5:17.

Keeling JW, Turner AM, Allen EE, Rowe SA, Merrill JA, Liddy ED et al. Development and evaluation of a prototype search engine to meet public health information needs. AMIA. Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2011;2011:693–700.

Kelly GC. A spatial decision support system for guiding focal indoor residual spraying interventions in a malaria elimination zone. Geospat Health. 2011;6:21–31.

Laberge M, Shachak A. Developing a tool to assess the quality of socio-demographic data in community health centres. Appl Clin Inf. 2013;4:1–11.

Liaw S-T, Ansari S, Zhou R, Gao J. A digital health profile & maturity assessment toolkit: cocreation and testing in the Pacific Islands. J Am Med Inf Assoc. 2021;28:494–503.

Livnat Y, Rhyne T-M, Samore MH. Epinome: a visual-analytics workbench for epidemiology data. IEEE Comput Graph Appl. 2012;32:89–95.

Loschen W, Coberly J, Sniegoski C, Holtry R, Sikes M, Happel Lewis S. Event communication in a regional disease surveillance system. AMIA. Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2007;483–7.

Loschen W, Seagraves R, Holtry R, Hung L, Lombardo J, Lewis S. INFOSHARE - an Information Sharing Tool for Public Health during the 2009 presidential inauguration and H1N1 outbreak. Online J Public Health Inf. 2010;2.

Maclachlan JC, Jerrett M, Abernathy T, Sears M, Bunch MJ. Mapping health on the internet: a new tool for environmental justice and public health research. Health Place. 2007;13:72–86.

Mansoor H, Gerych W, Alajaji A, Buquicchio L, Chandrasekaran K, Agu E, PLEADES: Population level observation of smartphone sensed symptoms for in-the-wild data using clustering BT – 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics, Theory, Applications VISIGRAPP et al. 2021, February 8, 2021. Department of Data Science, Worcester Polytechnic Institute, Worcester; MA, United StatesDepartment of Computer Science, Worcester Polytechnic Institute, Worcester; MA, United States: SciTePress; 2021. pp. 64–75.

Margevicius KJ, Generous N, Abeyta E, Castro L, Daughton A, Del Valle SY, et al. The biosurveillance analytics resource directory (BARD): facilitating the use of epidemiological models for infectious disease surveillance. PLoS ONE. 2016;11:e0146600.

Bögl M, Aigner W, Filzmoser P, Lammarsch T, Miksch S, Rind A. Visual analytics for Model Selection in Time Series Analysis. IEEE Trans Vis Comput Graph. 2013;19:2237–46.

Merkord CL, Liu Y, Mihretie A, Gebrehiwot T, Awoke W, Bayabil E et al. Integrating malaria surveillance with climate data for outbreak detection and forecasting: the EPIDEMIA system. Malar J. 2017;16.

Millery M, Ramos W, Lien C, Kukafka R, Aguirre AN. Design of a Community-Engaged Health Informatics Platform with an Architecture of Participation. AMIA Annu Symp Proc. 2015;2015:905–14.

Mukhtar Q, Mehta P, Brody ER, Camponeschi J, Friedrichs M, Kemple AM, et al. Development of the diabetes indicators and data sources Internet Tool (DIDIT). Prev Chronic Dis. 2006;3:A20.

Nagykaldi Z, Mold JW, Bradley KK, Bos JE. Bridging the gap between public and private healthcare: influenza-like illness surveillance in a practice-based research network. J Public Health Manag Pract. 2006;12:356–64.

Ngo TD, Canavati SE, Dinh HS, Ngo TD, Tran DT, Martin NJ et al. Addressing operational challenges of combatting malaria in a remote forest area of Vietnam using spatial decision support system approaches. Geospat Health. 2019;14.

Nguyen LH, LeFevre AE, Jennings L, Agarwal S, Labrique AB, Mehl G, et al. Perceptions of data processes in mobile-based versus paper-based health information systems for maternal, newborn and child health: a qualitative study in Andhra Pradesh, India. BMJ Innov. 2015;1:167–73.

Olingson C, Hallberg N, Timpka T, Lindqvist K. Requirements engineering for inter-organizational health information systems with functions for spatial analyses: modeling a WHO safe community applying use case maps. Methods Inf Med. 2002;41:299–304.

Chen M, Trefethen A, Bañares-Alcántara R, Jirotka M, Coecke B, Ertl T, et al. From data analysis and visualization to causality discovery. Comput (Long Beach Calif). 2011;44:84–7.

CAS   Google Scholar  

Park S, Gil-Garcia JR. Understanding transparency and accountability in open government ecosystems: The case of health data visualizations in a state government BT – 18th Annual International Conference on Digital Government Research, DG.O. 2017, June 7, 2017 - June 9, 2017. University at Albany, State University of New York, 187 Wolf Road, Suite 301, Albany; NY; 12205, United States: Association for Computing Machinery; 2017. pp. 39–47.

Patel R, Ahn E, Baldacchino T, Mullavey T, Kim J, Liu N et al. A Mobile App and Dashboard for Early Detection of Infectious Disease Outbreaks: Development Study. JMIR Public Health Surveill [Internet]. 2021;7:e14837. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emexb&NEWS=N&AN=634537554

Pelat C, Bonmarin I, Ruello M, Fouillet A, Caserio-Schonemann C, Levy-Bruhl D et al. Improving regional influenza surveillance through a combination of automated outbreak detection methods: the 2015/16 season in France. Eurosurveillance. 2017;22.

Rachmani E, Lin M-C, Hsu CY, Jumanto J, Iqbal U, Shidik GF et al. The implementation of an integrated e-leprosy framework in a leprosy control program at primary health care centers in Indonesia. Int J Med Inf. 2020;140.

Rajamani S, Bieringer A, Sowunmi S, Muscoplat M. Stakeholder Use and Feedback on Vaccination History and Clinical Decision Support for Immunizations Offered by Public Health. AMIA Annu Symp Proc. 2017;2017:1450–7.

Rajvanshi H, Telasey V, Soni C, Jain D, Surve M, Gangamwar V, et al. A comprehensive mobile application tool for disease surveillance, workforce management and supply chain management for Malaria Elimination Demonstration Project. Malar J. 2021;20:91.

Reeder B, Hills RA, Turner AM, Demiris G. Participatory design of an integrated information system design to support public health nurses and nurse managers. Public Health Nurs. 2014;31:183–92.

Reeder B, Turner AM. Scenario-based design: a method for connecting information system design with public health operations and emergency management. J Biomed Inf. 2011;44:978–88.

Rezaei-hachesu P, Samad-Soltani T, Yaghoubi S, GhaziSaeedi M, Mirnia K, Masoumi-Asl H, et al. The design and evaluation of an antimicrobial resistance surveillance system for neonatal intensive care units in Iran. Int J Med Inf. 2018;115:24–34.

Roberton T, Litvin K, Self A, Stegmuller AR. All things to all people: trade-offs in pursuit of an ideal modeling tool for maternal and child health. BMC Public Health. 2017;17:785.

Robinson AC, MacEachren AM, Roth RE. Designing a web-based learning portal for geographic visualization and analysis in public health. Health Inf J. 2011;17:191–208.

Sahar L, Faler G, Hristov E, Hughes S, Lee L, Westnedge C, et al. Development of the Inventory Management and Tracking System (IMATS) to Track the Availability of Public Health Department Medical Countermeasures during Public Health Emergencies. Online J Public Health Inf. 2015;7:e212.

Semwanga AR, Nakubulwa S, Adam T. Applying a system dynamics modelling approach to explore policy options for improving neonatal health in Uganda. Health Res Policy Syst. 2016;14:35.

Sopan A, Noh ASI, Karol S, Rosenfeld P, Lee G, Shneiderman B. Community Health Map: a geospatial and multivariate data visualization tool for public health datasets. Gov Inf Q. 2012;29:223–34.

Sorge J, Klassen B, Higgins R, Tooley L, Ablona A, Jollimore J, et al. Democratizing Access to Community-based survey findings through dynamic data visualizations. Arch Sex Behav. 2021;50:119–28.

Stegmuller AR, Self A, Litvin K, Roberton T. How is the lives Saved Tool (LiST) used in the global health community? Results of a mixed-methods LiST user study. BMC Public Health. 2017;17:773.

Struik LL, Abramowicz A, Riley B, Oliffe JL, Bottorff JL, Stockton Laura L. ORCID: http://orcid.org/0000-0001-7175-7308, Bottorff, Joan L.; ORCID: http://orcid.org/0000-0001-9724-5351 LDAI-O http://orcid.org/Struik. Evaluating a tool to support the integration of gender in programs to promote men’s health. Affleck Bottorff, Bunn, Damschroder, Dworkin, Gahagan, Galdas, Gelb, Heidari, Heilman, Kiselica, Langley, Lefkowich, Mackenzie, McIntosh, Ogrodniczuk, Oliffe, Oliffe, Oliffe, Paretz, Pirkis, Robertson, Robertson, Robertson, Robertson, Rycroft-Malone, San B, editor. Am J Mens Health. 2019;13.

Studnicki J, Fisher JW, Eichelberger C, Bridger C, Angelon-Gaetz K, Nelson D. NC CATCH: Advancing Public Health Analytics. Online J Public Health Inf. 2010;2.

Sutcliffe A, De Bruijn O, Thew S, Buchan I, Jarvis P, McNaught J, et al. Developing visualization-based decision support tools for epidemiology. Inf Vis. 2014;13:3–17.

Svoronos T, Jillson IA, Nsabimana MM. TRACnet’s absorption into the Rwandan HIV/AIDS response. Int J Healthc Technol Manage. 2008;9:430–45.

Swoboda CM, Griesenbrock T, Gureddygari HR, Aldrich A, Fareed N, Jonnalagadda P. Visualizing Opportunity Index Data Using a Dashboard Application: A Tool to Communicate Infant Mortality-Based Area Deprivation Index Information. Appl Clin Inform [Internet]. 2020;11:515–27. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emexa&NEWS=N&AN=632546907

Thew SL, Sutcliffe A, de Bruijn O, McNaught J, Procter R, Jarvis P, et al. Supporting creativity and appreciation of uncertainty in exploring geo-coded public health data. Methods Inf Med. 2011;50:158–65.

Article   CAS   PubMed   Google Scholar  

Tilahun B, Kauppinen T, Kesler C, Fritz F. Design and development of a linked open data-based health information representation and visualization system: potentials and preliminary evaluation. JMIR Med Inf. 2014;2:e31.

Tobgay T, Samdrup P, Jamtsho T, Mannion K, Thriemer K, Ortega L, et al. Performance and user acceptance of the Bhutan febrile and malaria information system: report from a pilot study. Malar J. 2016;15:52.

Tom-Aba D, Toikkanen SE, Glockner S, Denecke K, Silenou BC, Krause G et al. User Evaluation Indicates High Quality of the Surveillance Outbreak Response Management and Analysis System (SORMAS) After Field Deployment in Nigeria in 2015 and 2018. Stud Health Technol Inform [Internet]. 2018;253:233–7. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emed19&NEWS=N&AN=624845037

Travers D, Crouch J, Haas SW, Mostafa J, Waller AE, Schwartz TA et al. Implementation of Emergency Medical Text Classifier for syndromic surveillance. AMIA. Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2013;2013:1365–74.

Turner AM, Reeder B, Wallace JC. A resource management tool for public health continuity of operations during disasters. Disaster Med Public Health Prep. 2013;7:146–52.

Velicko I, Riera-Montes M. The Chlamydia surveillance system in Sweden delivers relevant and accurate data: results from the system evaluation, 1997–2008. Eurosurveillance. 2011;16.

Wang E-H, Zhou L, Watzlaf V, Abernathy PA, Web-Based Social Network Analysis System for Guiding Behavioral Interventions Delivery in Medically Underserved Communities BT – 2017 International Conference on Computational Science and, Intelligence C. CSCI 2017, December 14, 2017 - Dec. FPFHC, FOCUS Pittsburgh, Pittsburgh; PA, United StatesDepartment of HIM, University of Pittsburgh, Pittsburgh; PA, United States: Institute of Electrical and Electronics Engineers Inc.; 2017. pp. 840–5.

Wang KH, Marenco L, Madera JE, Aminawung JA, Wang EA, Cheung K-H. Using a community-engaged health informatics approach to develop a web analytics research platform for sharing data with community stakeholders. AMIA Annu Symp Proc. 2017;2017:1715–23.

Wongsapai M, Suebnukarn S, Rajchagool S, Kijsanayotin B. Health-oriented electronic oral health record for health surveillance. Stud Health Technol Inform [Internet]. 2013;192:763–7. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emed14&NEWS=N&AN=603583684

Wu E, Davis A, Villani J, Fareed N, Huerta TR, Harris DR, et al. Community dashboards to support data-informed decision-making in the HEALing communities study. Drug Alcohol Depend. 2020;217:108331.

Cole BL, Yancey AK, McCarthy WJ. A graphical, computer-based decision-support tool to help decision makers evaluate policy options relating to physical activity. Am J Prev Med. 2010;39:273–9.

Yang J-A, Block J, Jankowska MM, Baer RJ, Chambers CD, Jelliffe-Pawlowski LL, et al. An Online Geographic Data Visualization Tool to relate Preterm births to Environmental factors. Prev Chronic Dis. 2019;16:E102.

PubMed   PubMed Central   Google Scholar  

Kenealy T, et al. A whole of system approach to compare options for CVD interventions in Counties Manukau. Aust N Z J Public Health. 2012;36:263–8.

Geyer NR, Kessler FC, Lengerich EJ, United States. LionVu 2.0 usability assessment for Pennsylvania,. ISPRS Int J Geoinf [Internet]. 2020;9. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85094968202&doi=10.3390%2Fijgi9110619&partnerID=40&md5=bbbc5fe3300e82f8555c82bfcd150bec

Jinpon P, Jaroensutasinee M, Jaroensutasinee K. Integrated information visualization to support decision making for health promotion in Chonburi, Thailand. Walailak J Sci Technol. 2019;16:551–60.

Karlsson D, Ekberg J, Spreco A, Eriksson H, Timpka T. Visualization of infectious disease outbreaks in routine practice. Stud Health Technol Inf. 2013. pp. 697–701.

McGladrey M, Noar S, Crosby R, Young A, Webb E. Creating project CREATE: lessons learned and best practices for developing web-based resources for public health practitioners. Am J Health Educ. 2012;43:341–8.

Mittelstädt S, Hao MC, Dayal U, Hsu M-C, Terdiman J, Keim DA. Advanced visual analytics interfaces for adverse drug event detection. Proceedings of the Workshop on Advanced Visual Interfaces AVI. 2014. pp. 237–44.

Osborn AW, Peters LR. Vaccination Data when the outbreak happens: a qualitative evaluation of Oregon’s Rapid Response Tool. Disaster Med Public Health Prep. 2019;13:682–5.

Parks AL, Walker B, Pettey W, Benuzillo J, Gesteland P, Grant J et al. Interactive agent based modeling of public health decision-making. AMIA. Annual Symposium proceedings / AMIA Symposium AMIA Symposium [Internet]. 2009;2009:504–8. https://www.scopus.com/inward/record.uri?eid=2-s2.0-79953795548&partnerID=40&md5=253273e41c383b4ef358fa638b39a708

Pontin D, Thomas M, Jones G, O’Kane J, Wilson L, Dale F, et al. Developing a family resilience assessment tool for health visiting/public health nursing practice using virtual commissioning, high-fidelity simulation and focus groups. J Child Health Care. 2020;24:195–206.

Schooley B, Feldman S, Tipper B. A Unified Framework for Human Centered Design of a Substance Use, Abuse, and Recovery Support System. Advances in Intelligent Systems and Computing. Health Information Technology Program, College of Engineering and Computing, University of South Carolina, 550 Assembly Street, Columbia, SC 29208, United States; 2020. pp. 175–82.

Sinclair S, Hagen NA, Chambers C, Manns B, Simon A, Browman GP. Accounting for reasonableness: exploring the personal internal framework affecting decisions about cancer drug funding. Health Policy. 2008;86:381–90.

Thew S, Sutcliffe A, Procter R, de Bruijn O, McNaught J, Venters CC, et al. Requirements engineering for E-science: experiences in epidemiology. IEEE Softw. 2009;26:80–7.

Timpka T, Morin M, Jenvald J, Eriksson H, Gursky E. Towards a simulation environment for modeling of local influenza outbreaks. AMIA. Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2005;729–33.

Zakkar M, Sedig K. Interactive visualization of public health indicators to support policymaking: an exploratory study. Online J Public Health Inf. 2017;9.

Aburto NJennings, Rogers L, De-Regil LM, aria, Kuruchittham V, Rob G, Arif R et al. An evaluation of a global vitamin and mineral nutrition surveillance system. Arch Latinoam Nutr [Internet]. 2013;63:105–13. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emed14&NEWS=N&AN=604450009

Rajamani S, Chakoian H, Bieringer A, Lintelmann A, Sanders J, Ostadkar R, et al. Development and implementation of an interoperability tool across state public health agency’s disease surveillance and immunization information systems. JAMIA Open. 2023;6:ooad055.

Akre S, Liu PY, Friedman JR, Bui AAT, International. COVID-19 mortality forecast visualization: Covidcompare.io. JAMIA Open [Internet]. 2021;4. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85140846026&doi=10.1093%2fjamiaopen%2fooab113&partnerID=40&md5=9fc1e6ff64aafb515889c20b67f06e44

Alminana A, Bayeh A, Girma D, Kanagat N, Oot L, Prosser W et al. Early lessons from Ethiopia in establishing a data triangulation process to analyze immunization program and Supply data for decision making. Glob Health Sci Pract. 2022;10.

Alpers R, Kuhne L, Truong H-P, Zeeb H, Westphal M, Jackle S. Evaluation of the EsteR Toolkit for COVID-19 decision support: sensitivity analysis and usability study. JMIR Form Res. 2023;7:e44549.

Altura KAP, Madjalis HEC, Sungahid MDG, Serrano EA, Rodriguez RL. Development of a Web-Portal Health Information System for Barangay. Fujisawa, Japan: Institute of Electrical and Electronics Engineers Inc.; 2023. pp. 544–50.

Ansari B, Martin EG. Integrating human-centered design in public health data dashboards: lessons from the development of a data dashboard of sexually transmitted infections in New York State. J Am Med Inf Assoc. 2023.

Backonja U, Park S, Kurre A, Yudelman H, Heindel S, Schultz M et al. Supporting rural public health practice to address local-level social determinants of health across Northwest states: Development of an interactive visualization dashboard. J Biomed Inform [Internet]. 2022;129. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85127144845&doi=10.1016%2fj.jbi.2022.104051&partnerID=40&md5=07f1b9bec7b466f6d0ff6bbe52f77c93

Burgess H, Gutierrez-Mock L, Moghadassi Y-XH, Lesh M, Krueger N et al. E,. Implementing a digital system for contact tracing and case investigation during COVID-19 pandemic in San Francisco: A qualitative study. JAMIA Open [Internet]. 2021;4:ooab093-. https://academic.oup.com/jamiaopen

Delcher C, Horne N, McDonnell C, Bae J, Surratt H. Overdose Detection Mapping Application Program expansion evaluation—A qualitative study. Criminol Public Policy [Internet]. 2023;22:491–516. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85159118390&doi=10.1111%2f1745-9133.12628&partnerID=40&md5=7c72655dfe4bc641d9dbf531a8794f8d

Doyle M, Ainsworth P, Boul S, Lee D. Evaluation of a system for Real-Time Surveillance of Suicide in England. Crisis. 2023;44:341–8.

Agbemafle EE, Kubio C, Bandoh D, Odikro MA, Azagba CK, Issahaku RG, et al. Evaluation of the malaria surveillance system - Adaklu District, Volta Region, Ghana, 2019. Public Health Pract (Oxf). 2023;6:100414.

Filos D, Lekka I, Kilintzis V, Stefanopoulos L, Karavidopoulou Y, Maramis C, et al. Exploring associations between Children’s obesogenic behaviors and the local Environment using Big Data: development and evaluation of the obesity Prevention Dashboard. JMIR Mhealth Uhealth. 2021;9:e26290.

Guimarães EADA, Morato YC, Carvalho DBF, Oliveira VCD, Pivatti VMS, Cavalcante RB et al. Evaluation of the Usability of the Immunization Information System in Brazil: A Mixed-Method Study. Telemedicine and e-Health [Internet]. 2021;27:551–60. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85105752793&doi=10.1089%2ftmj.2020.0077&partnerID=40&md5=83f9a036c1d51f62785e1a03f461baf5

Hintermeier M, Gold AW, Erdmann S, Perplies C, Bozorgmehr K, Biddle L. From Research into Practice: Converting Epidemiological Data into Relevant Information for Planning of Regional Health Services for Refugees in Germany. Int J Environ Res Public Health [Internet]. 2022;19:8049. https://www.mdpi.com/1660-4601/19/13/8049/pdf?version=1656585138

Hollis S, Stolow J, Rosenthal M, Morreale SE, Moses L. Go.Data as a digital tool for case investigation and contact tracing in the context of COVID-19: a mixed-methods study. BMC Public Health. 2023;23:1717.

Said SIM, Aminuddin R, Abidin NAZ, Nasir SDNM, Ibrahim AZM. Visualizing COVID-19 Vaccination Rate and Vaccination Centre in Malaysia using DBSCAN Clustering model. 2022 IEEE International Power and Renewable Energy Conference (IPRECON). 2022. pp. 1–6.

Ising A, Waller A, Frerichs L. Evaluation of an Emergency Department Visit Data Mental Health Dashboard. J Public Health Manag Pract. 2023.

Jonnalagadda P, Swoboda C, Singh P, Gureddygari H, Scarborough S, Dunn I, et al. Developing dashboards to address children’s Health disparities in Ohio. Appl Clin Inf. 2022;13:100–12.

Lardi EA, Khader SAKSAAAMAASA. The Rotavirus Surveillance System in Yemen: evaluation study. JMIR Public Health Surveill. 2021;7:e27625.

Lechner C, Rumpler M, Dorley MC, Li Y, Ingram A, Fryman H. Developing an Online Dashboard to Visualize Performance Data—Tennessee Newborn Screening Experience. Int J Neonatal Screen [Internet]. 2022;8. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85138468949&doi=10.3390%2fijns8030049&partnerID=40&md5=e3824922762dd77b6b5004fe604b73c7

Li Y, Albarrak AS. An informatics-driven intelligent system to improve healthcare accessibility for vulnerable populations. J Biomed Inform [Internet]. 2022;134. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85137174603&doi=10.1016%2fj.jbi.2022.104196&partnerID=40&md5=d6ea5cf40f3b559a4ca1233bb856ae72

Mansoor H, Gerych W, Alajaji A, Buquicchio L, Chandrasekaran K, Agu E, et al. PLEADES: Population level observation of smartphone sensed symptoms for in-the-wild data using clustering. Virtual, Online: SciTe; 2021. pp. 64–75.

Meidani Z, Moravveji A, Gohari S, Ghaffarian H, Zare S, Vaseghi F, et al. Development and Testing requirements for an Integrated Maternal and Child Health Information System in Iran: A Design thinking Case Study. Methods Inf Med. 2022;61:e64–72.

O’Flaherty M, Lloyd-Williams F, Capewell S, Boland A, Maden M, Collins B et al. Modelling tool to support decision-making in the NHS Health Check programme: Workshops, systematic review and co-production with users. Health Technol Assess (Rockv) [Internet]. 2021;25:1–233. https://www.journalslibrary.nihr.ac.uk/hta/hta25350/#/abstract

O’Leary MC, Mayorga KHL, Hicklin ME, Davis K, Brenner MM et al. AT,. Engaging stakeholders in the use of an interactive simulation tool to support decision-making about the implementation of colorectal cancer screening interventions. Cancer Causes and Control [Internet]. 2023; https://www.springer.com/journal/10552

Praharaj S, Solis P, Wentz EA. Deploying geospatial visualization dashboards to combat the socioeconomic impacts of COVID-19. Environ Plan B Urban Anal City Sci [Internet]. 2023;50:1262–79. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85148503498&doi=10.1177%2f23998083221142863&partnerID=40&md5=f1c96f0c706aa744fb4a99f1150a1787

Rivers Z, Roth JA, Wright W, Rim SH, Richardson LC, Thomas CC, et al. Translating an Economic Analysis into a Tool for Public Health Resource Allocation in Cancer Survivorship. MDM Policy Pract. 2023;8:23814683231153376.

Swift B, Imohe A, Perez CH, Mwirigi L. An in-depth review of the UNICEF NutriDash platform, lessons learnt and future perspectives: a mixed-methods study. BMJ Open [Internet]. 2023;13:e062684-. http://bmjopen.bmj.com/content/early/by/section

Tchoualeu DD, Elmousaad HE, Osadebe LU, Adegoke OJ, Nnadi C, Haladu SA, et al. Use of a district health information system 2 routine immunization dashboard for immunization program monitoring and decision making, Kano State, Nigeria. Pan Afr Med J. 2021;40:2.

Tegegne HA, Bogaardt C, Collineau L, Cazeau G, Lailler R, Reinhardt J et al. OH-EpiCap: A semi-quantitative tool for the evaluation of One Health epidemiological surveillance capacities and capabilities. medRxiv [Internet]. 2023; https://www.medrxiv.org/

Tennant R, Tetui M, Grindrod K, Burns CM. Multi-disciplinary Design and implementation of a Mass Vaccination Clinic Mobile Application to support decision-making. IEEE J Transl Eng Health Med. 2023;11:60–9.

Vázquez Noguera JL, Ho Shin H, Sauer Ayala C, Grillo S, Pérez-Estigarribia P, Torales R et al. Epymodel: A User-Friendly Web Application for Visualising COVID-19 Projections for Paraguay Including Under-Reporting and Vaccination. Communications in Computer and Information Science [Internet]. 2023. pp. 58–72. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85169037861&doi=10.1007%2f978-3-031-36357-3_5&partnerID=40&md5=5a6a2c39c2f97a9230239d0b54f42116

Wells J, Grant R, Chang J, Kayyali R. Evaluating the usability and acceptability of a geographical information system (GIS) prototype to visualise socio-economic and public health data. BMC Public Health. 2021;21:2151.

Zheng S, Edwards JR, Dudeck MA, Patel PR, Wattenmaker L, Mirza M, et al. Building an Interactive Geospatial Visualization Application for National Health Care-Associated Infection Surveillance: Development Study. JMIR Public Health Surveill. 2021;7:e23528.

Yang C, Zhang Z, Fan Z, Jiang R, Chen Q, Song X, et al. EpiMob: interactive visual analytics of Citywide Human mobility restrictions for Epidemic Control. IEEE Trans Vis Comput Graph. 2023;29:3586–601.

Shimpi N, Glurich I, Hegde H, Steinmetz A, Kuester R, Crespin M et al. DentaSeal: a school-based dental sealant efficiency assessment tool to support statewide monitoring and reporting: a field report. Technol Health Care. 2023.

Rabiee F. Focus-group interview and data analysis. Proceedings of the nutrition society. 2004;63:655–60.

Lu S, Christie GA, Nguyen TT, Freeman JD, Hsu EB. Applications of artificial intelligence and machine learning in disasters and public health emergencies. Disaster Med Public Health Prep. 2022;16:1674–81.

Equator Network. Enhancing the quality and transparency of health research. 2016. https://www.equator-network.org/

Download references

Acknowledgements

Not applicable.

This work was funded by the XSeed 2020–2021 Interdivisional Research Funding Program and the Data Sciences Institute at the University of Toronto. BD is supported by a Canada Research Chair in Human Factors and Transportation. LCR is supported by a Canada Research Chair in Population Health Analytics and the Stephen Family Chair in Community Health from Trillium Health Partners. The funders played no role in study design, data collection, analysis and interpretation of data, or the writing of this manuscript.

Author information

Authors and affiliations.

Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, Canada

Holland M. Vasquez & Birsen Donmez

Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

Emilie Pianarosa, Renee Sirbu, Lori M. Diemert, Vinyas Harish & Laura C. Rosella

Gerstein Science Information Centre, University of Toronto, Toronto, Ontario, Canada

Heather Cunningham

Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada

Vinyas Harish & Laura C. Rosella

Institute for Better Health, Trillium Health Partners, Mississauga, Ontario, Canada

Laura C. Rosella

You can also search for this author in PubMed   Google Scholar

Contributions

LCR, BD were responsible for study design. HC, HV, LMD developed the search strategy. HV, RS, EP, LMD conducted the search, screening, and data extraction. HV, EP, LMD, VH, BD, LCR analyzed and interpreted the data. HV and VH drafted the original manuscript. All authors contributed to and approved of the manuscript.

Corresponding author

Correspondence to Laura C. Rosella .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Vasquez, H.M., Pianarosa, E., Sirbu, R. et al. Human factors methods in the design of digital decision support systems for population health: a scoping review. BMC Public Health 24 , 2458 (2024). https://doi.org/10.1186/s12889-024-19968-8

Download citation

Received : 03 April 2024

Accepted : 02 September 2024

Published : 10 September 2024

DOI : https://doi.org/10.1186/s12889-024-19968-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Human factors engineering
• Public health
• Decision-making tool
• User-centered design
• Literature review

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

A Critical Analysis of Decision Support Systems Research Revisited: The Rise of Design Science

Cite this chapter.

• David Arnott 4 &
• Graham Pervan 5  

1494 Accesses

7 Citations

4 Altmetric

Decision support systems (DSS) is the area of the information systems (IS) discipline that is focused on supporting and improving managerial decision making. In 2005 the Journal of Information Technology ( JIT ) published our paper that critically analyzed DSS research from 1990 to 2003 (Arnott and Pervan, 2005). That paper used bibliometric content analysis as its method and analyzed 1020 articles in 14 journals. The analysis illuminated a vibrant and important part of IS research. Personal DSS and group support systems (GSS) dominated DSS research and two-thirds of DSS research was empirical, a higher proportion than general IS research. Interpretive DSS research was growing from a low base while design-science research (DSR) and laboratory experiments were major research categories. Unfortunately, it was found that DSS research to 2003 was relatively poorly founded on judgment and decision-making theory and faced what was described as ‘a crisis of relevance.’

The contribution of the authors was equal

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Similar content being viewed by others

A critical analysis of decision support systems research

Computerized Decision Support Case Study Research: Concepts and Suggestions

Group Support Systems: Past, Present, and Future

Alavi, M. and Carlson, P. (1992). A Review of MIS Research and Disciplinary Development, Journal of Management Information Systems 8 (4): 45–62.

Article   Google Scholar  

Anthony, R.N. (1965). Planning and Control Systems: A framework for analysis , Cambridge, MA: Harvard University Press.

Google Scholar  

Arnott, D. and Pervan, G. (2005). A Critical Analysis of Decision Support Systems Research, Journal of Information Technology 20 (2): 67–87.

Arnott, D. and Pervan, G. (2008). Eight Key Issues for the Decision Support Systems Discipline, Decision Support Systems 44 (3): 657–672.

Arnott, D. and Pervan, G. (2010). How Relevant is Fieldwork to DSS Design-Science Research? in A. Respicio, F. Adam, G. Phillips-Wren, C Teixeira and J. Telhada (eds.) Bridging the Socio-Technical Gap in Decision Support Systems: Challenges for the next decade , Amsterdam: IOS Press, pp. 199–210.

Arnott, D. and Pervan, G. (2012). Design Science in Decision Support Systems Research: An assessment using the Hevner, March, Park, and Ram guidelines, Journal of the Association for Information Systems 13 (11): 923–949.

Arnott, D., Pervan, G. and Dodson, G. (2005). Who Pays for Decision Support Systems Research? Review, Directions and Issues, Communications of the Association for Information Systems 16 : 356–380.

Baskerville, R. (2008). What Design Science is Not, European Journal of Information Systems 17 (5): 441–443.

Benbasat, I. and Barki, H. (2007). Quo Vadis, TAM? Journal of the Association for Information Systems 8 (4): 211–218.

Benbasat, I. and Nault, B. (1990). An Evaluation of Empirical Research in Managerial Support Systems, Decision Support Systems 6 (3): 203–226.

Bragge, J., Korhonen, P., Wallenius, H. and Wallenius, J. (2012). Scholarly Communities of Research in Multiple Criteria Decision Making: A bibliometric research profiling study, International Journal of Information Technology and Decision Making 11(2): 401–426.

Briggs, R.O., de Vreede, G.-J. and Nunamaker, Jr. J.F. (2003). Collaboration engineering with ThinkLets to Pursue Sustained Success with Group Support Systems, Journal of Management Information Systems 19(4): 31–64.

Brown, B., Court, D. and Willmott, P. (2013). Mobilizing your C-suite for Big Data Analytics, McKinsey Quarterly , November.

Cavaye, A.L.M. (1996). Case Study Research: A multi-faceted research approach for IS, Information Systems Journal 6 (3): 227–242.

Chen, H., Chiang, R. and Storey, V. (2012). Business Intelligence and Analytics: From big data to big impact, MIS Quarterly 36 (4): 1165–1188.

Chen, W.S. and Hirschheim, R. (2004). A Paradigmatic and Methodological Examination of Information Systems Research from 1991 to 2001, Information Systems Journal 14 (3): 197–235.

Chiang, R.H.L., Goes, P. and Stohr, E.A. (2012). Business Intelligence and Analytics Education, and Program Development: A unique opportunity for the information systems discipline, ACM Transactions on Management Information Systems 3 (3): 1–13.

Clark, Jr T.D., Jones, M.C. and Armstrong, C.P. (2007). The Dynamic Structure of Management Support Systems: Theory development, research focus, and direction, MIS Quarterly 31 (3): 579–615.

Davenport, T.H. (2006). Competing on Analytics, Harvard Business Review 84 (1): 98–107.

Davenport, T.H. and Harris, J.G. (2007). Competing on Analytics: The new science of winning , Boston, MA: Harvard Business School Press.

Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly 13 (3): 319–339.

Delen, D. and Crossland, M.D. (2008). Seeding the Survey and Analysis of Research Literature with Text Mining, Expert Systems with Applications 34 (3): 1707–1720.

DeLone, W.H. and McLean, E.R. (2003). The DeLone and McLean Model of Information Systems Success: A ten-year update, Journal of Management Information Systems 19 (4): 9–30.

Dennis, A.R., Carte, T.A. and Kelly, G.G. (2003). Breaking the Rules: Success and failure in groupware-supported business process reengineering, Decision Support Systems 36 (1): 31–47.

Dennis, A.R., Tyran, C.K., Vogel, D.R. and Nunamaker, Jr J.F. (1997). Group Support Systems for Strategic Planning, Journal of Management Information Systems 14 (1): 155–184.

Galliers, R.D. (1991). Choosing Appropriate Information Systems Research Approaches: A revised taxonomy, in H.-E. Nissen, H.K. Klein and R. Hirschheim (eds.) Information Systems Research: Contemporary approaches and emergent traditions , Amsterdam: North-Holland, pp. 327–345.

Galliers, R.D. and Meadows, M. (2003). A Discipline Divided: Globalization and parochialism in information systems research, Communications of the Association for Information Systems 11: 108–117.

Gartner (2007). Creating Enterprise Leverage: The 2007 CIO agenda (Gartner EXP CIO Report) , Stamford, CT: Gartner Inc.

Gartner (2008). Gartner says worldwide business intelligence platform market grew 13 percent in 2007, Gartner Newsroom , [WWW document] http://www.gartner.com /it/page.jsp?id=700410 (accessed 7 November 2012).

Gartner (2009). Gartner says worldwide business intelligence, analytics and performance management grew 22 percent in 2008, Gartner Newsroom , [WWW document] http://www.gartner.com /it/page.jsp?id=1017812 (accessed 7 November 2012).

Gartner (2010). Gartner says worldwide business intelligence, analytics and performance management software market grew 4 percent in 2009, Gartner Newsroom , [WWW document] http://www.gartner.com /it/page.jsp?id=1357514 (accessed 7 November 2012).

Gartner (2011). Gartner says worldwide business intelligence, analytics and performance management software market surpassed the $10 billion mark in 2010, Gartner Newsroom , [WWW document] http://www.gartner.com /it/page.jsp?id=1642714 (accessed 7 November 2012).

Gartner (2012a). Amplifying the Enterprise: The 2012 Gartner CIO agenda report , Stamford, CT: Gartner Inc.

Gartner (2012b). Gartner says worldwide business intelligence, analytics and performance management software market surpassed the $12 billion mark in 2011, Gartner Newsroom , [WWW document] http://www.gartner.com /it/page.jsp?id=1971516 (accessed 7 November 2012).

Gartner (2013a). Gartner predicts business intelligence and analytics will remain top focus for CIOs through 2017, Gartner Newsroom , [WWW document] http://www.gartner.com /newsroom/id/2637615 (accessed 18 February 2014).

Gartner (2013b). Gartner says worldwide business intelligence software revenue to grow 7 percent in 2013, Gartner Newsroom , [WWW document] http://www.gartner.com /newsroom/id/2340216 (accessed 18 February 2014).

Germonprez, M., Hovorka, D. and Gal, U. (2011). Secondary Design: A case of behavioral design science research, Journal of the Association for Information Systems 12 (10): 662–683.

Goodhue, D.L. and Thompson, R.L. (1995). Task-Technology Fit and Individual Performance, MIS Quarterly 19 (2): 213–236.

Gorry, G.A. and Scott Morton, M.S. (1971). A Framework for Management Information Systems, Sloan Management Review 13 (1): 1–22.

Gottschalk, P. (2000). Predictors of IT Support For Knowledge Management in the Professions: An empirical study of law firms in Norway, Journal of Information Technology 15 (1): 69–78.

Gregor, S. and Jones, D. (2007). The Anatomy of a Design Theory, Journal of the Association for Information Systems 8 (5): 312–335.

Guo, Z. and Sheffield, J. (2008). A Paradigmatic and Methodological Examination of Knowledge Management Research: 2000 to 2004, Decision Support Systems 44 (3): 673–688.

Hevner, A.R. (2007). The Three Cycle View of Design Science Research, Scandinavian Journal of Information Systems 19 (2): 87–92.

Hevner, A.R., March, S.T., Park, J. and Ram, S. (2004). Design Science in Information Systems Research, MIS Quarterly 28 (1): 75–106.

Hirschheim, R. (2007). Introduction to the Special Issue on ‘Quo Vadis TAM — Issues and reflections on technology acceptance research’, Journal of the Association for Information Systems 8 (4): 203–205.

Hosack, B., Hall, D., Paradice, D. and Courtney, J.F. (2012). A Look Toward the Future: Decision support is alive and well, Journal of the Association for Information Systems 13 (5): 315–340.

Hwang, H.-G., Ku, C.-Y., Yen, D. and Cheng, C.C. (2004). Critical Factors Influencing the Adoption of Data Warehouse Technology: A study of the banking industry in Taiwan, Decision Support Systems 37 (1): 1–21.

Indulska, M. and Recker, J.C. (2008). Design Science in IS Research: A literature analysis, in S. Gregor and S. Ho (eds.) Proceedings of the 4th Biennial ANU Workshop on Information Systems Foundations , Canberra, Australia: ANU.

Iivari, J. (2007). A Paradigmatic Analysis of Information Systems as a Design Science, Scandinavian Journal of Information Systems 19 (2): 39–64.

Jourdan, Z., Rainer, R.K. and Marshall, T.E. (2008). Business Intelligence: An analysis of the literature, Information Systems Management 25 (2): 121–131.

Kahneman, D. (2011). Thinking Fast and Slow , New York: Farrar, Straus and Giroux.

Kuechler, W. and Vaishnavi, V. (2012). A Framework for Theory Development in Design Science Research: Multiple perspectives, Journal of the Association for Information Systems 13 (6): 395–423.

Lee, A.S. (2010). Retrospect and Prospect: Information systems research in the last and next 25 years, Journal of Information Technology 25 (4): 336–348.

Leidner, D.E., Carlsson, S., Elam, J. and Corrales, M. (1999). Mexican and Swedish Managers’ Perceptions of the Impact of EIS on Organizational Intelligence, Decision Making, and Structure, Decision Sciences 30 (3): 633–658.

Lewin, K. (1943). Defining the ‘Field at a Given Time’, Psychological Review 50 (3): 292–310.

Lipshitz, R. and Bar-Ilan, O. (1996). How Problems are Solved: Reconsidering the phase theorem, Organizational Behavior and Human Decision Processes 65 (1): 48–60.

March, S. and Smith, G.F. (1995). Design and Natural Science Research on Information Technology, Decision Support Systems 15 (4): 251–266.

Marginson, D., King, M. and McAulay, L. (2000). Executives’ Use of Information Technology: Comparison of electronic mail and an accounting information system, Journal of Information Technology 15 (2): 149–164.

McAfee, A. and Brynjolfsson, E. (2012). Big Data: The management revolution, Harvard Business Review (October): 61–68.

Nandhakumar, J. (1996). Design for Success?: Critical success factors in executive information systems development, European Journal of Information Systems 5 (1): 62–72.

Neuman, W.L. (2000). Social Research Methods: Qualitative and quantitative approaches , 4th edn, Needham Heights, MA: Allyn and Bacon.

Osareh, F. (1996). Bibliometrics, Citation Analysis and Co-Citation Analysis: A review of literature I, Libri 46 (3): 149–158.

Power, D.J. (2012). A brief history of decision support systems (version 4.1). [www document] http://dssresources.com /history/dsshistory.html, accessed 3 July 2012.

Ross, J.W., Beath, C.M. and Quaadgras, A. (2013). You May Not Need Big Data After All, Harvard Business Review (December): 90–98.

Rouibah, K. and Ould-ali, S. (2002). PUZZLE: A concept and prototype for linking business intelligence to business strategy, Journal of Strategic Information Systems 11 (2): 133–152.

Salo, A. and Kakola, T.K. (2005). Groupware Support for Requirements Management in New Product Development, Journal of Organizational Computing and Electronic Commerce 15 (4): 253–284.

Shander, B. (2013). Does your company actually need data visualization? HBR blog network [www document] http://blogs.hbr.org (accessed 14 November 2013).

Shanks, G., Jagielska, I. and Jayaganesh, M. (2009). A Framework for Understanding Customer Relationship Management Systems Benefits, Communications of the Association for Information Systems 25 : 263–288.

Simon, H.A. (1960). The New Science of Management Decision , New York: Harper.

Book   Google Scholar  

Taylor, H., Dillon, S. and Van Wingen, M. (2010). Focus and Diversity in Information Systems Research: Meeting the dual demands of a healthy applied discipline, MIS Quarterly 34 (4): 647–667.

Tversky, A. and Kahneman, D. (1981). The Framing of Decisions and the Psychology of Choice, Science 211 (4487): 453–458.

Vaishnavi, V.K. and Kuechler, Jnr W. (2008). Design Science Research Methods and Patterns: Innovating information and communication technology , Boca Raton, FL: Auerbach Publications.

Venable, J., Pries-Heje, J. and Baskerville, R. (2012). A Comprehensive Framework for Evaluation in Design Science Research, in K. Peffers, M. Rothenberger and B Kuechler (eds.) Proceedings of the 7th Conference on Design Science in Information Systems (DESRIST’12) Berlin: Springer-Verlag, pp. 423–438.

Watson, H.J., Fuller, C. and Ariyachandra, T. (2004). Data Warehouse Governance: Best practices at blue cross and blue shield of North Carolina, Decision Support Systems 38 (3): 435–450.

Watson, H.J., Goodhue, D.L. and Wixom, B.H. (2002). The Benefits of Data Warehousing: Why some organizations realize exceptional payoffs, Information & Management 39 (6): 491–502.

Walls, J.G., Widmeyer, G.R. and El Sawy, O.A. (1992). Building an Information Systems Design Theory for Vigilant EIS, Information Systems Research 3 (1): 36–59.

Weber, R.P. (1990). Basic Content Analysis , 2nd edn, Newbury Park, CA: Sage Publications.

Webster, J. and Watson, R.T. (2002). Analyzing the Past to Prepare for the Future: Writing a literature review, MIS Quarterly 26 (2): xiii–xxiii.

Winter, R. (2008). Design Science Research in Europe, European Journal of Information Systems 17 (5): 470–475.

Yin, R.K. (1994). Case Study Research: Design and methods , 2nd edn, Newbury Park, CA: Sage Publications.

Zikmund, W.G., Babin, B.J., Carr, J.C. and Griffin, M. (2010). Business Research Methods , 8th edn, Mason, OH: South-Western, Cengage Learning.

Download references

Author information

Authors and affiliations.

Monash University, Australia

David Arnott

School of Information Systems, Curtin Business School, Curtin University, Australia

Graham Pervan

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

London School of Economics and Political Science, London, UK

Leslie P. Willcocks

Oxford University, Oxford, UK

Chris Sauer

University of Missouri-St Louis, St Louis, Missouri, USA

Mary C. Lacity

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Association for Information Technology Trust

About this chapter

Arnott, D., Pervan, G. (2016). A Critical Analysis of Decision Support Systems Research Revisited: The Rise of Design Science. In: Willcocks, L.P., Sauer, C., Lacity, M.C. (eds) Enacting Research Methods in Information Systems. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-29272-4_3

Download citation

DOI : https://doi.org/10.1007/978-3-319-29272-4_3

Publisher Name : Palgrave Macmillan, Cham

Print ISBN : 978-3-319-29271-7

Online ISBN : 978-3-319-29272-4

eBook Packages : Business and Management Business and Management (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Towards effective clinical decision support systems: A systematic review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Algoritmi Research Center, University of Minho, Braga, Portugal

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – review & editing

Roles Conceptualization, Project administration, Supervision, Validation, Writing – review & editing

• Francini Hak, 
• Tiago Guimarães, 
• Manuel Santos

• Published: August 15, 2022
• https://doi.org/10.1371/journal.pone.0272846
• Peer Review
• Reader Comments

Clinical Decision Support Systems (CDSS) are used to assist the decision-making process in the healthcare field. Developing an effective CDSS is an arduous task that can take advantage from prior assessment of the most promising theories, techniques and methods used at the present time.

To identify the features of Clinical Decision Support Systems and provide an analysis of their effectiveness. Thus, two research questions were formulated: RQ1—What are the most common trend characteristics in a CDSS? RQ2—What is the maturity level of the CDSS based on the decision-making theory proposed by Simon?

AIS e-library, Decision Support Systems journal, Nature, PlosOne and PubMed were selected as information sources to conduct this systematic literature review. Studies from 2000 to 2020 were chosen covering search terms in CDSS, selected according to defined eligibility criteria. The data were extracted and managed in a worksheet, based on the defined criteria. PRISMA statements were used to report the systematic review.

The outcomes showed that rule-based module was the most used approach regarding knowledge management and representation. The most common technological feature adopted by the CDSS were the recommendations and suggestions. 19,23% of studies adopt the type of system as a web-based application, and 51,92% are standalone CDSS. Temporal evolution was also possible to visualize. This study contributed to the development of a Maturity Staging Model, where it was possible to verify that most CDSS do not exceed level 2 of maturity.

The trend characteristics addressed in the revised CDSS were identified, compared to the four predefined groups. A maturity stage model was developed based on Simon’s decision-making theory, allowing to assess the level of maturity of the most common features of the CDSS. With the application of the model, it was noticed that the phases of choice and implementation are underrepresented. This constitutes the main gap in the development of an effective CDSS.

Citation: Hak F, Guimarães T, Santos M (2022) Towards effective clinical decision support systems: A systematic review. PLoS ONE 17(8): e0272846. https://doi.org/10.1371/journal.pone.0272846

Editor: Gabriele Oliva, University Campus Bio-Medico of Rome, ITALY

Received: December 6, 2020; Accepted: July 27, 2022; Published: August 15, 2022

Copyright: © 2022 Hak et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting information files.

Funding: This study was supported by FCT – Fundação para a Ciência e Tecnologia, within the scope of the project DSAIPA/DS/0084/2018, FH work was supported by the grant 2021.06230.BD. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

In the day-to-day routine of healthcare units, the domain professionals interact with hundreds of patients. Naturally, it is of great importance that all this interaction is transformed into records, whether clinical or administrative, which in turn are transformed into data, information, and, hopefully, knowledge. Despite technological developments, these records are still kept in physical format, which creates a delay in the work performed when compared to digital versions [ 1 ].

Friedman [ 2 ] boasted a theorem in which it shows that an individual or group working with the contribution of a technological resource of information, has a better performance compared to a job without such assistance. For this functional theorem to be verified, the information resources must be valid and reliable and the user must know how to handle it properly.

The electronic representation of clinical records are embodied by Electronic Health Records (EHR), which aim, in particular, to eliminate the use of paper in the healthcare field. Notwithstanding, the representation of electronic health data tends to be more than just a substitute for paper [ 1 ]. HIMSS [ 3 ] shows that a healthcare information system must have the ability to generate a complete view of the clinical record at a meeting with a patient, including decision evidence-based support, quality management and results reporting, apart from the support of other activities related to direct or indirect care by means of a shared area. Thereby, it is evident that for a healthcare information system to be effective, it is crucial the existence of an integrated component of support in decision-making processes.

The concept of Decision Support (DS) is equivalent to the activity that assists a given user following a certain purpose. Thus, a Decision Support System (DSS) turns this activity into a system-based format in an efficient and reliable way, composing models and techniques through a knowledge representation infrastructure [ 4 ]. In this sense, a DSS portrays a system designed to support a professional in obtaining knowledge and making decisions in the specific area, therefore, diminishing uncertainties during the decision-making process [ 5 ].

In the healthcare field, the decision-making support activity is designed as Clinical Decision Support (CDS). The representation of this activity in a computerized-based format is translated into a Clinical Decision Support System (CDSS), where it provides all the information and desired knowledge that facilitate the daily tasks of healthcare providers and guarantee an improvement in the quality of services.

As a clinical knowledge-based system, the treatment of information and the process of knowledge extraction are considerable aspects for a Clinical Decision Support System to achieve its objectives. Some features are mandatory in a CDSS, but not all are known and some may be missed and contribute to system failures [ 6 ].

According to Simon, the decision-making process is the heart of any organization and it influences all processes integrated into it [ 7 ]. The decision-making process can be outlined using various models and theorems. Within the scope of this study, the Bounded Rationality theory was adopted as a reference to Herbert Simon’s work [ 8 ]. It is stated that people act in a rational way according to the knowledge and perception that they get. In an initial phase, this model approached three stages: Intelligence, Design and Choice. Later on, a new stage was added by Sprague and Carlson [ 9 ], resulting in the implementation phase. Thus, the four phases resulted by Simon’s work [ 8 ], redound to model decision making process of an Intelligent Decision Support System.

Previous systematic review studies (SRs) have addressed the use of a Clinical Decision Support System to assess its use and effectiveness [ 10 – 12 ], but applied to a specific workflow. Instead, this systematic review aims to analyse a set of articles that address a CDSS in order to identify the trends of the characteristics for its conception. In addition, we also identify other aspects such as the purpose of care and the recipient of the intervention. Furthermore, in order to enrich the outcomes obtained from the review, we trace a general trend of a CDSS, relating it with the four phases proposed by Simon.

This article is structured in four sections. First, an introduction to the topic of study is presented. Secondly, the landscape of the methods used to make the Systematic Review. The third section presents the results of the Systematic Review. The fourth section is dedicated to discussing the results obtained. At last, in section five, conclusions are drawn while leaving open doors for future work.

Decision-making theory

Decision-making is one of the main processes dealt within an organization [ 13 ]. This process can be approached using different models and theories. In the present study, the bounded rationality theory proposed by Simon will be focused.

Herbert Simon [ 8 ] was an economist who carried out research involving several areas, such as psychology, computing and management. One of Simon’s great contributions to scientific research was the theory of bounded rationality, where Simon redefines human rationality arguing that people act rationally according to the knowledge and perception they obtain. To formalize his theory, Simon established four phases that define the decision-making process, as shown in Fig 1 .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0272846.g001

In Intelligence phase, problems are identified and the purpose of action involving the decision is determined. In the Design phase, possible solutions are designed and alternatives are created to solve the problem. Consequently, in the Choice phase, the alternative that most satisfies the purpose of action will be chosen. Years later, Turban [ 14 ] was responsible for reformulating Simon’s theory by adding a fourth phase in order to assess the implementation process. Thus, the Implementation phase results from the joining of the three previous phases put into practice. It should be noted that the decision-making is a sequential and iterative process, which requires the completion of all phases to reach the final phase and an ongoing review of the same.

• Intelligence—Process of formalization of the problem and definition of decision conditions. The decisions to be made with the help of the proposed system and the benefits it will bring are still unclear and not understood. In this phase, reality is examined in order to model the existing information for decomposing the problem and determining its properties. The intelligence phase ends with a problem statement.
• Design—At this stage it is necessary to have the problem defined to search for alternatives or available options, where possible courses of action are analysed and validated. Problem-situation models are built to explore alternative solutions. For this, it is necessary to have well-defined and declared selection criteria as a result for the evaluation of potential solutions that will be identified (includes techniques, technologies, format, integration, functionalities, etc). Several possible outcomes can be considered for each alternative, each with a certain probability of occurrence. In cases where decisions are made with uncertainty, there are multiple outcomes for each alternative.
• Choice—In the choice phase, all alternatives are researched, evaluated, and one is defined as the recommended solution. In pursuit of goals, a course of action that is good enough is selected. Computer models or critical success factors can be used as techniques to evaluate the recommended solution. Normative models can use either the analytical model or an exhaustive and complete enumeration model. The solution is tested and once the proposed solution appears viable, a decision can be made.
• Implementation—This phase is the validation of all previous phases. If the adopted solution seems good enough, it can be implemented. Successful implementation results in solving the original problem. In case of failure, it must be returned to the previous phases.

The systematic review was conducted based on Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statements [ 15 ], followed by a checklist and a flow diagram.

Information sources

Five online data sources related to research in health information systems were selected. First, a search was performed on Scimago [ 16 ] comprising two areas of study: information systems and multidisciplinary. From this, journals with Q1 quartile in the last 5 years and with an impact factor greater than 150 were analysed. Journals that had the potential to have articles more related to the purpose of this study and that were already known by the authors were selected: Decision Support Systems, Nature Research Journal and Plos One. To complement the study, two public repositories were chosen, such as: the PubMed library (medline) and the Association of Information Systems (AIS) e-library. The Table 1 shows information about the selected data source types.

Source: Scimago Journal and Country Rank via www.scimagojr.com , accessed on May 13, 2022.

https://doi.org/10.1371/journal.pone.0272846.t001

Eligibility criteria and search strategy

To select the studies for the development of the systematic review, eligibility criteria were defined: (i) published studies in article format; (ii) open access or free full text; (iii) written in English; (iii) articles addressing a decision support systems in healthcare; (iv) computerized or electronic decision support systems. To meet these criteria, search filters were applied. The selected articles must address practical cases of a specific clinical decision support system, therefore, articles of the literature review type or that did not address a particular CDS system were excluded.

The search strategy was performed considering an ordering from the most recent publications to the oldest ones, restricting to a range from 2000/01/01 to 2020/12/31. To identify studies of the desired scope, the query terms were: ((everything:“Clinical Decision Support System”) OR everything:“Clinical Decision Support”) OR everything:“Decision Support System in Healthcare”. The title, abstract and keywords were the primary strategy for analyzing each scientific publication. When these parameters were not sufficient, a complete reading of each article was made, in order to guarantee the eligibility of inclusion criteria.

Data extraction and management

The screening process was carried out by two authors (FH and TG) independently, by reading the titles, abstracts and keywords of the publications of the search results. When the criteria were met, the full text was read. In moments of disagreement or doubt, the third reviewer (MS) took a stand and contributed to the discussion.

Through the review of eligible articles, data extraction was performed by two reviewers (FH and TG) and focused on four characteristics of a CDSS, defined as: knowledge management and representation technique; technological resources integrated into the system; the type of system; and system integration. These four categories were considered by the authors as the most relevant for the purpose of this study. In addition, the following information was extracted: clinical setting, study design, recipient of intervention, and purpose of care. Data extraction was performed through the textual analysis of the articles and the search for keywords. For better management and analysis, the extracted data were recorded on an appropriate sheet (see S1 File ).

To answer the second research question of this study, it was identified in which phase of the decision-making theory the CDSS of the reviewed articles was found. Thus, based on the literature on decision-making theory [ 17 , 18 ] and the definitions of each phase, the authors considered a set of criteria for the assignment of each phase, as shown in Table 2 . For a system to be classified in a certain phase, it must fulfill at least one of the conditions indicated.

https://doi.org/10.1371/journal.pone.0272846.t002

Quality assessment and data synthesis

All the reviewed studies portray a Clinical Decision Support System (CDSS) designed for a specific purpose. The key points for conducting this systematic review were to identify the trends of the CDSS developed regarding their type of system, integration, representation and formulation of knowledge, and their technological features.

The reporting quality was made following the PRISMA checklist. Publications considered eligible for the analysis were assessed for the methodological quality, meeting the inclusion criteria, risk of bias, assumptions and simplifications, and clarified evidence-based results. The reviewers did not apply any methodological assessment tool.

After defining the five data sources, the reviewers selected a set of publications within the scope of Clinical Decision Support Systems, through the search strategy and the inclusion criteria previously defined. The outcomes presented through tables and narrative summary characterize the different Clinical Decision Support Systems presented in the selected studies, in order to trace a global trend. The data from the matching results were pooled out and evaluated, based on the total value of studies and the probability of occurrence. The reviewers noted the presence of clinical and statistical heterogeneity among the studies.

Study selection

The study selection was carried out following the PRISMA statement [ 15 ], using a checklist (see S2 File ) and a flow diagram represented in Fig 2 ( S3 File ). 768 records were identified by searching five databases considered most influential for the purpose. After removing duplicate records, 690 records were screened to read the title, abstract and keywords. Lastly, the articles that did not meet the selection and search criteria were excluded, comprising: non-article format; unpublished articles; not in english; out of period; not open access. The selected articles were read in full to assess their eligibility. Finally, 52 articles met the research questions and were elected eligible and appropriate to carry out the systematic review, embracing a specific Clinical Decision Support System and not literature review articles.

https://doi.org/10.1371/journal.pone.0272846.g002

Study characteristics

The studies that met the inclusion criteria were characterized according to a set of attributes. The period covered between the studies was from 2000 to 2020, with 50% from 2020, giving priority to the most recent ones.

The studies were searched in the five previously selected data sources, with eighteen studies from PubMed (34,62%), eleven studies from Decision Support Systems journal (21,15%), ten studies from PlosOne (19,23%), and five studies from Nature (9,62%). We note a diversity in the countries that developed the studies, covering four different continents. However, the majority stand out to the United States of America, representing nineteen studies (34,62%).

Studies realized in any healthcare setting were the most frequent ones (28,85%), followed by hospital setting (21,15%) and hospital academic centers (21,15%). The most common recipients of the intervention are, directly, general practitioners (32,69%). Studies reveal that CDSS also has an action on patients, but mostly, indirectly.

In general, the study design of the reviews articles, demonstrates the evaluation of the effectiveness of the CDSS regarding its cost, implementation and usability. In addition, the review showed that the CDSS addressed have an associated clinical intervention, with the most common purpose of care being a specific workflow (32,69%), followed by a specific patient disease (15,09%).

The information extracted from the studies that were considered relevant for the desired purpose generated a set of outcomes, based on the characterization of a Clinical Decision Support System (CDSS).

Knowledge management . The representation and formulation of the acquired knowledge is one of the most important steps in the development of a CDSS. In order to identify the most used techniques for knowledge management, we have identified the approaches used in studies of the systematic review, as shown in Table 3 . Clinical Practice Guidelines (38,46%), rule-based module (40,39%) and algorithmic logic (38,46%), were the most used approaches to knowledge management in the designing of a CDSS. These techniques were used individually or in combination with others. The formulation of a knowledge base was also identified in thirteen studies (25%), in combination with other techniques. Inference engines were identified in seven studies (13,46%), which were also considered in some studies, as reasoning engines. Three studies (5,77%) provided if/then statements, and eight studies (15,39%) applied methods based on variables. Terminology standards and clinical classification systems were present in five studies (9,62%), applying, in particular, the ICD-10 system and the HL7 communication protocol. Bayesian and neural networks were used in two different articles (3,85% each). Finally, three studies (5,77%) used data mining techniques to reproduce the desired knowledge. Two studies [ 19 , 20 ] were excluded from pooling, as the method used in the knowledge representation process was not identified.

https://doi.org/10.1371/journal.pone.0272846.t003

Technological features . Technological interventions represent the features that contribute to the system achieving its purpose. The most common technological feature approached in the CDSS is recommendation and suggestion feature, identified in twenty four studies (46,15%), as shown in Table 4 . Information management and monitoring are the second most common feature in the systems, covering eighteen studies (46,15%). The third most desired feature is alerts, notifications and reminders, covering fourteen studies (26,92%). Therefore, it follows the purpose of reducing errors as mentioned in eleven studies (21,15%). Eight articles (15,38%) design the CDSS for assessment purposes. The prediction feature is also used for eight studies for different purposes (15,38%). The automation and prioritization of processes is considered to be of great importance and is addressed by seven articles (13,46%). The triggering of events is addressed by four studies (7,69%). The standardization of the clinical process is desired by four studies (7,69%). Three articles (5,77%) highlighted the calculation and scoring methods as features of a CDSS. Finally, the cost and time reduction is seen as a main feature in two studies (3,84%).

https://doi.org/10.1371/journal.pone.0272846.t004

System integration . The results related to the integration of the Clinical Decision Support System (CDSS), showed that 51,92% of the articles (twenty-seven) use a standalone system, as shown in Table 5 . The remaining studies show that their CDSS are integrated with another system, as well as using the information from these systems. Thus, eleven studies (21,15%) reveal that they integrate their CDSS with an Electronic Health Record (EHR) system; five studies (9,62%) integrate their system with a specific health information system; four studies (7,69%) integrate with a Computerized Provider Order Entry (CPOE) system; in this sequence, three studies (5,77%) integrate their CDSS with both EHR and CPOE; two studies differentiate their integration with an Electronic Medical Record (EMR) system.

https://doi.org/10.1371/journal.pone.0272846.t005

Type of system . The typology of a Clinical Decision Support System (CDSS) differs in several technological aspects, considering both the presentation of the user interface and the technique it is based on. The type of systems identified in the review of studies is represented in Table 6 . Most studies (19.23%) use a web-based application to present the system interface to the user. Nine studies (17,31%) describe their CDSS as a computerized tool, not specifying the type of software or interface used. Eight studies (15,38%) develop their decision support system in a software application. Regarding the logic used, five studies (9,62%) are based on machine learning techniques, and four studies (7,69%) are artificial intelligence-based. Three studies (5,77%) develop their CDSS in a mobile application and, still, four studies (7,69%) cover a mobile and web application. Two studies (3,85%) describe their CDSS as knowledge-based, and other two studies (3,85%) present the CDSS as data analytics. Other types of systems were considered in unique studies, such as cloud computing (1,92%), data-layer infraestructure (1,92%), image retrieval expert system (1,92%), user interface (1,92%), and web application integrated with data analytic (1,92%).

https://doi.org/10.1371/journal.pone.0272846.t006

The outcomes for each group were varied regarding the type of the CDSS, knowledge management, integration of systems and the technological features. Despite this diversity, we were able to notice a trend in results based on their frequent presence in the reviewed studies. As shown in Fig 3 , the most common knowledge representation and management techniques were the rule-based module, clinical practice guidelines and algorithmic logic. Regarding to the technological intervention or feature, the three top trends were recommendation and suggestion, information management and monitoring, and alerts, notifications and reminders. Standalone CDSS were the most common one, following integration with Electronic Healthcare Records (EHR) systems and specific healthcare information systems. The most frequent type of system were, respectively, web-based application, specific computerized tool, and software application. Despite the individual results, we also noticed a trend towards a combination of results. The most frequent sequence related to knowledge management was the mix of algorithmic logic with clinical practice guidelines. For technological features, the most common combination was the joining of recommendation and suggestion with alerts, notifications and reminders.

https://doi.org/10.1371/journal.pone.0272846.g003

Temporal evolution

Fig 4 demonstrates the temporal evolution of knowledge management and representation techniques used in studies. It is possible to verify that rule-based module and the algorithmic logic have been present since 2000 until today, mostly. As for clinical practice guidelines, on the other hand, proved to be a current issue due to their presence from 2013 to 2020. In general, the characteristics remain in existence over time, with a greater occurrence in 2020 due to the number of articles analysed that year. In contrast, the characteristics of the Bayesian network, if / then statements and neural networks were considered more recent, emerging from 2014.

https://doi.org/10.1371/journal.pone.0272846.g004

According to Fig 5 , we can see that the integration of standalone CDSS has been increasing in recent years. The integration with EHR systems, oscillated between 2014 to 2020, standing out also in the last year. Specific health information systems have been integrated into the CDSS from 2000 to 2020, on a regular basis.

https://doi.org/10.1371/journal.pone.0272846.g005

According to Fig 6 , the specific computerized tools have been present since the beginning of the time interval and were more present in the year 2020. The software application, machine learning-based, was already present. The mobile application remained constant. Web-based application was more prevalent in 2013.

https://doi.org/10.1371/journal.pone.0272846.g006

As showed in Fig 7 the alerts, notifications and reminders and the information management and monitoring feature were more present in 2020, but also present in some previous year. The recommendations and suggestions feature was present from 2003 to 2020. Standardization was present in 2000 and only returned in 2020. The calculation and scoring, cost and time reduction and prediction features were considered the most recent ones.

https://doi.org/10.1371/journal.pone.0272846.g007

Maturity staging model

For a CDSS to be considered effective, it has to reach the prestigious level of maturity. Thus, it is important to recognize the degree of maturity associated with the characterization of the CDSSs. In order to complement the study, a cross-checking was made with the trends characteristics facing the four phases of Simon’s decision-making theory [ 8 ]. Thus, four stages were classified aiming to represent the level of maturity of a system, based on Simon’s phases:

• Stage 4: Implementation + Choice + Design + Intelligence
• Stage 3: Choice + Design + Intelligence
• Stage 2: Design + Intelligence
• Stage 1: Intelligence

Table 7 represents the crossing of the Simon’s phases with the three most common characteristics of each group, in order to assess their stage of maturity. The columns referring to the maturity stages, correspond to the number of articles of the respective characteristic, given their presence in each Simon phase. It should be noted that the values are cumulative, that is, for a stage to be reached, it must contain the previous stage. All studies corresponding to each characteristic are present in the initial phase of Intelligence. Maturity is calculated using the weighted average (WAVG), which should vary between 1 and 4, corresponding to each stage.

https://doi.org/10.1371/journal.pone.0272846.t007

Main findings

This article aimed to develop a systematic review within the scope of Clinical Decision Support Systems (CDSS). The first research question (RQ1) was to identify the tendency of adopted approaches in CDSS development addressed in the reviewed studies, as shown in Fig 3 . To classify the revised studies, information about four predefined groups were extracted: knowledge management, technological intervention, system integration and type of system. Although there are other characteristics associated with a CDSS, the authors chose to extract specific information that go beyond the main objective of a CDSS, which is to assist the decision maker in the decision-making process. The results showed that the most used techniques for the knowledge representation in the systems are the creation of modules based on rules, clinical practice guidelines and logic algorithms. The technological features most present in CDSS are recommendation and suggestion, monitoring and information management, and alerts and reminders. Usually, CDSS are used standalone, following the integration with Electronic Health Systems or a specific information system. It was also identified that the most common types of systems are web applications, specific computer tools, and software applications.

In order to answer the second research question (RQ2), a preliminary classification during the review was carried out identifying which phase of the decision-making process model the CDSS was in. According to Simon’s bounded rationality theory [ 8 ], it was considered: Intelligence phase as responsible for analysis, exploration and description of the problem to be faced; the Design phase as the development and analysis of possible solutions to the problem at hand; in the Choice phase, the most appropriate solution is chosen to solve the problem; the Implementation seeks to apply the solution to the problem in question.

Regarding the maturity staging model proposed in Table 7 , when the stage 4 is achieved, the system has reached its prestigious level of maturity. However, this is not verified in the analysis. The weighted average (WAVG) of the characteristics showed very similar values, meaning that the CDSS of the reviewed studies predominates in Simon’s Intelligence and Design phases, equivalent to stage 2 of maturity. The recommendation and suggestion (1.55) and software application (1.53) characteristics have the lowest values nearly reaching the previous phase of Intelligence. The algorithmic logic characteristic stands out the most and shows that it is closer to achieving the upper stage of maturity (stage 3), meeting the Choice phase.

There are other systematic reviews that study the role of CDSS. However, to the best of our knowledge, existing studies analyse CDSS applied to a specific clinical context and do not evaluate its effectiveness as a whole (as [ 11 , 12 , 71 ]). The present systematic review selected a set of studies that approach the development of a CDSS characterizing them in order to trace a global trend, as well as assess their level of maturity. In this sense, it was possible to analyse the CDSS in its completeness, covering different approaches, techniques and purposes of use. The main contribution of this work was the proposed maturity staging model, that allowed to identify a gap between the state-of-art and the desirable stage of maturity in order to provide an effective CDSS. The results showed that the revised CDSS do not go beyond stage 2, meaning that CDSS are not succeeding in the healthcare arena due to the lack of maturity, i.e., as CDSS are not capable of supporting the choice of actions in clinical settings and are also not involved in the implementation of these actions.

This study allowed to raise a concern in the development of CDSS and to raise awareness that limited systems are being created and that may be far from being optimized. The projection of this study allowed us to portray a reality of many decision support systems in the health area, demonstrating the opposite of what it should be. When a system reaches the model implementation phase, it should be closer to reaching its optimization and not going back to the previous phases. This immaturity may be due to a lack of understanding of the real problem, difficulty in choosing the ideal solution, and failures in usability tests. An in-depth analysis must be done to discover the main constraints that prevent the inclusion of the decision-making phases in the CDSS.

Limitations and future research

The maturity staging model, combined with the phases of the decision-making process, serves to assess the effectiveness of the decision. When an implemented decision does not produce the desired results, there are likely to be several causes, such as incorrect problem definition, poor evaluation of alternatives, or inadequate implementation [ 18 ]. The proposed alternative may not be successful, which will lead to a new analysis of the problem, evaluation of alternatives and selection of a new alternative. Thus, evaluation is a key factor in the process because decision-making is a continuous and never-ending process.

Some limitations should be stressed. First, we were unable to perform a meta-analysis due to the variation in the type of studies analysed, as well as to use a quality assessment methodology to assess the quality of studies. The sample number of eligible studies was also limited. Another limitation was regarding the characterization of the CDSS, due to some studies not directly explaining the respective approach used. There may also be some inconsistency in the review carried out due to the personal opinions of each reviewer.

It is known that in the clinical area regulations and ethical issues (e.g. computer based control of infusion pumps) limit the application of the most advanced phases of the decision-making process (e.g. choice and implementation). This means that the highest value for the maturity stage might be lower than 4. Thus, some difficulties encountered in the adoption of CDSS may be related to the medical context, such as: the development of methods for supporting choice phase of decision making; interoperability among medical devices and health care information systems; technology acceptance; ethical and regulatory restrictions; poor involvement of professionals and organizations. A lot of work should be done in the future to mitigate those limitations, starting with: i) consider a larger sample of studies; ii) determine the accepted range of values for the maturity stage in particular clinical context of application; and iii) depict what it should be the focus of research in order to fulfil the actual maturity gap and develop effective CDSS.

Clinical Decision Support Systems (CDSS) represent the decision support activity and can be translated into a machine-readable computerized format. Most healthcare information systems are encouraged to or already include clinical decision support practices to organize clinical knowledge and improve the decision-making process [ 3 ]. In order to answer the research questions formulated in this work, a systematic review was developed to identify the techniques and approaches used in CDSS from 52 studies. The outcomes were varied and did not show a main pattern, leading to limited evidence. Nevertheless, it was identified the top three trends of the four defined groups: knowledge management, technological features, type of system, and system integration. In addition, this study allowed a more complete analysis to understand the state of maturity of the CDSS. A crossover of the identified trends was made to identify the maturity regarding the four phases proposed by Simon. The results demonstrated the lack of maturity of the CDSS presented by the reviewed studies.

Decision making is a process of making a choice between several alternatives to achieve a desired outcome. Among these possible causes, the most common and serious error is an inadequate definition of the problem. When the problem is not defined correctly, the alternative selected and implemented will not produce the desired result. That said, we believe that the biggest difficulties on the CDSS adoption are in the operational clinical environment, with the involvement of characteristics, processes, and stakeholders. Furthermore, based on the identified gap in this study, an agenda should be created for what an effective CDSS should be. There is an incentive for the scientific community to contribute to mitigate the limitations in order to achieve more effective Clinical Decision Support Systems.

Supporting information

S1 file. information extracted from the review..

https://doi.org/10.1371/journal.pone.0272846.s001

S2 File. PRISMA checklist.

https://doi.org/10.1371/journal.pone.0272846.s002

S3 File. PRISMA flow diagram of study selection.

https://doi.org/10.1371/journal.pone.0272846.s003

• View Article
• Google Scholar
• PubMed/NCBI
• 3. Handler T, Holtmeier R, Metzger J, Overhage M, Taylor S, Underwood C. HIMSS electronic health record definitional model. 2003;(January 2003):0–8.
• 5. Vasconcelos J, Rocha A, Gomes R. Sistemas de Informação de Apoio à Decisão Clínica: Estudo de um caso de uma Instituição de Saúde. 2004.
• 8. Simon HA. The New Science of Management Decision. USA: Prentice Hall PTR; 1960.
• 9. Sprague RH, Carlson ED. Building Effective Decision Support Systems. Prentice Hall Professional Technical Reference; 1982.
• 14. Turban E, Delen D, Sharda R. Business intelligence and analytics; 2014.
• 16. Scimago. Scimago Journal and Country Rank, year = 2022, url = https://www.scimagojr.com/ .
• 17. Kock E. A Broader Perspective to Decision Support Systems. 2003.
• 18. Lunenburg FC. THE DECISION MAKING PROCESS; 2010.
• 39. Berge GT, Granmo OC, Tveit TO. Combining unsupervised, supervised, and rule-based algorithms for text mining of electronic health records: A clinical decision support system for identifying and classifying allergies of concern for anesthesia during surgery. Information Systems Development: Advances in Methods, Tools and Management—Proceedings of the 26th International Conference on Information Systems Development, ISD 2017. 2017.
• 40. Wasylewicz ATM, Scheepers-Hoeks AMJW. Clinical decision support systems. Fundamentals of Clinical Data Science. 2018;(June):153–169.
• 41. Smit K, Koornneef P, Nysingh J, van Zwienen M, Berkhout M, Ravesteyn P. Transforming clinical practice guideline usage through the use of a clinical decision support system: An explorative study at the university medical centre Utrecht. 30th Bled eConference: Digital Transformation—From Connecting Things to Transforming our Lives, BLED 2017. 2017; p. 577–592.
• 62. Vargason T, Cohn J, Schultz O. The Open Repository @ Binghamton (The ORB) A Clinical Decision Support System for Malignant Pleural Effusion Analysis. 2016.

• Open access
• Published: 30 November 2022

Decision support system for handling control decisions and decision-maker related to supply chain

• Dimah Hussein Alahmadi 1 &
• Arwa A. Jamjoom   ORCID: orcid.org/0000-0002-5062-8942 1  

Journal of Big Data volume  9 , Article number:  114 ( 2022 ) Cite this article

5517 Accesses

4 Citations

1 Altmetric

Metrics details

The present study presents a knowledge-based DSS framework for supporting the decision-maker and handling control decisions related to supply chains.

Two binary variables were adopted for denoting at which time point a given task \(i\) starts and finishes. The scheduling issues are represented through the ontological model and appropriately interpreted using the Java environment. Regarding primary data, digital form of administration using google form platform took precedence over physical administration.

The findings might not be exact replication of the findings from previous studies that are limited to the influence of information and material flow on the performance of supply chain as there are concerns of what factors constitute information and material flow that need to be identified and considered. However, with the finding of associating factors of information and material flow may need to consider this in managing the flow and the supply chain. Associating factors such as information quality, information visibility, material cost, fund shortage and so on, play a role in information and material flow and the decisions made in an organization.

Conclusions

Factors associating with information and material flow need to be considered in decision making as well, as the cost in any of the elements affects the flow and this would impede supply chain performance of the organization.

Introduction

Information technology supports business activities in fast-flowing information and prompts changes of customer preference era [ 1 ]. The recent trend causes a paradigm shift in the production process, which consequently impacts the supply chain flows, with the risk of inefficiency and overexploitation from upstream to downstream. According to Carter and Rogers [ 2 ], sustainable supply chains focus on environmental, social, and economic aspects. In this regard, a decision support system (DSS) is emerged from the recent trend and is further competent in supporting the significant issues in the supply chains. Gorry and Scott Morton [ 3 ] have initially proposed DSS, and it has been broadly utilized in several realms. DSS aims to support decision-makers in aiding and enhancing their decisions about the process and the consequence of their business functions, which are in the representation of guidance for selecting the optimum sets of options in elevating the profit, customer satisfaction, and efficiency concerning the product.

Extant literature has focused on the use of DSS to support business-related procedures. For instance, these areas include the oil industry, fisheries, marine affairs, environmental sciences, transportation, tourism, and the health sector [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ]. This pattern indicates a very innovative application of DSS linked with numerous tools and classifications with other approaches for supporting the decision-making process [ 12 ]. DSS further experiences criticism when existing and potential consumers do not always take benefit of DSS in supporting their decision-making, either because of the DSS structure or knowledge and awareness. The consumer repeatedly and often utilizes the DSS when the usefulness and easiness are there. Therefore, DSS has to be customized based on problems and activities [ 13 ]. DSS has adopted data mining, business intelligence, statistical analysis, and data warehouse. The existing function of DSS is not merely restricted to the database system but also an expert system that aids decision-makers in solving the issues.

The efficacy of DSS is further reliant on the characteristics and construction, specifically in the supply chains, where it requires availability and information for transferring supply and demand between each stratum from downstream to upstream, enabling DSS to aid the decision-maker in the supply chains. There is a broad horizon for developing each decision support system in the supply chains since previous literature has grown significantly in the supply chain realm. Therefore, a knowledge-based DSS framework has been presented in this paper for supporting the decision-maker and handling control decisions related to supply chains.

Literature review

Organizations have always had the development of efficient supply chain systems as their focus. According to Attaran and Attaran [ 14 ], collaboration in the supply chain practice is becoming the crux of successful and long-lasting management of business operations. They posited that the inclination to produce quality goods and services is driving up the supply chain cost, affecting the supplier’s financial performance. Therefore, the significance of the supply chain to the production and eventual financial strength of the business makes it a top issue for the organization’s management [ 15 ]. For an organization to run a sustainable and successful supply chain, though, management has to ensure collaborative planning due to its effect on the movement of goods and services. As indicated by Cassivi [ 16 ], collaborative planning, forecasting, and replenishment (CPFR) is the bone of the business process that strengthens the management of the supply chain in an organization.

A successful supply chain management goes a long way in benefiting a company in a competition. Some of these benefits are evident in improving the cost of production, distribution, inventory, and the flexibility associated with the ‘production of goods and services, and improvement in market share [ 17 ] and customer relations [ 18 ]. Simatupang and Sridharan [ 19 ] posited that flexibility is vital in measuring the success of the supply chain in the organization, and it’s one of the benefits of having effective supply chain management. According to Hsu [ 20 ], the benefits of supply chain management can be either tangible or intangible. Tangible benefits are exemplified in the cost and reduction of inventory and its effective management. The time saved when the inventory is delivered quickly, while the intangible information accuracy, consistency, flow, service quality, and response time [ 21 ]. These, however, can only be realized when there is an integration of various functions and stakeholders in the organization.

According to Stevenson and Spring [ 22 ], correct and instantaneous information flow in the supply chain is equally as vital to the business as material flow. “An information-enriched supply chain would have a single customer entity connected to every scheduling process, showing order information flowing to all links, while for a non-enriched supply chain, the customer entity might connect only to the final scheduling link, leaving the remainder of the supply chain hidden from the customer” [ 23 ].

Sharing of information is essential due to its reflection of teamwork within the supply chain of an organization [ 24 ]. Simatupang and Sridharan [ 25 ] refer to information sharing as “the ability to see private data in a partner’s systems and monitor the progress of products as they pass through each process in the supply chain; the activity includes monitoring (data capturing), processing, and dissemination of customer data, end-to-end inventory status and locations, order status, costs-related data, and performance status”. Simatupang and Sridharan [ 19 ] believe that sharing information among supply chain partners allows short time order fulfillment within the order cycle times due to the shared information. Supply chain partners sharing of information generates supply chain information flow management that aids effective decision making among partners. Li et al. [ 24 ] stated that information flow is categorized based on the area of operation it is generated and needed. Therefore, the categories of information flow include production plan, inventory, order state, demand forecasting, and sales [ 26 ]. In their paper, Koh, Saad, and Arunachalam [ 27 ] stated that information flow is needed to support the management of activities like procurement of raw materials, schedule for production, and physical distribution system [ 28 ].

For information flow to be complete, two-way communication needs to be conducted, which involves contents, medium/channel, and systems [ 29 ]. The content is the actual information to be passed; the medium/channel is the pathway for the information, while the system allows the management of both information and the channel. According to Kembro and Selviaridis [ 30 ], information in supply chain information is sharable into three levels within the organization: strategic, tactical, and operational. At each of these levels, different types of information are communicated while various associated advantages and hindrances are encountered in sharing the information in the supply chain. Hsu et al. [ 31 ] also separated information shared in the organization into diverse levels, which are: strategic information (e.g., long-term objective, marketing, and customer information) and tactical (e.g., purchasing, operations scheduling, and logistics).

Information exchanged can also be classified into managerial and transactional information. Transactional refers to information required for an organization to conduct procurement or supplies. This class of information is related to payment order, receipt, inventory, transportation, and delivery. This information class relates to the technology needed for operations, quality, costs, and profitability.

From the perspective of supply chain management, information flow and its management are critical activities of the leaders in an organization. The flow of information in the supply chain is bi-directional. This is because other forms of activities, including materials and money flows, are activated by the movement of information to achieve the set objective. This means that material and money flows effective management is positively related to effective management of information flow. Therefore, the huge interest in these flows in literature and supply chain practice is understandable. Several supply chain practitioners have identified the significance of materials flow management as a critical strategic achievement issue [ 32 ].

Kuck et al. [ 33 ] suggested a data-driven simulation-based optimization approach for dynamic manufacturing system control. The study devised a method for rescheduling production in response to changing circumstances, taking into account aspects that may confuse, such as the simultaneous delivery of many orders. Ersoz et al. [ 34 ] attempted to bridge the scheduling theory and practice gap. They tailored their planning operations to the real-time information supplied by the process control and control systems. The dynamic structure of the production environment is quickly sensed in the offered procedure, and the schedule is modified in response to the changing conditions. In manufacturing, the traceability of the parts improved. Furthermore, needless waiting or downtimes were reduced.

Xiong et al. [ 35 ] suggested a simulation-based methodology for deciding dispatching rules in a dynamic scheduling issue with task release times and extended technical priority limitations. The proposed approach decreased total delay and the number of late tasks. Zhang et al. [ 36 ] conducted a literature study on job-shop scheduling issues and explored fresh viewpoints in the context of Industry 4.0. They added that under Industry 4.0, scheduling issues are addressed using new methodologies and approaches. The findings suggest that scheduling research should focus on smart distributed scheduling modeling and optimization. According to their assessment, this may be accomplished through two methods: combining old techniques and presenting a new way, as well as proposing new algorithms for smart distributed scheduling (Fig.  1 ).

Supply chain process

In their study, Rossit et al. [ 37 ] introduced the notion of intelligent manufacturing that arose with Industry 4.0. They have addressed the subject of smart scheduling, which they feel has a significant role in today’s product knowledge. They created the notion of tolerance scheduling in a dynamic environment to avoid production rescheduling. Likewise, Tao et al. [ 38 ] investigated contemporary advancements in production systems and smart manufacturing technologies, as well as Industry 4.0 models and stated that dynamic scheduling is one of the significant research undertaken in the literature in the context of Industry 4.0.

Jiang et al. [ 39 ] investigated the topic of energy-efficient job-shop scheduling to reduce the total cost of energy use and finishing time. However, the problem at hand was deemed NP-Hard. As a result, they created an enhanced whale optimization technique to address this issue. They improved the whale optimization method by using dispatching rules, nonlinear convergence factors, and mutation operations. They ran simulations to demonstrate the algorithm’s usefulness. According to the findings of simulations, the algorithm delivered benefits in terms of efficiency. Ortiz et al. [ 40 ] investigated a flexible job-shop problem and offered a novel methodology for solving it. They created a novel algorithm that reduces average tardiness and discovered better solutions than the existing dispatching rules. They developed a real-world production-scheduling challenge and an effective solution for solving it.

Ding and Jiang [ 41 ] examined the impact of IoT technologies in an industrial setting. They claim that while IoT has boosted production data, these data are sometimes discontinuous, uncorrelated, and challenging to use. As a result, they devised a strategy for using priceless data. They developed an RFID-based production data analysis system for production control in IoT-enabled smart businesses. Leusin et al. [ 42 ] developed a multi-agent system in a cyber-physical approach to handling the dynamic job-shop scheduling problem. The suggested system included self-configuring characteristics in the manufacturing process. This was accomplished through the usage of agents and IoT. In the shop, real-time data was used to make more informed decisions. A real-world case study was used to test the concept. The benefits of utilizing dynamic data and IoT in industrial applications are explored (Fig.  2 ).

Supply chain planning using DSS

Following a review of the relevant literature, it was discovered that numerous models were created to establish a decision support system for continuous process improvement based on IoT-enabled data analytics. There is a need to improve the decision support system for dynamic settings that can function with various dispatching criteria. The main goal is to improve the efficiency of production management and the job-shop.

Methodology

Knowledge-based decision support systems and model management systems have used tools like artificial intelligence to provide smarter decision-maker support. In addition, a model is presented toward closing the gap between analytical and transactional models, which are utilized in the organizational and technical aspects. It further implements different hierarchical levels throughout the enterprise structure, making information quality available. In particular, different and multiple decision insights might be sufficient throughout the decision-making task, which increases the speed response of the decision-support system.

This study tackles process control decisions associated with coordination and procedural control. Thereby, the integration of control sequence steps in the equipment modules is dealt with along with the transition between control recipes. Such decisions are mentioned in the control aspects, which receive data from the scheduling point, and offer data with the actual phase. Afterward, the information flow procedures were presented at this decision level and their association with other decision levels. The decisions are associated with the integration of the control recipe. The batch operation is managed through coordination control using the control recipe scheme in the ontological model. MATLAB managed data, and the JAVA environment offers the control function from the scheduling function with the relevant data. Consequently, it is an iterative process for decision-making, which encompasses the scheduling and control levels.

Moreover, the continuous-time STN illustration relies on explaining a common time grid that is variable and authentic for all shared resources. The model implies that all tasks commencing at a predefined time guarantee the authenticity of the material balances. Two binary variables were adopted to denote when a given task starts and finishes. The scheduling issues are represented through the ontological model and appropriately interpreted using the Java environment. GAMS implemented the formulation, whereas MILP solver was utilized to execute this formulation as the implied issues were lineal.

Concept integration

The DSS framework is intended to aid in the decision-making process. Many alternatives exist, particularly those linked with Industry 4.0 technologies; nonetheless, researchers’ objective is to build a more robust system to reduce the risk of human mistakes, particularly during the data entry phase. However, the critical components of DSS and technology employed are data management, communication, the user (decision-maker), and the simulation model. The Knowledge-Based Engineering (KBE) technique was used to create the framework, which is ideal for lean products. Furthermore, knowledge in DSS and Industry 4.0 indicates the use of data management tools via IoT in this idea where human aspects (users/decision-makers) continue in developing innovation to boost productivity (Prasad, 2014) (Fig.  3 ). In conjunction, LM principles functioned as a bridge.

Concept of DSS design based on KBE (Source: Prasad, 2014)

Developing framework

Combining physical and virtual processes results in smart manufacturing (Godfrey, 2002). IoT uses the internet networking idea to collect data from sensors. The data is collected using a barcode sensor. They are then installed in the database before being sent to the server through the network. The MySQL command is provided to compile and code the data to match the simulation required input. These step procedures give the core of an Industry 4.0-compliant networking system. Simulation-based Knowledge-Based Modeling (KBM) allows users to forecast and produce reliable results based on simulation data (Prasad and Rogers, 2005). Figure  4 illustrates the enhancement of proposed system in the context of Industry 4.0.

The proposed framework

Data requirement

The data needed are specified to take decisions at the scheduling level. The phases consist of the usability to schedule predefined in this ontological model regarding all cases. In particular, the scheduling function needs the information regarding capacity, demand, due date, product stage-unit, quantities in/out, processing time, stage-process, time horizon, and unit availability. The Java code was programmed to generate the input files to schedule optimization tools.

The investigation is made for academic purposes only and not for the organizations’ promotion or human resources appraisal. The questionnaire was e-mailed to the respondents along with a consent letter which doubles as a letter of introduction of the research and what is expected of the participants. Participants were assured of their anonymity and the confidentiality of their responses. The participants were also assured that no harm would befall them on their participation in the study. The participants were informed to fill and submit the form online only if they were interested in participating in the study. Otherwise, they were imploring to ignore the mail. The responses from the participant were downloaded after a couple of weeks of initiation, and the result was analyzed. The study also searched online, via the Google search engine, for articles related to information and material flow in organizations published in reputable journals, reviewed them for collation of data on associating factors of information and material flow. The result of the review formed the basis of identifying factors related to information and material flow for the study.

Data administration

Data collected from the company staff were subjected to analysis using SPSS v20.0. Descriptive statistics, factor analysis, correlation, and regression tools were used to analyze the data. Factor analysis was done to illustrate the strength of items or associating factors of information and material flow and supply chain performance in their groups. The correlation was used to test the relationships between information and material flow and supply chain performance. On the other hand, regression analysis was deployed to understand the predictability of information and material flow of supply chain performance and which of the independent variables has a stronger contribution to supply chain performance.

The descriptive analysis explains the percentage distribution of the respondents on the characteristics of the demographic variables (Table 1 ).

The result of factor analysis of the data collated from the sampled organizations is presented. Factor analysis is the calculation and explanation of the strength of individual items concerning the group that forms the whole of the factor being investigated. Tables 2 and 3 present the strength of things that make up information and material flow and supply chain performance within organizations.

The primary data quantitatively obtained from the sampled organization was used for the analysis due to its validity in evaluating the extent of information and material flow and level of supply chain performance in the organizations. Correlation analysis explains the relationships between information and material flow and organizations’ supply chains. On the other hand, regression analysis illustrates the strength of the contribution of information and material flow on supply chain performance to explicitly ascertain the causal association in line with the third objective of the study. The correlation and regression analysis results are presented in Tables 4 and 5 .

Table 4 presents the correlation matrix analysis showing the inter-correlation between information flow, material flow, and supply chain performance. First, the table shows the mean of Information flow to be (M = 19.53; SD = 1.98); Material flow (M = 27.06; SD = 2.61); Supply chain performance (M = 38.40; SD = 2.96). The relationship between the variables shows that there are significant positive relationships between information flow and supply chain performance (r = 0.237*). Also, the result in the table shows that material flow is significant in its positive relationship with supply chain performance (r = 0.411**). This result implies that the less hampered the flow of information and material experienced, the higher the level of supply chain performance. To get a clearer picture of each of the information and material flow and their strength of prediction supply chain performance, the data is further subjected to regression analysis, and the result is presented in Table 5 .

Table 5 presents a regression result showing the relationship between information and material flow and supply chain performance. Recall that the elements of information flow considered in the study are: information quality, information accuracy, information adequacy, credibility, information timeliness, and visibility. The elements that makeup material flow include material price fluctuation, imperfect sorting, delivery delay, poor planning, fund shortage, non-alignment specification, unnecessary paperwork, and in-house logistic problem. The result showed that material flow represents the strongest factor in predicting supply chain performance. It had a significant positive relationship and accounted for about 52.3% of the variance of supply chain performance (Beta = 0.523; t = 5.648; pv = 0.000). Information flow also had a significant positive relationship, but account less for about 42.3% of supply chain performance (Beta = 0.423; t = 4.534; pv = 0.000). Collectively, information and material flow had a significant positive relationship of 54.0% with supply chain performance of organizations (R2 = 0.540; Fcal = 31.236; pv = 0.000); although there is a significant difference in the degree of contribution of the individual factors to supply chain performance as pointed out by the Fcal.

Data mining techniques can process the data present in dynamic databases to determine the problems faced in production, generate rules to control output, improve product quality, and develop automation based on work intelligence. A better understanding of production systems may aid researchers in developing sophisticated DSS and decision-makers in making better judgments. Controlling the production environment with real-time data may also assist researchers in modeling the system without making any assumptions. This technique can bridge the gap between theory and experience, resulting in more realistic solutions.

Furthermore, real job processing times might be validated using job data from the past. This study would aid in predicting the system’s behavior under various scenarios, any subsystem could simply integrate into this system. Combining artificial intelligence or machine learning-based subsystems might form the basis of future smart factories. In this regard, this study indicates the benefits of such subsystem integrations, such that even technology integration in a real job-shop may take considerable time and effort.

The findings might not be an exact replication of the results from previous studies that are limited to the influence of information and material flow on the performance of the supply chain as there are concerns of what factors constitute information and material flow that need to be identified considered. However, with the finding of associating information and material flow may need to consider this in managing the flow and the supply chain. Associating factors such as information quality, information visibility, material cost, fund shortage, and so on, play a role in information and material flow and the decisions made in an organization, which in turn impact supply chain performance, as cost of material, for example, significantly impact the flow of materials. Factors associated with information and material flow need to be considered in decision making as well, as the cost in any of the elements affects the flow, which would impede the organization's supply chain performance. As a supply chain involves a network of supply chain partners, organizations need to ensure that information quality, information visibility, material price, funding, and so on are managed effectively through a centralized policy as they provide significant insights into the health of the supply chain.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Dellino G, Laudadio T, Mari R, Mastronardi N, Meloni C. A reliable decision support system for fresh food supply chain management. Int J Prod Res. 2018;56(4):1458–85.

Article   MATH   Google Scholar  

Carter CR, Rogers DS. A framework of sustainable supply chain management: moving toward new theory. Int J Phys Distrib Logist Manag. 2008. https://doi.org/10.1108/09600030810882816 .

Article   Google Scholar  

Gorry GA, Morton MSS. A framework for management information systems. Sloan Manag Rev 1971;13:55–70.

Hemmat M, Ayatollahi H, Maleki M, Saghafi F. Health information technologies in Iran: opportunities for development. Med J Islam Repub Iran. 2019;33:103.

Google Scholar  

Belgium S, Gorunescu F. How can intelligent decision support systems help the medical research? In: Belciug S, Gorunescu F, editors. Intelligent decision support systems—a journey to smarter healthcare. Berlin: Springer; 2020. p. 71–102.

Rico N, Díaz I, Villar JR, de la Cal E. Intelligent decision support to determine the best sensory guardrail locations. Neurocomputing. 2019;18(354):41–8.

El Abdallaoui HE, El Fazziki A, Ennaji FZ, Sadgal M. Decision support system for the analysis of traffic accident Big Data. In: 2018 14th International Conference on Signal-Image Technology and Internet-Based Systems (SITIS). IEEE. 2018. pp. 514–521.

Tan WJ, Yang CF, Château PA, Lee MT, Chang YC. Integrated coastal zone management for sustainable tourism using a decision support system based on system dynamics: a case study of Cijin, Kaohsiung. Taiwan Ocean Coast Manag. 2018;1(153):131–9.

Sholahuddin A, Shadriyah Y. Analysis of students’ process skills and chemistry learning outcomes. In: The 5th South East Asia Development Research Conference (SEA-DR) International Conference. Amsterdam: Altlantis Press; 2017. p. 364–370.

Garmendia E, Gamboa G, Franco J, Garmendia JM, Liria P, Olazabal M. Social multi-criteria evaluation as a decision support tool for integrated coastal zone management. Ocean Coast Manag. 2010;53(7):385–403.

Amir-Heidari P, Arneborg L, Lindgren JF, Lindhe A, Rosén L, Raie M, Axell L, Hassellöv IM. A state-of-the-art model for spatial and stochastic oil spill risk assessment: a case study of oil spill from a shipwreck. Environ Int. 2019;1(126):309–20.

Govindan K, Mina H, Alavi B. A decision support system for demand management in healthcare supply chains considering the epidemic outbreaks: A case study of coronavirus disease 2019 (COVID-19). Transp Res Part E: Logist Transp Rev. 2020;1(138):101967.

Koh SL, Genovese A, Acquaye AA, Barratt P, Rana N, Kuylenstierna J, Gibbs D. Decarbonising product supply chains: design and development of an integrated evidence-based decision support system–the supply chain environmental analysis tool (SCEnAT). Int J Prod Res. 2013;51(7):2092–109.

Attaran M, Attaran S. Collaborative supply chain management: the most promising practice for building efficient and sustainable supply chains. Bus Proc Manag J. 2007. https://doi.org/10.1108/14637150710752308 .

Salmon K. Análisis estratégico de la cadena de suministros fibras-textil-vestido (México).

Cassivi L. Collaboration planning in a supply chain. Supply Chain Manag: Int J. 2006. https://doi.org/10.1108/13598540610662158 .

Shepherd C, Günter H. Measuring supply chain performance: current research and future directions. In: Fransoo JC, Waefler T, Wilson JR, editors. Behavioral operations in planning and scheduling. Berlin: Springer; 2010. p. 105–21.

Chapter   Google Scholar  

Ferguson BR. Implementing supply chain management. Prod Invent Manag J. 2000;41(2):64.

Simatupang TM, Sridharan R. A benchmarking scheme for supply chain collaboration. Benchmarking: Int J. 2004. https://doi.org/10.1108/14635770410520285 .

Hsu PH, Wee HM. Horizontal suppliers coordination with uncertain suppliers deliveries. Int J Oper Res. 2005;2(2):17–30.

MATH   Google Scholar  

Alam A, Bagchi PK. Supply chain capability as a determinant of FDI. Multinatl Bus Rev. 2011. https://doi.org/10.1108/15253831111172658 .

Stevenson M, Spring M. Flexibility from a supply chain perspective: definition and review. Int J Oper Prod Manag. 2007. https://doi.org/10.1108/01443570710756956 .

Hull B. A structure for supply-chain information flows and its application to the Alaskan crude oil supply chain. Logist Inf Manag. 2002. https://doi.org/10.1108/09576050210412639 .

Li G, Yan H, Wang S, Xia Y. Comparative analysis on value of information sharing in supply chains. Supply Chain Manag: Int J. 2005. https://doi.org/10.1108/13598540510578360 .

Simatupang TM, Sridharan R. The collaborative supply chain. Int J Logist Manag. 2002;13(1):15–30.

Lee H, Whang S. Decentralized multi-echelon supply chains: incentives and information. Manage Sci. 1999;45(5):633–40.

Koh SL, Saad S, Arunachalam S. Competing in the 21st century supply chain through supply chain management and enterprise resource planning integration. Int J Phys Distrib Logist Manag. 2006. https://doi.org/10.1108/09600030610677401 .

Bovet D, Martha J. Supply chain hidden profits. Mercer Management Consulting. 2003. https://books.google.com/books?id=dJsFh-LIKEkC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false . Accessed 8 Aug 2003.

Durugbo C, Tiwari A, Alcock JR. Managing integrated information flow for delivery reliability. Ind Manag Data Syst. 2014. https://doi.org/10.1108/IMDS-10-2013-0430 .

Kembro J, Selviaridis K. Exploring information sharing in the extended supply chain: an interdependence perspective. Supply Chain Manag: Int J. 2015. https://doi.org/10.1108/SCM-07-2014-0252 .

Hsu CW, Hu AH. Green supply chain management in the electronic industry. Int J Environ Sci Technol. 2008;5(2):205–16.

Tummala VR, Phillips CL, Johnson M. Assessing supply chain management success factors: a case study. Supply Chain Manag: Int J. 2006. https://doi.org/10.1108/13598540610652573 .

Kück M, Ehm J, Hildebrandt T, Freitag M, Frazzon EM. Potential of data-driven simulation-based optimization for adaptive scheduling and control of dynamic manufacturing systems. In: 2016 Winter Simulation Conference (WSC). IEEE. 2016. p. 2820–2831.

Ersöz S, Türker AK, Aktepe A. Üretim Süreçlerinin Optimizasyonunda RFID Teknolojisi ve Uzman Sistem Temelli Tümle¸sik Yapının ERP Sistemine Entegrasyonu ve FNSS Savunma Sistemleri A. ¸S.’de Uygulanması; San-Tez Project Report; Ankara, Turkey, 2016. https://adnanaktepe.com/projeler/ . Accessed 14 Mar 2019.

Xiong H, Fan H, Jiang G, Li G. A simulation-based study of dispatching rules in a dynamic job shop scheduling problem with batch release and extended technical precedence constraints. Eur J Oper Res. 2017;257(1):13–24.

Article   MathSciNet   MATH   Google Scholar  

Zhang J, Ding G, Zou Y, Qin S, Fu J. Review of job shop scheduling research and its new perspectives under Industry 4.0. J Intell Manuf. 2019;30(4):1809–30.

Rossit DA, Tohmé F, Frutos M. Industry 4.0: smart scheduling. Int J Prod Res. 2019;57(12):3802–13.

Tao F, Qi Q, Liu A, Kusiak A. Data-driven smart manufacturing. J Manuf Syst. 2018;1(48):157–69.

Jiang T, Zhang C, Zhu H, Gu J, Deng G. Energy-efficient scheduling for a job shop using an improved whale optimization algorithm. Mathematics. 2018;6(11):220.

Ortíz MA, Betancourt LE, Negrete KP, De Felice F, Petrillo A. Dispatching algorithm for production programming of flexible job-shop systems in the smart factory industry. Ann Oper Res. 2018;264(1):409–33.

Ding K, Jiang P. RFID-based production data analysis in an IoT-enabled smart job-shop. IEEE/CAA J Autom Sin. 2017;5(1):128–38.

Leusin ME, Frazzon EM, Uriona Maldonado M, Kück M, Freitag M. Solving the job-shop scheduling problem in the industry 4.0 era. Technologies. 2018;6(4):107.

Download references

Acknowledgements

The author is highly thankful for all the associated personnel who contributed to the completion of this study. This research holds no conflict of interest.

This research received no external funding.

Author information

Authors and affiliations.

Information System Department, King Abdulaziz University, Jeddah, Saudi Arabia

Dimah Hussein Alahmadi & Arwa A. Jamjoom

You can also search for this author in PubMed   Google Scholar

Contributions

DHA: analysis & drafting. AAJ: conception, administration, analysis, editing & approval. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Arwa A. Jamjoom .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Alahmadi, D.H., Jamjoom, A.A. Decision support system for handling control decisions and decision-maker related to supply chain. J Big Data 9 , 114 (2022). https://doi.org/10.1186/s40537-022-00653-9

Download citation

Received : 25 October 2021

Accepted : 19 September 2022

Published : 30 November 2022

DOI : https://doi.org/10.1186/s40537-022-00653-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Decision support system
• Continuous process improvement
• Data analytics
• Supply chains

• Open access
• Published: 12 September 2024

Vaccination governance in protracted conflict settings: the case of northwest Syria

• Ronja Kitlope Baatz 1   na1 ,
• Abdulkarim Ekzayez 2 , 3   na1 ,
• Yasser Najib 4 ,
• Munzer Alkhalil 5 , 6 ,
• Mohammad Salem 4 ,
• Mohammed Ayman Alshiekh 7 &
• Preeti Patel 2  

BMC Health Services Research volume  24 , Article number:  1056 ( 2024 ) Cite this article

Metrics details

Effective vaccination governance in conflict-affected regions poses unique challenges. This study evaluates the governance of vaccination programs in northwest Syria, focusing on effectiveness, efficiency, inclusiveness, data availability, vision, transparency, accountability, and sustainability.

Using a mixed-methods approach, and adapting Siddiqi’s framework for health governance, data were collected through 14 key informant interviews (KIIs), a validating workshop, and ethnographic observations. Findings were triangulated to provide a comprehensive understanding of vaccination governance.

The study highlights innovative approaches used to navigate the complex health governance landscape to deliver vaccination interventions, which strengthened sub-national vaccination structures such as The Syria Immunisation Group (SIG). The analysis revealed several key themes. Effectiveness and efficiency were demonstrated through cold-chain reliability and extensive outreach activities, though formal reports lacked detailed analysis of vaccine losses and linkage between disease outbreak data and coverage statistics. Key informants and workshop participants rated the vaccination strategy positively but identified inefficiencies due to irregular funding and bureaucracy. Inclusiveness and data availability were prioritised, with outreach activities targeting vulnerable groups. However, significant gaps in demographic data and reliance on paper-based systems hindered comprehensive coverage analysis. Digitalisation efforts were noted but require further support. The SIG demonstrated a clear strategic vision supported by international organizations such as the World Health Organization, yet limited partner participation in strategic planning raised concerns about broader ownership and engagement. While the SIG was perceived as approachable, the lack of public documentation and financial disclosure limited transparency. Internal information sharing was prevalent, but public communication strategies were insufficient. Accountability and sustainability faced challenges due to a decentralized structure and reliance on diverse donors. Despite stabilizing factors such as decentralization and financial continuity, fragmented oversight and reliance on donor funding remained significant concerns.

The study highlights the complexities of vaccination governance in conflict-affected areas. Comparisons with other conflict zones underscore the importance of local organisations and international support. The SIG’s role is pivotal, but its legitimacy, transparency, and inclusivity require improvement. The potential transition to early recovery in Syria poses additional challenges to SIG’s sustainability and integration into national programs.

The governance of vaccination in northwest Syria is multifaceted, involving multiple stakeholders and lacking a legitimate government. Enhancing transparency, local ownership, and participatory decision-making are crucial for improving governance. The role of international bodies is essential, emphasising the need for structured feedback mechanisms and transparent monitoring processes to ensure the program’s success and sustainability.

Key message

• A hybrid governance model that combines top-down and bottom-up approaches effectively improves immunisation programs in conflict settings and promotes local ownership.

• In conflict settings, immunisation programmes require strong and direct intervention from the WHO with central management and coordination of the vaccine activities.

• In conflict areas, when the government is a party to the conflict or has limited access to some areas, United Nations institutions must intervene to manage or support vaccine activities in partnership with local entities, regardless of notions of national sovereignty.

• In Syria, the reluctance of United Nations institutions to fill the void left by the state in areas outside its control and provide vaccines led to the emergence of many diseases, including polio in 2013 and measles in 2017.

Peer Review reports

Introduction

Immunisation services are essential for any health system to ensure protection against major transmissible diseases. Armed conflicts often influence the availability, quality, accessibility, and uptake of vaccination services, which can lead to the emergence of outbreaks and epidemics [ 1 , 2 ]. The restoration of regular immunisation services in emergency contexts has not been extensively studied, and protracted crises “underscore the need to consider matters beyond the emergency mindset” [ 3 ]. Furthermore, health partnerships remain largely centred on national governments [ 4 ], raising the question of how areas beyond state control can best organise routine vaccination services.

The Syrian conflict, which started in March 2011, has had a devastating impact on the health system of the country; with vaccination coverage dropping from more than 90% for the Diphtheria, Tetanus & Pertussis (DTP) vaccine pre-conflict, to less than 10% in some areas [ 5 , 6 ]. With the fall of some areas under opposition control, the Syrian government began to withhold vaccinations from these areas, while simultaneously attacking healthcare facilities and infrastructure [ 7 ]. The decline in vaccine coverage resulted in outbreaks of Vaccine Preventable Diseases (VPDs), including polio (2013, 2017) and measles (2017, 2018) [ 8 , 9 ]. This led to vaccination becoming a priority for the humanitarian sector following the outbreak of wild poliovirus in October 2013.

Syria is now roughly divided into three main areas of control: the self-administration region of northeast Syria controlled by Kurdish majority forces, the governmental areas in the central, coastal and southern regions, and various opposition forces in the northwest. These delineations are visually depicted in Fig.  1 , where the regions are represented by the colours yellow, red, and green, respectively [ 10 ]. Opposition controlled areas in northwest Syria has a population of about 4.5 million people, of whom over a third, 1.8 million, live in camps, which is the area of focus in this study [ 11 ]. According to The United Nations Office for the Coordination of Humanitarian Affairs (OCHA), about 90% of the population is dependent on donor aid for their subsistence, including for health care [ 12 ]. Northwest Syria is governed by two main forces, the opposition forces with Turkish support in northern Aleppo, and Hayat Tahrir Al-Sham (HTS) in Idlib Governorate [ 13 ]. HTS is listed as a terrorist organisation by the US, UN, EU and Turkey, preventing aid organisations from working with them [ 14 ]. As there is no recognised government in northwest Syria and no clear end in sight to the conflict, international aid organisations are facing a long-term problem of coordination, particularly in programmes which require stability and effective governance, such as routine immunisation. Humanitarian access to northwest Syria has been using border crossing points from Turkey under annually renewed Resolutions by the UN Security Council since July 2014 [ 15 ]. However, this crossing became limited to only one crossing point, the Bab el Hawa border in 2019, and later this crossing has expired with the failure to renew this UN Resolution after being vetoed by Russia and China. The Security Council’s failure to reauthorise the long-standing cross border humanitarian aid mechanism in July 2023, has laid bare the implications for the humanitarian situation in Syria coupled with a deepening divide on the Security Council’s engagement on the issue. There is now uncertainty about the future of the aid mechanism and other UN operations in the region [ 16 ].

Areas of control in Syria as of April 2023. Source: Liveuamap, 2023

Prior to the conflict, Syria had advanced vaccination governance and high immunization coverage, with World Health Organization (WHO) and United Nations International Children’s Emergency Fund (UNICEF) estimating DTP vaccine coverage at over 89% [ 17 ]. During the conflict, vaccination activities faced significant challenges following the withdrawal of the Syrian Government from opposition-controlled territories in 2012. This led to disruptions in the supply chain, human resource shortages, and governance collapse, resulting in reduced vaccination coverage and outbreaks of diseases such as Polio and Measles [ 18 ]. Emergency vaccination campaigns were initiated by local and international actors to address these outbreaks, with the establishment of entities such as the Polio Task Force and Measles Task Force. Since 2016, vaccination efforts have been led by the Syria Immunisation Group (SIG), formed by local humanitarian actors and co-chaired by WHO and UNICEF. Please see Table  1 for the vaccination schedule in Syria before and after the conflict.

Despite Syria’s eligibility for Global Alliance for Vaccines and Immunization (GAVI) support in 2019, actual funding received remains lower than pledged, making it challenging to assess the total cost of vaccine activities [ 19 , 20 ]. The literature on vaccination governance in northwest Syria is scant, with limited distinction between northwest Syria and government-controlled areas. Comprehensive accounts of SIG’s work are rare, with the WHO 2020 report on Syria providing one notable exception [ 21 ]. This lack of literature may reflect the complex political economy context, as government withholding of vaccinations prompted alternative actors to facilitate vaccination and governance [ 22 ].

This study aims to explore the effectiveness and efficiency of vaccination governance in northwest Syria (NWS), its responsiveness, inclusivity, and informed decision-making processes, as well as its vision, strategy, transparency, and accountability. By examining these aspects, the research seeks to provide a comprehensive understanding of how vaccination programs operate in conflict-affected areas and the unique challenges they face.

Methodology

This study employed a mixed-methods approach consisting of semi-structured qualitative interviews, a validation workshop, and ethnographic observations to comprehensively investigate vaccination governance in northwest Syria.

Firstly, we adapted the Siddiqi framework for health governance [ 23 ] with modifications to accommodate the unique challenges and dynamics present in northwest Syria. Its six key principles offer a structured approach to assess governance effectiveness, inclusivity, transparency, and accountability, which were central to the study’s objectives. This adapted framework guided the data collection, analysis, and interpretation processes, providing a structured approach to examining vaccination governance from a health system perspective.

Secondly, we conducted 14 semi-structured qualitative Key Informant Interviews (KIIs) with key informants involved in vaccination governance in northwest Syria. Purposive sampling was used to select participants representing various stakeholders, including representatives from local health directorates, international organizations, and community leaders - please see Table  2 . Participants were identified based on their expertise and roles in vaccination delivery. We approached potential participants through email and phone calls, explaining the purpose of the study and inviting them to participate. Those who agreed to participate were scheduled for interviews at their convenience. The semi-structured interview guide (see Supplementary Material) aimed to explore participants’ experiences, perspectives, and challenges related to vaccination governance. The interviews were audio-recorded with participants’ consent and transcribed verbatim for analysis. Thematic analysis was conducted using both deductive and inductive approaches, with the Siddiqi framework guiding the thematic grouping and coding process. Notably, only two of the interviewees identified as female. This gender disparity reflects broader gender imbalances in leadership positions within the context of conflict-affected areas and may influence the perspectives and priorities discussed during the workshop.

Thirdly, a validation workshop was conducted in Gaziantep in November 2023 to validate the findings from the interviews and gather additional insights from stakeholders. The 15 participants in the workshop included key informants who had been interviewed, as well as other relevant stakeholders – please see Table  2 . An overview of the key findings per theme identified in the interviews was presented, followed by a discussion to validate and elaborate on these findings. The workshop facilitated a collaborative process to prioritize the main achievements and challenges identified in the interviews.

In addition, ethnographic observations were conducted alongside the field data collection to provide contextual insights into vaccination delivery and governance practices in northwest Syria. These observations involved daily immersion in the field, engaging in informal conversations with stakeholders, and documenting observations through field notes. This approach was used to build trust with key stakeholders, helping them understand the importance of our research and encouraging them to openly share their views and participate in research activities. The informal conversations and daily immersion provided rich qualitative data on the local context, practices, and challenges, which were crucial for interpreting the collected data. Additionally, relevant documents, such as reports and policy documents, were collected and analysed to complement the ethnographic data.

The three sets of data—interviews, workshop discussions, and ethnographic observations—were triangulated to enhance the validity and reliability of the findings. Triangulation was conducted through comparing and cross-referencing information from each data source. Initially, key themes and findings from the interviews were identified and categorised. These themes were then cross-checked against insights gathered from workshop discussions and ethnographic observations to identify common patterns, discrepancies, and unique contributions. Any discrepancies were further investigated through follow-up discussions or additional document analysis to resolve inconsistencies and confirm findings.

Ethical approval was obtained from the Institutional Review Board of King’s College London (MRA-22/23-34048) and, due to the sensitive nature of the subject, anonymity of participants was deemed critical. Informed consent was signed by all interviewees and interview records were deleted within two days after the interview, with notes being de-identified. All records and code-keys were stored on a password-protected secure drive.

This section presents five key themes that emerged from the data: effectiveness and efficiency, inclusiveness and data availability, clear vision with limited participatory strategy development, limited transparency, and accountability and sustainability. For each theme, findings are triangulated from interviews, workshop discussions, and ethnographic observations to provide a comprehensive understanding of vaccination governance in northwest Syria.

Effectiveness and efficiency

Field observations highlighted the operational success of the vaccination strategy, particularly in maintaining cold-chain reliability and conducting extensive outreach activities. Researchers noted that cold-chain facilities appeared well-maintained and outreach teams were active in various communities.

Document analysis corroborated these observations, although it revealed a lack of detailed analysis in formal reports regarding vaccine losses and linkage between disease outbreak data and coverage statistics. The annual report for 2021 noted the distribution of over 1.5 million routine vaccines and approximately 350,000 COVID-19 vaccines (SIG, 2021).

KIIs provided subjective assessments of effectiveness, with most participants rating the vaccination strategy very positively. For example, one key informant stated, “Cold-chain is very complicated, and (…) we have never faced gaps in the cold-chain. The outreach activities too, they are amazing in screening the whole community” (K-07). Another participant commented, “I think there are three successful entities in Syria. White Helmets, Early Warning and Response Network (EWARN) and SIG. Basically, they are performing governmental performance, without being a government” (K-10).

The workshop echoed these sentiments, emphasising the reliability of cold-chain logistics and the effectiveness of outreach programs. Participants highlighted the comprehensive knowledge outreach teams had about the communities, such as culture and health seeking behaviour, which facilitated high vaccine coverage.

Analysis suggests that while the subjective assessments are positive, the lack of detailed data in formal documents indicates a need for more robust quantitative evaluation mechanisms to fully substantiate these claims.

Efficiency was qualitatively explored through factors such as human resources, bureaucracy, corruption, and the non-governmental nature of the program. Field observations noted strong capacity among staff and stable governance structures.

Documents reviewed pointed to significant bureaucracy but suggested it was a necessary component to prevent corruption. KIIs reinforced this, with one participant noting, “You can’t do any humanitarian process without this paperwork, to be honest. It is the right way, because otherwise you are corrupted” (K-01). Another added that corruption was low due to the nature of the resources involved, stating, “There are few reasons for people to steal from this programme. It isn’t food baskets or money, it’s vaccines” (K-01).

Workshops confirmed these findings but also highlighted inefficiencies due to the lack of government services and irregular funding, which led to service discontinuations. One workshop participant explained, “The Expanded Programme for Immunisation (EPI) is continuous, it should be a 2 or 3 year project. For example, the first project ends by the end of May and the next project starts mid-June. So, there is a gap for staff, so they don’t receive their salaries” (W-02).

In conclusion, while the vaccination governance seems to be efficient with limited observed effectiveness, challenges remain in documentation and the impacts of funding irregularities, short termism and uncertainty.

Inclusiveness, responsiveness, and data availability

Field observations indicated that accessibility and inclusiveness are prioritized in vaccination efforts, with outreach activities playing a crucial role in reaching vulnerable groups. Researchers observed that outreach sessions outnumbered fixed sessions, reflecting the emphasis on inclusivity.

Document analysis revealed systematic data collection efforts to identify reasons for missed vaccinations to target vulnerable groups, including zero-dose children, people with disabilities, female-headed households, and those living in remote areas. However, significant gaps in demographic data and reliance on paper-based systems were noted, hindering comprehensive coverage analysis.

KIIs highlighted the challenges in data availability. One participant mentioned, “The most reliable approximations of vaccine coverage come from last year’s vaccination data and the door-to-door polio campaign” (K-05). Another added, “Alternative population data is available from OCHA, but it is considered inferior to the more comprehensive and up-to-date polio data” (K-06). This reliance on figures from previous Polio vaccination campaigns is confirmed by our document analysis. In 2021 the SIG vaccinated 134,083 children with Bacillus Calmette–Guérin (BCG). The Polio campaign in the previous year vaccinated a total of 155.378 children under 1. According to third party monitoring, the coverage rate of this polio campaign was 93%. Assuming that the age-distribution of the coverage is equal, this would make the total number of children under 1 in northwest Syria 167.073. Accordingly, the coverage rate for BCG would then be 80.3%. Similar statistics currently being used as coverage data, but these are suboptimal.

Workshop participants echoed these concerns, emphasizing the need for digitalization of medical and vaccination records. A participant remarked, “Paper vaccination cards are often lost, and manual data collection is prone to error. Digital systems are urgently needed” (W-03).

Our analysis indicates that while inclusivity is a stated priority and efforts are made to collect relevant data, the effectiveness of these efforts is limited by significant data availability challenges. Digitalization initiatives are a positive step but require more support and implementation.

Clear vision with limited participatory strategy development

Field observations showed the SIG’s active involvement in strategic planning, supported by WHO and GAVI. Researchers noted clear mission statements and detailed strategies in the SIG’s multi-year plan, though awareness among partners was limited.

Document analysis confirmed the existence of structured strategic plans but indicated fragmented decision-making processes involving multiple stakeholders, including donors, partners, and the SIG. The SIG was observed to function as a central coordination and mediation platform.

KIIs provided insights into the strategic planning processes, with participants acknowledging sufficient opportunities for input but noting limited participation from partners. One participant stated, “I don’t think the NGOs are participating in finding solutions. Mainly the SIG is doing this. The SIG is doing a good job, so we feel relaxed somehow, so we don’t want to interfere in the system” (K-11). Another added, “It is positive that the implementing partners are only implementing the central plans” (K-06).

Workshop participants supported these findings, expressing trust in the SIG’s strategic planning but also highlighting the lack of engagement from partners in the decision-making process. One participant noted, “The SIG maintains the strategy and the quality of the strategy. In humanitarian crises and the Syrian context, we operate as organizations, but we established a central team” (W-04).

Our analysis suggests that while the SIG has a clear vision and structured strategic plans, the limited participatory strategy development may hinder broader ownership and engagement from all partners.

Limited transparency

Field observations noted a general perception of the SIG being approachable, but with limited transparency in documentation. Researchers observed that information sharing was mostly internal, with minimal public disclosure.

Document analysis highlighted the lack of an internet presence, financial disclosure, and public availability of strategic plans and annual reports. Information was primarily disseminated through internal reports and meetings, limiting access for external stakeholders.

KIIs revealed a discrepancy between perceived and actual transparency. One participant commented, “A normal Ministry of Health would not separately publish their vaccination results in so much detail” (K-03). Another stated, “Partners funded through the WHO share their financial data with the SIG, but privately funded partners do not” (K-02).

Workshop participants emphasized the need for greater transparency, particularly for stakeholders not directly involved in the SIG’s network. A participant remarked, “It is difficult to obtain information about the topic if one is not part of the network. Only the WHO and the Assistant Coordination Unit (ACU) additionally report on selected aspects of vaccination” (W-01).

Our analysis indicates that while the SIG is considered transparent by partners due to its approachability, the lack of public documentation and financial disclosure limits overall transparency. Enhanced public communication strategies could improve transparency and accountability.

Accountability and sustainability

Field observations underscored the complex collaboration of stakeholders underpinning vaccine provision, with no single body having legitimate oversight. Filed researchers noted the decentralised structure and reliance on various donors.

Document analysis highlighted the lack of enforcement mechanisms for medical guidelines and protocols. The SIG’s Statement of Principle lacked enforceable standards, leaving de facto power with diverse donors. This patchwork funding approach posed challenges to accountability and sustainability.

KIIs pointed to the absence of a central governance body, with one participant noting, “The donors know that the SIG is not officially on the papers, but they know there is a body called SIG responsible for reaching the target, achieving the indicators, and supervising technically” (K-07). Another participant identified potential risks, stating, “The cut of funds, war, and lack of stability of the security situation. We have the scenario, but we don’t know what will happen” (K-08).

Workshop participants discussed stabilising factors such as the system’s size, decentralized structure, and financial continuity. One participant remarked, “The system grows and becomes a stable system. Everyone is aware of how the system is growing, and this assists the continuity” (W-05).

Our analysis concludes that while there are significant challenges to accountability and sustainability, including fragmented oversight and reliance on diverse donors, stabilizing factors such as decentralization and financial continuity offer some resilience against potential disruptions. Capacity building at district and governorate levels is crucial for ensuring long-term stability and effectiveness.

The primary themes under investigation in this study encompassed the effectiveness and efficiency of the vaccination governance in northwest Syria; its responsiveness, inclusivity, and informed decision-making; its vision and strategy; transparency; and accountability and sustainability.

The management and coordination of vaccination in conflict-affected areas pose significant challenges to effectiveness and efficiency. In regions like northwest Syria, where government control is limited, the discontinuation of routine vaccination services exacerbates these challenges. Comparisons with other conflict-affected areas, such as Myanmar and Somalia, highlight the role of local organizations and international support in filling governance gaps [ 24 , 25 ]. However, research on vaccination coordination in northwest Syria remains sparse, underscoring the need for a deeper understanding of local structures and operations.

Prior to 2016, the health governance model followed a bottom-up approach, with local entities playing significant roles in vaccination activities. With the establishment of SIG, a hybrid top-down and bottom-up model emerged, shifting the focus to international support and coordination while preserving field connection. This model change reflects the unique challenges of vaccination services in conflict-affected regions and underscores the need for a collaborative approach under the United Nations’ umbrella.

The Syria Immunisation Group (SIG) plays a pivotal role in vaccination governance in northwest Syria, aiming to address these challenges. While SIG has gained internal legitimacy through collaboration with health directorates (HDs) and external legitimacy through collaborating with WHO and UNICEF, concerns regarding accountability and inclusivity persist. The lack of transparency and involvement of partners in strategic planning processes hinder informed decision-making. These finding are in line with a study by Alaref et al. in 2023 which evaluated six governance principles for central quasi-governmental institutions in northwest Syria, including SIG, and found that its legitimacy is fair and requires improvement, scoring 41–60% on a health system governance scale adapted for this paper. Accountability, transparency, effectiveness and efficiency were poor and required significant improvement, scoring 21–40%, while strategic vision was very poor or inactive, scoring 0–20% [ 26 ].

Despite having a strategic plan and receiving support from international organisations like the WHO and GAVI, SIG faces contradictions in its effectiveness and efficiency. The transition from emergency task forces to SIG was marked by power dynamics and challenges to local ownership, raising questions about sustainability and integration into national vaccination programs [ 9 ]. The potential transition of WHO operations further complicates the future of SIG, posing a key challenge to early recovery in Syria.

These findings raise questions about the future of the SIG body in light of the political and military changes in the region and the constant threat associated with cross-border operations. What would happen if the WHO ceased operations in Gaziantep and moved to Damascus, where a national vaccine program has been in place for decades? In such a scenario, would the SIG continue to carry out its activities in northwest Syria, or would it become a part of the national vaccine program? This is a key challenge for the transition to early recovery in Syria.

In conclusion, the governance of vaccination in conflict-affected areas of northwest Syria is complex, with multiple stakeholders involved and a lack of a legitimate government to fulfil essential functions. The success of the vaccination program heavily relies on the efforts of the Syria Immunisation Group (SIG), which acts as a trusted mediator between various stakeholders. However, the lack of transparency and accountability hinders the ability to assess the program’s effectiveness and efficiency. This calls for a push towards more localised ownership and transparency, with a hybrid top-down and bottom-up approach that addresses the unique context of conflict settings. Engaging local partners in decision-making and capacity building can improve sustainability and address issues surrounding legitimacy. Moreover, the responsibility to protect public health goes beyond national sovereignty, and the role of international bodies like the WHO becomes crucial in conflict areas. Inaction or delayed action can have catastrophic consequences, as witnessed in Syria with the emergence of diseases like polio and measles. It is essential to implement a structured feedback mechanism and transparent monitoring and evaluation processes to address challenges and foster trust among stakeholders and the community. Ultimately, the findings of this study inform debates around health governance in conflict settings, highlighting the need for more inclusive, transparent, and context-sensitive approaches to ensure the success and sustainability of vaccination programs.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to the sensitive nature of the data, but are available from the corresponding author on reasonable request.

Abbreviations

Syria Immunisation Group

Early Warning and Response Network

Expanded Programme on Immunization

World Health Organization

Global Alliance for Vaccines and Immunization

United Nations Office for the Coordination of Humanitarian Affairs

Health Directorates

Global Alliance for Vaccines and Immunisation

Key Informant Interviews

United Nations International Children’s Emergency Fund

Sato R. Effect of armed conflict on vaccination: evidence from the Boko Haram insurgency in northeastern Nigeria. Confl Health. 2019;13(1):1–10.

Article   Google Scholar  

Ngo NV, Pemunta NV, Muluh NE, Adedze M, Basil N, Agwale S. Armed conflict, a neglected determinant of childhood vaccination: some children are left behind. 2019;16(6):1454–63. https://doi.org/10.1080/2164551520191688043

Lam E, McCarthy A, Brennan M. Vaccine-preventable diseases in humanitarian emergencies among refugee and internally-displaced populations. Hum Vaccin Immunother. 2015;11(11):2627–36.

Article   PubMed   PubMed Central   Google Scholar  

Kennedy J, Michailidou D. Civil war, contested sovereignty and the limits of global health partnerships: a case study of the Syrian polio outbreak in 2013. Health Policy Plan. 2017;32(5):690–8.

Article   PubMed   Google Scholar  

Pereira A, de Southgate L, Ahmed R, O’Connor H, Cramond P, Lenglet V. A. Infectious Disease Risk and Vaccination in Northern Syria after 5 years of Civil War: the MSF experience. PLoS Curr. 2018;10.

Tajaldin B, Almilaji K, Langton P, Sparrow A. Defining polio: closing the gap in global surveillance. Ann Glob Health. 2015;81(3):386–95.

Ahmad B, Bhattacharya S. Polio eradication in Syria. Lancet Infect Dis. 2014;14(7):547–8.

Meiqari L, Hoetjes M, Baxter L, Lenglet A. Impact of war on child health in northern Syria: the experience of Médecins sans Frontières. Eur J Pediatr. 2018;177(3):371–80.

Initiative GPE. In. Syrian Arab Republic. 2021. p. 191–191.

Alkhalil M, Alaref M, Mkhallalati H, Alzoubi Z, Ekzayez A. An analysis of humanitarian and health aid alignment over a decade (2011–2019) of the Syrian conflict. Confl Health. 2022.

Alkhalil M, Ekzayez A, Rayes D, Abbara A. Inequitable access to aid after the devastating earthquake in Syria. Lancet Glob Health. 2023;0(0).

OCHA. Northwest Syria Humanitarian Readiness and Response Plan. 2020.

Zulfiqar ABBC, Reality C. 2020 [cited 2020 May 2]. Syria: Who’s in control of Idlib? - BBC News. https://www.bbc.co.uk/news/world-45401474

EUAA. 1.3. Anti-government armed groups | European Union Agency for Asylum [Internet]. 2020 [cited 2023 Sep 10]. https://euaa.europa.eu/country-guidance-syria/13-anti-government-armed-groups

Alkhalil M, Alaref M, Ekzayez A, Mkhallalati H, El Achi N, Alzoubi Z, et al. Health aid displacement during a decade of conflict (2011–19) in Syria: an exploratory analysis. BMC Public Health. 2023;23(1):1–16.

Security Council Report. In. Hindsight: the demise of the Syria cross-border aid mechanism, August 2023 Monthly Forecast. Security Council Report; 2023.

WHO, Unicef. Immunization Summary: A statistical reference containing data through 2010. Vol. 2011. 2011.

Ekzayez A, Alkhalil M, Patel P, Bowsher G. Pandemic governance and community mobilization in conflict: a case study of Idlib, Syria. Inoculating cities: Case studies of the Urban response to the COVID-19 pandemic. 2024;61–80.

OECD. Creditor Reporting System (CRS) [Internet]. 2023 [cited 2023 Dec 1]. https://stats.oecd.org/Index.aspx?DataSetCode=CRS1

Kaddar M, Saxenian H, Senouci K, Mohsni E, Sadr-Azodi N. Vaccine procurement in the Middle East and North Africa region: challenges and ways of improving program efficiency and fiscal space. Vaccine. 2019;37(27):3520–8.

World Health Organization. World Health Organization Syrian Arab Republic [Internet]. 2020 [cited 2023 Sep 10]. http://apps.who.int/bookorders

ACU. Annual report 2019. Vol. 5, AIMS Mathematics. 2019.

Siddiqi S, Masud TI, Nishtar S, Peters DH, Sabri B, Bile KM, et al. Framework for assessing governance of the health system in developing countries: gateway to good governance. Health Policy. 2009;90(1):13–25.

Hugh Guan T, Htut HN, Davison CM, Sebastian S, Bartels SA, Aung SM, et al. Implementation of a neonatal hepatitis B immunization program in rural Karenni State, Myanmar: a mixed-methods study. PLoS ONE. 2021;16(12 December):e0261470.

Hugh Guan T, Htut HN, Davison CM, Sebastian S, Bartels SA, Aung SM et al. Implementation of a neonatal hepatitis B immunization program in rural Karenni State, Myanmar: A mixed-methods study. PLoS One [Internet]. 2021 Dec 1 [cited 2023 May 9];16(12 December):e0261470. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0261470

Alaref M, Al-Abdulla O, Al Zoubi Z, Al Khalil M, Ekzayez A. Health system governance assessment in protracted crisis settings: Northwest Syria. Health Res Policy Syst. 2023;21(1):1–13.

Download references

Acknowledgements

The authors acknowledge invaluable contributions of several staff based in Turkey and Syria for their input, access and support. We also wish to acknowledge in particularly contribution from Dr. Mahmoud Daher, then Head of the Gaziantep (Turkey) Office. Furthermore, appreciation is expressed for the contributions of the Assistance Coordination Unit staff, for the documents they made available for this study and their input in the analysis.

This publication is funded through the National Institute for Health Research (NIHR) 131207, Research for Health Systems Strengthening in northern Syria (R4HSSS), using UK aid from the UK Government to support global health research. The views expressed in this publication are those of the author(s) and do not necessarily reflect those of the NIHR or the UK government.

Author information

Ronja Kitlope Baatz and Abdulkarim Ekzayez are equal contributors to this work and designated as co-first authors.

Authors and Affiliations

Deventer Hospital, Deventer, Netherlands

Ronja Kitlope Baatz

Research for Health System Strengthening in northern Syria (R4HSSS), The Centre for Conflict & Health Research (CCHR), King’s College London, Strand, WC2R 2LS, London, UK

Abdulkarim Ekzayez & Preeti Patel

Syria Development Centre (SyriaDev), London, UK

Abdulkarim Ekzayez

Syria Immunisation Group (SIG), Gaziantep, Turkey

Yasser Najib & Mohammad Salem

Syria Public Health Network, London, UK

Munzer Alkhalil

Research for Health System Strengthening in Northern Syria (R4HSSS), UOSSM, Gaziantep, Turkey

Vascular Senior Clinical Fellow, Manchester Royal Infirmary, Manchester, UK

Mohammed Ayman Alshiekh

You can also search for this author in PubMed   Google Scholar

Contributions

The initial framing, literature review, data collection and drafting of the study were carried out by RB and AE. AE contributed to the design, supervision, data collection, data analysis, and multiple rounds of editing. YN contributed to access to data, data collection, and data analysis. MS contributed to access to data and data analysis. PP contributed to analysis and multiple rounds of editing. Mohammed Ayman Alshiekh (MA) contributed to analysis and multiple rounds of editing. Munzer Alkhalil contributed to analysis and multiple rounds of editing. All authors read and approved the final manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Ronja Kitlope Baatz or Abdulkarim Ekzayez .

Ethics declarations

Ethics approval and consent to participate.

Ethical approval was obtained from the Institutional Review Board of King’s College London, under the approval number MRA-22/23-34048. Informed consent was obtained from all participants involved in the study. Participants were provided with detailed information regarding the study’s objectives, procedures, potential risks, and benefits. They were assured of their right to withdraw from the study at any time without any repercussions. All data collected were anonymised to ensure the confidentiality and privacy of the participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Baatz, R.K., Ekzayez, A., Najib, Y. et al. Vaccination governance in protracted conflict settings: the case of northwest Syria. BMC Health Serv Res 24 , 1056 (2024). https://doi.org/10.1186/s12913-024-11413-1

Download citation

Received : 16 January 2024

Accepted : 07 August 2024

Published : 12 September 2024

DOI : https://doi.org/10.1186/s12913-024-11413-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Immunisation
• Vaccination
• Health governance
• Conflict setting
• Localisation

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Research on energy scheduling optimization strategy with compressed air energy storage.

1. Introduction

• A hierarchical scheduling model for CAES systems is constructed and transformed into a Markov decision process. The coordinated scheduling problem of wind farms and energy storage is balanced using the DRL algorithm.
• In order to achieve the efficient learning of intelligences, a deterministic policy gradient-based DDPG algorithm is used. The algorithm effectively improves the learning ability of the intelligent body in continuous action space to adapt to its complex environment in the power system.
• This paper introduces a combined algorithm that merges the NEAT algorithm with the DDPG algorithm to enhance the effectiveness of the algorithm. By utilizing the adaptive network structure of NEAT, the combined approach improves adaptability in complex environments and efficiency in renewable energy utilization.

2. Integrated Energy Framework

2.1. aa-caes structure, 2.2. hierarchical energy optimization strategy, 2.3. drl description, 3. cooperative control framework of source–storage–grid system, 3.1. wind farm model, 3.2. aa-caes model.

• It is assumed that air is an ideal gas and satisfies the ideal gas equation of state;
• The reservoir is modeled using an isothermal constant volume model, where the temperature of the air in the reservoir is equal to the ambient temperature, and the volume of the reservoir is exploded to be constant;
• The compressor and expander are modeled adiabatically;
• Heat loss from the heat storage tank and heat loss from the heat exchange process are excluded.

3.3. Energy Scheduling Model

3.4. markov model, 3.4.1. state space s t, 3.4.2. action space a t, 3.4.3. reward r t, 4. deep reinforcement learning algorithms, 4.1. actor–critic algorithm, 4.2. deep deterministic policy gradient, 4.3. neuroevolution of augmenting topologies.

5. Case Studies and Results

6. discussion, 7. conclusions.

• Deep reinforcement learning algorithms can play an important role in the intelligent scheduling of power systems containing AA-CAES.
• The effectiveness of the algorithm is verified by analyzing the simulation results. The algorithm realizes the cooperative scheduling in the source–storage network and ensures the safe operation of the power grid. Even in the case of unstable wind power generation, the system operation can be made smoother by scheduling AA-CAES.
• The experimental results also show the better performance of the improved DDPG algorithm with DDPG-NEAT compared to the other two DRL algorithms. The comparison of the power scheduling data of the three algorithms shows that the DDPG-NEAT algorithm can perform the scheduling task better and improve the energy utilization efficiency.

Author Contributions

Institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest, abbreviations.

• Karmaker, A.K.; Rahman, M.M.; Hossain, M.A.; Ahmed, M.R. Exploration and Corrective Measures of Greenhouse Gas Emission from Fossil Fuel Power Stations for Bangladesh. J. Clean. Prod. 2020 , 244 , 118645. [ Google Scholar ] [ CrossRef ]
• Xie, H.; Yu, Y.; Wang, W.; Liu, Y. The Substitutability of Non-Fossil Energy, Potential Carbon Emission Reduction and Energy Shadow Prices in China. Energy Policy 2017 , 107 , 63–71. [ Google Scholar ] [ CrossRef ]
• Ming, Z.; Song, X.; Mingjuan, M.; Xiaoli, Z. New Energy Bases and Sustainable Development in China: A Review. Renew. Sustain. Energy Rev. 2013 , 20 , 169–185. [ Google Scholar ] [ CrossRef ]
• Argyrou, M.C.; Christodoulides, P.; Kalogirou, S.A. Energy Storage for Electricity Generation and Related Processes: Technologies Appraisal and Grid Scale Applications. Renew. Sustain. Energy Rev. 2018 , 94 , 804–821. [ Google Scholar ] [ CrossRef ]
• Michaelides, E.E. Thermodynamics, Energy Dissipation, and Figures of Merit of Energy Storage Systems—A Critical Review. Energies 2021 , 14 , 6121. [ Google Scholar ] [ CrossRef ]
• Huang, Y.; Keatley, P.; Chen, H.S.; Zhang, X.J.; Rolfe, A.; Hewitt, N.J. Techno-Economic Study of Compressed Air Energy Storage Systems for the Grid Integration of Wind Power. Int. J. Energy Res. 2018 , 42 , 559–569. [ Google Scholar ] [ CrossRef ]
• Zhang, X.; Li, Y.; Gao, Z.; Chen, S.; Xu, Y.; Chen, H. Overview of Dynamic Operation Strategies for Advanced Compressed Air Energy Storage. J. Energy Storage 2023 , 66 , 107408. [ Google Scholar ] [ CrossRef ]
• Xu, W.; Zhang, W.; Hu, Y.; Yin, J.; Wang, J. Multi Energy Flow Optimal Scheduling Model of Advanced Adiabatic Compressed Air Energy Storage. Trans. China Electrotech. Soc. 2022 , 37 , 5944–5955. [ Google Scholar ]
• Wang, X.; Zhou, J.; Qin, B.; Guo, L. Coordinated Power Smoothing Control Strategy of Multi-Wind Turbines and Energy Storage Systems in Wind Farm Based on MADRL. IEEE Trans. Sustain. Energy 2024 , 15 , 368–380. [ Google Scholar ] [ CrossRef ]
• Zhou, X.; Wang, J.; Wang, X.; Chen, S. Optimal Dispatch of Integrated Energy System Based on Deep Reinforcement Learning. Energy Rep. 2023 , 9 , 373–378. [ Google Scholar ] [ CrossRef ]
• Sheng, Y.; Yang, J.; Ma, S.; Wang, Y.; Li, H. Research on Optimal Dispatching of Integrated Energy System Based on Demand-supply Interaction. Power Demand Side Manag. 2019 , 21 , 48–54. [ Google Scholar ]
• Zhuo, Y.; Chen, J.; Zhu, J.; Ye, H.; Wang, Z. Optimal Scheduling of Park-level Integrated Energy Systems Based on Improved Approximate Dynamic Programming. High Volt. Eng. 2022 , 51 , 2597–2606. [ Google Scholar ]
• Yan, K.; Zhang, J.; He, Y.; Zhang, Y.; Liu, Y.; Li, X. The Optimal Dispatching of Mixed Integer Programming Based on Opportunity Constraint of Microgrid. Electr. Power Sci. Eng. 2021 , 37 , 17–24. [ Google Scholar ]
• Li, Y.; Yao, F.; Zhang, S.; Liu, Y.; Miao, S. An Optimal Dispatch Model of Adiabatic Compressed Air Energy Storage System Considering Its Temperature Dynamic Behavior for Combined Cooling, Heating and Power Microgrid Dispatch. High Volt. Eng. 2022 , 51 , 104366. [ Google Scholar ] [ CrossRef ]
• Lin, J.; Liu, Y.; Chen, B.; Chen, R.; Chen, Y.; Dai, X. Micro-grid Energy Optimization Dispatch of Combined Cold and Heat Power Supply Based on Stochastic Chance-constrained Programming. Electr. Meas. Instrum. 2019 , 56 , 85–90. [ Google Scholar ]
• Li, Y.; Miao, S.; Yin, B.; Han, J.; Zhang, S.; Wang, J.; Luo, X. Combined Heat and Power Dispatch Considering Advanced Adiabatic Compressed Air Energy Storage for Wind Power Accommodation. Energy Convers. Manag. 2019 , 200 , 112091. [ Google Scholar ] [ CrossRef ]
• Naidji, I.; Ben Smida, M.; Khalgui, M.; Bachir, A.; Li, Z.; Wu, N. Efficient Allocation Strategy of Energy Storage Systems in Power Grids Considering Contingencies. IEEE Access 2019 , 7 , 186378–186392. [ Google Scholar ] [ CrossRef ]
• Men, J. Bi-Level Optimal Scheduling Strategy of Integrated Energy System Considering Adiabatic Compressed Air Energy Storage and Integrated Demand Response. J. Electr. Eng. Technol. 2024 , 19 , 97–111. [ Google Scholar ] [ CrossRef ]
• Long, F.; Jin, B.; Yu, Z.; Xu, H.; Wang, J.; Bhola, J.; Shavkatovich, S.N. Research on Multi-Objective Optimization of Smart Grid Based on Particle Swarm Optimization. Electrica 2023 , 23 , 222–230. [ Google Scholar ] [ CrossRef ]
• Torkan, R.; Ilinca, A.; Ghorbanzadeh, M. A Genetic Algorithm Optimization Approach for Smart Energy Management of Microgrid. Renew. Energy 2022 , 197 , 852–863. [ Google Scholar ] [ CrossRef ]
• Fathy, A.; Alanazi, T.M.; Rezk, H.; Yousri, D. Optimal Energy Management of Micro-Grid Using Sparrow Search Algorithm. Energy Rep. 2022 , 8 , 758–773. [ Google Scholar ] [ CrossRef ]
• Wang, Y.; Zheng, Y.; Xue, H.; Mi, Y. Optimal Dispatch of Mobile Energy Storage for Peak Load Shifting Based on Enhanced Firework Algorithm. Autom. Electr. Power Syst. 2021 , 45 , 48–56. [ Google Scholar ]
• Ma, Y.; Zhou, J.; Dong, X.; Wang, H.; Zhang, W.; Tan, Z. Multi-objective Optimal Scheduling Model for Multi-energy System Considering Uncertainty and Hybrid Energy Storage Devices. J. Electr. Power Sci. Technol. 2022 , 37 , 19–32. [ Google Scholar ]
• Lu, L.; Chu, G.; Zhang, T.; Yang, Z. Optimal Configuration of Energy Storage in a Microgrid Based on Improved Multi-objective Particle Swarm Optimization. Power Syst. Prot. Control 2020 , 48 , 116–124. [ Google Scholar ]
• Liu, X.; Xie, S.; Tian, J.; Wang, P. OTwo-Stage Scheduling Strategy for Integrated Energy Systems Considering Renewable Energy Consumption. IEEE Access 2022 , 10 , 83336–83349. [ Google Scholar ] [ CrossRef ]
• Xu, Z.; Han, G.; Liu, L.; Martínez-García, M.; Wang, Z. Multi-Energy Scheduling of an Industrial Integrated Energy System by Reinforcement Learning-Based Differential Evolution. IEEE Trans. Green Commun. Netw. 2021 , 5 , 1077–1090. [ Google Scholar ] [ CrossRef ]
• Li, Y.; Zhang, Z.; Meng, K.; Wei, H. Energy Optimal Dispatch of Microgrid Based on Improved Depth Deterministic Strategy Gradient Algorithm. Electron. Meas. Technol. 2023 , 46 , 73–80. [ Google Scholar ]
• Wang, B.; Li, Y.; Ming, W.; Wang, S. Deep Reinforcement Learning Method for Demand Response Management of Interruptible Load. IEEE Trans. Smart Grid 2020 , 11 , 3146–3155. [ Google Scholar ] [ CrossRef ]
• Chen, L.; Wu, J.; Tang, H.; Jin, F.; Wang, Y. A Q-Learning Based Optimization Method of Energy Management for Peak Load Control of Residential Areas with CCHP Systems. Electr. Power Syst. Res. 2023 , 214 , 108895. [ Google Scholar ]
• Luo, J.; Zhang, W.; Wang, H.; Wei, W.; He, J. Research on Data-Driven Optimal Scheduling of Power System. Energies 2023 , 16 , 2926. [ Google Scholar ] [ CrossRef ]
• Chang, Y.; Liu, S.; Wang, L.; Cong, W.; Zhang, Z.; Qi, S. Research on Low-Carbon Economic Operation Strategy of Renewable Energy-Pumped Storage Combined System. Math. Probl. Eng. 2022 , 13 , 9202625. [ Google Scholar ] [ CrossRef ]
• Bai, Y.; Chen, S.; Zhang, J.; Xu, J.; Gao, T.; Wang, X.; Gao, D.W. An Adaptive Active Power Rolling Dispatch Strategy for High Proportion of Renewable Energy Based on Distributed Deep Reinforcement Learning. Appl. Energy 2023 , 330 , 120294. [ Google Scholar ] [ CrossRef ]
• Dolatabadi, A.; Abdeltawab, H.; Mohamed, Y.A.-R.I. Deep Reinforcement Learning-Based Self-Scheduling Strategy for a CAES-PV System Using Accurate Sky Images-Based Forecasting. IEEE Trans. Power Syst. 2023 , 38 , 1608–1618. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Wang, R.; Zhang, Z.; Meng, K.; Lei, P.; Wang, K.; Yang, W.; Liu, Y.; Lin, Z. Research on Energy Scheduling Optimization Strategy with Compressed Air Energy Storage. Sustainability 2024 , 16 , 8008. https://doi.org/10.3390/su16188008

Wang R, Zhang Z, Meng K, Lei P, Wang K, Yang W, Liu Y, Lin Z. Research on Energy Scheduling Optimization Strategy with Compressed Air Energy Storage. Sustainability . 2024; 16(18):8008. https://doi.org/10.3390/su16188008

Wang, Rui, Zhanqiang Zhang, Keqilao Meng, Pengbing Lei, Kuo Wang, Wenlu Yang, Yong Liu, and Zhihua Lin. 2024. "Research on Energy Scheduling Optimization Strategy with Compressed Air Energy Storage" Sustainability 16, no. 18: 8008. https://doi.org/10.3390/su16188008

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

A Look Toward the Future: Decision Support Systems Research is Alive and Well

• Journal of the Association for Information Systems 13(5):315-340
• 13(5):315-340

• Penske Logistics

• Auburn University

• Louisiana Tech University

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Marifel Grace Capili-Kummer
• BMC MED INFORM DECIS
• Geir Thore Berge

• Tor Oddbjørn Tveit

• ARTIF INTELL MED

• Felix Weber
• JINGRAN WANG

• Sarah E. Walsh

• Yung-Hsin Wang
• Kun Chang Lee
• Soon Jae Kwon
• George P. Huber
• J OPER RES SOC
• Howard Carter
• ROBERT H. BONCZEK

• Daniel Saunders
• Wanda J. Orlikowski
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up