Development and Evaluation of an Artificial Intelligence–Assisted Decision Support System for Public Health Emergency Classification and Escalation in Kenya

ResearchMark Nanyingi, Eric Osoro, Geoffrey H. Siwo, Isaac Ngere, Samuel Kadivane, James Magige, Joseph Kamau, Shreya Jain, Bryan O. Nyawanda, Joseph Njoroge, Ian Njeru, Kadondi Kasera, Victoria Kanana, Kamene Kimenye4 min read

Development and Evaluation of an Artificial Intelligence–Assisted Decision Support System for Public Health Emergency Classification and Escalation in Kenya

A new preprint manuscript describes the development and evaluation of the DMT-PHE AI Agent, an artificial intelligence–assisted decision support system designed to operationalize Kenya's nationally validated Decision-Making Tool for Public Health Emergencies. The pilot evaluation demonstrated high concordance with expert assessments, strong usability, and no major safety concerns.

Background

Timely assessment, classification, and escalation of public health events are essential for effective outbreak response, yet decision-making after event detection remains challenging because of fragmented guidance and variable interpretation of escalation criteria. To strengthen public health emergency management, Kenya developed the Decision-Making Tool for Public Health Emergencies (DMT-PHE), a framework for event assessment, classification, notification, and escalation. An artificial intelligence (AI)-enabled version, the DMT-PHE AI Agent, was subsequently developed to operationalize the framework through decision support. This study describes the development of the DMT-PHE AI Agent and evaluates its performance, usability, safety, and user acceptability.

Methods

The DMT-PHE AI Agent was developed using a retrieval-augmented generation (RAG) architecture supported by a curated knowledge base derived from the validated DMT-PHE framework and related public health guidance. The AI Agent employs a retrieval-augmented generation architecture that combines the reasoning capabilities of a large language model (Anthropic Claude Sonnet 4.5) with a curated, domain-specific knowledge base. Rather than relying solely on information embedded within the foundation model, the retrieval component identifies relevant content from authoritative reference documents before generating responses. This approach enables recommendations to be grounded in validated public health guidance, reduces dependence on parametric model knowledge, and improves the transparency and contextual relevance of generated outputs.

The knowledge base comprised a version-controlled collection of reference materials derived from the validated DMT-PHE framework together with supporting national and international guidance. These resources included the International Health Regulations (2005), Kenya public health emergency management policies, Public Health Emergency Operations Centre (PHEOC) procedures, disease-specific technical guidance, One Health frameworks, and the DMT-PHE Trigger Cards that provide standardized operational guidance for priority public health hazards.

When a user submitted an event description, the AI Agent interpreted the information, retrieved the most relevant guidance documents, and synthesized the retrieved evidence to generate structured recommendations. Outputs included event classification, escalation level, notification pathways, priority response actions, coordination mechanisms, and supporting references where available. Recommendations were generated using a hierarchical decision logic that prioritized the DMT-PHE framework, followed by nationally approved public health emergency management guidance, International Health Regulations (2005) requirements, and One Health guidance where applicable.

Where multiple knowledge sources addressed the same decision point, recommendations were resolved using a predefined hierarchy in which the validated DMT-PHE framework served as the primary authority. National public health emergency management policies were given precedence over international guidance where operational procedures differed, while the International Health Regulations (2005) and One Health guidance provided complementary decision support for notification, multisectoral coordination, and risk assessment. This approach ensured consistency with Kenya's public health emergency management architecture while maintaining alignment with international best practices.

The DMT-PHE AI Agent was intentionally designed as a human-in-the-loop decision-support system rather than an autonomous decision-maker.

A simulation-based pilot evaluation was conducted among 11 public health professionals who independently assessed three standardized outbreak scenarios. Participants were purposively recruited from the Kenya National Public Health Institute (KNPHI) and selected county surveillance teams to ensure representation of professionals routinely involved in public health emergency preparedness and response. Eligible participants included epidemiologists, disease surveillance officers, public health emergency management personnel, and other technical staff with responsibilities related to event detection, risk assessment, outbreak investigation, or emergency coordination.

Three standardized outbreak scenarios were developed to evaluate the performance of the DMT-PHE AI Agent across distinct public health emergency contexts. Scenario selection was informed by the hazard categories and included Public Health Events of Initially Unknown Etiology (PHEIUE), Rift Valley fever, and Mpox.

AI-generated recommendations were compared with expert-defined gold standards. Outcomes included concordance, response-action coverage, citation performance, safety, usability, and user acceptability.

Results

Thirty-three scenario evaluations were completed. The AI Agent achieved an overall weighted concordance score of 0.924, with exact agreement of 90.9% for Public Health Events of Initially Unknown Etiology, 81.8% for Rift Valley fever, and 90.9% for Mpox. Citation support was provided in 78.8% of interactions, with no incorrect citations or major safety concerns identified. The mean System Usability Scale score was 85.2, while participants reported high trust (4.27/5), contextual relevance (4.55/5), and perceived time savings (4.82/5).

Conclusions

The DMT-PHE AI Agent demonstrated that a nationally validated public health emergency decision framework can be successfully translated into an AI-enabled decision-support system. These findings provide early evidence that AI can augment public health emergency decision-making by delivering structured, transparent, and context-specific recommendations while maintaining human oversight, offering a practical model for operationalizing national public health guidance.

Share

Comments

Loading comments…

Leave a comment

← All news and research