AI virtual subject matter expert for healthcare org
Executive summary
We partnered with a national healthcare organization to develop an AI-based virtual subject matter expert (SME) capable of answering internal and public-facing queries. The project addressed the complex data structures and regulatory demands unique to healthcare by creating an intelligent, modular system that integrates information from multiple, siloed systems while maintaining explainability. We developed a RAG-based virtual SME that achieved 98% accuracy on frequently asked questions.
Clean up the documents
Our data preprocessing services prepared a diverse range of document types to be ingested by AI and standardized info.
Build the KB
We ingested vast numbers of documents into a knowledge base able to be queried by AI for instant answers.
Test and refine
Once the knowledge base was built, we refined the AI with fine-tuning and human guidance to get high accuracy.
Background
Healthcare organizations are bound by stringent regulatory and data integrity requirements and must reconcile data inconsistencies across multiple internal systems. Traditional approaches often resulted in conflicting information. The client's need for unified knowledge access drove the development of a virtual subject matter expert (SME) for fast and accurate responses to questions about regulatory compliance, company policy, and procedures. Components included modular data integration, explainable AI, and a feedback loop for continuous improvement.
Objectives
The project aimed to:
- Develop an AI-based virtual SME that could respond accurately to internal and external queries
- Integrate diverse datasets and resolve data inconsistencies across the organization's systems
- Maintain high explainability standards for transparency and trust (essential in the healthcare sector)
- Utilize human-in-the-loop oversight to fine-tune responses and align AI outputs with industry requirements
Methodology
Leveraging the Cognitive Hive AI (CHAI) framework, the virtual SME was designed as a modular system. CHAI’s approach enabled the system to integrate structured data from siloed systems with the flexibility for future expansion. The system included:
- Data integration with knowledge graphs and vector databases: By using knowledge graphs, the SME created a structured understanding of the client's policies and guidelines. Meanwhile, vector databases processed unstructured text for similarity-based querying.
- Human-in-the-loop fine-tuning: Continuous HITL feedback refined the model’s responses. Subject matter experts reviewed and adjusted outputs, enhancing accuracy and aligning answers with regulatory standards.
- Explainability: Each response was backed by a traceable decision path, allowing users to understand how conclusions were reached, which is particularly critical in healthcare.
Results
The virtual SME achieved the following outcomes:
- High accuracy: Through HITL fine-tuning, the system achieved over 98% accuracy on common queries
- Cross-system data consistency: By integrating data across previously siloed systems, the AI significantly reduced inconsistencies in information retrieval
- Fast answers: Queries that previously required extensive manual cross-checking were now answered in seconds, supporting faster decision-making and reducing the burden on human resources
Challenges and solutions
- Data inconsistencies: The project faced challenges with conflicting data across systems. Talbot West addressed this by implementing conflict resolution protocols in the AI, which flagged and reconciled inconsistencies using HITL oversight.
- Transparency requirements: Given the regulatory burden on healthcare organizations, maintaining a transparent and explainable system was essential. The CHAI architecture facilitated traceable decision paths so that users could understand and trust AI recommendations.
Lessons learned
- Modular, explainable AI is crucial: The success of the CHAI-based modular approach underscored the value of explainability, especially for sensitive data environments like healthcare.
- Human-AI collaboration enhances accuracy: HITL fine-tuning ensured that complex cases received the oversight necessary to improve response quality and compliance alignment over time.
Conclusion
Talbot West’s AI solution exemplifies the integration of modular, explainable AI to meet regulatory and operational needs in healthcare. By streamlining data access, improving accuracy, and reducing response times, the virtual SME supports our client's commitment to compliant, patient-centered operations.
Ready to take action?
AI won't implement itself. If you want to reap the rewards, it's time to get down to brass tacks with a feasibility study. Contact us to discuss.
