Toward Transparent and Safe Clinical AI: A Framework for Explainable Large Language Models in Medicine

Abstract

Large Language Models (LLMs), such as GPT-4, are being increasingly adopted in clinical settings to support tasks like diagnosis, patient communication, and medical summarization. However, their black-box nature raises ethical and safety concerns, particularly regarding hallucinations, omissions, and lack of interpretability. This study aims to evaluate the performance of LLMs in clinical tasks and propose a transparent framework that integrates explainability and risk assessment tools. We benchmarked LLM performance using the RJUA-SP dataset, focusing on diagnostic reasoning, therapy recommendation, and multi-turn dialogues. The CREOLA framework was applied to classify hallucination and omission likelihoods in clinical outputs, and post-hoc explainability methods especially AMPLIFY were used to assess their impact on model interpretability and trust. Results show that GPT-4 achieves 63.63% accuracy in single-turn diagnostic QA and 18.18% in therapy recommendation, with reasoning performance peaking at 20.15% and multi-turn dialogue completeness below 16%. CREOLA identified high-risk errors in over 90% of outputs, underscoring the need for human oversight. Integrating AMPLIFY improved reasoning accuracy by nearly 10 percentage points and enhanced clinician trust. These findings suggest that while LLMs offer valuable clinical support, they must be paired with transparent mechanisms to ensure safe deployment. This research contributes a multi-layered framework combining benchmarking, risk evaluation, and explainability strategies for responsible LLM use in high-risk healthcare domains.