Using computers to help make clinical diagnoses has been artificial intelligence's goal since its inception. The adoption of electronic health record (EHR) systems by hospitals in the US has resulted in an unprecedented amount of digital data associated with patient encounters. Such EHRs data can be rich resources for computer-assisted clinical decision support systems (CDSS). CDSS is typically formulated as an NLP system that extracts diagnosis and procedure codes and a predictive modeling system that predicts patient future outcomes.
Extracting medical codes is difficult as some of them are not specifically mentioned in the medical note. We first introduced a challenging evaluation benchmark of unmentioned codes. To improve the coding accuracy, we proposed a clinical language model pretrained with a customized objective: to generate patient diagnosis given pateint basic info such as symptoms. Experiments showed that pretraining significantly helps this challenging benchmark.
Extraction is also challenging due to a high-dimensional space of multi-label assignment (tens of thousands of medical codes) and the long-tail challenge: only a few codes (common diseases) are frequently assigned while most codes (rare diseases) are infrequently assigned. We addressed the long-tail challenge by adapting a prompt-based fine-tuning technique with label semantics, which has been shown to be effective under few-shot setting. To further enhance the performance in medical domain, we proposed a knowledge-enhanced language model by injecting three domain-specific knowledge: hierarchy, synonym, and abbreviation with additional pretraining using contrastive learning. Experiments showed that our approach achieved the state-of-the-art performance on both common and rare diseases.
In the previous section, we saw the benefit of pretraining a language model, could this benefit migrate to predictive model? If LLMs are tools that take a sequence of words and predict the next one, what if we could make a tool that takes a sequence of medical codes and predicts the next one? To answer this question, we proposed TransformEHR, an innovative denoising sequence-to-sequence transformer model that was pretrained on 6.5 million patients' medical codes to predict complete codes of a visit. Results showed that TransformEHR outperformed SOTA BERT models on both common and uncommon code predictions. TransformEHR, with a high predictive value for identifying self-harm risk in PTSD patients, exceeded the threshold for cost-effectiveness, indicating its potential for clinical use in suicide prevention.
Finally, I will conclude this talk by discussing future work that combines the two methods, with the hope of providing further textual explanations to support decisions.
Advisor: Hong Yu