Generative AI model effectively highlights social determinants of health in doctor notes

Generative AI model effectively highlights social determinants of health in doctor notes

This article has been reviewed in accordance with Science X’s editorial procedures and policies. Editors have emphasized the following attributes while ensuring the credibility of the content:

Fact checked

peer-reviewed publications

Trusted source

Proofreading


An infographic summarizing new research led by Danielle Bitterman, M.D., using large language models to identify social determinants of health from doctor visit notes.Image source: Massachusetts Brigham General Hospital

× closure


An infographic summarizing new research led by Danielle Bitterman, M.D., using large language models to identify social determinants of health from doctor visit notes.Image source: Massachusetts Brigham General Hospital

Where we live and work, our age, and the conditions in which we grow up all influence our health and contribute to disparities, but these factors can be difficult for clinicians and researchers to capture and address.

A new study by researchers at Massachusetts General Hospital Brigham shows that large language models (LLMs), a type of generative artificial intelligence (AI), can be trained to automatically extract health-related information from clinicians’ notes. Social determinants (SDoH) information, which can enhance efforts to identify patients who may benefit from resource support.

The research results were published in npj digital medicine The study showed that the fine-tuned model could identify 93.8% of patients with adverse SDoH, whereas official diagnostic codes contained this information in only 2% of cases. These specialized models are less prone to bias than general-purpose models such as GPT-4.

“Our goal was to identify patients who could benefit from resources and social work support and to draw attention to the impact of social factors on health outcomes,” said corresponding author Danielle Bittman, MD, a faculty member in the School of Artificial Intelligence in Medicine explain. (AIM) program at Massachusetts General Hospital and physicians in the Department of Radiation Oncology at Brigham and Women’s Hospital.

“Algorithms that can pass major medical exams get a lot of attention, but that’s not what doctors need in the clinic to help take better care of patients every day. Algorithms that can notice things that doctors may miss in the ever-increasing amount of data Algorithmic medical records will be more clinically relevant and therefore more powerful for improving health.”

Health disparities are broadly related to SDoH and include employment, housing, and other non-medical conditions that impact health care. For example, a cancer patient’s distance from a major medical center or the support they receive from their partner can have a significant impact on treatment outcomes. Although clinicians may summarize relevant SDoH in visit notes, this important information is rarely systematically organized in electronic health records (EHRs).

To create a LM capable of extracting SDoH information, the researchers manually reviewed 800 clinician notes from 770 cancer patients who received radiation therapy in the Department of Radiation Oncology at Brigham and Women’s Hospital. They marked sentences that addressed one or more of six predetermined SDoHs: employment status, housing, transportation, parental status (if the patient has children under 18 years of age), interpersonal relationships, and the presence of social support.

Using this “annotated” dataset, the researchers trained an existing LM to identify references to SDoH in clinician notes. They tested their model using 400 clinical records from patients receiving immunotherapy at Dana-Farber Cancer Institute and patients in the intensive care unit at Beth Israel Deaconess Medical Center.

The researchers found that fine-tuned LMs, especially the Flan-T5 LM, can consistently identify rare SDoH citations in clinicians’ notes. The “learning ability” of these models was limited by the rarity of SDoH documents in the training set, with the researchers finding that only 3% of sentences in clinician notes contained mentions of SDoH.

To address this issue, the researchers used another LM ChatGPT to generate an additional 900 synthetic examples of SDoH sentences, which can be used as additional training data sets.

A major criticism of generative AI models in healthcare is that they may perpetuate bias and widen health disparities. The researchers found that their fine-tuned LM was less likely than OpenAI’s GPT-4 (general-purpose LM) to change its decision on SDoH based on an individual’s race/ethnicity and gender.

Researchers say it’s difficult to understand how biases form and deconstruct in humans and computer models. Understanding the origins of algorithmic bias is an ongoing effort among researchers.

“If we don’t monitor algorithmic biases as we develop and implement large language models, we could make existing health disparities much worse than they currently are,” Bittman said. “This study suggests that fine-tuning LMs may be a strategy to reduce algorithmic bias, but more research is needed in this area.”

More information:
Large-scale language models for identifying social determinants of health in electronic health records, npj digital medicine (2024). DOI: 10.1038/s41746-023-00970-0

Journal information:
npj digital medicine

Leave a Comment

Your email address will not be published. Required fields are marked *