A Deep-Learning Computer Model for Predicting Pediatric Patient Health Care Risks and Costs

A deep-learning computer model was better at predicting the health care risk of 112,000 children in a Medicaid accountable care organization (ACO) than traditional risk prediction models, researchers at Nationwide Children’s Hospital have shown.

“Using this approach, we were able to more accurately identify high-cost patients compared to other popular predictive models,” says Deborah Lin, PhD, MPH, a senior project scientist in Information Technology Research & Development at the Abigail Wexner Research Institute at Nationwide Children’s, and a leader of the research.

After the researchers fed the deep-learning model (DL) 2014 and 2015 patient claims data from Partners For Kids, Nationwide Children’s ACO, the model learned from the data in just a couple of hours. The investigators then compared DL’s performance to the DxCG Intelligence model, which some consider the “gold standard” of predictive models.

In minutes, the DL identified the top 1% of users—medically complex patients who are the most at risk for hospitalization. Their actual next year’s health-care cost was $5 million higher compared to the top 1% identified by DxCG.

“Analysts typically report model performance in metrics, such as AUC (area under the curve), that seem to have small differences and are rather abstract. In our case, the AUC for DL was 3% higher than DxCG,” Dr. Lin says. “The actual healthcare utilization cost of the high-risk patient population identified by DL demonstrates the impact of this improvement in model performance.”.

“That’s important because, like all ACOs, we want to know which members are most at risk, enabling us to develop programs and allocate resources for kids who need it the most,” says Brad Stamm, MBA, data manager at Partners For Kids, with whom the researchers consulted. “That’s where this model surpassed the commercially available products – by more accurately stratifying children by risk of hospitalization.”

The DL required significantly less time and domain expertise to develop compared to traditional commercial prediction models such as DxG and Clinical Classifications Software (CCS), which took years for subject-matter experts to develop and refine, says Simon Lin, MD, MBA, chief research information officer at Nationwide Children’s and a leader of the investigation.

A study of the DL is in the American Journal of Managed Care, and a technical paper is posted on arXiv.

Deep learning is a form of artificial intelligence, or AI, using digital neural networks that mimic aspects of the human brain in order to solve complex problems. For this risk prediction model, the researchers developed a novel architecture designed to learn from claims.

“In daily life, artificial intelligence is applied in a lot of places. When you go shopping on Amazon, AI engines are behind the recommendations you receive – the machine knows your preferences,” Simon Lin says. “But, we believe this is the first application of artificial intelligence to risk stratification by looking at claims data.”

The claims data includes patients’ sex, age and zip code; medical codes for their diagnosis, procedures and medications; their corresponding medical costs; and the visit date, type of service, location and provider.

The researchers employed an unsupervised learning approach called “Skip-Gram” to allow the neural networks to learn from the visit information and medical codes. The short time the model took to learn from the dataset and predict risk indicates that as new information is added, the DL will provide a quicker turnaround compared to traditional risk modeling, Deborah Lin says.

Another important difference, according to Stamm, is the traditional models are based on adult patients while this DL is based on children.

“I was surprised that the model developed by Simon Lin’s team surpassed the other models by such a large amount. I can’t speak to the science behind why it was so successful. However, it’s my experience that most population health solutions are specifically designed for adults managed by commercial health plans. Those solutions intentionally bypass both pediatric and Medicaid populations. Our children seem to fall into a “black hole” in the market and it’s exciting to see a solution specifically designed for this demographic”, Stamm says. “We’re not applying it at Partners For Kids — yet. There’s still work to be done. But the model is promising, and I see no reason it couldn’t be used by any ACO.”

The work resulted from a regional collaboration between Nationwide Children’s Hospital, The Ohio State University, and Ohio University. Xianlong Zeng, a graduate student in computer science, led the development of the deep learning model. The researchers are now experimenting with other parameters, such as the income level of patient families and social determinants of health, to determine if they can further improve the DL’s predictive capabilities.

References:

1. Lin E-JD, Hefner JL, Zeng X, Moosavinasab S, Huber T, Klima J, Liu C, Lin SM. A deep learning model for pediatric patient risk stratification. American Journal of Managed Care.2019 Oct 1;25(10):e310-e315
2. Zeng X, Moosanivinasab S, Lin E-JD, Bunescu R, Liu C, Lin S. Distributed representation of patients and its use for medical risk adjustment. arXiv:1909.07157