Predictive Analytics in Child Welfare — Benefits and Challenges
By Kate Jackson
Social Work Today
Vol. 18 No. 2 P. 10

Predictive risk modeling tools have the potential to help social workers and other child welfare workers anticipate and prevent child abuse and neglect, but refinement is needed.

Each day in the United States, according to a 2016 report of the Commission to Eliminate Child Abuse and Neglect Fatalities, "four to eight children die from abuse or neglect at the hands of their parents or caretakers." Social workers and other child welfare workers are largely tasked with the prevention of these tragedies; around the country, they staff child protection hotlines and struggle to field calls alerting them to children at risk of neglect or abuse and to determine whether suspicions or allegations warrant further investigation. They must determine quickly and accurately, and typically based on limited information, whether the assertions are valid—which calls can be screened out, requiring no further action, and which must be screened in, generating an investigation. In every instance, they're aware that the cost of an error could be a child's life.

In a process that's both time consuming and potentially inefficient, the caseworkers must manually cull information from disparate databases and multiple additional sources, data that may or may not be predictive. Children at risk easily can be overlooked. But technology has contributed what many experts believe is a better way.

Predictive risk modeling (PRM), or predictive analytics, has emerged as a strategy with which social workers and other child welfare workers might better infer which referrals require investigation and which do not. In addition to potentially increasing the accuracy of their decision-making processes, predictive analytics tools can reduce the time it takes to determine the need for investigation—a potentially lifesaving combination.

PRM draws on vast data mines to identify patterns—in the case of child welfare, patterns of family characteristics or behaviors and their associated outcomes. It can then be used to compare information about an individual or a family against those patterns revealed in historical data to help predict an outcome and pinpoint those at greatest risk. The data used might include, for example, criminal history, welfare status, hospital reports, alcohol and substance use, school records, behavioral health issues, and past allegations of abuse. The tools use statistical processes to crunch the data and produce a score for a given situation; low scores indicate less probability of harm, and the higher the score, the more likely the future will bring a negative outcome.

Although PRM tools have been employed for decades in insurance, finance, medicine, and other fields, their use in child welfare is nascent, due in part to concerns about the ways in which sensitive personal data might be used. Nevertheless, tools for PRM for social services have been created by a variety of developers, from IBM to SAS, and they are being explored, or have been implemented, in a variety of locations, including Los Angeles County; Hillsborough County, Florida; Douglas County, Colorado; and, most notably, Allegheny County, Pennsylvania.

Leading the Way
A pioneer in the use of PRM in child welfare is Allegheny County, which encompasses Pittsburgh. In August 2016, the county launched the Allegheny Family Screening Tool (AFST), which lets screeners extract information from an existing deep data warehouse to produce a score from one to 20. Several years ago, when the county was exploring the use of predictive analytics for its child welfare hotline, it partnered with researchers led by Rhema Vaithianathan, PhD, a professor of economics and codirector of the Centre for Social Data Analytics at Auckland University of Technology in New Zealand, and Emily Putnam-Hornstein, PhD, MSW, director of the Children's Data Network at the University of Southern California Suzanne Dworak-Peck School of Social Work. Previously, Vaithianathan and Putnam-Hornstein led a research team commissioned by the Ministry of Social Development of New Zealand to explore PRM as a strategy for early intervention to help prevent adverse outcomes for children younger than 5 years old. "That project, which delivered an accurate prototype for an 'at-birth' predictive risk model, marked a turning point for the field of PRM in child maltreatment and the start of my ongoing effort to convert the potential of PRM into measurable social impact," Vaithianathan says.

Application of the New Zealand work, Vaithianathan says, began with the AFST, which has now been implemented, "and Colorado and California are currently engaged in exploratory replications."

"What happens in call screening in child protection services," says Marc Cherna, MSW, director of the Allegheny County Department of Human Services, "is that you get an allegation and you have to do your due diligence on that call—speaking to collaterals, consulting with [the] source of the referral, determining whether the family is known in your system, and creating a risk score that influences whether you're going to give the case to an investigator or screen it out." A high score doesn't automatically indicate that a child needs to be removed from a home, but it demands an investigation, he says.

The county normally screens out about one-half of the cases that come in, Cherna says. When his department looked retrospectively at the screenings done before the use of predictive analytics, it became clear that a considerable number of individuals who, with analytics, would have high scores but were being screened out, and there were some who, with analytics, would have low scores, "whose chances of going into placement were one in a thousand" but who were being investigated. According to the department's website, "an analysis found that 27% of highest-risk cases were screened out, and 48% of the lowest-risk cases were screened in." Cherna estimates that about "80% of the cases that come in are gray area cases, so it's a judgment call whether you screen in or out. Using the score helps determine and influence the decisions." Generally, he says, "With the high scores, investigators go out and the low scores they don't." But the scores don't overrule human judgment. "Obviously if you get an allegation that's egregious, you go out no matter what the score," and if the claim is exceedingly minor or insubstantial, there'd be no investigation, even if there were a high score. He points, for an example of the latter, to a case of a 171/2-year-old boy reported as being truant. Even though an individual that age isn't required to go to school, the score can be a 20 because the boy had a long history in the system. "Because he's almost 18, almost out of the system, a manager can override the score." Cherna's quick to point out, however, "we always have to lay eyes on babies and little ones."

The bulk of the calls, he says, are situations about which "reasonable, serious people can disagree" over whether action is required. PRM offers an edge and contributes to consistency, but the role of the social worker remains paramount. "Social worker judgment is critical," says Ira M. Schwartz, MSW, a consultant and adjunct professor at Barry University of Social Work who's completed multiple large-scale studies of the use of predictive analytics and child welfare. "Analytics will help them make more informed decisions. As with physicians, the more they know about a specific case, the more accurate the diagnosis. Social workers will also be able to make more targeted intervention decisions leading to better outcomes." The social worker at the call center, he adds, "will have a much clearer idea about which cases are the most serious as opposed to those that are not. "

Effectiveness
Independent evaluations are designed to assess the effectiveness of analytics. Two separate proposals were awarded to evaluate the AFST, one from a public sector evaluation, research, and consulting firm and the other from Stanford University.

"The fact that a predictive model will be independently evaluated helps to build trust and support for the project," Vaithianathan says. "It also forces the researchers and agency to be clear about what the tool is setting out to achieve from the start, creating an agreed-upon measure of success." The independent impact evaluation, she adds, "will tell us how effective the AFST is, in particular how it improves accuracy of referrals by call screeners, reduces unwarranted variation in whether similar children are referred for investigation, reduces disparities for similar children from minority groups, and maintains reasonable workload in terms of the overall rate of referral."

It's always a work in progress, Cherna says, never stagnant because it's always being refined. "We don't have the evaluations completed by Stanford yet," he says, "but so far, preliminary results are that the AFST is helping us and it's positive."

Concern About Bias
Concern has been voiced about the possibility that PRM could be used disproportionally in a manner that could heighten racial bias and stigmatize, marginalize, and put undue attention on certain parents, particularly those of color, that might lead to the unjust wrenching of children from their families. But that viewpoint is not universal, and some who acknowledge the risk point to the ways it is being mitigated.

Putnam-Hornstein notes that "no one thinks the current system is free of bias operating via human decision-makers," pointing out that having standardized risk scores through predictive analytics "might help surface disparate treatment." A low score, for example, might cause a caseworker to rethink an assumption, or a supervisor to consult more closely with a caseworker before a decision is made. Furthermore, in the case of the AFST, only the screeners and their supervisors see the scores the analytics produce, never the investigators. "The investigators don't get the score because you don't want to bias the investigation," Cherna says. In this way, low scores won't allow investigators to overlook negative factors and high scores won't bias the investigators and lead them to judge a situation unnecessarily harshly.

To address issues about bias, Allegheny County commissioned an ethical review, and the authors determined not only that the tool was ethically appropriate but also that not using an accurate predictive tool would itself constitute an ethical quagmire. The analysis also asserted that existing methods are not free from the possibility of stigmatization and that new methods must be compared against the status quo. At the same time, it pointed to strategies for responding to stigmatization that can mitigate its impact and even give predictive analysis an edge. These include ensuring that scores are protected from those who don't require them, as well as providing training to reduce stigmatization and overcome confirmation bias.

"The use of predictive risk modeling raises important questions around topics including consent, family information, false-positives and -negatives, stigmatization, racial disparity, professional competence, suitable interventions, monitoring, and resourcing," Vaithianathan says. "By commissioning an independent ethical review, as Allegheny County did for the AFST, an agency can understand how a proposed model measures up ethically, in key areas of concern and overall. Commissioning an ethics report is a good first step, but most important is how the agency responds to an ethics report, and, where appropriate, incorporates the advice given into the model that is implemented."

"We try to be very transparent," Cherna says, "and I think we've set the bar for that with what we've done." He points out that extensive community engagement work occurred before launching the AFST. "We got input from parents and consumers of the system as well as from lawyers and advocacy groups to make sure people were comfortable with what we were doing before we did it."

Other Obstacles to Widespread Use
Though many believe it has great potential to make a significant difference in child protective services, PRM has not been widely implemented in the United States. "Allegheny County is the first jurisdiction to use a PRM that relies purely on administrative data to generate a screening score to guide decisions on how to respond to incoming allegations of child maltreatment," Vaithianathan says.

According to Putnam-Hornstein, "It is very early days for tools like this. And we must be careful to not oversell their accuracy or the impact they will have—the AFST is not a crystal ball. But I truly believe that PRM has the potential to fundamentally shift practice on the front end of our child protection systems—helping reduce the number of families subjected to unnecessary investigations, while also supporting social workers to prioritize referrals where action is needed."

What's standing in the way of more widespread implementation? "The field of PRM for social services currently lacks agreed 'best practice' principles for the development, implementation, and governance of PRM tools, Vaithianathan says. "This means agencies, funders, and communities do not tend to have a clear understanding of how PRM can help, what a 'good' PRM application looks like, or how to procure a PRM tool. This gap leaves each agency—and the provider it selects—to cut its own path, making a range of decisions around aspects like methodology, transparency, ethics, fairness, and use policy. Many important filters, like acceptance by agency leadership, community, stakeholders, and frontline staff, sit between even the most accurate theoretical model, and an effective and ethical use case." This, she says, "is why there exist highly accurate tools within [the] health and human services field that remain unused."

Although Vaithianathan notes a lack of agreed best practice principles in the field, she observes that "there's movement toward developing principles now from many directions. There's growing awareness among communities and agencies of these concerns, making them more likely to hold PRM providers to account on these topics. It's up to each agency/provider how they navigate these concerns and up to stakeholders including the media to ensure agencies are held to account on those issues." Agencies, she says, "can proactively address these concerns by commissioning and responding to an independent ethics review."

According to Schwartz, "Social workers will slowly become users because they tend to be reluctant to use technology to improve practice and fear anything that they think will impede or limit their discretion." A serious issue, he says, is that people who work in child welfare tend to be BA-educated social workers. He points to the need for greater understanding, noting that MSW students also need to become familiar with analytics in their graduate training. "The biggest challenge," Schwartz says, "is to educate administrators, supervisors, and line social workers about how analytics can improve practice, and social work educators need to be trained and educated in analytics as well—at least about its benefits."

Schwartz predicts that analytic tools "will play a gradually increasing and important role in improving the accuracy and utility of risk and safety assessments in child welfare." And, according to Vaithianathan, there will be "cautious adoption where there is a compelling case for its use." She doesn't, however, see the widespread adoption that's occurred in other fields. "Child welfare is an area where we cannot afford to get things wrong, so we are naturally cautious. But I also think it is an area where there are a lot of challenges that have nothing to do with predictive analytics but rather with more simple things like getting information to the right people to make good decisions."

Future endeavors will build on current successes, Vaithianathan says. "Comprehensive guidelines around PRM development, implementation, and governance will make a big difference to this process in the future, removing the need for every project to 'reinvent the wheel' and reducing the risk of failure." Once stronger guidance is in place, she says, "we can expect wider implementation."

— Kate Jackson is an editor and freelance writer based in Milford, PA, and a frequent contributor to Social Work Today.