By: AIF Staff
Earlier this week, the American Idea Foundation and the University of Notre Dame’s Lab for Economic Opportunities (LEO) hosted a virtual policy panel examining how evidence-based child welfare programs can assist younger Americans meet their short-term needs and be oriented for long-term success. The panel discussion, moderated by former Speaker Paul Ryan, featured perspectives from leading academics, practitioners, and thought-leaders who are working on innovative social programs targeted to at-risk youth and their families.
Throughout the conversation, the panelists reinforced the importance of assisting America’s youth by utilizing evidence-based strategies and interventions with track records of success. The conversation highlighted how building supporting evidence for social programs is a long-term, collaborative effort between policymakers, researchers, and academics, all of whom share the same goal of helping children and families out of poverty.
Joining Speaker Ryan in conversation were:
- Bill Evans, Keough-Hesburgh Professor at the University of Notre Dame
- Erin Kelley-Siel, Chief Office of Expansion and Policy, Friends of the Children
- David Sanders, Executive Vice President of Systems Improvement, Casey Family Foundation
Video of the panel discussion is available here. Notable excerpts from the conversation, edited lightly for clarity, follow.
Speaker Ryan: Evidence-based reforms to our child welfare system are necessary & overdue
“The child welfare system works with some of the most vulnerable members of our society. And, for too long, its treatment of our children has been a scandal.
“Until recently, policymakers have ignored the child welfare space. The recent passage of the Family First legislation was the first major reform to this area since the early 1980s. This isn’t for a lack of problems in the system. Far too many children are taken out of their homes too quickly, while other children are left to languish in difficult situations.
“Thankfully, there are a number of hardworking individuals in this space working to provide safe, permanent homes to children in need. And, even more importantly, we are working to prevent the need for youth to enter the foster care system in the first place.
“Unfortunately, one of the major issues with this space, like many others, is the lack of evidence on what works and what doesn’t work. And, even where there is evidence, it is all too difficult for caseworkers to identify programs that work. It’s one reason why the American Idea Foundation is working on a user-designed clearinghouse that would enable case workers to identify programs that work and easily refer families and children to them. As the Family First legislation continues to be implemented, we will continue to need individuals to do the hard work of developing an evidence base for programs that are successful.
“We have, in some other areas, seen ways that policymakers have overcome some of the challenges that these topics present. For example, the American Idea Foundation recently visited a Nurse-Family Partnership program in South Carolina. This program has solved a number of these issues – they developed an evidence base, they have scaled their programs, and they have federal legislation, called MIECHV, which was started under President Bush and continued on under Presidents Obama and Trump.
“It surpassed political parameters because it was carried forward with a body of evidence that proved it’s a good way to run this program. This is what we’re trying to do with the American Idea Foundation clearing house and it is exactly what LEO does on a day-to-day basis. These are the types of programs that policymakers need to be looking at and that’s why I’m so excited about the panelists that we have.”
Erin Kelley-Seil: Friends of the Children’s ‘secret sauce’ is love, building a base of evidence is a time-consuming but worthwhile endeavor
“First of all, for those who may not know, Friends of the Children is a national nonprofit, and what we do is we invite children and families who are facing the greatest obstacles when their children are between the ages of 4-6, and we give them a paid, professional, salaried mentor to work alongside that child and family from that time in their development all the way through high school graduation. We are a 30-year-old non-profit. We’re currently in 24 locations so we’re close to meeting our goal of 25 locations and really the Family First legislation has been very catalytic for what we do.
“We currently have a randomized control trial underway that’s focused on our outcomes. The trial will be one of the longest in the country that ever will happen. By the time it is completed, it will take 17 years and as I mentioned, we are a 12-plus year intervention and we have preliminary evidence of favorable effects of our program.
“We also had a qualitative study that was funded by the Casey Foundation that looked specifically at our foster care impact because historically for us, 40% of the youth that we are serving have experienced either formal foster care or informal kinship care and 100% are at-risk when using factors to assess children who are most in need of programs.
“Even with all this evidence – and you have with decades of third-party evaluation that has been underway, our advisors and researchers in the evaluation space from the University of Washington who are focused on evidence-based practice said you still don’t have enough to make it on to the Family First Clearinghouse. It was actually well-timed and painful, but really well-timed because it led us to talk about scaling and innovation. We had been innovating with regard to our model… and we had early impact data so the [RCT] thought process really wasn’t hard for us. It was because evaluation has been part of our DNA. For us, the [RCT] was a natural extension of what we were already doing now. Many of us that work here are still a little bit frustrated because we thought we would have had enough evidence to make it on the Family First Clearing House but we’re grateful for the chance to work with LEO.”
**
“We really believe there are three things that I’ll say that are the secret sauce. Let me just pick one or two. We have a paid professional in their lives and they stay with them for the long haul. We just believe those kids that are at the most risk need a longer duration of service than a 12-month service intervention. These are families who have been hurt by generational poverty and systemic obstacles and that is who we’re really working to serve.
“We actually had a researcher tell us once that our secret sauce is love. We build relationships. The mentors in our program have enough agility and adaptability to highly individualize their service approach and so everything we do is really grounded in love and one of our core values is to build relationships.
“The average tenure of one of our mentors is five to six years so when we introduce ourselves and our caregivers to families, we say that you should expect to have probably two mentors in the life cycle of your child and part of that, for us, is modeling and making a healthy transition. We’re working, as I mentioned, with a lot of youth who have had a lot of transitions in their lives and modeling that as an adult and building social capital for them with more than one adult is part of our model.”
Professor Bill Evans: Amid opioid epidemic, Notre Dame’s Laboratory for Economic Opportunities is helping local child welfare providers develop evidence of success
“If we take a look at children in foster care, there’s probably no group in the United States that has worse economic outcomes just in terms of basic socio-economic indicators, like the graduation rate from high school, the percentage that go on to college, the income levels at age 30, this group performs incredibly poorly across all different characteristics so we have to figure out not only how to deal with them when they’re in foster care but how to help them adjust as they are moving from foster care to early adulthood. We made some great legislative changes in 2018. We expanded the foster care program, but we really haven’t spent much time evaluating what’s going on with that at all.
“I think more children are at risk than ever before because of the opioid crisis, and some preliminary estimates are about 90,000 people died from drug overdoses in the past year, 20,000 more than the previous year. Some of our own work suggests there is about 1.5 million kids that are living away from their parents because the opioid crisis, so the risk to these kids is incredibly staggering. We have to figure out what works, but… we know very little about what policies can do to change these levers….
“For example, across states, the structure of the foster care system is quite different. In some states, it’s all government-run. In some states, it is privatized. In some states, it’s a mixture of the two. We have very little information about whether outcomes are different across states because of the way that these systems are structured. And that would seem to be a fundamental question to identify: Are we getting different outcomes based on the way this program is structured?
“We know very little about the extension of benefits to people after they age out or after they turn 18 and whether they’re on extended benefits or not, are better things happening with regard to outcomes? So, if we can, at least within legislative circles when we adopt these changes, require that there’ll be some evaluation program set aside to look at effectiveness, that would be useful.
“Recent legislation was passed trying to encourage the evaluation of federal programs but the way we found that change is actually occurring was primarily by working with courageous, local agencies. The local social service groups that have innovative programs and that are helping children out are willing to put the program under the microscope and allow people to examine it through randomized controlled experiments and determine whether the program is working or not. This is the group that we found to be the most receptive to evaluation so the more that we could work with local social service agencies that are actually engaged in helping us see who is at risk, I think the better off we’re going to be…. We need to figure out ways to encourage governments at all levels to take a look at these programs and get a better sense of what is working and what is not.”
Professor Bill Evans: Partnerships with researchers and social service providers are critical to evaluating programs
“The first part is to have you convinced that you want to be part of the RCT and given all the barriers, I think the way in which we do this at LEO is a pretty good model. Social service organizations are there to provide services. They don’t necessarily have the skills to do an evaluation. They might understand how to do it, but it’s not their job. Our job isn’t to provide social service to youths. Our job is to evaluate programs and to understand incentives and understand outcomes. And so, the partnership between an academic institution and social service providers can be a lot more than the sum of its parts.
“When we do an evaluation, we typically are taking on all the costs of tracking clients in order to do surveys or trying to find administrative data that allows us to track outcomes. We don’t want to interfere with the provision of services at all and it’s too much of a burden for the organization to take on those costs, so we’re going to pick up those costs now. Most of the time, we’re looking for projects we think are going to move the needle on poverty so eventually, we think we’re going to be able to find funding from the federal government or foundations that are going to help pay for the research costs. But I think that this dual nature of “you’re providing services and we’re providing evaluation” is a good way to think about it.
“One thing I’ll say about academics is that academics are looking for good ideas — that’s our business, and finding great partners out there where we could move the needle in this particular sector has been amazing. When we started and were trying to figure out one way that we can be different from other poverty labs across the country, we decided we were going to be different was in examining ideas are at the local level, where people are interacting with the client and a lot of times, there was no way for those ideas to bubble up. And so, what we’re trying to do is to put some empirical content on those great ideas.
“We are finding these people and finding these groups that not only have the good ideas but also have the bandwidth to actually do an evaluation — because sometimes it’s difficult and so we have a whole group that tries to find the innovative providers and tries to find the people that have quality staff who have the enthusiasm and the willingness to take a look under the hood.
“The second question was about cost. One thing that is useful when you’re doing an evaluation is you learn a lot of things about the way that the program operates. One of our evaluations was on emergency financial assistance for homelessness and what we found is that the people that are receiving the greatest benefits are those people that are actually the most at risk. Therefore, if you want to have a bigger impact, you actually had to target the program a lot more to lower income families, and maybe be more aggressive funneling money to that specific group. And so, if the goal is to try to have as much impact with fixed dollars that you have, a lot of times what you learn along the way can be very beneficial….
“I think the process of evaluation gets you thinking about how can I make my program better and who is benefiting the most? Are there specific character traits that are going to benefit most from the intervention? It gets you thinking in that way and making evaluation a much more important part of your day-to-day operation.”
David Sanders on how federal policymakers can assist social service providers by incentivizing evaluation and evidence:
“I think, first, the federal government needs to encourage payments for the things that we know work and I think that there are many sound ways of doing this, but I’ll give two examples.
“You mentioned earlier the concern about too many children being removed from their homes and at the same time too many children being left in dangerous situations. Well, the process in the health protection system is to do an investigation and that investigation is to determine the needs of the family and as far as I know, we have done little if any research on the effectiveness of if investigations accomplish exactly what we want them to. Yes, we paid for the investigation without question but did it accomplish what we wanted it to do?
“A second example is congregate care. Until Families First, a child could stay in congregate care for as long as the agency decided that they needed to be in here and they would be paid by the federal government for that. We have little, if any, evidence that congregate care actually works beyond a certain period of time, but we will pay for it regardless of what the evidence suggests.
“I think, first and foremost, we should actually pay for things that are accurate. Second, I think when we find that there are effective support programs that we also pay for them and that we do it quickly…. If innovations are demonstrated to work, they need to move quickly into the field and be supported and we should really offset that by not paying for things that we know don’t work.”
Speaker Ryan on the next steps of the Evidence Act and linking federal agencies with practitioners and researchers
“I toured the Nurse Family Partnership in Kenosha, Wisconsin many years ago and saw the results of my own eyes, not just anecdotally, but I saw the data, because it was the one program that the federal government actually used evidence. It is more or less why we wrote this bill, called the Evidence Act, that is in place now. To your point David, I couldn’t have said it better myself.
“Let’s go find out what works. Let’s find what works and by the way, since money and dollars are finite, take from what we now know does not work and put those dollars into what does work. We’re just now on the cusp of what I would call a “research renaissance” to learn about those things. And where this matters the most, and why the American Idea Foundation is focusing on this area, is because our children are the area that is in the greatest need. It also has the biggest potential to make the biggest difference.”
This panel discussion was part of a quarterly series of policy conversations hosted by the American Idea Foundation to draw attention to evidence-based policies aimed at expanding economic opportunities. Past policy conversations have focused on building a 21st century workforce, reforming the Earned Income Tax Credit, reducing recidivism and promoting 2nd chances, and properly implementing Opportunity Zones.
Note: Former Speaker Paul Ryan is a visiting lecturer at the University of Notre Dame.