By: AIF Staff
In 2014, a year that coincided with the 50th Anniversary of the War on Poverty, then-Chairman of the House Budget Committee Paul Ryan conducted a review of the federal government’s poverty-fighting efforts and disappointingly found that federal programs often fail to align dollars spent with outcomes achieved. All too frequently, well-intended federal programs did not meet their stated aims and as a result, communities and participants suffered due to lack-luster performances.
The 2014 assessment of the federal government’s poverty-fighting efforts revealed a simple truth: For decades, the federal government has attempted to structure large-scale programs in a way that achieves positive employment outcomes and there are lessons, both good and bad, to be learned from these efforts.
The desire to improve the federal government’s poverty-fighting efforts so they better assist more Americans is why Speaker Ryan and the American Idea Foundation participated in a project, led by Social Finance and conducted in partnership with the Federal Reserves of Philadelphia, Atlanta, and San Francisco, to identify incentive models that have demonstrated track records of success.
The result of this project was a book, Workforce Realigned: How New Partnerships are Advancing Economic Mobility, that contains a series of 19 case studies about how results-based financing can create benefits to the government, employers, service providers, and participants. Speaker Ryan’s chapter focused specifically on the mixed results of past performance-based programs and on a promising new innovation in this space, Social Impact Bonds.
Key excerpts from Speaker Ryan’s chapter follows. You can read his chapter in its entirety here.
An Introduction to Performance-Based Contracting
America is at a moment of great need and great opportunity in the fight against poverty. Amid a global pandemic and recession, the importance of disrupting the stale institutions in place to tackle these challenges has become clearer than ever before…
Our efforts have too often originated in Washington D.C. with little input from the individuals on the ground working to expand opportunity and those with lived experience, leading to approaches that further displace and marginalize those living in poverty. It’s when we innovate together to solve this problem–combining the vibrancy of community-based solutions, the know-how of the private sector, and the scale of government policy–that we have the greatest potential to make a difference.
Social impact bonds bring together the best of the public and private sectors to address the most critical issues our country is facing. The goal of a social public good–a world in which far fewer Americans live in poverty–is central to their execution. So too is the expertise and capital of the private sector, which provides the funding, the strategic thinking, and the energy to deploy resources where most needed. When executed properly, programs like these have enormous promise.
Unfortunately, the long history of performance-based contracting in American civic life includes frequent examples of programs that have not achieved their desired results. Over the past four decades, the government has attempted to structure several programs that offer payouts based on provider performance to drive better outcomes. Although these programs have been created with the best intentions, they have driven little improvement to the status quo. Identifying and addressing the challenges they have faced will be critical to designing the next generation of performance-based contracts….
When performance-based contracts are set up well, shortfalls are mitigated and the programs have significant potential to improve lives. Well-executed performance-based contracting offers benefits for all parties involved, by shifting spending risk away from governments, creating positive feedback loops based on provider effectiveness, and facilitating the collection of data on intervention outcomes.
As the next generation of performance-based contracts takes hold–strengthened by groundbreaking federal legislation, such as the Social Impact Partnerships to Pay for Results Act–it is essential that we learn from the challenges that past initiatives have faced. Historical examples from workforce development and health care offer lessons on how to mitigate typical shortcomings and fully unlock the potential of performance-based contracting.
The Job Training Partnership Act (1982)
Going back to the 1980s, federal legislation has tied payments to employment outcomes achieved by program participants…. The Job Training Partnership Act (JTPA) of 1982 was one such program. Developed through a bipartisan effort led by Senators Dan Quayle, Edward Kennedy, Paula Hawkins, and Claiborne Pell and by Representatives Augustus Hawkins and James Jeffords, it was signed into law by then President Reagan. The bill aimed to improve employment rates for low-income Americans by providing budgetary rewards and sanctions to jurisdictions based on the near-term labor market outcome levels achieved by participants.
To carry out its purpose, the JTPA established federal assistance for adult and youth programs, federally administered programs (such as training for migrant workers and veterans), summer youth employment and training programs, and training assistance for workers affected by layoffs. The program established a performance management system that provided rankings of 620 Service Delivery Areas (SDAs) and set aside funding to reward SDAs that performed particularly well relative to the overall labor market.
To evaluate outcomes, JTPA originally considered four performance measures: rate of entering employment, average wage at placement, cost per participant who entered employment, and rate of entering employment among welfare recipients. However, states were given considerable flexibility to select comparison data and define favorable terms. Where improved results existed, it became clear that they had been driven by the selection of participants who had fewer needs, and, therefore, were easier to serve…. The JTPA was suffering from several common challenges, as identified in an analysis by Burt Barnow & Jeffrey Smith:
- The program provided stranger incentives to serve less-vulnerable populations: JTPA incentives treated all program participants equally, which led to higher margins for service providers who chose to serve lower-need individuals. The program did not serve groups such as women and people of color in proportion to their share of the eligible population, while individuals who would likely have achieved high post-training earnings regardless of the quality of the training were disproportionately represented. A structure that assigns different levels of value based upon the need of the population might have addressed this challenge.
- The timing of performance incentives skewed services provided: In some cases, the length of the training programs was influenced by program managers’ desire to count participants in their data for a particular program year. These arbitrary timing changes were found to reduce the overall mean impact of the training services the program provided. Updates to monitoring and reporting systems may limit the extent to which programs are able to manipulate data in this way.
- There was limited support that incentives improved individual performance: It is unclear that the project improved the individual efficiency of employees in the absence of incentives at the individual employee level. Future performance-based contracts may explore how service providers can pass on incentive payments to their employees and how they can track individual performance without adding significant overhead costs.
- Some providers gamed the compensation system: There is strong evidence that JTPA service providers developed strategies to earn higher payments by gaming the performance system. A common gaming strategy involved formally enrolling participants in the program only after they had found jobs, and then quickly terminating them, in order to increase the proportion of employed individuals. Adjusting reporting requirements and improving metrics and evaluation systems could help to reduce the extent to which gaming can yield higher payments.
As a result of its structural challenges and the limited improvement to participants’ employment outcomes, the JTPA was repealed in 1998. The program’s failure to segment target populations, its focus on measurements that were not linked to individual performance, and its lack of safeguards to avoid gaming are valuable reminders of the potential risks of performance-based contracting in workforce development. However, these mistakes also offer lessons regarding critical areas of focus for other pay-for-performance programs to succeed in the future.
Ticket to Work (1999)
In 1999, another performance-based workforce development program emerged that aimed to increase the number of low-income Americans achieving economic self-sufficiency. At the time, only 0.5% of Social Security Disability beneficiaries were leaving the benefit rolls because they secured jobs. Legislators hoped to create a better market to meet the diverse return-to-work service needs of beneficiaries and increase the rate of exiting the program due to work to 1%.
The Ticket to Work (TTW) and Work Improvement Incentives Act of 1999 was designed to support this mission by promoting flexible, customizable services to help disability insurance beneficiaries secure self-supporting jobs. The program incentivized private organizations and state agencies to deliver quality services by providing large payments for each client who secured a job and retained it for long enough to stop receiving Social Security Disability benefits….
Despite the program’s intent to reach a broad group of beneficiaries, its early success was limited: By 2005, only 2% of the individuals who received Tickets in the mail had used them, and only 45% of the 1,300 enrolled employer networks had accepted a Ticket. Like the JTPA, TTW’s outcomes suffered from a range of shortfalls:
- Providers perceived the system as too financially risky: TTW tied 100% of provider compensation to outcomes, which caused significant uncertainty as to whether payouts would be achieved. Research showed that after the first two years of program operations, employer networks relying on TTW payments as their sole source of revenue would have lost money: The cost of service delivery far exceeded TTW revenues for most providers. Offering upfront operating capital to providers in addition to outcomes-based payments, as many social impact bonds now do, might have helped to mitigate this challenge.
- The program provided higher payouts to providers serving less-vulnerable populations: Like other unsuccessful performance-based contracts, TTW created selection bias against harder-to-serve individuals and services less likely to lead to quick employment. Providers could refuse to serve individuals who they thought were unlikely to maintain high enough earnings to stop receiving benefits, and, therefore, unlikely to trigger outcome payments. They could also choose to offer only services that aligned with the outcomes payments they were likely to receive. Differentiating payment amounts based on participants’ level of need could have helped to avoid rewarding providers for serving the lowest-need clients.
- The benefits structure discouraged some beneficiaries from returning to work: The program did nothing to address that participants would lose 100% of their Social Security Disability benefits once their monthly earnings exceeded a certain threshold, which created a significant barrier for returning to work. Structuring the program to scale the reduction of benefits more gradually might have increased the value proposition of returning to the workforce for participants.
- Reporting and administrative burdens fell on service providers and participants: Finally, there were significant administrative challenges that delayed outcomes measurement and provider repayment. Beneficiaries were expected to submit salary documentation to employer networks but given no incentive to do so, which made it difficult for employer networks to demonstrate that monthly earnings had reached the designated threshold. Establishing data-sharing provisions upfront might have minimized administrative burdens and streamlined the system for triggering repayment….
In response to provider feedback, a set of revisions passed in 2008 that increased the number and total value of provider payments, shortened the period of participant employment for employer networks to receive full payment from 60 to 36 months, and revamped payment procedures to reduce administrative burden. The revised system was significantly more attractive to providers, and the number of employer networks that accepted at least one Ticket doubled from 2007 to 2010.
The increase in participation in TTW following the 2008 legislation reform affirms the importance of seeking service provider input to mitigate unforeseen barriers to entry….
Physician Pay for Performance (2005-2017)
Health care is another area in which performance-based contracting offers both the potential to improve service quality, and the risk of gaming and poorly structured incentives. The United States spends more on doctors, pharmaceuticals, and health administration as a percentage of GDP than any other high-income nation, yet does not enjoy better health outcomes. To combat rising costs and improve quality, states, healthcare systems, insurance companies, and federal agencies have piloted Pay-For-Performance (PFP) programs for physicians and hospitals across the country. The success of these programs, however, has been largely uneven.
In November 2017, researchers from the University of Pittsburgh and Harvard published a study reporting that Medicare PFP programs failed to improve healthcare quality or reduce costs. Rather than promote better outcomes, the program penalized physicians who cared for lower-income and sicker patients because the doctors’ “quality scores,” and, therefore payment, decreased. The program’s structure emphasized health outputs over baseline improvement, creating financial disincentives for doctors to treat patients who were less healthy.
Providing higher payouts to those who serve healthier patients is a key issue in physician PFP programs, in which financial incentives often fail to promote health improvements over specific health outputs. While physician skill is an important component of health quality, factors such as the patient’s baseline health, socio-economic status, access to insurance, and exercise habits all contribute to health outcomes and are largely outside of the doctor’s control. PFP programs that exclusively target physician pay without supporting other interventions draw a direct link from individual clinician skill to patient health that can create financial disincentives to treat the sickest patients….
The lack of success of physician and hospital PFP programs has led many critics to call for an end to PFP in healthcare. However, it’s possible that an outcomes-based funding system could be effective in the absence of this program’s poor project design, weak measurement, incorrect outcome criteria, and flawed linkages between the intervention and outcomes. Past PFP programs struggled because they were structured around the underlying concept that financial rewards to physicians could improve outcomes in a vacuum. A stronger design could rescue the core concept.
Social Impact Partnerships to Pay for Results (2018)
As we’ve seen, performance-based contracts can fall victim to predictable design errors. But, when structured well, these programs have the potential for impressive results. In support of improving the effectiveness of social services, the Social Impact Partnerships to Pay for Results Act (SIPPRA) was signed into law in 2018.
SIPPRA brings great promise for the next generation of performance-based contracts. The Act appropriates $100 million to the U.S. Department of the Treasury, $15 million of which is set aside for evaluation costs to support state and local governments in building a foundation for outcomes-based decision making. Funding can be used across a range of issue areas, including child and family welfare, health, education, and employment, creating extensive opportunities to address the country’s most pressing needs.
I’m personally incredibly proud of SIPPRA and the principles it follows. First and foremost, SIPPRA takes a clearly evidence-based approach to lifting Americans out of poverty: Funding flows to programs whose methods have been evaluated using data, supporting real-world efforts that achieve positive results.
In building programs based on evidence of what works, SIPPRA has the potential to finance the most effective solutions for fighting poverty, which originate not from Washington D.C., but from leaders on the ground in communities across the country. SIPPRA funding will support individuals and organizations that have been making a difference in their communities for decades, while bringing their ideas to policymakers to expand their reach. This intersection of community-based approaches and government support is what will ultimately most improve the lives of Americans in need.