In June 2021, The U.S. Social Security Administration held a full-day virtual event dedicated to discussing lessons learned from its past tests of policies and programs to improve beneficiary work outcomes and directions for future research.
At the event, disability and social policy researchers presented papers on lessons learned about which policies, programs, and other operational decisions could provide effective supports for disability beneficiaries who want to work. Following each presentation, academic and policy discussants provided additional context for the research.
Please see the event agenda and presentation summaries below to learn more.
Time
Session
9:45 – 10:00 a.m. EDT
Virtual Event Opens
10:00 – 10:10 a.m. EDT
Opening Remarks
Jeffrey Hemmeter, U.S. Social Security Administration
Austin Nichols, Abt Associates
10:10 – 10:45 a.m. EDT
Design of Demonstrations
Panel Chair: Laura Peck, Abt Associates
Presenters: Burt Barnow, The George Washington University
David Greenberg, University of Maryland, Baltimore County
Discussants: Jesse Rothstein, University of California, Berkeley
Jack Smalligan, The Urban Institute
10:50 – 11:25 a.m. EDT
Use of Demonstrations
Panel Chair: Laura Peck, Abt Associates
Presenters: Austin Nichols, Abt Associates
Robert Weathers, U.S. Social Security Administration
Discussants: Jonah Gelbach, University of California, Berkeley
Elizabeth Curda, U.S. Government Accountability Office
11:25 – 11:45 a.m. EDT
Design & Use of Demonstrations Audience Q&A
11:45 am – 12:00 p.m. EDT
Break
12:00 – 12:30 p.m. EDT
Keynote
Kilolo Kijakazi, U.S. Social Security Administration
12:35 – 1:10 p.m. EDT
Return to Work
Panel Chair: Sarah Prenovitz, Abt Associates
Presenters: Robert Moffitt, Johns Hopkins University
Jesse Gregory, University of Wisconsin – Madison
Discussants: Hilary Hoynes, University of California, Berkeley
Kathleen Romig, Center on Budget and Policy Priorities
1:15 – 1:50 p.m. EDT
Early Intervention
Panel Chair: Sarah Prenovitz, Abt Associates
Presenter: Kevin Hollenbeck, W.E. Upjohn Institute for Employment Research
Discussants: Jeffrey Liebman, Harvard University
Jennifer Sheehy, U.S. Department of Labor
1:50 – 2:10 p.m. EDT
Return to Work & Early Intervention Audience Q&A
2:10 – 2:20 p.m. EDT
Break
2:20 – 3:05 p.m. EDT
Youth Transition
Panel Chair: Daniel Gubits, Abt Associates
Presenters: David Wittenburg, Mathematica
Gina Livermore, Mathematica
Discussants: Lucie Schmidt, Williams College
Manasi Deshpande, University of Chicago
Jennifer Sheehy, U.S. Department of Labor
3:10 – 3:45 p.m. EDT
Heterogeneity: Subgroup Findings
Panel Chair: Daniel Gubits, Abt Associates
Presenter: Till von Wachter, University of California, Los Angeles
Discussants: Howard Goldman, University of Maryland
Nick Hart, The Data Foundation
3:45 – 4:05 p.m. EDT
Youth Transition & Heterogeneity: Subgroup Findings Audience Q&A
4:05 – 4:15 p.m. EDT
Break
4:15 – 4:50 p.m. EDT
Benefits Counseling & Case Management
Panel Chair: Sarah Prenovitz, Abt Associates
Presenter: Vidya Sundar, University of New Hampshire
Discussants: John Kregel, Virginia Commonwealth University
Leslynn Angel, Michigan United Cerebral Palsy
4:55 – 5:30 p.m. EDT
Implementation
Panel Chair: Sarah Prenovitz, Abt Associates
Presenters: Michelle Wood, Abt Associates
Debra Goetz Engler, U.S. Social Security Administration
Discussants: David Stapleton, Tree House Economics
Calvin Johnson, U.S. Department of Housing and Urban Development
5:30 – 5:50 p.m. EDT
Benefits Counseling & Case Management,
Implementation Audience Q&A
5:50 – 6:00 p.m. EDT
Closing Remarks
Jeffrey Hemmeter, U.S. Social Security Administration
Austin Nichols, Abt Associates
Burt Barnow and David Greenberg reviewed the design of evaluations that are typically part of a demonstration. They reviewed the designs of past demonstrations’ evaluations and discussed the implications for what questions those designs can address. They also offered thoughts on alternative designs that the U.S. Social Security Administration (SSA) might consider in the future for greater learning. They encouraged SSA to consider multiple, varied treatment arms and factorial designs to help determine the role of the component parts of the interventions. Additionally, Barnow and Greenberg suggested additional designs, for example stepped wedge designs, that could help provide additional estimates. They also suggested going beyond the intent-to-treat (ITT) estimates that have been the default in SSA’s existing demonstrations and encouraged treatment-on-the-treated (TOT) estimates as well. Finally, they noted how important process analyses are and suggested that an increased use of fidelity measures could help SSA learn more about the specific interventions.
Robert Weathers and Austin Nichols discussed ways to improve the use of evaluation findings and implications for how demonstrations should be used. They focused on questions policymakers have historically wanted answered and how to better communicate findings to meet those needs. Weathers and Nichols encouraged strong theoretical models (including logic models) to underpin the demonstrations and clarify the goals of the demonstration. While acknowledging the existence of tradeoffs, they also urged SSA to consider going beyond the single intervention tests by looking at broad ranges of similar policy options. Looking inside the black box, through multiple treatment arms, factorial designs, etc., is important to answering questions related to why something worked. They also encouraged additional uses of qualitative findings and reanalyzing the data from past demonstrations to extend the analyses included in evaluation contract reports.
Jesse Gregory and Robert Moffitt reviewed the lessons from SSA’s return-to-work demonstrations such as the Benefit Offset National Demonstration and similar demonstrations. They described the incentives that individuals receiving Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) benefits face. They described the history of efforts to improve this population’s work outcomes, including how the demonstrations align with economic theory and what researchers expected to find. They also proposed several considerations for SSA’s next generation of demonstrations, including ideas related to the prior two presentations. One important lesson they noted is that most efforts to increase employment, earnings, and labor force engagement do not have large effects. As a result, it may be necessary to reconsider expectations about how many beneficiaries will go back to work. Promising areas to explore could include work incentives like the earned-income tax credit that provide additional income above and beyond the current benefit (as opposed to just not taking as much away) at first.
Kevin Hollenbeck provided an overview of SSA’s efforts to explore policies related to individuals not yet receiving disability benefits. This included early interventions that might prevent some of them from needing the SSDI or SSI programs for support. Programs that effectively reduce the need for SSDI or SSI could both improve individuals’ economic well-being and reduce government expenditures on these programs, allowing for a more efficient use of program resources. Hollenbeck shared lessons from his review of both U.S. and international programs related to SSDI and SSI. He noted that Individualized Placement and Support has been tested in several settings and while it does appear to have some success in improving labor market outcomes, there is little evidence that this translates into reductions in SSI or SSDI benefits. He recommended testing interventions targeting older denied applicants rather than a large swath of potential applicants.
David Wittenburg and Gina Livermore reviewed lessons from SSA’s efforts to support youth receiving benefits in transitioning to a successful and more self-sufficient adulthood. These efforts generally estimate mixed effects on benefit receipt, but are promising in improving participants’ social connections. Wittenburg and Livermore discussed new policy directions and partnerships SSA could explore. They noted that youth needs are different from those of adults and are not separable from families. As a result, services that focus on family outcomes are important. Additionally, interagency collaborations, such as those utilized in the Promoting Readiness of Minors in SSI project are important. Given the large number of services and program models for youth, building on existing programs, like Job Corps, could provide fruitful next steps.
Till von Wachter reviewed the importance of looking at the heterogeneous impacts of the demonstrations. He showed why subgroup impacts are important to estimate. He also provided some suggestions for groups that might be important to look at more closely. von Wachter noted that each demonstration uses different definitions for earnings outcomes, age groups, disability groups, etc., which hinders cross-demonstration comparisons. He noted that standardized outcomes and subgroup definitions would be helpful when comparing across demonstrations. He also described the state of the art on identifying subgroups with different impacts.
Vidya Sundar reviewed the use of benefits counseling and case management in SSA’s demonstrations. Sundar explored the challenges in measuring the effectiveness of these commonly used but often differently implemented services. She noted that we need more information on the timing and nature of benefits counseling and its interaction with other services (e.g., vocational rehabilitation). Additionally, more information is needed on models that focus on sustaining employment rather than just getting a job.
Michelle Wood and Debra Goetz Engler drew lessons from demonstrations’ implementation reports. Wood and Engler delved into the qualitative and process analyses to glean lessons from how the beneficiaries were recruited, how the demonstrations were run, and how the interventions were delivered. They also highlighted the importance of recruitment and measurement of fidelity to a program model. On recruitment, they noted that dedicated recruitment staff often have advantages over staff responsible for both recruitment and delivery. They also noted that site selection and intervention fidelity are important for structured, specialized services. Wood and Engler pointed out tradeoffs between centralized and decentralized funding and implementation with respect to the feasibility of data systems, operational policies, monitoring and other factors. Finally, they noted that emergency and basic needs of participants may impede participation and engagement in the intervention. Their insights are helpful for understanding how SSA could implement future demonstration efforts as well as conduct general outreach.
