The official shutdown of the Justice Department’s Science Advisory Board was announced in a terse message to members this week. One member says it’s a step backwards from an ambitious attempt to apply scientific and evidence-based thinking to the federal justice structure.
On Wednesday morning, all members (including me) of the Department of Justice’s Science Advisory Board (SAB) for the Office of Justice Programs were notified that the Department is abolishing the SAB.
The news was not unexpected. Since the Obama Administration left town, there had been just one meeting of the SAB.
The next three meetings were never scheduled. With no official word about the Board’s fate, most of the members expected to hear that the SAB’s official charter would be allowed to lapse.
The news came in the form of an official letter from Matt Dammersmith, Principal Deputy Assistant Attorney General.
“At this time,” Dammersmith wrote. “[The Office of Justice Programs] OJP has decided to bring closure to the SAB.”
He added: “OJP plans to continue to seek the advice of scientists and practitioners from across to country to provide input into the scientific activities and priorities of the OJP.”
The SAB was formed by the Obama Administration in 2010. Its goal, according to the SAB charter, was to provide advice on science-related issues.
The charter explained the SAB’s mission in the following way:
The Board will provide the office of the Assistant Attorney General (AAG) of the Office of Justice Programs (OJP) with valuable advice in the areas of social science and statistics for the purpose of enhancing the overall impact and performance of its programs and activities in the areas of criminal and juvenile justice. The Board will help develop long-range plans, advise on program development, and recommend guidance to assist in OJP’s adherence to the highest levels of scientific rigor as appropriate. The Board will provide an important base of contact with the criminal justice and juvenile justice academic and practitioner communities.
Al Blumstein, Carnegie Mellon University. Founding Chair SAB.
The membership of the Board, including founding Chair Al Blumstein of the Carnegie Mellon University, and his successor Ed Mulvey of the University of Pittsburgh School of Medicine, represented a wide range of academic disciplines. That included social and behavioral sciences, as well as professional disciplines including law enforcement, corrections, treatment specialists, prosecutors, and the defense bar.
The initial members represented a broad selection of some of the country’s most noted criminal justice scholars and practitioners, including former NYPD Commissioner William Bratton, Frank Cullen, Mark Lipsey, Tracey Meares, Joan Petersilia, Rick Rosenfeld, Rob Sampson, and David Weisburd.
The most recent membership list maintained that tradition.
During their meetings from 2011 to 2017, the members of the SAB received updates from the leadership of OJP and its bureaus (including the National Institute of Justice, the Bureau of Justice Statistics, The Office of Juvenile Justice and Delinquency Prevention, and the Bureau of Justice Assistance) on the Office’s initiatives and funding plans.
The Board members, in turn, would provide or facilitate presentations about pressing issues in policy and practice, and the extent to which available solutions were grounded in science or could be better supported by new evidence.
The members of the SAB, who served without compensation, did not judge or recommend particular OJP policies or programs. The mission was to provide guidance as available from the scientific literature and the newest findings of evaluation research.
At the first meeting, then-Assistant Attorney General Laurie Robinson told SAB members she hoped the Board could play six key roles in support of OJP.
- Look at the broad role of science within OJP and how we can better integrate what we learn from science in to our programmatic design and spending;
- Think about ways, of course, to strengthen the research and statistical functions within OJP;
- Suggest broad priorities on which research might be focused.
- Consider and make recommendations about institutional ways to protect the science here going into the future.
- Think about ways that OJP can more effectively connect researchers with practitioners and policymakers – particularly on translation of evidence – and generally serve as an avenue for outreach to the field and a promoter of a two-way dialogue;
- And finally, play a helpful role in providing advice and counsel to us on
practical concerns, like improving the way we handle peer review.
The members of the SAB were very conscious of Robinson’s fourth point, which she described as “critical.” As she pointed out:
We may not enjoy the support for science in future Assistant Attorneys General and Attorneys General that we have today. The Justice Department is a lawyer culture and we know from history that it can be hostile to science. We need to build in protections.
During 2016 the Board devoted considerable effort to preparing advisory notes for the field.
We joked (before we knew the outcome of the 2016 election) that these advisory notes represented our “message in a bottle,” that would be set afloat on the broad ocean of political change, and we hoped they would be of service to future researchers and policymakers.
The statements represented the consensus views of the diverse membership of the Board, were approved by a vote of the Board, and then promulgated on the Justice Department’s website on January 17, 2017.
Members of the now-disbanded Science Advisory Board of the Department of Justice
Advisory Statement 1 clarified the important but limited role of randomized controlled trials in the development of effective policies and practices, including:
Randomized controlled trials (RCT) generally provide the strongest or most defensible causal evidence for programs and practices, but it may not always be possible to implement successful RCT evaluations in the field. Many important questions in the field of justice are not answerable using RCT studies—either for practical, economic, political, or ethical reasons.
Research questions that are very difficult or expensive to answer using experimental methods may merit the necessary investment if they have widespread or profound social consequences, just as research questions with only modest consequences still merit experimental investment if they can be answered easily and at little cost. Funding for RCT evaluations should be managed like an investment portfolio with resources concentrated on the most effective combinations of theoretical salience, research feasibility, and social benefit.
Advisory Statement 2 focused on the role of evidence in justice policy and practice, including:
The strength of evidence required to judge the value of programs and practices in the justice field is a question of balance. Judgments should be based on the best available evidence, but the strength of evidence required for any decision is gauged by the costs of error and the burden of increasing evidentiary quality.
Decisions with little consequence require less accurate evidence and less exhaustive evidence. Highly consequential decisions require more evidence. Navigating the continuum of evidence-supported decision-making is complex and subjective. The available evidence for any policy, program, or practice is not the product of a straightforward and untrammeled search for effectiveness. It emerges from a contentious and inherently political process that governs social investment in research.
Advisory Statement 3 examined the responsibility of policymakers and researchers to both identify and develop effective interventions and strategies, including:
Justice programs are generally designed to achieve specific purposes, and produce intended effects. The effects of an intervention on the target population, both intended and unintended, are known as outcomes.
Recidivism is the most common outcome of interest within the justice field, but other measures are used, including satisfaction with services, impacts on drug use, employment, educational success, and the legitimacy of criminal justice agencies across communities. Identifying the correct outcomes for a policy, program or practice is a basic task associated with program development, performance measurement, and program evaluation.
There are a number of considerations related to outcome measurement that OJP must consider in drawing inferences about the effectiveness of justice programs. These include: primary and secondary outcomes, unintended outcomes, cost/benefit, implementation fidelity, and efficacy versus effectiveness research.
The SAB and its meetings and products were an effective means of infusing scientific thinking inside of the federal justice structure with the hope that all aspects of criminal justice, juvenile justice, and the much larger concerns of social justice would be informed by evidence.
Jeffrey Butts, PhD.,director of John Jay College’s Center for Research and Evaluation
And, just as importantly, that the production of future evidence would be sensitive to a broader range of interests and considerations—not simply mirror the preferences of current office holders.
I miss it already.
Jeffrey Butts, PhD., is director of the Research and Evaluation Center at John Jay College of Criminal Justice. This essay was originally published in Dr. Butts’ blog. He welcomes readers’ comments.