Whenever the subject of science policy comes up, people are quick to tell you that there are two aspects to the subject.
First up is policy for science. This generally refers to subjects such as federal budget outlays for science, and the relative priorities and dollar amounts for different fields…the allocation for the life sciences versus particle physics, say. It speaks to preferences for in-house research in federal agencies versus research grants to universities. It covers the use of Institutional Review Boards and other instrumentalities for ensuring that human rights are protected in experiments. There’s a great deal more, but you get the idea.
Then there’s science for policy. This is the use of science in policy formulation. This usually is taken to cover topics such as the use of data on greenhouse gas emissions to inform climate-change policy, the use of research linking smoking to cancer to develop regulation for the tobacco industry, and more.
Today’s Washington Post Outlook section surfaced a third dimension to this general subject area. Dylan Matthews, a reporter for the Post’s Wonkblog, wrote an interesting article entitled Taking the guesswork out of policy. The subtitle tells the whole story: The Post’s Dylan Matthews says there’s a way for Congress to make better laws: Test them first.
Here’s an excerpt or two. Mr. Matthews starts this way: I’m a policy reporter. My job is to explain to my readers whether smaller class sizes help students learn, whether tax cuts boost economic growth and whether housing programs help families escape poverty.
In a perfect world, what I do would be a kind of science reporting. Just as my colleagues at the health desk often explain which medicines are effective and which are a bust, I’d ideally be able to describe what sociologists, economists and political scientists have discovered about which policies work.
With a few exceptions, however, I can’t really do that — at least not with the precision my health colleagues often can. More important, neither can policymakers in Congress and in many regulatory agencies. The Food and Drug Administration has more information available in deciding whether to approve a treatment that a few thousand people will receive than Congress does in considering a bill that will affect every American.
He then goes on to cite a few cases where researchers actually did run pilot tests on policies and collect data to help guide a permanent decision. Some are domestic, but where Mr. Matthews sees really interesting results is in the international development arena: With funding from individuals and foundations, but for the most part not governments, they [development-policy researchers] have learned, for instance, that spreading information about the benefits of education keeps students in class; remedial tutoring doesn’t. Giving away bed nets reduces malaria infections; charging even a small amount for them is much less effective.
The confidence with which development researchers can make these judgments is in stark contrast to the Talmudic reading of a handful of studies that characterizes debate about social policy in the United States.
Mr. Matthews points out that the analytical capability available to Congress is limited to the Government Accountability Office (GAO), the Congressional Budget Office (CBO), and the Congressional Research Service (CRS), but the tools, the remit, and the resources available to these offices are limited.
He closes this way: Most people would never dream of taking a pill that lacks FDA approval — that hasn’t made it through a randomized trial with a solid record. We shouldn’t have to settle for less from federal policy. With a congressional policy-evaluation office, we wouldn’t have to.
What a splendid notion! The concept is easy to grasp and immediately attractive and compelling. Please take a moment to read the full article.
At the same time, it’s also easy to see where objections will arise. This idea will take a great deal more socialization and debate before it’s accepted and put into wider practice.
All the more reason to start talking about it now.
This looks to be another of those crucial conversations we were discussing Saturday.
More on this intriguing idea in the next post.
In adaptive governance, appraisal is the critical function. . In climate change policy, the agreed-upon policies are adaptation and mitigation to reduce losses of the things we value. The respective criteria for these policies are reductions in net losses and present vulnerability to climate-related events, and reductions in greenhouse gases to prevent further and perhaps more dangerous changes in the climate system. Observations for appraisal purposes should be focused on these primary outcomes but not limited to them. Judgments of success and failure also depend on observations of related outcomes, including costs. Explanations of formal and effective responsibility for multiple relevant outcomes also depend on additional observations. In other words, the main outcomes must be understood in context.
(See Box 1.1 in our book Adaptive Governance and Climate Change: a procedurally rational approach to policy requires policy processes that rely on appraisals for terminating failed policies and building on successful ones…)
Great comment, Amanda! Thanks for taking the time to write and in so gentle a fashion reminding me once again that’s what’s new to me may not be new to the world…I guess what got me excited about Dylan Matthews’ piece was the idea of a Congressional policy-evaluation office. At least in my reading of his suggestion this office and its resources would be proactively brought to bear by members of Congress contemplating legislation as part of the routine hygiene and practice of policy development, as opposed to leaving this kind of work solely to outsiders…and allowing it to be catch-as-catch-can. I may have been guilty of reading too much into his comments…and was certainly guilty of giving insufficient shrift to work extant.
I think it is great that you are highlighting such an important idea! It would indeed be wonderful if Congress put this kind of thing in place. Of course, we would need to teach our politicians and the press that “failure” is a good thing. Technically rational policy approaches rely on formal methods and metrics to evaluate planned alternatives and avoid failure a priori. The cost of policy failure tends to be prohibitive, justifying the recruitment of experts for extensive research and planning that attempts to get policy right the first time, which we know of course is not possible in a complex, contextual world.
Policy actions inevitably are matters of trial and error in some degree; sometimes they pay off more or less as expected, and sometimes they do not. The diversity of circumstances, trials, and outcomes on the ground frustrates centralized decision making but that diversity is an asset in adaptive governance. Among other things, diversity provides opportunities for harvesting experience on what works, and for diffusing that experience to inform choices and decisions elsewhere. If we conduct many experiments at once, we can keep what works and discard what does not work. We must embrace failure – allow our politicians to fail – and learn from it. Evaluation and appraisal of course is how we learn.