Building the legitimacy of ensemble” approaches to policy formulation.

Tuesday’s post made a case for what might be called “ensemble” approaches to policy formulation. The idea is borrowed from ensemble weather forecasts. Comparing independent numerical weather predictions (NWP) both provides opportunity to improve accuracy (the “consensus”of the various model runs) and characterize the uncertainty (as revealed by the spread or divergence of the different runs). The extension to the policy arena suggests that society is better served by capturing the full diversity of policy opinion from independent sources, again with the aim of charting an optimal policy course and at the same time gauging the degree of polarization on any given policy issue. In 2004, James Surowiecki provided an engaging account bearing on all this in his wonderful book The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations.

When it comes to the ensemble forecasts, those using the technique are free to choose which numerical weather prediction (NWP) models will be included and which will be excluded. The choice might be made on the basis of the rigor of the science used in the model, its individual performance based on quantitative metrics, its cost, its transparency, accessibility, and other benchmarks. Those constructing the individual NWP models making up such ensembles are presumably motivated by similar criteria.

In the same way, when contributing our perspectives to any policy discussion, you and I might do well to discipline ourselves bit… to reflect on the legitimacy of our personal thought process, as well as the legitimacy of the overall procedure, involving collection and aggregation of many individual views. We shouldn’t engage irresponsibly. We shouldn’t feel free to suggest whatever comes to mind, or what serves our short-term interests at the expense of the others. We shouldn’t feel free to suppress dissent. And so on.

My colleague Paul Higgins, Director of the American Meteorological Society’s Policy Program and a Senior Policy Fellow, has given this some thought. He’s articulated a set of criteria for establishing legitimacy in the expression of views, and has passed it around for informal reaction.

Paul emphasizes that this material is “draft,” but to me and several others it looks pretty good! At my request, he’s graciously allowed me to reproduce his list here, in hopes of getting more feedback. In particular please look at it through the lens of its bearing on ensemble formulation of policy (regulation, or legislation, for example). Here’s what Paul has to say:

There is great need for the scientific community to effectively and credibly express views on a range of topics. Such views may include objective assessments of scientific knowledge and understanding or subjective expressions of opinion (e.g., community goals and priorities and what “should” be).

In broad general terms, the legitimacy (and power) of views expressed by any group is enhanced through a process that ensures:

1)      The full range of credible and defensible views contained within the community are fully and fairly considered in the assessment process

2)      Relevant external views are sought and considered fairly

3)      Minority held views are sought and, when credible, included

4)      All participants have an equal opportunity to contribute (NB: not the same as saying all participants will contribute equally because …)

5)      All suggestions are assessed and included (or excluded) based on their merit (i.e., their substantive contribution to the assessment’s validity)

6)      Views expressed are free from self-interest to the maximum extent possible

7)      Any notable remaining self-interest is identified explicitly and clearly within the assessment itself

8)      The potential for participation bias in the assessment process is recognized and accounted for (e.g., the AMS Policy Statement on Geoengineering recognized the possibility that only proponents of geoengineering would be willing to invest the time required of participants)

9)      The group possesses (or has access to) sufficient subject matter expertise to fairly and accurately assess all relevant information

10)   Opportunities for broader community input and review are included to the extent possible (e.g., member/public comment periods)

11)   Independent validation and oversight occurs when possible (e.g., the AMS Council provides this role in the AMS statement process)

12)   Criticism and dissent are welcome and encouraged

13)   Scientists wishing to provide criticism or dissenting views are able to do so in ways that guard against potential retribution or the potential perception of it

14)   Hard choices and sacrifices do not involve non-participants in the process (i.e., such calls are not credible)

15)   Assessment efforts are led by a credible, trusted, and capable source (i.e., individuals and institutions committed to the principles articulated here and capable of ensuring they are met to the maximum extent possible)

Your thoughts? Paul (and I) would welcome your comments and suggestions. You can reach him at or comment on this post. Or both!

This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Building the legitimacy of ensemble” approaches to policy formulation.

Leave a Reply

Your email address will not be published. Required fields are marked *