Peer review – reviewed.

We’re entering a time when the process of scientific peer review will likely undergo a careful scrutiny from all sides. For example, the UK Parliament has just launched one such inquiry. This is serious business, and merits considered thought, from all quarters, for an extended period. We want to get this right.

Why? Because peer review is a fundamental component of the hygiene of science.

By hygiene we mean the whole of those habits and ways of doing business that keep science healthy – that distinguish science from unfounded opinion and conjecture, from wishful thinking, from fantasy and delusion, from lies and falsehoods. These activities would include, for example, the development of verifiable hypotheses, the test of those hypotheses by experiment(s), the evaluation and articulation of the uncertainties in the work and conclusions, the publication of the findings in peer-reviewed literature for all to critique and evaluate, efforts to duplicate the findings in independent work, explorations of the implications of those findings and tests of those implications – and on and on.

What is peer review? Just what the name implies. It means this: before a paper is published, experts in the field are given an opportunity to examine the paper thoroughly, to check it for content, for errors in logic, for the consistency of its conclusions with data and experiment, for any mis-representations, for originality, for appropriate acknowledgment of relevant prior and contemporary work, etc. The list is extensive, and probably not completely captured here. But you get the flavor.

Note that experts in the field may often be in competition with the author or authors of the work in question: for funding, for scientific priority, for prestige and recognition. The stakes are high! Nevertheless, they’re expected to put these considerations aside. Because the stakes are high, reviewers are usually but not always guaranteed anonymity.

Sounds complicated, and tenuous at best. How can this possibly work? Here’s an analogy that might help – one that may be familiar to many readers.

When I was younger I played a lot of basketball. Only a small fraction of these games were actually run by unbiased third parties, a.k.a. referees. Virtually all my basketball was of the pick-up variety – street games.

We called our own fouls and kept our own score.

This worked well. At the time I didn’t think it in any way remarkable. In hindsight it seems amazing.

Most of the violations were called by the person fouled: “foul!” Maybe 95% of the time, these calls were accepted without argument. Many times, the foul would first be called by the person committing the foul: “I fouled him! Their ball!” without waiting for protest from the other side. Sometimes the foul would be called by another player. Occasionally, a call would be overruled by others, usually to their own sides’ detriment. And whether the score was close or not, whether it was near the end of the game or not, play would swiftly move on. There was very little argument. We all wanted to play ball!

Peer review is very much – though not entirely – like that. It’s pretty easy to see both the advantages and limitations of this approach. The situation is not unlike what Winston Churchill once famously said, in reference to the political arena: “It has been said that democracy is the worst form of government – except all the others that have been tried.”

This entry was posted in Uncategorized. Bookmark the permalink.

4 Responses to Peer review – reviewed.

  1. Roger Caiazza says:

    I do not disagree with anything you said. However something that should be kept in mind is that science for policy implementation requires another level of expectations for review. A very large percentage of published science addresses topics that do not have direct implications to society and the peer review system that you describe is sufficient. However, when the science is being used to justify a regulation that could lead to large direct costs or life-style changes then complete transparency is necessary. For example, the epidemiological studies used to justify the level of the limits for the National Ambient Air Quality Standards have been re-analyzed numerous times and each “side” in the debate provided enough information so that their work could be completely reproduced. Ultimately, the follow-up analyses identified problems with the original work that led to follow-up research programs which strengthened the science. Of course, we do not have the resources to review all science to that level but there are times when this is necessary.

    One of the climate change controversies is that the EPA endangerment finding that CO2 was a pollutant was unique in that it did not provide that level of transparency in its supporting documentation. Instead, EPA basically quoted the IPCC documents and said that because they were peer-reviewed that was sufficient. Given that the ultimate financial and societal stakes are arguably larger than the ambient standard rule-makings, I do not think that was appropriate. I strongly suspect that many of the climate scientists embroiled in the “Climategate” debate for example, did not understand the expectation of critical reviewers in a regulatory context where transparency beyond the peer review system you described is required. When there were calls for the data and code, they were unprepared. Sadly the result is that personal conflicts have gone beyond the pale and now there is even litigation involved.

    Consider one example in the climate change arena, the instrumental data record. I believe that the analyses used for regulatory proceedings should provide all the original raw data and sufficient information to be able to reproduce how the original raw data was modified for use in the analysis. If that information is not available for whatever reason then it should not be used as the basis for the rule. One clarification is that I think that the original code with sufficient comment statements to describe what the author was doing is the easiest way to do this. I don’t think it means that the author needs to provide a working computer program for a reviewer to use. The alternative is to provide a complete flow chart but in my experience that nearly doubles the work load (I freely admit that approach likely marks me as a terrible programmer).

  2. Zajko says:

    Some 20 years old now but still a relevant read on peer-review, its uses in policy formation, and associated problems:
    Peerless science: Peer review and U.S. science policy
    By Daryl E. Chubin, Edward J. Hackett

  3. Pingback: Your peers’ view of your work? Maybe that’s not the view that matters most. | Living on the Real World

  4. Pingback: “Error Cascade” | Living on the Real World

Leave a Reply to Zajko Cancel reply

Your email address will not be published. Required fields are marked *