Peer review, role of editors, open access and much more

John Hawks on the role of editors of scholarly journals:

The editor’s role is to evaluate the manuscript’s suitability for publication — does it conform to the journal guidelines? Is it scientifically valid? Does it cite the existing literature appropriately? Do the observations support the claims made? A few manuscripts may be rejected immediately, because they fail to meet basic criteria of scientific value or readability.

Most manuscripts require the editor to seek out the opinions of additional experts in the process of peer review.

Since the paper that Hawks discusses in the post is published in PLoS, you can also have an access to the peer reviews that the paper received (in its original format), here and here. As you can see, the reviews are very different in their conclusions. Here is the anonymous reviewer:

 This paper is unacceptable for a variety of reasons and has so many fatal flaws that there are no reasons to ask for revisions. When I accepted the invitation for review, I anticipated a well-reasoned, well-documented manuscript, but this paper is the opposite. I list only a few of the problems, but it is full of citation errors, unreliable correlations, statistical manipulations and lacks essential documentation.

In short, this paper is so full of errors and misinterpretations that it is completely unacceptable. At best they have discovered on Palau some small individuals, but this is not well documented by them and not especially important since they are also found on Flores and other SE Asian islands. It adds little but confusion to the issue of the taxonomic position of the Liang Bua material.

Here is the second reviewer Robert B Eckhardt:

To begin with, I suspect that many of the comments that will be written by others about “Small-bodied humans from Palau, Micronesia” by Berger, Churchill, De Klerk and Quinn will be devoted to critical comments focusing on what the Palau material described here is not: Very likely it will be said by more than a few paleoanthropologists that the Palau sample is not pertinent to tests of hypotheses about the Liang Bua Cave skeletons from Flores, particularly that of the most complete specimen found there, LB1. I would be surprised, in fact, if the majority of the comments on this paper are not negative. Since the beginning late in 2004 of the controversy over the Flores skeletons, my estimate is that roughly 80% of those who consider themselves to be paleoanthropologists think that “Homo floresiensis” is a valid new species of hominin. Judging from the array of papers and posters presented at the 2007 Annual Meeting of the American Association of Physical Anthropologists, this percentage distribution has remained fairly constant for about three years.

Against this background, from the experience of the last several years, members of our own international research group (for membership in which see Berger, et al., reference 4) often have encountered such illuminating scientific comments on our work from Morwood group collaborators as “rubbish” (too often to bother tabulating), and such fascinating morphological assessments as “Robert Eckhardt is thick as a plank,” (Peter Brown, January 2006 Discover magazine [this characterization has been falsified, however, since in a subsequent scientific meeting at the University of Pennsylvania, 10 February 2006, my wife, Carey, used an anthropometer to demonstrate that I am, in fact, thicker than two short planks]). Just to make sure that everyone in the game understood that what sport aficionados refer to as “trash talk” was officially endorsed, Nature (31 August 2006) “warmly welcomed” [their phrase] our group’s detailed paper in Proceedings of the National Academy of Sciences of the United States (PNAS) [reference 4 in Berger, et al.] under the editorial title “Rude paleoanthropology.” Against this background of experience, I suggest that Berger, et al., as well as readers of this journal, be ready for all sorts of attempts at dismissal of the work at hand and its importance. Forewarned is forearmed.

Rather than what it is not, though, we should begin with what the paper by Berger, et al., is; their own words serve just fine in this regard: “We feel that the most parsimonious, and most reasonable, interpretation of the human fossil assemblage from Palau is that they derive from a small-bodied population of H. sapiens (representing either rapid insular dwarfism or a small-bodied colonizing population), and that the primitive traits that they possess reflect either pliotropic [sic] or epigenetic correlates of developmental programs for small body size.” Much of the rest of their paper describes the geographic and temporal settings, plus some detailed, professionally competent, morphological descriptions of the Palau skeletal material. There is no need to repeat those descriptions here, but they are well worth reading, and re-reading.

Interesting, isn’t it?

In any case, there is more in Hawks’ post about the paper, its contents, the reviews it received, and the associated politics; for example, here is Hawks on the comment that the peer review of the paper was influenced by media reports:

Dalton emphasized the media attention to the find, particularly focusing on the role of the National Geographic Society. NGS produced a documentary about Berger’s work on Palau (he is an NGS grantee).

In this case, National Geographic funded the work and apparently produced a documentary about it. Their production wasn’t disclosed to the journal, and I view it as irrelevant to the scientific evaluation of the manuscript.

Paleoanthropologist Tim White is quoted in Dalton’s story, saying that it appears that the “review process [was] driven by popular media.” Since White was not involved in the review process of this paper, he obviously is just speculating.

I tend to give him the benefit of the doubt, since in this story it appears that Dalton was trying to play up any contrary quotes about the findings. Why else would he run otherwise-uninformed comments of the kind in the story?

I would tend instead to ask these questions: Does the Nature Publishing Group (NPG), in publishing Rex Dalton’s piece, have a vested interest in the credibility of their own journals, in comparison to open access outlets like PLoS? Do NPG journals regularly receive manuscripts and publish them based on the associated media attention? Do they have an interest in pressuring grant agencies, like the NGS, into encouraging submission of manuscripts to NPG journals instead of alternate outlets? Does NPG have a well-established record of running stories questioning the value of open access publications?

A very interesting piece; take a look!



2 Responses to “Peer review, role of editors, open access and much more”

  1. Training for peer review « Entertaining Research Says:

    […] can also be used not only to show how to review but also to teach how not to review. By the way, as I noted earlier, PLoS journals which give open access to reviews are also ideal for such a course, though the fact […]

  2. William Says:

    All very interesting, but consider using “because” (a reason) in place of “since” ( a measure of time).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: