OUP user menu

The role of the reviewer in editorial decision-making

Ruben Coronel, Tobias Opthof
DOI: http://dx.doi.org/10.1016/S0008-6363(99)00177-7 261-264 First published online: 1 August 1999

Abstract

Scientific journals are confronted with increasing numbers of submitted manuscripts. Because of page limitations, these journals have to revert to a high rejection rate. The standard form accompanying manuscripts submitted to Cardiovascular Research was changed in order to entice reviewers in an advisory behaviour more suited to the editorial rejection rate. All original manuscripts submitted to Cardiovascular Research during the previous four years were analyzed. The recommended rejection rate of individual reviewers was 23% and, with increasing experience, reviewers become more critical. The reviewer’s recommendations and indication of scientific priority did not match the journal’s rejection rate. An effort to lure the reviewers into a more critical advisory behaviour was effective. However, unanimity between reviewers is rare. The reviewer’s recommendations do not justify high rejection rates and a wider dissemination of scientific information in the biomedical field is called for.

1 Introduction

As the pressure to publish increases, scientific journals are confronted with increasing numbers of manuscripts submitted for publication. Publishers and editors have to meet page budgets and, consequently, increase rejection rates. In some fields of science, publishing takes place on the Internet, which precludes the need for page limitations and offers the opportunity to update the publication and to append comments to it. Editors of printed scientific journals, however, are challenged to select from the submitted papers those that best match their readership and will be cited best. In order to select the submitted material on the basis of quality criteria, they invoke the help of the author’s peers. This peer-review process usually involves a written report (which is in part confidential), an indication of the scientific priority of the submitted paper, and a recommendation.

In past years, Cardiovascular Research has been compelled to negotiate a 25% increase in submitted papers [1]. Of course, this means hardship to the authors as well as to the editors because, often, valuable manuscripts have to be returned to the authors. The decision to reject a paper for publication in the journal is particularly difficult if the reviewers are not unanimous in their recommendations and if the reviewer’s recommended rejection rate differs from the journal’s rejection rate. Recently, the editors of Cardiovascular Research have tried to tune the reviewer’s indication of priority of a manuscript to the journal’s targeted rejection rate by changing the standard form sent to reviewers.

This paper reports on the (combined) recommendations of our reviewers and on their response to a change in the predefined options for the indication of scientific priority.

2 Methods and results

Manuscripts submitted to Cardiovascular Research are initially evaluated by the editors. In about 6% of the manuscripts, this leads to a unanimous decision of unsuitability. Since the change of the editorship of Cardiovascular Research in 1995, 2607 original manuscripts have been submitted to the peer-review process. Cardiovascular Research sends each manuscript to three reviewers for evaluation and tries to retrieve the reports after 21 days [2]. From 2138 (82%) of the manuscripts, three (or four) reviewer’s reports were returned to the editors, in 447 (17%) cases, two reports and in the remaining 22 cases, only one report was returned. RCP Manuscript Tracking System (RCP Consultants Ltd., Oxford, UK) is used for administrative purposes and to monitor the editorial process.

Fig. 1 shows part of a standard form sent to the reviewers of each manuscript submitted to Cardiovascular Research. The reviewer is asked to tick one of the boxes for the indication of scientific priority and one of the options in the ‘recommendation’ box. Not all reviewers are satisfied with the two options and add a box indicating intermediate priority, or do not indicate a priority at all. Initially, the reviewers were offered the choice between ‘high’ and ‘low’ priority. In 1998, the form was changed in order to entice the reviewers into a distribution of priority that matched the Journal’s rejection rate (priority upper 20% or lower 80%) more closely. Table 1 shows that the change in the form caused a statistically significant increase in the percentage of low priority ratings and a statistically significant decrease in the percentage of indifferent (intermediate or absent) priority indications. The percentage of high priority indications did not change significantly.

Fig. 1

Section of the standard form accompanying a manuscript admitted to the peer-review process of Cardiovascular Research.

View this table:
Table 1

Priority ratings of manuscripts

Priority indication‘High’ or ‘upper 20’ (%)No/medium (%)‘Low’ or ‘lower 80’ (%)Total
Before change of form1623 (29.8)1178 (21.7)2638 (48.5)5439
After change of form583 (30.6)236 (12.4)1085 (57.0)1904
n.s.P<0.01P<0.01
RecommendationDo not rejectReject (%)Total
Before change of form42011238 (22.8)5439
After change of form1452452 (23.7)1904
n.s.n.s.

The recommendation section of the form was left unchanged and the distribution of the recommendation (reject/do not reject) before and after the change of the form did not alter.

Table 2 shows the distribution of the reviewer’s recommendations (manuscripts with three or four reports returned only). The majority of recommendations are aimed at eventually accepting the manuscript and only 23% are aimed at rejecting the paper. The overall recommended ‘rejection’ rate of the reviewers thus strongly deviates from the editorial rejection rate of 75% of original papers. Fig. 2 shows the paired indication of scientific priority (combined before and after changing the form) and of the reviewer’s recommendation. It is clear that the two parameters are not independent, a low priority is mostly associated with a recommendation of rejection.

Fig. 2

Graph showing the combined priority indications (‘high’ or ‘low’) and the reviewer’s recommendations (minor revision, major revision, major revision with experiments, reject) of all reviewers’ reports of original papers submitted to Cardiovascular Research between 1995 and 1999.

View this table:
Table 2

Distribution of reviewer’s recommendations

RecommendationNumber%
None2173.0
Accept with minor revision200927.4
Reconsider after major revision169823.1
Reconsider after major revision and experiments172923.5
Reject169023.0

The average rejection rate of experienced reviewers (n=108, with more than ten manuscripts in their track record) was 27.6% (range from 0–80). Reviewers with a single manuscript in their track record had a recommended rejection rate of 20.9% (chi2-test, P<0.01 vs. experienced reviewers).

A system of peer review based on three reviewers, as utilized by Cardiovascular Research, implicates the need to weigh the various recommendations, unless the reports are unanimous. Table 3 demonstrates how rarely unanimity is reached in priority indication (low and not low) and recommendation (reject and do not reject). In more than 80% of manuscripts, there is a majority amongst the reviewers against rejection of the manuscript and, in about 50% of the manuscripts, a majority of the reviewers considered the priority of the manuscript as not being low. Table 3 also includes the editorial rejection rates of the manuscripts in each category. Where unanimity amongst reviewers regarding rejection or a low priority is present (in a minority of manuscripts), the editors have followed the reviewer’s advise. However, in the other groups, the editorial rejection rates are much higher than those according to the reviewer’s recommendations and priority indications.

View this table:
Table 3

Recommendations and priority indications per manuscript (three reviewer reports received)

Reviewer’s recommendationsEditorial decision rejection rate (%)
RejectDo not rejectManuscripts
Number%
30673.1100
2128413.399.6
1269632.688.6
03109151.056.3
2138100
Reviewer’s priority indicationEditorial decision rejection rate (%)
LowNot lowManuscripts
Number%
3035316.599.4
2174034.689.6
1270032.763.4
0334516.135.7
2138100

3 Discussion

Editorial decision-making is directed at selecting the best manuscripts from the submitted material and, with printed journals, heavily depends on the peer-review process. In general, the reviewers provide confidential comments to the editors (including an indication of scientific priority and a recommendation) and comments to the authors. Many scientific journals are forced to reject the majority of the submitted material and their editors make decisions based on their own opinion of the paper in combination with the reviewer’s recommendations.

This study on the advisory behaviour of reviewers of a representative biomedical scientific journal demonstrates the overriding willingness of peers to contribute to the improvement of a submitted manuscript rather than to advise rejection of a manuscript. The editors are thus facing the task of rejecting more papers than seems justified based on the reviewer’s recommendations and their indication of priority. Of course, the confidential comments to the editors may provide other arguments that may help in the process of selecting the best papers. From the degree of unanimity between the reviewers (if multiple reviewer’s reports are available), the editors may extract additional information for the decision process.

This study also shows (1) that with increasing experience, reviewers become more critical, (2) that efforts to lure reviewers into a recommendation performance more fitting to the editorial selection process have an effect, albeit small, and (3) that unanimity between multiple reviewers is rare.

The (combined) reviewers’ recommendations do not seem to justify the high rejection rates of scientific journals but call for a wider dissemination of scientific information, at least in the biomedical field. Publication on the world wide web may offer the opportunity for this. The peer review process may then serve as a quality criterion, aiding in the recovery of publications from large electronic journals.

Acknowledgements

The authors gratefully acknowledge the other editors of Cardiovascular Research (M.J. Janse, J.M.T. de Bakker, J.W.T. Fiolet, M.M. Levi, M. Pfaffendorf, A.A.M. Wilde) for their help and the editorial managers Nicole Mommertz, Yvonne Zwiers, Anne van der Valk and Maaike van Dartel for their conscientious work on the database.

References

  1. [1]
  2. [2]
View Abstract