https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798

J R Soc Med. 2006 Apr; 99(4): 178–182.

doi: 10.1258/jrsm.99.4.178

Peer review: a flawed process at the heart of science and journals

同行评议:科学和期刊中的一个重要瑕疵

作者：Richard Smith；

英汉对照

Peer review is at the heart of the processes of not just medical journals but of all of science. It is the method by which grants are allocated, papers published, academics promoted, and Nobel prizes won. Yet it is hard to define. It has until recently been unstudied. And its defects are easier to identify than its attributes. Yet it shows no sign of going away. Famously, it is compared with democracy: a system full of problems but the least worst we have.

同行评议不仅是医学期刊的核心，也是所有科学研究的核心。它是分配拨款、发表论文、职称评定和获得诺贝尔奖的评判方法。然而，他能起到的真正作用很难确定。它的功能一直没有被认真研究过。它的缺陷比它的属性更容易识别。然而，没有迹象它将被抛弃。同行评议堪比与民主体制: 一个充满问题的制度，但似乎是最不坏的制度。

When something is peer reviewed it is in some sense blessed. Even journalists recognize this. When the BMJ published a highly controversial paper that argued that a new `disease', female sexual dysfunction, was in some ways being created by pharmaceutical companies, a friend who is a journalist was very excited—not least because reporting it gave him a chance to get sex onto the front page of a highly respectable but somewhat priggish newspaper (the Financial Times). `But,' the news editor wanted to know, `was this paper peer reviewed?'. The implication was that if it had been it was good enough for the front page and if it had not been it was not. Well, had it been? I had read it much more carefully than I read many papers and had asked the author, who happened to be a journalist, to revise the paper and produce more evidence. But this was not peer review, even though I was a peer of the author and had reviewed the paper. Or was it? (I told my friend that it had not been peer reviewed, but it was too late to pull the story from the front page.)

当某件事被同行评议时，被评审者祝福自己走运。就连记者也意识到了这一点。当《英国医学杂志》发表了一篇极具争议的论文，认为一种新的“疾病”——女性性功能障碍，在某种程度上是由制药公司制造出来的时候，一位记者朋友非常兴奋，尤其是因为报道这篇文章让他有机会在一家非常受人尊敬但有点自命堂皇的报纸(《金融时报》)的头版上报道性行为。“但是，”新闻编辑要的是，“这篇论文经过同行评审了吗?”言下之意是，如果它通过了同行评审，那就足够好可以出现在头版上;如果它没有经过同行评审，那就不应该出现在头版上。是吗?我对这篇文章的阅读要比阅读许多论文仔细得多，我还请作者修改这篇文章并提供更多的证据，而作者恰好是一名记者。但这不是同行评议，尽管我是作者的同行，也审阅了这篇论文。这篇文章通过了同行评议吗?(我告诉我的朋友，这篇文章没有经过同行评议，但要把它从头版撤下已经太晚了。)

WHAT IS PEER REVIEW

什么是同行评议

My point is that peer review is impossible to define in operational terms (an operational definition is one whereby if 50 of us looked at the same process we could all agree most of the time whether or not it was peer review). Peer review is thus like poetry, love, or justice. But it is something to do with a grant application or a paper being scrutinized by a third party—who is neither the author nor the person making a judgement on whether a grant should be given or a paper published. But who is a peer? Somebody doing exactly the same kind of research (in which case he or she is probably a direct competitor)? Somebody in the same discipline? Somebody who is an expert on methodology? And what is review? Somebody saying `The paper looks all right to me', which is sadly what peer review sometimes seems to be. Or somebody pouring all over the paper, asking for raw data, repeating analyses, checking all the references, and making detailed suggestions for improvement? Such a review is vanishingly rare.

从操作上看，同行评议没有定义(无论这是否能叫同行评议，如果50个人持基本相同的看法就通过)。因此，同行评议就像诗歌、爱情或正义。同行评议似乎是指：资助申请或论文被第三方审查，第三方既不是作者，也不是判断是否应该给予资助或是否应该发表论文的人。但谁是同行呢?做完全相同研究的人(在这种情况下，同行可能是直接竞争对手)?同一学科的人?是方法论方面的专家吗?什么是评议? 很遗憾同行评议有时似乎就是'我看这篇论文不错'。或者评议是指有人在论文上滔滔不绝，索要原始数据，要求重复分析，核对所有参考文献，并提出详细的改进建议?这样的评审非常罕见。

What is clear is that the forms of peer review are protean. Probably the systems of every journal and every grant giving body are different in at least some detail; and some systems are very different. There may even be some journals using the following classic system. The editor looks at the title of the paper and sends it to two friends whom the editor thinks know something about the subject. If both advise publication the editor sends it to the printers. If both advise against publication the editor rejects the paper. If the reviewers disagree the editor sends it to a third reviewer and does whatever he or she advises. This pastiche—which is not far from systems I have seen used—is little better than tossing a coin, because the level of agreement between reviewers on whether a paper should be published is little better than you'd expect by chance.

很明显，同行评议的形式千变万化。可能每个期刊和每个拨款机构的系统至少在某些细节上是不同的;有些系统是非常不同的。甚至可能有一些期刊使用以下经典系统。编辑看了看标题，把它发给了两个朋友，编辑认为他们对这方面有所了解。如果两人都建议发表，编辑就把它交付印刷。如果两人都反对文章发表，编辑就会拒绝这篇论文。如果两个审稿人意见不同，编辑将其发送给第三位审稿人，并按照他或她的建议去做。这是通常的作法，比抛硬币好不了多少，因为审稿人之间就一篇论文是否应该发表达成的一致程度，只偶然机遇问题。

That is why Robbie Fox, the great 20th century editor of the Lancet, who was no admirer of peer review, wondered whether anybody would notice if he were to swap the piles marked `publish' and `reject'. He also joked that the Lancet had a system of throwing a pile of papers down the stairs and publishing those that reached the bottom. When I was editor of the BMJ I was challenged by two of the cleverest researchers in Britain to publish an issue of the journal comprised only of papers that had failed peer review and see if anybody noticed. I wrote back `How do you know I haven't already done it?'

这就是为什么20世纪《柳叶刀》伟大的编辑罗比·福克斯(Robbie Fox)不喜欢同行评议，他想知道，如果他把标记为“发表”和“拒绝”的那堆东西交换一下，没有人会有人注意到。他还嘲笑《柳叶刀》的评审制度类似把一堆论文往楼梯下扔，然后发表落到楼下的论。当我还是《英国医学杂志》(BMJ)的编辑时，英国两位最聪明的研究人员向我挑战说，如果我出版一期杂志，全是同行评议被拒的论文，看看能不能有人注意到。我回信说:“你怎么知道我还没有做过?”

DOES PEER REVIEW `WORK' AND WHAT IS IT FOR

同行评议“有效”吗?它的目的是什么

But does peer review `work' at all? A systematic review of all the available evidence on peer review concluded that `the practice of peer review is based on faith in its effects, rather than on facts'.2 But the answer to the question on whether peer review works depends on the question `What is peer review for?'.

但是同行评议真的“有效”吗?对同行评议的所有现有证据进行系统评估后得出结论:“同行评议的实践是基于对其效果的盲目相信，而不是基于事实“。但在评估同行评议是否有效之前，先要回答“同行评议的目的是什么?”

One answer is that it is a method to select the best grant applications for funding and the best papers to publish in a journal. It is hard to test this aim because there is no agreed definition of what constitutes a good paper or a good research proposal. Plus what is peer review to be tested against? Chance? Or a much simpler process? Stephen Lock when editor of the BMJ conducted a study in which he alone decided which of a consecutive series of papers submitted to the journal he would publish. He then let the papers go through the usual process. There was little difference between the papers he chose and those selected after the full process of peer review. This small study suggests that perhaps you do not need an elaborate process. Maybe a lone editor, thoroughly familiar with what the journal wants and knowledgeable about research methods, would be enough. But it would be a bold journal that stepped aside from the sacred path of peer review.

同行评议似乎是一种选择最佳资助申请和最佳论文发表在期刊上的方法。但是这一目标很难验证，凭什么说一篇是好论文或一份研究计划是好的计划。另外，同行评议的测试依据是什么?偶然机会吗?还是同行评议是一个比较简单的方法?斯蒂芬·洛克(Stephen Lock)，当时《英国医学杂志》(BMJ)的编辑进行了一项研究，在这项研究中，他独自决定在连续一系列提交给该杂志的论文中发表哪一篇。然后他让这些文件走常规程序。他选择的论文与经过全程同行评议后选出的论文几乎没有差别。这项小型研究表明，也许你不需要一个复杂的同行评议过程。也许一个完全熟悉期刊的需求，了解研究方法的编辑就足够了。但这没有一本期刊敢于抛弃同行评议的神圣之路。

Another answer to the question of what is peer review for is that it is to improve the quality of papers published or research proposals that are funded. The systematic review found little evidence to support this, but again such studies are hampered by the lack of an agreed definition of a good study or a good research proposal.

同行评议似乎可以提高发表的论文或资助的研究计划的质量。系统评价发现几乎没有证据支持这一点。无法获得证据的原因是无法定义什么是“好研究”或“好研究计划”。

Peer review might also be useful for detecting errors or fraud. At the BMJ we did several studies where we inserted major errors into papers that we then sent to many reviewers. Nobody ever spotted all of the errors. Some reviewers did not spot any, and most reviewers spotted only about a quarter. Peer review sometimes picks up fraud by chance, but generally it is not a reliable method for detecting fraud because it works on trust. A major question, which I will return to, is whether peer review and journals should cease to work on trust.

同行评议被认为有助于发现错误或欺诈。在英国医学杂志，我们做了几项研究，在论文中插入主要错误，然后发给审稿人。没有人能发现所有的错误。一些审稿人一个也没有发现，一些审稿人发现了大约四分之一。同行评议有时会偶然发现欺诈行为，但一般来说，它不是一种可靠的检测欺诈的方法，因为这主要依赖于诚信。问题是，期刊是否应该停止信任同行评议。

THE DEFECTS OF PEER REVIEW

同行评议的缺陷

So we have little evidence on the effectiveness of peer review, but we have considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.

因此，我们几乎没有证据表明同行评议的有效性，但是关于其缺陷则有相当多的证据。同行评议除了在检测重大错误方面表现不佳，在检测欺诈方面几乎毫无用处之外，它还缓慢、昂贵、浪费学术时间、高度主观、有点像彩票、容易受偏见左右，而且很容易被滥用。

Slow and expensive

缓慢而昂贵

Many journals, even in the age of the internet, take more than a year to review and publish a paper. It is hard to get good data on the cost of peer review, particularly because reviewers are often not paid (the same, come to that, is true of many editors). Yet there is a substantial `opportunity cost', as economists call it, in that the time spent reviewing could be spent doing something more productive—like original research. I estimate that the average cost of peer review per paper for the BMJ (remembering that the journal rejected 60% without external review) was of the order of £100, whereas the cost of a paper that made it right though the system was closer to £1000.

即使在互联网时代，许多期刊也需要一年多的时间来评审和发表一篇论文。很难得到关于同行评议成本的数据，特别是审稿通常是无偿的(许多编辑也是如此)。特别是花在审稿上的时间本可以花在更有成就的事情上，比如原创性研究。我估计，《英国医学杂志》每篇论文的同行评议平均成本(该杂志有60%的稿件在没有进入外部评审)约为100英镑，而一篇通过同行评议的论文的成本接近1000英镑。

The cost of peer review has become important because of the open access movement, which hopes to make research freely available to everybody. With the current publishing model peer review is usually `free' to authors, and publishers make their money by charging institutions to access the material. One open access model is that authors will pay for peer review and the cost of posting their article on a website. So those offering or proposing this system have had to come up with a figure—which is currently between $500-$2500 per article. Those promoting the open access system calculate that at the moment the academic community pays about $5000 for access to a peer reviewed paper. (The $5000 is obviously paying for much more than peer review: it includes other editorial costs, distribution costs—expensive with paper—and a big chunk of profit for the publisher.) So there may be substantial financial gains to be had by academics if the model for publishing science changes.

由于开放获取 (open access movement)，同行评议的成本变得重要起来。开放获取希望让所有人都能免费获得研究成果。在目前的出版模式下，同行评议对作者来说通常是“免费的”，出版商通过向机构收取访问的费用来赚钱。一种开放获取模式是，作者将支付同行评议和在网站上发表文章的费用。因此，那些提供或提议开放获取的人需要同行评议的成本——目前每篇文章的价格在500到2500美元之间。根据那些推动开放获取系统的人的计算，目前学术界为一篇同行评议的论文支付大约5000美元。(这5000美元显然比同行评议支付的费用要多得多:它包括其他编辑成本、发行成本(纸张成本高昂)以及出版商的一大块利润。)因此，如果出版科学的模式发生变化，学术界可能会获得可观的经济收益。

There is an obvious irony in people charging for a process that is not proved to be effective, but that is how much the scientific community values its faith in peer review.

对一个没有被证明有效的过程收费，这显然是一种讽刺，但这就是科学界对同行评议的信仰而造成的。

Inconsistent

文章质量与同行评议结果不一致

People have a great many fantasies about peer review, and one of the most powerful is that it is a highly objective, reliable, and consistent process. I regularly received letters from authors who were upset that the BMJ rejected their paper and then published what they thought to be a much inferior paper on the same subject. Always they saw something underhand. They found it hard to accept that peer review is a subjective and, therefore, inconsistent process. But it is probably unreasonable to expect it to be objective and consistent. If I ask people to rank painters like Titian, Tintoretto, Bellini, Carpaccio, and Veronese, I would never expect them to come up with the same order. A scientific study submitted to a medical journal may not be as complex a work as a Tintoretto altarpiece, but it is complex. Inevitably people will take different views on its strengths, weaknesses, and importance.

人们对同行评议有很多幻想，其中最强大的幻想之一就是它是一个高度客观和可靠的评价过程。我经常收到一些作者的来信，他们对英国医学杂志拒绝了他们的论文却发表了比他们差得多的论文。他们总能看到一些不光彩的事。他们发现很难接受同行评议是一个主观的，因此不能给出正确评价的过程。但期望同行评议给出客观评价是不可能做到的。如果我让人们给画家Titian, Tintoretto, Bellini, Carpaccio, and Veronese排序，他们不会给出同样的顺序。提交给医学杂志的科学研究可能不像Tintoretto的祭坛画那样复杂，但它确实很复杂。不可避免地，人们会对它的优点、缺点和重要性有不同的看法。

So, the evidence is that if reviewers are asked to give an opinion on whether or not a paper should be published they agree only slightly more than they would be expected to agree by chance. (I am conscious that this evidence conflicts with the study of Stephen Lock showing that he alone and the whole BMJ peer review process tended to reach the same decision on which papers should be published. The explanation may be that being the editor who had designed the BMJ process and appointed the editors and reviewers it was not surprising that they were fashioned in his image and made similar decisions.)

因此，发表论文决定于审稿人的评价就如同投彩票一样。(我意识到，这一证据与斯蒂芬·洛克的研究相冲突，后者表明，他本人和整个《英国医学杂志》的同行评议过程往往会对哪些论文应该发表做出相同的决定。解释可能是，作为设计《英国医学杂志》同行评议并指派了编辑和审稿人的编辑，并不奇怪他为他的同行评议下的结论。)

Sometimes the inconsistency can be laughable. Here is an example of two reviewers commenting on the same papers.

可笑的是两个审稿人对同一篇稿件给出完全不同的评价：

Reviewer A: `I found this paper an extremely muddled paper with a large number of deficits'

审稿人A:“这篇论文非常混乱，有很多错误。”

Reviewer B: `It is written in a clear style and would be understood by any reader'.

审稿人B:“该文写作风格清晰，任何读者都能理解。”

This—perhaps inevitable—inconsistency can make peer review something of a lottery. You submit a study to a journal. It enters a system that is effectively a black box, and then a more or less sensible answer comes out at the other end. The black box is like the roulette wheel, and the prizes and the losses can be big. For an academic, publication in a major journal like Nature or Cell is to win the jackpot.

这种可能不可避免的不一致性会让同行评议变得有点像碰运气。你向期刊提交一份研究报告。它进入一个实际上是一个黑匣子，然后一个或多或少明智的答案从另一端出来。黑盒子就像轮盘赌，奖金和损失相差很大。对于一名学者来说，在《自然》或《细胞》等主要期刊上发表论文就等于中了头彩。

Bias

偏见

The evidence on whether there is bias in peer review against certain sorts of authors is conflicting, but there is strong evidence of bias against women in the process of awarding grants. The most famous piece of evidence on bias against authors comes from a study by DP Peters and SJ Ceci. They took 12 studies that came from prestigious institutions that had already been published in psychology journals. They retyped the papers, made minor changes to the titles, abstracts, and introductions but changed the authors' names and institutions. They invented institutions with names like the Tri-Valley Center for Human Potential. The papers were then resubmitted to the journals that had first published them. In only three cases did the journals realize that they had already published the paper, and eight of the remaining nine were rejected—not because of lack of originality but because of poor quality. Peters and Ceci concluded that this was evidence of bias against authors from less prestigious institutions.

关于同行评议中对某些类型的作者是否存在偏见的证据是相互矛盾的，但至少有强有力的证据表明，在授予资助的过程中对女性存在偏见。关于对作者的偏见，最著名的证据来自于DP Peters和SJ ceci的一项研究。他们选取了12项已经在心理学期刊上发表，作者来自知名机构的文章。他们对这些论文重新打印稿件，对标题、摘要和介绍做了小改动，但改变了作者的名字和机构。他们虚构了一些机构，比如三谷人类潜能研究中心。这些论文随后被提交给最初发表它们的期刊。期刊只发现3例是已经发表了论文，其余9例中的8个被拒稿——不是因为缺乏原创性，而是因为质量差。Peters and Ceci得出结论，这是对来自不知名机构的作者存在偏见的证据。

This is known as the Mathew effect: `To those who have, shall be given; to those who have not shall be taken away even the little that they have'. I remember feeling the effect strongly when as a young editor I had to consider a paper submitted to the BMJ by Karl Popper. I was unimpressed and thought we should reject the paper. But we could not. The power of the name was too strong. So we published, and time has shown we were right to do so. The paper argued that we should pay much more attention to error in medicine, about 20 years before many papers appeared arguing the same.

这就是Mathew效应:“有的，还要继续给;没有的，连他们很少的一点东西也要被夺去“。我记得当我还是一个年轻的编辑时，我强烈地感受到这种影响，我不得不考虑一篇由卡尔·波普尔提交给《英国医学杂志》的论文。这篇稿件没有感动我的地方，我认为因该拒稿。。但是我不能。这个名字的力量太强大了。所以我们发表了文章，时间证明我们这样做是正确的。这篇论文认为，我们应该更加关注医学中的错误。然而大约在20年前，许多被拒的稿件就提出过同样的观点。

The editorial peer review process has been strongly biased against `negative studies', i.e. studies that find an intervention does not work. It is also clear that authors often do not even bother to write up such studies. This matters because it biases the information base of medicine. It is easy to see why journals would be biased against negative studies. Journalistic values come into play. Who wants to read that a new treatment does not work? That's boring.

编辑同行评议对“负面研究”，即失败的研究，有强烈偏见。同样明显的是，作者通常甚至懒得写这样的失败研究的报告。这使医学基础信息产生偏差。很容易理解为什么期刊会对负面研究有偏见。这是新闻价值观在发挥作用，因为读一篇没有疗效的治疗方法很无聊。

We became very conscious of this bias at the BMJ; we always tried to concentrate not on the results of a study we were considering but on the question it was asking. If the question is important and the answer valid, then it must not matter whether the answer is positive or negative. I fear, however, that bias is not so easily abolished and persists.

在英国医学杂志，我们非常清楚这种偏见;我们总是试图把注意力集中在提出的问题上，而不是研究的结果上。如果这个问题很重要，答案没有错，那么答案是肯定的还是否定的就无关紧要了。然而，偏见不会那么容易消除，而是会持续存在。

The Lancet has tried to get round the problem by agreeing to consider the protocols (plans) for studies yet to be done. If it thinks the protocol sound and if the protocol is followed, the Lancet will publish the final results regardless of whether they are positive or negative. Such a system also has the advantage of stopping resources being spent on poor studies. The main disadvantage is that it increases the sum of peer reviewing—because most protocols will need to be reviewed in order to get funding to perform the study.

《柳叶刀》试图通过同意考虑发表尚未完成的研究的协议(计划)来绕过这个问题。如果《柳叶刀》认为该方案是合理的，并且研究是按该方案执行的，那么无论结果是正面还是负面的，《柳叶刀》都会发表最终结果。这样还可以防止资源被浪费在糟糕的研究上。主要的缺点是它增加了同行评议的数量——因为为了获得开展研究的资金，大多数协议都需要被评议。

Abuse of peer review

同行评议的滥用

There are several ways to abuse the process of peer review. You can steal ideas and present them as your own, or produce an unjustly harsh review to block or at least slow down the publication of the ideas of a competitor. These have all happened. Drummond Rennie tells the story of a paper he sent, when deputy editor of the New England Journal of Medicine, for review to Vijay Soman. Having produced a critical review of the paper, Soman copied some of the paragraphs and submitted it to another journal, the American Journal of Medicine. This journal, by coincidence, sent it for review to the boss of the author of the plagiarized paper. She realized that she had been plagiarized and objected strongly. She threatened to denounce Soman but was advised against it. Eventually, however, Soman was discovered to have invented data and patients, and left the country. Rennie learnt a lesson that he never subsequently forgot but which medical authorities seem reluctant to accept: those who behave dishonestly in one way are likely to do so in other ways as well.

同行评议可能被滥用。你可以窃取稿件中的创意并将其作为自己的创意发表，或者进行不公正的审稿，以阻止或至少减缓竞争对手观点的发表。这些都发生过。Drummond Rennie讲述了他的投稿，《新英格兰医学杂志》副主编让Vijay Soman审稿。Soman对这篇论文给出了批判性的评论后，复制了其中的一些段落，投稿另一家杂志《美国医学杂志》。凑巧的是，这家杂志选择了被剽窃论文作者的老板审稿。她意识到自己的作品被剽窃了，表示了强烈反对。她要投诉Soman，但被人劝阻。然而，Soman最终因为伪造了数据和病人而并离开了这个国家。Rennie学到了一个教训，他后来从未忘记，但医学权威似乎不愿接受的教训:那些在一方面表现不诚实的人，在其他方面也会表现得不诚实。

HOW TO IMPROVE PEER REVIEW

如何改进同行评议

The most important question with peer review is not whether to abandon it, but how to improve it.  Many ideas have been advanced to do so, and an increasing number have been tested experimentally.  The options include: standardizing procedures; opening up the process;  blinding reviewers to the identity of authors;  reviewing protocols;  training reviewers;  being more rigorous in selecting and deselecting reviewers;  using electronic review;  rewarding reviewers;  providing detailed feedback to reviewers;  using more checklists;  or creating professional review agencies.  It might be, however, that the best response would be to adopt a very quick and light form of peer review—and then let the broader world critique the paper or even perhaps rank it in the way that Amazon asks users to rank books and CDs.

目前对同行评议的质疑集中在如何改进它，而不是 是否抛弃它。为此提出了许多想法，而且正在对很多想法进行实验。其中包括:使同行评议标准化; 使同行评议开放透明;使作者对审稿人匿名;制定稿协议;培训审稿人;更严格地选择审稿人的策略;采用电子评审;为审稿人提供报酬。向审稿人提供详细的反馈;使用更多的规则;或者建立专业的审稿机构。然而，最好的方法可能是采用快速而简化的同行评议——以便来自更广泛的学术界的评论，甚至可能像亚马逊要求用户对书籍和cd进行排名那样处理文章。

I hope that it will not seem too indulgent if I describe the far from finished journey of the BMJ to try and improve peer review.  We tried as we went to conduct experiments rather than simply introduce changes.

我希望，对我描述《英国医学杂志》(BMJ)努力改进同行评议的远未完成的历程，不要太介意。我们只是尝试着进行实验，还没有得到要改变什么结果。

The most important step on the journey was realizing that peer review could be studied just like anything else. This was the idea of Stephen Lock, my predecessor as editor, together with Drummond Rennie and John Bailar. At the time it was a radical idea, and still seems radical to some—rather like conducting experiments with God or love.

像其他任何事情一样，需要认识到同行评议需要被研究。这是我的前任编辑Stephen Lock、Drummond Rennie和John Bailar的想法。在当时，这是一个激进的想法，现在对一些人来说仍然是激进的。

Blinding reviewers to the identity of authors

作者的身份对审稿人匿名

The next important step was hearing the results of a randomized trial that showed that blinding reviewers to the identity of authors improved the quality of reviews (as measured by a validated instrument). This trial, which was conducted by Bob McNutt, A T Evans, and Bob and Suzanne Fletcher, was important not only for its results but because it provided an experimental design for investigating peer review. Studies where you intervene and experiment allow more confident conclusions than studies where you observe without intervening.

一项随机实验的结果表明，让作者的身份对审稿人匿名可以提高审稿的质量(这个实验类似于用仪器测量同行评议得质量)。这项实验是Bob McNutt, A T Evans, 和 Bob 和 Suzanne Fletcher做的。它的重要性不仅在于它的结果，还在于它为评价同行评议提供了一种实验设计。提供干预得实验的研究结果比不干预的观察结论更有说服力。

This trial was repeated on a larger scale by the BMJ and by a group in the USA who conducted the study in many different journals. Neither study found that blinding reviewers improved the quality of reviews. These studies also showed that such blinding is difficult to achieve (because many studies include internal clues on authorship), and that reviewers could identify the authors in about a quarter to a third of cases. But even when the results were analysed by looking at only those cases where blinding was successful there was no evidence of improved quality of the review.

英国医学杂志和美国的一个小组在许多不同的期刊上进行了这项研究，在更大的范围内重复了这项试验。但是这些研究都没有发现盲审能提高审稿质量。而且这种盲法很难实现(文章的内部线索能泄露作者的身份)，四分之一到三分之一的稿件能让审稿人判断出作者是谁。但是，即使对那些不能辨别作者身份的稿件，也没有证据表明评审的质量有所提高。

Opening up peer review

使同行评议更透明

At this point we at the BMJ thought that we would change direction dramatically and begin to open up the process. We hoped that increasing the accountability would improve the quality of review. We began by conducting a randomized trial of open review (meaning that the authors but not readers knew the identity of the reviewers) against traditional review. It had no effect on the quality of reviewers' opinions. They were neither better nor worse. We went ahead and introduced the system routinely on ethical grounds: such important judgements should be open and acountable unless there were compelling reasons why they could not be—and there were not.

我们英国医学杂志认为同行评议应该向非常不同的方向改进，因此试图使同行评议更透明。我们希望增加问责制将提高评审质量。和传统评审不同，我们进行了一项开放评审的随机试验 (作者而不是读者知道评审人的身份)，但是评审质量没有因此提高，既没有更好也没有更差。除非涉及利益冲突，我们例行公事地引入了这一制度:如此重要的稿件质量判断应该是公开和负责任的。这是从道德的角度出发采取的措施，而实际上并没有使审稿人更负责人地审稿。

Our next step was to conduct a trial of our current open system against a system whereby every document associated with peer review, together with the names of everybody involved, was posted on the BMJ's website when the paper was published. Once again this intervention had no effect on the quality of the opinion. We thus planned to make posting peer review documents the next stage in opening up our peer review process, but that has not yet happened—partly because the results of the trial have not yet been published and partly because this step required various technical developments.

下一步是对照我们目前的开放系统进一步的实验，即当论文发表时，同行评议所有相关的文件，以及所有相关人员的姓名，都公布在英国医学杂志的网站上。这种措施仍然对评审意见的质量没有影响。因此，我们计划开放同行评议，审稿过程中将同行评审意见发布，但这还没有实现——部分原因是试验结果尚未公布，部分原因是这一步骤还需要各种技术到位。

The final step was, in my mind, to open up the whole process and conduct it in real time on the web in front of the eyes of anybody interested. Peer review would then be transformed from a black box into an open scientific discourse. Often I found the discourse around a study was a lot more interesting than the study itself. Now that I have left I am not sure if this system will be introduced.

在我看来，是公开整个同行评议过程最后步骤，同行评议实时地在网上进行，任何人可以看到。同行评议将从一个黑盒子转变为一个开放的科学对话。我经常发现，围绕一项研究的对话比文章本身有趣得多。现在我已经离开了，我不确定这个制度是否会被引入。

Training reviewers

培训评论者

The BMJ also experimented with another possible way to improve peer review—by training reviewers. It is perhaps extraordinary that there has been no formal training for such an important job. Reviewers learnt either by trial and error (without, it has to be said, very good feedback), or by working with an experienced reviewer (who might unfortunately be experienced but not very good).

英国医学杂志还试验了另一种改善同行评议的方法——培训审稿人。似乎很奇怪，对于如此重要的工作，审稿人没有受过正式的培训。审稿人要么通过试错来学习(这一方面还没有很好的反馈)，要么通过与经验丰富的审稿人合作来学习(但是经验丰富的不一定是很好的)。

Our randomized trial of training reviewers had three arms: one group got nothing; one group had a day's face-to-face training plus a CD-rom of the training; and the third group got just the CD-rom. The overall result was that training made little difference. The groups that had training did show some evidence of improvement relative to those who had no training, but we did not think that the difference was big enough to be meaningful. We cannot conclude from this that longer or better training would not be helpful. A problem with our study was that most of the reviewers had been reviewing for a long time. `Old dogs cannot be taught new tricks', but the possibility remains that younger ones could.

我们的随机培训试验将审稿人分成三组:一组没有培训;一组接受了一天的面对面培训，外加一张培训光盘;第三组只拿到培训光盘。经过培训的确实比没有培训的好点，但是改进并不大因而没有实际意义。我们不能由此得出更长或更好的训练没有帮助的结论。问题是，大多数审稿人已经审稿很长时间了。“老狗学不了新把戏”，但年轻人还是有可能学会的。

TRUST IN SCIENCE AND PEER REVIEW

信任科学界和同行评议是否应该被信任

One difficult question is whether peer review should continue to operate on trust. Some have made small steps beyond into the world of audit. The Food and Drug Administration in the USA reserves the right to go and look at the records and raw data of those who produce studies that are used in applications for new drugs to receive licences. Sometimes it does so. Some journals, including the BMJ, make it a condition of submission that the editors can ask for the raw data behind a study. We did so once or twice, only to discover that reviewing raw data is difficult, expensive, and time consuming. I cannot see journals moving beyond trust in any major way unless the whole scientific enterprise moves in that direction.

一个难题是同行评议是否应该继续被信任。一些公司已经在审计方面迈出了一小步。美国食品和药物管理局(Food and Drug Administration)保留审查那些用于新药申请许可的研究人员的实验记录和原始数据的权利。有时确实去检查。包括英国医学杂志在内的一些期刊，编辑可以要求提供研究背后的原始数据作为投稿条件。我们这样做了一两次，只是发现检查原始数据是困难的、昂贵的和耗时的。除非整个科学事业都朝着这个方向发展，否则我认为期刊的行为仍然是建立在对科学家和审稿人的信任基础之上的。

CONCLUSION

结论

So peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative, and scientists and editors have a continuing belief in peer review. How odd that science should be rooted in belief.

同行评议是一个有缺陷的过程，充满了容易识别的缺陷，几乎没有证据表明它有效。然而，它很可能仍然是科学和期刊的核心，因为没有明显的替代方案，科学家和编辑们将一如既往地相信同行评议。奇怪的是，科学竟然植根于信仰而非事实。

Notes

注释

Richard Smith was editor of the BMJ and chief executive of the BMJ Publishing Group for 13 years. In his last year at the journal he retreated to a 15th century palazzo in Venice to write a book. The book will be published by RSM Press [www.rsmpress.co.uk], and this is the second in a series of extracts that will be published in the JRSM.

Richard Smith是《英国医学杂志》的编辑，也是《英国医学杂志》出版集团的首席执行官，长达13年之久。在杂志社工作的最后一年，他躲到威尼斯一座15世纪的公寓里写一本书。这本书将由RSM出版社[www.rsmpress.co.uk]出版，这是将在JRSM上发表的一系列中第二本的摘录。