(For new reader and those who request 好友请求, please read my 公告栏 first).

A recent report from the US National Academy of Sciences addresses the above title.

Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency, Committee on Science, Engineering, and Public Policy, The National Academies; and Board on Environmental Studies and Toxicology, Division on Earth and Life Studies (2008, approx. 144 pp.; ISBN 0-309-11684-8; available from the National Academies Press, tel. 1-800-624-6242; $35.00 plus $4.50 shipping for single copies).

While the report is directed towards environmental research, the conclusions and suggestion are generally applicable. I summarize them below

Currently the Office of Management and Budget (OMB) urges the Environmental Protection Agency (EPA) and other agencies to gauge the efficiency of their research based on its ultimate outcomes -- for example, whether a research program eventually leads to cleaner air or fewer deaths. But measuring efficiency this way isn't achievable or valid for most programs since eventual outcome occurs far in the future and is not controlled by the research scientists.

it is easier to gauge efficiency accurately if they split the concept in two, the report says. Assessments of "process efficiency" should look at how well research processes are managed, and whether program managers exercise skill in using and conserving resources. This type of efficiency can be measured quantitatively -- for example, by comparing the number of grants awarded or journal articles published against benchmarks.

Expert review panels also are key to assessing "investment efficiency," a second aspect that concerns whether an agency is investing in high-quality research in areas that further its mission. So-called intermediate outcomes can also be helpful, the report adds; an evaluation might consider whether a program has increased the knowledge available for making regulatory decisions, for example.

Evaluations of research should not over-emphasize efficiency, however. Efficiency should be weighed as just one factor in the overall context of a program's quality, relevance, and effectiveness, the report stresses.

None of the conclusions are new or revolutionary. This is how researches are mostly judged in the US based on my own experience for the past half century. To implement this in China currently, I would recommend two additional necessary but probably impractical requirements .

1.      At least half of the review panel should be composed of foreign experts

2.      Quantitative count of journal publications should be based at least 50% on international non Chinese journals.

Otherwise, despite the best intentions of everyone, we all know what will bound to occur.

 

如何评价科研效率？

 

　　美国国家科学院的一项最新报告，解答了“如何评价科研效率”的问题（报告链接：http://books.nap.edu/catalog.php?record_id=12150）。尽管该报告主要针对环境方面的研究，但其中的结论和建议还是具有普遍的意义。我将它们归纳如下：

 

　　眼下，美国管理与预算办公室（OMB）正敦促美国环境保护局（EPA）和其它一些机构对它们负责的研究进行效率测评，而这种评估要基于研究的最终结果。比如，一项研究能否最终使空气更加清洁，降低死亡率等。然而，对于大多数研究项目而言，用这种方式检测效率既不可行，也不合理。因为最终的结果往往发生在遥远的未来，而且不是科学家能够控制的。

 

　　报告指出，如果把评估理念一分为二，就能够更加准确地进行测评。其中一个方面就是“过程效率”（process efficiency），它评价的主要是研究过程的管理情况，项目管理者在利用和节约研究资源中是否采用了一些技巧。这一效率是可以定量测量的，比如将获得基金的数量或者论文的数量与一定标准相对照。

 

　　另一个重要方面就是“投资效率”（investment efficiency），它主要侧重于评价环保局等机构投资的项目是否是高质量的项目，而且这些研究是否有助于实现这些机构的长远目标。在这个评审过程中，专家评审团的意见就非常重要了。该报告还说，所谓的“中期成果”（intermediate outcomes）也是十分有用的，比如，评价一项研究的时候，可以考虑该研究是否创造了新的知识，有助于决策调整。

 

　　然而，科研的评价不能过分强调效率。该报告强调，效率只能作为对研究质量、意义以及有效性等综合评价中的一个方面而已。

 

　　可以看出，以上这些结论并不新奇，也不具有革命性。这仅仅是基于我过去半个世纪的经历的得出的美国通行做法。如果要移植到中国，我还有两点额外的建议，它们是必不可少的，但不一定现实。

 

　　1．项目评审团中至少有一半的成员是外国专家。

　　2．在对发表文章的定量统计中，至少有50%是基于国际期刊，而非中国期刊。

 

　　否则，纵使大家意图不错，我们也都知道肯定会发生些什么。

 

　　（科学网/任霄鹏译 何姣校）

 

转载本文请联系原作者获取授权，同时请注明本文来自何毓琦科学网博客。
链接地址：https://blog.sciencenet.cn/blog-1565-23016.html

上一篇：开博一年了
下一篇：On Research and Education (#11) Changing Research Direction and Field of Endeavo