mleoking的个人博客分享 http://blog.sciencenet.cn/u/mleoking

博文

LeoTask 快速可靠可扩展的计算研究框架 (开源轻量多核MapReduce)

已有 1667 次阅读 2015-11-21 11:24 |系统分类:科研笔记|关键词:MapReduce,LeoTask,可靠,快速,多核,并行,灾难恢复| 多核, 快速, mapreduce, 可靠, LeoTask

LeoTask 快速可靠可扩展的计算研究框架 (可靠的轻量级多核MapReduce框架)

源码:https://github.com/mleoking/LeoTask


特别适合于编写长时间运行的多参数计算/模拟程序。LeoTask自动遍历多个参数的组合参数空间,自动并行运行和统计数据,找出最优值,格式化结果输出,高质量画图(Gnuplot)等。LeoTask,不需要任何额外的代码,就能够自动备份程序并行统计的数据,在服务器发生异常(重启,断电等)后,可以重新继续从断点并行运行程序。


LeoTask

LeoTask is a parallel task running and results aggregation (MapReduce) framework. It is a free and open-source project designed to facilitate running computational intensive tasks [1]. The framework implements the MapReduce model, allocating tasks to multi-cores of a computer and aggregating results according to a XML based configuration file. The framework includes mechanisms to automatically recover applications from interruptions caused by accidents (e.g. Power Cut). Applications using the framework can continue running after an interruption without losing its calculated results.

Download | Introduction | Applications | Discussion

Features:

  • Automatic & parallel parameter space exploration.

  • Flexible & configuration-based result aggregation.

  • Programming model focusing only on the key logic.

  • Reliable & automatic interruption recovery.

  • Ultra lightweight ~ 300KB Jar.

Utilities:

Example Application:

Please refer to the introduction for building an example application using the framework.


Code (RollDice.java):


public class RollDice extends Task {
   public Integer nSide; //Number of dice sides
   public Integer nDice; //Number of dices to roll
   public Integer sum;//Sum of the results of nDice dices

   public boolean prepTask() {
       boolean rtn = nSide > 0 && nDice > 0;
       return rtn;
   }

   public void beforeRept() {
       super.beforeRept();
       sum = 0;
   }

   public boolean step() {
       boolean rtn = iStep <= nDice;
       if (rtn) {
           sum += (int) (rand.nextDouble() * nSide + 1);
       }
       return rtn;
   }
}


Configuration (rolldice.xml):


<Tasks>
   <name val="task-rolldice"/><usage val="0.9"/><nRepeats val="2000"/><checkInterval val="4"/>
   <variables class="org.leores.task.app.RollDice">    
       <nSide val="2;4;6"/>
       <nDice val="2:1:5"/><!--from 2 to 5 with a step of 1, i.e. 2;3;4;5 -->
   </variables>
   <statistics>
       <members>
           <i><info val="Fig1%pltm+@afterRept@"/><valVar val="sum;#$sum$/$nDice$#"/>
              <parVars val="nSide;nDice"/></i>
           <i><info val="Fig2%plt+@afterRept@"/><valVar val="sum"/><parVars val="nSide"/></i>
           <i><info val="Fig3%plt+@afterRept@"/><valVar val="sum"/><parVars val="nDice"/></i>
       </members>
   </statistics>
</Tasks>


Before running the example application, please install Java and include the the directory of the command java in the system's PATH environment variable. Windows system users can alternatively download and install (install.bat) the all-in-one runtime environment package: LeoTaskRunEnv


Chang the current directory to the "Demo" folder and then execute the following commnad

   java -jar leotask.jar -load=rolldice.xml

If you are using a MS windows system, you can also execute "rolldice.bat".


References:

[1] Changwang Zhang, Shi Zhou, Benjamin M. Chain (2015). "LeoTask: a fast, flexible and reliable framework for computational research", arXiv:1501.01678. (PDF)




http://blog.sciencenet.cn/blog-571128-937101.html

上一篇:MedRotation: medical student rotation to U.S.

0

该博文允许注册用户评论 请点击登录 评论 (3 个评论)

数据加载中...
扫一扫,分享此博文

Archiver|手机版|科学网 ( 京ICP备14006957 )

GMT+8, 2019-10-21 18:21

Powered by ScienceNet.cn

Copyright © 2007- 中国科学报社

返回顶部