|||
最近数据交付开始使用OSS了,以前都是快递硬盘。比较了下,发现OSS超级快,1小时下载了近200g的数据,极大节省了数据上传下载时间。
传统的做法是测序公司邮寄移动硬盘(约1-2天),从移动硬盘拷贝到本地服务器(1-2h,若是云服务器的话可能还需要1-2天)。现在直接1h搞定。加快了50-100倍。
centos使用记录
1,安装python,及各种依赖
1.1 python安装略
1.2 依赖包
yum install python-devel
2,安装oss-python SDK包
下载https://github.com/aliyun/aliyun-oss-python-sdk
然后解压
python setup.py install
3,存配置(一次就行)
python osscmd config --host=oss-cn-shanghai.aliyuncs.com --id=自己的id --key=自己的key
4,下载
python osscmd ls oss://你的目录
python osscmd get oss://你的目录/test.fastq.gz test.fastq.gz
就将test.fastq.gz下载到本地了,名字还是test.fastq.gz
批量下载:
python osscmd downloadallobject oss://你的目录 本地目录 --replace=false --thread_num=5
更多命令:
getallbucket(gs)
createbucket(cb,mb,pb) oss://bucket --acl=[acl] --location=[location]
deletebucket(db) oss://bucket
deletewholebucket oss://bucket
getbucketlocation(gl) oss://bucket
putcors oss://bucket localfile
getcors oss://bucket
deletecors oss://bucket
putlogging oss://source_bucket oss://target_bucket/[prefix]
getlogging oss://bucket
deletelogging oss://bucket
putwebsite oss://bucket indexfile [errorfile]
getwebsite oss://bucket
deletewebsite oss://bucket
putlifecycle oss://bucket localfile
getlifecycle oss://bucket
deletelifecycle oss://bucket
putreferer oss://bucket --allow_empty_referer=true --referer="referer1,referer2,...,refererN"
getreferer oss://bucket
getacl oss://bucket
setacl oss://bucket --acl=[acl]
allow private, public-read, public-read-write
getobjectacl oss://bucket/object
setobjectacl oss://bucket/object --acl=[acl]
allow private, public-read, public-read-write
ls(list) oss://bucket/[prefix] [marker] [delimiter] [maxkeys] [encoding_type]
ls(list) oss://bucket/[prefix] --marker=xxx --delimiter=xxx --maxkeys=xxx --encoding_type=url
mkdir oss://bucket/dirname
listallobject oss://bucket/[prefix]
listalldir oss://bucket/[prefix]
deleteallobject oss://bucket/[prefix] --force=false [--suffix=xxx]
downloadallobject oss://bucket/[prefix] localdir --replace=false --thread_num=5
downloadtodir oss://bucket/[prefix] localdir --replace=false --temp_dir=xxx --thread_num=5
uploadfromdir localdir oss://bucket/[prefix] --check_point=check_point_file --replace=false --check_md5=false --thread_num=5
put localfile oss://bucket/object --content_type=[content_type] --headers="key1:value1#key2:value2" --check_md5=false
appendfromfile localfile oss://bucket/object --content_type=[content_type] --headers="key1:value1#key2:value2"
upload localfile oss://bucket/object --content_type=[content_type] --check_md5=false
get oss://bucket/object localfile
multiget(multi_get) oss://bucket/object localfile --thread_num=5
cat oss://bucket/object
meta oss://bucket/object
info oss://bucket/object
copy oss://source_bucket/source_object oss://target_bucket/target_object --headers="key1:value1#key2:value2"
copylargefile oss://source_bucket/source_object oss://target_bucket/target_object --part_size=10*1024*1024 --upload_id=xxx
copybucket oss://source_bucket/[prefix] oss://target_bucket/[prefix] --headers="key1:value1" --replace=false
rm(delete,del) oss://bucket/object --encoding_type=url
signurl(sign) oss://bucket/object --timeout=[timeout_seconds]
createlinkfromfile oss://bucket/object object_name_list_file
createlink oss://bucket/object object1 object2 ... objectN
getlinkindex oss://bucket/object
options oss://bucket/[object] --origin=xxx --method=[GET, PUT, DELETE, HEAD, POST]
uploaddisk localdir oss://bucket/[prefix] [--check_point=check_point_file --filename=filename_file --replace=false --content_type=xxx --skip_dir=false --skip_suffix=false --out=xxx] --device_id=xxx --check_md5=false
init oss://bucket/object
listpart oss://bucket/object --upload_id=xxx
listparts oss://bucket [--encoding_type=url]
getallpartsize oss://bucket
cancel oss://bucket/object --upload_id=xxx [--encoding_type=url]
multiupload(multi_upload,mp) localfile oss://bucket/object --check_md5=false --thread_num=10
multiupload(multi_upload,mp) localfile oss://bucket/object --upload_id=xxx --thread_num=10 --max_part_num=1000 --check_md5=false
uploadpartfromfile (upff) localfile oss://bucket/object --upload_id=xxx --part_number=xxx
uploadpartfromstring(upfs) oss://bucket/object --upload_id=xxx --part_number=xxx --data=xxx
config --host=oss.aliyuncs.com --id=accessid --key=accesskey --sts_token=token
Archiver|手机版|科学网 ( 京ICP备07017567号-12 )
GMT+8, 2024-11-23 19:19
Powered by ScienceNet.cn
Copyright © 2007- 中国科学报社