-H followed by the host name. If it is an IP, you need to add http in front, for example http://172.21. . -a followed by access_key, for example YT********************** -s followed by Add secret_key such as JlNVoLfLuJJ******************************
(1) Delete a single bucket. The script corresponding to this function is thread-delete-bucket.py. When using, specify the -b parameter followed by the bucket name.
[wuchen@manager delete]$ python thread-delete-bucket.py -H http://172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test7
['test1', 'test2', 'test7']
[{'Key': u'osd.84.log'}] 1
(2) Delete the bucket list. The script corresponding to this function is thread-delete-buckets.py. Before use, you need to write one bucket name for each line of the bucket list into a configuration file. The script corresponding to this function is thread-delete-buckets. py, when using, specify the -f parameter followed by the path to the configuration file (including the configuration file name)
[wuchen@manager delete]$ cat bucket.txt
test10
test3
test4
test5
test6
test7
test8
test9
[wuchen@manager delete]$ python thread-delete-buckets.py -H http://172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -f /home/wuchen/wuchen/delete/bucket.txt
deleting test10
[{'Key': u'osd.84.log'}] 1
deleting test3
[{'Key': u'osd.84.log'}] 1
deleting test4
[{'Key': u'osd.84.log'}] 1
deleting test5
[{'Key': u'osd.84.log'}] 1
deleting test6
[{'Key': u'osd.84.log'}] 1
deleting test7
[{'Key': u'osd.84.log'}] 1
deleting test8
[{'Key': u'osd.84.log'}] 1
deleting test9
[{'Key': u'osd.84.log'}] 1
(3) Delete all buckets of a user. The script corresponding to this function is thread-delete-user.py. It only requires three basic parameters to delete all buckets of the user.
[wuchen@manager delete]$ python thread-delete-user.py -H http://172.21.*.* -a YT***************** -s JlNVoLfLuJJ***********************
Use the script thread-download.py to download large files (larger than 8M) in fragmented multi-threads. If you want to download small files less than 8M, you only need to specify the -t parameter as 0.
(1) The required parameter -H is the host address. If you write IP, you don’t need to add http in front, for example, 172.21. . -a User’s access_key, for example, YT**********************
-s user's secret_key such as JlNVoLfLuJJ******************************
-b The name of the bucket where the object is to be downloaded, such as test12 -o The name of the object to be downloaded, such as python.tar.gz If it contains a multi-layer folder directory, you need to write the complete path -f The download object is written to the local file path, for example /home/wuchen/wuchen/download/download-file
(2) The optional parameter -t specifies the number of threads. The default is 10 threads.
-c specifies the fragment size in M, the default is 8M
Use the script download-folder.py to download a folder in the bucket. This function will recursively download all subdirectories in the directory.
The parameter analysis is as follows: -H If the host address is written as IP, there is no need to add http in front, for example, 172.21. . -a User's access_key, for example, YT**********************
-s user's secret_key such as JlNVoLfLuJJ******************************
-b The name of the bucket where the directory to be downloaded is located
-f The download directory data is stored in the local path location -p The s3 directory to be downloaded. If it is a multi-level directory, the complete path needs to be given. For example, there is a directory test1 under the bucket, and test1 has a directory test2. If you want to download test1, then - Add test1 after p, to download test2, add test1/test2 after -p
Download using default parameters
[wuchen@manager download]$ python thread-download.py -H 172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -o python.tar.gz -f /home/wuchen/wuchen/download/download-file
Download using 15 threads, sharding by default
[wuchen@manager download]$ python thread-download.py -H 172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -o python.tar.gz -f /home/wuchen/wuchen/download/download-file -t 20
Use 20 threads to download, and the fragmentation is 20M
[wuchen@manager download]$ python thread-download.py -H 172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -o python.tar.gz -f /home/wuchen/wuchen/download/download-file -t 20 -c 20
Download a single small file
wuchen@manager download]$ python thread-download.py -H 172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -o python.tar.gz -f /home/wuchen/wuchen/download/download-file -t 0
[wuchen@manager download]$ python download-folder.py -H http://172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -f /home/wuchen/wuchen/s3-tool/download/result22 -p werfefdsa/dffds -b test12
After this command is executed, the local result22 directory will download werfefdsa/dffds in the bucket name test12.
[wuchen@manager download]$ python download-folder.py -H http://172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -f /home/wuchen/wuchen/s3-tool/download/result22 -p werfefdsa -b test12
After the command is executed, the local result22 directory will download werfefdsa/ in the bucket name test12.
Use the script thread-upload.py to upload large files (larger than 8M) in multiple threads. By default, 8M slices will be uploaded in 10 threads. If you want to upload a small file smaller than 8M, you only need to specify the -t parameter as 0.
(1) The required parameter -H is the host address. If you write IP, you don’t need to add http in front, for example, 172.21. . -a User’s access_key, for example, YT**********************
-s user's secret_key such as JlNVoLfLuJJ******************************
-b The name of the bucket where the object is to be uploaded, such as test12 -o The path of the object to be uploaded. If you upload it directly to the bucket, you do not need to write the -o parameter. If you upload it to a folder under the bucket, you need to specify -o as the folder path -f to upload. The local file path of the object, such as /home/wuchen/wuchen/download/download-file
(2) The optional parameter -t specifies the number of threads. The default is 10 threads. It can be omitted. -c specifies the fragment size in M. The default is 8M. It can be omitted.
Use the script upload-folder.py to recursively upload a folder with a local specified path to the bucket. You can upload it directly to the bucket, or you can upload it to a directory under the bucket. The parameter parsing is as follows: -H Host address If you write IP If so, there is no need to add http in front, for example 172.21. . -a user’s access_key, for example YT******************
-s user's secret_key such as JlNVoLfLuJJ******************************
-b The bucket name of the directory to be uploaded
-l The upload directory is stored in the local path location. -d Recursively uploads the folder with the local specified path to the path location below the bucket. If it is uploaded directly to the bucket, specify -d root. If it is uploaded to a directory path below the bucket, , then the complete path is given. For example, when uploading to the subdirectory test2 under the test1 directory under the bucket, specify -d test1/test2
Upload using default parameters
python thread-upload.py -H 172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -f /home/wuchen/wuchen/upload/iflyek_url_tesy-1.0.1.100247.jar
Use 15 threads to download, sharding by default, and upload to the test1 directory under the bucket
python thread-upload.py -H 172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test1 -o test1 -f /home/wuchen/wuchen/upload/iflyek_url_tesy-1.0.1.100247.jar -t 15
Use 20 threads to upload, upload in 20M fragments, and upload to the /test1/test2/test3 subdirectory under the bucket
python thread-upload.py -H 172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -o /test1/test2/test3 -f /home/wuchen/wuchen/upload/iflyek_url_tesy-1.0.1.100247.jar -t 20 -c 20
Upload the local directory directly to bucket test12
[wuchen@manager upload]$ python upload-folder.py -H http://172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -l /home/wuchen/wuchen/s3-tool/upload/download -d root
Upload the local directory to the werfefdsa directory under bucket test12
[wuchen@manager upload]$ python upload-folder.py -H http://172.21.*.* -a YT***************** -s JlNVoLfLuJJ*********************** -b test12 -l /home/wuchen/wuchen/s3-tool/upload/download -d werfefdsa
Use the jar package java-upload-download.jar to upload and download. The command line parameters are specified as follows: (1) Upload files or directories java -jar iflytek_url_tesy-1.0.1.10023.jar. The following parameters are the bucket name + object name (including Directory path uploaded to the bucket) + number of threads + fragment size + uploaded file or directory path + upload string (in order to distinguish whether it is uploading or downloading)
(2) Download files (the download directory is still prone to errors)
The parameters following java -jar s3-tool-java.jar are the bucket name + the name of the file to be downloaded (including the full path under the bucket) + the number of threads + the fragment size + the file or directory path to download to the local + download string
Usage examples
1.上传文件夹，将文件夹 /lustre2/dlp/esxu/lustre-soft/上传到bucket下面
[root@jobsub-9-073 wuchen]# java -jar s3-tool-java.jar test-bucket test5 6 5242880 /lustre2/dlp/esxu/lustre-soft/ upload
Uploading etc
[########################################]: Completed
thread upload run time： 34s
2.上传文件 将文件/lustre2/dlp/esxu/test1.tar 上传到bucket test-bucket 下面，名称为test1.tar
[root@jobsub-9-073 wuchen]# java -Xmx10m -jar s3-tool-java.jar test-bucket test1.tar 10 5242880 /lustre2/dlp/esxu/test1.tar upload
Uploading etc
[########################################]: Completed
thread upload run time： 17s
3.下载文件 下载bucket下面的test1.tar 到本地重命名为 /lustre2/dlp/esxu/test2.tar
[root@jobsub-9-073 wuchen]# java -Xmx10m -jar s3-tool-java.jar test-bucket test1.tar 10 5242880 /lustre2/dlp/esxu/test2.tar download
Downloading etc
[########################################]: Completed
thread upload run time： 17s
