Modify Small Python Script
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

Currently I have a small Python script which downloads all folders/files recursively from a target bucket in Upyun (a cloud storage/CDN service) and save folders/files to a target local folder on server.

I need it modified to be able to:

  1. Select all folders/files recursively from a target bucket in Aliyun OSS (another cloud storage/CDN service) and download folders/files to a target local folder on local server
  2. Select all folders/files recursively from a target folder on local server and upload folders/files to a target bucket in Aliyun OSS

Just with different script parameters to switch between the two functions.

The current script that I have is in the link below:
https://yolobit.com/v/84e5d5

The Aliyun Python installation doc is here:
https://help.aliyun.com/document_detail/32026.html?spm=5176.doc32031.6.665.yqinHV

The Aliyun Python download doc is here:
https://help.aliyun.com/document_detail/32031.html?spm=5176.doc32032.6.670.zdk5vH

The Aliyun Python upload doc is here:
https://help.aliyun.com/document_detail/32030.html?spm=5176.doc32031.6.669.yqinHV

The docs are unfortunately not in English but the SDK installation and execution are relatively straightforward. You can use Google translate for a general translation and if necessary, I can provide extra tips :)

Thanks!

awarded to tomtoump

Crowdsource coding tasks.

1 Solution

Winning solution

I've found this project on GitHub that does exactly what you want. It is written in Ruby.

Will this work for your needs?

EDIT: Can you try this script?

This script allows both upload and download? But I don't have Ruby in my current environment and I've never used it before. Would it be possible to convert that script into Python?
user856 10 months ago
Because I need another small script later on which will download images from URL's and then upload to Aliyun OSS and I already have a Python script that does something similar. So if you manage to get the Python script working, I can also accept your solution later on for that script as well :D
user856 10 months ago
Can you give me any credentials with limited access for testing?
tomtoump 10 months ago
I sent you the necessary info to your contact email :)
user856 10 months ago
Can you tell me what's the parameter/command to run the script?
user856 10 months ago
When I ran the script I got this error: File "aliyun_oss.py", line 3, in import configargparse ImportError: No module named configargparse
user856 10 months ago
Edit lines 89-91 with your data and run the script with the -h switch to see the available options.
tomtoump 10 months ago
Run pip install configargparse
tomtoump 10 months ago
I installed configarparse and got this success message: Successfully built configargparse Installing collected packages: configargparse Successfully installed configargparse-0.11.0
user856 10 months ago
But when I run the script I still get the same error
user856 10 months ago
It is a system specific issue. Can you try python -m pip install configargparse?
tomtoump 10 months ago
Okay now the script is running but only partial files are downloaded. Also the script currently doesn't recursively download folders in the bucket along with the files inside those folders.
user856 10 months ago
I think the problem is with the recursive folder issue. Because in terminal I see that the script stops running when encountering the folder and subsequent files aren't downloaded
user856 10 months ago
Currently I'm testing download but I need recursive download and recursive upload in the script
user856 10 months ago
In my testing, with the token you gave me, both recursive download and upload work. Can you try with that bucket?
tomtoump 10 months ago
Can you try this bucket 'site-eumt' and see if the folder 'aux-files-dom' has any files downloaded to it?
user856 10 months ago
I've made a small modification and seems to work now. Can you try again?
tomtoump 10 months ago
I tried again and recursive folders are being downloaded but still only a portion of the files in target bucket are being downloaded for some reason. About 50% of files are downloaded and the rest aren't.
user856 10 months ago
It seems the script downloads all files starting with 'a' and then when it encounters a folder, it tries to download the files in the folders recursively. But not all files are downloaded and then the rest of the files in root directory aren't downloaded as well.
user856 10 months ago
Actually it had a default limit of 100 files. I updated the script to get all the files.
tomtoump 10 months ago
Can you please add a parameter for user to enter 'endpoint' value?
user856 10 months ago
Updated the script.
tomtoump 10 months ago
For the 'upload' function, is there anywhere or anyway I can set an upload path for the files to be uploaded to bucket?
user856 10 months ago
Also for 'download' function is it possible to set bucket download path as well?
user856 10 months ago
Added the -bp parameter.
tomtoump 10 months ago
When I run the upload function I get this error: File: ./2016-12-13-23-18-03.zip doesn't exist. -> Uploading ... {'status': 400, 'details': {'HostId': 'some-bucket.oss-cn-beijing.aliyuncs.com', 'Message': 'The specified object is not valid.', 'Code': 'InvalidObjectName', 'RequestId': '585011764F29D03537EB44C4', 'ObjectName': '/File-Backups/2016-12-13-23-18-03.zip'}}
user856 10 months ago
I'm trying to upload a .zip file from this local folder '/File-Backups'
user856 10 months ago
Try -bp 'File-Backups'
tomtoump 10 months ago
Okay thanks a lot! Everything is now working great :) Do you have time for another really similar script? It's the same upload function but just for images. I'll post a new bounty :)
user856 10 months ago
I'll check it out! Thanks!
tomtoump 10 months ago
The new bounty is here: https://bountify.co/small-python-script
user856 10 months ago