Modify Small Python Script
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

Currently I have a small Python script which downloads all folders/files recursively from a target bucket in Upyun (a cloud storage/CDN service) and save folders/files to a target local folder on server.

I need it modified to be able to:

  1. Select all folders/files recursively from a target bucket in Aliyun OSS (another cloud storage/CDN service) and download folders/files to a target local folder on local server
  2. Select all folders/files recursively from a target folder on local server and upload folders/files to a target bucket in Aliyun OSS

Just with different script parameters to switch between the two functions.

The current script that I have is in the link below:
https://yolobit.com/v/84e5d5

The Aliyun Python installation doc is here:
https://help.aliyun.com/document_detail/32026.html?spm=5176.doc32031.6.665.yqinHV

The Aliyun Python download doc is here:
https://help.aliyun.com/document_detail/32031.html?spm=5176.doc32032.6.670.zdk5vH

The Aliyun Python upload doc is here:
https://help.aliyun.com/document_detail/32030.html?spm=5176.doc32031.6.669.yqinHV

The docs are unfortunately not in English but the SDK installation and execution are relatively straightforward. You can use Google translate for a general translation and if necessary, I can provide extra tips :)

Thanks!

awarded to tomtoump

Crowdsource coding tasks.

1 Solution

Winning solution

I've found this project on GitHub that does exactly what you want. It is written in Ruby.

Will this work for your needs?

EDIT: Can you try this script?

This script allows both upload and download? But I don't have Ruby in my current environment and I've never used it before. Would it be possible to convert that script into Python?
user856 1 year ago
Because I need another small script later on which will download images from URL's and then upload to Aliyun OSS and I already have a Python script that does something similar. So if you manage to get the Python script working, I can also accept your solution later on for that script as well :D
user856 1 year ago
Can you give me any credentials with limited access for testing?
tomtoump 1 year ago
I sent you the necessary info to your contact email :)
user856 1 year ago
Can you tell me what's the parameter/command to run the script?
user856 1 year ago
When I ran the script I got this error: File "aliyun_oss.py", line 3, in import configargparse ImportError: No module named configargparse
user856 1 year ago
Edit lines 89-91 with your data and run the script with the -h switch to see the available options.
tomtoump 1 year ago
Run pip install configargparse
tomtoump 1 year ago
I installed configarparse and got this success message: Successfully built configargparse Installing collected packages: configargparse Successfully installed configargparse-0.11.0
user856 1 year ago
But when I run the script I still get the same error
user856 1 year ago
It is a system specific issue. Can you try python -m pip install configargparse?
tomtoump 1 year ago
Okay now the script is running but only partial files are downloaded. Also the script currently doesn't recursively download folders in the bucket along with the files inside those folders.
user856 1 year ago
I think the problem is with the recursive folder issue. Because in terminal I see that the script stops running when encountering the folder and subsequent files aren't downloaded
user856 1 year ago
Currently I'm testing download but I need recursive download and recursive upload in the script
user856 1 year ago
In my testing, with the token you gave me, both recursive download and upload work. Can you try with that bucket?
tomtoump 1 year ago
Can you try this bucket 'site-eumt' and see if the folder 'aux-files-dom' has any files downloaded to it?
user856 1 year ago
I've made a small modification and seems to work now. Can you try again?
tomtoump 1 year ago
I tried again and recursive folders are being downloaded but still only a portion of the files in target bucket are being downloaded for some reason. About 50% of files are downloaded and the rest aren't.
user856 1 year ago
It seems the script downloads all files starting with 'a' and then when it encounters a folder, it tries to download the files in the folders recursively. But not all files are downloaded and then the rest of the files in root directory aren't downloaded as well.
user856 1 year ago
Actually it had a default limit of 100 files. I updated the script to get all the files.
tomtoump 1 year ago
Can you please add a parameter for user to enter 'endpoint' value?
user856 1 year ago
Updated the script.
tomtoump 1 year ago
For the 'upload' function, is there anywhere or anyway I can set an upload path for the files to be uploaded to bucket?
user856 1 year ago
Also for 'download' function is it possible to set bucket download path as well?
user856 1 year ago
Added the -bp parameter.
tomtoump 1 year ago
When I run the upload function I get this error: File: ./2016-12-13-23-18-03.zip doesn't exist. -> Uploading ... {'status': 400, 'details': {'HostId': 'some-bucket.oss-cn-beijing.aliyuncs.com', 'Message': 'The specified object is not valid.', 'Code': 'InvalidObjectName', 'RequestId': '585011764F29D03537EB44C4', 'ObjectName': '/File-Backups/2016-12-13-23-18-03.zip'}}
user856 1 year ago
I'm trying to upload a .zip file from this local folder '/File-Backups'
user856 1 year ago
Try -bp 'File-Backups'
tomtoump 1 year ago
Okay thanks a lot! Everything is now working great :) Do you have time for another really similar script? It's the same upload function but just for images. I'll post a new bounty :)
user856 1 year ago
I'll check it out! Thanks!
tomtoump 1 year ago
The new bounty is here: https://bountify.co/small-python-script
user856 1 year ago