Node - Download a file from box ftp site and upload to google cloud bucket
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

I want to accomplish this in google cloud functions triggered by pubsub. I do not want to store the file locally before uploading to google cloud bucket. I've tried multiple methods but I keep having issues piping the file over.

did you have a chance to look my solution?
ocanal 13 days ago
awarded to ocanal

Crowdsource coding tasks.

1 Solution

Winning solution

Hey, I wrote my solution and tested, it works like a charm,

I used jsftp to read file from ftp, and @google-cloud/storage to upload bucket.

Before starting,

  • You have to create and download json service account key to your project folder, which is named cloud-private-key.json in my code.

  • install jsftp with npm i jsftp

  • install @google-cloud/storage with npm i @google-cloud/storage

  • don't forget to replace your ftp settings, your-project-id and your-bucket-name

Here is my working solution for file.txt

const jsftp = require("jsftp");

const ftp = new jsftp({
  host: "ftp_host", 
  port: 21, 
  user: "ftp_user", 
  pass: "ftp_pass" 
});

const {Storage} = require('@google-cloud/storage');

const GOOGLE_CLOUD_PROJECT_ID = 'your-project-id'; // Replace with your project ID
const GOOGLE_CLOUD_KEYFILE = 'cloud-private-key.json'; // Replace with the path to the downloaded private key

const storage = new Storage({
    projectId: GOOGLE_CLOUD_PROJECT_ID,
    keyFilename: GOOGLE_CLOUD_KEYFILE,
});

jsftp.prototype.getBuffer = function(remotePath, callback) {
  var buffer = new Buffer(0);

  this.get(remotePath, function(err, socket) {
    if (err) {
      return callback(err);
    }

    socket.on('data', function(data) {
      buffer = Buffer.concat([buffer, data]);
    });

    socket.on('close', function() {
      if (err) {
        return callback(err);
      }

      callback(null, buffer);
    });

    socket.resume();
  });
};

ftp.getBuffer('file.txt', function(hadErr, data) {
    if (hadErr)
      console.error(hadErr);
    else {
        console.log(data.toString());
        var fs = require('fs');
        var myBucket = storage.bucket('your-bucket-name'); //replace your bucket name

        var file = myBucket.file('file.txt');

        file.createWriteStream()
          .on('error', function(err) {
            console.log(err);
          })
          .on('finish', function() {
            console.log('file is uploaded');
          })
          .end(data);
    }

  });