Pass a large numpy array to another process with twisted
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

I have a class that contains a large numpy array self.data (multiple gigabytes). I want to use twisted to pass this data to another process on command.

I need a server and a client function/outline that will call a function on self.data and serialize the results (a numpy array) across the network connection to the client process.

If you can show a way to do this generally with the necessary code, that's all I need.

awarded to hashme33

Crowdsource coding tasks.

1 Solution

Winning solution

Some clarifications would be good. Who is listening? Server or client? Or it's longpolling clients that do connect to server? How do we identify, to which client do we send?

And here is a variation, where client -worker is listening to connections and server connects to him.
https://dl.dropbox.com/u/10235407/public/consumer_producer.zip
computing_server.py is the thing which is to be run on computer that is going to compute your function ("call a function on self.data and serialize the results")
So, you shoould change foo_routine in it to your function.
You have to launch computing_server.py first, then you can use computing_client.py.
MyClass in computing_client.py is your "class that contains a large numpy array self.data" and do_work is a function that sends self.data to another process through the network.

Hi there, thank you very much your reply and I'm sorry it's taken me so long to get to this! I am going to award you the bounty, but wanted to reach out before I do--can you help with some fairly simple freelance work related to Twisted? Basically, I just have a few questions about how it works in my code. Please let me know, my e-mail is dave.sullivan@blackcloudbsg.com if you want to contact me directly. Thanks.
davebs 7 years ago
View Timeline