Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane
From: Jon Olav Vik <jonovik <at> gmail.com>
Subject: Iterator version of TaskClient.map() that returns results as they become available?
Newsgroups: gmane.comp.python.ipython.user
Date: Thursday 11th February 2010 09:37:37 UTC (over 7 years ago)
I have lots of trivially parallel computations and will store the results
to 
disk. Output order does not matter.

The TaskClient.map() function or TaskClient.parallel() decorator is
convenient 
and load-balanced, but blocks until all the results are in. What I would
like 
instead is a map()-like iterator that returns each result as it becomes 
available; similarly, an iparallel() decorator that returns an iterator.
Then I 
could do:

from IPython.kernel import client
tc = client.TaskClient()
# Tasks that take an varying amount of time
import numpy as np
import time
@tc.iparallel()   # <-- nifty feature to be written
def f(x):
    time.sleep(np.random.random())
    return -x
N = 1000
for result in f(range(N)):
    print result # or save to file, or plot a data point

By flushing the output from time to time, I could then watch results take
shape 
as they get computed.

I tried digging into the source code for TaskClient.map(), but was
overwhelmed 
by the layers and intricacies of Twisted. Any pointers in the right
direction 
would be highly appreciated.

Regards,
Jon Olav
 
CD: 3ms