Subject: Iterator version of TaskClient.map() that returns results as they become available?
Date: Thursday 11th February 2010 09:37:37 UTC (over 7 years ago)
I have lots of trivially parallel computations and will store the results to disk. Output order does not matter. The TaskClient.map() function or TaskClient.parallel() decorator is convenient and load-balanced, but blocks until all the results are in. What I would like instead is a map()-like iterator that returns each result as it becomes available; similarly, an iparallel() decorator that returns an iterator. Then I could do: from IPython.kernel import client tc = client.TaskClient() # Tasks that take an varying amount of time import numpy as np import time @tc.iparallel() # <-- nifty feature to be written def f(x): time.sleep(np.random.random()) return -x N = 1000 for result in f(range(N)): print result # or save to file, or plot a data point By flushing the output from time to time, I could then watch results take shape as they get computed. I tried digging into the source code for TaskClient.map(), but was overwhelmed by the layers and intricacies of Twisted. Any pointers in the right direction would be highly appreciated. Regards, Jon Olav