Melding Technology

Vancouver, BC


20 October, 2014

During some recent data processing, a nifty built-in library was brought to my attention. Multiprocessing function, short little snippet below.

from multiprocessing import Pool

def process(data):
    print "{+} Processing: %s" % data
    #do something useful

if __name__ == '__main__':
    data = [1,2,3,4,5,6]
    pool = Pool(processes=6), data)

In short, you can with one line almost turn any array that needs to be process into a multiprocess map reduce party.

From a python processing, if you have isolated data, this will allow you to scale in a pretty aggresive way.