python - Multiprocessing IOError: bad message length -

i ioerror: bad message length when passing large arguments map function. how can avoid this? error occurs when set n=1500 or bigger.

the code is:

import numpy np import multiprocessing  def func(args):     i=args[0]     images=args[1]     print     return 0  n=1500       #n=1000 works fine  images=[] in np.arange(n):     images.append(np.random.random_integers(1,100,size=(500,500)))  iter_args=[] in range(0,1):     iter_args.append([i,images])  pool=multiprocessing.pool() print pool pool.map(func,iter_args) 

in docs of multiprocessing there function recv_bytes raises ioerror. because of this? (https://python.readthedocs.org/en/v2.7.2/library/multiprocessing.html)

edit if use images numpy array instead of list, different error: systemerror: null result without error in pyobject_call. bit different code:

import numpy np import multiprocessing  def func(args):     i=args[0]     images=args[1]     print     return 0  n=1500       #n=1000 works fine  images=[] in np.arange(n):     images.append(np.random.random_integers(1,100,size=(500,500))) images=np.array(images)                                            #new  iter_args=[] in range(0,1):     iter_args.append([i,images])  pool=multiprocessing.pool() print pool pool.map(func,iter_args) 

edit2 actual function use is:

def func(args):     i=args[0]     images=args[1]     image=np.mean(images,axis=0)     np.savetxt("image%d.txt"%(i),image)     return 0 

additionally, iter_args not contain same set of images:

iter_args=[] in range(0,1):     rand_ind=np.random.random_integers(0,n-1,n)     iter_args.append([i,images[rand_ind]]) 

you're creating pool , sending images @ once func(). if can away working on single image @ once, try this, runs completion n=10000 in 35s python 2.7.10 me:

import numpy np import multiprocessing  def func(args):     = args[0]     img = args[1]     print "{}: {} {}".format(i, img.shape, img.sum())     return 0  n=10000  images = ((i, np.random.random_integers(1,100,size=(500,500))) in xrange(n)) pool=multiprocessing.pool(4) pool.imap(func, images) pool.close() pool.join() 

the key here use iterators don't have hold data in memory @ once. instance converted images array holding data generator expression create image when needed. modify load images disk or whatever. used pool.imap instead of pool.map.

if can, try load image data in worker function. right have serialize data , ship across process. if image data larger, might bottleneck.

[update know func has handle images @ once]

you iterative mean on images. here's solution without using multiprocessing. use multiprocessing, divide images chunks, , farm chunks out pool.

import numpy np  n=10000 shape = (500,500)  def func(images):     average = np.full(shape, 0)     i, img in images:         average += img / n     return average  images = ((i, np.full(shape,i)) in range(n))  print func(images)