Positionnal argument issue when using partial from funcgroups and multiprocessing

I have quite an intensive python script and I want to speed up the process. So I’ve been trying to implement multiprocessing. I’ve turned part of my script into a function to use the multriprocessing Pool.map method over. The problem is my function takes in 13 positional arguments, 12 of them are constants of different type (list, dataframes, array,etc…) and 1 is the iterator.

So I’ve used partial from funcgroups to define a partial function with those 12 arguments and give it to the multiprocessing along with the iterator.

My problem is that when I do so I’ve got an error message saying I passed too many positional arguments.

This is what I’ve tried:

def multiproc_PG_simu(summary,file,grain_size,grain_delta,delta_range,elem,grain,beam,boxcar,ind_anomalous,Nb_PG,all_data,axres,k):

...

return summary,fres,poucet, sim, all_data,closest_match_final

With K being an iterator that replaces a ‘for’ loop and the argument I want the multiprocessing to be done over. The rest can be considered as constants.

if __name__=="__main__":
    ...
        num_cpu_to_use = max(1,cpu_count() -1)
        partial_function=partial(multiproc_PG_simu,summary,file,grain_size,grain_delta,
                                 delta_range,elem,grain,beam,boxcar,ind_anomalous,Nb_PG,all_data,axres)
        with Pool(num_cpu_to_use) as pool:
            res=pool.map(partial_function,range(iterations))

This is the error message

TypeError: multiproc_PG_simu() takes 13 positional arguments but 14 were given

I’ve passed the exact right amount of arguments so I don’t understand. I’ve tried adding ‘self’ as the first argument of my function in its definition but it doesn’t work either.

  • Can’t reproduce… look for typos in your own code

    – 




Leave a Comment