I have quite an intensive python script and I want to speed up the process. So I’ve been trying to implement multiprocessing. I’ve turned part of my script into a function to use the multriprocessing Pool.map method over. The problem is my function takes in 13 positional arguments, 12 of them are constants of different type (list, dataframes, array,etc…) and 1 is the iterator.
So I’ve used partial from funcgroups to define a partial function with those 12 arguments and give it to the multiprocessing along with the iterator.
My problem is that when I do so I’ve got an error message saying I passed too many positional arguments.
This is what I’ve tried:
def multiproc_PG_simu(summary,file,grain_size,grain_delta,delta_range,elem,grain,beam,boxcar,ind_anomalous,Nb_PG,all_data,axres,k):
...
return summary,fres,poucet, sim, all_data,closest_match_final
With K being an iterator that replaces a ‘for’ loop and the argument I want the multiprocessing to be done over. The rest can be considered as constants.
if __name__=="__main__":
...
num_cpu_to_use = max(1,cpu_count() -1)
partial_function=partial(multiproc_PG_simu,summary,file,grain_size,grain_delta,
delta_range,elem,grain,beam,boxcar,ind_anomalous,Nb_PG,all_data,axres)
with Pool(num_cpu_to_use) as pool:
res=pool.map(partial_function,range(iterations))
This is the error message
TypeError: multiproc_PG_simu() takes 13 positional arguments but 14 were given
I’ve passed the exact right amount of arguments so I don’t understand. I’ve tried adding ‘self’ as the first argument of my function in its definition but it doesn’t work either.
Can’t reproduce… look for typos in your own code