Python multiprcessing on Windows with dynamic types

Question 1

import multiprocessing
class Handler:
    pass

handler = type('HandlerClass', (int, Handler),{"const": [123]})
def stream_process(H):
    h=H()
    print(h.const)

processes = [ multiprocessing.Process(target=stream_process, args=( handler,), name=f"server #{i}") for i in range(4) ]

for process in processes:
    process.start()

for process in processes:
    process.join()

raises error

Traceback (most recent call last):                                                                                                      
  File "C:\Projects\winmp\test.py", line 18, in <module>
    process.start()
  File "C:\Programs\Python\Python312\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Programs\Python\Python312\Lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programs\Python\Python312\Lib\multiprocessing\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Programs\Python\Python312\Lib\multiprocessing\popen_spawn_win32.py", line 94, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class '__main__.HandlerClass'>: attribute lookup HandlerClass on __main__ failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Programs\Python\Python312\Lib\multiprocessing\spawn.py", line 108, in spawn_main
    source_process = _winapi.OpenProcess(
                     ^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 87] The parameter is incorrect

this code also rises error

import multiprocessing
import sys
class Handler:
    pass


def stream_process(H):
    h = H()
    print(h.from_bytes(b'a', "big"))
    print(h.const)


def main():
    global HandlerClass
    HandlerClass = type('HandlerClass', (int, Handler),{"const": [123]})
    setattr(sys.modules[__name__], "HandlerClass",HandlerClass )
    setattr(sys.modules['test'], "HandlerClass",HandlerClass )
    
    processes = [ multiprocessing.Process(target=stream_process, args=( HandlerClass,), name=f"server #{i}") for i in range(4) ]

    for process in processes:
        process.start()

    for process in processes:
        process.join()

error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Programs\Python\Python312\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programs\Python\Python312\Lib\multiprocessing\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'HandlerClass' on <module 'test' from 'C:\\Projects\\winmp\\test.py'>

I need construct type in main process, but init it in subprocess. How to solve this error?

On Linux and WSL it works.

this works, but class constructed 5 times. {"const": [123]} is very slow in real code, but constant. i not want to construct it on each process.

import multiprocessing
import sys
class Handler:
    pass


def stream_process(H):
    h = H()
    print(h.from_bytes(b'a', "big"))
    print(h.const)


HandlerClass = type('HandlerClass', (int, Handler),{"const": [123]})
print('HandlerClass constructed here', HandlerClass)

if __name__ == "__main__":
    processes = [ multiprocessing.Process(target=stream_process, args=( HandlerClass,), name=f"server #{i}") for i in range(4) ]

    for process in processes:
        process.start()

    for process in processes:
        process.join()

Question 2

This happens because the variable name and the __qualname__ attribute of the class you create dynamically are mismatched (so "handler" != "HandlerClass"). That is to say, the attribute “HandlerClass” doesn’t even exist in the module’s namespace (only “handler” does). You can check this by running:

# AttributeError: module '__main__' has no attribute 'HandlerClass'
print(getattr(sys.modules[__name__], "HandlerClass"))

So the fix here is to change the line

handler = type('HandlerClass', (int, Handler),{"const": [123]})

to:

HandlerClass = type('HandlerClass', (int, Handler),{"const": [123]})

Question 3

This occurs due to a mismatch between the variable name and the qualname attribute of the dynamically created class. In other words, “handler” does not match “HandlerClass.” Specifically, the attribute “HandlerClass” does not exist in the module’s namespace; only “handler” does. You can verify this by executing:

handler = type(‘HandlerClass’, (int, Handler),{“const”: [123]})

Leave a Comment Cancel reply