What exactly is Python multiprocessing Module's .join() Method Doing?
In an old tutorial from 2008 it states that without the
p.join() call in the code below, "the child process will sit idle and not terminate, becoming a zombie you must manually kill".
from multiprocessing import Process def say_hello(name='world'): print "Hello, %s" % name p = Process(target=say_hello) p.start() p.join()
I added a printout of the
PID as well as a
time.sleep to test and as far as I can tell, the process terminates on its own:
from multiprocessing import Process import sys import time def say_hello(name='world'): print "Hello, %s" % name print 'Starting:', p.name, p.pid sys.stdout.flush() print 'Exiting :', p.name, p.pid sys.stdout.flush() time.sleep(20) p = Process(target=say_hello) p.start() # no p.join()
within 20 seconds:
936 ttys000 0:00.05 /Library/Frameworks/Python.framework/Versions/2.7/Reso 938 ttys000 0:00.00 /Library/Frameworks/Python.framework/Versions/2.7/Reso 947 ttys001 0:00.13 -bash
after 20 seconds:
947 ttys001 0:00.13 -bash
Behavior is the same with
p.join() added back at end of the file. Python Module of the Week offers a very readable explanation of the module; "To wait until a process has completed its work and exited, use the join() method.", but it seems like at least OS X was doing that anyway.
Am also wondering about the name of the method. Is the
.join() method concatenating anything here? Is it concatenating a process with it's end? Or does it just share a name with Python's native
join() method, when used with
multiprocessing, is not related to
str.join() - it's not actually concatenating anything together. Rather, it just means "wait for this [thread/process] to complete". The name
join is used because the
multiprocessing module's API is meant to look as similar to the
threading module's API, and the
threading module uses
join for its
Thread object. Using the term
join to mean "wait for a thread to complete" is common across many programming languages, so Python just adopted it as well.
Now, the reason you see the 20 second delay both with and without the call to
join() is because by default, when the main process is ready to exit, it will implicitly call
join() on all running
multiprocessing.Process instances. This isn't as clearly stated in the
multiprocessing docs as it should be, but it is mentioned in the Programming Guidelines section:
Remember also that non-daemonic processes will be automatically be joined.
You can override this behavior by setting the
daemon flag on the
True prior to starting the process:
p = Process(target=say_hello) p.daemon = True p.start() # Both parent and child will exit here, since the main process has completed.
If you do that, the child process will be terminated as soon as the main process completes:
The process’s daemon flag, a Boolean value. This must be set before start() is called.
The initial value is inherited from the creating process.
When a process exits, it attempts to terminate all of its daemonic child processes.