The multiprocessing package supports spawning processes using an API similar to the threading module. It also offers both local and remote concurrency. This tutorial will discuss multiprocessing in Python and how to use multiprocessing to communicate between processes and perform synchronization between processes, as well as logging.
Introduction to Multiprocessing
from multiprocessing import Process def greeting(): print 'hello world' if __name__ == '__main__': p = Process(target=greeting) p.start() p.join()
In the example code above, we first import the Process class and then instantiate the Process object with the greeting function which we want to run.
We then tell the process to begin using the
start() method, and we finally complete the process with the
Additionally, you can also pass arguments to the function by providing the
args keyword argument like so:
from multiprocessing import Process def greeting(name): print 'hello' + " " + name if __name__ == '__main__': p = Process(target=greeting, args=('world',)) p.start() p.join()
Let's look at a more detailed example that covers all the concepts we have discussed above.
In this example, we are going to create a process that calculates the square of numbers and prints the results to the console.
from multiprocessing import Process def square(x): for x in numbers: print('%s squared is %s' % (x, x**2)) if __name__ == '__main__': numbers = [43, 50, 5, 98, 34, 35] p = Process(target=square, args=('x',)) p.start() p.join print "Done" #result Done 43 squared is 1849 50 squared is 2500 5 squared is 25 98 squared is 9604 34 squared is 1156 35 squared is 1225
You can also create more than one process at the same time, as shown in the example below, in which process p1 gets the results of numbers squared, while the second process p2 checks if the given numbers are even.
from multiprocessing import Process def square(x): for x in numbers: print('%s squared is %s' % (x, x**2)) def is_even(x): for x in numbers: if x % 2 == 0: print('%s is an even number ' % (x)) if __name__ == '__main__': numbers = [43, 50, 5, 98, 34, 35] p1 = Process(target=square, args=('x',)) p2 = Process(target=is_even, args=('x',)) p1.start() p2.start() p1.join() p2.join() print "Done" #result 43 squared is 1849 50 squared is 2500 5 squared is 25 98 squared is 9604 34 squared is 1156 35 squared is 1225 50 is an even number 98 is an even number 34 is an even number Done
Communication Between Processes
Multiprocessing supports two types of communication channels between processes:
Queue objects are used to pass data between processes. They can store any pickle-able Python object, and you can use them as shown in the example below:
import multiprocessing def is_even(numbers, q): for n in numbers: if n % 2 == 0: q.put(n) if __name__ == "__main__": q = multiprocessing.Queue() p = multiprocessing.Process(target=is_even, args=(range(20), q)) p.start() p.join() while q: print(q.get())
In the above example, we first create a function that checks if a number is even and then put the result at the end of the queue. We then instantiate a queue object and a process object and begin the process.
Finally, we check if the queue is empty, and if not, we get the values from the front of the queue and print them to the console.
We have shown how to share data between two processes using a queue, and the result is as shown below.
# result 0 2 4 6 8 10 12 14 16 18
It's also important to note that Python has a Queue module which lives in the process module and is used to share data between threads, unlike the multiprocessing queue which lives in shared memory and is used to share data between processes.
Pipes in multiprocessing are primarily used for communication between processes. Usage is as simple as:
from multiprocessing import Process, Pipe def f(conn): conn.send(['hello world']) conn.close() if __name__ == '__main__': parent_conn, child_conn = Pipe() p = Process(target=f, args=(child_conn,)) p.start() print parent_conn.recv() p.join()
Pipe() returns two connection objects which represent the two ends of the pipe. Each connection object has
recv() methods. Here we create a process that prints the string
hello world and then shares the data across.
# result ['hello world']
Locks work by ensuring that only one process is executed at a time, hence blocking other processes from executing similar code. This allows the process to be completed, and only then can the lock be released.
The example below shows a pretty straightforward usage of the Lock method.
from multiprocessing import Process, Lock def greeting(l, i): l.acquire() print 'hello', i l.release() if __name__ == '__main__': lock = Lock() names = ['Alex', 'sam', 'Bernard', 'Patrick', 'Jude', 'Williams'] for name in names: Process(target=greeting, args=(lock, name)).start() #result hello Alex hello sam hello Bernard hello Patrick hello Jude hello Williams
In this code, we first import the Lock method, acquire it, execute the print function, and then release it.
The multiprocessing module also provides support for logging, although the logging package doesn't use locks so messages between processes might end up being mixed up during execution.
Usage of logging is as simple as:
import multiprocessing, logging logger = multiprocessing.log_to_stderr() logger.setLevel(logging.INFO) logger.warning('Error has occurred')
Here we first import the logging and multiprocessing modules, and we then define the
multiprocessing.log_to_stderr() method, which performs a call to
get_logger() as well as adding a handler which sends output to
sys.stderr. Finally, we set the logger level and the message we want to convey.
This tutorial has covered what is necessary to get started with multiprocessing in Python. Multiprocessing overcomes the problem of GIL (Global Interpreter Lock) since it leverages the use of subprocesses instead of threads.
There is much more in the Python documentation that isn’t covered in this tutorial, so feel free to visit the Python multiprocessing docs and utilize the full power of this module.