{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Multi-processing example\n", "\n", "We’ll start with code that is clear, simple, and executed top-down. It’s easy to develop and incrementally testable:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('https://cusy.io/en', 29771)\n", "('https://jupyter-tutorial.readthedocs.io/en/latest/', 43637)\n", "('https://pyviz-tutorial.readthedocs.io/de/latest/', 37252)\n", "('https://github.com/veit/jupyter-tutorial/', 366286)\n", "('https://github.com/veit/pyviz-tutorial/', 334712)\n" ] } ], "source": [ "from multiprocessing.pool import ThreadPool as Pool\n", "\n", "import requests\n", "\n", "\n", "sites = [\n", " \"https://github.com/veit/jupyter-tutorial/\",\n", " \"https://jupyter-tutorial.readthedocs.io/en/latest/\",\n", " \"https://github.com/veit/pyviz-tutorial/\",\n", " \"https://pyviz-tutorial.readthedocs.io/de/latest/\",\n", " \"https://cusy.io/en\",\n", "]\n", "\n", "\n", "def sitesize(url):\n", " with requests.get(url) as u:\n", " return url, len(u.content)\n", "\n", "\n", "pool = Pool(10)\n", "for result in pool.imap_unordered(sitesize, sites):\n", " print(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "**Note:**\n", " \n", "A good development strategy is to use [map](https://docs.python.org/3/library/functions.html#map), to test your code in a single process and thread before moving to multi-processing.\n", "
\n", "\n", "
\n", "\n", "**Note:**\n", " \n", "In order to better assess when `ThreadPool` and when process `Pool` should be used, here are some rules of thumb:\n", "\n", "* For CPU-heavy jobs, `multiprocessing.pool.Pool` should be used. Usually we start here with twice the number of CPU cores for the pool size, but at least 4.\n", "\n", "* For I/O-heavy jobs, `multiprocessing.pool.ThreadPool` should be used. Usually we start here with five times the number of CPU cores for the pool size.\n", "\n", "* If we use Python 3 and do not need an interface identical to `pool`, we use [concurrent.future.Executor](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor) instead of `multiprocessing.pool.ThreadPool`; it has a simpler interface and was designed for threads from the start. Since it returns instances of `concurrent.futures.Future`, it is compatible with many other libraries, including `asyncio`.\n", "\n", "* For CPU- and I/O-heavy jobs, we prefer `multiprocessing.Pool` because it provides better process isolation.\n", "
" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('https://github.com/veit/jupyter-tutorial/', 366286)\n", "('https://jupyter-tutorial.readthedocs.io/en/latest/', 43637)\n", "('https://github.com/veit/pyviz-tutorial/', 334706)\n", "('https://pyviz-tutorial.readthedocs.io/de/latest/', 37252)\n", "('https://cusy.io/en', 29771)\n" ] } ], "source": [ "from multiprocessing.pool import ThreadPool as Pool\n", "\n", "import requests\n", "\n", "\n", "sites = [\n", " \"https://github.com/veit/jupyter-tutorial/\",\n", " \"https://jupyter-tutorial.readthedocs.io/en/latest/\",\n", " \"https://github.com/veit/pyviz-tutorial/\",\n", " \"https://pyviz-tutorial.readthedocs.io/de/latest/\",\n", " \"https://cusy.io/en\",\n", "]\n", "\n", "\n", "def sitesize(url):\n", " with requests.get(url) as u:\n", " return url, len(u.content)\n", "\n", "\n", "for result in map(sitesize, sites):\n", " print(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What can be parallelised?\n", "\n", "### Amdahl’s law\n", "\n", "> The increase in speed is mainly limited by the sequential part of the problem, since its execution time cannot be reduced by parallelisation. In addition, parallelisation creates additional costs, such as for communication and synchronisation of the processes.\n", "\n", "In our example, the following tasks can only be processed serially:\n", "\n", "* UDP DNS request request for the URL\n", "* UDP DNS response\n", "* Socket from the OS\n", "* TCP-Connection\n", "* Sending the HTTP request for the root resource\n", "* Waiting for the TCP response\n", "* Counting characters on the site" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('https://github.com/veit/jupyter-tutorial/', 366286)\n", "('https://jupyter-tutorial.readthedocs.io/en/latest/', 43637)\n", "('https://pyviz-tutorial.readthedocs.io/de/latest/', 37252)\n", "('https://github.com/veit/pyviz-tutorial/', 334706)\n", "('https://cusy.io/en', 29771)\n" ] } ], "source": [ "from multiprocessing.pool import ThreadPool as Pool\n", "\n", "import requests\n", "\n", "\n", "sites = [\n", " \"https://github.com/veit/jupyter-tutorial/\",\n", " \"https://jupyter-tutorial.readthedocs.io/en/latest/\",\n", " \"https://github.com/veit/pyviz-tutorial/\",\n", " \"https://pyviz-tutorial.readthedocs.io/de/latest/\",\n", " \"https://cusy.io/en\",\n", "]\n", "\n", "\n", "def sitesize(url):\n", " with requests.get(url, stream=True) as u:\n", " return url, len(u.content)\n", "\n", "\n", "pool = Pool(4)\n", "for result in pool.imap_unordered(sitesize, sites):\n", " print(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "**Note:**\n", " \n", "[imap_unordered](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.imap_unordered) is used to improve responsiveness. However, this is only possible because the function returns the argument and result as a tuple.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tips\n", "\n", "* Don’t make too many trips back and forth\n", "\n", " If you get too many iterable results, this is a good indicator of too many trips, such as in\n", "\n", " ```python\n", " >>> def sitesize(url, start):\n", " ... req = urllib.request.Request()\n", " ... req.add_header('Range:%d-%d' % (start, start+1000))\n", " ... u = urllib.request.urlopen(url, req)\n", " ... block = u.read()\n", " ... return url, len(block)\n", " ```\n", "\n", "* Make relevant progress on every trip\n", "\n", " Once you get the process, you should make significant progress and not get bogged down. The following example illustrates intermediate steps that are too small:\n", "\n", " ```python\n", " >>> def sitesize(url, results):\n", " ... with requests.get(url, stream=True) as u:\n", " ... while True:\n", " ... line = u.iter_lines()\n", " ... results.put((url, len(line)))\n", " ```\n", "\n", "* Don’t send or receive too much data\n", "\n", " The following example unnecessarily increases the amount of data:\n", "\n", " ```python\n", " >>> def sitesize(url):\n", " ... with requests.get(url) as u:\n", " ... return url, u.content\n", " ```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.13 Kernel", "language": "python", "name": "python313" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.0" }, "latex_envs": { "LaTeX_envs_menu_present": true, "autoclose": false, "autocomplete": true, "bibliofile": "biblio.bib", "cite_by": "apalike", "current_citInitial": 1, "eqLabelWithNumbers": true, "eqNumInitial": 1, "hotkeys": { "equation": "Ctrl-E", "itemize": "Ctrl-I" }, "labels_anchors": false, "latex_user_defs": false, "report_style_numbering": false, "user_envs_cfg": false }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }