{"id":15460917,"url":"https://github.com/machawk1/parallelizing-python","last_synced_at":"2025-04-01T11:35:49.589Z","repository":{"id":173383298,"uuid":"650686249","full_name":"machawk1/parallelizing-python","owner":"machawk1","description":null,"archived":false,"fork":false,"pushed_at":"2023-06-07T17:02:30.000Z","size":714,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T09:45:19.745Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-sa-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/machawk1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-07T15:32:26.000Z","updated_at":"2023-06-07T16:13:31.000Z","dependencies_parsed_at":"2023-09-27T02:25:07.581Z","dependency_job_id":null,"html_url":"https://github.com/machawk1/parallelizing-python","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"5e85c7894e3d905d2b92afdce6cd0f0afb18fe1a"},"previous_names":["machawk1/parallelising-python"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fparallelizing-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fparallelizing-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fparallelizing-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fparallelizing-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/machawk1","download_url":"https://codeload.github.com/machawk1/parallelizing-python/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246634864,"owners_count":20809311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T23:40:08.921Z","updated_at":"2025-04-01T11:35:49.558Z","avatar_url":"https://github.com/machawk1.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parallelizing Python code\n\nIt is sometimes stated that parallel code is difficult in Python. However, for most scientific applications, we can achieve the level of parallelism without much effort. In this notebook I will show some simple ways to get parallel code execution in Python.\n\n1. With NumPy\n2. With Joblib and multiprocessing\n3. With Numba\n4. With Cython\n\nThese methods range in complexity from easiest to most difficult. \n\nAfter discussing Cython, there is a short example with Numba vectorize, which can be used for functions that should be applied element-wise on numerical NumPy arrays.\n\n\n## Parallel code with NumPy\n\nBy default, NumPy will dispatch the computations to an efficient BLAS (basic linear algebra subproblem) and LAPACK (Linear Algebra PACKage) implementation. BLAS and LAPACK routines are very efficient linear algebra routines that are implemented by groups of people that are experts getting as much speed as humanly possible out of the CPU, and we cannot compete with those for linear algebra computations.\n\nA benefit and downside with NumPy is that it will likely parallelize the code for you without your knowledge. Try to compute the matrix product between two large matrices and look at your CPU load. It will likely use all of your CPU hardware threads (hardware threads are essentially cores).\n\n## Moving the parallelism to the outer loop\n\nOften, when we program, we have nested loops. Like below\n\n\n```python\nx = [[] for _ in range(10)]\nfor i in range(10):\n    for j in range(10):\n        x[i].append(i*j)\n\nfor row in x:\n    print(row)      \n```\n\n    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n    [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]\n    [0, 3, 6, 9, 12, 15, 18, 21, 24, 27]\n    [0, 4, 8, 12, 16, 20, 24, 28, 32, 36]\n    [0, 5, 10, 15, 20, 25, 30, 35, 40, 45]\n    [0, 6, 12, 18, 24, 30, 36, 42, 48, 54]\n    [0, 7, 14, 21, 28, 35, 42, 49, 56, 63]\n    [0, 8, 16, 24, 32, 40, 48, 56, 64, 72]\n    [0, 9, 18, 27, 36, 45, 54, 63, 72, 81]\n\n\nHere, we have nested loop, and there are two ways to make this parallel, either by doing multiple iterations of the outer loop (`for i in range(10)`) at the same time or by doing multiple iterations of the inner loop (`for j in range(10)`) at the same time. \n\nGenerally, we prefer to have the parallel code on the outer loop, as that is the most work per iteration, which means that there is less likelihood for our cores to stay idle. If there are more hardware threads available than there are iterations on the outer loop, we may split it up and have some of the parallelism on the inner loop as well. However, it is important to make sure that we don't try to do more things in parallel than we have hardware threads available, as otherwise, much time will be spent switching between tasks rather than actually performing the computations.\n\n## Disabling parallel code execution in NumPy routines\nUnfortunately, we have a loop where we use a NumPy function, then that function likely runs in parallel using all the available hardware threads. To avoid this from happening, we have to set some envionment variables *before* importing NumPy. Specifically, we need to set\n\n```\nOMP_NUM_THREADS=#THREADS\nOPENBLAS_NUM_THREADS=#THREADS\nMKL_NUM_THREADS=#THREADS\nVECLIB_MAXIMUM_THREADS=#THREADS\nNUMEXPR_NUM_THREADS=#THREADS\n```\n\nThe first variable sets the number of OpenMP threads to `#THREADS`. OpenMP is used by many software packages that implement parallel code. The next three variables sets the number of threads for NumPy for various different BLAS backends. Finally, the last line sets the number of threads for a useful package called numexpr, which can optimise operations on the form `a*x + b*x - c`, which with pure numpy would entail four separate loops, but with numexpr is compiled to a single parallel loop.\n\nWe can either set these variables directly from Python, but then we MUST do it before any library has imported NumPy. Or, alternatively, we can set it as global environment variables. On Linux, you can add these lines to your [`~/.profile` file](https://www.quora.com/What-is-profile-file-in-Linux):\n\n```\nOPENBLAS_NUM_THREADS=1\nMKL_NUM_THREADS=1\nVECLIB_MAXIMUM_THREADS=1\nNUMEXPR_NUM_THREADS=1\n```\n\nNotice how we did not set the number of OpenMP threads to 1 in the `~/.profile` file, as that would likely disable parallelism for most programs that use OpenMP for parallel code execution.\n\n**Note that if we set `OMP_NUM_THREADS` to 1, then parallelism with Numba and Cython will not work.**\n\n\n```python\nimport os\n\ndef set_threads(\n    num_threads,\n    set_blas_threads=True,\n    set_numexpr_threads=True,\n    set_openmp_threads=False\n):\n    num_threads = str(num_threads)\n    if not num_threads.isdigit():\n        raise ValueError(\"Number of threads must be an integer.\")\n    if set_blas_threads:\n        os.environ[\"OPENBLAS_NUM_THREADS\"] = num_threads\n        os.environ[\"MKL_NUM_THREADS\"] = num_threads\n        os.environ[\"VECLIB_MAXIMUM_THREADS\"] = num_threads\n    if set_numexpr_threads:\n        os.environ[\"NUMEXPR_NUM_THREADS\"] = num_threads\n    if set_openmp_threads:\n        os.environ[\"OMP_NUM_THREADS\"] = num_threads\n\nset_threads(1)\n```\n\nNow, we can import numpy to our code and it will only run on only one core.\n\n\n```python\nimport numpy as np\n```\n\n## Parallel code with Joblib and multiprocessing\n\nPython does not support parallel threading. This means that each Python process can only do one thing at a time. The reason for this lies with the way Python code is run on your computer. Countless hours has been spent trying to remove this limitation, but all sucessfull attempts severly impaired the speed of the language (the most well known attempt is Larry Hasting's [gilectomy](https://github.com/larryhastings/gilectomy)). \n\nSince we cannot run code in parallel within a single process, we need to start new processes for each task we wish to compute in parallel and send the relevant information to these processes. This leads to a lot of overhead, and if we hope to have any performance gain, then we should parallelize substantial tasks if we wish to do it with multiple processes.\n\n### The best approach: Joblib\n\nThe best approach to multiprocessing in Python is through the Joblib library. It overcomes some of the shortcomings of multiprocessing (that you may not realise is a problem until you encounter them) at the cost of an extra dependency in your code. Below, we see an example of parallel code with joblib\n\n\n```python\nfrom joblib import Parallel, delayed\n\ndef f(x):\n    return x + 2\n\nnumbers1 = Parallel(n_jobs=2)(delayed(f)(x) for x in range(10))\n\nprint(numbers1)\n```\n\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n\n\nHere we see how Joblib can help us parallelize simple for loops. We wrap what we wish to compute in a function and use it in a list comprehension. The `n_jobs` argument specifies how many processes to spawn. If it is a positive number (1, 2, 4, etc.) then it is the number of processes to spawn and if it is a negative number then joblib will spawn (n_cpu_threads + 1 - n_processes). Thus `n_jobs=-1` will spawn as many processes as there are CPU threads available, `n_jobs=-2` will spawn n-1 CPU threads, etc. \n\nI recommend setting `n_jobs=-2` so you have one CPU thread free to surf the web while you run hard-core experiments on your computer.\n\n\n```python\nnumbers1 = Parallel(n_jobs=-2)(delayed(f)(x) for x in range(10))\n\nprint(numbers1)\n```\n\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n\n\nIf we cannot wrap all the logic within a single function, but need to have two separate parallel loops, then we should use the `Parallel` object in a slightly different fashion. If we do the following:\n\n\n```python\nfrom joblib import Parallel, delayed\n\ndef f(x):\n    return x + 2\n\nnumbers1 = Parallel(n_jobs=2)(delayed(f)(x) for x in range(10))\nnumbers2 = Parallel(n_jobs=2)(delayed(f)(x) for x in range(20))\n\n\nprint(numbers1)\nprint(numbers2)\n```\n\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]\n\n\nThen we will first create two new Python processes, compute the parallel list comprehension, close these two processess before spawning two new Python processes and computing the second parallel list comprehension. This is obviously not ideal, and we can reuse the pool of processes with a context manager:\n\n\n```python\nwith Parallel(n_jobs=2) as parallel:\n    numbers1 = parallel(delayed(f)(x) for x in range(10))\n    numbers2 = parallel(delayed(f)(x) for x in range(20))\n\nprint(numbers1)\nprint(numbers2)\n```\n\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]\n\n\nHere, the same processes are used for both list comprehensions!\n\n## Async operations with multiprocessing\nAn alternative to using Joblib for multiprocessing in Python is to use the builtin `multiprocessing` module.\nThis module is not as user friendly as joblib, and may break with weird error messages. \n\n\n```python\nimport multiprocessing\n\ndef add_2(x):\n    return x + 2\n\nwith multiprocessing.Pool(4) as p:\n    print(p.map(add_2, range(10)))\n```\n\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n\n\nHere, we see that multiprocessing also requires us to wrap the code we wish to run in parallel in a function. \n\nHowever, one particular of multiprocessing is that it requires all inputs to be picklable. That means that we cannot use output a factory function and you may also have problems with using multiprocessing with instance methods. Below is an example that fails.\n\n\n```python\nimport multiprocessing\n\ndef add(x):\n    def add_x(y):\n        return x + y\n    return add_x\n\nadd_2 = add(2)\nprint(add_2(2))\n\nwith multiprocessing.Pool(4) as p:\n    p.map(add_2, range(10))\n```\n\n    4\n\n\n\n    ---------------------------------------------------------------------------\n\n    AttributeError                            Traceback (most recent call last)\n\n    \u003cipython-input-9-6e36bedeabae\u003e in \u003cmodule\u003e\n         10 \n         11 with multiprocessing.Pool(4) as p:\n    ---\u003e 12     p.map(add_2, range(10))\n    \n\n    /usr/lib/python3.8/multiprocessing/pool.py in map(self, func, iterable, chunksize)\n        362         in a list that is returned.\n        363         '''\n    --\u003e 364         return self._map_async(func, iterable, mapstar, chunksize).get()\n        365 \n        366     def starmap(self, func, iterable, chunksize=None):\n\n\n    /usr/lib/python3.8/multiprocessing/pool.py in get(self, timeout)\n        766             return self._value\n        767         else:\n    --\u003e 768             raise self._value\n        769 \n        770     def _set(self, i, obj):\n\n\n    /usr/lib/python3.8/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)\n        535                         break\n        536                     try:\n    --\u003e 537                         put(task)\n        538                     except Exception as e:\n        539                         job, idx = task[:2]\n\n\n    /usr/lib/python3.8/multiprocessing/connection.py in send(self, obj)\n        204         self._check_closed()\n        205         self._check_writable()\n    --\u003e 206         self._send_bytes(_ForkingPickler.dumps(obj))\n        207 \n        208     def recv_bytes(self, maxlength=None):\n\n\n    /usr/lib/python3.8/multiprocessing/reduction.py in dumps(cls, obj, protocol)\n         49     def dumps(cls, obj, protocol=None):\n         50         buf = io.BytesIO()\n    ---\u003e 51         cls(buf, protocol).dump(obj)\n         52         return buf.getbuffer()\n         53 \n\n\n    AttributeError: Can't pickle local object 'add.\u003clocals\u003e.add_x'\n\n\nWe see that local functions aren't picklable, however, the same code runs with joblib:\n\n\n```python\nprint(Parallel(n_jobs=4)(delayed(add_2)(x) for x in range(10)))\n```\n\n    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n\n\n### So why use multiprocessing?\nUnfortunately, Joblib blocks the python interpreter, so that while the other processess run, no work can be done on the mother process. See the example below:\n\n\n```python\nfrom time import sleep, time\n\ndef slow_function(x):\n    sleep(3)\n    return x\n\nstart_time = time()\nParallel(n_jobs=-2)(delayed(slow_function)(i) for i in range(10))\nprint(time() - start_time)\n```\n\n    6.969935655593872\n\n\nMeanwhile, with multiprocessing, we can start the processes, let those run in the background, and do other tasks while waiting. Here is an example\n\n\n```python\nwith multiprocessing.Pool(6) as p:\n    # Start ten processes.\n    # The signature for the apply_async method is as follows\n    # apply_async(function, args, kwargs)\n    # the args iterable is fed into the function using tuple unpacking\n    # the kwargs iterable is fed into the function using dictionary unpacking\n    tasks = [p.apply_async(slow_function, [i]) for i in range(10)]\n\n    prev_ready = 0\n    num_ready = sum(task.ready() for task in tasks)\n    while num_ready != len(tasks):\n        if num_ready != prev_ready:\n            print(f\"{num_ready} out of {len(tasks)} completed tasks\")\n        prev_ready = num_ready\n        num_ready = sum(task.ready() for task in tasks)\n    results = [task.get() for task in tasks]\nprint(results)\n```\n\n    1 out of 10 completed tasks\n    2 out of 10 completed tasks\n    5 out of 10 completed tasks\n    6 out of 10 completed tasks\n    7 out of 10 completed tasks\n    8 out of 10 completed tasks\n    9 out of 10 completed tasks\n    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n\n\nThis means that, if you have to do some post processing of the output of the parallel loop, then you can start doing that with the elements that are done. Here is a very simple example\n\n\n```python\nwith multiprocessing.Pool(6) as p:\n    # Start ten processes.\n    # The signature for the apply_async method is as follows\n    # apply_async(function, args, kwargs)\n    # the args iterable is fed into the function using tuple unpacking\n    # the kwargs iterable is fed into the function using dictionary unpacking\n    tasks = [p.apply_async(slow_function, [i]) for i in range(10)]\n\n    finished = {}\n    while len(finished) != len(tasks):\n        for i, task in enumerate(tasks):\n            if task.ready():\n                if i not in finished:\n                    print(f\"Task {i} just finished, its result was {task.get()}\")\n                finished[i] = task.get()\n\nprint([finished[i] for i in range(10)])\n```\n\n    Task 0 just finished, its result was 0\n    Task 1 just finished, its result was 1\n    Task 2 just finished, its result was 2\n    Task 3 just finished, its result was 3\n    Task 4 just finished, its result was 4\n    Task 5 just finished, its result was 5\n    Task 6 just finished, its result was 6\n    Task 7 just finished, its result was 7\n    Task 8 just finished, its result was 8\n    Task 9 just finished, its result was 9\n    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n\n\n## Parallelizing with Numba\n\nNumba is a magical tool that will let us write Python code that is just in time compiled (JIT) to machine code using LLVM. This means that we can get C-speed with our Python code!\n\nUnfortunately, the price of this black magic is that Numba doesn't support the whole Python language. Rather, it supports a subset of it. Especially if you enable `nopython` mode (doing work outside the Python virtual machine) to get the best speedups.\n\nLet us start by looking at some simple non-parallel Numba tricks\n\n\n```python\nimport numba\n\ndef python_sum(A):\n    s = 0\n    for i in range(A.shape[0]):\n        s += A[i]\n\n    return s\n\n@numba.jit\ndef numba_normal_sum(A):\n    s = 0\n    for i in range(A.shape[0]):\n        s += A[i]\n\n    return s\n```\n\n\n```python\nx = np.random.randn(10000)\nprint(\"Pure python\")\n%timeit python_sum(x)\nprint(\"Numba\")\n%timeit numba_normal_sum(x)\n```\n\n    Pure python\n    2.09 ms ± 31.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n    Numba\n    10.5 µs ± 254 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n\n\nWe see that the numba compiled code is much faster than the plain python code (more than x100). The downside with writing code this way is that error messages are cryptic  and a function that is JIT compiled can only call a subset of all Python, NumPy and SciPy functions in addition to other JIT compiled functions. See the documentation for more info on this.\n\nHowever, we can also parallelize code with Numba, using the `numba.prange` (parallel range) function. This code is cheekily stolen from the [documentation](https://numba.pydata.org/numba-doc/0.11/prange.html)\n\n\n```python\n@numba.jit(parallel=True, nogil=True)\ndef numba_parallel_sum(A):\n    s = 0\n    for i in numba.prange(A.shape[0]):\n        s += A[i]\n\n    return s\n```\n\n\n```python\nx = np.random.randn(10000)\nnumba_normal_sum(x)  # compile it once\nnumba_parallel_sum(x)  # compile it once\nprint(\"Pure python\")\n%timeit python_sum(x)\nprint(\"\\nNumba\")\n%timeit numba_normal_sum(x)\nprint(\"\\nParallel Numba\")\n%timeit numba_parallel_sum(x)\n```\n\n    Pure python\n    2.09 ms ± 16.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n    \n    Numba\n    10.4 µs ± 97.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n    \n    Parallel Numba\n    28.7 µs ± 462 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n\n\nHere, we see that the performance actually deteriorates by parallising the code! This is because of the extra overhead needed to organise multiple workers. However, sometimes parallelizing code this way can lead to significant speedups (especially if each iteration is costly).\n\nWe can use Cython to reduce the overhead that we experience with the parallel sum.\n\n**Note:** It is difficult to use Numba on the outer loop, so if you have costly outer loops, then you should use Joblib to have the parallelism there instead.\n\n## Parallel code with Cython\nFinally, we look at Cython to optimise and paralellise code. Cython is a language that will let us write Python-like code that is transpiled into a Python C extension. This means several things:\n\n1. We can get C speed without much effort\n2. Cython is a superset of Python, so any Python code can be compiled\n3. It is easier to write Cython, but it requires manual compilation.\n\nThe first two points here make Cython a very attractive alternative. However, the final point can be very problematic. Whenever you make a change to a Cython file, you need to compile it again. This is generally done via a `setup.py` file that contains build instructionsf for your Cython files.\n\nLuckily, we can prototype some Cython code in a notebook, using the `%%cython` magic command.\n\n\n```python\n%load_ext Cython\n```\n\nThe code below is just copy pasted from above, but the inclusion of the `%%cython` cell magic means that the code is now compiled and can run faster its pure Python counterpart. Just copy pasting code this way will not massively improve runtime.\n\n\n```cython\n%%cython\nimport numpy as np\n\ndef cython_sum(A):\n    s = 0\n    for i in range(A.shape[0]):\n        s += A[i]\n\n    return s\n```\n\nUnfortunately, the code above is sill running in the CPython virtual machine which are a lot slower than a pure C function. Let us fix this, to do that, we avoid any Python data types, and only use the C counterparts.\n\n\n```cython\n%%cython\ncimport cython\ncimport numpy as np\nimport numpy as np\n\n@cython.boundscheck(False)  # Do not check if Numpy indexing is valid\n@cython.wraparound(False)   # Deactivate negative Numpy indexing.\ncpdef smart_cython_sum(np.ndarray[np.float_t] A):  \n# ^ Notice cpdef instead of def. Define it as a C function and Python function simultaneously.\n    cdef float s = 0\n    cdef int i\n    for i in range(A.shape[0]):\n        s += A[i]\n\n    return s\n```\n\nNow, we can look at how to make this run in parallel. To do this, we need OpenMP (which runs a bit differently on linux and windows).\n\n\n```cython\n%%cython --compile-args=-fopenmp --link-args=-fopenmp --force\nfrom cython.parallel import prange\nimport numpy as np\n\ncimport cython\ncimport numpy as np\ncimport openmp\n\n\n@cython.boundscheck(False)  # Do not check if Numpy indexing is valid\n@cython.wraparound(False)   # Deactivate negative Numpy indexing.\ncpdef parallel_cython_sum(np.ndarray[np.float_t] A):  \n# ^ Notice cpdef instead of def. Define it as a C function and Python function simultaneously.\n    cdef float s = 0\n    cdef int i\n    for i in prange(A.shape[0], nogil=True, num_threads=8):\n        s += A[i]\n\n\n    return s\n\n```\n\n\n```python\nx = np.random.randn(10000)\nprint(\"Pure python\")\n%timeit python_sum(x)\nprint(\"\\nNumba\")\n%timeit numba_normal_sum(x)\nprint(\"\\nParallel Numba\")\n%timeit numba_parallel_sum(x)\nprint(\"\\nNaive Cython\")\n%timeit cython_sum(x)\nprint(\"\\nSmart Cython\")\n%timeit smart_cython_sum(x)\nprint(\"\\nParallel Cython\")\n%timeit parallel_cython_sum(x)\n```\n\n    Pure python\n    2.25 ms ± 137 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n    \n    Numba\n    10.6 µs ± 108 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n    \n    Parallel Numba\n    33.3 µs ± 2.58 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n    \n    Naive Cython\n    2.47 ms ± 88.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n    \n    Smart Cython\n    44.4 µs ± 1.63 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n    \n    Parallel Cython\n    14.4 µs ± 3.55 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n\n\nFrom this, we see a couple of things\n\n 1. There is little difference between pure Python and naive Cython\n 2. Pure Numba is faster than sophisticated Cython for this example\n 3. The parallel code in Cython has much less overhead than the parallel code in Numba\n \nIn general, it is difficult to say if Numba or Cython will be fastest. The reason for this is that Numba may or may not be able to lift the CPython virtual machine. If it isn't able to do so, it will often be much slower than Cython.\n\nThus, if Numba works, it is often as good as, if not better than Cython. You should therefore start with Numba, and if that doesn't provide a good enough speed up, then you can try Cython.\n\nLong functions are often easier to implement in Cython and small ones are often best implemented in Numba. A big downside with Cython (that cannot be stressed enough) is that it adds huge overhead for building and distributing the code. I therefore discourage the use of Cython for code that will be made public unless all other options are tested first.\n\n## Vectorization with Numba\nFinally, we will look at vectorization of functions with Numba. Vectorization is when we wish to apply the same function to all elements of an array. For example, the `exp` function in NumPy is a vectorized function.\n\nLet us create a vectorised function to compute the Mandelbrot set. \n\n\n```python\n@np.vectorize\ndef compute_mandelbrot_np(x):\n    C = x\n    for i in range(20):\n        if abs(x) \u003e= 4:\n            return i\n\n        x = x**2 + C\n    return -1\n\n\n@numba.vectorize([numba.int32(numba.complex128)], target=\"cpu\")\ndef compute_mandelbrot_cpu(x):\n    C = x\n    for i in range(20):\n        if abs(x) \u003e= 4:\n            return i\n\n        x = x**2 + C\n    return -1\n\n@numba.vectorize([numba.int32(numba.complex128)], target=\"parallel\")\ndef compute_mandelbrot_parallel(x):\n    C = x\n    for i in range(20):\n        if abs(x) \u003e= 4:\n            return i\n\n        x = x**2 + C\n    return -1\n```\n\n\n```python\nX = -0.235\nY = 0.827\nR = 4.0e-1\n\nx = np.linspace(X - R, X + R, 100)\ny = np.linspace(Y - R, Y + R, 100)\n\nxx, yy = np.meshgrid(x, y)\nzz = xx + 1j*yy\n\ncompute_mandelbrot_cpu(zz)  # Compile once\ncompute_mandelbrot_parallel(zz)  # Compile once\nprint(\"Single core NumPy\")\n%timeit compute_mandelbrot_np(zz)\nprint(\"Single core Numba\")\n%timeit compute_mandelbrot_cpu(zz)\nprint(\"Multi core Numba\")\n%timeit compute_mandelbrot_parallel(zz)\n```\n\n    Single core NumPy\n    10.5 ms ± 3.1 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)\n    Single core Numba\n    421 µs ± 41.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n    Multi core Numba\n    396 µs ± 224 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n\n\nHere, we see not only the effect of just in time compiling our function but also that all our CPU cores are fully utilised when vectorizing functions! Let us plot a section of the mandelbrot set!\n\n\n```python\nimport matplotlib.pyplot as plt\n```\n\n\n```python\n%config InlineBackend.figure_format = 'retina'\n```\n\n\n```python\nX = -0.235\nY = 0.827\nR = 4.0e-1\n\nx = np.linspace(X - R, X + R, 1000)\ny = np.linspace(Y - R, Y + R, 1000)\n\nxx, yy = np.meshgrid(x, y)\nzz = xx + 1j*yy\nmandelbrot = compute_mandelbrot_parallel(zz)\n\nplt.figure(figsize=(10, 10), dpi=300)\nplt.imshow(mandelbrot)\nplt.axis('off')\nplt.show()\n```\n\n\n![Mandelbrot set](resources/mandelbrot.png)\n\n\n## Credits\n\nContents of this repo based off of work by [@yngvem](https://github.com/yngvem/parallelising-python).\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmachawk1%2Fparallelizing-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmachawk1%2Fparallelizing-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmachawk1%2Fparallelizing-python/lists"}