{"id":20710183,"url":"https://github.com/oxylabs/concurrency-vs-parallelism","last_synced_at":"2026-03-17T13:55:19.278Z","repository":{"id":134336563,"uuid":"554193102","full_name":"oxylabs/concurrency-vs-parallelism","owner":"oxylabs","description":"Learn the difference between concurrency and parallelism","archived":false,"fork":false,"pushed_at":"2025-02-11T12:57:12.000Z","size":11,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-11T13:44:10.587Z","etag":null,"topics":["concurrency","paralelism-vs-concurrency","parallel","parallel-vs-concurent","threading","web-scraping-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-19T12:03:46.000Z","updated_at":"2025-02-11T12:57:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"822736ec-1455-4362-996f-99aaf7931a5f","html_url":"https://github.com/oxylabs/concurrency-vs-parallelism","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fconcurrency-vs-parallelism","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fconcurrency-vs-parallelism/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fconcurrency-vs-parallelism/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fconcurrency-vs-parallelism/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/concurrency-vs-parallelism/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242980798,"owners_count":20216299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","paralelism-vs-concurrency","parallel","parallel-vs-concurent","threading","web-scraping-api"],"created_at":"2024-11-17T02:10:25.218Z","updated_at":"2026-03-17T13:55:19.267Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Concurrency vs. Parallelism\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=concurrency-vs-parallelism-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n\n[\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Concurrency+vs+Parallelism\u0026color=brightgreen\" /\u003e](https://github.com/topics/Concurrency-vs-Parallelism)\n\n- [What is concurrency?](#what-is-concurrency)\n- [What is a thread?](#what-is-a-thread)\n- [Practical example](#practical-example)\n- [Using concurrency to speed up processes](#using-concurrency-to-speed-up-processes)\n- [What is parallelism?](#what-is-parallelism)\n- [Using parallelism to speed up processes](#using-parallelism-to-speed-up-processes)\n\nThis article gives you an overview of the differences between concurrency and parallelism. \n\nThis article gives you an overview of cURL.\n\nFor a detailed explanation, see our [blog post](https://oxylabs.io/blog/concurrency-vs-parallelism).\n\n## What is concurrency?\n\nConcurrency is pausing and resuming threads.\n\nThis capability of modern CPUs to pause and resume tasks so fast gives an illusion that the tasks are running in parallel. **However, this is not parallel. This is concurrent.**\n\nConcurrency can be broadly understood as multi-threading. There are usually many ways of creating concurrent applications, and threading is just one of them. \n\n### What is a thread?\n\nIn broad terms, a thread is the smallest set of tasks that can be handled and managed by the operating system without any dependencies on each other. \n\nPython provides a powerful threading module for creating and managing threads. \n\n## Practical example\n\nTo understand how concurrency works, let’s solve a practical problem. The task is to process over 200 pages as fast as possible. Here are the details:\n\n**Step 1.** Go to the Wikipedia page with [a list of countries by population](https://en.wikipedia.org/wiki/List_of_countries_by_population_(United_Nations)) and get the links of all the 233 countries listed on this page.\n\n**Step 2.** Go to all these 233 pages and save the HTML locally.\n\nLet’s create a function to get all the links. At first, we won’t involve concurrency or parallelism here.\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\nfrom urllib.parse import urljoin\n\ndef get_links():\n    countries_list = 'https://en.wikipedia.org/wiki/List_of_countries_by_population_(United_Nations)'\n    all_links = []\n    response = requests.get(countries_list)\n    soup = BeautifulSoup(response.text, \"lxml\")\n    countries_el = soup.select('td .flagicon+ a')\n    for link_el in countries_el:\n        link = link_el.get(\"href\")\n        link = urljoin(countries_list, link)\n        all_links.append(link)\n    return all_links\n```\n\nCreate a function to fetch and save a link.\n\n```python\ndef fetch(link):\n    response = requests.get(link)\n    with open(link.split(\"/\")[-1]+\".html\", \"wb\") as f:\n        f.write(response.content)\n```\n\nFinally, let’s call this function in a loop:\n\n```python\nimport time\n\nif __name__ == '__main__':\n    links = get_links()\n    print(f\"Total pages: {len(links)}\")\n    start_time = time.time()\n    # This for loop will be optimized\n    for link in links:\n        fetch(link)\n\n    duration = time.time() - start_time\n    print(f\"Downloaded {len(links)} links in {duration} seconds\")\n```\n\nWith our computer, this took **137.37 seconds.** Our objective is to bring this time down.\n\n### Using concurrency to speed up processes\n\nAlthough we can create threads manually, we’ll have to start them manually and call the join method on each thread so that the main program waits for all these threads to complete.\n\nThe better approach is to use the `ThreadPoolExecutor` class. \n\nFirst, we need to import ThreadPoolExecutor:\n\n```\nfrom concurrent.futures import ThreadPoolExecutor\n```\n\nNow, the for loop written above can be changed to the following.\n\n```\nwith ThreadPoolExecutor(max_workers=16) as executor:\n      executor.map(fetch, links)\n```\n\nThe final result is astonishing! All these 233 links were downloaded in **11.23 seconds**. \n\n## What is parallelism?\n\nThis is a type of computation in which multiple processors carry out many processes simultaneously. \n\nParallelism is multiple threads running on multiple CPUs.\n\n### Using parallelism to speed up processes\n\nLet’s start with importing the required module:\n\n```python\nfrom multiprocessing import Pool, cpu_count\n```\n\nNow we can replace the for loop in the synchronous code with this code:\n\n```python\nwith Pool(cpu_count()) as p:\n        p.map(fetch, links)\n```\n\nThis fetches all 233 links in **18.10 seconds**. It’s also noticeably faster than the synchronous version which took around 138 seconds.\n\nIf you wish to learn more about Concurrency vs. Parallelism, see our [blog post](https://oxylabs.io/blog/concurrency-vs-parallelism).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fconcurrency-vs-parallelism","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fconcurrency-vs-parallelism","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fconcurrency-vs-parallelism/lists"}