{"id":17028618,"url":"https://github.com/kevinknights29/concurrent-and-parallel-programming-in-python","last_synced_at":"2026-01-19T00:33:20.816Z","repository":{"id":232733686,"uuid":"785005297","full_name":"kevinknights29/Concurrent-and-Parallel-Programming-in-Python","owner":"kevinknights29","description":"This project implements the code examples from the course: `Concurrent and Parallel Programming in Python` by Maximilian Schallwig","archived":false,"fork":false,"pushed_at":"2024-06-15T01:42:15.000Z","size":66,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-05T20:44:58.766Z","etag":null,"topics":["concurrent-programming","parallel-programming","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kevinknights29.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-11T01:58:21.000Z","updated_at":"2024-06-15T01:41:03.000Z","dependencies_parsed_at":"2024-12-19T03:42:51.511Z","dependency_job_id":"d18ad75f-9636-47f3-9d0e-606f2b4b4eb6","html_url":"https://github.com/kevinknights29/Concurrent-and-Parallel-Programming-in-Python","commit_stats":null,"previous_names":["kevinknights29/concurrent-and-parallel-programming-in-python"],"tags_count":0,"template":false,"template_full_name":"kevinknights29/Python-Dev-Container-Template","purl":"pkg:github/kevinknights29/Concurrent-and-Parallel-Programming-in-Python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinknights29%2FConcurrent-and-Parallel-Programming-in-Python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinknights29%2FConcurrent-and-Parallel-Programming-in-Python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinknights29%2FConcurrent-and-Parallel-Programming-in-Python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinknights29%2FConcurrent-and-Parallel-Programming-in-Python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kevinknights29","download_url":"https://codeload.github.com/kevinknights29/Concurrent-and-Parallel-Programming-in-Python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinknights29%2FConcurrent-and-Parallel-Programming-in-Python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28554394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T23:48:35.235Z","status":"ssl_error","status_checked_at":"2026-01-18T23:47:49.178Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrent-programming","parallel-programming","python"],"created_at":"2024-10-14T07:55:29.916Z","updated_at":"2026-01-19T00:33:20.800Z","avatar_url":"https://github.com/kevinknights29.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Concurrent-and-Parallel-Programming-in-Python\n\nThis project implements the code examples from the course: `Concurrent and Parallel Programming in Python` by Maximilian Schallwig.\n\n## Topics\n\n1. [Overview](#overview)\n2. [Goals](#goals)\n3. [Scope and Context](#scope-and-context)\n4. [System Design](#system-design)\n5. [Alternatives Considered](#alternatives-considered)\n6. [Learning Logs](#learning-logs)\n7. [Resources](#resources)\n\n---\n\n## Overview\n\nThe project involves building a system that fetches the list of companies from the S\u0026P 500 and retrieves the stock information for each of those companies using Yahoo Finance.\n\nThe stock data is then inserted into a PostgreSQL database. The system leverages concurrent and parallel programming in Python to efficiently manage the flow of data between different components: fetching the list of companies, retrieving stock prices, and storing the data in the database.\n\n### Pipeline Feature\n\nThis project now includes a pipeline feature that allows the process to be configured from a configuration file.\n\nThe pipeline executor initializes and manages queues, workers, and schedulers based on the provided configuration, making the system highly flexible and easy to modify.\n\n### Processing Logs\n\n![image](https://github.com/kevinknights29/Concurrent-and-Parallel-Programming-in-Python/assets/74464814/12ebb4a7-5039-4746-8083-34efb062fd62)\n\n### Results from DB\n\n```sql\nselect * from public.prices;\n```\n\n![image](https://github.com/kevinknights29/Concurrent-and-Parallel-Programming-in-Python/assets/74464814/484255e0-90ac-49c6-a5f2-4571bc6e5abc)\n\n## Goals\n\nThe primary goal of this project is to learn and demonstrate the concepts of concurrent and parallel programming in Python.\n\nBy building a simple yet practical application, we aim to understand how to manage multiple tasks simultaneously, efficiently handle inter-process communication, and effectively utilize system resources.\n\nThe project showcases the use of Python's built-in queue functionality, multiprocessing, and logging modules to create a robust and scalable application.\n\n## Scope and Context\n\nThe scope of this project includes:\n\n1. Fetching the list of S\u0026P 500 companies from Wikipedia.\n2. Using multiple worker instances to retrieve stock price information concurrently from Yahoo Finance.\n3. Storing the retrieved stock data in a PostgreSQL database using multiple database worker instances.\n4. Implementing a logging system to monitor and debug the application.\n\nThe project is designed to be a learning exercise, focusing on the practical application of concurrent and parallel programming concepts. It provides a hands-on approach to understanding how to build and manage a system that performs multiple tasks simultaneously, highlighting the challenges and solutions associated with such an approach. The context of this project is educational, aimed at enhancing the developer's skills in Python and system design.\n\n## System Design\n\n### High Level Process\n\n```mermaid\ngraph TD;\n    A[Wikipedia] --\u003e|Fetches tickers| B[WikiWorker]\n    B --\u003e|Puts tickers into| C[Tickers Queue]\n    C --\u003e|Fetches tickers| D[YahooFinancePriceScheduler]\n    D --\u003e|Puts stock data into| E[Postgres Queue]\n    E --\u003e|Fetches stock data| F[PostgresScheduler]\n    D --\u003e|STOP_SIGNAL| C\n\n    subgraph System Design\n        direction TB\n        B --\u003e C\n        D --\u003e E\n        E --\u003e F\n    end\n\n    subgraph Scheduler Instances\n        direction LR\n        D[YahooFinancePriceScheduler]\n        F[PostgresScheduler]\n    end\n\n    subgraph Queues\n        direction TB\n        C[Tickers Queue]\n        E[Postgres Queue]\n    end\n```\n\n### Pipeline Design\n\n```mermaid\ngraph TD;\n    A[PipelineExecutor Initialization] --\u003e B[Initialize Queues]\n    B --\u003e C[Create Queues from Config]\n    C --\u003e D[Assign Queue Instances]\n\n    A --\u003e E[Initialize Workers]\n    E --\u003e F[Create Workers from Config]\n    F --\u003e G[Import Worker Classes]\n    G --\u003e H[Assign Input/Output Queues]\n    H --\u003e I[Instantiate Worker Classes]\n\n    A --\u003e J[Initialize Schedulers]\n    J --\u003e K[Create Schedulers from Config]\n    K --\u003e L[Import Scheduler Classes]\n    L --\u003e M[Assign Input/Output Queues]\n    M --\u003e N[Instantiate Scheduler Instances]\n\n    A --\u003e O[Join Schedulers]\n    O --\u003e P[Collect Schedulers]\n    P --\u003e Q[Join Scheduler Instances]\n\n    A --\u003e R[Setup Pipeline]\n    R --\u003e B\n    R --\u003e E\n    R --\u003e J\n```\n\n## Alternatives Considered\n\n## Learning Logs\n\n| Date | Learning |\n|------|----------|\n| 06-08-2024 | Psycopg3 (psycopg) has been realized and offers optimizations for psycopg2. |\n| 06-09-2024 | DB_HOST can be set to db (service) name in the docker-compose.yaml for connectivity between the db and the app. |\n\n## Resources\n\n- [Concurrent and Parallel Programming in Python](https://learning.oreilly.com/course/concurrent-and-parallel/9781804611944/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinknights29%2Fconcurrent-and-parallel-programming-in-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevinknights29%2Fconcurrent-and-parallel-programming-in-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinknights29%2Fconcurrent-and-parallel-programming-in-python/lists"}