{"id":17151233,"url":"https://github.com/soda480/mpmq","last_synced_at":"2026-02-05T18:31:10.347Z","repository":{"id":38376744,"uuid":"354447837","full_name":"soda480/mpmq","owner":"soda480","description":"The mpmq module enables seamless interprocess communication between a parent and child processes when parallelizing a task across multiple workers","archived":false,"fork":false,"pushed_at":"2025-01-03T00:31:57.000Z","size":436,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-27T18:46:06.503Z","etag":null,"topics":["dockerfile","message-queue","multiprocessing","pooling","pybuilder","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soda480.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-04-04T03:36:10.000Z","updated_at":"2025-01-03T00:31:07.000Z","dependencies_parsed_at":"2025-04-13T12:01:14.153Z","dependency_job_id":"a73075ee-0491-47bb-adac-ff3954cb5ec0","html_url":"https://github.com/soda480/mpmq","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/soda480/mpmq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmpmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmpmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmpmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmpmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soda480","download_url":"https://codeload.github.com/soda480/mpmq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmpmq/sbom","scorecard":{"id":835722,"data":{"date":"2025-08-11","repo":{"name":"github.com/soda480/mpmq","commit":"2d023f5eec244f0c13f37819594fc7a8ecb48cc2"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/main.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/17 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/soda480/mpmq/main.yml/main?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:2","Warn: pipCommand not pinned by hash: Dockerfile:7","Warn: pipCommand not pinned by hash: Dockerfile:7","Info:   0 out of   1 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 26 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-23T18:59:30.665Z","repository_id":38376744,"created_at":"2025-08-23T18:59:30.665Z","updated_at":"2025-08-23T18:59:30.665Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29128681,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T18:31:02.507Z","status":"ssl_error","status_checked_at":"2026-02-05T18:29:43.962Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dockerfile","message-queue","multiprocessing","pooling","pybuilder","python3"],"created_at":"2024-10-14T21:37:33.989Z","updated_at":"2026-02-05T18:31:10.328Z","avatar_url":"https://github.com/soda480.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mpmq\n[![GitHub Workflow Status](https://github.com/soda480/mpmq/workflows/build/badge.svg)](https://github.com/soda480/mpmq/actions)\n[![vulnerabilities](https://img.shields.io/badge/vulnerabilities-None-brightgreen)](https://pypi.org/project/bandit/)\n[![coverage](https://img.shields.io/badge/coverage-99%25-brightgreen)](https://pybuilder.io/)\n[![complexity](https://img.shields.io/badge/complexity-A-brightgreen)](https://radon.readthedocs.io/en/latest/api.html#module-radon.complexity)\n[![PyPI version](https://badge.fury.io/py/mpmq.svg)](https://badge.fury.io/py/mpmq)\n[![python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-teal)](https://www.python.org/downloads/)\n\nThe mpmq module enables seamless interprocess communication between a parent and child processes when parallelizing a task across multiple workers. The `MPmq` class defines a custom log handler that sends all log messages from child workers to a thread-safe queue that the parent can consume and handle. This is helpful in cases where you want the parent to show real-time progress of child workers as they execute a task.\n\n### Installation\n```bash\npip install mpmq\n```\n\n### `MPmq class`\n```\nmpmq.MPmq(function, process_data=None, shared_data=None, processes_to_start=None)\n```\n\u003e `function` - the function represents the task you wish the child workers to execute\n\n\u003e `process_data` - list of dictionaries where each dictionary contains the arguments that will be sent to each background child process executing the function; the length of the list dictates the total number of processes that will be executed\n\n\u003e `shared_data` - a dictionary containing arbitrary data that will be sent to all processes as key word arguments\n\n\u003e `process_to_start` - the number of processes to initially start; this represents the number of concurrent processes that will be running. If the total number of processes is greater than this \nnumber then execution will be queued and executed to ensure that this concurrency is maintained\n\n\u003e **execute(raise_if_error=False)**\n\u003e\u003e Start execution the process’s activity. If `raise_if_error` is set to True, an exception will be raised if any function encountered an error during execution.\n\n\u003e **process_message(offset, message)**\n\u003e\u003e Process a message sent from one of the background workers executing the function. The `offset` represents the index of the executing Process; this number is the same as the corresponding index within the `process_data` list that was sent to the constructor. The `message` represents the message that was logged by the function. \n\n### Examples\n\n The primary intent is for the MPmq class to be used as a superclass where the subclass ovverrides the `process_message` method to handle messages coming in from the child workers. The following example demonstrate how this can be done.\n\n#### [Worker Status as a Progress Bar](https://github.com/soda480/mpmq/blob/main/examples/example1.py)\n\nThe example parallizezes a task across multiple processes using a pool of worker processes. Status of each worker is shown as a Progress Bar, as each Child worker in the pool completes an item defined in the task the Parent updates a Progress Bar.\n\n![example](https://raw.githubusercontent.com/soda480/mpmq/main/docs/images/example1.gif)\n\n\n#### [Worker Status as a List](https://github.com/soda480/mpmq/blob/main/examples/example2.py)\n\nThe example parallizezes a task across multiple processes using a pool of worker processes. Status of each worker is shown using an array where each index of the array represents an individual worker, as each Child worker in the pool completes the associated item in the List is updated with the completed message.\n\n![example](https://raw.githubusercontent.com/soda480/mpmq/main/docs/images/example2.gif)\n\n\n### Projects using `mpmq`\n\n* [`mpcurses`](https://pypi.org/project/mpcurses/) An abstraction of the Python curses and multiprocessing libraries providing function execution and runtime visualization capabilities\n\n* [`mppbars`](https://pypi.org/project/mppbar/) Scale execution of a function across multiple across a number of background processes while displaying their execution status via a progress bar\n\n* [`mp4ansi`](https://pypi.org/project/mp4ansi/) A simple ANSI-based terminal emulator that provides multi-processing capabilities\n\n### Development\n\nClone the repository and ensure the latest version of Docker is installed on your development server.\n\nBuild the Docker image:\n```sh\ndocker image build \\\n-t mpmq:latest .\n```\n\nRun the Docker container:\n```sh\ndocker container run \\\n--rm \\\n-it \\\n-v $PWD:/code \\\nmpmq:latest \\\nbash\n```\n\nExecute the build:\n```sh\npyb -X\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoda480%2Fmpmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoda480%2Fmpmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoda480%2Fmpmq/lists"}