{"id":21453434,"url":"https://github.com/lapets/mr4mp","last_synced_at":"2026-03-11T13:12:53.654Z","repository":{"id":47453889,"uuid":"115942961","full_name":"lapets/mr4mp","owner":"lapets","description":"Thin MapReduce-like layer that wraps the Python multiprocessing library.","archived":false,"fork":false,"pushed_at":"2023-06-09T18:19:57.000Z","size":75,"stargazers_count":7,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-26T23:13:30.535Z","etag":null,"topics":["library","mapreduce","multiprocessing","multiprocessing-library","parallel-programming","parallel-python","python","python-library"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/mr4mp","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lapets.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-01-01T19:45:06.000Z","updated_at":"2024-07-01T19:56:38.000Z","dependencies_parsed_at":"2023-09-27T03:24:59.024Z","dependency_job_id":null,"html_url":"https://github.com/lapets/mr4mp","commit_stats":{"total_commits":87,"total_committers":4,"mean_commits":21.75,"dds":0.2068965517241379,"last_synced_commit":"a978657cb5b4167be378fa56858692cff15e2c11"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/lapets/mr4mp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapets%2Fmr4mp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapets%2Fmr4mp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapets%2Fmr4mp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapets%2Fmr4mp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lapets","download_url":"https://codeload.github.com/lapets/mr4mp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapets%2Fmr4mp/sbom","scorecard":{"id":578860,"data":{"date":"2025-08-11","repo":{"name":"github.com/lapets/mr4mp","commit":"a978657cb5b4167be378fa56858692cff15e2c11"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 1/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/lint-test-cover-docs.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/lint-test-cover-docs.yml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/lapets/mr4mp/lint-test-cover-docs.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/lint-test-cover-docs.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/lapets/mr4mp/lint-test-cover-docs.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/lint-test-cover-docs.yml:21","Warn: pipCommand not pinned by hash: .github/workflows/lint-test-cover-docs.yml:26","Warn: pipCommand not pinned by hash: .github/workflows/lint-test-cover-docs.yml:34","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T18:36:48.680Z","repository_id":47453889,"created_at":"2025-08-20T18:36:48.680Z","updated_at":"2025-08-20T18:36:48.680Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30382546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T12:49:11.341Z","status":"ssl_error","status_checked_at":"2026-03-11T12:46:41.342Z","response_time":84,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["library","mapreduce","multiprocessing","multiprocessing-library","parallel-programming","parallel-python","python","python-library"],"created_at":"2024-11-23T04:39:37.300Z","updated_at":"2026-03-11T13:12:53.638Z","avatar_url":"https://github.com/lapets.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"=====\nmr4mp\n=====\n\nThin MapReduce-like layer that wraps the Python multiprocessing library.\n\n|pypi| |readthedocs| |actions| |coveralls|\n\n.. |pypi| image:: https://badge.fury.io/py/mr4mp.svg\n   :target: https://badge.fury.io/py/mr4mp\n   :alt: PyPI version and link.\n\n.. |readthedocs| image:: https://readthedocs.org/projects/mr4mp/badge/?version=latest\n   :target: https://mr4mp.readthedocs.io/en/latest/?badge=latest\n   :alt: Read the Docs documentation status.\n\n.. |actions| image:: https://github.com/lapets/mr4mp/workflows/lint-test-cover-docs/badge.svg\n   :target: https://github.com/lapets/mr4mp/actions/workflows/lint-test-cover-docs.yml\n   :alt: GitHub Actions status.\n\n.. |coveralls| image:: https://coveralls.io/repos/github/lapets/mr4mp/badge.svg?branch=main\n   :target: https://coveralls.io/github/lapets/mr4mp?branch=main\n   :alt: Coveralls test coverage summary.\n\nPurpose\n-------\nThis package provides a streamlined interface for the built-in Python `multiprocessing \u003chttps://docs.python.org/3/library/multiprocessing.html\u003e`__ library. The interface makes it possible to parallelize in a succinct way (sometimes using only one line of code) a data workflow that can be expressed in a  `MapReduce \u003chttps://en.wikipedia.org/wiki/MapReduce\u003e`__-like form. More background information about this package's design and implementation, as well a detailed use case, can be found in a `related article \u003chttps://github.com/python-supply/map-reduce-and-multiprocessing\u003e`__.\n\nInstallation and Usage\n----------------------\nThis library is available as a `package on PyPI \u003chttps://pypi.org/project/mr4mp\u003e`__:\n\n.. code-block:: bash\n\n    python -m pip install mr4mp\n\nThe library can be imported in the usual way:\n\n.. code-block:: python\n\n    import mr4mp\n\nExamples\n^^^^^^^^\n\nIn addition to the `use case in a related article \u003chttps://github.com/python-supply/map-reduce-and-multiprocessing\u003e`__ and the example below, smaller examples for each of the methods can be found in the `documentation \u003chttps://mr4mp.readthedocs.io\u003e`__.\n\nWord-Document Index\n~~~~~~~~~~~~~~~~~~~\n\nAssume there exists a collection of documents and that each document contains a collection of 7-character \"words\". This example demonstrates how a dictionary that associates each word to the collection of documents in which that word appears can be built by leveraging `multiprocessing \u003chttps://docs.python.org/3/library/multiprocessing.html\u003e`__ and the `MapReduce \u003chttps://en.wikipedia.org/wiki/MapReduce\u003e`__ paradigm. Suppose the function definitions below are found within a module ``example.py``:\n\n.. code-block:: python\n\n    from random import choice\n    from string import ascii_lowercase\n    from uuid import uuid4\n\n    def word():\n        \"\"\"Generate a random 7-character 'word'.\"\"\"\n        return ''.join(choice(ascii_lowercase) for _ in range(7))\n\n    def doc():\n        \"\"\"Generate a random 25-word 'document' and its identifier.\"\"\"\n        return ([word() for _ in range(25)], uuid4())\n\n    def docs():\n        \"\"\"Generate list of 50 random 'documents'.\"\"\"\n        return [doc() for _ in range(50)]\n\n    def word_to_doc_id_dict(document):\n        \"\"\"Build a dictionary mapping the 'words' in a 'document' to its identifier.\"\"\"\n        (words, identifier) = document\n        return {w: {identifier} for w in words}\n\n    def merge_dicts(d, e):\n        \"\"\"Merge two dictionaries ``d`` and ``e``.\"\"\"\n        return {w: (d.get(w, set()) | e.get(w, set())) for w in d.keys() | e.keys()}\n\n.. |pool| replace:: ``pool``\n.. _pool: https://mr4mp.readthedocs.io/en/2.7.1/_source/mr4mp.html#mr4mp.mr4mp.pool\n\nThe code below (also included in ``example.py``) constructs a dictionary that maps each individual word to the set of document identifiers in which that word appears. The code does so by incrementally building up larger and larger dictionaries (starting from one dictionary per document via the ``word_to_doc_id_dict`` function and merging them via the ``merge_dicts`` function), all while using the maximum number of processes supported by the system: \n\n.. code-block:: python\n\n    if __name__ == '__main__':\n        import mr4mp\n        from timeit import default_timer\n\n        start = default_timer()\n        p = mr4mp.pool()\n        p.mapreduce(word_to_doc_id_dict, merge_dicts, docs())\n        p.close()\n        print(\n            \"Finished in \" + str(default_timer()-start) + \"s \" +\n            \"using \" + str(len(p)) + \" process(es).\"\n        )\n\nNote that any code invoking library methods must be protected inside an ``if __name__ == '__main__':`` block to ensure that the `multiprocessing \u003chttps://docs.python.org/3/library/multiprocessing.html\u003e`__ library methods can `safely load the module \u003chttps://docs.python.org/3/library/multiprocessing.html#the-process-class\u003e`__ without causing side effects. Executing the module might yield the output below:\n\n.. code-block:: bash\n\n    python example.py\n    Finished in 0.664681524217187s using 2 process(es).\n\nSuppose that it is explicitly indicated (by adding ``processes=1`` to the invocation of |pool|_) that only one process can be used:\n\n.. code-block:: python\n\n    p = mr4mp.pool(processes=1)\n\nAfter the above modification, executing the module might yield the output below:\n\n.. code-block:: bash\n\n    python example.py\n    Finished in 2.23329004518571s using 1 process(es).\n\nDevelopment\n-----------\nAll installation and development dependencies are fully specified in ``pyproject.toml``. The ``project.optional-dependencies`` object is used to `specify optional requirements \u003chttps://peps.python.org/pep-0621\u003e`__ for various development tasks. This makes it possible to specify additional options (such as ``docs``, ``lint``, and so on) when performing installation using `pip \u003chttps://pypi.org/project/pip\u003e`__:\n\n.. code-block:: bash\n\n    python -m pip install .[docs,lint]\n\nDocumentation\n^^^^^^^^^^^^^\nThe documentation can be generated automatically from the source files using `Sphinx \u003chttps://www.sphinx-doc.org\u003e`__:\n\n.. code-block:: bash\n\n    python -m pip install .[docs]\n    cd docs\n    sphinx-apidoc -f -E --templatedir=_templates -o _source .. \u0026\u0026 make html\n\nTesting and Conventions\n^^^^^^^^^^^^^^^^^^^^^^^\nAll unit tests are executed and their coverage is measured when using `pytest \u003chttps://docs.pytest.org\u003e`__ (see the ``pyproject.toml`` file for configuration details):\n\n.. code-block:: bash\n\n    python -m pip install .[test]\n    python -m pytest\n\nSome unit tests are included in the module itself and can be executed using `doctest \u003chttps://docs.python.org/3/library/doctest.html\u003e`__:\n\n.. code-block:: bash\n\n    python src/mr4mp/mr4mp.py -v\n\nStyle conventions are enforced using `Pylint \u003chttps://pylint.readthedocs.io\u003e`__:\n\n.. code-block:: bash\n\n    python -m pip install .[lint]\n    python -m pylint src/mr4mp test/test_mr4mp.py\n\nContributions\n^^^^^^^^^^^^^\nIn order to contribute to the source code, open an issue or submit a pull request on the `GitHub page \u003chttps://github.com/lapets/mr4mp\u003e`__ for this library.\n\nVersioning\n^^^^^^^^^^\nBeginning with version 0.1.0, the version number format for this library and the changes to the library associated with version number increments conform with `Semantic Versioning 2.0.0 \u003chttps://semver.org/#semantic-versioning-200\u003e`__.\n\nPublishing\n^^^^^^^^^^\nThis library can be published as a `package on PyPI \u003chttps://pypi.org/project/mr4mp\u003e`__ by a package maintainer. First, install the dependencies required for packaging and publishing:\n\n.. code-block:: bash\n\n    python -m pip install .[publish]\n\nEnsure that the correct version number appears in ``pyproject.toml``, and that any links in this README document to the Read the Docs documentation of this package (or its dependencies) have appropriate version numbers. Also ensure that the Read the Docs project for this library has an `automation rule \u003chttps://docs.readthedocs.io/en/stable/automation-rules.html\u003e`__ that activates and sets as the default all tagged versions. Create and push a tag for this version (replacing ``?.?.?`` with the version number):\n\n.. code-block:: bash\n\n    git tag ?.?.?\n    git push origin ?.?.?\n\nRemove any old build/distribution files. Then, package the source into a distribution archive:\n\n.. code-block:: bash\n\n    rm -rf build dist src/*.egg-info\n    python -m build --sdist --wheel .\n\nFinally, upload the package distribution archive to `PyPI \u003chttps://pypi.org\u003e`__:\n\n.. code-block:: bash\n\n    python -m twine upload dist/*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flapets%2Fmr4mp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flapets%2Fmr4mp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flapets%2Fmr4mp/lists"}