{"id":41186305,"url":"https://github.com/adpena/achilles","last_synced_at":"2026-01-22T20:17:13.306Z","repository":{"id":57407953,"uuid":"196734811","full_name":"adpena/achilles","owner":"adpena","description":"Distributed/parallel computing in modern Python based on the multiprocessing.Pool API (map, imap, imap_unordered).","archived":false,"fork":false,"pushed_at":"2019-10-17T23:22:53.000Z","size":238,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-01T22:51:49.453Z","etag":null,"topics":["achilles","clusters","distributed","distributed-computing","distributed-systems","python","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adpena.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-13T15:07:27.000Z","updated_at":"2019-10-17T23:22:55.000Z","dependencies_parsed_at":"2022-09-13T04:50:39.165Z","dependency_job_id":null,"html_url":"https://github.com/adpena/achilles","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"purl":"pkg:github/adpena/achilles","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adpena%2Fachilles","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adpena%2Fachilles/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adpena%2Fachilles/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adpena%2Fachilles/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adpena","download_url":"https://codeload.github.com/adpena/achilles/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adpena%2Fachilles/sbom","scorecard":{"id":167575,"data":{"date":"2025-08-11","repo":{"name":"github.com/adpena/achilles","commit":"ec1cc38c27f4ff6ddcf5fc4a08ccdbe8717d1100"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"14 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2021-142 / GHSA-8q59-q68h-6hv4","Warn: Project is vulnerable to: PYSEC-2018-49 / GHSA-rprw-h62v-c2w7","Warn: Project is vulnerable to: GHSA-32gv-6cf3-wcmq","Warn: Project is vulnerable to: PYSEC-2020-214 / GHSA-3gqj-cmxr-p4x2","Warn: Project is vulnerable to: PYSEC-2019-129 / GHSA-65rm-h285-5cc5","Warn: Project is vulnerable to: PYSEC-2019-128 / GHSA-6cc5-2vg4-cc7m","Warn: Project is vulnerable to: GHSA-8r99-h8j2-rw64","Warn: Project is vulnerable to: PYSEC-2022-195 / GHSA-c2jg-hw38-jrqq","Warn: Project is vulnerable to: GHSA-c8m8-j448-xjx7","Warn: Project is vulnerable to: PYSEC-2024-75 / GHSA-cf56-g6w6-pqq2","Warn: Project is vulnerable to: PYSEC-2020-259 / GHSA-h96w-mmrf-2h6v","Warn: Project is vulnerable to: PYSEC-2020-260 / GHSA-p5xh-vx83-mxcj","Warn: Project is vulnerable to: PYSEC-2023-224 / GHSA-xc8x-vp79-p3wm","Warn: Project is vulnerable to: PYSEC-2019-212"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-16T15:20:28.370Z","repository_id":57407953,"created_at":"2025-08-16T15:20:28.370Z","updated_at":"2025-08-16T15:20:28.370Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28670366,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T19:36:09.361Z","status":"ssl_error","status_checked_at":"2026-01-22T19:36:05.567Z","response_time":144,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["achilles","clusters","distributed","distributed-computing","distributed-systems","python","python3"],"created_at":"2026-01-22T20:17:13.252Z","updated_at":"2026-01-22T20:17:13.300Z","avatar_url":"https://github.com/adpena.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `achilles`\nDistributed/parallel computing in modern Python based on the `multiprocessing.Pool` API (`map`, `imap`, `imap_unordered`).\n\n## What/why is it?\nThe purpose of `achilles` is to make distributed/parallel computing as easy as possible by limiting the required configuration, hiding the details (server/node/controller architecture) and exposing a simple interface based on the popular `multiprocessing.Pool` API.\n\n\u003e `achilles` provides developers with entry-level capabilities for concurrency across a network of machines (see PEP 372 on the intent behind adding `multiprocessing` to the standard library -\u003e https://www.python.org/dev/peps/pep-0371/) using a server/node/controller architecture.\n\nThe `achilles_server`, `achilles_node` and `achilles_controller` are designed to run cross-platform/cross-architecture. The server/node/controller may be hosted on a single machine (for development) or deployed across heterogeneous resources.\n\n`achilles` is comparable to excellent Python packages like `pathos/pyina`, `Parallel Python` and `SCOOP`, but different in certain ways:\n- Designed for developers familiar with the `multiprocessing` module in the standard library with simplicity and ease of use in mind.\n- In addition to the blocking `map` API which requires that developers wait for all computation to be finished before accessing results (common in such packages), `imap`/`imap_unordered` allow developers to process results as they are returned to the `achilles_controller` by the `achilles_server`.\n- `achilles` allows for composable scalability and novel design patterns as:\n    - Iterables including lists, lists of lists and generator functions (as first-class object - generator expressions will not work as generators cannot be serialized by `pickle`/`dill`) are accepted as arguments.\n        - TIP: Use generator functions together with `imap` or `imap_unordered` to perform distributed computation on arbitrarily large data.\n    - The `dill` serializer is used to transfer data between the server/node/controller and `multiprocess` (fork of `multiprocessing` that uses the `dill` serializer instead of `pickle`) is used to perform `Pool.map` on the `achilles_nodes`, so developers are freed from some of the constraints of the `pickle` serializer.\n    \u003cbr/\u003e\n\n### Install\n`pip install achilles`\n\n### Quick Start\nStart an `achilles_server` listening for connections from `achilles_nodes` at a certain endpoint specified as arguments or in an `.env` file in the `achilles` package's directory.\n\nThen simply import `map`, `imap`, and/or `imap_unordered` from `achilles_main` and use them dynamically in your own code (under the hood they create and close `achilles_controller`s).\n\n`map`, `imap` and `imap_unordered` will distribute your function to each `achilles_node` connected to the `achilles_server`. Then, the `achilles_server` will distribute arguments to each `achilles_node` (load balanced and made into a list of arguments if the arguments' type is not already a list) which will then perform your function on the arguments using `multiprocess.Pool.map`.\n\nEach `achilles_node` finishes its work, returns the results to the `achilles_server` and waits to receive another argument. This process is repeated until all of the arguments have been exhausted.\n\n1) `runAchillesServer(host=None, port=None, username=None, secret_key=None)` -\u003e run on your local machine or on another machine connected to your network\n\n    `in:`\n    ```python\n    from achilles.lineReceiver.achilles_server import runAchillesServer\n    \n   # host = IP address of the achilles_server\n   # port = port to listen on for connections from achilles_nodes (must be an int)\n   # username, secret_key used for authentication with achilles_controller\n    \n   runAchillesServer(host='127.0.0.1', port=9999, username='foo', secret_key='bar')\n   ```\n   \n   ```python\n      \n   # OR generate an .env file with a default configuration so that\n   # arguments are no longer required to runAchillesServer()\n   \n   # use genConfig() to overwrite\n\n   from achilles.lineReceiver.achilles_server import runAchillesServer, genConfig\n   \n   genConfig(host='127.0.0.1', port=9999, username='foo', secret_key='bar')\n   runAchillesServer()\n    ```\n    \u003cbr/\u003e\n    \n    `out:`\n    ```\n    ALERT: achilles_server initiated at 127.0.0.1:9999\n    Listening for connections...\n    ```\n\n2) `runAchillesNode(host=None, port=None)` -\u003e run on your local machine or on another machine connected to your network\n    \n    `in:`\n    ```python\n    from achilles.lineReceiver.achilles_node import runAchillesNode\n   \n   # genConfig() is also available in achilles_node, but only expects host and port arguments\n    \n    runAchillesNode(host='127.0.0.1', port=9999)\n    ```\n    \u003cbr/\u003e   \n    \n    `out:`\n    ```\n    GREETING: Welcome! There are currently 1 open connections.\n    \n    Connected to achilles_server running at 127.0.0.1:9999\n    CLIENT_ID: 0\n    ```\n        \n3) Examples of how to use the 3 most commonly used `multiprocessing.Pool` methods in `achilles`:\n    \u003cbr/\u003e\n    \u003e\u003e Note: `map`, `imap` and `imap_unordered` currently accept iterables including - but not limited - to lists, lists of lists, and generator functions as `achilles_args`.\n                                                         \n    \u003e\u003e Also note:  if there isn't already a `.env` configuration file in the `achilles` package directory, must use `genConfig(host, port, username, secret_key)` before using or include `host`, `port`, `username` and `secret_key` as arguments when using `map`, `imap`, `imap_unordered`.\n    \n    1) `map(func, args, callback=None, chunksize=1, host=None, port=None, username=None, secret_key=None)`\u003cbr/\u003e\u003cbr/\u003e\n        `in:`\n        ```python\n       from achilles.lineReceiver.achilles_main import map\n       \n       def achilles_function(arg):\n           return arg ** 2\n       \n       def achilles_callback(result):\n           return result ** 2\n       \n       if __name__ == \"__main__\":\n           results = map(achilles_function, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], achilles_callback, chunksize=1)\n           print(results)\n       ```\n       \u003cbr/\u003e\n       \n       `out:`\n       ```\n       ALERT: Connection to achilles_server at 127.0.0.1:9999 and authentication successful.\n       \n       [[1, 16, 81, 256, 625, 1296, 2401, 4096], [6561, 10000]]\n       ```\n       \n   2) `imap(func, args, callback=None, chunksize=1, host=None, port=None, username=None, secret_key=None)`\u003cbr/\u003e\u003cbr/\u003e\n        `in:`\n        ```python\n        from achilles.lineReceiver.achilles_main import imap\n      \n      def achilles_function(arg):\n          return arg ** 2 \n      \n      def achilles_callback(result):\n          return result ** 2\n      \n      if __name__ == \"__main__\":\n          for result in imap(achilles_function, [1,2,3,4,5,6,7,8,9,10], achilles_callback, chunksize=1):\n              print(result)\n        ```\n      \u003cbr/\u003e\n      \n      `out:`\n      ``` \n      ALERT: Connection to achilles_server at 127.0.0.1:9999 and authentication successful.\n\n      {'ARGS_COUNTER': 0, 'RESULT': [1, 16, 81, 256, 625, 1296, 2401, 4096]}\n      {'ARGS_COUNTER': 8, 'RESULT': [6561, 10000]}\n      ```\n\n     3) `imap_unordered(func, args, callback=None, chunksize=1, host=None, port=None, username=None, secret_key=None)`\u003cbr/\u003e\u003cbr/\u003e\n     `in:`\n        ```python\n        from achilles.lineReceiver.achilles_main import imap_unordered\n        \n        def achilles_function(arg):\n            return arg ** 2\n        \n        def achilles_callback(result):\n            return result ** 2\n      \n        if __name__ == \"__main__\":\n            for result in imap_unordered(achilles_function, [1,2,3,4,5,6,7,8,9,10], achilles_callback, chunksize=1):\n                print(result)\n        ```\n        \u003cbr/\u003e\n        \n        `out:`\n        ``` \n        ALERT: Connection to achilles_server at 127.0.0.1:9999 and authentication successful.\n\n        {'ARGS_COUNTER': 8, 'RESULT': [6561, 10000]}\n        {'ARGS_COUNTER': 0, 'RESULT': [1, 16, 81, 256, 625, 1296, 2401, 4096]}\n        ```\n\n## How `achilles` works \n\n### Under the hood\n- `Twisted`\n    - An event-driven networking engine written in Python and MIT licensed.\n- `dill`\n    - `dill` extends Python’s `pickle` module for serializing and de-serializing Python objects to the majority of the built-in Python types.\n- `multiprocess`\n    - multiprocess is a fork of multiprocessing that uses `dill` instead of `pickle` for serialization. `multiprocessing` is a package for the Python language which supports the spawning of processes using the API of the standard library’s threading module.\n\n### Examples\nSee the `examples` directory for tutorials on various use cases, including:\n- Square numbers/run multiple jobs sequentially\n- Word count (TO DO)\n\n### How to kill cluster\n```python\nfrom achilles.lineReceiver.achilles_main import killCluster\n\n# simply use the killCluster() command and verify your intent at the prompt\n# killCluster() will search for an .env configuration file in the achilles package's directory\n\n# if it does not exist, specify host, port, username and secret_key as arguments\n# a command is sent to all connected achilles_nodes to stop the Twisted reactor and exit() the process\n\n# optionally, you can pass command_verified=True to proceed directly with killing the cluster\n\nkillCluster(command_verified=True)\n```\n\n### Caveats/Things to know\n- `achilles_node`s use all of the CPU cores available on the host machine to perform `multiprocess.Pool.map` (`pool = multiprocess.Pool(multiprocess.cpu_count())`).\n- `achilles` leaves it up to the developer to ensure that the correct packages are installed on `achilles_node`s to perform the function distributed by the `achilles_server` on behalf of the `achilles_controller`. Current recommended solution is to SSH into each machine and `pip install` a `requirements.txt` file.\n- All import statements required by the developer's function, arguments and callback must be included in the definition of the function.\n- The `achilles_server` is currently designed to handle one job at a time. For more complicated projects, I highly recommend checking out `Dask` (especially `dask.distributed`) and learning more about directed acyclic graphs (DAGs).\n- Fault tolerance: if some `achilles_node` disconnects before returning expected results, the argument will be distributed to another `achilles_node` for computation instead of being lost.\n- `callback_error` argument has yet to be implemented, so detailed information regarding errors can only be gleaned from the interpreter used to launch the `achilles_server`, `achilles_node` or `achilles_controller`. Deploying the server/node/controller on a single machine is recommended for development.\n- `achilles` performs load balancing at runtime and assigns `achilles_node`s arguments by `cpu_count` * `chunksize`.\n    - Default `chunksize` is 1.\n    - Increasing the `chunksize` is an easy way to speed up computation and reduce the amount of time spent transferring data between the server/node/controller.\n- If your arguments are already lists, the `chunksize` argument is not used.\n    - Instead, one argument/list will be distributed to the connected `achilles_node`s at a time.\n- If your arguments are load balanced, the results returned are contained in lists of length `achilles_node's cpu_count` *  `chunksize`.\n    - `map`:\n        - Final result of `map` is an ordered list of load balanced lists (the final result is not flattened).\n    - `imap`:\n        - Results are returned as computation is finished in dictionaries that include the following keys:\n            - `RESULT`: load balanced list of results.\n            - `ARGS_COUNTER`: index of first argument (0-indexed).\n         - Results are ordered.\n             - The first result will correspond to the next result after the last result in the preceding results packet's list of results.\n             - Likely to be slower than `immap_unordered` due to `achilles_controller` yielding ordered results. `imap_unordered` (see below) yields results as they are received, while `imap` yields results as they are received only if the argument's `ARGS_COUNTER` is expected based on the length of the `RESULT` list in the preceding results packet. Otherwise, a `result_buffer` is checked for the results packet with the expected `ARGS_COUNTER` and the current results packet is added to the `result_buffer`. If it is not found, `achilles_controller` will not yield results until a results packet with the expected `ARGS_COUNTER` is received.\n    - `imap_unordered`:\n        - Results are returned as computation is finished in dictionaries that include the following keys:\n            - `RESULT`: load balanced list of results.\n            - `ARGS_COUNTER`: index of first argument (0-indexed).\n         - Results are not ordered.\n             - Results packets are yielded as they are received (after any `achilles_callback` has been performed on it).\n             - Fastest way of consuming results received from the `achilles_server`.\n             \n\n\u003chr/\u003e\n\n`achilles` is in the early stages of active development and your suggestions/contributions are kindly welcomed.\n\n`achilles` is written and maintained by Alejandro Peña. Email me at adpena at gmail dot com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadpena%2Fachilles","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadpena%2Fachilles","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadpena%2Fachilles/lists"}