{"id":13533505,"url":"https://github.com/polluxio/pollux-payload","last_synced_at":"2026-03-14T04:03:25.498Z","repository":{"id":236089453,"uuid":"446375048","full_name":"polluxio/pollux-payload","owner":"polluxio","description":"Pollux payload core files and examples","archived":false,"fork":false,"pushed_at":"2025-02-12T14:45:16.000Z","size":20057,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-12T14:46:08.939Z","etag":null,"topics":["cloud","message-passing","parallel-computing","parallel-programming"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/polluxio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null}},"created_at":"2022-01-10T10:23:01.000Z","updated_at":"2025-02-12T14:27:35.000Z","dependencies_parsed_at":"2024-04-25T16:31:16.477Z","dependency_job_id":"9d351c14-5d3e-497f-81e4-110bfb003b8d","html_url":"https://github.com/polluxio/pollux-payload","commit_stats":null,"previous_names":["polluxio/pollux-payload"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polluxio%2Fpollux-payload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polluxio%2Fpollux-payload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polluxio%2Fpollux-payload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polluxio%2Fpollux-payload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/polluxio","download_url":"https://codeload.github.com/polluxio/pollux-payload/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246713477,"owners_count":20821895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","message-passing","parallel-computing","parallel-programming"],"created_at":"2024-08-01T07:01:20.488Z","updated_at":"2026-03-14T04:03:23.583Z","avatar_url":"https://github.com/polluxio.png","language":"C++","funding_links":[],"categories":["Software"],"sub_categories":["Trends"],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg width=\"150\" alt=\"Pollux Logo\" src=\"./docs/images/pollux-logo.jpg\"\u003e\n\u003c/div\u003e\n\n---\n\n# Pollux\n\n![build](https://github.com/polluxio/pollux-payload/actions/workflows/test_local_mode.yml/badge.svg)\n\n**Pollux** is a Message Passing Cloud orchestrator designed to:\n\n- distribute and manage complex algorithms, including NP-Hard and Complete problems, across multiple Cloud-based machines.\n- span algorithms on the Cloud, ensuring synchronization and continuous communication among distributed components.\n- ensure **robustness** by offering recover from local crash capabilities.\n- take advantage of Cloud cheaper 'Spot' instances by designing Cloud distributed applications where part of the application can be killed while the overall application survives.\n- support multiple payload types.\n- support multiple payloads using multiple programming languages.\n- support multiple payload hardwares: design applications where CPU, GPU, FPGAs, ... are collaborating live.\n- offer easy reporting for algorithmic dimensioning and fine tuning.\n\nFollowing video shows a [PSO - Particle Swarm Optimization](https://github.com/polluxio/pollux-payload/tree/main/src/c%2B%2B/examples/pso) application example developed using **Pollux** API. This application is available for testing (see below).\n\n![Pollux PSO](./docs/images/pollux-PSO.gif)\n\n:information_desk_person: If you have any questions, send us a [mail](mailto:christophe.alexandre@getpollux.io).\n\n:star: If you find Pollux interesting, and would like to stay up-to-date, consider starring this repo to help spread the word.\n\n:exclamation: Pollux payload API is distributed under Apache 2 license.\nAdditionally, the Pollux orchestrator is made available through binary releases, which can be accessed either as direct binaries or Docker images.\n\n---\n\n## Motivation\n\nIn today's world many computational problems from network routing to semiconductors to AI are becoming more and more complex.\n\nWe believe that the next step for optimization of such complex grand scale computation is to leverage the massive parallelization the Cloud can offer.\n\nWe believe that this step has not been reached yet, as there is no genuine Cloud suitable framework for this type of computation; most of the frameworks used for similar purposes are actually meant for supercomputers, not the Cloud.\n\nOur goal is to provide such a framework, designed for Cloud architecture while easy enough to use for engineers from any field.\nThrough a clear interface they should be able to parallelize any algorithm without being Cloud experts.\n\nOur goal is not to treat the Cloud as a super computer, it is not.\nOur goal is to virtualize a super computer on the Cloud and finally open these capabilities to a wider audience.\n\n**Pollux**'s general principle is summarized in the following figure.\n\n![Pollux Schema](./docs/images/pollux-general.png)\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n\n---\n\n## Use Cases\n\nCurrently, the following **Pollux** use cases are provided:\n\n- [Pollux payload example](https://github.com/polluxio/pollux-payload/blob/main/src/c%2B%2B/examples/test): a simple test application deploying a configurable number of workers and exchanging random messages between them.\n- [Pollux PSO - Particle Swarm Optimization](https://github.com/polluxio/pollux-payload/tree/main/src/c%2B%2B/examples/pso): a [PSO](https://en.wikipedia.org/wiki/Particle_swarm_optimization) Pollux implementation (This application has been used to create the upper video).\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n\n---\n\n## Architecture\n\n**Pollux** is both:\n\n- a distributed Payload API (C++ for the moment, more to come in the future).\n- and a Cloud orchestrator.\nThe overall architecture is summarized in the following picture.\n![Pollux architecture](./docs/images/pollux-architecture.png)\n\nA REST API allowing users live access to the running application is under development.\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n\n---\n\n## Building and Installing\n\n### Getting sources\n\n```bash\n# First clone the repository\ngit clone https://github.com/polluxio/pollux-payload\n# go inside it and init submodules\ncd pollux-payload\ngit submodule update --init --recursive\n```\n\nIf you have access to a Cloud provider (we ony support [Qarnot](#cloud---qarnot) for the moment, more to come soon), \nat this point, you are good to test pollux, locally if you have docker installed or on the Cloud if you have an account.\n\n## Running\n\nRunning modes are managed through the **Pollux** [launcher](#pollux-launcher) '-m' option. The following modes are supported:\n\n### Local\n\nIf you have **Docker** installed on your system, we suggest to directly use the [local docker](#local-docker) mode.\n\nTo use this mode, **Pollux** binaries need to be installed locally. We provide binaries compiled for Ubuntu\nin the pollux-payload release.\n\nIf you face any compatibility issue or you need binaries for another Linux version,\nplease contact [us](mailto:christophe.alexandre@getpollux.io). \nOnce you have downloaded the binaries (`pollux` and `zebulon`),\nuntar the archive in a directory and set a $POLLUX_INSTALL environment variable pointing to that directory.\n\n```bash\nmkdir pollux_install\ncd pollux_install\ntar xvzf pollux.tar.gz\nexport POLLUX_INSTALL pollux_install\n```\n\nOnce this is done, run the local mode:\n\n```bash\n# Following command will launch 4 workers\npython3 \u003cpollux_payload_sources\u003e/src/launcher/pollux.py -p \u003cpath_to_payload\u003e -m local -n 4\n```\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n\n---\n\n### Local docker\n\nThis is the easiest mode to use to get started and a good first step to prepare for Cloud deployment.\n\nUse the pollux test [image](https://hub.docker.com/repository/docker/polluxio/pollux-payload-examples/general)\nto quickly test Pollux by running the following Pollux launcher command:\n\n```bash\n# Following command will launch 4 workers\n# -c : docker image containing the payload to run as a container.\n# -p : path to payload in payload container to run (usually payloads are located in /root)\npython3 \u003cpollux-payload-sources\u003e/pollux.py -m local_docker -c polluxio/pollux-payload-examples -p /root/pollux-payload-test -n 4 \n```\n\nThe following command launches the **PSO** example with 8 workers:\n\n```bash\npython3 \u003cpollux-payload-sources\u003e~/pollux.py -m local_docker -c polluxio/pollux-payload-examples -p /root/pollux-payload-pso -n 8 \n```\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n\n---\n\n### Cloud - Qarnot\n\n[Qarnot](https://qarnot.com) is a Cloud provider with a unique approach to Cloud computing in which the wasted heat generated by computer servers is recycled to heat buildings.\n\n**Qarnot** offers free trial credits accessible in a few clicks.\nTo try deployment on **Qarnot**, first create an account on this [page](https://tasq.qarnot.com/login/).\n\nRetrieve your access token, save it locally ('qarnot.conf) and then launch Pollux using the following command:\n\n```bash\n#Launching on Qarnot with 3 workers\npython3 \u003cpollux-payload-sources\u003e/pollux.py -m qarnot -c polluxio/pollux-payload-examples -p /root/pollux-payload-pso -n 3 \n```\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n\n---\n\n## Pollux launcher\n\nTo launch Pollux applications, a `Python` script launcher is provided\n[here](https://github.com/polluxio/pollux-payload/blob/main/src/launcher/pollux.py).\n\nA convenient way to proceed is to create a `Python` virtual environment. With Python 3:\n\n```bash\napt-get install python3-venv\npip3 install virtualenv\n```\n\nIn a project directory, create an environment:\n\n```bash\npython3 -m venv pollux\n```\n\nActivate it:\n\n```bash\n. pollux/bin/activate\n```\n\nNow that the environment has been created and launched, you can install the following `Python` libraries:\n\n```bash\npip install pyyaml\n#to use local Docker pollux mode\npip install docker\n#To use Qarnot Pollux mode \npip install qarnot\n```\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n\n---\n\n## Write your own Pollux payload\n\n### Supported payload languages\n\nCurrently, only `C++` is supported. `Python` support, for instance, could easily be added (some early tests have been done).\n\n### C++\n\nFirst install following dependencies, On Ubuntu for instance:\n\n```bash\nsudo apt-get install cmake\nsudo apt-get install g++\nsudo apt-get install protobuf-compiler-grpc\nsudo apt-get install libgrpc++-dev\n```\n\nCopy relevant files:\n\n```bash\n#create a thirdparty dir\nmkdir thirdparty\ncd thirdparty\ngit submodule add https://github.com/polluxio/pollux-payload\n```\n\n### Dockerfile\n\nOnce your **Pollux** payload is ready to be tested, in order to launch in [Local Docker](#local-docker)\nor on the Cloud, you need to write your own `Dockerfile`.\n\nThis [Dockerfile](https://github.com/polluxio/pollux-payload/blob/main/Dockerfile) is the best starting point:\n\n```Dockerfile\nFROM ubuntu:latest AS builder\n\n# Update Ubuntu Software repository\n# And install needed components for your payload\nRUN apt-get update \u0026\u0026 apt-get -y install \\\n    cmake \\\n    make  \\\n    g++ \\\n    protobuf-compiler-grpc\n\nWORKDIR /my-pollux-payload\n#COPY you application components\nCOPY cmake cmake\nCOPY src src\nCOPY thirdparty thirdparty\nCOPY CMakeLists.txt .\nRUN cmake . \u0026\u0026 make -j$(nproc)\n\nFROM polluxio/pollux:latest AS my-pollux-payload\n#add eventual runtime libraries in below command\nRUN apt-get update \u0026\u0026 apt-get -y install libgrpc++ \n\nWORKDIR /root\nCOPY --from=builder /my-pollux-payload/src/c++/my-payload\n#For Pollux Communication\nEXPOSE 50000\n```\n\nOnce the Dockerfile written, run the folowing commands to create your docker image:\n\n```bash\ndocker build --target my-pollux-payload --file Dockerfile . -t my-pollux-payload\ndocker tag my-pollux-payload mydockerrepository/my-pollux-payload\ndocker push mydockerrepository/my-pollux-payload\n```\n\nYou should then be able to run your own payload by referencing it with the\n'-p' switch of the **Pollux** launcher.\n\n### Runtime configuration\nThe runtime behavior can be controlled using a pollux.yaml file.\n\n#### Global options\n##### Verbose mode\nVerbose mode can be set to info (default), debug or trace.\n```yaml\nverbose: trace # will set verbose to trace (can be info, debug or trace).\n```\n\n#### Example 1: Launching Multiple Identical Payloads\nThe following example launches five payloads, all executing the same command with the same options:\n```yaml\nsynchronized: true    # Enable synchronized mode\nexecutor: local       # Run payloads in local mode\n\n# Default command for all payloads\nglobal_payload_command: build/src/c++/examples/test/pollux-payload-test\n\npayloads_nb: 5        # Number of payloads to launch\n\n# Options applied to all payloads\nglobal_user_options:\n  nb_iterations:\n    type: int\n    value: 2\n```\n\n#### Example 2: Launching Payloads with Different Options\n\nThis example runs two payloads, both using the same command but with different user options:\n```yaml\nsynchronized: true\nexecutor: local\n\npayloads:\n  1:  \n    command: build/src/c++/examples/test/pollux-payload-test\n    user_options:\n      user0:\n        type: int\n        value: 2\n    \n  2:  \n    command: build/src/c++/examples/test/pollux-payload-test\n    user_options:\n      user0:\n        type: int\n        value: 3\n```\n\n\n\u003cdiv align=\"right\"\u003e[ \u003ca href=\"#pollux\"\u003e↑ Back to top ↑\u003c/a\u003e ]\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolluxio%2Fpollux-payload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpolluxio%2Fpollux-payload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolluxio%2Fpollux-payload/lists"}