{"id":21357014,"url":"https://github.com/gjbex/worker-ng","last_synced_at":"2025-08-26T12:19:50.481Z","repository":{"id":186110827,"uuid":"128186887","full_name":"gjbex/worker-ng","owner":"gjbex","description":"Next-generation worker implementation","archived":false,"fork":false,"pushed_at":"2025-04-01T13:41:37.000Z","size":3477,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-01T14:37:51.417Z","etag":null,"topics":["hpc","parallel-computing","workflow"],"latest_commit_sha":null,"homepage":"https://gjbex.github.io/worker-ng/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gjbex.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-05T09:46:02.000Z","updated_at":"2025-02-13T10:33:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"3a0706d4-32ec-4e36-9d39-32bad7cc1cc8","html_url":"https://github.com/gjbex/worker-ng","commit_stats":null,"previous_names":["gjbex/worker-ng"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/gjbex/worker-ng","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbex%2Fworker-ng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbex%2Fworker-ng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbex%2Fworker-ng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbex%2Fworker-ng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gjbex","download_url":"https://codeload.github.com/gjbex/worker-ng/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbex%2Fworker-ng/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272219692,"owners_count":24894474,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","parallel-computing","workflow"],"created_at":"2024-11-22T04:42:50.736Z","updated_at":"2025-08-26T12:19:50.454Z","avatar_url":"https://github.com/gjbex.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# worker-ng\n\nNext-generation worker implementation of worker framework.\n\nWhereas the original worker framework was based on the\nMessage Passing Interface (MPI) this version uses ZeroMQ\nfor communication.  This has several advantages:\n    * better robustness: if a client crashes, that has no\n      impact on eiter the server or other clients;\n    * MPI work items are now trivial to handle;\n    * server and clients are decoupled, allowing for more\n      flexible scenarios.\n\nThis version of worker is also easier to adapt to various\nschedulers.  It has currently support for both slurm and PBS\ntorque, but can be extended to other schedulers if necessary.\n\nAlthough worker-ng shares many features with the original\nworker framework, it is not entirely a drop-in replacement.\nThe user-facing scripts of the original framework where\nwritten in Perl, whereas this version is based on Python.\nThis has some implications on how command line options are\nhandled.\n\n\n## What is it?\n\nFor users:\n\n1. `examples`: a directory containing several examples of how to\n   use the worker framework.\n1. `docs_user`: user documentation on how to use the worker\n   framework.\n1. `LICENSE`: the license file for users and developers.\n\nFor developers:\n\n1. `conf`: worker configuration files, required at runtime.\n1. `docs_dev`: developer documentation.\n1. `scripts`: Python scripts to submit worker jobs, monitor\n   progress, analyze performance and resume jobs.  The `worker`\n   subdirectory contains the Python library for the framework,\n   as well as the templates for the job scripts.\n1. `src`: source code for the C++ server and client and the\n   supporting library.\n1. `test`: tests for the Python library.\n1. `CMakeLists.txt`: top-level CMake build script.\n1. `environment.yml`: conda environment file to build and run\n   the worker framework.\n1  `development_worker.recipe`: Singularity/Apptainer recipe to\n   build an image that contains all tools and libraries required\n   for development.\n1. Various convenience scripts\n    * `run_test.sh`: script to run the tests.\n    * `build_genius.sh`: script to build worker on the KU Leuven genius\n      cluster.\n    * `setup_genius.sh`: script to be sourced to set up your environment\n      to run worker on the genius cluster.\n    * `build_wice.sh`: script to build worker on the KU Leuven wice\n      cluster.\n    * `setup_wice.sh`: script to be sourced to set up your environment\n      to run worker on the wice cluster.\n\n\n## Installation\n\nInstallion is done using CMake:\n\n```bash\n$ cmake -S . -B build/ -DSCHEDULER_NAME=slurm -DCMAKE_INSTALL_PREFIX=\u003cwhere-you-want\u003e\n$ cmake --build build/\n$ cmake --install build/\n```\n\nRequirements:\n1. Boost 1.76.0\n1. cppzmq 4.8.1\n1. CMake \u003e= 3.20\n1. doxygen \u003e= 1.9\n1. Python \u003e= 3.9\n\n\n## Contributors\n\n1. Geert Jan Bex ([geertjan.bex@uhasselt.be](mailto:geertjan.bex@uhasselt.be)):\n   main developer.\n1. Mag Selwa: testing and feedback.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgjbex%2Fworker-ng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgjbex%2Fworker-ng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgjbex%2Fworker-ng/lists"}