{"id":24625295,"url":"https://github.com/link89/job-queue-lambda","last_synced_at":"2025-03-19T08:45:36.221Z","repository":{"id":273837206,"uuid":"921027715","full_name":"link89/job-queue-lambda","owner":"link89","description":"Use job queue (Slurm, PBS, etc) as a remote function executor.","archived":false,"fork":false,"pushed_at":"2025-03-05T08:03:04.000Z","size":114,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-05T09:20:41.565Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/link89.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-23T07:46:46.000Z","updated_at":"2025-03-05T08:03:07.000Z","dependencies_parsed_at":"2025-03-05T09:19:59.492Z","dependency_job_id":"30fa6318-68a0-466b-bc28-b7786a2b91f3","html_url":"https://github.com/link89/job-queue-lambda","commit_stats":null,"previous_names":["link89/job-queue-lambda"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Fjob-queue-lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Fjob-queue-lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Fjob-queue-lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/link89%2Fjob-queue-lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/link89","download_url":"https://codeload.github.com/link89/job-queue-lambda/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244394443,"owners_count":20445631,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-25T04:13:50.620Z","updated_at":"2025-03-19T08:45:36.216Z","avatar_url":"https://github.com/link89.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# job-queue-lambda\n\n\n[![PyPI version](https://badge.fury.io/py/job-queue-lambda.svg)](https://badge.fury.io/py/job-queue-lambda)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/job-queue-lambda)](https://pypi.org/project/job-queue-lambda/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/job-queue-lambda)](https://pypi.org/project/job-queue-lambda/)\n\nUse job queue (Slurm, PBS, etc) as a remote function executor, just like AWS Lambda.\n\n## Installation\n\n```bash\npip install job-queue-lambda\n```\n## Getting Started\n\n`job-queue-lambda` allows you to forward a HTTP request to a service that running on a remote job queue. Currently only Slurm is supported.\n\nFor example, you can use the following configuration:\n\n```yaml\n# ./examples/config.yaml\nclusters:\n  - name: ikkem-hpc\n    # if running on login node, then ssh section is not needed\n    ssh:\n      host: ikkem-hpc\n      # it use ssh dynamic port forwarding to connect to the cluster, so socks_port is required\n      socks_port: 10801\n\n    lambdas:\n      - name: python-http\n        forward_to: http://{NODE_NAME}:8080/\n        cwd: ./jq-lambda-demo\n        script: |\n          #!/bin/bash\n          #SBATCH -N 1\n          #SBATCH --job-name=python-http\n          #SBATCH --partition=cpu\n          set -e\n          timeout 30 python3 -m http.server 8080\n\n    job_queue:\n      slurm: {}\n\n```\n\nAnd then you can start the server by running:\n```bash\njq-lambda ./examples/config.yaml\n```\n\nNow you can use browser to access the following URL:  http://localhost:9000/clusters/ikkem-hpc/lambdas/python-http\n\nor using `curl`:\n```bash\ncurl http://localhost:9000/clusters/ikkem-hpc/lambdas/python-http\n```\n\nThe request will be forwarded to the remote job queue, and the response will be returned to you.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flink89%2Fjob-queue-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flink89%2Fjob-queue-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flink89%2Fjob-queue-lambda/lists"}