{"id":33923033,"url":"https://github.com/rvhonorato/job-orchestrator","last_synced_at":"2026-07-11T19:31:29.928Z","repository":{"id":273886695,"uuid":"887186776","full_name":"rvhonorato/job-orchestrator","owner":"rvhonorato","description":" Asynchronous job orchestrator with quota-based load balancing for distributing computational workloads across heterogeneous computing resources","archived":false,"fork":false,"pushed_at":"2025-12-10T19:36:42.000Z","size":143,"stargazers_count":2,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-11T01:55:29.697Z","etag":null,"topics":["async","distributed-computing","hpc","job-orchestrator","job-scheduler","load-balancing","quota-management","rest-api","rust","scientific-computing","tokio","workflow-engine"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rvhonorato.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-12T10:12:11.000Z","updated_at":"2025-12-10T19:06:16.000Z","dependencies_parsed_at":"2025-07-30T14:29:09.190Z","dependency_job_id":"e9b92b6e-e050-45a6-a9a7-5c5e871b0b8a","html_url":"https://github.com/rvhonorato/job-orchestrator","commit_stats":null,"previous_names":["rvhonorato/orchestrator","rvhonorato/job-orchestrator"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/rvhonorato/job-orchestrator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjob-orchestrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjob-orchestrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjob-orchestrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjob-orchestrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rvhonorato","download_url":"https://codeload.github.com/rvhonorato/job-orchestrator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjob-orchestrator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27680569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-12T02:00:06.775Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","distributed-computing","hpc","job-orchestrator","job-scheduler","load-balancing","quota-management","rest-api","rust","scientific-computing","tokio","workflow-engine"],"created_at":"2025-12-12T09:58:30.567Z","updated_at":"2025-12-12T09:58:31.982Z","avatar_url":"https://github.com/rvhonorato.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# job-orchestrator\n\n![GitHub License](https://img.shields.io/github/license/rvhonorato/job-orchestrator)\n![GitHub Release](https://img.shields.io/github/v/release/rvhonorato/job-orchestrator)\n[![ci](https://github.com/rvhonorato/job-orchestrator/actions/workflows/ci.yml/badge.svg)](https://github.com/rvhonorato/job-orchestrator/actions/workflows/ci.yml)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/7f2a8816886645d28cbaac0fead038f9)](https://app.codacy.com/gh/rvhonorato/job-orchestrator/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_grade)\n[![Crates.io](https://img.shields.io/crates/v/job-orchestrator)](https://crates.io/crates/job-orchestrator)\n\n\u003e An asynchronous job orchestration system for managing and distributing computational workloads across heterogeneous computing resources with intelligent quota-based load balancing.\n\n## Overview\n\njob-orchestrator is a central component of [WeNMR](https://wenmr.science.uu.nl), a worldwide e-Infrastructure for structural biology operated by the [BonvinLab](https://bonvinlab.org) at [Utrecht University](https://uu.nl). It serves as a reactive middleware layer that connects web applications to diverse computing resources, enabling efficient job distribution for scientific computing workflows.\n\n### Key Features\n\n- **Asynchronous Job Management**: Built with Rust and Tokio for high-performance async operations\n- **Quota-Based Load Balancing**: Per-user, per-service quotas prevent resource exhaustion\n- **Dual-Mode Architecture**: Runs as server (job orchestration) or client (job execution)\n- **Multiple Backend Support**: Extensible to integrate with various computing resources:\n  - Native client mode for local job execution\n  - [DIRAC Interware](https://dirac.readthedocs.io/en/latest/index.html) _(planned)_\n  - SLURM clusters _(planned)_\n  - Educational cloud services _(planned)_\n- **RESTful API**: Simple HTTP interface for job submission and retrieval\n- **Automatic Cleanup**: Configurable retention policies for completed jobs\n\n### Architecture\n\n```mermaid\nflowchart LR\n    B([User]) --\u003e C[Web Application]\n    C[Web Application] \u003c--\u003e Y[(Database)]\n    C --\u003e X{{Orchestrator Server\u003cbr/\u003ePort 5000}}\n    X --\u003e|REST API| D[Orchestrator Client\u003cbr/\u003eprodigy:9000]\n    X --\u003e|REST API| E[Orchestrator Client\u003cbr/\u003edisvis:9000]\n    X --\u003e|REST API| G[Orchestrator Client\u003cbr/\u003eother services:9000]\n    E -.-\u003e|optional| H[HPC/SLURM]\n```\n\n**Components:**\n\n- **Orchestrator Server** (port 5000): Receives job submissions, tracks state, manages quotas, and distributes work\n- **Orchestrator Client** (port 9000): Receives jobs from server, executes them locally, and returns results\n- **Database**: SQLite for job tracking and state management\n- **Filesystem**: Local storage for job files and results\n\nBoth server and client modes are provided by the same binary, configured via command-line arguments.\n\n## Quick Start\n\n### Running with Docker Compose\n\n```bash\ndocker compose up --build\n```\n\nThis starts both the orchestrator server (port 5000) and an example client (port 9000).\n\n### Submitting a Job\n\n```bash\ncurl -X POST http://localhost:5000/upload \\\n  -F \"file=@example/run.sh\" \\\n  -F \"file=@example/2oob.pdb\" \\\n  -F \"user_id=1\" \\\n  -F \"service=example\" | jq\n```\n\nResponse:\n\n```json\n{\n  \"id\": 1,\n  \"user_id\": 1,\n  \"service\": \"example\",\n  \"status\": \"Queued\",\n  \"loc\": \"/opt/data/978e5a14-dc94-46ab-9507-fe0a94d688b8\",\n  \"dest_id\": \"\"\n}\n```\n\n### Checking Job Status\n\nUse HTTP HEAD to check status without downloading:\n\n```bash\ncurl -I http://localhost:5000/download/1\n```\n\n**Status Codes:**\n\n- `200` - Job completed, ready to download\n- `202` - Job queued or running\n- `204` - Job failed or cleaned up\n- `404` - Job not found\n- `500` - Internal server error\n\n### Downloading Results\n\nOnce the job completes (status `200`):\n\n```bash\ncurl -o results.zip http://localhost:5000/download/1\n```\n\n## How It Works\n\n### Job Lifecycle\n\n1. **Submission**: User uploads files via `/upload` endpoint with `user_id` and `service` parameters\n2. **Queuing**: Job enters queue; orchestrator checks user quotas before dispatching\n3. **Distribution**: Server sends job to available client matching the service type\n4. **Execution**: Client executes the job and signals completion\n5. **Retrieval**: Server fetches results from client and stores locally\n6. **Download**: User downloads results as a ZIP archive\n7. **Cleanup**: Completed jobs are automatically removed after configured retention period (default: 48 hours)\n\n### Quota System\n\nThe orchestrator enforces per-user, per-service quotas to ensure fair resource allocation. Configuration example:\n\n```bash\nSERVICE_EXAMPLE_RUNS_PER_USER=5  # Max 5 concurrent jobs per user for \"example\" service\n```\n\nThis prevents any single user from monopolizing computing resources.\n\n### Testing the Queue\n\nSubmit multiple jobs to observe quota-based throttling:\n\n```bash\nfor i in {1..250}; do\n  cat \u003c\u003cEOF \u003e run.sh\n#!/bin/bash\nsleep \\$((RANDOM % 36 + 25))\necho 'Computation complete!' \u003e output.txt\nEOF\n  curl -s -X POST http://localhost:5000/upload \\\n    -F \"file=@run.sh\" \\\n    -F \"user_id=1\" \\\n    -F \"service=example\" \u003e /dev/null\n  echo \"Submitted job $i\"\ndone\n```\n\nMonitor the orchestration in real-time:\n\n```bash\ndocker compose logs server --follow\n```\n\nYou'll see jobs dispatched gradually according to the configured quota limits.\n\n## Use Cases\n\nOrchestrator is designed for scenarios requiring:\n\n- **Scientific Computing Workflows**: Distribute computational biology/chemistry jobs across clusters\n- **Multi-Tenant Systems**: Fair resource allocation with per-user quotas\n- **Heterogeneous Computing**: Route jobs to appropriate backends (local, HPC, cloud)\n- **Web-Based Science Platforms**: Decouple frontend from compute infrastructure\n- **Batch Processing**: Handle high-throughput job submissions with automatic queuing\n\n## Project Status\n\n**Current State**: Production-ready with server/client architecture\n\n**Planned Features**:\n\n- DIRAC Interware integration\n- SLURM direct integration\n- Enhanced monitoring and metrics\n- Job priority queues\n- Advanced scheduling policies\n- Client auto-discovery and registration\n\n## Documentation\n\n- **API Documentation**: Available via Swagger UI at `http://localhost:5000/swagger-ui/` when running\n- **Technical Documentation**: Coming soon\n- **Configuration Guide**: Coming soon\n\n## Contributing\n\nContributions, bug reports, and feature requests are welcome via GitHub issues.\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Contact\n\nFor questions, collaborations, or if you think this project could benefit your use case:\n\n- **Issues**: [GitHub Issues](https://github.com/rvhonorato/job-orchestrator/issues)\n- **Email**: Rodrigo V. Honorato \u003crvhonorato@protonmail.com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvhonorato%2Fjob-orchestrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frvhonorato%2Fjob-orchestrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvhonorato%2Fjob-orchestrator/lists"}