{"id":21423123,"url":"https://github.com/rvhonorato/jobd","last_synced_at":"2026-05-17T02:33:30.430Z","repository":{"id":264215583,"uuid":"892671389","full_name":"rvhonorato/jobd","owner":"rvhonorato","description":"A lightweight Golang REST API daemon for seamless file transfer and job management in containerized research software environments.","archived":false,"fork":false,"pushed_at":"2025-10-13T09:41:53.000Z","size":126,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-25T05:43:48.742Z","etag":null,"topics":["containerization","docker","golang","microservices","research-software","rest-api"],"latest_commit_sha":null,"homepage":"https://rvhonorato.github.io/jobd/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rvhonorato.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-22T14:55:00.000Z","updated_at":"2025-10-13T09:41:50.000Z","dependencies_parsed_at":"2025-01-20T12:20:45.088Z","dependency_job_id":"9400f06d-fb37-4e4f-bdcf-717e30c23877","html_url":"https://github.com/rvhonorato/jobd","commit_stats":null,"previous_names":["rvhonorato/jobd"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rvhonorato/jobd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjobd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjobd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjobd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjobd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rvhonorato","download_url":"https://codeload.github.com/rvhonorato/jobd/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvhonorato%2Fjobd/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33125301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T18:38:32.183Z","status":"online","status_checked_at":"2026-05-17T02:00:05.366Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containerization","docker","golang","microservices","research-software","rest-api"],"created_at":"2024-11-22T21:14:13.350Z","updated_at":"2026-05-17T02:33:30.412Z","avatar_url":"https://github.com/rvhonorato.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `jobd` (Job Daemon)\n\n![GitHub License](https://img.shields.io/github/license/rvhonorato/jobd)\n[![ci](https://github.com/rvhonorato/jobd/actions/workflows/ci.yml/badge.svg)](https://github.com/rvhonorato/jobd/actions/workflows/ci.yml)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/ae50eb8e1303415f981ec755f0b8a28f)](https://app.codacy.com/gh/rvhonorato/jobd/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_grade)\n[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/ae50eb8e1303415f981ec755f0b8a28f)](https://app.codacy.com/gh/rvhonorato/jobd/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_coverage)\n\nThis is a central component [WeNMR](https://wenmr.science.uu.nl), a worldwide\ne-Infrastructure for NMR and structural biology - operated by\nthe [BonvinLab](https://bonvinlab.org) at the [Utrecht University](https://uu.nl).\n\nIt enables interaction between the web backend and the\n[research software developed in the Bonvinlab](https://github.com/haddocking) which\nare offered as web services for a community of over\n[52.000 users accross 154 countries](https://rascar.science.uu.nl/new/stats).\n\n`jobd` is a lightweight Golang application designed to facilitate interaction with\nresearch software through REST APIs. It is specifically engineered to be deployed\nin multi-stage Docker builds, providing a flexible and portable solution for job\nmanagement and file transfer.\n\n```mermaid\nflowchart LR\n    B([User]) --\u003e C[Web App]\n    C[Web App] \u003c--\u003e Y[(Database)]\n    C[Web App] --\u003e X{{Orchestrator}}\n    X --\u003e|jobd| D[[prodigy]]\n    X --\u003e|jobd| E[[disvis]]\n    X --\u003e|jobd| G[[other_service]]\n    E --\u003e|slurml| H[local HPC]\n```\n\n🚧 Documentation is still a work in progess 🚧\n\n## Features\n\nImplements two primary REST API endpoints:\n\n- `POST /api/upload` Allows backend systems or scripts to upload files to the container\n- `GET /api/get/:id` Enables retrieval of files (results) from the container\n\nCheck the [API docs](https://rvhonorato.github.io/jobd/) for more information\n\nUse Cases\n\n- Microservice-based job submission and file handling\n- Simplified API interfaces for research software workflows\n\n## Usage\n\nThe purpose of `jobd` is to be an adapter to allow backend/scripts to interact with\n_any_ sort of command-line based application.\n\nIt takes a base64 encoded `.zip` file that **must** contain a `run.sh` script.\n\nOnce a `POST` request is made to `/api/upload`, `jobd` creates an entry in its\nmicro(embedded) database and saves the `input` to disk (inside the container).\n\nIt will check every second for tasks that are `QUEUED` in the database, then\nmake a system call to the `run.sh` script and report its exit code.\n\nAll the resulting contents are then compressed (also base64 `.zip`) and\nreturned as the `output`.\n\n### Example\n\n\u003e ⚠️ `jobd` was not designed for this type of interaction, but rather via automated\n\u003e scripts. The workflow below is just to demonstrate how it works.\n\n[PRODIGY](https://github.com/haddocking/prodigy) is a command-line application\nto calculate binding energies between proteins, developed by the BonvinLab.\nIt takes as input a single protein structure and is executed simply with:\n\n```bash\nprodigy input.pdb\n```\n\nThis application can be installed from `pip`, see below how we can containerize\n`prodigy` and add `jobd` as the interface.\n\n```dockerfile\nFROM ubuntu:latest AS build\n\nRUN pip install prodigy\n\n# Here you could also clone a repository and install it;\n## WORKDIR /src/app\n## RUN git clone https://github.com/username/code . \u0026\u0026 \\\n##  bash install.sh ## or pip install, or anything like that\n\n# Get `jobd` from the github image repository\nFROM ghcr.io/rvhonorato/jobd:latest AS jobd\n\nWORKDIR /data\nCOPY --from=jobd /jobd /bin/jobd\n\nENTRYPOINT [ \"/bin/jobd\" ]\n```\n\nAfter building and running this image, you can then interact with the container\nvia its [APIs](https://rvhonorato.me/jobd).\n\nThen prepare the payload, it must contain the `run.sh` file and the input files.\n\n- `run.sh` file:\n\n  ```bash\n  #!/bin/bash\n\n  prodigy input.pdb\n\n  # any other steps you want to add here\n  ```\n\nPrepare the payload;\n\n```bash\n# Make a zip file called `files.zip`\n$ zip files.zip run.sh input.pdb\n```\n\nThen prepare the input `.json` file - it is expected to be in the following format:\n\n```json\n{\n  \"id\": \"name-of-my-job\",\n  \"input\": \"BASE64_STRING_HERE\"\n}\n```\n\nThis can easily be done with more scripting (or using any other method)\n\n```bash\njq -n --arg id \"name-of-my-job\" --arg input \"$(base64 files.zip)\" \\\n   '{id: $id, input: $input}' \u003e job.json\n```\n\nFinally submit the job to the container:\n\n```bash\ncurl -X POST http://your.server:8080/api/upload \\\n     -H \"Content-Type: application/json\" \\\n     -d @job.json\n```\n\nAnd later download the results by making a `GET` request to `/api/get/name-of-my-job`\n\n## Setup\n\nThe application is optimized for containerized environments,\nsupporting multi-stage build patterns or simple binary execution.\n\nIn both ways the `jobd` version can be passed as a build argument:\n\n```bash\ndocker build --build-arg JOBD_VERSION=v0.1.0 .\n```\n\n### Multi-stage build\n\nFirst stage pulls the jobd executable from a specific image\nSecond stage incorporates the executable into a base research container\nEnables seamless integration of job management capabilities into existing research\nsoftware containers\n\n```dockerfile\nARG JOBD_VERSION=latest\nFROM ghcr.io/rvhonorato/jobd:${JOBD_VERSION} AS jobd\n\nFROM ghcr.io/haddocking/arctic3d:v0.5.1 AS base\n\nWORKDIR /data\nCOPY --from=jobd /jobd /bin/jobd\n\nENTRYPOINT [ \"/bin/jobd\" ]\n```\n\n### Binary execution\n\n```dockerfile\nFROM ghcr.io/haddocking/arctic3d:v0.5.1\nWORKDIR /data\n\nARG JOBD_VERSION=v0.1.0\nARG JOBD_ARCH=linux_386\n\n# Download and extract jobd binary\nADD https://github.com/rvhonorato/jobd/releases/download/${JOBD_VERSION}/jobd_${JOBD_VERSION}_${JOBD_ARCH}.tar.gz /tmp/\nRUN tar -xzf /tmp/jobd_${JOBD_VERSION}_${JOBD_ARCH}.tar.gz -C /bin/ \\\n    \u0026\u0026 chmod +x /bin/jobd \\\n    \u0026\u0026 rm /tmp/jobd_${JOBD_VERSION}_${JOBD_ARCH}.tar.gz\n\nENTRYPOINT [ \"/bin/jobd\" ]\n```\n\n## Key points\n- Language: Golang\n- Type: Lightweight REST API-based job management microservice\n- Primary Purpose: Facilitate interaction between web backends and research software\n- Containerization-native design\n- Embedded microdatabase for job tracking\n- Polls database every second for QUEUED tasks\n- Executes run.sh script via system call\n- Captures and reports script exit code\n- Compresses and returns job results\n- Multi-stage Docker builds\n- Direct binary execution\n- Research computing platforms\n- Microservice architectures\n- Scientific software workflows\n- Minimal system overhead\n- Flexible integration with research software\n\n## Contact\n\nIf you think this project would be useful for your use case or would like to suggest something, please reach out either via issue here or via email. (:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvhonorato%2Fjobd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frvhonorato%2Fjobd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvhonorato%2Fjobd/lists"}