{"id":23635170,"url":"https://github.com/willdady/foreman","last_synced_at":"2025-04-10T00:36:27.945Z","repository":{"id":269499384,"uuid":"907125022","full_name":"willdady/foreman","owner":"willdady","description":"Foreman is a Rust based job scheduler and executor agent","archived":false,"fork":false,"pushed_at":"2025-02-28T01:35:56.000Z","size":107,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T02:22:08.774Z","etag":null,"topics":["agent","docker","job-scheduler","runner","rust","worker"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/willdady.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-22T21:54:56.000Z","updated_at":"2025-02-28T01:33:47.000Z","dependencies_parsed_at":"2024-12-24T02:40:15.175Z","dependency_job_id":"e5946087-832d-4c1c-96a3-f65b97796cbc","html_url":"https://github.com/willdady/foreman","commit_stats":null,"previous_names":["willdady/foreman"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willdady%2Fforeman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willdady%2Fforeman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willdady%2Fforeman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willdady%2Fforeman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/willdady","download_url":"https://codeload.github.com/willdady/foreman/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137996,"owners_count":21053773,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","docker","job-scheduler","runner","rust","worker"],"created_at":"2024-12-28T05:31:39.882Z","updated_at":"2025-04-10T00:36:27.930Z","avatar_url":"https://github.com/willdady.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# foreman\n\nForeman is a Rust based job scheduler and executor agent.\n\n## Features\n\n- **💬 Language agnostic**: Jobs are processed in containerised environments.\n- **🔐 Secure by default**: Self-hostable behind a NAT gateway, without the need to be exposed publically over the internet.\n- **🚀 Fast, efficient and lightweight**: Compiles to a single binary executable\n\n## Installation\n\nInstall the Rust toolchain via rustup.\n\n```bash\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n```\n\nCompile release build.\n\n```bash\ncargo build --release\n```\n\n## Usage\n\nThe foreman binary expects to find a configuration file named `foreman.toml` in one of the following locations:\n\n- At a path specified by the `FOREMAN_CONFIG` environment variable. e.g. `FOREMAN_CONFIG=/path/to/foreman.toml`\n- The current working directory\n- `/etc/foreman/foreman.toml`\n- `$HOME/.foreman/foreman.toml`\n\nRefer to [example.foreman.toml](example.foreman.toml) for an explanation of the various configuration options and their defaults.\n\nAlternatively, config values can be specified via environment variables.\nEach environment variable should be prefixed with `FOREMAN_`.\n\nFor example ...\n\n```toml\n[core]\nport = 8080\n```\n\n... can be specified via the environment variable `FOREMAN_CORE_PORT=8080`.\n\nValues set via environment variables will override any values specified in `foreman.toml`.\n\n### Labels\n\nLabels are optional key/value pairs which you can define in the `[core.labels]` section of `foreman.toml`.\nThese labels will be sent to your control server in the header `x-foreman-labels` when polling for jobs.\n\nYour control server MAY discriminate requests based on these labels and only deliver matching jobs.\nFor example, you may choose to identify features of the forman agent host by defining labels such as `cpu = 4`, `gpu = true` etc.\n\nSimilarly, your control server may maintain multiple job queues.\nLabels allow you to assign agents to specific queues e.g. `queue = high-priority`, `queue = low-priority` etc.\n\nLabels are formatted as a comma-separated list of key/value pairs e.g `cpu=4,gpu=true,queue=high-priority`.\nBoth keys and values will be URL-encoded so you are free to use '=' and ',' in your labels.\nThe trade-off here is you need to remember to URL-decode before usage on your control server.\nThe order of the key/values is NOT guaranteed.\n\n## Concepts\n\n### Foreman\n\nForeman (this project) is a Rust-based job scheduling agent which retrieves jobs from a control server and executes them inside a containerised environment.\nIt is designed to be run in private subnets behind a NAT gateway, without the need to be exposed to the internet.\n\nForeman is similar in spirit to a CI/CD agent but more generic.\n\n### Control Server\n\nAt a high level, a control server is a responsible for the following:\n\n- Responds to requests from foreman agents returning zero-or-more jobs in response to a request\n- Retrieves job execution statuses from foreman agents\n\nThe implementation of a control server is not within the scope of this project, though a reference implementation is included for development purposes.\nSee the Development section below for more information.\n\n### Job\n\nA job defines a single task that needs to be executed.\nIt can be anything from running a script to deploying an application.\n\n### Executor\n\nAn executor is responsible for executing jobs on behalf of a foreman agent.\n\nForeman manages Docker as it's job executor.\nA custom bridge network is created on start-up which all containers created by foreman are added to.\nBy default the network is named `foreman`.\nThis can be changed via the `core.network_name` configuration option.\n\n## Sequence diagram\n\nThe following sequence diagram illustrates the flow of a job execution request between foreman, a control server and an executor.\n\n```mermaid\nsequenceDiagram\n    participant CS as Control Server\n    participant F as Foreman\n    participant E as Executor (Docker)\n    F-\u003e\u003eCS: GET /job\n    CS--\u003e\u003eF: JSON\n    F-\u003e\u003eE: Start container\n    E-\u003e\u003eF: GET /job\n    F--\u003e\u003eE: JSON\n    E-\u003e\u003eE: Execute job\n    E-\u003e\u003eF: PUT /job/\u003cjob-id\u003e\n    F-\u003e\u003eCS: PUT /job/\u003cjob-id\u003e\n    CS--\u003e\u003eF: OK\n    F-\u003e\u003eE: Stop container\n    F-\u003e\u003eF: Wait\n    F-\u003e\u003eE: Remove container\n```\n\n## Job schema\n\nA job returned by a control server is expected to conform to the following schema (denoted here as a Typescript interface):\n\n```typescript\ninterface Job {\n    /**\n     * Unique identifier for the job\n     */\n    id: string;\n\n    /**\n     * Docker image to use for the job\n     */\n    image: string;\n\n    /**\n     * Command to run in the container\n     */\n    command?: string[];\n\n    /**\n     * Body of the job, which can be any type\n     */\n    body: any;\n\n    /**\n     * Environment variables for the job\n     */\n    env?: { [key: string]: string };\n\n    /**\n     * Callback URL for the job\n     */\n    callbackUrl: string;\n\n    /**\n     * Whether to always pull the Docker image before creating a container\n     */\n    alwaysPull?: boolean;\n}\n```\n\nSome things to note:\n\n- The `id` is used to uniquely identify each job and should be unique within your control server.\n  Using a UUID is recommended.\n- The `callbackUrl` does not need to be the same server as your control server (though you will likely still need to signal back to your control server when the job completes).\n- Avoid setting `alwaysPull: true` as it will slow down the creation of job containers. \n  You should only need this if your image tags are **mutable** which is generally considered bad practice.\n- The job schema is also available in JSON schema format in [job.schema.json](job.schema.json).\n\n## Authoring a job processor image\n\nForeman will create a container based on the `image` field defined in a job, pulling the image if necessary.\n\nThe foreman agent exposes a simple REST API which job containers are expected to communicate with when dealing with their associated job.\n\nWhen a container is ready it MUST perform a GET request to the URL contained in the `FOREMAN_GET_JOB_ENDPOINT` environment variable.\nThis endpoint returns a JSON object containing the job `id` and `body` fields from the original job received from the control server.\n\nLikewise the container MUST perform a PUT request to the URL contained in the `FOREMAN_PUT_JOB_ENDPOINT` environment variable with updates to the job's status.\nWhen sending requests to this endpoint the only requirement is the following headers must be set in the request.\n\n| name                   | required | description                                                                                 |\n| ---------------------- | -------- | ------------------------------------------------------------------------------------------- |\n| x-foreman-job-status   | YES      | MUST be either 'running' or 'completed'                                                     |\n| x-foreman-job-progress | NO       | A floating point number representing the progress of the job. Defaults to 0.0 if undefined. |\n\nRequests sent to this endpoint are forwarded to the job's `callbackUrl` as-is.\nThe `completed` status is a terminal state and can be set at-most once per job.\nIt is invalid to send a PUT request with `x-foreman-job-status` set to `running` on a completed job.\n\nA container becomes eligible for removal once it's status changes to `completed`.\n\n## Development\n\n### 1. Build the Test image\n\nBuild the test job image so it is available on your local machine.\nThis image is used by jobs produced by the reference control server (see next step).\n\n```\ncd examples/test_job_image\ndocker build -t foreman-test-job-image:latest .\n```\n\n### 2. Run the reference control server\n\nA reference control server, using Typescript and Deno, is defined in `examples/control_server`.\n\nTo run the server, `cd` into the `examples/control_server` directory and run:\n\n```bash\ndeno run -A index.ts\n```\n\nThe server runs on port `8888` and listens for `GET` requests on the `/job` endpoint.\n\n### 3. Configure foreman\n\nUpdate your `foreman.toml` file to contain the following configuration.\nThis allows code running inside the test image to reach the foreman process running on your host machine.\n\n```toml\n[core]\nurl = 'http://localhost:8888/job'\ntoken = 'MY-SUPER-SECRET-TOKEN'\nhostname = \"host.docker.internal\"\nextra_hosts = [\"host.docker.internal:host-gateway\"]\n```\n\n### 4. Run foreman\n\nIn a separate terminal, start foreman.\n\n```bash\ncargo run\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilldady%2Fforeman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwilldady%2Fforeman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilldady%2Fforeman/lists"}