{"id":16223962,"url":"https://github.com/kohkimakimoto/hq","last_synced_at":"2025-07-07T03:04:24.606Z","repository":{"id":42537718,"uuid":"63310435","full_name":"kohkimakimoto/hq","owner":"kohkimakimoto","description":"HQ is a simplistic, language agnostic job queue engine communicated by HTTP messages.","archived":false,"fork":false,"pushed_at":"2023-03-06T23:34:20.000Z","size":5056,"stargazers_count":71,"open_issues_count":7,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-27T08:21:07.140Z","etag":null,"topics":["go","queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kohkimakimoto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-14T06:55:33.000Z","updated_at":"2025-01-25T07:23:26.000Z","dependencies_parsed_at":"2024-06-19T00:22:31.358Z","dependency_job_id":"da5b0184-9dbc-41ae-8b30-0245d3ee5ff0","html_url":"https://github.com/kohkimakimoto/hq","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kohkimakimoto%2Fhq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kohkimakimoto%2Fhq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kohkimakimoto%2Fhq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kohkimakimoto%2Fhq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kohkimakimoto","download_url":"https://codeload.github.com/kohkimakimoto/hq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243814880,"owners_count":20352037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","queue"],"created_at":"2024-10-10T12:21:06.587Z","updated_at":"2025-03-16T12:31:37.837Z","avatar_url":"https://github.com/kohkimakimoto.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# HQ \u003c!-- omit in toc --\u003e\n\nHQ is a simplistic, language agnostic job queue engine communicated by HTTP messages.\n\nHQ is implemented as a standalone JSON over HTTP API server. When you push a job to the HQ server, it stores the job in the internal queue database and sends asynchronous HTTP POST request to a URL that specified in the job.\n\nWorker applications that actually run the jobs are web applications. So you can implement applications for the jobs in Any programming language that can talk HTTP (such as Go, PHP, Python, etc).\n\n```\n                   ┌────────────────────────────────────────────────────────────────┐\n┌───┐              │HQ server                                               ┌──────┐│              ┌──────────┐\n│app│──POST /job──\u003e┼┐          ┌───────────────────┐                      ┌\u003e│worker│┼──POST /xxx──\u003e│worker app│\n└───┘              ││          │queue              │                      │ └──────┘│              └──────────┘\n┌───┐              ││          │┌───┐ ┌───┐   ┌───┐│          ┌──────────┐│ ┌──────┐│              ┌──────────┐\n│app│──POST /job──\u003e┼┼─enqueue-\u003e││job│ │job│...│job││-dequeue-\u003e│dispatcher│┼\u003e│worker│┼──POST /xxx──\u003e│worker app│\n└───┘              ││          │└───┘ └───┘   └───┘│          └──────────┘│ └──────┘│              └──────────┘\n┌───┐              ││          │                   │                      │ ┌──────┐│              ┌──────────┐\n│app│──POST /job──\u003e┼┘          └───────────────────┘                      └\u003e│worker│┼──POST /xxx──\u003e│worker app│\n└───┘              │                                                        └──────┘│              └──────────┘\n                   └────────────────────────────────────────────────────────────────┘\n```\n\n## Table of Contents \u003c!-- omit in toc --\u003e\n\n  - [Installation](#installation)\n  - [Getting Started](#getting-started)\n  - [Configuration](#configuration)\n    - [Example](#example)\n    - [Parameters](#parameters)\n  - [Job](#job)\n  - [HTTP API](#http-api)\n    - [`GET /`](#get-)\n      - [Request](#request)\n      - [Response](#response)\n    - [`GET /stats`](#get-stats)\n      - [Request](#request-1)\n      - [Response](#response-1)\n    - [`POST /job`](#post-job)\n      - [Request](#request-2)\n      - [Response](#response-2)\n    - [`GET /job`](#get-job)\n      - [Request](#request-3)\n      - [Response](#response-3)\n    - [`GET /job/{id}`](#get-jobid)\n      - [Request](#request-4)\n      - [Response](#response-4)\n    - [`DELETE /job/{id}`](#delete-jobid)\n      - [Request](#request-5)\n      - [Response](#response-5)\n    - [`POST /job/{id}/restart`](#post-jobidrestart)\n      - [Request](#request-6)\n      - [Response](#response-6)\n    - [`POST /job/{id}/stop`](#post-jobidstop)\n      - [Request](#request-7)\n      - [Response](#response-7)\n  - [Commands](#commands)\n  - [Web UI](#web-ui)\n  - [Author](#author)\n  - [License](#license)\n\n## Installation\n\nHQ lives in a single binary without external dependencies. You can download a precompiled binary at the Github releases page.\n\n[Download latest version](https://github.com/kohkimakimoto/hq/releases/latest)\n\nYou can also use RPM package that is stored in the same releases page. It is useful because it configures systemd service automatically.\n\n## Getting Started\n\nIt is very easy to get started on HQ. After installing HQ, run `hq serve` in your terminal.\n\n```\n$ hq serve\n2019-04-18T18:56:25+09:00 WARN Your 'data_dir' configuration is not set. HQ server uses a temporary directory that is deleted after the process terminates.\n2019-04-18T18:56:25+09:00 WARN Created temporary data directory: /var/folders/7q/7yfsnkpj09n6s1pvlktkpp6h0000gn/T/hq_data_718224193\n2019-04-18T18:56:25+09:00 INFO Opened data directory: /var/folders/7q/7yfsnkpj09n6s1pvlktkpp6h0000gn/T/hq_data_718224193\n2019-04-18T18:56:25+09:00 INFO Opened boltdb: /var/folders/7q/7yfsnkpj09n6s1pvlktkpp6h0000gn/T/hq_data_718224193/server.bolt\n2019-04-18T18:56:25+09:00 INFO The server Listening on 0.0.0.0:19900 (pid: 74090)\n```\n\n\u003e Note: Running HQ server without any configuration like the above can cause to lose queued jobs, because HQ uses temporary directory to store jobs. Therefore this should be used only on DEV environment. When you use HQ on your production environment, You should set a proper configuration file. See [Configuration](#configuration).\n\nNext, you must launch your worker application. It is a web application that should be implemented for your purpose. But this time, you can use an example web app I made. Download [`workerapp.py`](./examples/workerapp.py) script and add it executable permission and then run the script like the following:\n\n```\n$ ./workerapp.py\nServing at port 8000\n```\n\nThis is a web application that just outputs HTTP post request info to the console.\nYou are ready to push a job. You can push a job to HQ by using the following `curl` command:\n\n```\n$ curl -XPOST http://localhost:19900/job -H \"Content-Type: application/json\" -d '{\"url\": \"http://localhost:8000/\", \"payload\": {\"message\": \"Hello world!\"}}'\n```\n\n`workerapp.py` will get a job from HQ and output like the following message.\n\n```\n--POST REQUEST BEGIN--\nPOST /\nHost: localhost:8000\nUser-Agent: HQ/1.0.0\nContent-Length: 27\nContent-Type: application/json\nX-Hq-Job-Id: 121128807380811776\nAccept-Encoding: gzip\n\n{\"message\": \"Hello world!\"}\n--POST REQUEST END----\n```\n\n## Configuration\n\nThe config file must be written in [TOML](https://github.com/toml-lang/toml). You can specify the config file by `-c` or `-config-file` option when HQ runs like the following.\n\n```\n$ hq serve -c /path/to/config.toml\n```\n\n### Example\n\n```toml\nserver_id = 0\naddr = \"0.0.0.0:19900\"\ndata_dir = \"/var/lib/hq\"\nlog_level = \"info\"\nlog_file = \"/var/log/hq/hq.log\"\naccess_log_file = \"/var/log/hq/access.log\"\nqueues = 8192\ndispatchers = 1\nmax_workers = 0\nshutdown_timeout = 10\njob_lifetime = 2419200\njob_list_default_limit = 0\nui = true\nui_basename = \"/ui\"\n```\n\n### Parameters\n\n* `server_id` (number): This is used to generate Job ID. HQ uses [go-katsubushi](https://github.com/kayac/go-katsubushi) to allocate unique ID. If you want Job ID to be unique on mulitple servers, You need to set `server_id` unique on each servers. The default is `0`.\n\n* `addr` (string): The listen address to the HQ server process. The default is `0.0.0.0:19900`.\n\n* `data_dir` (string): The data directory to store all generated data by the HQ sever. You should set the parameter to keep jobs persistantly. If you doesn't set it, HQ uses a temporary directory that is deleted after the process terminates.\n\n* `log_level` (string): The log level (`debug|info|warn|error`). The default is `info`.\n\n* `log_file` (string): The log file path. If you do not set, HQ writes log to STDOUT.\n\n* `access_log_file` (string):The access log file path. If you do not set, HQ writes log to STDOUT.\n\n* `queues` (number): Size of queue. The default is `8192`.\n\n* `dispatchers` (number): Number of dispatchers. The default is `runtime.NumCPU()`.\n\n* `max_workers` (number): Number of max workers. The dispacher can execute multiple workers concurrently. This config is limit how many each dispatcher can run workers concurrently. For example, If you set `dispatcher = 2` and `max_workers = 3`, HQ can run max `6` workers at the same time. If you set `max_workers = 0`, each dispacher run only one worker synchronously. The default is `0`.\n\n* `shutdown_timeout` (number): This is time how many seconds HQ waits executing jobs to finish in a shutdown process. If HQ server process receives `SIGINT` or `SIGTERM`, it try to shutdown itself. If HQ has executing workers, it waits workers to finish or number of seconds of this config. The default is `10`.\n\n* `job_lifetime` (number): HQ removes old finished jobs automatically. This config sets time how many seconds HQ keeps jobs. If you set it `0`, HQ does not remove any jobs. The default is `2419200` (28 days).\n\n* `job_list_default_limit` (number): The default `limit` value of [`GET /job`](#get-job). The default is `0` (no limit).\n\n* `ui` (boolean): Enables built-in Web UI. The default is `true`.\n\n* `ui_basename` (string): The built-in Web UI URL. For example, if you set it `/foo`, The Web UI will be provided on the url like `http://localhost:19900/foo`. The default is `/ui`.\n\n## Job\n\nJob in HQ is a JSON object as the following:\n\n```json\n{\n  \"canceled\": false,\n  \"comment\": \"This is an example job!\",\n  \"createdAt\": \"2019-10-29T07:32:26.054Z\",\n  \"err\": \"\",\n  \"failure\": false,\n  \"finishedAt\": \"2019-10-29T07:32:28.548Z\",\n  \"headers\": null,\n  \"id\": \"109192606348480512\",\n  \"name\": \"example-job\",\n  \"output\": \"OK\",\n  \"payload\": {\n    \"message\": \"Hello world!\"\n  },\n  \"running\": false,\n  \"startedAt\": \"2019-10-29T07:32:28.252Z\",\n  \"status\": \"success\",\n  \"statusCode\": 200,\n  \"success\": true,\n  \"timeout\": 0,\n  \"url\": \"http://your-worker-app-server/example\",\n  \"waiting\": false\n}\n```\n\nJobs are managed internal database in HQ. To create and push a new job, You can use [`POST /job`](#post-job) API.\nThe pushed job is stored in the queue and executed by the HQ worker. The HQ worker constructs HTTP POST request from the job. You can customize this request headers and JSON payload by the job properties.\n\nIf the above example job is executed, HQ will send like the following HTTP request:\n\n```http\nPOST /example HTTP/1.1\nHost: your-worker-app-server\nUser-Agent: HQ/1.0.0\nContent-Type: application/json\nContent-Length: 26\nX-Hq-Job-Id: 109192606348480512\nAccept-Encoding: gzip\n\n{\"message\":\"Hello world!\"}\n```\n\n## HTTP API\n\nHQ core functions are provided via RESTful HTTP API.\n\nOverview of endpoints:\n\n - [`GET /`](#get-): Gets HQ info.\n - [`GET /stats`](#get-stats): Gets the HQ server statistics.\n - [`POST /job`](#post-job): Pushes a new job.\n - [`GET /job`](#get-job): Lists jobs.\n - [`GET /job/{id}`](#get-jobid): Gets a job.\n - [`DELETE /job/{id}`](#delete-jobid): Deletes a job.\n - [`POST /job/{id}/restart`](#post-jobidrestart): Restarts a job.\n - [`POST /job/{id}/stop`](#post-jobidstop): Stops a job.\n\nBy default, the output of all HTTP API requests is minimized JSON. If the client passes `pretty` on the query string, formatted JSON will be returned.\n\n### `GET /`\n\n#### Request\n\nGets HQ info.\n\n```http\nGET /\n```\n\n#### Response\n\n```json\n{\n  \"version\": \"1.0.0\",\n  \"commitHash\": \"e78e5977ffecffce7f5118e002069dd05165deb6\"\n}\n```\n\n### `GET /stats`\n\nGets the HQ server statistics.\n\n#### Request\n\n```http\nGET /stats\n```\n\n#### Response\n\n\n```json\n{\n  \"queues\": 8192,\n  \"dispatchers\": 8,\n  \"maxWorkers\": 0,\n  \"numWorkers\": 0,\n  \"numJobsInQueue\": 0,\n  \"numJobsWaiting\": 0,\n  \"numJobsRunning\": 0,\n  \"numStoredJobs\": 67,\n  \"numJobsInLastMinute\": 0\n}\n```\n\n### `POST /job`\n\nPushes a new job.\n\n#### Request\n\n```http\nPOST /job\n```\n\n```json\n{\n  \"url\": \"https://your-worker-app-server/example\",\n  \"name\": \"example\",\n  \"comment\": \"This is an example job!\",\n  \"payload\": {\n    \"message\": \"Hello world!\"\n  },\n  \"headers\": {\n    \"X-Custom-Token\": \"xxxxxxx\"\n  },\n  \"timeout\": 0\n}\n```\n\n##### Parameters \u003c!-- omit in toc --\u003e\n\n- `url` (string,required): The URL to send HTTP request to a worker application.\n- `name` (string): The name of this job. You can set it an arbitrary string. This property is used by searching of the [`GET /job`](#get-job). If you do not set it. HQ sets it `default` automatically.\n- `comment` (string): The arbitrary text to describe this job.\n- `payload` (json): The payload on the HTTP request to a worker application.\n- `headers` (json): Custom HTTP headers on the HTTP request to a worker application.\n- `timeout` (number): timeout seconds of this job. The default is `0` (no timeout).\n\n#### Response\n\n```json\n{\n  \"canceled\": false,\n  \"comment\": \"This is an example job!\",\n  \"createdAt\": \"2019-10-29T23:57:08.713Z\",\n  \"err\": \"\",\n  \"failure\": false,\n  \"finishedAt\": null,\n  \"headers\": null,\n  \"id\": \"109440416981450752\",\n  \"name\": \"example\",\n  \"output\": \"\",\n  \"payload\": {\n    \"message\": \"Hello world!\"\n  },\n  \"running\": false,\n  \"startedAt\": null,\n  \"status\": \"unfinished\",\n  \"statusCode\": null,\n  \"success\": false,\n  \"timeout\": 0,\n  \"url\": \"https://your-worker-app-server/example\",\n  \"waiting\": false\n}\n```\n\n### `GET /job`\n\nLists jobs.\n\n#### Request\n\n```http\nGET /job?name={name}\u0026begin={id}\u0026reverse={true|false}\u0026status={status}\u0026limit={limit}\n```\n\n##### Parameters \u003c!-- omit in toc --\u003e\n\n- `name`: Specifies a regular expression string to filter the jobs with job's name\n- `term`: Specifies a regular expression string to filter the jobs with job's id name, comment, url or status\n- `begin`: Load the jobs from ID. (default: 0)\n- `reverse`: Sort by descending ID.\n- `status`: Specifies STATUS to filter the jobs with job's status (`running|waiting|canceling|failure|success|canceled|unfinished|unknown`).\n- `limit`: Max number of displaying jobs.\n\n#### Response\n\n```json\n{\n  \"jobs\": [\n    {\n      \"canceled\": false,\n      \"comment\": \"\",\n      \"createdAt\": \"2019-10-29T23:57:08.713Z\",\n      \"err\": \"failed to do http request: Post https://localhost/: dial tcp [::1]:443: connect: connection refused\",\n      \"failure\": true,\n      \"finishedAt\": \"2019-10-29T23:57:08.761Z\",\n      \"headers\": null,\n      \"id\": \"109440416981450752\",\n      \"name\": \"default\",\n      \"output\": \"\",\n      \"payload\": {\n        \"message\": \"Hello world!\"\n      },\n      \"running\": false,\n      \"startedAt\": \"2019-10-29T23:57:08.736Z\",\n      \"status\": \"failure\",\n      \"statusCode\": null,\n      \"success\": false,\n      \"timeout\": 0,\n      \"url\": \"https://localhost/\",\n      \"waiting\": false\n    },\n    // ...\n  ],\n  \"hasNext\": true,\n  \"next\": \"109592774310887424\",\n  \"count\": 1\n}\n```\n\n### `GET /job/{id}`\n\nGets a job.\n\n#### Request\n\n```http\nGET /job/{id}\n```\n\n##### Parameters \u003c!-- omit in toc --\u003e\n\n- `id`: Job ID to get.\n\n#### Response\n\n```json\n{\n  \"canceled\": false,\n  \"comment\": \"\",\n  \"createdAt\": \"2019-10-29T23:57:08.713Z\",\n  \"err\": \"failed to do http request: Post https://localhost/: dial tcp [::1]:443: connect: connection refused\",\n  \"failure\": true,\n  \"finishedAt\": \"2019-10-29T23:57:08.761Z\",\n  \"headers\": null,\n  \"id\": \"109440416981450752\",\n  \"name\": \"default\",\n  \"output\": \"\",\n  \"payload\": {\n    \"message\": \"Hello world!\"\n  },\n  \"running\": false,\n  \"startedAt\": \"2019-10-29T23:57:08.736Z\",\n  \"status\": \"failure\",\n  \"statusCode\": null,\n  \"success\": false,\n  \"timeout\": 0,\n  \"url\": \"https://localhost/\",\n  \"waiting\": false\n}\n```\n\n### `DELETE /job/{id}`\n\nDeletes a job.\n\n#### Request\n\n```http\nDELETE /job/{id}\n```\n\n##### Parameters \u003c!-- omit in toc --\u003e\n\n- `id`: Job ID to delete.\n\n#### Response\n\n```json\n{\n  \"id\": \"109440416981450752\"\n}\n```\n\n### `POST /job/{id}/restart`\n\nRestarts a job.\n\n#### Request\n\n```http\nPOST /job/{id}/restart\n```\n\n```json\n{\n  \"copy\": false\n}\n```\n\n##### Parameters \u003c!-- omit in toc --\u003e\n\n- `id`: Job ID to restart.\n- `copy`: If it set `true`, Restarts the copied job instead of updating the existed job.\n\n#### Response\n\n```json\n{\n  \"canceled\": false,\n  \"comment\": \"\",\n  \"createdAt\": \"2019-10-30T10:02:33.531Z\",\n  \"err\": \"\",\n  \"failure\": false,\n  \"finishedAt\": null,\n  \"headers\": null,\n  \"id\": \"109592774310887424\",\n  \"name\": \"default\",\n  \"output\": \"\",\n  \"payload\": {\n    \"message\": \"Hello world!\"\n  },\n  \"running\": false,\n  \"startedAt\": null,\n  \"status\": \"unfinished\",\n  \"statusCode\": null,\n  \"success\": false,\n  \"timeout\": 0,\n  \"url\": \"https://localhost/\",\n  \"waiting\": false\n}\n```\n\n### `POST /job/{id}/stop`\n\nStops a job.\n\n#### Request\n\n```http\nPOST /job/{id}/stop\n```\n\n##### Parameters \u003c!-- omit in toc --\u003e\n\n- `id`: Job ID to stop.\n\n#### Response\n\n```json\n{\n  \"id\": \"109440416981450752\"\n}\n```\n\n## Commands\n\nHQ also provides command-line interface to communicate HQ server. To view a list of the available commands, just run `hq` without any arguments:\n\n```\nNAME:\n   hq - Simplistic job queue engine\n\nUSAGE:\n   hq [global options] command [command options] [arguments...]\n\nVERSION:\n   2.0.0 (5bdbdaf31772c1f5cdd8feb2056e4d5fcafa7a51)\n\nCOMMANDS:\n   delete   Deletes a job\n   info     Displays a job detail\n   list     Lists jobs\n   push     Pushes a new job.\n   restart  Restarts a job\n   serve    Starts the HQ server process\n   stats    Displays the HQ server statistics.\n   stop     Stops a job\n   help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --help, -h     show help (default: false)\n   --version, -v  print the version (default: false)\n\nCOPYRIGHT:\n   Copyright (c) 2019 Kohki Makimoto\n```\n\nSee more detail, Run a sub command with `-h` option.\n\n## Web UI\n\nHQ includes built-in Web UI. The web ui is enabled at default. See `http://localhost:19900/ui` with your browser.\n\n![webui.png](https://raw.githubusercontent.com/kohkimakimoto/hq/master/webui.png)\n\n## Author\n\nKohki Makimoto \u003ckohki.makimoto@gmail.com\u003e\n\n## License\n\nThe MIT License (MIT)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkohkimakimoto%2Fhq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkohkimakimoto%2Fhq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkohkimakimoto%2Fhq/lists"}