{"id":19926619,"url":"https://github.com/microlinkhq/queue","last_synced_at":"2025-05-03T08:32:30.412Z","repository":{"id":40268282,"uuid":"174889059","full_name":"microlinkhq/queue","owner":"microlinkhq","description":"The high resilient queue for processing URLs.","archived":false,"fork":false,"pushed_at":"2024-07-31T21:39:33.000Z","size":120,"stargazers_count":11,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-01T01:45:10.198Z","etag":null,"topics":["consumer","message-queue","microservice","queue"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microlinkhq.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-10T22:30:49.000Z","updated_at":"2024-07-31T21:39:36.000Z","dependencies_parsed_at":"2022-09-11T02:30:47.731Z","dependency_job_id":"5789f946-2ad5-41c5-b839-8f1b15ddd584","html_url":"https://github.com/microlinkhq/queue","commit_stats":null,"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microlinkhq%2Fqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microlinkhq%2Fqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microlinkhq%2Fqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microlinkhq%2Fqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microlinkhq","download_url":"https://codeload.github.com/microlinkhq/queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224358040,"owners_count":17298038,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumer","message-queue","microservice","queue"],"created_at":"2024-11-12T22:29:58.415Z","updated_at":"2024-11-12T22:29:58.480Z","avatar_url":"https://github.com/microlinkhq.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/microlinkhq/cdn/raw/master/dist/logo/banner.png#gh-light-mode-only\" alt=\"microlink logo\"\u003e\n  \u003cimg src=\"https://github.com/microlinkhq/cdn/raw/master/dist/logo/banner-dark.png#gh-dark-mode-only\" alt=\"microlink logo\"\u003e\n  \u003cbr\u003e\n  \u003cbr\u003e\n\u003c/div\u003e\n\n[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)\n\n\u003e High resilient \u0026 reliable URLs processing queue.\n\n## Motivation\n\nWhen you are consuming an API (such as [Microlink API](https://docs.microlink.io/api/#introduction) or using any other API) you need to assume it can be a wide resource and some of them can eventually fail for many and varied reasons: rate limit, timeout, proxy rotation, etc.\n\nSome of these errors can regret simply retrying the original request after waiting a prudent period of time\n\nIn any case, you want to have the guaranteed the URL will be successfully processed in an indeterministic moment in the future.\n\nThis package presents a small but powerful architecture for processing URLs with guarantees.\n\n## Architecture\n\nFor getting guarantees that your requests are successfully processed, we are going to use [bull](https://github.com/OptimalBits/bull), a lightweight FIFO queue backed on redis.\n\nThe requests (called **jobs**) will remain in the queue until a consumer get it and verify is has been successfully processed.\n\n## Sender\n\n\u003e npm run start:sender\n\nThe sender is is who sends the work to process the consumers.\n\nIt's exposed using a HTTP server.\n\nFor sending a job into the queue, just send it using a `GET`\n\n```bash\n$ curl http://localhost:3000\\?url\\=http://microlink.com\\\u0026video\n```\n\nIf the job has been added successfully, you will have a `201 Created` and the `job.id` back.\n\n```bash\nHTTP/1.1 201 Created\n{\n  \"id\": \"7\"\n}\n```\n\nYou can provide two customizable things as query parameters:\n\n- **priority** (optional): Priority value. ranges from 1 (highest priority) to [MAX_INT](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER) (lowest priority).\n- **delay** (optional): An amount of miliseconds to wait until this job can be processed.\n\nAny other query parameter provided will be passed as part of the job.\n\n## Receiver\n\n\u003e npm run start:receiver\n\nThe receivers will process the job pushed by the consumer. \n\nThey are a pool of processes, waiting on idle until they have something to consume.\n\n```\n$ npm run start:receiver\n\nreceiver=0 status=listening\nreceiver=1 status=listening\nreceiver=2 status=listening\nreceiver=3 status=listening\nstatus=processing id=13\nreceiver=0 status=processed id=13 status=success\n```\n\nby default, You are going to have one receiver per physical CPU core. Since we delegated on [farm](https://github.com/Kikobeats/farm-cli#farm-cli), this is easily customizable.\n\n## License\n\n**queue** © [microlink.io](https://microlink.io), released under the [MIT](https://github.com/microlinkhq/queue/blob/master/LICENSE.md) License.\u003cbr\u003e\nAuthored and maintained by [Kiko Beats](https://kikobeats.com) with help from [contributors](https://github.com/microlinkhq/queue/contributors).\n\n\u003e [microlink.io](https://microlink.io) · GitHub [microlink.io](https://github.com/microlinkhq) · Twitter [@microlinkhq](https://twitter.com/microlinkhq)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrolinkhq%2Fqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrolinkhq%2Fqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrolinkhq%2Fqueue/lists"}