{"id":14956894,"url":"https://github.com/freekmencke/fmq","last_synced_at":"2026-01-19T22:32:02.209Z","repository":{"id":89640661,"uuid":"169623123","full_name":"FreekMencke/FMQ","owner":"FreekMencke","description":"FMQ is a threadsafe, simple message queuing system with little overhead, exposed by a REST api.","archived":false,"fork":false,"pushed_at":"2023-04-25T17:59:05.000Z","size":1119,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-13T14:38:53.067Z","etag":null,"topics":["message-queue","mongo-db","mq","queue","rest-api","thread-safe","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreekMencke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-07T18:45:49.000Z","updated_at":"2022-12-14T22:57:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"ccaeda35-572b-4011-bf65-eb3780710a01","html_url":"https://github.com/FreekMencke/FMQ","commit_stats":{"total_commits":73,"total_committers":4,"mean_commits":18.25,"dds":"0.31506849315068497","last_synced_commit":"7aeac91aab43480d7bf01c7ae703eae2d28a727a"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreekMencke%2FFMQ","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreekMencke%2FFMQ/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreekMencke%2FFMQ/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreekMencke%2FFMQ/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreekMencke","download_url":"https://codeload.github.com/FreekMencke/FMQ/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247648928,"owners_count":20972942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["message-queue","mongo-db","mq","queue","rest-api","thread-safe","typescript"],"created_at":"2024-09-24T13:13:41.389Z","updated_at":"2026-01-19T22:32:02.179Z","avatar_url":"https://github.com/FreekMencke.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"![CI](https://github.com/FreekMencke/fmq/workflows/CI/badge.svg)\n[![GitHub issues](https://img.shields.io/github/issues/FreekMencke/fmq.svg)](https://github.com/FreekMencke/fmq/issues)\n[![GitHub license](https://img.shields.io/github/license/FreekMencke/fmq.svg)](https://github.com/FreekMencke/fmq/blob/master/LICENSE)\n\n# FMQ\n\nFMQ is a simple message queuing system with little overhead.\nIt is exposed through an Express API and preserved through a Mongo database.\n\nFMQ is multithread and replica safe.\n\n## Getting started\n\nEither clone and build this repository, or pull the [image](https://hub.docker.com/r/freekmencke/fmq) from Docker hub.\n\n### Kubernetes/Docker\n\n`docker pull freekmencke/fmq:latest`\n\nTo run FMQ from a container, provide the Mongo DB connection details. The connection details can be provided as a secret (**recommended**) or an environment variable. If both sources are provided, the environment variables will be prioritized.\n\nThe container runs on port 8080, but can mapped to any port with docker. Metrics are available on port 8088 at `/metrics`.\n\n#### Environment variables\n| Name | Description |\n| ---- | ---- |\n| **ALLOWED_ORIGINS** | Comma separated list of allowed origins (CORS). |\n| **PORT** | The port FQM will listen for requests (default `8080`). |\n| **PORT_METRICS** | The port where metrics can be queried (default `8088`)|\n| **MONGO_USER** | The username used to connect to the Mongo DB. |\n| **MONGO_PASSWORD** | The password used to connect to the Mongo DB. |\n| **MONGO_URL** | The url used to connect to the Mongo DB (mongodb://...). |\n| **MONGO_AUTHSOURCE** | The database associated with the user credentials. |\n| **MONGO_AUTHMECHANISM**| The authentication mechanism that MongoDB will use for the connection. |\n\n## Concepts\n\n### Actions\n\nFMQ uses basic MQ concepts with a few extra features. The available actions are:\n\n- [AckMany](#ackmany)\n- [Peek](#peek)\n- [PingMany](#pingmany)\n- [PopOne](#popone)\n- [PopMany](#popmany)\n- [Push](#push)\n- [Size](#size)\n\n### Multi threading and replicating\n\nAll actions are **safe to be called simultaniously**, which makes FMQ perfect to use with docker replicas for example.\n\n### Command History\n\nFMQ also utilizes a concept called **Command history**. Command history can be used by **Push** actions to determine if a push with that data has already occured, and can therefore be ignored. Command history is used by sending a unique hash and a duration during which duplicate messages shouldn't be added to the queue (default 300 seconds).\n\n### Dead queue\n\nSome messages might not be able to complete successfully. To prevent the message from being permanently in the queue there is the **dead queue** system. After a message has had 5 unsuccessful attempts, the message will be moved to a dead queue. A dead queue is a queue with the same name as the normal queue, appended with `-dead`.\n\nFor example we have a `player-hiscore-scraper` queue we use to push players to we want to have scraped. Somehow a non-existing username gets pushed and it fails 5 times. The user then gets moved to the `player-hiscore-scraper-dead` queue.\n\nThis allows us to easily see which messages failed. We could implement a system which will notify us when a message would fail to process.\n\n## Queues\n\nAny number of queues can be used simultaneously.\nAll requests to perform actions on a queue have to start with `/queue/:queueName`. For example, we could check the size of the `test` queue by performing `GET /queue/test/size`.\n\n## API\n\n### AckMany\n\nAfter we processed messages successfully, we use this endpoint to remove the messages from the queue. This is important, otherwise after the expiry time of message has passed the message will be available again on the queue.\n\nEndpoint: `POST /queue/:queueName/ack`\n\nPossible status codes: `204` and `500`.\n\nExpected payload:\n\n```ts\nid: string[]\n```\n\n### Peek\n\nWe can look at the messages in a queue using this endpoint. We can use the `filter` parameter to pass a json with [Mongo query operators](https://docs.mongodb.com/manual/reference/operator/) to look for specific messages. We can also use the `amount` and `offset` parameters to further customize our results.\n\nEndpoint: `GET /queue/:queueName/peek(?filter=[json])(\u0026amount=[number])(\u0026offset=[number])`\n\nPossible status codes: `200`, `204` and `500`.\n\nReturns:\n\n```ts\n[{\n  \"_id\": string,\n  \"payload\": any,\n  \"attempts\"?: number,\n  \"expiryDate\"?: Date\n},...]\n```\n\n### PingMany\n\nWhen processing messages might take longer than the `expiresIn` parameter (standard 300s), we can use this endpoint to extend the time we have to process the message. We can use the optional `expiresIn` query parameter to choose how long we want to extend the messages (standard 300s).\nIf we don't and we pass the expiry date, the message will become available again on the queue before we can [AckMany](#ackmany) it.\n\nEndpoint: `POST /queue/:queueName/ping(?expiresIn=[number])`\n\nPossible status codes: `200` and `500`.\n\nExpected payload:\n\n```ts\nid: string[]\n```\n\n### PopOne\n\nWe can call this method to retrieve a message from a certain queue. We can use the optional `expiresIn` parameter (standard 300s). After the `expiresIn` time has passed without an [AckMany](#ackmany) call, we assume the message failed to process and it will become available on the queue again, unless we [PingMany](#pingmany) it.\n\nEndpoint: `POST /queue/:queueName/pop(?expiresIn=[number])`\n\nPossible status codes: `200`, `204` and `500`.\n\nReturns:\n\n```ts\n{\n  \"_id\": string,\n  \"payload\": any,\n  \"attempts\": number,\n  \"expiryDate\": Date\n}\n```\n\n### PopMany\n\nWe can call this method to retrieve multiple messages from a certain queue. The `amount` route parameter is used to choose the amount of messages to pop. We can use the optional `expiresIn` parameter (standard 300s). After the `expiresIn` time has passed without an [AckMany](#ackmany) call, we assume the message failed to process and it will become available on the queue again, unless we [PingMany](#pingmany) it.\n\nEndpoint: `POST /queue/:queueName/pop/:amount(?expiresIn=[number])`\n\nPossible status codes: `200`, `204` and `500`.\n\nReturns:\n\n```ts\n[{\n  \"_id\": string,\n  \"payload\": any,\n  \"attempts\": number,\n  \"expiryDate\": Date\n}, ...]\n```\n\n### Push\n\nThe push method is used to put a new message on the queue. This call accepts an object (PushOne) or an array (PushMany).\n\nTo make sure that multiple docker containers trying to push the same message simultaniously will only add 1 message to the queue, we can use the optional `hashCode` parameter. The `hashCode` parameter will add this command to the [Command History](#command-history).\n\nThe hashCode should be a hash of your payload (you can add more values like dates to make it more unique). We can use the optional `expiresIn` parameter (standard 300s) to decide how long this command will be in the [Command History](#command-history). While there is a hash already in the history, any attempt to push the message will be ignored.\n\nEndpoint: `POST /queue/queueName/push(?hashCode=[string])(\u0026expiresIn=[number])`\n\nPossible status codes: `201`, `204` and `500`.\n\nExpected payload:\n\n```ts\nOne:\n{}: any\n\nMany:\n[{}, ...]: any[]\n```\n\n### Size\n\nWe can use this endpoint to check howmany messages are left on a certain queue.\n\nEndpoint: `GET /queue/:queueName/size`\n\nPossible status codes: `200` and `500`.\n\nReturns:\n\n```ts\n{\n  \"size\": number\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreekmencke%2Ffmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreekmencke%2Ffmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreekmencke%2Ffmq/lists"}