{"id":21823253,"url":"https://github.com/orlandos-nl/mongoqueue","last_synced_at":"2025-04-14T04:05:59.895Z","repository":{"id":40322261,"uuid":"405607713","full_name":"orlandos-nl/MongoQueue","owner":"orlandos-nl","description":"MongoDB Job Queue in Swift","archived":false,"fork":false,"pushed_at":"2024-12-26T10:28:30.000Z","size":109,"stargazers_count":12,"open_issues_count":3,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T12:42:05.207Z","etag":null,"topics":["job-queue","mongodb","mongokitten","server-side-swift","swift-nio"],"latest_commit_sha":null,"homepage":"","language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orlandos-nl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-09-12T10:10:16.000Z","updated_at":"2025-02-01T23:03:58.000Z","dependencies_parsed_at":"2023-01-28T12:01:13.590Z","dependency_job_id":"a089395c-f82a-4466-a6ed-c6ec27efe65f","html_url":"https://github.com/orlandos-nl/MongoQueue","commit_stats":{"total_commits":47,"total_committers":4,"mean_commits":11.75,"dds":"0.17021276595744683","last_synced_commit":"b664673d50f6fd1ca10604fb9786cce95ccf22df"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orlandos-nl%2FMongoQueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orlandos-nl%2FMongoQueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orlandos-nl%2FMongoQueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orlandos-nl%2FMongoQueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orlandos-nl","download_url":"https://codeload.github.com/orlandos-nl/MongoQueue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248819389,"owners_count":21166477,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["job-queue","mongodb","mongokitten","server-side-swift","swift-nio"],"created_at":"2024-11-27T17:21:53.490Z","updated_at":"2025-04-14T04:05:59.847Z","avatar_url":"https://github.com/orlandos-nl.png","language":"Swift","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg height=\"200px\" style=\"float: right;\" src=\"https://user-images.githubusercontent.com/1951674/224335889-c6345081-fef9-4b01-95ba-c3a718aa91e9.png\" /\u003e MongoQueue\n\nA [MongoKitten](https://github.com/orlandos-nl/MongoKitten) based JobQueue for MongoDB.\n\n[Join our Discord](https://discord.gg/H6799jh) for any questions and friendly banter.\n\n[Read the Docs](https://swiftpackageindex.com/orlandos-nl/MongoQueue/main/documentation/mongoqueue) for more info.\n\n### Quick Start\n\nConnect to MongoDB with MongoKitten regularly:\n\n```swift\nlet db = try await MongoDatabase.connect(to: \"mongodb://localhost/my_database\")\n```\n\nSelect a collection for your job queue:\n\n```swift\nlet queue = MongoQueue(collection: db[\"tasks\"])\n```\n\nDefine your jobs by conforming to `ScheduledTask` (and implicitly `Codable`):\n\n```swift\nstruct RegistrationEmailTask: ScheduledTask {\n    // This type has no context\n    typealias ExecutionContext = Void\n\n    // Computed property, required by ScheduledTask\n    // This executed the task ASAP\n    var taskExecutionDate: Date { Date() }\n    \n    // Stored properties represent the metadata needed to execute the task\n    let recipientEmail: String\n    let userId: ObjectId\n    let fullName: String\n    \n    func execute(withContext context: ExecutionContext) async throws {\n        // TODO: Send the email\n        // Throwing an error triggers `onExecutionFailure`\n    }\n    \n    func onExecutionFailure(failureContext: QueuedTaskFailure\u003cExecutionContext\u003e) async throws -\u003e TaskExecutionFailureAction {\n        // Only attempt the job once. Failing to send the email cancels the job\n        return .dequeue()\n    }\n}\n```\n\nRegister the task to MongoQueue, before it starts.\n\n```swift\n// Context is `Void`, so we pass in a void here\nqueue.registerTask(RegistrationEmailTask.self, context: ())\n```\n\nStart the queue in the background - this is helpful in use inside HTTP applications - or when creating separate workers.\n\n```swift\ntry queue.runInBackground()\n```\n\nAlternatively, run the queue in the foreground and block until the queue is stopped. Only use this if your queue worker is only running as a worker. I.E., it isn't serving users on the side.\n\n```swift\ntry await queue.run()\n```\n\nQueue the task in MongoDB:\n\n```swift\nlet task = RegistrationEmailTask(\n  recipientEmail: \"joannis@orlandos.nl\",\n  userId: ...,\n  fullName: \"Joannis Orlandos\"\n)\ntry await queue.queueTask(task)\n```\n\nTada! Just wait for it to be executed.\n\n## Testing\n\nYou can run all currently active jobs in the queue one-by-one using:\n\n```swift\ntry await queue.runUntilEmpty()\n``` \n\nThis will not run any jobs scheduled for the future, and will exit once there are no current jobs available.\n\n## Parallelisation\n\nYou can set the parallelisation amount **per job queue instance** using the following code:\n\n```swift\nqueue.setMaxParallelJobs(to: 6)\n```\n\nIf you have two containers running an instance of MongoQueue, it will therefore be able to run `2 * 6 = 12` jobs simultaneously.\n\n## Integrate with Vapor\n\nTo access the `queue` from your Vapor Request, add the following snippet:\n\n```swift\nimport Vapor\nimport MongoKitten\nimport MongoQueue\n\nextension Request {\n  public var queue: MongoQueue {\n    return application.queue\n  }\n}\n\nprivate struct MongoQueueStorageKey: StorageKey {\n  typealias Value = MongoQueue\n}\n\nextension Application {\n  public var queue: MongoQueue {\n    get {\n      storage[MongoQueueStorageKey.self]!\n    }\n    set {\n      storage[MongoQueueStorageKey.self] = newValue\n    }\n  }\n\n  public func initializeMongoQueue(withCollection collection: MongoCollection) {\n    self.queue = MongoQueue(collection: collection)\n  }\n}\n```\n\nFrom here, you can add tasks as such:\n\n```swift\napp.post(\"tasks\") { req in\n  try await req.queue.queueTask(MyCustomTask())\n  return HTTPStatus.created\n}\n```\n\n# Advanced Use\n\nBefore diving into more (detailed) APIs, here's an overview of how this works:\n\nWhen you queue a task, it is used to derive the basic information for queueing the job. Parts of these requirements are in the protocol, but have a default value provided by MongoQueue.\n\n## Dequeing Process\n\nEach task has a **category**, a unique string identifying this task's type in the database. When you register your task with MongoQueue, the category is used to know how to decode \u0026 execute the task once it is acquired by a worker.\n\nMongoQueue regularly checks, on a timer (and if possible with Change Streams for better responsiveness) whether a new task is ready to grab. When it pulls a task from MongoDB, it takes the **highest priority** task that is scheduled for execution at this date.\n\nThe priority is `.normal` by default, but urgency can be increased or decreased in a tasks `var priority: TaskPriority { get }`.\n\nWhen the task is taken out of the queue, its `status` is set to `executing`. This means that other jobs can't execute this task right now. While doing so, the task model's `maxTaskDuration` is used as an indication of the expected duration of a task. The expected deadline is set on the model in MongoDB by adding `maxTaskDuration` to the current date.\n\nIf the deadline is reached, other workers can (and will) dequeue the task and put it back into `scheduled`. This assumes the worker has crashed. However, in cases where the task is taking an abnormal amount of time, the worker will update the deadline accordingly.\n\nDue to this system, it is adviced to set urgent and short-lived tasks to a shorter `maxTaskDuration`. But take network connectivity into consideration, as setting it very low (like 5 seconds) may cause the deadline to be reached before it can be prolonged.\n\nIf the task is dequeued, your task model gets a notification in `func onDequeueTask(withId taskId: ObjectId, withContext context: ExecutionContext, inQueue queue: MongoQueue) async throws`.\n\nLikewise, on execution failure you get a call on `func onExecutionFailure(failureContext: QueuedTaskFailure\u003cExecutionContext\u003e) async throws -\u003e TaskExecutionFailureAction` where you can decide whether to requeue, and whether to apply a maximum amount of attempts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forlandos-nl%2Fmongoqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forlandos-nl%2Fmongoqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forlandos-nl%2Fmongoqueue/lists"}