{"id":17696408,"url":"https://github.com/joker666/cogman","last_synced_at":"2025-05-06T20:42:42.432Z","repository":{"id":77573335,"uuid":"313049393","full_name":"Joker666/cogman","owner":"Joker666","description":"Simple, efficient background processing for Golang backed by RabbitMQ and Redis","archived":false,"fork":false,"pushed_at":"2022-08-30T14:34:03.000Z","size":230,"stargazers_count":41,"open_issues_count":1,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-29T06:33:49.567Z","etag":null,"topics":["background-jobs","golang","job-queue","mongodb","queue","rabbitmq","redis","task-scheduler","worker-queue"],"latest_commit_sha":null,"homepage":"https://joker666.github.io/cogman/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Joker666.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-11-15T14:30:07.000Z","updated_at":"2024-01-30T19:31:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"fe374711-447a-414d-9590-9e62a9c345dc","html_url":"https://github.com/Joker666/cogman","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joker666%2Fcogman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joker666%2Fcogman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joker666%2Fcogman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Joker666%2Fcogman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Joker666","download_url":"https://codeload.github.com/Joker666/cogman/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252768671,"owners_count":21801371,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","golang","job-queue","mongodb","queue","rabbitmq","redis","task-scheduler","worker-queue"],"created_at":"2024-10-24T14:44:15.085Z","updated_at":"2025-05-06T20:42:42.410Z","avatar_url":"https://github.com/Joker666.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/Joker666/cogman.svg?style=svg)](https://circleci.com/gh/Joker666/cogman) [![Go Report Card](https://goreportcard.com/badge/Joker666/cogman)](https://goreportcard.com/report/github.com/Joker666/cogman) [![GitHub](https://img.shields.io/github/license/Joker666/cogman)](https://github.com/Joker666/cogman/blob/master/LICENSE) [![godoc for Joker666/cogman](https://godoc.org/github.com/nathany/looper?status.svg)](http://godoc.org/github.com/Joker666/cogman)\n\n![logo](https://i.imgur.com/by0sIiG.png)\n\n## Table of Contents\n\n- [How to Use](#how-to-use)\n- [Motivation](#motivation)\n- [Requirements](#requirements)\n- [Features](#features)\n- [Examples](#examples)\n- [Setup](#setup)\n  - [Config](#config)\n  - [Client/Server](#clientserver)\n  - [Task](#ask)\n  - [Worker/Task Hander](#workertask-handler)\n  - [Register The Handlers](#register-the-handlers)\n  - [Send task](#send-task)\n- [Feature Comparison](#feature-comparison)\n- [Contribution](#contribution)\n- [License](#license)\n\n## How to use:\n\nFirst add it to `$GOPATH`\n\n```\ngo get github.com/Joker666/cogman\n```\n\nThen add [configuration](#Config) for rabbitmq for messaging, redis as backend, optionally mongodb as backend for re-enqueuing feature. \u003cbr /\u003e\nStart the server to consume the tasks and start the client session to send tasks to server. [Start server and client](#clientserver). \u003cbr /\u003e\nWrite [task handlers](#workertask-handler) and [register](#register-the-handlers) them.\n[Send the tasks](#send-task) to process.\nAnd voila!, you have set up the simplest background processing job server.\n\nYou should see something like this when everything is up and running\n![List of services](https://i.imgur.com/AyIJkW8.png)\n\n## Motivation\n\nIn python world you have [Celery](https://github.com/celery/celery), in Ruby world you have [Resque](https://github.com/resque/resque), [SideKiq](https://github.com/mperham/sidekiq), in C# [Hangfire](https://github.com/HangfireIO/Hangfire). All of them had one thing in common, simple interface to get started. When building products in Golang, this was apparent that, there is no library with a simple interface. We have Machinery, which is an excellent library, but has a steep learning curve. Also it has tonnes of features baked in that you do not need but is required anyway.\n\nAlso the way it handled processing of future tasks with RabbitMQ's [Dead Letter Exchange](https://www.rabbitmq.com/dlx.html), we were not very fond of it. So we decided to make our own job processing library. This is a opinionated library as it uses [RabbitMQ](https://www.rabbitmq.com/) as the message broker and [Redis](https://redis.io/) and optionally [MongoDB](https://www.mongodb.com/) backend for more features. This setup has worked great for us in production and under stress and I believe this can work for large tasks as well\n\n## Requirements\n\n- Go\n- RabbitMQ\n- Redis\n- MongoDB (optional)\n\n## Features\n\n- [x] Task Priority\n- [x] Persistence\n- [x] [Queue](#queue) type\n- [x] Retries\n- [x] Multiple consumer \u0026 server\n- [x] Concurrency\n- [x] Redis and Mongo log\n- [x] [Re-enqueue](#re-enqueue) recovered task\n- [x] Handle [Reconnection](#re-connection)\n- [ ] UI\n- [x] Rest API\n\n## Examples\n\n- [Simple use](https://github.com/Joker666/cogman/tree/master/example/simple)\n- [Queue type](https://github.com/Joker666/cogman/tree/master/example/queue)\n- [Client \u0026 Server](https://github.com/Joker666/cogman/tree/master/example/client-server)\n- [Task \u0026 Task Handler](https://github.com/Joker666/cogman/tree/master/example/tasks)\n\n## Setup\n\n### Config\n\nCogman api config example.\n\n```go\ncfg := \u0026config.Config{\n    ConnectionTimeout: time.Minute * 10, // default value 10 minutes\n    RequestTimeout   : time.Second * 5,  // default value 5 second\n\n\n    AmqpURI : \"amqp://localhost:5672\",                  // required\n    RedisURI: \"redis://localhost:6379/0\",               // required\n    MongoURI: \"mongodb://root:secret@localhost:27017/\", // optional\n\n    RedisTTL: time.Hour * 24 * 7,  // optional. default value 1 week\n    MongoTTL: time.Hour * 24 * 30, // optional. default value 1 month\n\n    HighPriorityQueueCount: 2,     // optional. default value 1\n    LowPriorityQueueCount : 4,     // optional. default value 1\n    StartRestServer:        true   // optional. default value false\n}\n```\n\nClient \u0026 Server also has individual config file to use them separately.\n\n### Client/Server\n\nThis Cogman api call will start a client and a server.\n\n```go\nif err := cogman.StartBackground(cfg); err != nil {\n    log.Fatal(err)\n}\n```\n\nInstead, if you want you can initiate Client \u0026 Server individually:\n\n```go\n// Client\nclient, err := cogman.NewSession(cfg)\nif err != nil {\n    log.Fatal(err)\n}\n\nif err := client.Connect(); err != nil {\n    log.Fatal(err)\n}\n\n// Server\nserver, err := cogman.NewServer(cfg)\nif err != nil {\n    log.Fatal(err)\n}\n\ngo func() {\n    defer server.Stop()\n    if err = server.Start(); err != nil {\n        log.Fatal(err)\n    }\n}()\n\n```\n\n### Task\n\nTasks are grouped by two priority level. Based on that it will be assigned to a queue.\n\n```go\ntype Task struct {\n    TaskID         string       // unique. ID should be assigned by Cogman.\n    Name           string       // required. And Task name must be registered with a task handler\n    OriginalTaskID string       // a retry task will carry it's parents ID.\n    PrimaryKey     string       // optional. Client can set any key to trace a task.\n    Retry          int          // default value 0.\n    Prefetch       int          // optional. Number of task fetch from queue by consumer at a time.\n    Payload        []byte       // required\n    Priority       TaskPriority // required. High or Low\n    Status         Status       // current task status\n    FailError      string       // default empty. If Status is failed, it must have a value.\n    Duration       *float64     // task execution time.\n    CreatedAt      time.Time    // create time.\n    UpdatedAt      time.Time    // last update time.\n}\n```\n\n### Worker/Task Handler\n\nAny struct can be passed as a handler it implements below `interface`:\n\n```go\ntype Handler interface {\n\tDo(ctx context.Context, payload []byte) error\n}\n```\n\nA function type `HandlerFunc` also can pass as handler.\n\n```go\ntype HandlerFunc func(ctx context.Context, payload []byte) error\n\nfunc (h HandlerFunc) Do(ctx context.Context, payload []byte) error {\n\treturn h(ctx, payload)\n}\n```\n\n### Register The Handlers\n\n```go\n// Register task handler from Server side\nserver.Register(taskName, handler)\nserver.Register(taskName, handlerFunc)\n```\n\n### Send Task\n\nSending task using Cogman API:\n\n```go\nif err := cogman.SendTask(*task, handler); err != nil {\n\tlog.Fatal(err)\n}\n\n// If a task handler is already registered, you can pass nil.\nif err := cogman.SendTask(*task, nil); err != nil {\n\tlog.Fatal(err)\n}\n\n```\n\nSending task using Cogman Client/Server:\n\n```go\n// Sending task from client\nif err := client.SendTask(task); err != nil {\n    return err\n}\n```\n\n### Queue\n\nCogman queue type:\n\n```\n- High_Priority_Queue [default Queue]\n- Low_Priority_Queue  [lazy Queue]\n```\n\nThere are two types queues that Cogman maintains. Default \u0026 Lazy queue.\nHigh priority tasks would be pushed to default queue and low priority task would be pushed to lazy queue.\nThe number of each type of queues can be set by client/server through configuration.\nQueue won't be lost after any sort of connection interruption.\n\n### Re-Connection\n\nCogman Client \u0026 Server both handles reconnection. If the client loses connection, it can still take tasks,\nand those will be processed immediate after Cogman client gets back the connection.\nAfter Server reconnects, it will start to consume tasks without losing any task.\n\n### Re-Enqueue\n\nRe-enqueue feature to recover all the initiated task those are lost for connection error.\nIf client somehow loses the amqp connection, Cogman can still take the task in offline.\nAll offline task will be re-queue after connection re-established.\nCogman fetches all the offline tasks from mongo logs, and re-initiate them. Mongo connection required here.\nFor re-enqueuing, task retry count would not change.\n\n## Feature Comparison\n\nComparison among the other job/task process runner.\n\n| Feature                  |   Cogman    | Machinery |\n| :----------------------- | :---------: | :-------: |\n| Backend                  | redis/mongo |   redis   |\n| Priorities               |      ✓      |     ✓     |\n| Re-Enqueue               |      ✓      |           |\n| Concurrency              |      ✓      |     ✓     |\n| Re-Connection            |      ✓      |           |\n| Delayed jobs             |             |     ✓     |\n| Concurrent client/server |      ✓      |           |\n| Re-try                   |      ✓      |     ✓     |\n| Persistence              |      ✓      |     ✓     |\n| UI                       |             |           |\n| Rest API                 |      ✓      |           |\n| Chain                    |             |     ✓     |\n| Chords                   |             |     ✓     |\n| Groups                   |             |     ✓     |\n\n## Contribution\n\nWant to contribute? Great!\n\nTo fix a bug or enhance an existing module, follow these steps:\n\n- Fork the repo\n- Create a new branch (`git checkout -b improve-feature`)\n- Make the appropriate changes in the files\n- Add changes to reflect the changes made\n- Commit your changes (`git commit -am 'Improve feature'`)\n- Push to the branch (`git push origin improve-feature`)\n- Create a Pull Request\n\n### Bug / Feature Request\n\nIf you find a bug, kindly open an issue [here](https://github.com/Joker666/cogman/issues/new).\u003cbr/\u003e\nIf you'd like to request/add a new function, feel free to do so by opening an issue [here](https://github.com/Joker666/cogman/issues/new).\n\n## [License](https://github.com/Joker666/cogman/blob/master/LICENSE.md)\n\nMIT © [MD Ahad Hasan](https://github.com/joker666)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoker666%2Fcogman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoker666%2Fcogman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoker666%2Fcogman/lists"}