{"id":14956619,"url":"https://github.com/mongodb/amboy","last_synced_at":"2025-11-11T21:02:30.224Z","repository":{"id":8658264,"uuid":"58073483","full_name":"mongodb/amboy","owner":"mongodb","description":"Amboy -- A Go(lang) Job Queue Tool","archived":false,"fork":false,"pushed_at":"2025-03-13T15:08:06.000Z","size":36033,"stargazers_count":97,"open_issues_count":9,"forks_count":30,"subscribers_count":107,"default_branch":"main","last_synced_at":"2025-04-09T17:19:59.326Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongodb.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-04T18:08:21.000Z","updated_at":"2025-03-15T23:20:24.000Z","dependencies_parsed_at":"2024-05-20T20:45:50.027Z","dependency_job_id":"5485e8a2-21fc-4b26-abf8-23d211bb7a77","html_url":"https://github.com/mongodb/amboy","commit_stats":{"total_commits":486,"total_committers":23,"mean_commits":"21.130434782608695","dds":0.6584362139917695,"last_synced_commit":"dec677a8b2e0736bb366b70a865a7b324a3cdc39"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Famboy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Famboy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Famboy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Famboy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongodb","download_url":"https://codeload.github.com/mongodb/amboy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248074922,"owners_count":21043490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T13:13:13.500Z","updated_at":"2025-11-11T21:02:30.102Z","avatar_url":"https://github.com/mongodb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"================================================\n``amboy`` -- Job and Worker Pool Infrastructure\n================================================\n\nOverview\n--------\n\nAmboy is a collection of interfaces and tools for running and managing\nasynchronous background work queues in the context of Go programs, and\nprovides a number of interchangeable and robust methods for running\njobs.\n\nFeatures\n--------\n\nQueues\n~~~~~~\n\nQueue implementations impose ordering and dispatching behavior, and\ndescribe the storage of jobs before and after work is\ncomplete. Current queue implementations include:\n\n- a limited size queue that keep a fixed number of completed jobs in\n  memory, which is ideal for long-running background processes.\n\n- remote queues that store all jobs in an external storage system\n  (e.g. a database) to support architectures where multiple processes\n  can service the same underlying queue.\n\nQueue Groups\n~~~~~~~~~~~~\n\nThe `QueueGroup \u003chttps://godoc.org/github.com/mongodb/amboy#QueueGroup\u003e`_\ninterface provides a mechanism to manage collections of queues. There are remote\nand local versions of the queue group possible, but these groups make it\npossible to create new queues at runtime, and improve the isolation of queues\nfrom each other.\n\nRetryable Queues\n~~~~~~~~~~~~~~~~\n\nThe `RetryableQueue\n\u003chttps://godoc.org/github.com/mongodb/amboy#RetryableQueue\u003e_` interface provides\na superset of the queue functionality. Along with regular queue operations, it\nalso supports jobs that can retry. When a job finishes executing and needs to\nretry (e.g. due to a transient error), the retryable queue will automatically\nre-run the job.\n\nRunners\n~~~~~~~\n\nRunners are the execution component of the worker pool, and are\nembedded within the queues, and can be injected at run time before\nstarting the queue pool. The `LocalWorkers\n\u003chttps://godoc.org/github.com/mongodb/amboy/pool#LocalWorkers\u003e`_\nimplementation executes jobs in a fixed-size worker pool, which is\nthe default of most queue implementations.\n\nAdditional implementation provide rate limiting, and it would be possible to\nimplement runners which used the REST interface to distribute workers to a\nlarger pool of processes, where existing runners simply use go routines.\n\nDependencies\n~~~~~~~~~~~~\n\nThe `DependencyManager\n\u003chttps://godoc.org/github.com/mongodb/amboy/dependency#Manager\u003e`_\ninterface makes it possible for jobs to express relationships to each\nother and to their environment so that Job operations can noop or\nblock if their requirements are not satisfied. The data about\nrelationships between jobs can inform job ordering.\n\nThe handling of dependency information is the responsibility of the\nqueue implementation. Most queue implementations do not support this, unless\nexplicitly stated.\n\nManagement\n~~~~~~~~~~\n\nThe `management package\n\u003chttps://godoc.org/github.com/mongodb/amboy/management\u003e`_ centers around a\n`management interface\n\u003chttps://godoc.org/github.com/mongodb/amboy/management#Manager\u003e`_ that provides\nmethods for reporting and safely interacting with the state of jobs.\n\nREST Interface\n~~~~~~~~~~~~~~\n\nThe REST interface provides tools to manage jobs in an Amboy queue provided as a\nservice. The rest package in Amboy provides the tools to build clients and\nservices, although any client that can construct JSON-formatted Job object can\nuse the REST API.\n\nAdditionally the REST package provides remote implementations of the `management\ninterface \u003chttps://godoc.org/github.com/mongodb/amboy/rest#ManagementService\u003e`_\nwhich makes it possible to manage and report on the jobs in an existing queue,\nand the `abortable pool\n\u003chttps://godoc.org/github.com/mongodb/amboy/rest#AbortablePoolManagementService\u003e`_\ninterface, that makes it possible to abort running jobs. These management tools\ncan help administrators of larger amboy systems gain insights into the current\nbehavior of the system, and promote safe and gentle operational interventions.\n\nSee the documentation of the `REST package\n\u003chttps://godoc.org/github.com/mongodb/amboy/rest\u003e`_\n\nLogger\n~~~~~~\n\nThe Logger package provides amboy.Queue backed implementation of the grip\nlogging system's sender interface for asynchronous log message delivery. These\njobs do not support remote-backed queues.\n\nPatterns\n--------\n\nThe following patterns have emerged during our use of Amboy.\n\nBase Job\n~~~~~~~~\n\nEmbed the `job.Base \u003chttps://godoc.org/github.com/mongodb/amboy/job/#Base\u003e`_\ntype in your Job implementations. This provides a number of helpers for\nbasic job definition, in addition to implementations of all general methods in\nthe interface. With the Base, you only need to implement a ``Run()`` method and\nwhatever application logic is required for the job.\n\nThe only case where embedding the Base type *may* be contraindicated is in\nconjunction with the REST interface, as the Base type may require more\ncomplicated initialization processes.\n\nChange Queue Implementations for Different Deployment Architectures\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf your core application operations are implemented in terms of Jobs, then\nyou can: execute them independently of queues by calling the ``Run()`` method,\nuse a locally backed queue for synchronous operation for short running queues,\nand use a limited size queue or remote-backed queue as part of a long running\nservice.\n\nPlease submit pull requests or `issues \u003chttps://github.com/mongodb/amboy\u003e`_ with\nadditional examples of amboy use.\n\nAPI and Documentation\n---------------------\n\nSee the `API documentation \u003chttps://godoc.org/github.com/mongodb/amboy\u003e`\nfor more information about amboy interfaces and internals.\n\nNotice for External Users\n-------------------------\nAmboy is being continuously developed for `Evergreen \u003chttps://github.com/evergreen-ci/evergreen\u003e`. This is not a stable\nlibrary and upgrades are at your own risk - it may be changed to add, remove, or modify functionality in a way that\nbreaks backward compatibility.\n\nDevelopment\n-----------\n\nGetting Started\n~~~~~~~~~~~~~~~\n\nAmboy uses Go modules. To download the modules ::\n\n    make mod-tidy\n\nAll project automation is managed by a makefile, with all output captured in the\n`build` directory. Consider the following operations: ::\n\n   make compile                 # runs a test compile\n   make test                    # tests all packages\n   make test-\u003cpackage\u003e          # runs the tests only for a specific packages\n   make lint                    # lints all packages\n   make lint-\u003cpackage\u003e          # lints a specific package\n   make html-coverage           # generates the HTML coverage report for all packages\n   make html-coverage-\u003cpackage\u003e # generates the HTML coverage report for a specific package\n\nThe buildsystem also has a number of flags, which may be useful for more\niterative development workflows: ::\n\n  RUN_TEST=\u003cTestName\u003e   # specify a test name or regex to run a subset of tests\n  RUN_COUNT=\u003cnum\u003e       # run a test more than once to isolate an intermittent failure\n  RACE_DETECTOR=true    # run specified tests with the race detector enabled. \n\nIssues\n~~~~~~\n\nPlease file all issues in the `EVG project\n\u003chttps://jira.mongodb.org/browse/EVG\u003e`_ in the `MongoDB Jira\n\u003chttps://jira.mongodb.org/\u003e`_ instance.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb%2Famboy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongodb%2Famboy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb%2Famboy/lists"}