{"id":14956580,"url":"https://github.com/mongodb/curator","last_synced_at":"2025-11-11T21:03:02.465Z","repository":{"id":10990202,"uuid":"58072252","full_name":"mongodb/curator","owner":"mongodb","description":"Curator -- a build and package automation tool","archived":false,"fork":false,"pushed_at":"2024-06-10T18:14:11.000Z","size":66478,"stargazers_count":19,"open_issues_count":10,"forks_count":22,"subscribers_count":108,"default_branch":"main","last_synced_at":"2025-01-29T09:43:55.031Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongodb.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-04T17:51:30.000Z","updated_at":"2024-06-10T18:14:12.000Z","dependencies_parsed_at":"2024-06-10T20:52:58.202Z","dependency_job_id":null,"html_url":"https://github.com/mongodb/curator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fcurator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fcurator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fcurator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fcurator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongodb","download_url":"https://codeload.github.com/mongodb/curator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237088437,"owners_count":19253565,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T13:13:10.073Z","updated_at":"2025-10-19T07:31:11.387Z","avatar_url":"https://github.com/mongodb.png","language":"Go","readme":"=================================================\n``curator`` -- Artifact and Repository Management\n=================================================\n\nOverview\n--------\n\nCurator is a tool that we use at MongoDB to generate our package\nrepositories (e.g. ``repo.mongodb.org`` and\n``repo.mongodb.com``). Additionally, curator provides tooling to\nsupport the automated publication and management of build artifacts as\npart of our continuous integration system.\n\nComponents\n----------\n\nPlease refer to the ``--help`` output for ``curator`` and its\nsub-commands. The following sections provides overviews of these\ncomponents and their use.\n\nS3 Tools\n~~~~~~~~\n\nCurator includes a basic set of AWS S3 operations on the\ncommand line. Although the get, put, and delete operations are faily\nbasic the ``sync-to`` and ``sync-from`` operations provide parallel\ndirectory tree sync operations between S3 buckets and the local file\nsystem. These operations compare objects using MD5 checksums, do\nmulti-threaded file uploads, and retry failed operations with\nexponential backoff, for efficient and robust transfers.\n\nRepobuilder\n~~~~~~~~~~~\n\nThe repobuilder is a helper package to submit Barque jobs to build RPM\nand DEB package repositories in S3. These jobs: sync files from an\nexisting repository, add packages from the local filesystem to the\nrepository, sign packages (for RPM), regenerate package metadata, sign\npackage metadata, generate html pages for web-based display, and sync\nthe changed files to the remote repository.\n\nThe current implementation of the repobuilder process depends on\nexternal repository generation tools (e.g. ``createrepo`` and\n``apt`` tools.) Additionally, the repobuilder currently depends on\nMongoDB's internal signing service.\n\nArtifacts\n~~~~~~~~~\n\nThe artifacts functionality uses the release metadata feeds\n(e.g. ``https://downloads.mongodb.org/full.json``) to fetch and\nextract release build artifacts for local use. It is particularly\nuseful for fetching artifacts for and maintaining local caches of\nMongoDB builds for multiple releases. Set the\n``CURATOR_ARTIFACTS_DIRECTORY`` environment variable or pass the\n``--path`` option to a flag, and then use the ``curator artifacts\ndownload`` command to download files.\n\nThe ``artifacts`` command also includes two exploration subcommands\nfor discovering available builds: Use the ``list-map`` for specific\nlists of available edition, target, and architecture combinations and\n``list-all`` for a list of available target and architectures. Both\nlist operations are specific to a single version.\n\nCombine the artifacts tool with the prune tool to avoid unbounded\ncache growth.\n\nPrune\n~~~~~\n\nPrune is based on the `lru \u003chttps://github.com/evergreen-ci/lru\u003e`_\nlibrary, and takes a file system tree and removes files, based on age,\nuntil the total size of the files is less than a specified maximum\nsize. Prune uses modified time for age, in an attempt to have\nconsistent behavior indepenent of operation system and file system.\n\nThere are two modes of operation, a recursive mode which removes\nobjects from the tree recursively, but skips directory objects, and\ndirectory mode, which does not collect objects recursively, but tracks\nthe size for the contents--recursively--of top-level directories.\n\nDevelopment\n-----------\n\nDesign Principles and Goals\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- All operations in the continuous integration environment should be\n  easily reproducible outside of the environment. In practice, curator\n  exists to build repositories inside of Evergreen tasks; however, it\n  is possible to run all stages of the repository building process by\n  hand. The manual abilities are useful and required for publishing\n  package revisions and repairing corrupt repositories.\n\n- Leverage, as possible, third party libraries and tools. For example,\n  the cache pruning and artifact management functionality is entirely\n  derived from third-party repositories maintained separately from\n  curator.\n\n- Major functionality is implemented and executed in terms of `amboy\n  \u003chttps://github.com/mongodb/amboy\u003e`_ jobs and queues. Not only does\n  this provide a framework for task execution, but leaves the door\n  open to provide curator functionality as a highly available service\n  with minimal modification.\n\nAPIs and Documentation\n~~~~~~~~~~~~~~~~~~~~~~\n\nSee the `godoc API documentation \u003chttp://godoc.org/github.com/mongodb/curator\u003e`_\nfor more information about curator interfaces and internals.\n\nIssues\n~~~~~~\n\nPlease file all issues in the `EVG project\n\u003chttps://jira.mongodb.org/browse/EVG\u003e`_ in the `MongoDB Jira\n\u003chttps://jira.mongodb.org/\u003e`_ instance.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb%2Fcurator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongodb%2Fcurator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb%2Fcurator/lists"}