{"id":14956417,"url":"https://github.com/mongodb/anser","last_synced_at":"2025-11-11T21:02:58.247Z","repository":{"id":26514981,"uuid":"100294206","full_name":"mongodb/anser","owner":"mongodb","description":"Data Transformation/Migration Tool","archived":false,"fork":false,"pushed_at":"2025-03-24T14:44:57.000Z","size":32712,"stargazers_count":20,"open_issues_count":8,"forks_count":17,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-05-07T18:09:00.800Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongodb.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-14T17:47:36.000Z","updated_at":"2025-03-24T14:45:01.000Z","dependencies_parsed_at":"2024-09-05T19:45:37.902Z","dependency_job_id":"a54cae08-2a15-46c6-a2f0-2b837ac9ea3e","html_url":"https://github.com/mongodb/anser","commit_stats":{"total_commits":165,"total_committers":12,"mean_commits":13.75,"dds":0.6666666666666667,"last_synced_commit":"44d490c05007153b9b0e456f61e28e2e59bb4854"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fanser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fanser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fanser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb%2Fanser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongodb","download_url":"https://codeload.github.com/mongodb/anser/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931555,"owners_count":21827112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T13:13:02.476Z","updated_at":"2025-11-11T21:02:53.222Z","avatar_url":"https://github.com/mongodb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"=======================================\n``anser`` -- Database Migration Toolkit\n=======================================\n\nSummary\n-------\n\nAnser is a toolkit for managing evolving data sets for\napplications. It focuses on on-line data transformations and providing\nhigher-level tools to support data modeling and access. \n\nFor the Evergreen project, anser allows us to treat these routine data\nmigrations, data back fills, and retroactively changing the schema of\nlegacy data as part of application code rather than one-off shell\nscripts.\n\nOverview\n--------\n\nIn general, anser migrations have a two-phase approach. First a\ngenerator runs with some configuration and an input query to collect\ninput documents and creation migration jobs. Then, the output of these\ngenerators, are executed in parallel \n\nYou can define generators either directly in your own code, *or* you\ncan use the configuration-file based approach for a more flexible\napproach.\n\nConcepts\n~~~~~~~~\n\nThere are three major types of migrations: \n\n- ``simple``: these migrations perform their transformations using\n  MongoDB's update syntax. Use these migrations for very basic\n  migrations, particularly when you want to throttle the rate of\n  migrations and avoid the use of larger difficult-to-index\n  multi-updates.\n  \n- ``manual``: these migrations call a user-defined function on a\n  ``bson.RawDoc`` representation of the document to migrate. Use these\n  migrations for more complex transformations or those migrations\n  that you want to write in application code. \n  \n- ``stream``: these migrations are similar to manual migrations;\n  however, they pass a database session *and* an iterator to all\n  documents impacted by the migration. These jobs offer ultimate\n  flexibility.\n  \nInternally these jobs execute using amboy infrastructure and make it\npossible to express dependencies between migrations. Additionally the\n`MovingAverageRateLimitedWorkers\n\u003chttps://godoc.org/github.com/mongodb/amboy/pool#NewMovingAverageRateLimitedWorkers\u003e`_\nand `SimpleRateLimitingWorkers\n\u003chttps://godoc.org/github.com/mongodb/amboy/pool#NewSimpleRateLimitedWorkers\u003e`_\nwere developed to support anser migrations, as well as the `adaptive\nordering local queue\n\u003chttps://godoc.org/github.com/mongodb/amboy/queue#NewAdaptiveOrderedLocalQueue\u003e`_\nwhich respects dependency-driven ordering.\n\nConsiderations\n~~~~~~~~~~~~~~\n\nWhile it's possible to do any kind of migration with anser, we have\nfound the following properties to be useful to keep in mind when\nbuilding migrations: \n\n- Write your migration implementations so that they are idempotent so\n  that it's possible to run them multiple times with the same effect.\n\n- Ensure that generator queries are supported by indexes, otherwise\n  the generator processes will force collection scans. \n\n- Rate-Limiting, provided by configuring the underlying amboy\n  infrastructure, focuses on limiting the number of migration (or\n  generator) jobs executed, rather than limiting the jobs based on\n  their impact. \n  \n- Use batch limits. Generators have limits to control the number of\n  jobs that they will produce. This is particularly useful for tests,\n  but may have adverse effects on job dependency, particularly if\n  logical migrations are split across more than one generator\n  function.  \n\nInstallation\n------------\n\nAnser uses Go modules. To download the modules ::\n\n    make mod-tidy\n\nResources\n---------\n\nPlease consult the godoc for most usage. Most of the API is in the `top\nlevel package \u003chttps://godoc.org/github.com/mongodb/anser\u003e`_; however,\nplease do also consider the `model\n\u003chttps://godoc.org/github.com/mongodb/anser/model\u003e`_ \nand `bsonutil \u003chttps://godoc.org/github.com/mongodb/anser/bsonutil\u003e`_ package.\n\nAdditionally you can use the interfaces `db\n\u003chttps://godoc.org/github.com/mongodb/anser/db\u003e`_\npackage as a wrapper for `mgo \u003chttps://godoc.org/github.com/mongodb/anser\u003e`_ to access\nMongoDB which allows you to use `mocks\n\u003chttps://godoc.org/github.com/mongodb/anser/mocks\u003e`_ as needed for\ntesting without depending on a running database instance.\n\nProject\n-------\n\nPlease file feature requests and bug reports in the `EVG project\n\u003chttps://jira.mongodb.com/browse/EVG\u003e`_ of the MongoDB Jira\ninstance. This is also the place to file related amboy and grip\nrequests.\n\nFuture anser development will focus on supporting additional migration\nworkflows, supporting additional MongoDB and BSON utilities, and\nproviding tools to support easier data-life-cycle management.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb%2Fanser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongodb%2Fanser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb%2Fanser/lists"}