{"id":22671506,"url":"https://github.com/cloudforet-io/db-migration","last_synced_at":"2025-06-24T10:36:47.241Z","repository":{"id":57752987,"uuid":"527465387","full_name":"cloudforet-io/db-migration","owner":"cloudforet-io","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-02T08:10:06.000Z","size":207,"stargazers_count":6,"open_issues_count":0,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-02T14:16:08.813Z","etag":null,"topics":["migration","mongodb","tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudforet-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-08-22T07:55:56.000Z","updated_at":"2025-06-02T08:10:09.000Z","dependencies_parsed_at":"2024-01-08T03:02:12.351Z","dependency_job_id":"5fc15583-44f2-45e8-b02e-225dfb0b6bde","html_url":"https://github.com/cloudforet-io/db-migration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cloudforet-io/db-migration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudforet-io%2Fdb-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudforet-io%2Fdb-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudforet-io%2Fdb-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudforet-io%2Fdb-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudforet-io","download_url":"https://codeload.github.com/cloudforet-io/db-migration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudforet-io%2Fdb-migration/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261652844,"owners_count":23190350,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["migration","mongodb","tools"],"created_at":"2024-12-09T16:15:16.736Z","updated_at":"2025-06-24T10:36:47.224Z","avatar_url":"https://github.com/cloudforet-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DB-migration\n\n---\n\n# Getting started with DB-migration\n\n## 1) Introduction\n\nDB-migration is a tool that executes DB patch contents for MongoDB used by Cloudforet.  \nThrough this tool, you can easily check which DB operation occurred in a specific version.\n\n\u003cbr\u003e\n\u003cbr\u003e\n\n## 2) How to use\n\n### 2-1) Install and Set Up DB-migration\n\n1) First, create a DB-migration clone in the folder you want to install (folder_path).\n\n```shell\n$ git clone https://github.com/cloudforet-io/db-migration.git\n```\n\n2) After creating and activating the virtual environment, install packages used for DB-migration.\n\n* This guide is based on running in a virtualenv environment.\n\n```shell\n$ virtualenv -p 3.8 venv\n\n$ source venv/bin/activate\n\n$ pip3 install -r {folder_path}/pip_requirements.txt\n```\n\n### 2-2) DB-migration execution\n\n**db-migration command**\n\n```shell\n$ {folder_path}/migrate.py version [-f \u003cexternal_config_path\u003e.yml]\n```\n\n- `version` : Version to use for migration (required)\n- `-f {external_config_path}.yml` : external files related to config (optional)\n\n### 2-3) Example of DB-migration\n\nLooking at the following example code, you can interpret it as follows:\n\n```shell\n$ {folder_path}/migrate.py v1.1.1\n```\n\n- It means that you are migrating v1.1.1 and don't use an external config file.\n- Logs generated by DB migration are accumulated in `~/db_migration_log/v1_1_1.log`.\n\n```shell\n$ {folder_path}/migrate.py v1.1.1 -f {external_config_path.yml}\n```\n\n- Migrate v1.1.1 and use an external config file that is `external_config_path.yml`.\n- Logs generated by DB migration are accumulated in `~/external_config_path/v1_1_1.log`.\n\n\u003cbr\u003e\n\u003cbr\u003e\n\n## 3) Settings\n\n```yaml\n---\nCONNECTION_URI: 'localhost:27017'\nPAGE_SIZE: 100\nLOG_PATH: '/var/log/external_log'\n\nDB_NAME_MAP:\n  # DB ALIAS: DB NAME\n  IDENTITY: dev-identity\n  MONITORING: dev-monitoring\n  STATISTICS: dev-statistics\n  SECRET: dev-secret\n  REPOSITORY: dev-repository\n  PLUGIN: dev-plugin\n  CONFIG: dev-config\n  INVENTORY: dev-inventory\n```\n\n- `CONNECTION_URI` : Connection String URI Format\n    - **Standard Connection URL**\n        - String URI Format follows MongoDB standard. Please refer to the following URL for more information.\n        - [Settings guide of standard connection URL](https://www.mongodb.com/docs/manual/reference/connection-string/#connection-string-uri-format)\n    - **Connection URL of DocumentDB**\n        - [Settings guide of DocumentDB connection URL](https://docs.aws.amazon.com/documentdb/latest/developerguide/connect_programmatically.html)\n        - The code below is an example of a Connection URL that supports TLS and connects to an Amazon DocumentDB\n          cluster.\n          ```\n            'mongodb://\u003csample-user\u003e:\u003cpassword\u003e@sample-cluster.node.us-east-1.docdb.amazonaws.com:27017/?replicaSet=rs0\u0026readPreference=secondaryPreferred\u0026retryWrites=false'\n          ```\n\n\n- `BATCH_SIZE` : This parameter is used when using the bulk_write method.\n- `LOG_PATH` : It corresponds to the location of the log file that occurs in DB-migration.\n- `DB_NAME_MAP` : Used as a DB wrapper that maps aliases to real names. In real MongoDB, IDENTITY has the name\n  dev-identity.\n\n\u003cbr\u003e\n\u003cbr\u003e\n\n## 4) Advanced\n\n### 4-1) folder structure\n\n```text\n├── .github\n├── .gitignore\n├── LICENSE\n├── README.md\n├── pip_requirements.txt\n└── src\n    ├── __init__.py\n    ├── conf\n    │   ├── __init__.py\n    │   └── default_conf.py\n    ├── lib\n    │   ├── __init__.py\n    │   ├── logger.py\n    │   ├── mongo_custom_client.py\n    │   └── util.py\n    ├── migrate.py\n    └── migration\n        ├── __init__.py\n        ├── v1_10_1.py\n        ├── v1_10_2.py\n        └── v1_10_3.py\n```\n\nThe file structure is as follows: The parts you need to understand for dual use correspond to the files below.\n\n* `migrate.py`\n  : Execute the db-migration command\n\n* `migration/{version}.py`\n  : The details of migration for each version are specified, and you can check what kind of work was done with what\n  collection of db actually.\n\n* `default_conf.py`\n  : If there is no external configuration file, the config specified in default_conf.py is executed.\n\n### 4-2) Function name\n\nLet's take a look at the detailed specification (version.py) used for the actual specific version of the db patch.\n\n**Example of version.py**\n\n```python\n@print_log\ndef identity_service_account_set_additional_fields(mongo_client: MongoCustomClient):\n    mongo_client.update_many('IDENTITY', 'service_account', {\"service_account_type\": {\"$ne\": \"TRUSTED\"}},\n                             {\"$set\": {'service_account_type': 'GENERAL', 'scope': 'PROJECT'}}, upsert=True)\n\n\ndef main(file_path, 'v1.1.1'):\n    mongo_client: MongoCustomClient = MongoCustomClient(file_path)\n    identity_service_account_set_additional_fields(mongo_client)\n```\n\n- First, def main() is executed. Declare mongo_client between executions and execute the function to work with DB.\n\n- function has `{db}_{collection}_{work content}` as the function name.\n\n- In the above example, `identity_service_account_set_additional_fields` means a DB operation that sets additional\n  fields in the service_account collection of the IDENTITY db.\n\n## 5) Release\n\n### 5-1) Release Note\n\n| Version | Description                                          | Related Images           | Release Date |\n|---------|------------------------------------------------------|--------------------------|--------------|\n| 2.0.7   | Refactor data structure and fields for new CostReport service features          | cost-analysis:2.0.dev223 | 2025-06-02   |\n| 2.0.6   | Add \"service_healthy\" field at Alert-Manager.Service | alert-manager:2.0.dev72  | 2025-04-22   |\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudforet-io%2Fdb-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudforet-io%2Fdb-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudforet-io%2Fdb-migration/lists"}