{"id":18976334,"url":"https://github.com/aliyevh/mad-migration","last_synced_at":"2025-07-30T15:06:50.400Z","repository":{"id":41961438,"uuid":"287740234","full_name":"AliyevH/mad-migration","owner":"AliyevH","description":"Database migration tool to migrate different structured databases.","archived":false,"fork":false,"pushed_at":"2024-04-29T14:32:08.000Z","size":1242,"stargazers_count":29,"open_issues_count":16,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-19T19:25:38.163Z","etag":null,"topics":["alembic","database","database-migration","database-structures","mariadb","mongdb","mysql","postgresql","python","sqlalchemy","transfered-data"],"latest_commit_sha":null,"homepage":"https://aliyevh.github.io/mad-migration/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AliyevH.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-15T12:14:28.000Z","updated_at":"2024-08-12T05:57:12.000Z","dependencies_parsed_at":"2024-04-26T23:24:23.942Z","dependency_job_id":"6191dbec-b10b-440d-8475-083d9d44cf3e","html_url":"https://github.com/AliyevH/mad-migration","commit_stats":null,"previous_names":["aliyevh/mad-migration"],"tags_count":22,"template":false,"template_full_name":null,"purl":"pkg:github/AliyevH/mad-migration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliyevH%2Fmad-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliyevH%2Fmad-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliyevH%2Fmad-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliyevH%2Fmad-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AliyevH","download_url":"https://codeload.github.com/AliyevH/mad-migration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliyevH%2Fmad-migration/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267889223,"owners_count":24161149,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alembic","database","database-migration","database-structures","mariadb","mongdb","mysql","postgresql","python","sqlalchemy","transfered-data"],"created_at":"2024-11-08T15:23:45.194Z","updated_at":"2025-07-30T15:06:50.345Z","avatar_url":"https://github.com/AliyevH.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\r\n  \u003ca href=\"\" rel=\"noopener\"\u003e\r\n \u003cimg width=400px height=400px src=\"https://github.com/MadeByMads/mad-migration/blob/master/docs/img/mm.jpg\" alt=\"Project logo\"\u003e\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n\u003ch3 align=\"center\"\u003eDatabase Migration Tool\u003c/h3\u003e\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\r\n[![Status](https://img.shields.io/badge/status-active-success.svg)]()\r\n[![GitHub Issues](https://img.shields.io/github/issues/kylelobo/The-Documentation-Compendium.svg)](https://github.com/MadeByMads/mad-migration/issues)\r\n[![GitHub Pull Requests](https://img.shields.io/github/issues-pr/kylelobo/The-Documentation-Compendium.svg)](https://github.com/MadeByMads/mad-migration/pulls)\r\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](/LICENSE)\r\n\r\n\u003c/div\u003e\r\n\r\n## 🧐 About \u003ca name = \"about\"\u003e\u003c/a\u003e\r\n\r\nThe Database Migration Tool was designed for those looking to migrate their data from one database to another. Additionally type casting is supported if you need to change column type in new version. Our main goal is to make data migration possible between different database servers. For example mysql, postgresql.\r\n\r\nCurrently only mysql and postgresql is supported. Future planning is to support migration from/to other databases.\r\n\r\n---\r\n**Documentation**: [Documentation](aliyevh.github.io/mad-migration/)\r\n---\r\n\r\n## 🏁 Getting Started \u003ca name = \"getting_started\"\u003e\u003c/a\u003e\r\n\r\n### Installing\r\n\r\n```bash\r\npip install madmigration\r\n```\r\n\r\n### Usage\r\n```bash\r\nmadmigration -f migration_schema.yaml\r\n```\r\n\r\n## Configuration \u003ca name=\"configuration\"\u003e\u003c/a\u003e\r\n\r\nAfter installation you should define YAML file where configuration will be taken in order to apply data to target database. Yaml file with list and dictionaries may contain following structures:\r\n\r\n\r\n![alt text](docs/img/db.jpg)\r\n\r\n#### Connection Config\r\n\r\n- SourceConfig is intented to be data from source database\r\n- DestinationConfig is intented to be transfered data to target database\r\n\r\n```yaml\r\nversion: 0.1.6\r\nConfigs:\r\n  - SourceConfig:\r\n       dbURI: \"postgres://admin:admin@127.0.0.1/oldDB\"\r\n  - DestinationConfig:\r\n       dbURI: \"mysql://admin:admin@127.0.0.1/newDB\"\r\n\r\nmigrationTables:\r\n  - migrationTable:\r\n      SourceTable:\r\n        name: users\r\n      DestinationTable:\r\n        name: persons\r\n        create: True\r\n\r\n      MigrationColumns:\r\n        - sourceColumn:\r\n            name: id\r\n          destinationColumn:\r\n            name: id\r\n            options:\r\n              type_cast: bigint\r\n              primary_key: true\r\n              autoincrement: true\r\n\r\n        - sourceColumn:\r\n            name: name\r\n          destinationColumn:\r\n            name: fname\r\n            options:\r\n              type_cast: varchar\r\n              length: 32\r\n      \r\n        - sourceColumn:\r\n            name: surname\r\n          destinationColumn:\r\n            name: lname\r\n            options:\r\n              type_cast: varchar\r\n              length: 32\r\n              index: true\r\n\r\n        - sourceColumn:\r\n            name: age\r\n          destinationColumn:\r\n            name: age\r\n            options:\r\n              type_cast: int\r\n\r\n        - sourceColumn:\r\n            name: createdAT\r\n          destinationColumn:\r\n            name: created_at\r\n            options:\r\n              type_cast: datetime\r\n\r\n        - sourceColumn:\r\n            name: updatedAT\r\n          destinationColumn:\r\n            name: updated_at\r\n            options:\r\n              type_cast: datetime\r\n\r\n```\r\n\r\n`Configs section`\r\n- `SourceConfig` set the source database configurations\r\n  - `dbURI` source database URI \r\n- `DestinationConfig` set the destination database configurations\r\n  - `dbURI` destination database URI \r\n```yml\r\nConfigs:\r\n  - SourceConfig:\r\n       dbURI: \"postgres://root:admin@127.0.0.1/oldDB\"  # set source database uri\r\n  - DestinationConfig:\r\n       dbURI: \"mysql://root:admin@127.0.0.1/newDB\"  # set destination database uri\r\n```\r\n`migrationTables section`\r\n- `migrationTables` in this configuration, you will write the source of the table that you have to migrate and the destination tables that will migrate the data. \r\n  - `migrationTable` specify the source and destination table name\r\n    - `SourceTable` information about source table\r\n      - `name` source table name\r\n    - `DestinationTable` information about destination table\r\n      - `name` destination table name\r\n      - `create` bool value. This parameter tells the program whether it should create a table or not. (`default false`)\r\n```yml\r\nmigrationTables:\r\n  - migrationTable:\r\n      SourceTable:\r\n        name: users\r\n      DestinationTable:\r\n        name: persons\r\n        create: True\r\n```\r\n`MigrationColumns section` \r\n- `MigrationColumns` specify source and destination column\r\n  - `sourceColumn`  information about source column\r\n    - `name` source column name\r\n  - `destinationColumn` information about destination column\r\n    - `name` destination column name\r\n    - `options` column options\r\n      - `type_cast` destination column type name varchar,integer etc. (`when we convert data we use this parameter`) \r\n\r\n```yml\r\nMigrationColumns:\r\n  - sourceColumn:\r\n      name: id\r\n    destinationColumn:\r\n      name: id\r\n      options:\r\n        type_cast: bigint\r\n        primary_key: true\r\n        autoincrement: true\r\n```\r\n\r\n**If you want to create a foreign key you can specify it in the column parameters**\r\n```yml\r\n- sourceColumn:\r\n    name: USERID\r\n  destinationColumn:\r\n    name: user_id\r\n    options:\r\n      type_cast: uuid\r\n      foreign_key:\r\n        table_name: users\r\n        column_name: id\r\n        ondelete: CASCADE\r\n```\r\n\r\n### You can split your .yaml files or import .json file into .yaml file.\r\nYou must create the main .yaml file and importing other files into main .yaml file. \r\n\r\n`main.yaml` file\r\n```yml \r\nversion: 1.1\r\nConfigs:\r\n  - SourceConfig:\r\n      dbURI: \"mysql://admin:admin@127.0.0.1/old\"\r\n  - DestinationConfig:\r\n      dbURI: \"postgresql://admin:admin@127.0.0.1/new\"\r\n\r\nmigrationTables:\r\n  - migrationTable: !import company.yaml\r\n  - migrationTable: !import op_cond.json\r\n\r\n```\r\n\r\n`company.yaml` file\r\n```yml\r\nSourceTable:\r\n  name: company\r\nDestinationTable:\r\n  name: company\r\n  create: true \r\n\r\nMigrationColumns:\r\n  - sourceColumn:\r\n      name: id\r\n    destinationColumn: \r\n      name: id\r\n      options:\r\n        primary_key: true\r\n        type_cast: uuid\r\n\r\n  - sourceColumn:\r\n      name: name\r\n    destinationColumn:\r\n      name: name\r\n      options:\r\n        length: 120\r\n        type_cast: varchar\r\n        nullable: false\r\n\r\n  - sourceColumn:\r\n      name: created\r\n    destinationColumn:\r\n      name: created\r\n      options:\r\n        type_cast: datetime\r\n  - sourceColumn:\r\n      name: updated\r\n    destinationColumn:\r\n      name: updated\r\n      options:\r\n        type_cast: datetime\r\n        nullable: true\r\n```\r\n\r\n`op_conds.json` file\r\n```json\r\n{\r\n    \"SourceTable\": {\r\n      \"name\": \"operation_conditions\"\r\n    },\r\n    \"DestinationTable\": {\r\n      \"name\": \"operation_conditions\",\r\n      \"create\": true\r\n    },\r\n    \"MigrationColumns\": [\r\n      {\r\n        \"sourceColumn\": {\r\n          \"name\": \"id\"\r\n        },\r\n        \"destinationColumn\": {\r\n          \"name\": \"id\",\r\n          \"options\": {\r\n            \"primary_key\": true,\r\n            \"type_cast\": \"uuid\"\r\n          }\r\n        }\r\n      },\r\n      {\r\n        \"sourceColumn\": {\r\n          \"name\": \"interest\"\r\n        },\r\n        \"destinationColumn\": {\r\n          \"name\": \"interest\",\r\n          \"options\": {\r\n            \"type_cast\": \"varchar\",\r\n            \"length\": 30,\r\n            \"nullable\": false\r\n          }\r\n        }\r\n      },\r\n      {\r\n        \"sourceColumn\": {\r\n          \"name\": \"FIFD\"\r\n        },\r\n        \"destinationColumn\": {\r\n          \"name\": \"FIFD\",\r\n          \"options\": {\r\n            \"type_cast\": \"varchar\",\r\n            \"length\": 30,\r\n            \"nullable\": false\r\n          }\r\n        }\r\n      },\r\n      {\r\n        \"sourceColumn\": {\r\n          \"name\": \"comission\"\r\n        },\r\n        \"destinationColumn\": {\r\n          \"name\": \"comission\",\r\n          \"options\": {\r\n            \"type_cast\": \"varchar\",\r\n            \"length\": 30,\r\n            \"nullable\": false\r\n          }\r\n        }\r\n      }\r\n    ]\r\n  }\r\n```\r\n\r\nSee also the list of [contributors](https://github.com/MadeByMads/mad-migration/graphs/contributors) who participated in this project.\r\n\r\n\r\n## Contributing\r\nWe are open to  new ideas, additions. If you have any we would be happy to recieve and diccuss.\r\n\r\n- [Contributing](https://github.com/AliyevH/mad-migration/blob/master/mdCONTRIBUTING.md)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faliyevh%2Fmad-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faliyevh%2Fmad-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faliyevh%2Fmad-migration/lists"}