{"id":21843670,"url":"https://github.com/tasmidur/sink-data-mysql-to-mongodb","last_synced_at":"2025-04-14T12:06:49.683Z","repository":{"id":46535988,"uuid":"502362408","full_name":"tasmidur/sink-data-mysql-to-mongodb","owner":"tasmidur","description":"sink-data-mysql-to-mongodb","archived":false,"fork":false,"pushed_at":"2022-06-22T17:11:42.000Z","size":2429,"stargazers_count":3,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-14T12:06:40.738Z","etag":null,"topics":["mongodb","mysql-binlog-events","mysql-database","nodejs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tasmidur.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-11T13:54:53.000Z","updated_at":"2022-07-19T03:53:33.000Z","dependencies_parsed_at":"2022-09-07T05:30:37.491Z","dependency_job_id":null,"html_url":"https://github.com/tasmidur/sink-data-mysql-to-mongodb","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasmidur%2Fsink-data-mysql-to-mongodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasmidur%2Fsink-data-mysql-to-mongodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasmidur%2Fsink-data-mysql-to-mongodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasmidur%2Fsink-data-mysql-to-mongodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tasmidur","download_url":"https://codeload.github.com/tasmidur/sink-data-mysql-to-mongodb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248877983,"owners_count":21176243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mongodb","mysql-binlog-events","mysql-database","nodejs"],"created_at":"2024-11-27T22:16:03.217Z","updated_at":"2025-04-14T12:06:49.654Z","avatar_url":"https://github.com/tasmidur.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sink data mysql to mongodb\nThe application is developed by inspired by Kafka debezium. Here sync MySQL data to mongodb database based on mysql event. To capture the mysql event, I use a npm package https://github.com/p80-ch/mysql-event-emitter.\n## Requirements\n\n* Node 12\n* Git\n* MySql 5.7 to 8.0.0\n* MongoDB\n\n## Common setup\n\n### Mysql Server\nEnable binary log replication in `/etc/mysql/my.cnf`\n```\n[mysqld]\nserver-id        = 1\nlog_bin          = /var/log/mysql/mysql-bin\nlog_bin_index    = /var/log/mysql/mysql-bin.index\nbinlog-format    = row\n```\n\n### Mysql User\nGive your user the rights to read binary logs\n```sql\nGRANT REPLICATION SLAVE, REPLICATION CLIENT, SELECT ON *.* TO '[USER]'@'[HOST]'\n\n```\n### NodeJs Application\n\nClone the repo and install the dependencies.\n\n```bash\ngit clone https://github.com/tasmidur/sink-data-mysql-to-mongodb.git\ncd sink-data-mysql-to-mongodb\n```\n\n```bash\nnpm install\n```\n\n## Project Setup Steps:\n\nStep 1: Copy env.example to .env\n\nStep 2: Edit Source Connector Config For Mysql\n```\nMYSQL_HOST=\"localhost\"\nMYSQL_PORT=\"3306\"\nMYSQL_USER=\"root\"\nMYSQL_PASSWORD=\"root\"\n\n```\nStep 3: Edit Sink Connector Config For MongDB\n```\nMONGO_DB_URL=\"mongodb://localhost:27017\"\nMONGO_DB_COLLECTION_NAME=\"collection_name\"\n```\n## Project Config\nEvent Config file configuration\n\nStep 1: Go to config directory and edit eventConfig.js File\n\n```\nconst eventConfig = {\n    /**\n     * Default: false\n     * Pass true to only emit binlog events that occur after ZongJi's instantiation.\n     * Must be used in start() method for effect.\n     */\n    startAtEnd: false,\n    /**\n     * Databases and tables to include (Only for row events).\n     * Use database names as the key and pass an array of table names or true (for the entire database)\n     * Example: { 'my_database': ['allow_table', 'another_table'], 'another_db': true }\n     */\n    includeSchema: {\n        cdc: true\n    },\n    /**\n     * Object describing which databases and tables to exclude (Same format as includeSchema)\n     * Example: { 'other_db': ['disallowed_table'], 'ex_db': true }\n     */\n    excludeSchema: {\n        mysql: true,\n        sys: true\n    }\n}\n```\n\n## Use Docker\nYou can also run this app as a Docker container:\n\nStep 1: Go to docker directory and run\n\n``` bash\ncd docker\ndocker-compose up\n```\n## Run Project\n\n``` bash\nnpm run start\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftasmidur%2Fsink-data-mysql-to-mongodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftasmidur%2Fsink-data-mysql-to-mongodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftasmidur%2Fsink-data-mysql-to-mongodb/lists"}