{"id":22521549,"url":"https://github.com/r13i/sync-elasticsearch-mysql","last_synced_at":"2025-08-03T20:31:11.436Z","repository":{"id":45063238,"uuid":"310561594","full_name":"r13i/sync-elasticsearch-mysql","owner":"r13i","description":"Using Logstash to synchronize an Elasticsearch index with MySQL data","archived":false,"fork":false,"pushed_at":"2022-01-11T16:58:15.000Z","size":83,"stargazers_count":94,"open_issues_count":1,"forks_count":60,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-24T12:15:17.052Z","etag":null,"topics":["docker","elasticsearch","kibana","logstash","mysql"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/r13i.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-06T10:14:21.000Z","updated_at":"2024-03-18T02:49:17.000Z","dependencies_parsed_at":"2022-09-02T05:40:35.629Z","dependency_job_id":null,"html_url":"https://github.com/r13i/sync-elasticsearch-mysql","commit_stats":null,"previous_names":["r13i/sync-elasticsearch-mysql"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Fsync-elasticsearch-mysql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Fsync-elasticsearch-mysql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Fsync-elasticsearch-mysql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Fsync-elasticsearch-mysql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/r13i","download_url":"https://codeload.github.com/r13i/sync-elasticsearch-mysql/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228562317,"owners_count":17937234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","elasticsearch","kibana","logstash","mysql"],"created_at":"2024-12-07T05:11:55.341Z","updated_at":"2024-12-07T05:11:56.072Z","avatar_url":"https://github.com/r13i.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# sync-elasticsearch-mysql\nUsing Logstash to synchronize an Elasticsearch index with MySQL data\n\n\u003e This project is described in details in this article: [How to synchronize Elasticsearch with MySQL](https://towardsdatascience.com/how-to-synchronize-elasticsearch-with-mysql-ed32fc57b339)\n\n## Introduction\n\nThis project is a working example demonstrating how to use Logstash to link Elasticsearch to a MySQL database in order to:\n- Build an Elasticsearch index from scratch\n- Continuously monitor changes on the database records and replicate any of those changes to Elasticsearch (`create`, `update`, `delete`)\n\nIt uses:\n- MySQL as the main database of a given business architecture (version 8.0)\n- Elasticsearch as a text search engine (version 7.9.3)\n- Logstash as a connector or data pipe from MySQL to Elasticsearch (version 7.9.3)\n- Kibana for monitoring, data visualization, and debuging tool (version 7.9.3)\n\n![Architecture of this project](./docs/sync-elasticsearch-mysql.png)\n\nThis repo is a valid prototype and works as it is, however it is not suitable for a production environment. Please refer to the official documentation of each of the above technologies for instructions on how to go live in your production environment.\n\n## Deployment\nOn your development/local environment, run the following commands on a terminal:\n\n\u003e Note: Make sure to install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/)\n\n```bash\n# Clone this project and cd into it\ngit clone https://github.com/redouane-dev/sync-elasticsearch-mysql.git \u0026\u0026 cd sync-elasticsearch-mysql\n\n# Start the whole architecture\ndocker-compose up # add -d for detached mode\n\n# To keep an eye on the logs\ndocker-compose logs -f --tail 111 \u003cservice-name\u003e\n```\n\nTo start services separately or in a different order, you can run:\n```bash\ndocker-compose up -d mysql\ndocker-compose up -d elasticsearch kibana\ndocker-compose up logstash\n```\n\n## Testing\nPlease refer to the above article for testing steps.\n\n## Resources\n- Inspiration by [How to keep Elasticsearch synchronized with a relational database using Logstash and JDBC](https://www.elastic.co/blog/how-to-keep-elasticsearch-synchronized-with-a-relational-database-using-logstash). However the article does not deal with indexing from scratch and deleted records.\n- Data used for this project is available in the Kaggle dataset [Goodreads-books](https://www.kaggle.com/jealousleopard/goodreadsbooks)\n- [Logstash JDBC input plugin](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html)\n- [Logstash Mutate filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html)\n- [Logstash Elasticsearch output plugin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr13i%2Fsync-elasticsearch-mysql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fr13i%2Fsync-elasticsearch-mysql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr13i%2Fsync-elasticsearch-mysql/lists"}